2024-03-18T14:51:52.442 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-18T14:51:52.442 DEBUG:teuthology.run:Teuthology command: teuthology --name pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi --owner scheduled_pdonnell@teuthology --archive /home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608866 --verbose --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/fsync-tester}} -- /home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608866/orig.config.yaml 2024-03-18T14:51:52.502 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-18T14:51:52.587 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608866 branch: wip-batrick-testing-20240314.183243 description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/fsync-tester}} email: pdonnell@redhat.com first_in_suite: false job_id: '7608866' kernel: client: branch: testing kdb: true sha1: distro ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-batrick-testing-20240314.183243 ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 5 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_bal_replicate_threshold: 1 mds_export_ephemeral_random_max: 0.1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - mons down, quorum - out of quorum - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - MON_DOWN sha1: 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af subvols: create: 2 subvol_options: --size 25000000000 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - min: 3 name: mds_cache.dir_update - min: 3 name: mds_cache.dir_update_receipt - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount dry_run: true install: ceph: flavor: default sha1: 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=crc - wsync syntax: v2 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-batrick-testing-20240314.183243 sha1: 4e19f704afd9957ea657ab858f9ecf76401fef6f owner: scheduled_pdonnell@teuthology priority: 75 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 4770 sha1: 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af sleep_before_teardown: 0 subset: 498/512 suite: fs suite_branch: wip-batrick-testing-20240314.183243 suite_path: /home/teuthworker/src/github.com_batrick_ceph_4e19f704afd9957ea657ab858f9ecf76401fef6f/qa suite_relpath: qa suite_repo: https://github.com/batrick/ceph suite_sha1: 4e19f704afd9957ea657ab858f9ecf76401fef6f syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi019.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMi36RYgvG4AraWlJgExqmzfxDhqmHf8HPuvRZse4mMaEql3bjgV0xyq7VJJUIa319dm8A6Liu5CDhgt73TBJrk= smithi033.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNr0GJOBuYiogmoLTbOrsspKiHUxCKK/NaMD1IBqLwCKxp+zCGAt81olsCTp/rSf0yudJmvPJRMj/tOTHKA4Sic= smithi104.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOJddSLJuC3b7IoFH5QS0IDfmdWmvBFATT9soZcSavH79wPqV8wo5g+uJXY8/ezG7NBZcD2rDiLcirZcvp4ZZZ4= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate false - ceph fs subvolumegroup pin cephfs qa random 0.10 - check-counter: {} - exec: mon.a: - ceph mgr module enable snap_schedule - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - workunit: clients: all: - suites/fsync-tester.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-18_14:19:46 tube: smithi user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-18T14:51:52.587 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_batrick_ceph_4e19f704afd9957ea657ab858f9ecf76401fef6f/qa; will attempt to use it 2024-03-18T14:51:52.588 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_batrick_ceph_4e19f704afd9957ea657ab858f9ecf76401fef6f/qa/tasks 2024-03-18T14:51:52.588 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-18T14:51:52.589 INFO:teuthology.task.internal:Checking packages... 2024-03-18T14:51:52.611 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af' 2024-03-18T14:51:52.611 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-18T14:51:52.611 INFO:teuthology.packaging:ref: None 2024-03-18T14:51:52.611 INFO:teuthology.packaging:tag: None 2024-03-18T14:51:52.611 INFO:teuthology.packaging:branch: wip-batrick-testing-20240314.183243 2024-03-18T14:51:52.611 INFO:teuthology.packaging:sha1: 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-18T14:51:52.612 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-batrick-testing-20240314.183243 2024-03-18T14:51:52.911 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2162.g3d78f5ba 2024-03-18T14:51:52.913 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-18T14:51:52.922 INFO:teuthology.task.internal:no buildpackages task found 2024-03-18T14:51:52.922 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-18T14:51:52.965 INFO:teuthology.task.internal:Saving configuration 2024-03-18T14:51:52.984 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-18T14:51:53.021 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-18T14:51:53.045 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi019.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608866', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-18 14:47:06.901319', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMi36RYgvG4AraWlJgExqmzfxDhqmHf8HPuvRZse4mMaEql3bjgV0xyq7VJJUIa319dm8A6Liu5CDhgt73TBJrk='} 2024-03-18T14:51:53.070 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi033.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608866', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-18 14:47:06.902738', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNr0GJOBuYiogmoLTbOrsspKiHUxCKK/NaMD1IBqLwCKxp+zCGAt81olsCTp/rSf0yudJmvPJRMj/tOTHKA4Sic='} 2024-03-18T14:51:53.092 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi104.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608866', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-18 14:47:06.899618', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOJddSLJuC3b7IoFH5QS0IDfmdWmvBFATT9soZcSavH79wPqV8wo5g+uJXY8/ezG7NBZcD2rDiLcirZcvp4ZZZ4='} 2024-03-18T14:51:53.092 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-18T14:51:53.102 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-03-18T14:51:53.102 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-03-18T14:51:53.102 INFO:teuthology.task.internal:roles: ubuntu@smithi104.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-03-18T14:51:53.102 INFO:teuthology.run_tasks:Running task console_log... 2024-03-18T14:51:53.230 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f23940d5160>, signals=[15]) 2024-03-18T14:51:53.230 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-18T14:51:53.241 INFO:teuthology.task.internal:Opening connections... 2024-03-18T14:51:53.241 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi019.front.sepia.ceph.com 2024-03-18T14:51:53.243 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi019.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T14:51:53.349 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi033.front.sepia.ceph.com 2024-03-18T14:51:53.350 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi033.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T14:51:53.453 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi104.front.sepia.ceph.com 2024-03-18T14:51:53.454 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T14:51:53.544 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-18T14:51:53.553 DEBUG:teuthology.orchestra.run.smithi019:> uname -m 2024-03-18T14:51:53.571 INFO:teuthology.orchestra.run.smithi019.stdout:x86_64 2024-03-18T14:51:53.571 DEBUG:teuthology.orchestra.run.smithi019:> cat /etc/os-release 2024-03-18T14:51:53.629 INFO:teuthology.orchestra.run.smithi019.stdout:NAME="CentOS Stream" 2024-03-18T14:51:53.629 INFO:teuthology.orchestra.run.smithi019.stdout:VERSION="9" 2024-03-18T14:51:53.629 INFO:teuthology.orchestra.run.smithi019.stdout:ID="centos" 2024-03-18T14:51:53.629 INFO:teuthology.orchestra.run.smithi019.stdout:ID_LIKE="rhel fedora" 2024-03-18T14:51:53.629 INFO:teuthology.orchestra.run.smithi019.stdout:VERSION_ID="9" 2024-03-18T14:51:53.629 INFO:teuthology.orchestra.run.smithi019.stdout:PLATFORM_ID="platform:el9" 2024-03-18T14:51:53.629 INFO:teuthology.orchestra.run.smithi019.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-18T14:51:53.629 INFO:teuthology.orchestra.run.smithi019.stdout:ANSI_COLOR="0;31" 2024-03-18T14:51:53.630 INFO:teuthology.orchestra.run.smithi019.stdout:LOGO="fedora-logo-icon" 2024-03-18T14:51:53.630 INFO:teuthology.orchestra.run.smithi019.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-18T14:51:53.630 INFO:teuthology.orchestra.run.smithi019.stdout:HOME_URL="https://centos.org/" 2024-03-18T14:51:53.630 INFO:teuthology.orchestra.run.smithi019.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-18T14:51:53.630 INFO:teuthology.orchestra.run.smithi019.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-18T14:51:53.630 INFO:teuthology.orchestra.run.smithi019.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-18T14:51:53.630 INFO:teuthology.lock.ops:Updating smithi019.front.sepia.ceph.com on lock server 2024-03-18T14:51:53.655 DEBUG:teuthology.orchestra.run.smithi033:> uname -m 2024-03-18T14:51:53.674 INFO:teuthology.orchestra.run.smithi033.stdout:x86_64 2024-03-18T14:51:53.674 DEBUG:teuthology.orchestra.run.smithi033:> cat /etc/os-release 2024-03-18T14:51:53.731 INFO:teuthology.orchestra.run.smithi033.stdout:NAME="CentOS Stream" 2024-03-18T14:51:53.732 INFO:teuthology.orchestra.run.smithi033.stdout:VERSION="9" 2024-03-18T14:51:53.732 INFO:teuthology.orchestra.run.smithi033.stdout:ID="centos" 2024-03-18T14:51:53.732 INFO:teuthology.orchestra.run.smithi033.stdout:ID_LIKE="rhel fedora" 2024-03-18T14:51:53.732 INFO:teuthology.orchestra.run.smithi033.stdout:VERSION_ID="9" 2024-03-18T14:51:53.732 INFO:teuthology.orchestra.run.smithi033.stdout:PLATFORM_ID="platform:el9" 2024-03-18T14:51:53.732 INFO:teuthology.orchestra.run.smithi033.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-18T14:51:53.732 INFO:teuthology.orchestra.run.smithi033.stdout:ANSI_COLOR="0;31" 2024-03-18T14:51:53.732 INFO:teuthology.orchestra.run.smithi033.stdout:LOGO="fedora-logo-icon" 2024-03-18T14:51:53.732 INFO:teuthology.orchestra.run.smithi033.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-18T14:51:53.732 INFO:teuthology.orchestra.run.smithi033.stdout:HOME_URL="https://centos.org/" 2024-03-18T14:51:53.732 INFO:teuthology.orchestra.run.smithi033.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-18T14:51:53.732 INFO:teuthology.orchestra.run.smithi033.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-18T14:51:53.732 INFO:teuthology.orchestra.run.smithi033.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-18T14:51:53.733 INFO:teuthology.lock.ops:Updating smithi033.front.sepia.ceph.com on lock server 2024-03-18T14:51:53.755 DEBUG:teuthology.orchestra.run.smithi104:> uname -m 2024-03-18T14:51:53.773 INFO:teuthology.orchestra.run.smithi104.stdout:x86_64 2024-03-18T14:51:53.773 DEBUG:teuthology.orchestra.run.smithi104:> cat /etc/os-release 2024-03-18T14:51:53.830 INFO:teuthology.orchestra.run.smithi104.stdout:NAME="CentOS Stream" 2024-03-18T14:51:53.830 INFO:teuthology.orchestra.run.smithi104.stdout:VERSION="9" 2024-03-18T14:51:53.830 INFO:teuthology.orchestra.run.smithi104.stdout:ID="centos" 2024-03-18T14:51:53.830 INFO:teuthology.orchestra.run.smithi104.stdout:ID_LIKE="rhel fedora" 2024-03-18T14:51:53.831 INFO:teuthology.orchestra.run.smithi104.stdout:VERSION_ID="9" 2024-03-18T14:51:53.831 INFO:teuthology.orchestra.run.smithi104.stdout:PLATFORM_ID="platform:el9" 2024-03-18T14:51:53.831 INFO:teuthology.orchestra.run.smithi104.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-18T14:51:53.831 INFO:teuthology.orchestra.run.smithi104.stdout:ANSI_COLOR="0;31" 2024-03-18T14:51:53.831 INFO:teuthology.orchestra.run.smithi104.stdout:LOGO="fedora-logo-icon" 2024-03-18T14:51:53.831 INFO:teuthology.orchestra.run.smithi104.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-18T14:51:53.831 INFO:teuthology.orchestra.run.smithi104.stdout:HOME_URL="https://centos.org/" 2024-03-18T14:51:53.831 INFO:teuthology.orchestra.run.smithi104.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-18T14:51:53.831 INFO:teuthology.orchestra.run.smithi104.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-18T14:51:53.831 INFO:teuthology.orchestra.run.smithi104.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-18T14:51:53.831 INFO:teuthology.lock.ops:Updating smithi104.front.sepia.ceph.com on lock server 2024-03-18T14:51:53.855 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-18T14:51:53.865 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-18T14:51:53.882 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-18T14:51:53.882 DEBUG:teuthology.orchestra.run.smithi019:> test '!' -e /home/ubuntu/cephtest 2024-03-18T14:51:53.885 DEBUG:teuthology.orchestra.run.smithi033:> test '!' -e /home/ubuntu/cephtest 2024-03-18T14:51:53.887 DEBUG:teuthology.orchestra.run.smithi104:> test '!' -e /home/ubuntu/cephtest 2024-03-18T14:51:53.903 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-18T14:51:53.913 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-18T14:51:53.913 DEBUG:teuthology.orchestra.run.smithi019:> test -z $(ls -A /var/lib/ceph) 2024-03-18T14:51:53.942 DEBUG:teuthology.orchestra.run.smithi033:> test -z $(ls -A /var/lib/ceph) 2024-03-18T14:51:53.944 DEBUG:teuthology.orchestra.run.smithi104:> test -z $(ls -A /var/lib/ceph) 2024-03-18T14:51:53.971 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-18T14:51:54.049 INFO:teuthology.run_tasks:Running task kernel... 2024-03-18T14:51:54.065 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'} 2024-03-18T14:51:54.065 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}} 2024-03-18T14:51:54.065 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}}, timeout 300 2024-03-18T14:51:54.066 DEBUG:teuthology.orchestra.run.smithi019:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-18T14:51:54.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T14:51:54.083 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2024-03-18T14:51:54.287 DEBUG:teuthology.task.kernel:sha1 for client.0 is 3845b7ac715a3ff198582638d199a90e30e254da 2024-03-18T14:51:54.287 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "3845b7ac715a3ff198582638d199a90e30e254da"... 2024-03-18T14:51:54.287 DEBUG:teuthology.orchestra.run.smithi019:> uname -r 2024-03-18T14:51:54.304 INFO:teuthology.orchestra.run.smithi019.stdout:5.14.0-361.el9.x86_64 2024-03-18T14:51:54.305 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-361.el9.x86_64 vs 3845b7ac715a3ff198582638d199a90e30e254da 2024-03-18T14:51:54.305 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-03-18T14:51:54.305 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum upgrade -y linux-firmware 2024-03-18T14:51:56.614 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-03-18T14:51:58.205 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS-9 - BaseOS 4.4 MB/s | 2.0 MB 00:00 2024-03-18T14:51:59.221 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.0 MB 00:00 2024-03-18T14:52:01.230 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS Stream 9 - AppStream 21 MB/s | 19 MB 00:00 2024-03-18T14:52:14.247 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS Stream 9 - CRB 671 kB/s | 6.1 MB 00:09 2024-03-18T14:52:15.839 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS Stream 9 - Extras packages 38 kB/s | 16 kB 00:00 2024-03-18T14:52:17.157 INFO:teuthology.orchestra.run.smithi019.stdout:Extra Packages for Enterprise Linux 17 MB/s | 21 MB 00:01 2024-03-18T14:52:22.307 INFO:teuthology.orchestra.run.smithi019.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-03-18T14:52:24.760 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-03-18T14:52:24.775 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-18T14:52:24.775 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2024-03-18T14:52:24.775 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-18T14:52:24.776 INFO:teuthology.orchestra.run.smithi019.stdout:Upgrading: 2024-03-18T14:52:24.776 INFO:teuthology.orchestra.run.smithi019.stdout: linux-firmware noarch 20240219-143.el9 CentOS-BaseOS 375 M 2024-03-18T14:52:24.776 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-18T14:52:24.776 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-03-18T14:52:24.776 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-18T14:52:24.776 INFO:teuthology.orchestra.run.smithi019.stdout:Upgrade 1 Package 2024-03-18T14:52:24.776 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-18T14:52:24.776 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 375 M 2024-03-18T14:52:24.776 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-03-18T14:52:33.033 INFO:teuthology.orchestra.run.smithi019.stdout:linux-firmware-20240219-143.el9.noarch.rpm 45 MB/s | 375 MB 00:08 2024-03-18T14:52:33.034 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-03-18T14:52:33.034 INFO:teuthology.orchestra.run.smithi019.stdout:Total 45 MB/s | 375 MB 00:08 2024-03-18T14:52:33.034 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-03-18T14:52:33.131 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-03-18T14:52:33.131 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-03-18T14:52:35.624 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-03-18T14:52:35.624 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-03-18T14:52:42.009 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-03-18T14:52:42.225 INFO:teuthology.orchestra.run.smithi019.stdout: Upgrading : linux-firmware-20240219-143.el9.noarch 1/2 2024-03-18T14:52:42.319 INFO:teuthology.orchestra.run.smithi019.stdout: Cleanup : linux-firmware-20230814-139.el9.noarch 2/2 2024-03-18T14:52:43.616 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: linux-firmware-20230814-139.el9.noarch 2/2 2024-03-18T14:52:43.616 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : linux-firmware-20240219-143.el9.noarch 1/2 2024-03-18T14:52:44.000 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : linux-firmware-20230814-139.el9.noarch 2/2 2024-03-18T14:52:44.000 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-18T14:52:44.000 INFO:teuthology.orchestra.run.smithi019.stdout:Upgraded: 2024-03-18T14:52:44.000 INFO:teuthology.orchestra.run.smithi019.stdout: linux-firmware-20240219-143.el9.noarch 2024-03-18T14:52:44.000 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-18T14:52:44.000 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-03-18T14:52:44.310 INFO:teuthology.task.kernel:Downloading kernel 3845b7ac715a3ff198582638d199a90e30e254da on client.0... 2024-03-18T14:52:44.311 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3845b7ac715a3ff198582638d199a90e30e254da 2024-03-18T14:52:44.532 INFO:teuthology.task.kernel:fetching, builder baseurl is https://2.chacra.ceph.com/r/kernel/testing/3845b7ac715a3ff198582638d199a90e30e254da/centos/9/flavors/default/x86_64/ 2024-03-18T14:52:44.532 DEBUG:teuthology.orchestra.run.smithi019:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.8.0_rc7_g3845b7ac715a-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://2.chacra.ceph.com/r/kernel/testing/3845b7ac715a3ff198582638d199a90e30e254da/centos/9/flavors/default/x86_64/ --input-file=- 2024-03-18T14:52:44.535 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi019.front.sepia.ceph.com to complete... 2024-03-18T14:52:56.153 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18 14:52:56 URL:https://2.chacra.ceph.com/r/kernel/testing/3845b7ac715a3ff198582638d199a90e30e254da/centos/9/flavors/default/x86_64/kernel-6.8.0_rc7_g3845b7ac715a-1.x86_64.rpm [336075563/336075563] -> "/tmp/kernel.x86_64.rpm" [1] 2024-03-18T14:52:56.153 INFO:teuthology.orchestra.run.smithi019.stderr:FINISHED --2024-03-18 14:52:56-- 2024-03-18T14:52:56.154 INFO:teuthology.orchestra.run.smithi019.stderr:Total wall clock time: 12s 2024-03-18T14:52:56.154 INFO:teuthology.orchestra.run.smithi019.stderr:Downloaded: 1 files, 321M in 11s (28.4 MB/s) 2024-03-18T14:52:56.156 INFO:teuthology.task.kernel:Installing kernel 3845b7ac715a3ff198582638d199a90e30e254da on client.0... 2024-03-18T14:52:56.157 DEBUG:teuthology.orchestra.run.smithi019:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-03-18T14:52:58.192 INFO:teuthology.orchestra.run.smithi019.stdout:Verifying... ######################################## 2024-03-18T14:52:58.251 INFO:teuthology.orchestra.run.smithi019.stdout:Preparing... ######################################## 2024-03-18T14:52:58.336 INFO:teuthology.orchestra.run.smithi019.stdout:Updating / installing... 2024-03-18T14:53:06.182 INFO:teuthology.orchestra.run.smithi019.stdout:kernel-6.8.0_rc7_g3845b7ac715a-1 ######################################## 2024-03-18T14:53:22.339 INFO:teuthology.orchestra.run.smithi019.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-18T14:53:45.546 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi019.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-03-18T14:53:45.547 DEBUG:teuthology.orchestra.run.smithi019:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-03-18T14:53:45.620 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a 2024-03-18T14:53:45.621 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/System.map 2024-03-18T14:53:45.621 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/config 2024-03-18T14:53:45.621 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel 2024-03-18T14:53:45.621 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch 2024-03-18T14:53:45.621 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86 2024-03-18T14:53:45.621 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto 2024-03-18T14:53:45.621 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-03-18T14:53:45.621 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-03-18T14:53:45.621 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-03-18T14:53:45.621 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-03-18T14:53:45.621 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-03-18T14:53:45.621 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-03-18T14:53:45.621 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-03-18T14:53:45.622 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/crc32c-intel.ko 2024-03-18T14:53:45.622 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-03-18T14:53:45.622 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-03-18T14:53:45.622 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-03-18T14:53:45.622 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-03-18T14:53:45.622 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/serpent-avx2.ko 2024-03-18T14:53:45.622 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-03-18T14:53:45.622 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-03-18T14:53:45.622 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-03-18T14:53:45.622 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-03-18T14:53:45.622 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-03-18T14:53:45.622 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-03-18T14:53:45.622 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-03-18T14:53:45.622 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/kernel 2024-03-18T14:53:45.623 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/kernel/cpu 2024-03-18T14:53:45.623 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/kernel/cpu/mce 2024-03-18T14:53:45.623 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-03-18T14:53:45.623 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/kvm 2024-03-18T14:53:45.623 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/kvm/kvm-amd.ko 2024-03-18T14:53:45.623 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/kvm/kvm-intel.ko 2024-03-18T14:53:45.623 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/kvm/kvm.ko 2024-03-18T14:53:45.623 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto 2024-03-18T14:53:45.623 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/ansi_cprng.ko 2024-03-18T14:53:45.623 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/anubis.ko 2024-03-18T14:53:45.623 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/arc4.ko 2024-03-18T14:53:45.623 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/async_tx 2024-03-18T14:53:45.623 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/async_tx/async_memcpy.ko 2024-03-18T14:53:45.623 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/async_tx/async_pq.ko 2024-03-18T14:53:45.624 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/async_tx/async_raid6_recov.ko 2024-03-18T14:53:45.624 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/async_tx/async_tx.ko 2024-03-18T14:53:45.624 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/async_tx/async_xor.ko 2024-03-18T14:53:45.624 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/async_tx/raid6test.ko 2024-03-18T14:53:45.624 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/authenc.ko 2024-03-18T14:53:45.624 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/authencesn.ko 2024-03-18T14:53:45.624 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/blake2b_generic.ko 2024-03-18T14:53:45.624 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/blowfish_common.ko 2024-03-18T14:53:45.624 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/blowfish_generic.ko 2024-03-18T14:53:45.624 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/camellia_generic.ko 2024-03-18T14:53:45.624 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/cast5_generic.ko 2024-03-18T14:53:45.624 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/cast6_generic.ko 2024-03-18T14:53:45.624 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/cast_common.ko 2024-03-18T14:53:45.624 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/ccm.ko 2024-03-18T14:53:45.625 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/cmac.ko 2024-03-18T14:53:45.625 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/crc32_generic.ko 2024-03-18T14:53:45.625 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/crypto_engine.ko 2024-03-18T14:53:45.625 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/crypto_user.ko 2024-03-18T14:53:45.625 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/cts.ko 2024-03-18T14:53:45.625 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/des_generic.ko 2024-03-18T14:53:45.625 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/dh_generic.ko 2024-03-18T14:53:45.625 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/echainiv.ko 2024-03-18T14:53:45.625 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/essiv.ko 2024-03-18T14:53:45.625 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/fcrypt.ko 2024-03-18T14:53:45.625 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/khazad.ko 2024-03-18T14:53:45.625 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/lz4.ko 2024-03-18T14:53:45.625 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/lz4hc.ko 2024-03-18T14:53:45.626 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/md4.ko 2024-03-18T14:53:45.626 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/michael_mic.ko 2024-03-18T14:53:45.626 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/pcbc.ko 2024-03-18T14:53:45.626 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/pcrypt.ko 2024-03-18T14:53:45.626 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/rmd160.ko 2024-03-18T14:53:45.626 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/seed.ko 2024-03-18T14:53:45.626 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/serpent_generic.ko 2024-03-18T14:53:45.626 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/tcrypt.ko 2024-03-18T14:53:45.626 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/tea.ko 2024-03-18T14:53:45.626 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/twofish_common.ko 2024-03-18T14:53:45.626 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/twofish_generic.ko 2024-03-18T14:53:45.626 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/vmac.ko 2024-03-18T14:53:45.626 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/wp512.ko 2024-03-18T14:53:45.626 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/xcbc.ko 2024-03-18T14:53:45.627 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/xor.ko 2024-03-18T14:53:45.627 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/xxhash_generic.ko 2024-03-18T14:53:45.627 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers 2024-03-18T14:53:45.627 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/acpi 2024-03-18T14:53:45.627 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/acpi/acpi_ipmi.ko 2024-03-18T14:53:45.627 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/acpi/acpi_pad.ko 2024-03-18T14:53:45.627 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/acpi/custom_method.ko 2024-03-18T14:53:45.627 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/acpi/ec_sys.ko 2024-03-18T14:53:45.627 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/acpi/platform_profile.ko 2024-03-18T14:53:45.627 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/acpi/sbs.ko 2024-03-18T14:53:45.627 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/acpi/sbshc.ko 2024-03-18T14:53:45.627 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/acpi/video.ko 2024-03-18T14:53:45.627 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata 2024-03-18T14:53:45.627 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/acard-ahci.ko 2024-03-18T14:53:45.628 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/ahci_platform.ko 2024-03-18T14:53:45.628 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/ata_generic.ko 2024-03-18T14:53:45.628 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/libahci_platform.ko 2024-03-18T14:53:45.628 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_acpi.ko 2024-03-18T14:53:45.628 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_ali.ko 2024-03-18T14:53:45.628 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_amd.ko 2024-03-18T14:53:45.628 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_artop.ko 2024-03-18T14:53:45.628 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_atiixp.ko 2024-03-18T14:53:45.628 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_atp867x.ko 2024-03-18T14:53:45.628 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_cmd640.ko 2024-03-18T14:53:45.628 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_cmd64x.ko 2024-03-18T14:53:45.628 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_cypress.ko 2024-03-18T14:53:45.628 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_efar.ko 2024-03-18T14:53:45.628 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_hpt366.ko 2024-03-18T14:53:45.629 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_hpt37x.ko 2024-03-18T14:53:45.629 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_hpt3x2n.ko 2024-03-18T14:53:45.629 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_hpt3x3.ko 2024-03-18T14:53:45.629 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_it8213.ko 2024-03-18T14:53:45.629 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_it821x.ko 2024-03-18T14:53:45.629 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_jmicron.ko 2024-03-18T14:53:45.629 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_marvell.ko 2024-03-18T14:53:45.629 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_mpiix.ko 2024-03-18T14:53:45.629 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_netcell.ko 2024-03-18T14:53:45.629 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_ninja32.ko 2024-03-18T14:53:45.629 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_ns87410.ko 2024-03-18T14:53:45.629 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_ns87415.ko 2024-03-18T14:53:45.629 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_oldpiix.ko 2024-03-18T14:53:45.630 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_opti.ko 2024-03-18T14:53:45.630 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_optidma.ko 2024-03-18T14:53:45.630 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_pcmcia.ko 2024-03-18T14:53:45.630 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_pdc2027x.ko 2024-03-18T14:53:45.630 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-03-18T14:53:45.630 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_piccolo.ko 2024-03-18T14:53:45.630 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_rdc.ko 2024-03-18T14:53:45.630 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_sch.ko 2024-03-18T14:53:45.630 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_serverworks.ko 2024-03-18T14:53:45.630 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_sil680.ko 2024-03-18T14:53:45.630 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_sis.ko 2024-03-18T14:53:45.630 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_sl82c105.ko 2024-03-18T14:53:45.630 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_triflex.ko 2024-03-18T14:53:45.631 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_via.ko 2024-03-18T14:53:45.631 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pdc_adma.ko 2024-03-18T14:53:45.631 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/sata_inic162x.ko 2024-03-18T14:53:45.631 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/sata_mv.ko 2024-03-18T14:53:45.631 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/sata_nv.ko 2024-03-18T14:53:45.631 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/sata_promise.ko 2024-03-18T14:53:45.631 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/sata_qstor.ko 2024-03-18T14:53:45.631 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/sata_sil.ko 2024-03-18T14:53:45.631 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/sata_sil24.ko 2024-03-18T14:53:45.631 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/sata_sis.ko 2024-03-18T14:53:45.631 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/sata_svw.ko 2024-03-18T14:53:45.631 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/sata_sx4.ko 2024-03-18T14:53:45.631 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/sata_uli.ko 2024-03-18T14:53:45.631 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/sata_via.ko 2024-03-18T14:53:45.632 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/sata_vsc.ko 2024-03-18T14:53:45.632 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/auxdisplay 2024-03-18T14:53:45.632 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/auxdisplay/ks0108.ko 2024-03-18T14:53:45.632 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/base 2024-03-18T14:53:45.632 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/base/regmap 2024-03-18T14:53:45.632 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/base/regmap/regmap-i2c.ko 2024-03-18T14:53:45.632 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/base/regmap/regmap-mmio.ko 2024-03-18T14:53:45.632 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/bcma 2024-03-18T14:53:45.632 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/bcma/bcma.ko 2024-03-18T14:53:45.632 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block 2024-03-18T14:53:45.632 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/aoe 2024-03-18T14:53:45.632 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/aoe/aoe.ko 2024-03-18T14:53:45.632 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/brd.ko 2024-03-18T14:53:45.633 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/drbd 2024-03-18T14:53:45.633 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/drbd/drbd.ko 2024-03-18T14:53:45.633 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/floppy.ko 2024-03-18T14:53:45.633 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/loop.ko 2024-03-18T14:53:45.633 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/mtip32xx 2024-03-18T14:53:45.633 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-03-18T14:53:45.633 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/nbd.ko 2024-03-18T14:53:45.633 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/null_blk 2024-03-18T14:53:45.633 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/null_blk/null_blk.ko 2024-03-18T14:53:45.633 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/pktcdvd.ko 2024-03-18T14:53:45.633 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/rbd.ko 2024-03-18T14:53:45.633 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/virtio_blk.ko 2024-03-18T14:53:45.633 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/xen-blkback 2024-03-18T14:53:45.633 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-03-18T14:53:45.634 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/xen-blkfront.ko 2024-03-18T14:53:45.634 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/zram 2024-03-18T14:53:45.634 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/zram/zram.ko 2024-03-18T14:53:45.634 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char 2024-03-18T14:53:45.634 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/hangcheck-timer.ko 2024-03-18T14:53:45.634 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/hw_random 2024-03-18T14:53:45.634 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/hw_random/amd-rng.ko 2024-03-18T14:53:45.634 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/hw_random/intel-rng.ko 2024-03-18T14:53:45.634 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-03-18T14:53:45.634 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/hw_random/via-rng.ko 2024-03-18T14:53:45.634 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/hw_random/virtio-rng.ko 2024-03-18T14:53:45.634 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/ipmi 2024-03-18T14:53:45.634 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-03-18T14:53:45.634 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-03-18T14:53:45.635 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-03-18T14:53:45.635 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/ipmi/ipmi_si.ko 2024-03-18T14:53:45.635 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-03-18T14:53:45.635 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-03-18T14:53:45.635 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/lp.ko 2024-03-18T14:53:45.635 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/mwave 2024-03-18T14:53:45.635 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/mwave/mwave.ko 2024-03-18T14:53:45.635 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/ppdev.ko 2024-03-18T14:53:45.635 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/tlclk.ko 2024-03-18T14:53:45.635 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/tpm 2024-03-18T14:53:45.635 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/tpm/tpm.ko 2024-03-18T14:53:45.635 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/tpm/tpm_atmel.ko 2024-03-18T14:53:45.635 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/tpm/tpm_infineon.ko 2024-03-18T14:53:45.635 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/tpm/tpm_nsc.ko 2024-03-18T14:53:45.636 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/tpm/tpm_tis.ko 2024-03-18T14:53:45.636 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-03-18T14:53:45.636 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/uv_mmtimer.ko 2024-03-18T14:53:45.636 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/virtio_console.ko 2024-03-18T14:53:45.636 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/cpufreq 2024-03-18T14:53:45.636 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-03-18T14:53:45.636 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-03-18T14:53:45.636 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/cpufreq/p4-clockmod.ko 2024-03-18T14:53:45.636 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-03-18T14:53:45.636 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/cpufreq/powernow-k8.ko 2024-03-18T14:53:45.636 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/cpufreq/speedstep-lib.ko 2024-03-18T14:53:45.636 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/crypto 2024-03-18T14:53:45.636 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/crypto/ccp 2024-03-18T14:53:45.636 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-03-18T14:53:45.636 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/crypto/ccp/ccp.ko 2024-03-18T14:53:45.637 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/crypto/intel 2024-03-18T14:53:45.637 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/crypto/intel/qat 2024-03-18T14:53:45.637 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/crypto/intel/qat/qat_common 2024-03-18T14:53:45.637 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-03-18T14:53:45.637 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-03-18T14:53:45.637 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-03-18T14:53:45.637 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/crypto/padlock-aes.ko 2024-03-18T14:53:45.637 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/crypto/padlock-sha.ko 2024-03-18T14:53:45.637 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/crypto/virtio 2024-03-18T14:53:45.637 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-03-18T14:53:45.637 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/dca 2024-03-18T14:53:45.637 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/dca/dca.ko 2024-03-18T14:53:45.637 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/devfreq 2024-03-18T14:53:45.637 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-03-18T14:53:45.637 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/dma 2024-03-18T14:53:45.638 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/dma/dw 2024-03-18T14:53:45.638 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/dma/dw/dw_dmac.ko 2024-03-18T14:53:45.638 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/dma/ioat 2024-03-18T14:53:45.638 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/dma/ioat/ioatdma.ko 2024-03-18T14:53:45.638 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/edac 2024-03-18T14:53:45.638 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/edac/amd64_edac.ko 2024-03-18T14:53:45.638 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/edac/e752x_edac.ko 2024-03-18T14:53:45.638 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/edac/edac_mce_amd.ko 2024-03-18T14:53:45.638 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/edac/i3000_edac.ko 2024-03-18T14:53:45.638 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/edac/i3200_edac.ko 2024-03-18T14:53:45.638 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/edac/i5100_edac.ko 2024-03-18T14:53:45.638 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/edac/i5400_edac.ko 2024-03-18T14:53:45.638 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/edac/i7300_edac.ko 2024-03-18T14:53:45.639 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/edac/i7core_edac.ko 2024-03-18T14:53:45.639 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/edac/i82975x_edac.ko 2024-03-18T14:53:45.639 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/edac/ie31200_edac.ko 2024-03-18T14:53:45.639 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/edac/sb_edac.ko 2024-03-18T14:53:45.639 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/edac/x38_edac.ko 2024-03-18T14:53:45.639 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/firmware 2024-03-18T14:53:45.639 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/firmware/edd.ko 2024-03-18T14:53:45.639 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/firmware/iscsi_ibft.ko 2024-03-18T14:53:45.639 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/gpio 2024-03-18T14:53:45.639 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/gpio/gpio-ich.ko 2024-03-18T14:53:45.639 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/gpio/gpio-viperboard.ko 2024-03-18T14:53:45.639 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid 2024-03-18T14:53:45.639 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-appleir.ko 2024-03-18T14:53:45.639 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-aureal.ko 2024-03-18T14:53:45.640 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-axff.ko 2024-03-18T14:53:45.640 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-dr.ko 2024-03-18T14:53:45.640 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-elecom.ko 2024-03-18T14:53:45.640 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-elo.ko 2024-03-18T14:53:45.640 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-emsff.ko 2024-03-18T14:53:45.640 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-gaff.ko 2024-03-18T14:53:45.640 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-gt683r.ko 2024-03-18T14:53:45.640 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-gyration.ko 2024-03-18T14:53:45.640 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-holtek-kbd.ko 2024-03-18T14:53:45.640 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-holtek-mouse.ko 2024-03-18T14:53:45.640 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-holtekff.ko 2024-03-18T14:53:45.640 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-hyperv.ko 2024-03-18T14:53:45.640 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-icade.ko 2024-03-18T14:53:45.641 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-keytouch.ko 2024-03-18T14:53:45.641 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-kye.ko 2024-03-18T14:53:45.641 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-lcpower.ko 2024-03-18T14:53:45.641 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-led.ko 2024-03-18T14:53:45.641 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-lenovo.ko 2024-03-18T14:53:45.641 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-logitech-dj.ko 2024-03-18T14:53:45.641 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-03-18T14:53:45.641 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-multitouch.ko 2024-03-18T14:53:45.641 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-ortek.ko 2024-03-18T14:53:45.641 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-penmount.ko 2024-03-18T14:53:45.641 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-petalynx.ko 2024-03-18T14:53:45.641 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-picolcd.ko 2024-03-18T14:53:45.641 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-pl.ko 2024-03-18T14:53:45.641 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-plantronics.ko 2024-03-18T14:53:45.642 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-primax.ko 2024-03-18T14:53:45.642 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-rmi.ko 2024-03-18T14:53:45.642 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-roccat-arvo.ko 2024-03-18T14:53:45.642 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-roccat-common.ko 2024-03-18T14:53:45.642 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-roccat-isku.ko 2024-03-18T14:53:45.642 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-roccat-kone.ko 2024-03-18T14:53:45.642 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-03-18T14:53:45.642 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-roccat-konepure.ko 2024-03-18T14:53:45.642 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-03-18T14:53:45.642 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-roccat-lua.ko 2024-03-18T14:53:45.642 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-roccat-pyra.ko 2024-03-18T14:53:45.642 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-roccat-ryos.ko 2024-03-18T14:53:45.642 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-roccat-savu.ko 2024-03-18T14:53:45.642 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-roccat.ko 2024-03-18T14:53:45.643 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-saitek.ko 2024-03-18T14:53:45.643 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-samsung.ko 2024-03-18T14:53:45.643 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-sensor-hub.ko 2024-03-18T14:53:45.643 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-sjoy.ko 2024-03-18T14:53:45.643 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-sony.ko 2024-03-18T14:53:45.643 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-speedlink.ko 2024-03-18T14:53:45.643 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-steelseries.ko 2024-03-18T14:53:45.643 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-sunplus.ko 2024-03-18T14:53:45.643 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-thrustmaster.ko 2024-03-18T14:53:45.643 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-tivo.ko 2024-03-18T14:53:45.643 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-tmff.ko 2024-03-18T14:53:45.643 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-topseed.ko 2024-03-18T14:53:45.643 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-twinhan.ko 2024-03-18T14:53:45.643 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-uclogic.ko 2024-03-18T14:53:45.643 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-waltop.ko 2024-03-18T14:53:45.643 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-wiimote.ko 2024-03-18T14:53:45.644 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-xinmo.ko 2024-03-18T14:53:45.644 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-zpff.ko 2024-03-18T14:53:45.644 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-zydacron.ko 2024-03-18T14:53:45.644 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/uhid.ko 2024-03-18T14:53:45.644 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/wacom.ko 2024-03-18T14:53:45.644 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hv 2024-03-18T14:53:45.644 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hv/hv_balloon.ko 2024-03-18T14:53:45.644 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hv/hv_utils.ko 2024-03-18T14:53:45.644 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hv/hv_vmbus.ko 2024-03-18T14:53:45.644 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon 2024-03-18T14:53:45.644 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/abituguru.ko 2024-03-18T14:53:45.644 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/abituguru3.ko 2024-03-18T14:53:45.644 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/acpi_power_meter.ko 2024-03-18T14:53:45.644 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/ad7414.ko 2024-03-18T14:53:45.644 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/ad7418.ko 2024-03-18T14:53:45.644 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/adc128d818.ko 2024-03-18T14:53:45.645 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/adm1025.ko 2024-03-18T14:53:45.645 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/adm1026.ko 2024-03-18T14:53:45.645 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/adm1029.ko 2024-03-18T14:53:45.645 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/adm1031.ko 2024-03-18T14:53:45.645 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/adm9240.ko 2024-03-18T14:53:45.645 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/ads7828.ko 2024-03-18T14:53:45.645 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/adt7410.ko 2024-03-18T14:53:45.645 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/adt7411.ko 2024-03-18T14:53:45.645 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/adt7462.ko 2024-03-18T14:53:45.645 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/adt7470.ko 2024-03-18T14:53:45.645 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/adt7475.ko 2024-03-18T14:53:45.645 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/adt7x10.ko 2024-03-18T14:53:45.645 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/amc6821.ko 2024-03-18T14:53:45.645 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/applesmc.ko 2024-03-18T14:53:45.645 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/asb100.ko 2024-03-18T14:53:45.645 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/asc7621.ko 2024-03-18T14:53:45.646 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/asus_atk0110.ko 2024-03-18T14:53:45.646 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/atxp1.ko 2024-03-18T14:53:45.646 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/coretemp.ko 2024-03-18T14:53:45.646 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-03-18T14:53:45.646 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/dme1737.ko 2024-03-18T14:53:45.646 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/ds1621.ko 2024-03-18T14:53:45.646 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/ds620.ko 2024-03-18T14:53:45.646 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/emc1403.ko 2024-03-18T14:53:45.646 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/emc6w201.ko 2024-03-18T14:53:45.646 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/f71805f.ko 2024-03-18T14:53:45.646 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/f71882fg.ko 2024-03-18T14:53:45.646 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/f75375s.ko 2024-03-18T14:53:45.646 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/fam15h_power.ko 2024-03-18T14:53:45.647 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/fschmd.ko 2024-03-18T14:53:45.647 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/g760a.ko 2024-03-18T14:53:45.647 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/g762.ko 2024-03-18T14:53:45.647 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/gl518sm.ko 2024-03-18T14:53:45.647 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/gl520sm.ko 2024-03-18T14:53:45.647 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/hwmon-vid.ko 2024-03-18T14:53:45.647 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/i5500_temp.ko 2024-03-18T14:53:45.647 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/i5k_amb.ko 2024-03-18T14:53:45.647 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/ibmaem.ko 2024-03-18T14:53:45.647 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/ibmpex.ko 2024-03-18T14:53:45.647 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/ina209.ko 2024-03-18T14:53:45.647 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/ina2xx.ko 2024-03-18T14:53:45.647 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/it87.ko 2024-03-18T14:53:45.647 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/k10temp.ko 2024-03-18T14:53:45.647 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/k8temp.ko 2024-03-18T14:53:45.647 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/lineage-pem.ko 2024-03-18T14:53:45.648 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/lm63.ko 2024-03-18T14:53:45.648 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/lm73.ko 2024-03-18T14:53:45.648 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/lm75.ko 2024-03-18T14:53:45.648 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/lm77.ko 2024-03-18T14:53:45.648 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/lm78.ko 2024-03-18T14:53:45.648 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/lm80.ko 2024-03-18T14:53:45.648 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/lm83.ko 2024-03-18T14:53:45.648 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/lm85.ko 2024-03-18T14:53:45.648 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/lm87.ko 2024-03-18T14:53:45.648 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/lm90.ko 2024-03-18T14:53:45.648 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/lm92.ko 2024-03-18T14:53:45.648 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/lm93.ko 2024-03-18T14:53:45.648 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/lm95234.ko 2024-03-18T14:53:45.648 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/lm95241.ko 2024-03-18T14:53:45.648 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/lm95245.ko 2024-03-18T14:53:45.648 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/ltc2945.ko 2024-03-18T14:53:45.649 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/ltc4151.ko 2024-03-18T14:53:45.649 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/ltc4215.ko 2024-03-18T14:53:45.649 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/ltc4222.ko 2024-03-18T14:53:45.649 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/ltc4245.ko 2024-03-18T14:53:45.649 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/ltc4260.ko 2024-03-18T14:53:45.649 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/ltc4261.ko 2024-03-18T14:53:45.649 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/max16065.ko 2024-03-18T14:53:45.649 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/max1619.ko 2024-03-18T14:53:45.649 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/max1668.ko 2024-03-18T14:53:45.649 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/max197.ko 2024-03-18T14:53:45.649 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/max6639.ko 2024-03-18T14:53:45.649 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/max6650.ko 2024-03-18T14:53:45.649 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/max6697.ko 2024-03-18T14:53:45.649 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/mcp3021.ko 2024-03-18T14:53:45.649 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/nct6683.ko 2024-03-18T14:53:45.649 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/nct6775-core.ko 2024-03-18T14:53:45.650 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/nct6775.ko 2024-03-18T14:53:45.650 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/nct7802.ko 2024-03-18T14:53:45.650 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/pc87360.ko 2024-03-18T14:53:45.650 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/pc87427.ko 2024-03-18T14:53:45.650 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/pcf8591.ko 2024-03-18T14:53:45.650 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/pmbus 2024-03-18T14:53:45.650 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-03-18T14:53:45.650 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-03-18T14:53:45.650 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-03-18T14:53:45.650 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/pmbus/max16064.ko 2024-03-18T14:53:45.650 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/pmbus/max34440.ko 2024-03-18T14:53:45.650 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/pmbus/max8688.ko 2024-03-18T14:53:45.650 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-03-18T14:53:45.651 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-03-18T14:53:45.651 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-03-18T14:53:45.651 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-03-18T14:53:45.651 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-03-18T14:53:45.651 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-03-18T14:53:45.651 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/powr1220.ko 2024-03-18T14:53:45.651 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/sch5627.ko 2024-03-18T14:53:45.651 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/sch5636.ko 2024-03-18T14:53:45.651 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/sch56xx-common.ko 2024-03-18T14:53:45.651 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/sht15.ko 2024-03-18T14:53:45.651 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/sht21.ko 2024-03-18T14:53:45.651 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/shtc1.ko 2024-03-18T14:53:45.651 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/sis5595.ko 2024-03-18T14:53:45.651 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/smsc47b397.ko 2024-03-18T14:53:45.652 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/smsc47m1.ko 2024-03-18T14:53:45.652 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/smsc47m192.ko 2024-03-18T14:53:45.652 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/thmc50.ko 2024-03-18T14:53:45.652 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/tmp102.ko 2024-03-18T14:53:45.652 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/tmp103.ko 2024-03-18T14:53:45.652 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/tmp401.ko 2024-03-18T14:53:45.652 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/tmp421.ko 2024-03-18T14:53:45.652 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/via-cputemp.ko 2024-03-18T14:53:45.652 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/via686a.ko 2024-03-18T14:53:45.652 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/vt1211.ko 2024-03-18T14:53:45.652 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/vt8231.ko 2024-03-18T14:53:45.652 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/w83627ehf.ko 2024-03-18T14:53:45.652 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/w83627hf.ko 2024-03-18T14:53:45.652 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/w83781d.ko 2024-03-18T14:53:45.652 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/w83791d.ko 2024-03-18T14:53:45.653 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/w83792d.ko 2024-03-18T14:53:45.653 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/w83793.ko 2024-03-18T14:53:45.653 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/w83795.ko 2024-03-18T14:53:45.653 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/w83l785ts.ko 2024-03-18T14:53:45.653 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/w83l786ng.ko 2024-03-18T14:53:45.653 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c 2024-03-18T14:53:45.653 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/algos 2024-03-18T14:53:45.653 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-03-18T14:53:45.653 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-03-18T14:53:45.653 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses 2024-03-18T14:53:45.653 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-03-18T14:53:45.653 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-03-18T14:53:45.653 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-03-18T14:53:45.653 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-03-18T14:53:45.653 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-03-18T14:53:45.654 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-03-18T14:53:45.654 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-03-18T14:53:45.654 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-03-18T14:53:45.654 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-i801.ko 2024-03-18T14:53:45.654 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-isch.ko 2024-03-18T14:53:45.654 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-03-18T14:53:45.654 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-03-18T14:53:45.654 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-03-18T14:53:45.654 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-parport.ko 2024-03-18T14:53:45.654 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-03-18T14:53:45.654 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-03-18T14:53:45.654 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-03-18T14:53:45.654 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-03-18T14:53:45.654 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-03-18T14:53:45.655 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-03-18T14:53:45.655 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-via.ko 2024-03-18T14:53:45.655 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-03-18T14:53:45.655 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-03-18T14:53:45.655 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/i2c-dev.ko 2024-03-18T14:53:45.655 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/i2c-mux.ko 2024-03-18T14:53:45.655 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-03-18T14:53:45.655 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/i2c-smbus.ko 2024-03-18T14:53:45.655 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/i2c-stub.ko 2024-03-18T14:53:45.655 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband 2024-03-18T14:53:45.655 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/core 2024-03-18T14:53:45.655 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/core/ib_cm.ko 2024-03-18T14:53:45.655 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/core/ib_core.ko 2024-03-18T14:53:45.655 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/core/ib_umad.ko 2024-03-18T14:53:45.656 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-03-18T14:53:45.656 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/core/iw_cm.ko 2024-03-18T14:53:45.656 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/core/rdma_cm.ko 2024-03-18T14:53:45.656 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-03-18T14:53:45.656 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/hw 2024-03-18T14:53:45.656 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/hw/cxgb4 2024-03-18T14:53:45.656 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-03-18T14:53:45.656 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/hw/mlx4 2024-03-18T14:53:45.656 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-03-18T14:53:45.656 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/hw/mlx5 2024-03-18T14:53:45.656 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-03-18T14:53:45.656 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/hw/mthca 2024-03-18T14:53:45.656 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-03-18T14:53:45.656 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/ulp 2024-03-18T14:53:45.656 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/ulp/ipoib 2024-03-18T14:53:45.657 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-03-18T14:53:45.657 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/ulp/iser 2024-03-18T14:53:45.657 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-03-18T14:53:45.657 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/ulp/isert 2024-03-18T14:53:45.657 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-03-18T14:53:45.657 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/ulp/srp 2024-03-18T14:53:45.657 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-03-18T14:53:45.657 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/ulp/srpt 2024-03-18T14:53:45.657 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-03-18T14:53:45.657 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input 2024-03-18T14:53:45.657 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/gameport 2024-03-18T14:53:45.657 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-03-18T14:53:45.657 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/gameport/fm801-gp.ko 2024-03-18T14:53:45.657 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/gameport/gameport.ko 2024-03-18T14:53:45.658 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/joydev.ko 2024-03-18T14:53:45.658 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/keyboard 2024-03-18T14:53:45.658 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/keyboard/gpio_keys.ko 2024-03-18T14:53:45.658 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/mouse 2024-03-18T14:53:45.658 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/mouse/appletouch.ko 2024-03-18T14:53:45.658 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/mouse/bcm5974.ko 2024-03-18T14:53:45.658 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/mouse/cyapatp.ko 2024-03-18T14:53:45.658 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/mouse/elan_i2c.ko 2024-03-18T14:53:45.658 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/mouse/sermouse.ko 2024-03-18T14:53:45.658 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-03-18T14:53:45.658 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/mouse/synaptics_usb.ko 2024-03-18T14:53:45.658 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/mouse/vsxxxaa.ko 2024-03-18T14:53:45.658 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/rmi4 2024-03-18T14:53:45.658 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/rmi4/rmi_core.ko 2024-03-18T14:53:45.659 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/serio 2024-03-18T14:53:45.659 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/serio/altera_ps2.ko 2024-03-18T14:53:45.659 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/serio/arc_ps2.ko 2024-03-18T14:53:45.659 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-03-18T14:53:45.659 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/serio/serio_raw.ko 2024-03-18T14:53:45.659 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/sparse-keymap.ko 2024-03-18T14:53:45.659 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/leds 2024-03-18T14:53:45.659 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/leds/leds-blinkm.ko 2024-03-18T14:53:45.659 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/leds/leds-lm3530.ko 2024-03-18T14:53:45.659 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/leds/leds-lp3944.ko 2024-03-18T14:53:45.659 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/leds/leds-ss4200.ko 2024-03-18T14:53:45.659 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/leds/trigger 2024-03-18T14:53:45.659 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-03-18T14:53:45.659 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-03-18T14:53:45.659 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-03-18T14:53:45.659 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-03-18T14:53:45.660 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-03-18T14:53:45.660 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-03-18T14:53:45.660 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-03-18T14:53:45.660 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-03-18T14:53:45.660 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-03-18T14:53:45.660 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md 2024-03-18T14:53:45.660 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/bcache 2024-03-18T14:53:45.660 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/bcache/bcache.ko 2024-03-18T14:53:45.660 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/dm-bio-prison.ko 2024-03-18T14:53:45.660 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/dm-cache-smq.ko 2024-03-18T14:53:45.660 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/dm-cache.ko 2024-03-18T14:53:45.660 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/dm-crypt.ko 2024-03-18T14:53:45.660 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/dm-delay.ko 2024-03-18T14:53:45.660 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/dm-flakey.ko 2024-03-18T14:53:45.661 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/dm-log-userspace.ko 2024-03-18T14:53:45.661 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/dm-multipath.ko 2024-03-18T14:53:45.661 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/dm-queue-length.ko 2024-03-18T14:53:45.661 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/dm-raid.ko 2024-03-18T14:53:45.661 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/dm-round-robin.ko 2024-03-18T14:53:45.661 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/dm-service-time.ko 2024-03-18T14:53:45.661 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/dm-switch.ko 2024-03-18T14:53:45.661 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/dm-thin-pool.ko 2024-03-18T14:53:45.661 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/dm-verity.ko 2024-03-18T14:53:45.661 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/persistent-data 2024-03-18T14:53:45.661 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-03-18T14:53:45.661 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/raid0.ko 2024-03-18T14:53:45.661 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/raid1.ko 2024-03-18T14:53:45.661 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/raid10.ko 2024-03-18T14:53:45.661 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/raid456.ko 2024-03-18T14:53:45.662 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/message 2024-03-18T14:53:45.662 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/message/fusion 2024-03-18T14:53:45.662 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/message/fusion/mptbase.ko 2024-03-18T14:53:45.662 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/message/fusion/mptctl.ko 2024-03-18T14:53:45.662 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/message/fusion/mptfc.ko 2024-03-18T14:53:45.662 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/message/fusion/mptlan.ko 2024-03-18T14:53:45.662 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/message/fusion/mptsas.ko 2024-03-18T14:53:45.662 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/message/fusion/mptscsih.ko 2024-03-18T14:53:45.662 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/message/fusion/mptspi.ko 2024-03-18T14:53:45.662 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/mfd 2024-03-18T14:53:45.662 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/mfd/lpc_ich.ko 2024-03-18T14:53:45.662 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/mfd/lpc_sch.ko 2024-03-18T14:53:45.662 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/mfd/mfd-core.ko 2024-03-18T14:53:45.663 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/mfd/sm501.ko 2024-03-18T14:53:45.663 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/mfd/viperboard.ko 2024-03-18T14:53:45.663 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/mfd/vx855.ko 2024-03-18T14:53:45.663 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/mfd/wl1273-core.ko 2024-03-18T14:53:45.663 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc 2024-03-18T14:53:45.663 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/altera-stapl 2024-03-18T14:53:45.663 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-03-18T14:53:45.663 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/apds9802als.ko 2024-03-18T14:53:45.663 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/apds990x.ko 2024-03-18T14:53:45.663 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/bh1770glc.ko 2024-03-18T14:53:45.663 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/cb710 2024-03-18T14:53:45.663 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/cb710/cb710.ko 2024-03-18T14:53:45.663 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/eeprom 2024-03-18T14:53:45.663 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/eeprom/at24.ko 2024-03-18T14:53:45.663 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-03-18T14:53:45.664 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/eeprom/max6875.ko 2024-03-18T14:53:45.664 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/enclosure.ko 2024-03-18T14:53:45.664 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/hpilo.ko 2024-03-18T14:53:45.664 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/isl29003.ko 2024-03-18T14:53:45.664 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/isl29020.ko 2024-03-18T14:53:45.664 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/lis3lv02d 2024-03-18T14:53:45.664 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-03-18T14:53:45.664 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-03-18T14:53:45.664 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/mei 2024-03-18T14:53:45.664 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/mei/mei-me.ko 2024-03-18T14:53:45.664 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/mei/mei-txe.ko 2024-03-18T14:53:45.664 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/mei/mei.ko 2024-03-18T14:53:45.664 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/sgi-gru 2024-03-18T14:53:45.664 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/sgi-gru/gru.ko 2024-03-18T14:53:45.664 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/sgi-xp 2024-03-18T14:53:45.665 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/sgi-xp/xp.ko 2024-03-18T14:53:45.665 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/sgi-xp/xpc.ko 2024-03-18T14:53:45.665 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-03-18T14:53:45.665 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/tifm_7xx1.ko 2024-03-18T14:53:45.665 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/tifm_core.ko 2024-03-18T14:53:45.665 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/tsl2550.ko 2024-03-18T14:53:45.665 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/vmw_balloon.ko 2024-03-18T14:53:45.665 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/vmw_vmci 2024-03-18T14:53:45.665 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-03-18T14:53:45.665 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net 2024-03-18T14:53:45.665 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/bonding 2024-03-18T14:53:45.665 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/bonding/bonding.ko 2024-03-18T14:53:45.665 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/dsa 2024-03-18T14:53:45.665 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/dsa/b53 2024-03-18T14:53:45.665 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/dsa/b53/b53_common.ko 2024-03-18T14:53:45.665 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/dsa/bcm-sf2.ko 2024-03-18T14:53:45.666 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/dsa/mv88e6060.ko 2024-03-18T14:53:45.666 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/dsa/mv88e6xxx 2024-03-18T14:53:45.666 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-03-18T14:53:45.666 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/dummy.ko 2024-03-18T14:53:45.666 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/eql.ko 2024-03-18T14:53:45.666 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet 2024-03-18T14:53:45.666 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/3com 2024-03-18T14:53:45.666 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-03-18T14:53:45.666 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-03-18T14:53:45.666 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-03-18T14:53:45.666 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-03-18T14:53:45.666 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/8390 2024-03-18T14:53:45.666 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/8390/8390.ko 2024-03-18T14:53:45.666 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-03-18T14:53:45.667 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-03-18T14:53:45.667 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-03-18T14:53:45.667 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/adaptec 2024-03-18T14:53:45.667 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-03-18T14:53:45.667 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/agere 2024-03-18T14:53:45.667 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/agere/et131x.ko 2024-03-18T14:53:45.667 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/alteon 2024-03-18T14:53:45.667 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-03-18T14:53:45.667 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/altera 2024-03-18T14:53:45.667 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-03-18T14:53:45.667 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/amd 2024-03-18T14:53:45.667 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-03-18T14:53:45.667 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-03-18T14:53:45.667 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-03-18T14:53:45.668 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/atheros 2024-03-18T14:53:45.668 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/atheros/alx 2024-03-18T14:53:45.668 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-03-18T14:53:45.668 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/atheros/atl1c 2024-03-18T14:53:45.668 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-03-18T14:53:45.668 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/atheros/atl1e 2024-03-18T14:53:45.668 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-03-18T14:53:45.668 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/atheros/atlx 2024-03-18T14:53:45.668 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-03-18T14:53:45.668 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-03-18T14:53:45.668 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/broadcom 2024-03-18T14:53:45.668 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-03-18T14:53:45.668 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-03-18T14:53:45.668 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-03-18T14:53:45.668 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-03-18T14:53:45.669 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-03-18T14:53:45.669 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/broadcom/genet 2024-03-18T14:53:45.669 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-03-18T14:53:45.669 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-03-18T14:53:45.669 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/brocade 2024-03-18T14:53:45.669 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/brocade/bna 2024-03-18T14:53:45.669 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-03-18T14:53:45.669 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/chelsio 2024-03-18T14:53:45.669 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/chelsio/cxgb 2024-03-18T14:53:45.669 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-03-18T14:53:45.669 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-03-18T14:53:45.669 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-03-18T14:53:45.669 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-03-18T14:53:45.669 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-03-18T14:53:45.670 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-03-18T14:53:45.670 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-03-18T14:53:45.670 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-03-18T14:53:45.670 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-03-18T14:53:45.670 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/cisco 2024-03-18T14:53:45.670 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/cisco/enic 2024-03-18T14:53:45.670 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-03-18T14:53:45.670 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/dec 2024-03-18T14:53:45.670 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/dec/tulip 2024-03-18T14:53:45.670 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-03-18T14:53:45.670 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-03-18T14:53:45.670 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-03-18T14:53:45.670 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-03-18T14:53:45.670 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-03-18T14:53:45.671 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-03-18T14:53:45.671 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/dlink 2024-03-18T14:53:45.671 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-03-18T14:53:45.671 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-03-18T14:53:45.671 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/dnet.ko 2024-03-18T14:53:45.671 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/emulex 2024-03-18T14:53:45.671 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/emulex/benet 2024-03-18T14:53:45.671 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-03-18T14:53:45.671 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/ethoc.ko 2024-03-18T14:53:45.671 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/fealnx.ko 2024-03-18T14:53:45.671 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel 2024-03-18T14:53:45.671 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/e100.ko 2024-03-18T14:53:45.671 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/e1000 2024-03-18T14:53:45.671 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-03-18T14:53:45.671 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/e1000e 2024-03-18T14:53:45.671 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-03-18T14:53:45.672 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/fm10k 2024-03-18T14:53:45.672 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-03-18T14:53:45.672 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/i40e 2024-03-18T14:53:45.672 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-03-18T14:53:45.672 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/iavf 2024-03-18T14:53:45.672 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-03-18T14:53:45.672 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/igb 2024-03-18T14:53:45.672 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-03-18T14:53:45.672 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/igbvf 2024-03-18T14:53:45.672 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-03-18T14:53:45.672 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/ixgbe 2024-03-18T14:53:45.672 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-03-18T14:53:45.672 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/ixgbevf 2024-03-18T14:53:45.672 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-03-18T14:53:45.672 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/jme.ko 2024-03-18T14:53:45.673 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/marvell 2024-03-18T14:53:45.673 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-03-18T14:53:45.673 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/marvell/skge.ko 2024-03-18T14:53:45.673 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-03-18T14:53:45.673 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/mellanox 2024-03-18T14:53:45.673 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/mellanox/mlx4 2024-03-18T14:53:45.673 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-03-18T14:53:45.673 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-03-18T14:53:45.673 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/mellanox/mlx5 2024-03-18T14:53:45.673 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-03-18T14:53:45.673 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-03-18T14:53:45.673 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/micrel 2024-03-18T14:53:45.673 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-03-18T14:53:45.673 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/myricom 2024-03-18T14:53:45.673 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/myricom/myri10ge 2024-03-18T14:53:45.673 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-03-18T14:53:45.674 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/natsemi 2024-03-18T14:53:45.674 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-03-18T14:53:45.674 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-03-18T14:53:45.674 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/neterion 2024-03-18T14:53:45.674 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-03-18T14:53:45.674 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/nvidia 2024-03-18T14:53:45.674 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-03-18T14:53:45.674 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/packetengines 2024-03-18T14:53:45.674 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-03-18T14:53:45.674 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-03-18T14:53:45.674 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/qlogic 2024-03-18T14:53:45.674 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/qlogic/netxen 2024-03-18T14:53:45.674 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-03-18T14:53:45.675 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-03-18T14:53:45.675 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-03-18T14:53:45.675 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-03-18T14:53:45.675 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/rdc 2024-03-18T14:53:45.675 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-03-18T14:53:45.675 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/realtek 2024-03-18T14:53:45.675 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-03-18T14:53:45.675 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-03-18T14:53:45.675 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/realtek/atp.ko 2024-03-18T14:53:45.675 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-03-18T14:53:45.675 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/rocker 2024-03-18T14:53:45.675 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-03-18T14:53:45.675 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/sfc 2024-03-18T14:53:45.675 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-03-18T14:53:45.676 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/silan 2024-03-18T14:53:45.676 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-03-18T14:53:45.676 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/sis 2024-03-18T14:53:45.676 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/sis/sis190.ko 2024-03-18T14:53:45.676 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/sis/sis900.ko 2024-03-18T14:53:45.676 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/smsc 2024-03-18T14:53:45.676 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-03-18T14:53:45.676 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-03-18T14:53:45.676 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-03-18T14:53:45.676 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-03-18T14:53:45.676 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/stmicro 2024-03-18T14:53:45.676 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/stmicro/stmmac 2024-03-18T14:53:45.676 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-03-18T14:53:45.676 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-03-18T14:53:45.676 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/sun 2024-03-18T14:53:45.677 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/sun/cassini.ko 2024-03-18T14:53:45.677 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/sun/niu.ko 2024-03-18T14:53:45.677 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/sun/sungem.ko 2024-03-18T14:53:45.677 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-03-18T14:53:45.677 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/tehuti 2024-03-18T14:53:45.677 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-03-18T14:53:45.677 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/ti 2024-03-18T14:53:45.677 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/ti/tlan.ko 2024-03-18T14:53:45.677 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/via 2024-03-18T14:53:45.677 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-03-18T14:53:45.677 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-03-18T14:53:45.677 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/wiznet 2024-03-18T14:53:45.677 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-03-18T14:53:45.677 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-03-18T14:53:45.677 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/xircom 2024-03-18T14:53:45.678 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-03-18T14:53:45.678 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/geneve.ko 2024-03-18T14:53:45.678 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/hyperv 2024-03-18T14:53:45.678 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-03-18T14:53:45.678 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ieee802154 2024-03-18T14:53:45.678 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ieee802154/fakelb.ko 2024-03-18T14:53:45.678 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ifb.ko 2024-03-18T14:53:45.678 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ipvlan 2024-03-18T14:53:45.678 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ipvlan/ipvlan.ko 2024-03-18T14:53:45.678 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/macvlan.ko 2024-03-18T14:53:45.678 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/macvtap.ko 2024-03-18T14:53:45.678 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/mdio 2024-03-18T14:53:45.678 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/mdio.ko 2024-03-18T14:53:45.678 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-03-18T14:53:45.679 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-03-18T14:53:45.679 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/mdio/mdio-regmap.ko 2024-03-18T14:53:45.679 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/mii.ko 2024-03-18T14:53:45.679 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/net_failover.ko 2024-03-18T14:53:45.679 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/netconsole.ko 2024-03-18T14:53:45.679 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/nlmon.ko 2024-03-18T14:53:45.679 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/pcs 2024-03-18T14:53:45.679 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/pcs/pcs-lynx.ko 2024-03-18T14:53:45.679 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-03-18T14:53:45.679 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy 2024-03-18T14:53:45.679 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/amd.ko 2024-03-18T14:53:45.679 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-03-18T14:53:45.679 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-03-18T14:53:45.679 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/bcm7xxx.ko 2024-03-18T14:53:45.679 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/bcm87xx.ko 2024-03-18T14:53:45.679 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/broadcom.ko 2024-03-18T14:53:45.680 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/cicada.ko 2024-03-18T14:53:45.680 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/davicom.ko 2024-03-18T14:53:45.680 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/dp83640.ko 2024-03-18T14:53:45.680 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/et1011c.ko 2024-03-18T14:53:45.680 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/icplus.ko 2024-03-18T14:53:45.680 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/lxt.ko 2024-03-18T14:53:45.680 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/marvell.ko 2024-03-18T14:53:45.680 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/micrel.ko 2024-03-18T14:53:45.680 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/national.ko 2024-03-18T14:53:45.680 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/phylink.ko 2024-03-18T14:53:45.680 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/qsemi.ko 2024-03-18T14:53:45.680 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/realtek.ko 2024-03-18T14:53:45.680 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/smsc.ko 2024-03-18T14:53:45.680 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/ste10Xp.ko 2024-03-18T14:53:45.680 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/vitesse.ko 2024-03-18T14:53:45.681 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ppp 2024-03-18T14:53:45.681 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ppp/bsd_comp.ko 2024-03-18T14:53:45.681 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ppp/ppp_async.ko 2024-03-18T14:53:45.681 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ppp/ppp_deflate.ko 2024-03-18T14:53:45.681 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ppp/ppp_generic.ko 2024-03-18T14:53:45.681 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ppp/ppp_mppe.ko 2024-03-18T14:53:45.681 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ppp/ppp_synctty.ko 2024-03-18T14:53:45.681 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ppp/pppoe.ko 2024-03-18T14:53:45.681 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ppp/pppox.ko 2024-03-18T14:53:45.681 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ppp/pptp.ko 2024-03-18T14:53:45.681 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/slip 2024-03-18T14:53:45.681 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/slip/slhc.ko 2024-03-18T14:53:45.681 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/slip/slip.ko 2024-03-18T14:53:45.681 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/sungem_phy.ko 2024-03-18T14:53:45.682 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/tap.ko 2024-03-18T14:53:45.682 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/team 2024-03-18T14:53:45.682 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/team/team.ko 2024-03-18T14:53:45.682 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/team/team_mode_activebackup.ko 2024-03-18T14:53:45.682 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/team/team_mode_broadcast.ko 2024-03-18T14:53:45.682 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-03-18T14:53:45.682 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/team/team_mode_random.ko 2024-03-18T14:53:45.682 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-03-18T14:53:45.682 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/tun.ko 2024-03-18T14:53:45.682 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/veth.ko 2024-03-18T14:53:45.682 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/virtio_net.ko 2024-03-18T14:53:45.682 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/vmxnet3 2024-03-18T14:53:45.682 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-03-18T14:53:45.683 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/vxlan 2024-03-18T14:53:45.683 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/vxlan/vxlan.ko 2024-03-18T14:53:45.683 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/xen-netback 2024-03-18T14:53:45.683 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/xen-netback/xen-netback.ko 2024-03-18T14:53:45.683 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/xen-netfront.ko 2024-03-18T14:53:45.683 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ntb 2024-03-18T14:53:45.683 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ntb/ntb.ko 2024-03-18T14:53:45.683 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/nvme 2024-03-18T14:53:45.683 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/nvme/host 2024-03-18T14:53:45.683 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/nvme/host/nvme-core.ko 2024-03-18T14:53:45.683 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/nvme/host/nvme.ko 2024-03-18T14:53:45.683 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/parport 2024-03-18T14:53:45.683 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/parport/parport.ko 2024-03-18T14:53:45.683 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/parport/parport_cs.ko 2024-03-18T14:53:45.683 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/parport/parport_pc.ko 2024-03-18T14:53:45.684 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/parport/parport_serial.ko 2024-03-18T14:53:45.684 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pci 2024-03-18T14:53:45.684 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pci/hotplug 2024-03-18T14:53:45.684 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-03-18T14:53:45.684 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pci/pcie 2024-03-18T14:53:45.684 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pci/pcie/aer_inject.ko 2024-03-18T14:53:45.684 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pci/xen-pcifront.ko 2024-03-18T14:53:45.684 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pcmcia 2024-03-18T14:53:45.684 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pcmcia/i82092.ko 2024-03-18T14:53:45.684 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pcmcia/pd6729.ko 2024-03-18T14:53:45.684 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pcmcia/yenta_socket.ko 2024-03-18T14:53:45.684 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pinctrl 2024-03-18T14:53:45.684 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pinctrl/intel 2024-03-18T14:53:45.684 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-03-18T14:53:45.684 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform 2024-03-18T14:53:45.684 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86 2024-03-18T14:53:45.685 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/acer-wmi.ko 2024-03-18T14:53:45.685 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/acerhdf.ko 2024-03-18T14:53:45.685 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/apple-gmux.ko 2024-03-18T14:53:45.685 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/asus-laptop.ko 2024-03-18T14:53:45.685 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-03-18T14:53:45.685 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/asus-wmi.ko 2024-03-18T14:53:45.685 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/classmate-laptop.ko 2024-03-18T14:53:45.685 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-03-18T14:53:45.685 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-03-18T14:53:45.685 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-03-18T14:53:45.685 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-03-18T14:53:45.685 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/hdaps.ko 2024-03-18T14:53:45.685 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/intel 2024-03-18T14:53:45.685 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-03-18T14:53:45.686 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/intel_ips.ko 2024-03-18T14:53:45.686 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/msi-wmi.ko 2024-03-18T14:53:45.686 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/mxm-wmi.ko 2024-03-18T14:53:45.686 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-03-18T14:53:45.686 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/samsung-laptop.ko 2024-03-18T14:53:45.686 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/samsung-q10.ko 2024-03-18T14:53:45.686 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/topstar-laptop.ko 2024-03-18T14:53:45.686 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-03-18T14:53:45.686 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/toshiba_haps.ko 2024-03-18T14:53:45.686 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/wmi-bmof.ko 2024-03-18T14:53:45.686 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/wmi.ko 2024-03-18T14:53:45.686 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/powercap 2024-03-18T14:53:45.686 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/powercap/intel_rapl_common.ko 2024-03-18T14:53:45.686 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/powercap/intel_rapl_msr.ko 2024-03-18T14:53:45.687 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pps 2024-03-18T14:53:45.687 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pps/clients 2024-03-18T14:53:45.687 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pps/clients/pps-gpio.ko 2024-03-18T14:53:45.687 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pps/clients/pps-ldisc.ko 2024-03-18T14:53:45.687 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pps/clients/pps_parport.ko 2024-03-18T14:53:45.687 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pps/pps_core.ko 2024-03-18T14:53:45.687 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ptp 2024-03-18T14:53:45.687 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ptp/ptp.ko 2024-03-18T14:53:45.687 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ptp/ptp_kvm.ko 2024-03-18T14:53:45.687 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc 2024-03-18T14:53:45.687 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-bq32k.ko 2024-03-18T14:53:45.687 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-ds1286.ko 2024-03-18T14:53:45.687 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-ds1307.ko 2024-03-18T14:53:45.687 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-ds1374.ko 2024-03-18T14:53:45.687 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-ds1511.ko 2024-03-18T14:53:45.688 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-ds1553.ko 2024-03-18T14:53:45.688 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-ds1672.ko 2024-03-18T14:53:45.688 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-ds1742.ko 2024-03-18T14:53:45.688 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-ds2404.ko 2024-03-18T14:53:45.688 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-ds3232.ko 2024-03-18T14:53:45.688 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-em3027.ko 2024-03-18T14:53:45.688 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-fm3130.ko 2024-03-18T14:53:45.688 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-isl12022.ko 2024-03-18T14:53:45.688 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-isl1208.ko 2024-03-18T14:53:45.688 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-m41t80.ko 2024-03-18T14:53:45.688 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-m48t35.ko 2024-03-18T14:53:45.688 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-m48t59.ko 2024-03-18T14:53:45.688 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-max6900.ko 2024-03-18T14:53:45.688 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-msm6242.ko 2024-03-18T14:53:45.688 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-pcf2127.ko 2024-03-18T14:53:45.688 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-pcf85063.ko 2024-03-18T14:53:45.689 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-pcf8523.ko 2024-03-18T14:53:45.689 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-pcf8563.ko 2024-03-18T14:53:45.689 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-pcf8583.ko 2024-03-18T14:53:45.689 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-rp5c01.ko 2024-03-18T14:53:45.689 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-rs5c372.ko 2024-03-18T14:53:45.689 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-03-18T14:53:45.689 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-rx8025.ko 2024-03-18T14:53:45.689 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-rx8581.ko 2024-03-18T14:53:45.689 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-03-18T14:53:45.689 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-x1205.ko 2024-03-18T14:53:45.689 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi 2024-03-18T14:53:45.689 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/3w-9xxx.ko 2024-03-18T14:53:45.689 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/3w-sas.ko 2024-03-18T14:53:45.689 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/3w-xxxx.ko 2024-03-18T14:53:45.689 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/BusLogic.ko 2024-03-18T14:53:45.690 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/a100u2w.ko 2024-03-18T14:53:45.690 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/aacraid 2024-03-18T14:53:45.690 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/aacraid/aacraid.ko 2024-03-18T14:53:45.690 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/advansys.ko 2024-03-18T14:53:45.690 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/aic7xxx 2024-03-18T14:53:45.690 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-03-18T14:53:45.690 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-03-18T14:53:45.690 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/am53c974.ko 2024-03-18T14:53:45.690 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/arcmsr 2024-03-18T14:53:45.690 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-03-18T14:53:45.690 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/atp870u.ko 2024-03-18T14:53:45.690 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/be2iscsi 2024-03-18T14:53:45.690 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-03-18T14:53:45.690 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/bfa 2024-03-18T14:53:45.691 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/bfa/bfa.ko 2024-03-18T14:53:45.691 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/bnx2fc 2024-03-18T14:53:45.691 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-03-18T14:53:45.691 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/bnx2i 2024-03-18T14:53:45.691 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-03-18T14:53:45.691 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/ch.ko 2024-03-18T14:53:45.691 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/csiostor 2024-03-18T14:53:45.691 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/csiostor/csiostor.ko 2024-03-18T14:53:45.691 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/cxgbi 2024-03-18T14:53:45.691 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/cxgbi/cxgb3i 2024-03-18T14:53:45.691 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-03-18T14:53:45.691 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/cxgbi/cxgb4i 2024-03-18T14:53:45.691 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-03-18T14:53:45.691 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-03-18T14:53:45.692 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/dc395x.ko 2024-03-18T14:53:45.692 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/device_handler 2024-03-18T14:53:45.692 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-03-18T14:53:45.692 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-03-18T14:53:45.692 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-03-18T14:53:45.692 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-03-18T14:53:45.692 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/dmx3191d.ko 2024-03-18T14:53:45.692 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/esas2r 2024-03-18T14:53:45.692 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/esas2r/esas2r.ko 2024-03-18T14:53:45.692 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/esp_scsi.ko 2024-03-18T14:53:45.692 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/fcoe 2024-03-18T14:53:45.692 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/fcoe/fcoe.ko 2024-03-18T14:53:45.692 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-03-18T14:53:45.692 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/fnic 2024-03-18T14:53:45.693 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/fnic/fnic.ko 2024-03-18T14:53:45.693 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/hpsa.ko 2024-03-18T14:53:45.693 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/hptiop.ko 2024-03-18T14:53:45.693 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/hv_storvsc.ko 2024-03-18T14:53:45.693 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/initio.ko 2024-03-18T14:53:45.693 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/ipr.ko 2024-03-18T14:53:45.693 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/ips.ko 2024-03-18T14:53:45.693 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/isci 2024-03-18T14:53:45.693 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/isci/isci.ko 2024-03-18T14:53:45.693 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-03-18T14:53:45.693 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/iscsi_tcp.ko 2024-03-18T14:53:45.693 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/libfc 2024-03-18T14:53:45.693 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/libfc/libfc.ko 2024-03-18T14:53:45.693 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/libiscsi.ko 2024-03-18T14:53:45.693 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/libiscsi_tcp.ko 2024-03-18T14:53:45.694 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/libsas 2024-03-18T14:53:45.694 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/libsas/libsas.ko 2024-03-18T14:53:45.694 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/lpfc 2024-03-18T14:53:45.694 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/lpfc/lpfc.ko 2024-03-18T14:53:45.694 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/megaraid 2024-03-18T14:53:45.694 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/megaraid.ko 2024-03-18T14:53:45.694 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-03-18T14:53:45.694 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-03-18T14:53:45.694 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-03-18T14:53:45.694 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/mpt3sas 2024-03-18T14:53:45.694 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-03-18T14:53:45.694 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/mvsas 2024-03-18T14:53:45.694 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/mvsas/mvsas.ko 2024-03-18T14:53:45.694 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/mvumi.ko 2024-03-18T14:53:45.695 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/pm8001 2024-03-18T14:53:45.695 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-03-18T14:53:45.695 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/pmcraid.ko 2024-03-18T14:53:45.695 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/qla1280.ko 2024-03-18T14:53:45.695 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/qla2xxx 2024-03-18T14:53:45.695 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-03-18T14:53:45.695 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-03-18T14:53:45.695 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/qla4xxx 2024-03-18T14:53:45.695 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-03-18T14:53:45.695 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/raid_class.ko 2024-03-18T14:53:45.695 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/scsi_debug.ko 2024-03-18T14:53:45.695 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/scsi_transport_fc.ko 2024-03-18T14:53:45.695 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-03-18T14:53:45.695 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/scsi_transport_sas.ko 2024-03-18T14:53:45.696 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/scsi_transport_spi.ko 2024-03-18T14:53:45.696 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/scsi_transport_srp.ko 2024-03-18T14:53:45.696 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/ses.ko 2024-03-18T14:53:45.696 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/st.ko 2024-03-18T14:53:45.696 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/stex.ko 2024-03-18T14:53:45.696 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/sym53c8xx_2 2024-03-18T14:53:45.696 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-03-18T14:53:45.696 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/virtio_scsi.ko 2024-03-18T14:53:45.696 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/vmw_pvscsi.ko 2024-03-18T14:53:45.696 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/wd719x.ko 2024-03-18T14:53:45.696 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ssb 2024-03-18T14:53:45.696 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ssb/ssb.ko 2024-03-18T14:53:45.696 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/target 2024-03-18T14:53:45.696 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/target/iscsi 2024-03-18T14:53:45.697 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-03-18T14:53:45.697 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/target/loopback 2024-03-18T14:53:45.697 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/target/loopback/tcm_loop.ko 2024-03-18T14:53:45.697 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/target/target_core_file.ko 2024-03-18T14:53:45.697 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/target/target_core_iblock.ko 2024-03-18T14:53:45.697 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/target/target_core_mod.ko 2024-03-18T14:53:45.697 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/target/target_core_pscsi.ko 2024-03-18T14:53:45.697 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/target/target_core_user.ko 2024-03-18T14:53:45.697 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/target/tcm_fc 2024-03-18T14:53:45.697 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-03-18T14:53:45.697 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal 2024-03-18T14:53:45.697 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel 2024-03-18T14:53:45.697 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/int340x_thermal 2024-03-18T14:53:45.697 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-03-18T14:53:45.698 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-03-18T14:53:45.698 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-03-18T14:53:45.698 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-03-18T14:53:45.698 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-03-18T14:53:45.698 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-03-18T14:53:45.698 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-03-18T14:53:45.698 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-03-18T14:53:45.698 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-03-18T14:53:45.698 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-03-18T14:53:45.698 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-03-18T14:53:45.698 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-03-18T14:53:45.698 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-03-18T14:53:45.698 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-03-18T14:53:45.698 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-03-18T14:53:45.699 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-03-18T14:53:45.699 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-03-18T14:53:45.699 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-03-18T14:53:45.699 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/tty 2024-03-18T14:53:45.699 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/tty/ipwireless 2024-03-18T14:53:45.699 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-03-18T14:53:45.699 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/tty/n_gsm.ko 2024-03-18T14:53:45.699 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/tty/n_hdlc.ko 2024-03-18T14:53:45.699 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/tty/nozomi.ko 2024-03-18T14:53:45.699 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/tty/serial 2024-03-18T14:53:45.699 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/tty/serial/8250 2024-03-18T14:53:45.699 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-03-18T14:53:45.699 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/tty/serial/arc_uart.ko 2024-03-18T14:53:45.699 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/tty/serial/jsm 2024-03-18T14:53:45.700 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/tty/serial/jsm/jsm.ko 2024-03-18T14:53:45.700 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ufs 2024-03-18T14:53:45.700 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ufs/core 2024-03-18T14:53:45.700 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ufs/core/ufshcd-core.ko 2024-03-18T14:53:45.700 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ufs/host 2024-03-18T14:53:45.700 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-03-18T14:53:45.700 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/uio 2024-03-18T14:53:45.700 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/uio/uio.ko 2024-03-18T14:53:45.700 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/uio/uio_aec.ko 2024-03-18T14:53:45.700 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/uio/uio_cif.ko 2024-03-18T14:53:45.700 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/uio/uio_pci_generic.ko 2024-03-18T14:53:45.700 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/uio/uio_sercos3.ko 2024-03-18T14:53:45.700 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb 2024-03-18T14:53:45.701 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/class 2024-03-18T14:53:45.701 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/class/cdc-acm.ko 2024-03-18T14:53:45.701 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/class/cdc-wdm.ko 2024-03-18T14:53:45.701 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/class/usblp.ko 2024-03-18T14:53:45.701 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/class/usbtmc.ko 2024-03-18T14:53:45.701 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/host 2024-03-18T14:53:45.701 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/host/sl811-hcd.ko 2024-03-18T14:53:45.701 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/image 2024-03-18T14:53:45.701 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/image/mdc800.ko 2024-03-18T14:53:45.701 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/image/microtek.ko 2024-03-18T14:53:45.701 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc 2024-03-18T14:53:45.701 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/adutux.ko 2024-03-18T14:53:45.701 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/appledisplay.ko 2024-03-18T14:53:45.701 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/emi26.ko 2024-03-18T14:53:45.702 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/emi62.ko 2024-03-18T14:53:45.702 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/ezusb.ko 2024-03-18T14:53:45.702 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/idmouse.ko 2024-03-18T14:53:45.702 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/iowarrior.ko 2024-03-18T14:53:45.702 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/isight_firmware.ko 2024-03-18T14:53:45.702 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/ldusb.ko 2024-03-18T14:53:45.702 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/legousbtower.ko 2024-03-18T14:53:45.702 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/sisusbvga 2024-03-18T14:53:45.702 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-03-18T14:53:45.702 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/trancevibrator.ko 2024-03-18T14:53:45.702 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/usb3503.ko 2024-03-18T14:53:45.702 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/usblcd.ko 2024-03-18T14:53:45.703 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/usbsevseg.ko 2024-03-18T14:53:45.703 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/uss720.ko 2024-03-18T14:53:45.703 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/yurex.ko 2024-03-18T14:53:45.703 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/phy 2024-03-18T14:53:45.703 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/phy/phy-generic.ko 2024-03-18T14:53:45.703 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial 2024-03-18T14:53:45.703 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/aircable.ko 2024-03-18T14:53:45.703 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/ark3116.ko 2024-03-18T14:53:45.703 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/belkin_sa.ko 2024-03-18T14:53:45.703 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/ch341.ko 2024-03-18T14:53:45.703 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/cp210x.ko 2024-03-18T14:53:45.703 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/cyberjack.ko 2024-03-18T14:53:45.703 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/cypress_m8.ko 2024-03-18T14:53:45.703 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/digi_acceleport.ko 2024-03-18T14:53:45.703 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/empeg.ko 2024-03-18T14:53:45.704 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/ftdi_sio.ko 2024-03-18T14:53:45.704 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/garmin_gps.ko 2024-03-18T14:53:45.704 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/io_edgeport.ko 2024-03-18T14:53:45.704 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/io_ti.ko 2024-03-18T14:53:45.704 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/ipaq.ko 2024-03-18T14:53:45.704 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/ipw.ko 2024-03-18T14:53:45.704 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/ir-usb.ko 2024-03-18T14:53:45.704 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-03-18T14:53:45.704 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/keyspan.ko 2024-03-18T14:53:45.704 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/keyspan_pda.ko 2024-03-18T14:53:45.704 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/kl5kusb105.ko 2024-03-18T14:53:45.704 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/kobil_sct.ko 2024-03-18T14:53:45.704 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/mct_u232.ko 2024-03-18T14:53:45.704 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/mos7720.ko 2024-03-18T14:53:45.704 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/mos7840.ko 2024-03-18T14:53:45.705 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/navman.ko 2024-03-18T14:53:45.705 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/omninet.ko 2024-03-18T14:53:45.705 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/opticon.ko 2024-03-18T14:53:45.705 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/option.ko 2024-03-18T14:53:45.705 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/oti6858.ko 2024-03-18T14:53:45.705 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/pl2303.ko 2024-03-18T14:53:45.705 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/qcaux.ko 2024-03-18T14:53:45.705 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/qcserial.ko 2024-03-18T14:53:45.705 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/quatech2.ko 2024-03-18T14:53:45.705 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/safe_serial.ko 2024-03-18T14:53:45.705 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/sierra.ko 2024-03-18T14:53:45.705 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/spcp8x5.ko 2024-03-18T14:53:45.705 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/ssu100.ko 2024-03-18T14:53:45.705 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/symbolserial.ko 2024-03-18T14:53:45.705 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-03-18T14:53:45.706 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-03-18T14:53:45.706 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/usb_debug.ko 2024-03-18T14:53:45.706 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/usb_wwan.ko 2024-03-18T14:53:45.706 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/visor.ko 2024-03-18T14:53:45.706 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/whiteheat.ko 2024-03-18T14:53:45.706 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/xsens_mt.ko 2024-03-18T14:53:45.706 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/storage 2024-03-18T14:53:45.706 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/storage/uas.ko 2024-03-18T14:53:45.706 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/storage/ums-alauda.ko 2024-03-18T14:53:45.706 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/storage/ums-cypress.ko 2024-03-18T14:53:45.706 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/storage/ums-datafab.ko 2024-03-18T14:53:45.706 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-03-18T14:53:45.706 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/storage/ums-freecom.ko 2024-03-18T14:53:45.706 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/storage/ums-isd200.ko 2024-03-18T14:53:45.707 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-03-18T14:53:45.707 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/storage/ums-karma.ko 2024-03-18T14:53:45.707 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/storage/ums-onetouch.ko 2024-03-18T14:53:45.707 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/storage/ums-realtek.ko 2024-03-18T14:53:45.707 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/storage/ums-sddr09.ko 2024-03-18T14:53:45.707 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/storage/ums-sddr55.ko 2024-03-18T14:53:45.707 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/storage/ums-usbat.ko 2024-03-18T14:53:45.707 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/storage/usb-storage.ko 2024-03-18T14:53:45.707 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/usbip 2024-03-18T14:53:45.707 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/usbip/usbip-core.ko 2024-03-18T14:53:45.707 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/usbip/usbip-host.ko 2024-03-18T14:53:45.707 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-03-18T14:53:45.707 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/vfio 2024-03-18T14:53:45.707 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/vfio/pci 2024-03-18T14:53:45.708 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-03-18T14:53:45.708 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/vfio/pci/vfio-pci.ko 2024-03-18T14:53:45.708 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/vfio/vfio.ko 2024-03-18T14:53:45.708 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-03-18T14:53:45.708 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/vhost 2024-03-18T14:53:45.708 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/vhost/vhost.ko 2024-03-18T14:53:45.708 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/vhost/vhost_iotlb.ko 2024-03-18T14:53:45.708 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/vhost/vhost_net.ko 2024-03-18T14:53:45.708 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/vhost/vhost_scsi.ko 2024-03-18T14:53:45.708 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/video 2024-03-18T14:53:45.708 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/video/backlight 2024-03-18T14:53:45.708 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/video/backlight/apple_bl.ko 2024-03-18T14:53:45.708 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/video/backlight/lcd.ko 2024-03-18T14:53:45.708 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/video/backlight/platform_lcd.ko 2024-03-18T14:53:45.708 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/virtio 2024-03-18T14:53:45.708 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/virtio/virtio.ko 2024-03-18T14:53:45.709 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/virtio/virtio_balloon.ko 2024-03-18T14:53:45.709 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/virtio/virtio_mmio.ko 2024-03-18T14:53:45.709 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/virtio/virtio_pci.ko 2024-03-18T14:53:45.709 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-03-18T14:53:45.709 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-03-18T14:53:45.709 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/virtio/virtio_ring.ko 2024-03-18T14:53:45.709 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1 2024-03-18T14:53:45.709 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1/masters 2024-03-18T14:53:45.709 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1/masters/ds2482.ko 2024-03-18T14:53:45.709 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1/masters/ds2490.ko 2024-03-18T14:53:45.709 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1/slaves 2024-03-18T14:53:45.709 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-03-18T14:53:45.709 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-03-18T14:53:45.709 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-03-18T14:53:45.709 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-03-18T14:53:45.710 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-03-18T14:53:45.710 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-03-18T14:53:45.710 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-03-18T14:53:45.710 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-03-18T14:53:45.710 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-03-18T14:53:45.710 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1/slaves/w1_smem.ko 2024-03-18T14:53:45.710 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1/slaves/w1_therm.ko 2024-03-18T14:53:45.710 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1/wire.ko 2024-03-18T14:53:45.710 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog 2024-03-18T14:53:45.710 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/alim1535_wdt.ko 2024-03-18T14:53:45.710 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/alim7101_wdt.ko 2024-03-18T14:53:45.710 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/f71808e_wdt.ko 2024-03-18T14:53:45.710 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/hpwdt.ko 2024-03-18T14:53:45.710 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/i6300esb.ko 2024-03-18T14:53:45.711 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-03-18T14:53:45.711 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/iTCO_wdt.ko 2024-03-18T14:53:45.711 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/ib700wdt.ko 2024-03-18T14:53:45.711 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/ibmasr.ko 2024-03-18T14:53:45.711 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-03-18T14:53:45.711 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/it8712f_wdt.ko 2024-03-18T14:53:45.711 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/it87_wdt.ko 2024-03-18T14:53:45.711 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/machzwd.ko 2024-03-18T14:53:45.711 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/nv_tco.ko 2024-03-18T14:53:45.711 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/pcwd_pci.ko 2024-03-18T14:53:45.711 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/pcwd_usb.ko 2024-03-18T14:53:45.711 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-03-18T14:53:45.711 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/sch311x_wdt.ko 2024-03-18T14:53:45.711 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/softdog.ko 2024-03-18T14:53:45.712 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/sp5100_tco.ko 2024-03-18T14:53:45.712 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/via_wdt.ko 2024-03-18T14:53:45.712 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-03-18T14:53:45.712 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/w83877f_wdt.ko 2024-03-18T14:53:45.712 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/w83977f_wdt.ko 2024-03-18T14:53:45.712 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/wdt_pci.ko 2024-03-18T14:53:45.712 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/xen_wdt.ko 2024-03-18T14:53:45.712 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/xen 2024-03-18T14:53:45.712 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/xen/xen-acpi-processor.ko 2024-03-18T14:53:45.712 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/xen/xen-evtchn.ko 2024-03-18T14:53:45.712 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/xen/xen-gntalloc.ko 2024-03-18T14:53:45.712 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/xen/xen-gntdev.ko 2024-03-18T14:53:45.712 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/xen/xen-pciback 2024-03-18T14:53:45.712 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-03-18T14:53:45.712 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/xen/xen-privcmd.ko 2024-03-18T14:53:45.713 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/xen/xenfs 2024-03-18T14:53:45.713 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/xen/xenfs/xenfs.ko 2024-03-18T14:53:45.713 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs 2024-03-18T14:53:45.713 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/9p 2024-03-18T14:53:45.713 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/9p/9p.ko 2024-03-18T14:53:45.713 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/affs 2024-03-18T14:53:45.713 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/affs/affs.ko 2024-03-18T14:53:45.713 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/befs 2024-03-18T14:53:45.713 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/befs/befs.ko 2024-03-18T14:53:45.713 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/binfmt_misc.ko 2024-03-18T14:53:45.713 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/btrfs 2024-03-18T14:53:45.713 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/btrfs/btrfs.ko 2024-03-18T14:53:45.713 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/cachefiles 2024-03-18T14:53:45.713 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/cachefiles/cachefiles.ko 2024-03-18T14:53:45.713 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ceph 2024-03-18T14:53:45.713 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ceph/ceph.ko 2024-03-18T14:53:45.714 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/coda 2024-03-18T14:53:45.714 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/coda/coda.ko 2024-03-18T14:53:45.714 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/cramfs 2024-03-18T14:53:45.714 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/cramfs/cramfs.ko 2024-03-18T14:53:45.714 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/dlm 2024-03-18T14:53:45.714 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/dlm/dlm.ko 2024-03-18T14:53:45.714 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ecryptfs 2024-03-18T14:53:45.714 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ecryptfs/ecryptfs.ko 2024-03-18T14:53:45.714 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/f2fs 2024-03-18T14:53:45.714 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/f2fs/f2fs.ko 2024-03-18T14:53:45.714 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/fat 2024-03-18T14:53:45.714 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/fat/fat.ko 2024-03-18T14:53:45.714 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/fat/msdos.ko 2024-03-18T14:53:45.714 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/fat/vfat.ko 2024-03-18T14:53:45.715 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/fuse 2024-03-18T14:53:45.715 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/fuse/cuse.ko 2024-03-18T14:53:45.715 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/fuse/fuse.ko 2024-03-18T14:53:45.715 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/gfs2 2024-03-18T14:53:45.715 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/gfs2/gfs2.ko 2024-03-18T14:53:45.715 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/hfs 2024-03-18T14:53:45.715 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/hfs/hfs.ko 2024-03-18T14:53:45.715 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/hfsplus 2024-03-18T14:53:45.715 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/hfsplus/hfsplus.ko 2024-03-18T14:53:45.715 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/isofs 2024-03-18T14:53:45.715 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/isofs/isofs.ko 2024-03-18T14:53:45.715 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/jfs 2024-03-18T14:53:45.715 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/jfs/jfs.ko 2024-03-18T14:53:45.715 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/lockd 2024-03-18T14:53:45.715 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/lockd/lockd.ko 2024-03-18T14:53:45.715 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/minix 2024-03-18T14:53:45.716 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/minix/minix.ko 2024-03-18T14:53:45.716 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/netfs 2024-03-18T14:53:45.716 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/netfs/netfs.ko 2024-03-18T14:53:45.716 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nfs 2024-03-18T14:53:45.716 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nfs/blocklayout 2024-03-18T14:53:45.716 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-03-18T14:53:45.716 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nfs/filelayout 2024-03-18T14:53:45.716 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-03-18T14:53:45.716 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nfs/flexfilelayout 2024-03-18T14:53:45.716 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-03-18T14:53:45.716 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nfs/nfs.ko 2024-03-18T14:53:45.716 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nfs/nfsv3.ko 2024-03-18T14:53:45.716 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nfs/nfsv4.ko 2024-03-18T14:53:45.716 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nfs_common 2024-03-18T14:53:45.716 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nfs_common/grace.ko 2024-03-18T14:53:45.717 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nfs_common/nfs_acl.ko 2024-03-18T14:53:45.717 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nfsd 2024-03-18T14:53:45.717 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nfsd/nfsd.ko 2024-03-18T14:53:45.717 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nilfs2 2024-03-18T14:53:45.717 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nilfs2/nilfs2.ko 2024-03-18T14:53:45.717 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls 2024-03-18T14:53:45.717 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/mac-celtic.ko 2024-03-18T14:53:45.717 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/mac-centeuro.ko 2024-03-18T14:53:45.717 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/mac-croatian.ko 2024-03-18T14:53:45.717 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/mac-cyrillic.ko 2024-03-18T14:53:45.717 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/mac-gaelic.ko 2024-03-18T14:53:45.717 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/mac-greek.ko 2024-03-18T14:53:45.717 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/mac-iceland.ko 2024-03-18T14:53:45.717 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/mac-inuit.ko 2024-03-18T14:53:45.717 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/mac-roman.ko 2024-03-18T14:53:45.717 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/mac-romanian.ko 2024-03-18T14:53:45.718 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/mac-turkish.ko 2024-03-18T14:53:45.718 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp1250.ko 2024-03-18T14:53:45.718 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp1251.ko 2024-03-18T14:53:45.718 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp1255.ko 2024-03-18T14:53:45.718 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp737.ko 2024-03-18T14:53:45.718 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp775.ko 2024-03-18T14:53:45.718 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp850.ko 2024-03-18T14:53:45.718 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp852.ko 2024-03-18T14:53:45.718 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp855.ko 2024-03-18T14:53:45.718 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp857.ko 2024-03-18T14:53:45.718 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp860.ko 2024-03-18T14:53:45.718 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp861.ko 2024-03-18T14:53:45.718 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp862.ko 2024-03-18T14:53:45.718 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp863.ko 2024-03-18T14:53:45.718 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp864.ko 2024-03-18T14:53:45.719 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp865.ko 2024-03-18T14:53:45.719 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp866.ko 2024-03-18T14:53:45.719 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp869.ko 2024-03-18T14:53:45.719 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp874.ko 2024-03-18T14:53:45.719 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp932.ko 2024-03-18T14:53:45.719 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp936.ko 2024-03-18T14:53:45.719 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp949.ko 2024-03-18T14:53:45.719 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp950.ko 2024-03-18T14:53:45.719 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_euc-jp.ko 2024-03-18T14:53:45.719 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_iso8859-1.ko 2024-03-18T14:53:45.719 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_iso8859-13.ko 2024-03-18T14:53:45.719 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_iso8859-14.ko 2024-03-18T14:53:45.719 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_iso8859-15.ko 2024-03-18T14:53:45.719 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_iso8859-2.ko 2024-03-18T14:53:45.719 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_iso8859-3.ko 2024-03-18T14:53:45.720 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_iso8859-4.ko 2024-03-18T14:53:45.720 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_iso8859-5.ko 2024-03-18T14:53:45.720 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_iso8859-6.ko 2024-03-18T14:53:45.720 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_iso8859-7.ko 2024-03-18T14:53:45.720 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_iso8859-9.ko 2024-03-18T14:53:45.720 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_koi8-r.ko 2024-03-18T14:53:45.720 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_koi8-ru.ko 2024-03-18T14:53:45.720 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_koi8-u.ko 2024-03-18T14:53:45.720 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_ucs2_utils.ko 2024-03-18T14:53:45.720 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_utf8.ko 2024-03-18T14:53:45.720 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ocfs2 2024-03-18T14:53:45.720 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ocfs2/cluster 2024-03-18T14:53:45.720 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-03-18T14:53:45.720 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ocfs2/dlm 2024-03-18T14:53:45.720 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-03-18T14:53:45.720 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ocfs2/dlmfs 2024-03-18T14:53:45.721 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-03-18T14:53:45.721 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ocfs2/ocfs2.ko 2024-03-18T14:53:45.721 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-03-18T14:53:45.721 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-03-18T14:53:45.721 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-03-18T14:53:45.721 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/overlayfs 2024-03-18T14:53:45.721 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/overlayfs/overlay.ko 2024-03-18T14:53:45.721 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/pstore 2024-03-18T14:53:45.721 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/pstore/ramoops.ko 2024-03-18T14:53:45.721 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/reiserfs 2024-03-18T14:53:45.721 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/reiserfs/reiserfs.ko 2024-03-18T14:53:45.721 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/romfs 2024-03-18T14:53:45.721 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/romfs/romfs.ko 2024-03-18T14:53:45.721 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/smb 2024-03-18T14:53:45.722 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/smb/client 2024-03-18T14:53:45.722 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/smb/client/cifs.ko 2024-03-18T14:53:45.722 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/smb/common 2024-03-18T14:53:45.722 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/smb/common/cifs_arc4.ko 2024-03-18T14:53:45.722 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/smb/common/cifs_md4.ko 2024-03-18T14:53:45.722 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/squashfs 2024-03-18T14:53:45.722 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/squashfs/squashfs.ko 2024-03-18T14:53:45.722 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/sysv 2024-03-18T14:53:45.722 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/sysv/sysv.ko 2024-03-18T14:53:45.722 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/udf 2024-03-18T14:53:45.722 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/udf/udf.ko 2024-03-18T14:53:45.722 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ufs 2024-03-18T14:53:45.722 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ufs/ufs.ko 2024-03-18T14:53:45.722 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/xfs 2024-03-18T14:53:45.723 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/xfs/xfs.ko 2024-03-18T14:53:45.723 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/kernel 2024-03-18T14:53:45.723 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/kernel/rcu 2024-03-18T14:53:45.723 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/kernel/rcu/rcutorture.ko 2024-03-18T14:53:45.723 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/kernel/torture.ko 2024-03-18T14:53:45.723 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/kernel/trace 2024-03-18T14:53:45.723 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-03-18T14:53:45.723 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib 2024-03-18T14:53:45.723 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/asn1_encoder.ko 2024-03-18T14:53:45.723 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/crc-itu-t.ko 2024-03-18T14:53:45.723 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/crc8.ko 2024-03-18T14:53:45.723 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/crypto 2024-03-18T14:53:45.723 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/crypto/libarc4.ko 2024-03-18T14:53:45.723 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/crypto/libdes.ko 2024-03-18T14:53:45.724 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/libcrc32c.ko 2024-03-18T14:53:45.724 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/lru_cache.ko 2024-03-18T14:53:45.724 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/lz4 2024-03-18T14:53:45.724 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/lz4/lz4_compress.ko 2024-03-18T14:53:45.724 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/lz4/lz4hc_compress.ko 2024-03-18T14:53:45.724 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/math 2024-03-18T14:53:45.724 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/math/cordic.ko 2024-03-18T14:53:45.724 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/raid6 2024-03-18T14:53:45.724 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/raid6/raid6_pq.ko 2024-03-18T14:53:45.724 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/reed_solomon 2024-03-18T14:53:45.724 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/reed_solomon/reed_solomon.ko 2024-03-18T14:53:45.724 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/ts_bm.ko 2024-03-18T14:53:45.724 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/ts_fsm.ko 2024-03-18T14:53:45.724 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/ts_kmp.ko 2024-03-18T14:53:45.724 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/zstd 2024-03-18T14:53:45.725 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/zstd/zstd_compress.ko 2024-03-18T14:53:45.725 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/mm 2024-03-18T14:53:45.725 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/mm/hwpoison-inject.ko 2024-03-18T14:53:45.725 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net 2024-03-18T14:53:45.725 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/6lowpan 2024-03-18T14:53:45.725 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/6lowpan/6lowpan.ko 2024-03-18T14:53:45.725 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/6lowpan/nhc_dest.ko 2024-03-18T14:53:45.725 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/6lowpan/nhc_fragment.ko 2024-03-18T14:53:45.725 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/6lowpan/nhc_hop.ko 2024-03-18T14:53:45.725 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/6lowpan/nhc_ipv6.ko 2024-03-18T14:53:45.725 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/6lowpan/nhc_mobility.ko 2024-03-18T14:53:45.725 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/6lowpan/nhc_routing.ko 2024-03-18T14:53:45.725 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/6lowpan/nhc_udp.ko 2024-03-18T14:53:45.725 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/802 2024-03-18T14:53:45.725 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/802/garp.ko 2024-03-18T14:53:45.725 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/802/mrp.ko 2024-03-18T14:53:45.726 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/802/p8022.ko 2024-03-18T14:53:45.726 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/802/psnap.ko 2024-03-18T14:53:45.726 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/802/stp.ko 2024-03-18T14:53:45.726 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/8021q 2024-03-18T14:53:45.726 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/8021q/8021q.ko 2024-03-18T14:53:45.726 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/9p 2024-03-18T14:53:45.726 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/9p/9pnet.ko 2024-03-18T14:53:45.726 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/9p/9pnet_fd.ko 2024-03-18T14:53:45.726 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/9p/9pnet_rdma.ko 2024-03-18T14:53:45.726 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/9p/9pnet_virtio.ko 2024-03-18T14:53:45.726 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/batman-adv 2024-03-18T14:53:45.726 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/batman-adv/batman-adv.ko 2024-03-18T14:53:45.726 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge 2024-03-18T14:53:45.727 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/br_netfilter.ko 2024-03-18T14:53:45.727 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/bridge.ko 2024-03-18T14:53:45.727 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter 2024-03-18T14:53:45.727 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-03-18T14:53:45.727 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_among.ko 2024-03-18T14:53:45.727 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_arp.ko 2024-03-18T14:53:45.727 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-03-18T14:53:45.727 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-03-18T14:53:45.727 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_ip.ko 2024-03-18T14:53:45.727 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-03-18T14:53:45.727 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_limit.ko 2024-03-18T14:53:45.727 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_log.ko 2024-03-18T14:53:45.727 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_mark.ko 2024-03-18T14:53:45.727 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-03-18T14:53:45.727 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-03-18T14:53:45.727 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-03-18T14:53:45.728 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-03-18T14:53:45.728 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_snat.ko 2024-03-18T14:53:45.728 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_stp.ko 2024-03-18T14:53:45.728 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-03-18T14:53:45.728 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-03-18T14:53:45.728 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-03-18T14:53:45.728 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-03-18T14:53:45.728 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebtables.ko 2024-03-18T14:53:45.728 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-03-18T14:53:45.728 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-03-18T14:53:45.728 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-03-18T14:53:45.728 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ceph 2024-03-18T14:53:45.728 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ceph/libceph.ko 2024-03-18T14:53:45.729 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/core 2024-03-18T14:53:45.729 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/core/failover.ko 2024-03-18T14:53:45.729 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/core/pktgen.ko 2024-03-18T14:53:45.729 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/dccp 2024-03-18T14:53:45.729 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/dccp/dccp.ko 2024-03-18T14:53:45.729 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/dccp/dccp_diag.ko 2024-03-18T14:53:45.729 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/dccp/dccp_ipv4.ko 2024-03-18T14:53:45.729 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/dccp/dccp_ipv6.ko 2024-03-18T14:53:45.729 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/dns_resolver 2024-03-18T14:53:45.729 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/dns_resolver/dns_resolver.ko 2024-03-18T14:53:45.729 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/dsa 2024-03-18T14:53:45.729 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/dsa/dsa_core.ko 2024-03-18T14:53:45.729 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/dsa/tag_brcm.ko 2024-03-18T14:53:45.729 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/dsa/tag_dsa.ko 2024-03-18T14:53:45.730 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/dsa/tag_none.ko 2024-03-18T14:53:45.730 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/dsa/tag_trailer.ko 2024-03-18T14:53:45.730 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ieee802154 2024-03-18T14:53:45.730 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ieee802154/6lowpan 2024-03-18T14:53:45.730 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-03-18T14:53:45.730 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ieee802154/ieee802154.ko 2024-03-18T14:53:45.730 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ieee802154/ieee802154_socket.ko 2024-03-18T14:53:45.730 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4 2024-03-18T14:53:45.730 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/ah4.ko 2024-03-18T14:53:45.730 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/esp4.ko 2024-03-18T14:53:45.730 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/fou.ko 2024-03-18T14:53:45.730 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/gre.ko 2024-03-18T14:53:45.730 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/inet_diag.ko 2024-03-18T14:53:45.731 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/ip_gre.ko 2024-03-18T14:53:45.731 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/ip_tunnel.ko 2024-03-18T14:53:45.731 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/ip_vti.ko 2024-03-18T14:53:45.731 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/ipcomp.ko 2024-03-18T14:53:45.731 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/ipip.ko 2024-03-18T14:53:45.731 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter 2024-03-18T14:53:45.731 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/arp_tables.ko 2024-03-18T14:53:45.731 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-03-18T14:53:45.731 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-03-18T14:53:45.731 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-03-18T14:53:45.731 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-03-18T14:53:45.731 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-03-18T14:53:45.731 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-03-18T14:53:45.731 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-03-18T14:53:45.732 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-03-18T14:53:45.732 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-03-18T14:53:45.732 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/iptable_security.ko 2024-03-18T14:53:45.732 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-03-18T14:53:45.732 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-03-18T14:53:45.732 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-03-18T14:53:45.732 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-03-18T14:53:45.732 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-03-18T14:53:45.732 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-03-18T14:53:45.732 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-03-18T14:53:45.732 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-03-18T14:53:45.732 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-03-18T14:53:45.732 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-03-18T14:53:45.733 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/tcp_bic.ko 2024-03-18T14:53:45.733 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/tcp_dctcp.ko 2024-03-18T14:53:45.733 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/tcp_diag.ko 2024-03-18T14:53:45.733 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/tcp_highspeed.ko 2024-03-18T14:53:45.733 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/tcp_htcp.ko 2024-03-18T14:53:45.733 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/tcp_hybla.ko 2024-03-18T14:53:45.733 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/tcp_illinois.ko 2024-03-18T14:53:45.733 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/tcp_lp.ko 2024-03-18T14:53:45.733 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/tcp_scalable.ko 2024-03-18T14:53:45.733 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/tcp_vegas.ko 2024-03-18T14:53:45.733 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/tcp_veno.ko 2024-03-18T14:53:45.733 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/tcp_westwood.ko 2024-03-18T14:53:45.733 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/tcp_yeah.ko 2024-03-18T14:53:45.733 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/tunnel4.ko 2024-03-18T14:53:45.734 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/udp_diag.ko 2024-03-18T14:53:45.734 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/udp_tunnel.ko 2024-03-18T14:53:45.734 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/xfrm4_tunnel.ko 2024-03-18T14:53:45.734 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6 2024-03-18T14:53:45.734 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/ah6.ko 2024-03-18T14:53:45.734 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/esp6.ko 2024-03-18T14:53:45.734 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/fou6.ko 2024-03-18T14:53:45.734 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/ip6_tunnel.ko 2024-03-18T14:53:45.734 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-03-18T14:53:45.734 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/ip6_vti.ko 2024-03-18T14:53:45.734 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/ipcomp6.ko 2024-03-18T14:53:45.734 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter 2024-03-18T14:53:45.734 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-03-18T14:53:45.735 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-03-18T14:53:45.735 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-03-18T14:53:45.735 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-03-18T14:53:45.735 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-03-18T14:53:45.735 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-03-18T14:53:45.735 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-03-18T14:53:45.735 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-03-18T14:53:45.735 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-03-18T14:53:45.735 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-03-18T14:53:45.735 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-03-18T14:53:45.735 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-03-18T14:53:45.735 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-03-18T14:53:45.735 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-03-18T14:53:45.735 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-03-18T14:53:45.736 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-03-18T14:53:45.736 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-03-18T14:53:45.736 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-03-18T14:53:45.736 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-03-18T14:53:45.736 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-03-18T14:53:45.736 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-03-18T14:53:45.736 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-03-18T14:53:45.736 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-03-18T14:53:45.736 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-03-18T14:53:45.736 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/sit.ko 2024-03-18T14:53:45.736 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/tunnel6.ko 2024-03-18T14:53:45.736 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/xfrm6_tunnel.ko 2024-03-18T14:53:45.736 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/key 2024-03-18T14:53:45.736 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/key/af_key.ko 2024-03-18T14:53:45.737 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/l2tp 2024-03-18T14:53:45.737 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/l2tp/l2tp_core.ko 2024-03-18T14:53:45.737 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/l2tp/l2tp_debugfs.ko 2024-03-18T14:53:45.737 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/l2tp/l2tp_eth.ko 2024-03-18T14:53:45.737 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/l2tp/l2tp_ip.ko 2024-03-18T14:53:45.737 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/l2tp/l2tp_ip6.ko 2024-03-18T14:53:45.737 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/l2tp/l2tp_netlink.ko 2024-03-18T14:53:45.737 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/l2tp/l2tp_ppp.ko 2024-03-18T14:53:45.737 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/llc 2024-03-18T14:53:45.737 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/llc/llc.ko 2024-03-18T14:53:45.737 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/mac802154 2024-03-18T14:53:45.737 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/mac802154/mac802154.ko 2024-03-18T14:53:45.737 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/mpls 2024-03-18T14:53:45.737 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/mpls/mpls_gso.ko 2024-03-18T14:53:45.738 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter 2024-03-18T14:53:45.738 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset 2024-03-18T14:53:45.738 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset/ip_set.ko 2024-03-18T14:53:45.738 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-03-18T14:53:45.738 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-03-18T14:53:45.738 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-03-18T14:53:45.738 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-03-18T14:53:45.738 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-03-18T14:53:45.738 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-03-18T14:53:45.738 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-03-18T14:53:45.738 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-03-18T14:53:45.738 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-03-18T14:53:45.738 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-03-18T14:53:45.739 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-03-18T14:53:45.739 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-03-18T14:53:45.739 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-03-18T14:53:45.739 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-03-18T14:53:45.739 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-03-18T14:53:45.739 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipvs 2024-03-18T14:53:45.739 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipvs/ip_vs.ko 2024-03-18T14:53:45.739 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-03-18T14:53:45.739 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-03-18T14:53:45.739 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-03-18T14:53:45.739 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-03-18T14:53:45.739 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-03-18T14:53:45.739 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-03-18T14:53:45.740 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-03-18T14:53:45.740 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-03-18T14:53:45.740 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-03-18T14:53:45.740 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-03-18T14:53:45.740 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-03-18T14:53:45.740 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-03-18T14:53:45.740 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-03-18T14:53:45.740 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_conncount.ko 2024-03-18T14:53:45.740 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_conntrack.ko 2024-03-18T14:53:45.740 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-03-18T14:53:45.740 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-03-18T14:53:45.740 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-03-18T14:53:45.740 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_conntrack_h323.ko 2024-03-18T14:53:45.741 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_conntrack_irc.ko 2024-03-18T14:53:45.741 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-03-18T14:53:45.741 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-03-18T14:53:45.741 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-03-18T14:53:45.741 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_conntrack_sane.ko 2024-03-18T14:53:45.741 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_conntrack_sip.ko 2024-03-18T14:53:45.741 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-03-18T14:53:45.741 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-03-18T14:53:45.741 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_dup_netdev.ko 2024-03-18T14:53:45.741 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_flow_table.ko 2024-03-18T14:53:45.741 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_flow_table_inet.ko 2024-03-18T14:53:45.741 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_log_syslog.ko 2024-03-18T14:53:45.741 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_nat.ko 2024-03-18T14:53:45.742 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_nat_amanda.ko 2024-03-18T14:53:45.742 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_nat_ftp.ko 2024-03-18T14:53:45.742 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_nat_irc.ko 2024-03-18T14:53:45.742 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_nat_sip.ko 2024-03-18T14:53:45.742 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_nat_tftp.ko 2024-03-18T14:53:45.742 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_synproxy_core.ko 2024-03-18T14:53:45.742 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_tables.ko 2024-03-18T14:53:45.742 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nfnetlink.ko 2024-03-18T14:53:45.742 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nfnetlink_acct.ko 2024-03-18T14:53:45.742 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nfnetlink_log.ko 2024-03-18T14:53:45.742 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nfnetlink_osf.ko 2024-03-18T14:53:45.742 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nfnetlink_queue.ko 2024-03-18T14:53:45.742 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_chain_nat.ko 2024-03-18T14:53:45.743 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_compat.ko 2024-03-18T14:53:45.743 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_ct.ko 2024-03-18T14:53:45.743 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_dup_netdev.ko 2024-03-18T14:53:45.743 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_fib.ko 2024-03-18T14:53:45.743 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_fwd_netdev.ko 2024-03-18T14:53:45.743 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_hash.ko 2024-03-18T14:53:45.743 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_limit.ko 2024-03-18T14:53:45.743 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_log.ko 2024-03-18T14:53:45.743 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_masq.ko 2024-03-18T14:53:45.743 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_nat.ko 2024-03-18T14:53:45.743 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_numgen.ko 2024-03-18T14:53:45.743 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_queue.ko 2024-03-18T14:53:45.743 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_quota.ko 2024-03-18T14:53:45.744 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_redir.ko 2024-03-18T14:53:45.744 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_reject.ko 2024-03-18T14:53:45.744 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_reject_inet.ko 2024-03-18T14:53:45.744 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_socket.ko 2024-03-18T14:53:45.744 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_tunnel.ko 2024-03-18T14:53:45.744 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_AUDIT.ko 2024-03-18T14:53:45.744 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_CHECKSUM.ko 2024-03-18T14:53:45.744 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_CLASSIFY.ko 2024-03-18T14:53:45.744 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-03-18T14:53:45.744 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_CT.ko 2024-03-18T14:53:45.744 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_DSCP.ko 2024-03-18T14:53:45.744 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_HL.ko 2024-03-18T14:53:45.744 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_HMARK.ko 2024-03-18T14:53:45.744 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_IDLETIMER.ko 2024-03-18T14:53:45.745 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_LED.ko 2024-03-18T14:53:45.745 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_LOG.ko 2024-03-18T14:53:45.745 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_MASQUERADE.ko 2024-03-18T14:53:45.745 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_NETMAP.ko 2024-03-18T14:53:45.745 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_NFLOG.ko 2024-03-18T14:53:45.745 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_NFQUEUE.ko 2024-03-18T14:53:45.745 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_RATEEST.ko 2024-03-18T14:53:45.745 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_REDIRECT.ko 2024-03-18T14:53:45.745 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_SECMARK.ko 2024-03-18T14:53:45.745 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_TCPMSS.ko 2024-03-18T14:53:45.745 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-03-18T14:53:45.745 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_TEE.ko 2024-03-18T14:53:45.745 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_TPROXY.ko 2024-03-18T14:53:45.745 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_TRACE.ko 2024-03-18T14:53:45.746 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_addrtype.ko 2024-03-18T14:53:45.746 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_bpf.ko 2024-03-18T14:53:45.746 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_cgroup.ko 2024-03-18T14:53:45.746 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_cluster.ko 2024-03-18T14:53:45.746 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_comment.ko 2024-03-18T14:53:45.746 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_connbytes.ko 2024-03-18T14:53:45.746 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_connlabel.ko 2024-03-18T14:53:45.746 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_connlimit.ko 2024-03-18T14:53:45.746 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_connmark.ko 2024-03-18T14:53:45.746 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_conntrack.ko 2024-03-18T14:53:45.746 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_cpu.ko 2024-03-18T14:53:45.746 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_dccp.ko 2024-03-18T14:53:45.746 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_devgroup.ko 2024-03-18T14:53:45.746 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_dscp.ko 2024-03-18T14:53:45.747 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_ecn.ko 2024-03-18T14:53:45.747 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_esp.ko 2024-03-18T14:53:45.747 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_hashlimit.ko 2024-03-18T14:53:45.747 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_helper.ko 2024-03-18T14:53:45.747 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_hl.ko 2024-03-18T14:53:45.747 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_ipcomp.ko 2024-03-18T14:53:45.747 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_iprange.ko 2024-03-18T14:53:45.747 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_ipvs.ko 2024-03-18T14:53:45.747 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_l2tp.ko 2024-03-18T14:53:45.747 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_length.ko 2024-03-18T14:53:45.747 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_limit.ko 2024-03-18T14:53:45.747 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_mac.ko 2024-03-18T14:53:45.747 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_mark.ko 2024-03-18T14:53:45.747 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_multiport.ko 2024-03-18T14:53:45.747 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_nat.ko 2024-03-18T14:53:45.748 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_nfacct.ko 2024-03-18T14:53:45.748 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_osf.ko 2024-03-18T14:53:45.748 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_owner.ko 2024-03-18T14:53:45.748 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_physdev.ko 2024-03-18T14:53:45.748 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_pkttype.ko 2024-03-18T14:53:45.748 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_policy.ko 2024-03-18T14:53:45.748 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_quota.ko 2024-03-18T14:53:45.748 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_rateest.ko 2024-03-18T14:53:45.748 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_realm.ko 2024-03-18T14:53:45.748 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_recent.ko 2024-03-18T14:53:45.748 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_sctp.ko 2024-03-18T14:53:45.748 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_set.ko 2024-03-18T14:53:45.748 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_socket.ko 2024-03-18T14:53:45.748 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_state.ko 2024-03-18T14:53:45.748 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_statistic.ko 2024-03-18T14:53:45.749 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_string.ko 2024-03-18T14:53:45.749 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_tcpmss.ko 2024-03-18T14:53:45.749 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_time.ko 2024-03-18T14:53:45.749 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_u32.ko 2024-03-18T14:53:45.749 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netlink 2024-03-18T14:53:45.749 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netlink/netlink_diag.ko 2024-03-18T14:53:45.749 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/nsh 2024-03-18T14:53:45.749 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/nsh/nsh.ko 2024-03-18T14:53:45.749 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/openvswitch 2024-03-18T14:53:45.749 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/openvswitch/openvswitch.ko 2024-03-18T14:53:45.749 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/openvswitch/vport-geneve.ko 2024-03-18T14:53:45.749 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/openvswitch/vport-gre.ko 2024-03-18T14:53:45.749 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/openvswitch/vport-vxlan.ko 2024-03-18T14:53:45.749 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/packet 2024-03-18T14:53:45.750 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/packet/af_packet_diag.ko 2024-03-18T14:53:45.750 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/rds 2024-03-18T14:53:45.750 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/rds/rds.ko 2024-03-18T14:53:45.750 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/rds/rds_rdma.ko 2024-03-18T14:53:45.750 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/rds/rds_tcp.ko 2024-03-18T14:53:45.750 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched 2024-03-18T14:53:45.750 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/act_csum.ko 2024-03-18T14:53:45.750 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/act_gact.ko 2024-03-18T14:53:45.750 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/act_mirred.ko 2024-03-18T14:53:45.750 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/act_nat.ko 2024-03-18T14:53:45.750 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/act_pedit.ko 2024-03-18T14:53:45.750 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/act_police.ko 2024-03-18T14:53:45.750 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/act_simple.ko 2024-03-18T14:53:45.750 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/act_skbedit.ko 2024-03-18T14:53:45.751 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/act_vlan.ko 2024-03-18T14:53:45.751 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/cls_basic.ko 2024-03-18T14:53:45.751 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/cls_bpf.ko 2024-03-18T14:53:45.751 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/cls_flow.ko 2024-03-18T14:53:45.751 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/cls_fw.ko 2024-03-18T14:53:45.751 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/cls_route.ko 2024-03-18T14:53:45.751 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/cls_u32.ko 2024-03-18T14:53:45.751 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/em_cmp.ko 2024-03-18T14:53:45.751 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/em_ipset.ko 2024-03-18T14:53:45.751 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/em_meta.ko 2024-03-18T14:53:45.751 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/em_nbyte.ko 2024-03-18T14:53:45.751 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/em_text.ko 2024-03-18T14:53:45.751 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/em_u32.ko 2024-03-18T14:53:45.751 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_choke.ko 2024-03-18T14:53:45.752 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_codel.ko 2024-03-18T14:53:45.752 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_drr.ko 2024-03-18T14:53:45.752 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_fq.ko 2024-03-18T14:53:45.752 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_gred.ko 2024-03-18T14:53:45.752 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_hfsc.ko 2024-03-18T14:53:45.752 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_hhf.ko 2024-03-18T14:53:45.752 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_htb.ko 2024-03-18T14:53:45.752 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_ingress.ko 2024-03-18T14:53:45.752 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_mqprio.ko 2024-03-18T14:53:45.752 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_mqprio_lib.ko 2024-03-18T14:53:45.752 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_multiq.ko 2024-03-18T14:53:45.752 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_netem.ko 2024-03-18T14:53:45.752 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_pie.ko 2024-03-18T14:53:45.753 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_plug.ko 2024-03-18T14:53:45.753 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_prio.ko 2024-03-18T14:53:45.753 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_qfq.ko 2024-03-18T14:53:45.753 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_red.ko 2024-03-18T14:53:45.753 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_sfb.ko 2024-03-18T14:53:45.753 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_sfq.ko 2024-03-18T14:53:45.753 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_tbf.ko 2024-03-18T14:53:45.753 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_teql.ko 2024-03-18T14:53:45.753 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sctp 2024-03-18T14:53:45.753 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sctp/sctp.ko 2024-03-18T14:53:45.753 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sctp/sctp_diag.ko 2024-03-18T14:53:45.753 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sunrpc 2024-03-18T14:53:45.753 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sunrpc/auth_gss 2024-03-18T14:53:45.753 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-03-18T14:53:45.753 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-03-18T14:53:45.754 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sunrpc/sunrpc.ko 2024-03-18T14:53:45.754 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sunrpc/xprtrdma 2024-03-18T14:53:45.754 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-03-18T14:53:45.754 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/tipc 2024-03-18T14:53:45.754 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/tipc/diag.ko 2024-03-18T14:53:45.754 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/tipc/tipc.ko 2024-03-18T14:53:45.754 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/unix 2024-03-18T14:53:45.754 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/unix/unix_diag.ko 2024-03-18T14:53:45.754 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/vmw_vsock 2024-03-18T14:53:45.754 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-03-18T14:53:45.754 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-03-18T14:53:45.754 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/vmw_vsock/vsock.ko 2024-03-18T14:53:45.754 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/vmw_vsock/vsock_diag.ko 2024-03-18T14:53:45.754 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/vmw_vsock/vsock_loopback.ko 2024-03-18T14:53:45.755 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/xfrm 2024-03-18T14:53:45.755 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/xfrm/xfrm_ipcomp.ko 2024-03-18T14:53:45.755 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/security 2024-03-18T14:53:45.755 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/security/keys 2024-03-18T14:53:45.755 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/security/keys/encrypted-keys 2024-03-18T14:53:45.755 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-03-18T14:53:45.755 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/security/keys/trusted-keys 2024-03-18T14:53:45.755 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/security/keys/trusted-keys/trusted.ko 2024-03-18T14:53:45.755 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/virt 2024-03-18T14:53:45.755 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/virt/lib 2024-03-18T14:53:45.755 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/virt/lib/irqbypass.ko 2024-03-18T14:53:45.755 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/modules.alias 2024-03-18T14:53:45.755 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/modules.alias.bin 2024-03-18T14:53:45.755 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/modules.builtin 2024-03-18T14:53:45.755 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/modules.builtin.alias.bin 2024-03-18T14:53:45.756 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/modules.builtin.bin 2024-03-18T14:53:45.756 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/modules.builtin.modinfo 2024-03-18T14:53:45.756 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/modules.dep 2024-03-18T14:53:45.756 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/modules.dep.bin 2024-03-18T14:53:45.756 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/modules.devname 2024-03-18T14:53:45.756 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/modules.order 2024-03-18T14:53:45.756 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/modules.softdep 2024-03-18T14:53:45.756 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/modules.symbols 2024-03-18T14:53:45.756 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/modules.symbols.bin 2024-03-18T14:53:45.756 INFO:teuthology.orchestra.run.smithi019.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/vmlinuz 2024-03-18T14:53:45.760 DEBUG:teuthology.task.kernel:get_image_version: 6.8.0-rc7-g3845b7ac715a 2024-03-18T14:53:45.760 DEBUG:teuthology.orchestra.run.smithi019:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-03-18T14:53:45.784 INFO:teuthology.orchestra.run.smithi019.stdout:postinstall scriptlet (using /bin/sh): 2024-03-18T14:53:45.784 INFO:teuthology.orchestra.run.smithi019.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-03-18T14:53:45.784 INFO:teuthology.orchestra.run.smithi019.stdout: /usr/bin/kernel-install add 6.8.0-rc7-g3845b7ac715a /lib/modules/6.8.0-rc7-g3845b7ac715a/vmlinuz 2024-03-18T14:53:45.784 INFO:teuthology.orchestra.run.smithi019.stdout:fi 2024-03-18T14:53:45.784 INFO:teuthology.orchestra.run.smithi019.stdout:for file in vmlinuz System.map config; do 2024-03-18T14:53:45.784 INFO:teuthology.orchestra.run.smithi019.stdout: if ! cmp --silent "/lib/modules/6.8.0-rc7-g3845b7ac715a/${file}" "/boot/${file}-6.8.0-rc7-g3845b7ac715a"; then 2024-03-18T14:53:45.784 INFO:teuthology.orchestra.run.smithi019.stdout: cp "/lib/modules/6.8.0-rc7-g3845b7ac715a/${file}" "/boot/${file}-6.8.0-rc7-g3845b7ac715a" 2024-03-18T14:53:45.784 INFO:teuthology.orchestra.run.smithi019.stdout: fi 2024-03-18T14:53:45.784 INFO:teuthology.orchestra.run.smithi019.stdout:done 2024-03-18T14:53:45.784 INFO:teuthology.orchestra.run.smithi019.stdout:preuninstall scriptlet (using /bin/sh): 2024-03-18T14:53:45.785 INFO:teuthology.orchestra.run.smithi019.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2024-03-18T14:53:45.785 INFO:teuthology.orchestra.run.smithi019.stdout:new-kernel-pkg --remove 6.8.0-rc7-g3845b7ac715a --rminitrd --initrdfile=/boot/initramfs-6.8.0-rc7-g3845b7ac715a.img 2024-03-18T14:53:45.785 INFO:teuthology.orchestra.run.smithi019.stdout:elif [ -x /usr/bin/kernel-install ]; then 2024-03-18T14:53:45.785 INFO:teuthology.orchestra.run.smithi019.stdout:kernel-install remove 6.8.0-rc7-g3845b7ac715a 2024-03-18T14:53:45.785 INFO:teuthology.orchestra.run.smithi019.stdout:fi 2024-03-18T14:53:45.785 INFO:teuthology.orchestra.run.smithi019.stdout:postuninstall scriptlet (using /bin/sh): 2024-03-18T14:53:45.785 INFO:teuthology.orchestra.run.smithi019.stdout:if [ -x /sbin/update-bootloader ]; then 2024-03-18T14:53:45.785 INFO:teuthology.orchestra.run.smithi019.stdout:/sbin/update-bootloader --remove 6.8.0-rc7-g3845b7ac715a 2024-03-18T14:53:45.785 INFO:teuthology.orchestra.run.smithi019.stdout:fi 2024-03-18T14:53:45.786 DEBUG:teuthology.orchestra.run.smithi019:> sudo rpm -qi grub2-tools 2024-03-18T14:53:45.866 INFO:teuthology.orchestra.run.smithi019.stdout:Name : grub2-tools 2024-03-18T14:53:45.866 INFO:teuthology.orchestra.run.smithi019.stdout:Epoch : 1 2024-03-18T14:53:45.866 INFO:teuthology.orchestra.run.smithi019.stdout:Version : 2.06 2024-03-18T14:53:45.866 INFO:teuthology.orchestra.run.smithi019.stdout:Release : 68.el9 2024-03-18T14:53:45.866 INFO:teuthology.orchestra.run.smithi019.stdout:Architecture: x86_64 2024-03-18T14:53:45.866 INFO:teuthology.orchestra.run.smithi019.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-18T14:53:45.866 INFO:teuthology.orchestra.run.smithi019.stdout:Group : Unspecified 2024-03-18T14:53:45.866 INFO:teuthology.orchestra.run.smithi019.stdout:Size : 8264254 2024-03-18T14:53:45.866 INFO:teuthology.orchestra.run.smithi019.stdout:License : GPLv3+ 2024-03-18T14:53:45.867 INFO:teuthology.orchestra.run.smithi019.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-18T14:53:45.867 INFO:teuthology.orchestra.run.smithi019.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-18T14:53:45.867 INFO:teuthology.orchestra.run.smithi019.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-18T14:53:45.867 INFO:teuthology.orchestra.run.smithi019.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-18T14:53:45.867 INFO:teuthology.orchestra.run.smithi019.stdout:Packager : builder@centos.org 2024-03-18T14:53:45.867 INFO:teuthology.orchestra.run.smithi019.stdout:Vendor : CentOS 2024-03-18T14:53:45.867 INFO:teuthology.orchestra.run.smithi019.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-18T14:53:45.867 INFO:teuthology.orchestra.run.smithi019.stdout:Summary : Support tools for GRUB. 2024-03-18T14:53:45.867 INFO:teuthology.orchestra.run.smithi019.stdout:Description : 2024-03-18T14:53:45.867 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-18T14:53:45.867 INFO:teuthology.orchestra.run.smithi019.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-18T14:53:45.867 INFO:teuthology.orchestra.run.smithi019.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-18T14:53:45.867 INFO:teuthology.orchestra.run.smithi019.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-18T14:53:45.867 INFO:teuthology.orchestra.run.smithi019.stdout:hardware devices. 2024-03-18T14:53:45.867 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-18T14:53:45.868 INFO:teuthology.orchestra.run.smithi019.stdout:This subpackage provides tools for support of all platforms. 2024-03-18T14:53:45.868 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-18T14:53:45.868 INFO:teuthology.task.kernel:Updating grub on smithi019 to boot 6.8.0-rc7-g3845b7ac715a 2024-03-18T14:53:45.869 DEBUG:teuthology.orchestra.run.smithi019:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-18T14:53:46.515 INFO:teuthology.orchestra.run.smithi019.stderr:Generating grub configuration file ... 2024-03-18T14:53:47.991 INFO:teuthology.orchestra.run.smithi019.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-18T14:53:48.006 INFO:teuthology.orchestra.run.smithi019.stderr:done 2024-03-18T14:53:48.008 DEBUG:teuthology.orchestra.run.smithi019:> mktemp 2024-03-18T14:53:48.030 INFO:teuthology.orchestra.run.smithi019.stdout:/tmp/tmp.qAOgJyga2f 2024-03-18T14:53:48.030 DEBUG:teuthology.orchestra.run.smithi019:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.qAOgJyga2f 2024-03-18T14:53:48.103 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod 0666 /tmp/tmp.qAOgJyga2f 2024-03-18T14:53:48.244 DEBUG:teuthology.orchestra.remote:smithi019:/tmp/tmp.qAOgJyga2f is 6KB 2024-03-18T14:53:48.292 DEBUG:teuthology.orchestra.run.smithi019:> rm -fr /tmp/tmp.qAOgJyga2f 2024-03-18T14:53:48.307 DEBUG:teuthology.orchestra.run.smithi019:> sudo /bin/ls /boot/loader/entries || true 2024-03-18T14:53:48.375 INFO:teuthology.orchestra.run.smithi019.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-18T14:53:48.375 INFO:teuthology.orchestra.run.smithi019.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-18T14:53:48.375 INFO:teuthology.orchestra.run.smithi019.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-18T14:53:48.375 INFO:teuthology.orchestra.run.smithi019.stdout:15949e3c88704300882482eddfd7baae-6.8.0-rc7-g3845b7ac715a.conf 2024-03-18T14:53:48.376 DEBUG:teuthology.orchestra.run.smithi019:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-6.8.0-rc7-g3845b7ac715a 2024-03-18T14:53:48.532 DEBUG:teuthology.orchestra.run.smithi019:> sudo shutdown -r now 2024-03-18T14:54:18.537 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-03-18T14:54:18.538 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2024-03-18T14:54:18.539 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi019.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T14:54:37.086 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.19 2024-03-18T14:54:46.087 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2024-03-18T14:54:46.088 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi019.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T14:55:46.090 DEBUG:teuthology.orchestra.remote:timed out 2024-03-18T14:55:58.091 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2024-03-18T14:55:58.093 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi019.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T14:55:58.580 DEBUG:teuthology.orchestra.run.smithi019:> true 2024-03-18T14:55:59.273 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi019.front.sepia.ceph.com' 2024-03-18T14:55:59.273 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.8.0-rc7-g3845b7ac715a"... 2024-03-18T14:55:59.273 DEBUG:teuthology.orchestra.run.smithi019:> uname -r 2024-03-18T14:55:59.330 INFO:teuthology.orchestra.run.smithi019.stdout:6.8.0-rc7-g3845b7ac715a 2024-03-18T14:55:59.331 DEBUG:teuthology.task.kernel:current kernel version is 6.8.0-rc7-g3845b7ac715a vs 6.8.0-rc7-g3845b7ac715a 2024-03-18T14:55:59.331 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-18T14:55:59.331 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-18T14:56:00.332 DEBUG:teuthology.parallel:result is None 2024-03-18T14:56:00.333 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-18T14:56:00.342 INFO:teuthology.task.internal:Creating test directory... 2024-03-18T14:56:00.342 DEBUG:teuthology.orchestra.run.smithi019:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-18T14:56:00.350 DEBUG:teuthology.orchestra.run.smithi033:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-18T14:56:00.353 DEBUG:teuthology.orchestra.run.smithi104:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-18T14:56:00.400 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-18T14:56:00.413 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-18T14:56:00.456 INFO:teuthology.task.internal:Creating archive directory... 2024-03-18T14:56:00.456 DEBUG:teuthology.orchestra.run.smithi019:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-18T14:56:00.464 DEBUG:teuthology.orchestra.run.smithi033:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-18T14:56:00.466 DEBUG:teuthology.orchestra.run.smithi104:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-18T14:56:00.534 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-18T14:56:00.541 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-18T14:56:00.542 DEBUG:teuthology.orchestra.run.smithi033:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-18T14:56:00.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T14:56:00.559 DEBUG:teuthology.orchestra.run.smithi104:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-18T14:56:00.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T14:56:00.575 DEBUG:teuthology.orchestra.run.smithi019:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-18T14:56:00.582 DEBUG:teuthology.orchestra.run.smithi033:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-18T14:56:00.601 DEBUG:teuthology.orchestra.run.smithi104:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-18T14:56:00.641 INFO:teuthology.orchestra.run.smithi033.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-18T14:56:00.662 INFO:teuthology.orchestra.run.smithi033.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-18T14:56:00.674 INFO:teuthology.orchestra.run.smithi104.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-18T14:56:00.697 INFO:teuthology.orchestra.run.smithi104.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-18T14:56:00.735 INFO:teuthology.orchestra.run.smithi019.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-18T14:56:00.805 INFO:teuthology.orchestra.run.smithi019.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-18T14:56:00.810 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-18T14:56:00.818 INFO:teuthology.task.internal:Configuring sudo... 2024-03-18T14:56:00.818 DEBUG:teuthology.orchestra.run.smithi019:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-18T14:56:00.824 DEBUG:teuthology.orchestra.run.smithi033:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-18T14:56:00.826 DEBUG:teuthology.orchestra.run.smithi104:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-18T14:56:00.922 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-18T14:56:00.933 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-18T14:56:00.933 DEBUG:teuthology.orchestra.run.smithi019:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-18T14:56:00.941 DEBUG:teuthology.orchestra.run.smithi033:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-18T14:56:00.943 DEBUG:teuthology.orchestra.run.smithi104:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-18T14:56:00.993 DEBUG:teuthology.orchestra.run.smithi019:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-18T14:56:01.079 DEBUG:teuthology.orchestra.run.smithi019:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-18T14:56:01.193 DEBUG:teuthology.orchestra.run.smithi019:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-18T14:56:01.280 DEBUG:teuthology.orchestra.run.smithi019:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-18T14:56:01.377 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-03-18T14:56:01.377 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-18T14:56:01.469 DEBUG:teuthology.orchestra.run.smithi033:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-18T14:56:01.515 DEBUG:teuthology.orchestra.run.smithi033:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-18T14:56:01.575 DEBUG:teuthology.orchestra.run.smithi033:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-18T14:56:01.616 DEBUG:teuthology.orchestra.run.smithi033:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-18T14:56:01.668 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-18T14:56:01.668 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-18T14:56:01.741 DEBUG:teuthology.orchestra.run.smithi104:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-18T14:56:01.785 DEBUG:teuthology.orchestra.run.smithi104:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-18T14:56:02.069 DEBUG:teuthology.orchestra.run.smithi104:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-18T14:56:02.114 DEBUG:teuthology.orchestra.run.smithi104:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-18T14:56:02.160 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-03-18T14:56:02.161 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-18T14:56:02.230 DEBUG:teuthology.orchestra.run.smithi019:> sudo service rsyslog restart 2024-03-18T14:56:02.238 DEBUG:teuthology.orchestra.run.smithi033:> sudo service rsyslog restart 2024-03-18T14:56:02.240 DEBUG:teuthology.orchestra.run.smithi104:> sudo service rsyslog restart 2024-03-18T14:56:02.284 INFO:teuthology.orchestra.run.smithi033.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-18T14:56:02.317 INFO:teuthology.orchestra.run.smithi104.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-18T14:56:02.365 INFO:teuthology.orchestra.run.smithi019.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-18T14:56:02.787 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-18T14:56:02.796 INFO:teuthology.task.internal:Starting timer... 2024-03-18T14:56:02.796 INFO:teuthology.run_tasks:Running task pcp... 2024-03-18T14:56:02.852 INFO:teuthology.run_tasks:Running task selinux... 2024-03-18T14:56:02.875 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-03-18T14:56:02.876 DEBUG:teuthology.orchestra.run.smithi019:> sudo service auditd rotate 2024-03-18T14:56:02.997 INFO:teuthology.orchestra.run.smithi019.stdout:Rotating logs: 2024-03-18T14:56:03.001 DEBUG:teuthology.orchestra.run.smithi033:> sudo service auditd rotate 2024-03-18T14:56:03.082 INFO:teuthology.orchestra.run.smithi033.stdout:Rotating logs: 2024-03-18T14:56:03.083 DEBUG:teuthology.orchestra.run.smithi104:> sudo service auditd rotate 2024-03-18T14:56:03.146 INFO:teuthology.orchestra.run.smithi104.stdout:Rotating logs: 2024-03-18T14:56:03.147 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-18T14:56:03.148 DEBUG:teuthology.orchestra.run.smithi019:> /usr/sbin/getenforce 2024-03-18T14:56:03.208 INFO:teuthology.orchestra.run.smithi019.stdout:Permissive 2024-03-18T14:56:03.208 DEBUG:teuthology.orchestra.run.smithi033:> /usr/sbin/getenforce 2024-03-18T14:56:03.226 INFO:teuthology.orchestra.run.smithi033.stdout:Permissive 2024-03-18T14:56:03.227 DEBUG:teuthology.orchestra.run.smithi104:> /usr/sbin/getenforce 2024-03-18T14:56:03.247 INFO:teuthology.orchestra.run.smithi104.stdout:Permissive 2024-03-18T14:56:03.247 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi019.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi033.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi104.front.sepia.ceph.com': 'permissive'} 2024-03-18T14:56:03.247 DEBUG:teuthology.orchestra.run.smithi019:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-18T14:56:03.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T14:56:03.327 DEBUG:teuthology.orchestra.run.smithi033:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-18T14:56:03.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T14:56:03.360 DEBUG:teuthology.orchestra.run.smithi104:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-18T14:56:03.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T14:56:03.388 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-18T14:56:03.388 DEBUG:teuthology.orchestra.run.smithi019:> sudo /usr/sbin/setenforce permissive 2024-03-18T14:56:03.479 DEBUG:teuthology.orchestra.run.smithi033:> sudo /usr/sbin/setenforce permissive 2024-03-18T14:56:03.512 DEBUG:teuthology.orchestra.run.smithi104:> sudo /usr/sbin/setenforce permissive 2024-03-18T14:56:03.548 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-18T14:56:03.560 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-18T14:56:03.581 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-18T14:56:03.582 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi019.front.sepia.ceph.com,smithi033.front.sepia.ceph.com,smithi104.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-18T15:02:09.801 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi019.front.sepia.ceph.com'), Remote(name='ubuntu@smithi033.front.sepia.ceph.com'), Remote(name='ubuntu@smithi104.front.sepia.ceph.com')] 2024-03-18T15:02:09.802 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2024-03-18T15:02:09.804 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi019.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T15:02:09.960 DEBUG:teuthology.orchestra.run.smithi019:> true 2024-03-18T15:02:10.115 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi019.front.sepia.ceph.com' 2024-03-18T15:02:10.116 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2024-03-18T15:02:10.116 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi033.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T15:02:10.220 DEBUG:teuthology.orchestra.run.smithi033:> true 2024-03-18T15:02:10.290 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi033.front.sepia.ceph.com' 2024-03-18T15:02:10.291 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-03-18T15:02:10.291 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T15:02:10.380 DEBUG:teuthology.orchestra.run.smithi104:> true 2024-03-18T15:02:10.449 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-03-18T15:02:10.451 INFO:teuthology.run_tasks:Running task clock... 2024-03-18T15:02:10.463 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-18T15:02:10.464 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-18T15:02:10.464 DEBUG:teuthology.orchestra.run.smithi019:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-18T15:02:10.471 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-18T15:02:10.472 DEBUG:teuthology.orchestra.run.smithi033:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-18T15:02:10.474 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-18T15:02:10.474 DEBUG:teuthology.orchestra.run.smithi104:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-18T15:02:10.508 INFO:teuthology.orchestra.run.smithi033.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-18T15:02:10.521 INFO:teuthology.orchestra.run.smithi104.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-18T15:02:10.530 INFO:teuthology.orchestra.run.smithi033.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-18T15:02:10.538 INFO:teuthology.orchestra.run.smithi104.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-18T15:02:10.567 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: ntpd: command not found 2024-03-18T15:02:10.568 INFO:teuthology.orchestra.run.smithi033.stderr:sudo: ntpd: command not found 2024-03-18T15:02:10.574 INFO:teuthology.orchestra.run.smithi019.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-18T15:02:10.581 INFO:teuthology.orchestra.run.smithi104.stdout:506 Cannot talk to daemon 2024-03-18T15:02:10.585 INFO:teuthology.orchestra.run.smithi033.stdout:506 Cannot talk to daemon 2024-03-18T15:02:10.598 INFO:teuthology.orchestra.run.smithi104.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-18T15:02:10.607 INFO:teuthology.orchestra.run.smithi033.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-18T15:02:10.614 INFO:teuthology.orchestra.run.smithi104.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-18T15:02:10.628 INFO:teuthology.orchestra.run.smithi033.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-18T15:02:10.631 INFO:teuthology.orchestra.run.smithi019.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-18T15:02:10.658 INFO:teuthology.orchestra.run.smithi104.stderr:bash: line 1: ntpq: command not found 2024-03-18T15:02:10.661 INFO:teuthology.orchestra.run.smithi104.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-18T15:02:10.661 INFO:teuthology.orchestra.run.smithi104.stdout:=============================================================================== 2024-03-18T15:02:10.662 INFO:teuthology.orchestra.run.smithi104.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T15:02:10.662 INFO:teuthology.orchestra.run.smithi104.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T15:02:10.662 INFO:teuthology.orchestra.run.smithi104.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T15:02:10.662 INFO:teuthology.orchestra.run.smithi104.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T15:02:10.681 INFO:teuthology.orchestra.run.smithi033.stderr:bash: line 1: ntpq: command not found 2024-03-18T15:02:10.684 INFO:teuthology.orchestra.run.smithi033.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-18T15:02:10.685 INFO:teuthology.orchestra.run.smithi033.stdout:=============================================================================== 2024-03-18T15:02:10.685 INFO:teuthology.orchestra.run.smithi033.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T15:02:10.685 INFO:teuthology.orchestra.run.smithi033.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T15:02:10.685 INFO:teuthology.orchestra.run.smithi033.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T15:02:10.685 INFO:teuthology.orchestra.run.smithi033.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T15:02:10.729 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: ntpd: command not found 2024-03-18T15:02:10.773 INFO:teuthology.orchestra.run.smithi019.stdout:506 Cannot talk to daemon 2024-03-18T15:02:10.831 INFO:teuthology.orchestra.run.smithi019.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-18T15:02:10.887 INFO:teuthology.orchestra.run.smithi019.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-18T15:02:11.026 INFO:teuthology.orchestra.run.smithi019.stderr:bash: line 1: ntpq: command not found 2024-03-18T15:02:11.036 INFO:teuthology.orchestra.run.smithi019.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-18T15:02:11.036 INFO:teuthology.orchestra.run.smithi019.stdout:=============================================================================== 2024-03-18T15:02:11.036 INFO:teuthology.orchestra.run.smithi019.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T15:02:11.036 INFO:teuthology.orchestra.run.smithi019.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T15:02:11.036 INFO:teuthology.orchestra.run.smithi019.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T15:02:11.036 INFO:teuthology.orchestra.run.smithi019.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T15:02:11.037 INFO:teuthology.run_tasks:Running task install... 2024-03-18T15:02:11.046 DEBUG:teuthology.task.install:project ceph 2024-03-18T15:02:11.046 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af'}} 2024-03-18T15:02:11.046 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af'} 2024-03-18T15:02:11.046 INFO:teuthology.task.install:Using flavor: default 2024-03-18T15:02:11.052 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-18T15:02:11.052 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-03-18T15:02:11.052 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af', 'tag': None, 'wait_for_package': False} 2024-03-18T15:02:11.053 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-18T15:02:11.055 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af', 'tag': None, 'wait_for_package': False} 2024-03-18T15:02:11.055 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-18T15:02:11.056 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af', 'tag': None, 'wait_for_package': False} 2024-03-18T15:02:11.057 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-18T15:02:11.201 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/ 2024-03-18T15:02:11.201 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2162.g3d78f5ba 2024-03-18T15:02:11.214 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/ 2024-03-18T15:02:11.214 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2162.g3d78f5ba 2024-03-18T15:02:11.275 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/ 2024-03-18T15:02:11.275 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2162.g3d78f5ba 2024-03-18T15:02:11.532 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-18T15:02:11.532 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-03-18T15:02:11.532 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-18T15:02:11.564 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-18T15:02:11.565 DEBUG:teuthology.orchestra.run.smithi104:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-18T15:02:11.567 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-18T15:02:11.567 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-03-18T15:02:11.567 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-18T15:02:11.616 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-18T15:02:11.616 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-18T15:02:11.617 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-18T15:02:11.635 DEBUG:teuthology.orchestra.run.smithi104:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-18T15:02:11.648 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-18T15:02:11.649 DEBUG:teuthology.orchestra.run.smithi033:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-18T15:02:11.672 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-18T15:02:11.673 DEBUG:teuthology.orchestra.run.smithi019:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-18T15:02:11.716 DEBUG:teuthology.orchestra.run.smithi104:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-18T15:02:11.719 DEBUG:teuthology.orchestra.run.smithi033:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-18T15:02:11.783 INFO:teuthology.orchestra.run.smithi104.stdout:check_obsoletes = 1 2024-03-18T15:02:11.785 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum clean all 2024-03-18T15:02:11.806 DEBUG:teuthology.orchestra.run.smithi033:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-18T15:02:11.816 DEBUG:teuthology.orchestra.run.smithi019:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-18T15:02:11.836 INFO:teuthology.orchestra.run.smithi033.stdout:check_obsoletes = 1 2024-03-18T15:02:11.837 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum clean all 2024-03-18T15:02:11.958 DEBUG:teuthology.orchestra.run.smithi019:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-18T15:02:12.018 INFO:teuthology.orchestra.run.smithi104.stdout:74 files removed 2024-03-18T15:02:12.044 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-18T15:02:12.055 INFO:teuthology.orchestra.run.smithi019.stdout:check_obsoletes = 1 2024-03-18T15:02:12.060 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum clean all 2024-03-18T15:02:12.083 INFO:teuthology.orchestra.run.smithi033.stdout:74 files removed 2024-03-18T15:02:12.109 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-18T15:02:12.433 INFO:teuthology.orchestra.run.smithi019.stdout:74 files removed 2024-03-18T15:02:12.473 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-18T15:02:13.186 INFO:teuthology.orchestra.run.smithi104.stdout:ceph packages for x86_64 97 kB/s | 83 kB 00:00 2024-03-18T15:02:13.298 INFO:teuthology.orchestra.run.smithi033.stdout:ceph packages for x86_64 94 kB/s | 83 kB 00:00 2024-03-18T15:02:13.937 INFO:teuthology.orchestra.run.smithi104.stdout:ceph noarch packages 14 kB/s | 10 kB 00:00 2024-03-18T15:02:13.963 INFO:teuthology.orchestra.run.smithi019.stdout:ceph packages for x86_64 86 kB/s | 83 kB 00:00 2024-03-18T15:02:14.040 INFO:teuthology.orchestra.run.smithi033.stdout:ceph noarch packages 14 kB/s | 10 kB 00:00 2024-03-18T15:02:14.587 INFO:teuthology.orchestra.run.smithi104.stdout:ceph source packages 3.1 kB/s | 1.9 kB 00:00 2024-03-18T15:02:14.590 INFO:teuthology.orchestra.run.smithi033.stdout:ceph source packages 3.7 kB/s | 1.9 kB 00:00 2024-03-18T15:02:14.755 INFO:teuthology.orchestra.run.smithi019.stdout:ceph noarch packages 14 kB/s | 10 kB 00:00 2024-03-18T15:02:15.065 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-03-18T15:02:15.159 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-18T15:02:15.547 INFO:teuthology.orchestra.run.smithi019.stdout:ceph source packages 2.7 kB/s | 1.9 kB 00:00 2024-03-18T15:02:16.258 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-03-18T15:02:16.431 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS-9 - BaseOS 7.6 MB/s | 2.0 MB 00:00 2024-03-18T15:02:16.560 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS-9 - BaseOS 7.6 MB/s | 2.0 MB 00:00 2024-03-18T15:02:18.084 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS-9 - BaseOS 3.9 MB/s | 2.0 MB 00:00 2024-03-18T15:02:18.800 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS Stream 9 - BaseOS 4.2 MB/s | 8.0 MB 00:01 2024-03-18T15:02:19.088 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.0 MB 00:00 2024-03-18T15:02:19.122 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS Stream 9 - BaseOS 3.4 MB/s | 8.0 MB 00:02 2024-03-18T15:02:21.098 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS Stream 9 - AppStream 21 MB/s | 19 MB 00:00 2024-03-18T15:02:22.981 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS Stream 9 - AppStream 7.1 MB/s | 19 MB 00:02 2024-03-18T15:02:25.065 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS Stream 9 - CRB 14 MB/s | 6.1 MB 00:00 2024-03-18T15:02:26.424 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS Stream 9 - Extras packages 74 kB/s | 16 kB 00:00 2024-03-18T15:02:27.382 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.1 MB 00:00 2024-03-18T15:02:29.641 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS Stream 9 - Extras packages 18 kB/s | 16 kB 00:00 2024-03-18T15:02:30.100 INFO:teuthology.orchestra.run.smithi104.stdout:Extra Packages for Enterprise Linux 5.8 MB/s | 21 MB 00:03 2024-03-18T15:02:31.160 INFO:teuthology.orchestra.run.smithi019.stdout:Extra Packages for Enterprise Linux 16 MB/s | 21 MB 00:01 2024-03-18T15:02:35.000 INFO:teuthology.orchestra.run.smithi104.stdout:lab-extras 25 kB/s | 1.7 kB 00:00 2024-03-18T15:02:37.226 INFO:teuthology.orchestra.run.smithi019.stdout:lab-extras 15 kB/s | 1.7 kB 00:00 2024-03-18T15:02:37.471 INFO:teuthology.orchestra.run.smithi104.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-18T15:02:37.472 INFO:teuthology.orchestra.run.smithi104.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-18T15:02:37.486 INFO:teuthology.orchestra.run.smithi104.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-18T15:02:37.487 INFO:teuthology.orchestra.run.smithi104.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-18T15:02:37.488 INFO:teuthology.orchestra.run.smithi104.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-18T15:02:37.489 INFO:teuthology.orchestra.run.smithi104.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-18T15:02:37.491 INFO:teuthology.orchestra.run.smithi104.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-18T15:02:37.494 INFO:teuthology.orchestra.run.smithi104.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-18T15:02:37.495 INFO:teuthology.orchestra.run.smithi104.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-18T15:02:37.576 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-03-18T15:02:37.599 INFO:teuthology.orchestra.run.smithi104.stdout:============================================================================================ 2024-03-18T15:02:37.599 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repository Size 2024-03-18T15:02:37.599 INFO:teuthology.orchestra.run.smithi104.stdout:============================================================================================ 2024-03-18T15:02:37.599 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-03-18T15:02:37.599 INFO:teuthology.orchestra.run.smithi104.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-18T15:02:37.599 INFO:teuthology.orchestra.run.smithi104.stdout: ceph x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 6.6 k 2024-03-18T15:02:37.599 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-base x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 5.4 M 2024-03-18T15:02:37.599 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-fuse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 890 k 2024-03-18T15:02:37.599 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 145 k 2024-03-18T15:02:37.599 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 1.5 M 2024-03-18T15:02:37.599 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 139 k 2024-03-18T15:02:37.600 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 1.8 M 2024-03-18T15:02:37.600 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 7.4 M 2024-03-18T15:02:37.600 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-rook noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 49 k 2024-03-18T15:02:37.600 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-radosgw x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 9.8 M 2024-03-18T15:02:37.600 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-test x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 48 M 2024-03-18T15:02:37.600 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-volume noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 264 k 2024-03-18T15:02:37.600 INFO:teuthology.orchestra.run.smithi104.stdout: cephadm noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 748 k 2024-03-18T15:02:37.600 INFO:teuthology.orchestra.run.smithi104.stdout: cephfs-mirror x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 221 k 2024-03-18T15:02:37.600 INFO:teuthology.orchestra.run.smithi104.stdout: cephfs-shell x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 27 k 2024-03-18T15:02:37.600 INFO:teuthology.orchestra.run.smithi104.stdout: cephfs-top noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 18 k 2024-03-18T15:02:37.600 INFO:teuthology.orchestra.run.smithi104.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-18T15:02:37.600 INFO:teuthology.orchestra.run.smithi104.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-18T15:02:37.600 INFO:teuthology.orchestra.run.smithi104.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-18T15:02:37.601 INFO:teuthology.orchestra.run.smithi104.stdout: libcephfs-devel x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 32 k 2024-03-18T15:02:37.601 INFO:teuthology.orchestra.run.smithi104.stdout: libcephfs2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 747 k 2024-03-18T15:02:37.601 INFO:teuthology.orchestra.run.smithi104.stdout: librados-devel x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 127 k 2024-03-18T15:02:37.601 INFO:teuthology.orchestra.run.smithi104.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-18T15:02:37.601 INFO:teuthology.orchestra.run.smithi104.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-18T15:02:37.601 INFO:teuthology.orchestra.run.smithi104.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-18T15:02:37.601 INFO:teuthology.orchestra.run.smithi104.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-18T15:02:37.601 INFO:teuthology.orchestra.run.smithi104.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-18T15:02:37.601 INFO:teuthology.orchestra.run.smithi104.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-18T15:02:37.601 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cephfs x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 162 k 2024-03-18T15:02:37.601 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rados x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 320 k 2024-03-18T15:02:37.601 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rbd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 298 k 2024-03-18T15:02:37.601 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rgw x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 99 k 2024-03-18T15:02:37.602 INFO:teuthology.orchestra.run.smithi104.stdout: rbd-fuse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 85 k 2024-03-18T15:02:37.602 INFO:teuthology.orchestra.run.smithi104.stdout: rbd-mirror x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.1 M 2024-03-18T15:02:37.602 INFO:teuthology.orchestra.run.smithi104.stdout: rbd-nbd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 170 k 2024-03-18T15:02:37.602 INFO:teuthology.orchestra.run.smithi104.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-18T15:02:37.602 INFO:teuthology.orchestra.run.smithi104.stdout:Upgrading: 2024-03-18T15:02:37.602 INFO:teuthology.orchestra.run.smithi104.stdout: NetworkManager x86_64 1:1.46.0-1.el9 baseos 2.3 M 2024-03-18T15:02:37.602 INFO:teuthology.orchestra.run.smithi104.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 baseos 1.8 M 2024-03-18T15:02:37.602 INFO:teuthology.orchestra.run.smithi104.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 baseos 42 k 2024-03-18T15:02:37.602 INFO:teuthology.orchestra.run.smithi104.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 baseos 248 k 2024-03-18T15:02:37.602 INFO:teuthology.orchestra.run.smithi104.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-18T15:02:37.602 INFO:teuthology.orchestra.run.smithi104.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-18T15:02:37.602 INFO:teuthology.orchestra.run.smithi104.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-18T15:02:37.602 INFO:teuthology.orchestra.run.smithi104.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-18T15:02:37.603 INFO:teuthology.orchestra.run.smithi104.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-18T15:02:37.603 INFO:teuthology.orchestra.run.smithi104.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-18T15:02:37.603 INFO:teuthology.orchestra.run.smithi104.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-18T15:02:37.603 INFO:teuthology.orchestra.run.smithi104.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-18T15:02:37.603 INFO:teuthology.orchestra.run.smithi104.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-18T15:02:37.603 INFO:teuthology.orchestra.run.smithi104.stdout: librados2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.4 M 2024-03-18T15:02:37.603 INFO:teuthology.orchestra.run.smithi104.stdout: librbd1 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.1 M 2024-03-18T15:02:37.603 INFO:teuthology.orchestra.run.smithi104.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-18T15:02:37.603 INFO:teuthology.orchestra.run.smithi104.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-18T15:02:37.603 INFO:teuthology.orchestra.run.smithi104.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-18T15:02:37.603 INFO:teuthology.orchestra.run.smithi104.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-18T15:02:37.603 INFO:teuthology.orchestra.run.smithi104.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-18T15:02:37.603 INFO:teuthology.orchestra.run.smithi104.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-18T15:02:37.603 INFO:teuthology.orchestra.run.smithi104.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-18T15:02:37.604 INFO:teuthology.orchestra.run.smithi104.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-18T15:02:37.604 INFO:teuthology.orchestra.run.smithi104.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-18T15:02:37.604 INFO:teuthology.orchestra.run.smithi104.stdout:Installing dependencies: 2024-03-18T15:02:37.604 INFO:teuthology.orchestra.run.smithi104.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-18T15:02:37.604 INFO:teuthology.orchestra.run.smithi104.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-18T15:02:37.604 INFO:teuthology.orchestra.run.smithi104.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-18T15:02:37.604 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-common x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 21 M 2024-03-18T15:02:37.604 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 34 k 2024-03-18T15:02:37.604 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mds x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 2.4 M 2024-03-18T15:02:37.604 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 248 k 2024-03-18T15:02:37.604 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mon x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 4.7 M 2024-03-18T15:02:37.604 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-osd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 17 M 2024-03-18T15:02:37.605 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 16 k 2024-03-18T15:02:37.605 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-selinux x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 25 k 2024-03-18T15:02:37.605 INFO:teuthology.orchestra.run.smithi104.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-18T15:02:37.605 INFO:teuthology.orchestra.run.smithi104.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-18T15:02:37.605 INFO:teuthology.orchestra.run.smithi104.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-18T15:02:37.605 INFO:teuthology.orchestra.run.smithi104.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-18T15:02:37.605 INFO:teuthology.orchestra.run.smithi104.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-18T15:02:37.605 INFO:teuthology.orchestra.run.smithi104.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-18T15:02:37.605 INFO:teuthology.orchestra.run.smithi104.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-18T15:02:37.605 INFO:teuthology.orchestra.run.smithi104.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-18T15:02:37.605 INFO:teuthology.orchestra.run.smithi104.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-18T15:02:37.605 INFO:teuthology.orchestra.run.smithi104.stdout: libcephsqlite x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 163 k 2024-03-18T15:02:37.605 INFO:teuthology.orchestra.run.smithi104.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-18T15:02:37.605 INFO:teuthology.orchestra.run.smithi104.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-18T15:02:37.606 INFO:teuthology.orchestra.run.smithi104.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-18T15:02:37.606 INFO:teuthology.orchestra.run.smithi104.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-18T15:02:37.606 INFO:teuthology.orchestra.run.smithi104.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-18T15:02:37.606 INFO:teuthology.orchestra.run.smithi104.stdout: libradosstriper1 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 501 k 2024-03-18T15:02:37.606 INFO:teuthology.orchestra.run.smithi104.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-18T15:02:37.606 INFO:teuthology.orchestra.run.smithi104.stdout: librgw2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 5.1 M 2024-03-18T15:02:37.606 INFO:teuthology.orchestra.run.smithi104.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-18T15:02:37.606 INFO:teuthology.orchestra.run.smithi104.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-18T15:02:37.606 INFO:teuthology.orchestra.run.smithi104.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-18T15:02:37.606 INFO:teuthology.orchestra.run.smithi104.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-18T15:02:37.606 INFO:teuthology.orchestra.run.smithi104.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-18T15:02:37.606 INFO:teuthology.orchestra.run.smithi104.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-18T15:02:37.606 INFO:teuthology.orchestra.run.smithi104.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-18T15:02:37.607 INFO:teuthology.orchestra.run.smithi104.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-18T15:02:37.607 INFO:teuthology.orchestra.run.smithi104.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-18T15:02:37.607 INFO:teuthology.orchestra.run.smithi104.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-18T15:02:37.607 INFO:teuthology.orchestra.run.smithi104.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-18T15:02:37.607 INFO:teuthology.orchestra.run.smithi104.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-18T15:02:37.607 INFO:teuthology.orchestra.run.smithi104.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-18T15:02:37.607 INFO:teuthology.orchestra.run.smithi104.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-18T15:02:37.607 INFO:teuthology.orchestra.run.smithi104.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-18T15:02:37.607 INFO:teuthology.orchestra.run.smithi104.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-18T15:02:37.607 INFO:teuthology.orchestra.run.smithi104.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-18T15:02:37.607 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-18T15:02:37.607 INFO:teuthology.orchestra.run.smithi104.stdout: python3-ceph-argparse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 45 k 2024-03-18T15:02:37.607 INFO:teuthology.orchestra.run.smithi104.stdout: python3-ceph-common x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 131 k 2024-03-18T15:02:37.607 INFO:teuthology.orchestra.run.smithi104.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-18T15:02:37.608 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-18T15:02:37.608 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-18T15:02:37.608 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-18T15:02:37.608 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-18T15:02:37.608 INFO:teuthology.orchestra.run.smithi104.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-18T15:02:37.608 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-18T15:02:37.608 INFO:teuthology.orchestra.run.smithi104.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-18T15:02:37.608 INFO:teuthology.orchestra.run.smithi104.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-18T15:02:37.608 INFO:teuthology.orchestra.run.smithi104.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-18T15:02:37.608 INFO:teuthology.orchestra.run.smithi104.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-18T15:02:37.608 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-18T15:02:37.608 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-18T15:02:37.608 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-18T15:02:37.609 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-18T15:02:37.609 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-18T15:02:37.609 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-18T15:02:37.609 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-18T15:02:37.609 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-18T15:02:37.609 INFO:teuthology.orchestra.run.smithi104.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-18T15:02:37.609 INFO:teuthology.orchestra.run.smithi104.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-18T15:02:37.609 INFO:teuthology.orchestra.run.smithi104.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-18T15:02:37.609 INFO:teuthology.orchestra.run.smithi104.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-18T15:02:37.609 INFO:teuthology.orchestra.run.smithi104.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-18T15:02:37.609 INFO:teuthology.orchestra.run.smithi104.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-18T15:02:37.609 INFO:teuthology.orchestra.run.smithi104.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-18T15:02:37.609 INFO:teuthology.orchestra.run.smithi104.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-18T15:02:37.609 INFO:teuthology.orchestra.run.smithi104.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-18T15:02:37.609 INFO:teuthology.orchestra.run.smithi104.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-18T15:02:37.610 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-18T15:02:37.610 INFO:teuthology.orchestra.run.smithi104.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-18T15:02:37.610 INFO:teuthology.orchestra.run.smithi104.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-18T15:02:37.610 INFO:teuthology.orchestra.run.smithi104.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-18T15:02:37.610 INFO:teuthology.orchestra.run.smithi104.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-18T15:02:37.610 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-18T15:02:37.610 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-18T15:02:37.610 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-18T15:02:37.610 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-18T15:02:37.610 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-18T15:02:37.610 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-18T15:02:37.610 INFO:teuthology.orchestra.run.smithi104.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-18T15:02:37.610 INFO:teuthology.orchestra.run.smithi104.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-18T15:02:37.611 INFO:teuthology.orchestra.run.smithi104.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-18T15:02:37.611 INFO:teuthology.orchestra.run.smithi104.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-18T15:02:37.611 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-18T15:02:37.611 INFO:teuthology.orchestra.run.smithi104.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-18T15:02:37.611 INFO:teuthology.orchestra.run.smithi104.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-18T15:02:37.611 INFO:teuthology.orchestra.run.smithi104.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-18T15:02:37.611 INFO:teuthology.orchestra.run.smithi104.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-18T15:02:37.611 INFO:teuthology.orchestra.run.smithi104.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-18T15:02:37.611 INFO:teuthology.orchestra.run.smithi104.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-18T15:02:37.611 INFO:teuthology.orchestra.run.smithi104.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-18T15:02:37.611 INFO:teuthology.orchestra.run.smithi104.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-18T15:02:37.611 INFO:teuthology.orchestra.run.smithi104.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-18T15:02:37.611 INFO:teuthology.orchestra.run.smithi104.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-18T15:02:37.611 INFO:teuthology.orchestra.run.smithi104.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-18T15:02:37.612 INFO:teuthology.orchestra.run.smithi104.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-18T15:02:37.612 INFO:teuthology.orchestra.run.smithi104.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-18T15:02:37.612 INFO:teuthology.orchestra.run.smithi104.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-18T15:02:37.612 INFO:teuthology.orchestra.run.smithi104.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-18T15:02:37.612 INFO:teuthology.orchestra.run.smithi104.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-18T15:02:37.612 INFO:teuthology.orchestra.run.smithi104.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-18T15:02:37.612 INFO:teuthology.orchestra.run.smithi104.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-18T15:02:37.612 INFO:teuthology.orchestra.run.smithi104.stdout:Installing weak dependencies: 2024-03-18T15:02:37.612 INFO:teuthology.orchestra.run.smithi104.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-18T15:02:37.612 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-18T15:02:37.612 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-03-18T15:02:37.612 INFO:teuthology.orchestra.run.smithi104.stdout:============================================================================================ 2024-03-18T15:02:37.612 INFO:teuthology.orchestra.run.smithi104.stdout:Install 150 Packages 2024-03-18T15:02:37.613 INFO:teuthology.orchestra.run.smithi104.stdout:Upgrade 24 Packages 2024-03-18T15:02:37.613 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-18T15:02:37.613 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 224 M 2024-03-18T15:02:37.613 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-03-18T15:02:38.234 INFO:teuthology.orchestra.run.smithi104.stdout:(1/174): ceph-19.0.0-2162.g3d78f5ba.el9.x86_64. 28 kB/s | 6.6 kB 00:00 2024-03-18T15:02:38.551 INFO:teuthology.orchestra.run.smithi104.stdout:(2/174): ceph-fuse-19.0.0-2162.g3d78f5ba.el9.x8 2.7 MB/s | 890 kB 00:00 2024-03-18T15:02:38.626 INFO:teuthology.orchestra.run.smithi104.stdout:(3/174): ceph-immutable-object-cache-19.0.0-216 1.9 MB/s | 145 kB 00:00 2024-03-18T15:02:38.743 INFO:teuthology.orchestra.run.smithi104.stdout:(4/174): ceph-base-19.0.0-2162.g3d78f5ba.el9.x8 7.2 MB/s | 5.4 MB 00:00 2024-03-18T15:02:38.918 INFO:teuthology.orchestra.run.smithi104.stdout:(5/174): ceph-mgr-19.0.0-2162.g3d78f5ba.el9.x86 8.5 MB/s | 1.5 MB 00:00 2024-03-18T15:02:39.318 INFO:teuthology.orchestra.run.smithi104.stdout:(6/174): ceph-mds-19.0.0-2162.g3d78f5ba.el9.x86 3.4 MB/s | 2.4 MB 00:00 2024-03-18T15:02:39.419 INFO:teuthology.orchestra.run.smithi104.stdout:(7/174): ceph-mon-19.0.0-2162.g3d78f5ba.el9.x86 9.4 MB/s | 4.7 MB 00:00 2024-03-18T15:02:39.919 INFO:teuthology.orchestra.run.smithi104.stdout:(8/174): ceph-common-19.0.0-2162.g3d78f5ba.el9. 11 MB/s | 21 MB 00:01 2024-03-18T15:02:39.961 INFO:teuthology.orchestra.run.smithi104.stdout:(9/174): ceph-selinux-19.0.0-2162.g3d78f5ba.el9 606 kB/s | 25 kB 00:00 2024-03-18T15:02:40.079 INFO:teuthology.orchestra.run.smithi019.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-18T15:02:40.080 INFO:teuthology.orchestra.run.smithi019.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-18T15:02:40.102 INFO:teuthology.orchestra.run.smithi019.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-18T15:02:40.104 INFO:teuthology.orchestra.run.smithi019.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-18T15:02:40.105 INFO:teuthology.orchestra.run.smithi019.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-18T15:02:40.107 INFO:teuthology.orchestra.run.smithi019.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-18T15:02:40.109 INFO:teuthology.orchestra.run.smithi019.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-18T15:02:40.113 INFO:teuthology.orchestra.run.smithi019.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-18T15:02:40.114 INFO:teuthology.orchestra.run.smithi019.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-18T15:02:40.209 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-03-18T15:02:40.231 INFO:teuthology.orchestra.run.smithi019.stdout:============================================================================================ 2024-03-18T15:02:40.231 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2024-03-18T15:02:40.231 INFO:teuthology.orchestra.run.smithi019.stdout:============================================================================================ 2024-03-18T15:02:40.232 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-03-18T15:02:40.232 INFO:teuthology.orchestra.run.smithi019.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-18T15:02:40.232 INFO:teuthology.orchestra.run.smithi019.stdout: ceph x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 6.6 k 2024-03-18T15:02:40.232 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-base x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 5.4 M 2024-03-18T15:02:40.232 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-fuse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 890 k 2024-03-18T15:02:40.232 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 145 k 2024-03-18T15:02:40.232 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 1.5 M 2024-03-18T15:02:40.232 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 139 k 2024-03-18T15:02:40.232 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 1.8 M 2024-03-18T15:02:40.232 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 7.4 M 2024-03-18T15:02:40.232 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-rook noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 49 k 2024-03-18T15:02:40.232 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-radosgw x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 9.8 M 2024-03-18T15:02:40.232 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-test x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 48 M 2024-03-18T15:02:40.233 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-volume noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 264 k 2024-03-18T15:02:40.233 INFO:teuthology.orchestra.run.smithi019.stdout: cephadm noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 748 k 2024-03-18T15:02:40.233 INFO:teuthology.orchestra.run.smithi019.stdout: cephfs-mirror x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 221 k 2024-03-18T15:02:40.233 INFO:teuthology.orchestra.run.smithi019.stdout: cephfs-shell x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 27 k 2024-03-18T15:02:40.233 INFO:teuthology.orchestra.run.smithi019.stdout: cephfs-top noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 18 k 2024-03-18T15:02:40.233 INFO:teuthology.orchestra.run.smithi019.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-18T15:02:40.233 INFO:teuthology.orchestra.run.smithi019.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-18T15:02:40.233 INFO:teuthology.orchestra.run.smithi019.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-18T15:02:40.233 INFO:teuthology.orchestra.run.smithi019.stdout: libcephfs-devel x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 32 k 2024-03-18T15:02:40.233 INFO:teuthology.orchestra.run.smithi019.stdout: libcephfs2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 747 k 2024-03-18T15:02:40.233 INFO:teuthology.orchestra.run.smithi019.stdout: librados-devel x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 127 k 2024-03-18T15:02:40.234 INFO:teuthology.orchestra.run.smithi019.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-18T15:02:40.234 INFO:teuthology.orchestra.run.smithi019.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-18T15:02:40.234 INFO:teuthology.orchestra.run.smithi019.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-18T15:02:40.234 INFO:teuthology.orchestra.run.smithi019.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-18T15:02:40.234 INFO:teuthology.orchestra.run.smithi019.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-18T15:02:40.234 INFO:teuthology.orchestra.run.smithi019.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-18T15:02:40.234 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cephfs x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 162 k 2024-03-18T15:02:40.234 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rados x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 320 k 2024-03-18T15:02:40.234 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rbd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 298 k 2024-03-18T15:02:40.234 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rgw x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 99 k 2024-03-18T15:02:40.234 INFO:teuthology.orchestra.run.smithi019.stdout: rbd-fuse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 85 k 2024-03-18T15:02:40.234 INFO:teuthology.orchestra.run.smithi019.stdout: rbd-mirror x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.1 M 2024-03-18T15:02:40.234 INFO:teuthology.orchestra.run.smithi019.stdout: rbd-nbd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 170 k 2024-03-18T15:02:40.235 INFO:teuthology.orchestra.run.smithi019.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-18T15:02:40.235 INFO:teuthology.orchestra.run.smithi019.stdout:Upgrading: 2024-03-18T15:02:40.235 INFO:teuthology.orchestra.run.smithi019.stdout: NetworkManager x86_64 1:1.46.0-1.el9 baseos 2.3 M 2024-03-18T15:02:40.235 INFO:teuthology.orchestra.run.smithi019.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 baseos 1.8 M 2024-03-18T15:02:40.235 INFO:teuthology.orchestra.run.smithi019.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 baseos 42 k 2024-03-18T15:02:40.235 INFO:teuthology.orchestra.run.smithi019.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 baseos 248 k 2024-03-18T15:02:40.235 INFO:teuthology.orchestra.run.smithi019.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-18T15:02:40.235 INFO:teuthology.orchestra.run.smithi019.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-18T15:02:40.235 INFO:teuthology.orchestra.run.smithi019.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-18T15:02:40.235 INFO:teuthology.orchestra.run.smithi019.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-18T15:02:40.235 INFO:teuthology.orchestra.run.smithi019.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-18T15:02:40.235 INFO:teuthology.orchestra.run.smithi019.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-18T15:02:40.235 INFO:teuthology.orchestra.run.smithi019.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-18T15:02:40.235 INFO:teuthology.orchestra.run.smithi019.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-18T15:02:40.236 INFO:teuthology.orchestra.run.smithi019.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-18T15:02:40.236 INFO:teuthology.orchestra.run.smithi019.stdout: librados2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.4 M 2024-03-18T15:02:40.236 INFO:teuthology.orchestra.run.smithi019.stdout: librbd1 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.1 M 2024-03-18T15:02:40.236 INFO:teuthology.orchestra.run.smithi019.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-18T15:02:40.236 INFO:teuthology.orchestra.run.smithi019.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-18T15:02:40.236 INFO:teuthology.orchestra.run.smithi019.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-18T15:02:40.236 INFO:teuthology.orchestra.run.smithi019.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-18T15:02:40.236 INFO:teuthology.orchestra.run.smithi019.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-18T15:02:40.236 INFO:teuthology.orchestra.run.smithi019.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-18T15:02:40.236 INFO:teuthology.orchestra.run.smithi019.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-18T15:02:40.236 INFO:teuthology.orchestra.run.smithi019.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-18T15:02:40.236 INFO:teuthology.orchestra.run.smithi019.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-18T15:02:40.236 INFO:teuthology.orchestra.run.smithi019.stdout:Installing dependencies: 2024-03-18T15:02:40.236 INFO:teuthology.orchestra.run.smithi019.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-18T15:02:40.237 INFO:teuthology.orchestra.run.smithi019.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-18T15:02:40.237 INFO:teuthology.orchestra.run.smithi019.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-18T15:02:40.237 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-common x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 21 M 2024-03-18T15:02:40.237 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 34 k 2024-03-18T15:02:40.237 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mds x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 2.4 M 2024-03-18T15:02:40.237 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 248 k 2024-03-18T15:02:40.237 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mon x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 4.7 M 2024-03-18T15:02:40.237 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-osd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 17 M 2024-03-18T15:02:40.237 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 16 k 2024-03-18T15:02:40.237 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-selinux x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 25 k 2024-03-18T15:02:40.237 INFO:teuthology.orchestra.run.smithi019.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-18T15:02:40.237 INFO:teuthology.orchestra.run.smithi019.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-18T15:02:40.237 INFO:teuthology.orchestra.run.smithi019.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-18T15:02:40.237 INFO:teuthology.orchestra.run.smithi019.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-18T15:02:40.237 INFO:teuthology.orchestra.run.smithi019.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-18T15:02:40.238 INFO:teuthology.orchestra.run.smithi019.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-18T15:02:40.238 INFO:teuthology.orchestra.run.smithi019.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-18T15:02:40.238 INFO:teuthology.orchestra.run.smithi019.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-18T15:02:40.238 INFO:teuthology.orchestra.run.smithi019.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-18T15:02:40.238 INFO:teuthology.orchestra.run.smithi019.stdout: libcephsqlite x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 163 k 2024-03-18T15:02:40.238 INFO:teuthology.orchestra.run.smithi019.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-18T15:02:40.238 INFO:teuthology.orchestra.run.smithi019.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-18T15:02:40.238 INFO:teuthology.orchestra.run.smithi019.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-18T15:02:40.238 INFO:teuthology.orchestra.run.smithi019.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-18T15:02:40.238 INFO:teuthology.orchestra.run.smithi019.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-18T15:02:40.238 INFO:teuthology.orchestra.run.smithi019.stdout: libradosstriper1 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 501 k 2024-03-18T15:02:40.238 INFO:teuthology.orchestra.run.smithi019.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-18T15:02:40.238 INFO:teuthology.orchestra.run.smithi019.stdout: librgw2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 5.1 M 2024-03-18T15:02:40.238 INFO:teuthology.orchestra.run.smithi019.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-18T15:02:40.238 INFO:teuthology.orchestra.run.smithi019.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-18T15:02:40.239 INFO:teuthology.orchestra.run.smithi019.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-18T15:02:40.239 INFO:teuthology.orchestra.run.smithi019.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-18T15:02:40.239 INFO:teuthology.orchestra.run.smithi019.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-18T15:02:40.239 INFO:teuthology.orchestra.run.smithi019.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-18T15:02:40.239 INFO:teuthology.orchestra.run.smithi019.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-18T15:02:40.239 INFO:teuthology.orchestra.run.smithi019.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-18T15:02:40.239 INFO:teuthology.orchestra.run.smithi019.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-18T15:02:40.239 INFO:teuthology.orchestra.run.smithi019.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-18T15:02:40.239 INFO:teuthology.orchestra.run.smithi019.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-18T15:02:40.239 INFO:teuthology.orchestra.run.smithi019.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-18T15:02:40.239 INFO:teuthology.orchestra.run.smithi019.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-18T15:02:40.239 INFO:teuthology.orchestra.run.smithi019.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-18T15:02:40.239 INFO:teuthology.orchestra.run.smithi019.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-18T15:02:40.239 INFO:teuthology.orchestra.run.smithi019.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-18T15:02:40.240 INFO:teuthology.orchestra.run.smithi019.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-18T15:02:40.240 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-18T15:02:40.240 INFO:teuthology.orchestra.run.smithi019.stdout: python3-ceph-argparse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 45 k 2024-03-18T15:02:40.240 INFO:teuthology.orchestra.run.smithi019.stdout: python3-ceph-common x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 131 k 2024-03-18T15:02:40.240 INFO:teuthology.orchestra.run.smithi019.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-18T15:02:40.240 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-18T15:02:40.240 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-18T15:02:40.240 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-18T15:02:40.240 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-18T15:02:40.240 INFO:teuthology.orchestra.run.smithi019.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-18T15:02:40.240 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-18T15:02:40.240 INFO:teuthology.orchestra.run.smithi019.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-18T15:02:40.241 INFO:teuthology.orchestra.run.smithi019.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-18T15:02:40.241 INFO:teuthology.orchestra.run.smithi019.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-18T15:02:40.241 INFO:teuthology.orchestra.run.smithi019.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-18T15:02:40.241 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-18T15:02:40.241 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-18T15:02:40.241 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-18T15:02:40.241 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-18T15:02:40.241 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-18T15:02:40.241 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-18T15:02:40.241 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-18T15:02:40.241 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-18T15:02:40.241 INFO:teuthology.orchestra.run.smithi019.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-18T15:02:40.241 INFO:teuthology.orchestra.run.smithi019.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-18T15:02:40.241 INFO:teuthology.orchestra.run.smithi019.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-18T15:02:40.242 INFO:teuthology.orchestra.run.smithi019.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-18T15:02:40.242 INFO:teuthology.orchestra.run.smithi019.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-18T15:02:40.242 INFO:teuthology.orchestra.run.smithi019.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-18T15:02:40.242 INFO:teuthology.orchestra.run.smithi019.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-18T15:02:40.242 INFO:teuthology.orchestra.run.smithi019.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-18T15:02:40.242 INFO:teuthology.orchestra.run.smithi019.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-18T15:02:40.242 INFO:teuthology.orchestra.run.smithi019.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-18T15:02:40.242 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-18T15:02:40.242 INFO:teuthology.orchestra.run.smithi019.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-18T15:02:40.242 INFO:teuthology.orchestra.run.smithi019.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-18T15:02:40.242 INFO:teuthology.orchestra.run.smithi019.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-18T15:02:40.242 INFO:teuthology.orchestra.run.smithi019.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-18T15:02:40.242 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-18T15:02:40.242 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-18T15:02:40.242 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-18T15:02:40.243 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-18T15:02:40.243 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-18T15:02:40.243 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-18T15:02:40.243 INFO:teuthology.orchestra.run.smithi019.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-18T15:02:40.243 INFO:teuthology.orchestra.run.smithi019.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-18T15:02:40.243 INFO:teuthology.orchestra.run.smithi019.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-18T15:02:40.243 INFO:teuthology.orchestra.run.smithi019.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-18T15:02:40.243 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-18T15:02:40.243 INFO:teuthology.orchestra.run.smithi019.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-18T15:02:40.243 INFO:teuthology.orchestra.run.smithi019.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-18T15:02:40.243 INFO:teuthology.orchestra.run.smithi019.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-18T15:02:40.243 INFO:teuthology.orchestra.run.smithi019.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-18T15:02:40.244 INFO:teuthology.orchestra.run.smithi019.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-18T15:02:40.244 INFO:teuthology.orchestra.run.smithi019.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-18T15:02:40.244 INFO:teuthology.orchestra.run.smithi019.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-18T15:02:40.244 INFO:teuthology.orchestra.run.smithi019.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-18T15:02:40.244 INFO:teuthology.orchestra.run.smithi019.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-18T15:02:40.244 INFO:teuthology.orchestra.run.smithi019.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-18T15:02:40.244 INFO:teuthology.orchestra.run.smithi019.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-18T15:02:40.244 INFO:teuthology.orchestra.run.smithi019.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-18T15:02:40.244 INFO:teuthology.orchestra.run.smithi019.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-18T15:02:40.244 INFO:teuthology.orchestra.run.smithi019.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-18T15:02:40.244 INFO:teuthology.orchestra.run.smithi019.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-18T15:02:40.244 INFO:teuthology.orchestra.run.smithi019.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-18T15:02:40.244 INFO:teuthology.orchestra.run.smithi019.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-18T15:02:40.244 INFO:teuthology.orchestra.run.smithi019.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-18T15:02:40.245 INFO:teuthology.orchestra.run.smithi019.stdout:Installing weak dependencies: 2024-03-18T15:02:40.245 INFO:teuthology.orchestra.run.smithi019.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-18T15:02:40.245 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-18T15:02:40.245 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-03-18T15:02:40.245 INFO:teuthology.orchestra.run.smithi019.stdout:============================================================================================ 2024-03-18T15:02:40.245 INFO:teuthology.orchestra.run.smithi019.stdout:Install 150 Packages 2024-03-18T15:02:40.245 INFO:teuthology.orchestra.run.smithi019.stdout:Upgrade 24 Packages 2024-03-18T15:02:40.245 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-18T15:02:40.246 INFO:teuthology.orchestra.run.smithi104.stdout:(10/174): ceph-radosgw-19.0.0-2162.g3d78f5ba.el 12 MB/s | 9.8 MB 00:00 2024-03-18T15:02:40.246 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 224 M 2024-03-18T15:02:40.246 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-03-18T15:02:40.311 INFO:teuthology.orchestra.run.smithi104.stdout:(11/174): cephfs-mirror-19.0.0-2162.g3d78f5ba.e 3.2 MB/s | 221 kB 00:00 2024-03-18T15:02:40.370 INFO:teuthology.orchestra.run.smithi104.stdout:(12/174): cephfs-shell-19.0.0-2162.g3d78f5ba.el 466 kB/s | 27 kB 00:00 2024-03-18T15:02:40.420 INFO:teuthology.orchestra.run.smithi104.stdout:(13/174): libcephfs-devel-19.0.0-2162.g3d78f5ba 634 kB/s | 32 kB 00:00 2024-03-18T15:02:40.553 INFO:teuthology.orchestra.run.smithi104.stdout:(14/174): libcephfs2-19.0.0-2162.g3d78f5ba.el9. 5.5 MB/s | 747 kB 00:00 2024-03-18T15:02:40.629 INFO:teuthology.orchestra.run.smithi104.stdout:(15/174): libcephsqlite-19.0.0-2162.g3d78f5ba.e 2.1 MB/s | 163 kB 00:00 2024-03-18T15:02:40.721 INFO:teuthology.orchestra.run.smithi104.stdout:(16/174): librados-devel-19.0.0-2162.g3d78f5ba. 1.4 MB/s | 127 kB 00:00 2024-03-18T15:02:40.846 INFO:teuthology.orchestra.run.smithi104.stdout:(17/174): libradosstriper1-19.0.0-2162.g3d78f5b 3.9 MB/s | 501 kB 00:00 2024-03-18T15:02:41.404 INFO:teuthology.orchestra.run.smithi019.stdout:(1/174): ceph-19.0.0-2162.g3d78f5ba.el9.x86_64. 13 kB/s | 6.6 kB 00:00 2024-03-18T15:02:41.530 INFO:teuthology.orchestra.run.smithi104.stdout:(18/174): librgw2-19.0.0-2162.g3d78f5ba.el9.x86 7.5 MB/s | 5.1 MB 00:00 2024-03-18T15:02:41.605 INFO:teuthology.orchestra.run.smithi104.stdout:(19/174): python3-ceph-argparse-19.0.0-2162.g3d 602 kB/s | 45 kB 00:00 2024-03-18T15:02:41.697 INFO:teuthology.orchestra.run.smithi104.stdout:(20/174): python3-ceph-common-19.0.0-2162.g3d78 1.4 MB/s | 131 kB 00:00 2024-03-18T15:02:41.789 INFO:teuthology.orchestra.run.smithi104.stdout:(21/174): python3-cephfs-19.0.0-2162.g3d78f5ba. 1.7 MB/s | 162 kB 00:00 2024-03-18T15:02:41.889 INFO:teuthology.orchestra.run.smithi104.stdout:(22/174): python3-rados-19.0.0-2162.g3d78f5ba.e 3.1 MB/s | 320 kB 00:00 2024-03-18T15:02:41.989 INFO:teuthology.orchestra.run.smithi104.stdout:(23/174): python3-rbd-19.0.0-2162.g3d78f5ba.el9 2.9 MB/s | 298 kB 00:00 2024-03-18T15:02:42.028 INFO:teuthology.orchestra.run.smithi019.stdout:(2/174): ceph-fuse-19.0.0-2162.g3d78f5ba.el9.x8 1.4 MB/s | 890 kB 00:00 2024-03-18T15:02:42.081 INFO:teuthology.orchestra.run.smithi104.stdout:(24/174): python3-rgw-19.0.0-2162.g3d78f5ba.el9 1.1 MB/s | 99 kB 00:00 2024-03-18T15:02:42.145 INFO:teuthology.orchestra.run.smithi019.stdout:(3/174): ceph-immutable-object-cache-19.0.0-216 1.2 MB/s | 145 kB 00:00 2024-03-18T15:02:42.174 INFO:teuthology.orchestra.run.smithi104.stdout:(25/174): rbd-fuse-19.0.0-2162.g3d78f5ba.el9.x8 928 kB/s | 85 kB 00:00 2024-03-18T15:02:42.615 INFO:teuthology.orchestra.run.smithi104.stdout:(26/174): ceph-osd-19.0.0-2162.g3d78f5ba.el9.x8 5.1 MB/s | 17 MB 00:03 2024-03-18T15:02:42.724 INFO:teuthology.orchestra.run.smithi104.stdout:(27/174): rbd-nbd-19.0.0-2162.g3d78f5ba.el9.x86 1.5 MB/s | 170 kB 00:00 2024-03-18T15:02:42.849 INFO:teuthology.orchestra.run.smithi104.stdout:(28/174): rbd-mirror-19.0.0-2162.g3d78f5ba.el9. 4.6 MB/s | 3.1 MB 00:00 2024-03-18T15:02:42.874 INFO:teuthology.orchestra.run.smithi104.stdout:(29/174): ceph-grafana-dashboards-19.0.0-2162.g 226 kB/s | 34 kB 00:00 2024-03-18T15:02:42.916 INFO:teuthology.orchestra.run.smithi104.stdout:(30/174): ceph-mgr-cephadm-19.0.0-2162.g3d78f5b 2.0 MB/s | 139 kB 00:00 2024-03-18T15:02:42.920 INFO:teuthology.orchestra.run.smithi019.stdout:(4/174): ceph-mds-19.0.0-2162.g3d78f5ba.el9.x86 3.0 MB/s | 2.4 MB 00:00 2024-03-18T15:02:43.245 INFO:teuthology.orchestra.run.smithi019.stdout:(5/174): ceph-base-19.0.0-2162.g3d78f5ba.el9.x8 2.3 MB/s | 5.4 MB 00:02 2024-03-18T15:02:43.313 INFO:teuthology.orchestra.run.smithi019.stdout:(6/174): ceph-mgr-19.0.0-2162.g3d78f5ba.el9.x86 3.8 MB/s | 1.5 MB 00:00 2024-03-18T15:02:43.509 INFO:teuthology.orchestra.run.smithi104.stdout:(31/174): ceph-test-19.0.0-2162.g3d78f5ba.el9.x 13 MB/s | 48 MB 00:03 2024-03-18T15:02:43.567 INFO:teuthology.orchestra.run.smithi104.stdout:(32/174): ceph-mgr-dashboard-19.0.0-2162.g3d78f 2.6 MB/s | 1.8 MB 00:00 2024-03-18T15:02:43.592 INFO:teuthology.orchestra.run.smithi104.stdout:(33/174): ceph-mgr-modules-core-19.0.0-2162.g3d 2.9 MB/s | 248 kB 00:00 2024-03-18T15:02:43.651 INFO:teuthology.orchestra.run.smithi104.stdout:(34/174): ceph-mgr-rook-19.0.0-2162.g3d78f5ba.e 592 kB/s | 49 kB 00:00 2024-03-18T15:02:43.676 INFO:teuthology.orchestra.run.smithi104.stdout:(35/174): ceph-prometheus-alerts-19.0.0-2162.g3 196 kB/s | 16 kB 00:00 2024-03-18T15:02:43.826 INFO:teuthology.orchestra.run.smithi104.stdout:(36/174): ceph-volume-19.0.0-2162.g3d78f5ba.el9 1.5 MB/s | 264 kB 00:00 2024-03-18T15:02:43.851 INFO:teuthology.orchestra.run.smithi104.stdout:(37/174): cephadm-19.0.0-2162.g3d78f5ba.el9.noa 4.2 MB/s | 748 kB 00:00 2024-03-18T15:02:43.910 INFO:teuthology.orchestra.run.smithi104.stdout:(38/174): cephfs-top-19.0.0-2162.g3d78f5ba.el9. 218 kB/s | 18 kB 00:00 2024-03-18T15:02:43.985 INFO:teuthology.orchestra.run.smithi104.stdout:(39/174): automake-1.16.2-8.el9.noarch.rpm 4.8 MB/s | 662 kB 00:00 2024-03-18T15:02:44.019 INFO:teuthology.orchestra.run.smithi104.stdout:(40/174): bison-3.7.4-5.el9.x86_64.rpm 8.4 MB/s | 920 kB 00:00 2024-03-18T15:02:44.044 INFO:teuthology.orchestra.run.smithi104.stdout:(41/174): boost-program-options-1.75.0-8.el9.x8 1.8 MB/s | 106 kB 00:00 2024-03-18T15:02:44.078 INFO:teuthology.orchestra.run.smithi104.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 5.2 MB/s | 307 kB 00:00 2024-03-18T15:02:44.098 INFO:teuthology.orchestra.run.smithi104.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 599 kB/s | 32 kB 00:00 2024-03-18T15:02:44.190 INFO:teuthology.orchestra.run.smithi104.stdout:(44/174): ceph-mgr-diskprediction-local-19.0.0- 5.8 MB/s | 7.4 MB 00:01 2024-03-18T15:02:44.240 INFO:teuthology.orchestra.run.smithi104.stdout:(45/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 18 MB/s | 3.0 MB 00:00 2024-03-18T15:02:44.271 INFO:teuthology.orchestra.run.smithi104.stdout:(46/174): flexiblas-openblas-openmp-3.0.4-8.el9 101 kB/s | 17 kB 00:00 2024-03-18T15:02:44.288 INFO:teuthology.orchestra.run.smithi019.stdout:(7/174): ceph-mon-19.0.0-2162.g3d78f5ba.el9.x86 4.5 MB/s | 4.7 MB 00:01 2024-03-18T15:02:44.299 INFO:teuthology.orchestra.run.smithi104.stdout:(47/174): libaio-devel-0.3.111-13.el9.x86_64.rp 94 kB/s | 10 kB 00:00 2024-03-18T15:02:44.325 INFO:teuthology.orchestra.run.smithi104.stdout:(48/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.9 MB/s | 159 kB 00:00 2024-03-18T15:02:44.350 INFO:teuthology.orchestra.run.smithi104.stdout:(49/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 562 kB/s | 44 kB 00:00 2024-03-18T15:02:44.392 INFO:teuthology.orchestra.run.smithi104.stdout:(50/174): librdkafka-1.6.1-102.el9.x86_64.rpm 7.1 MB/s | 661 kB 00:00 2024-03-18T15:02:44.417 INFO:teuthology.orchestra.run.smithi104.stdout:(51/174): libtool-2.4.6-45.el9.x86_64.rpm 8.6 MB/s | 578 kB 00:00 2024-03-18T15:02:44.442 INFO:teuthology.orchestra.run.smithi104.stdout:(52/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 510 kB/s | 46 kB 00:00 2024-03-18T15:02:44.467 INFO:teuthology.orchestra.run.smithi104.stdout:(53/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.7 MB/s | 282 kB 00:00 2024-03-18T15:02:44.492 INFO:teuthology.orchestra.run.smithi104.stdout:(54/174): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 187 kB 00:00 2024-03-18T15:02:44.518 INFO:teuthology.orchestra.run.smithi104.stdout:(55/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.8 MB/s | 217 kB 00:00 2024-03-18T15:02:44.552 INFO:teuthology.orchestra.run.smithi104.stdout:(56/174): openblas-0.3.21-2.el9.x86_64.rpm 410 kB/s | 34 kB 00:00 2024-03-18T15:02:44.668 INFO:teuthology.orchestra.run.smithi104.stdout:(57/174): postgresql-13.11-1.el9.x86_64.rpm 10 MB/s | 1.5 MB 00:00 2024-03-18T15:02:44.722 INFO:teuthology.orchestra.run.smithi019.stdout:(8/174): ceph-osd-19.0.0-2162.g3d78f5ba.el9.x86 12 MB/s | 17 MB 00:01 2024-03-18T15:02:44.744 INFO:teuthology.orchestra.run.smithi104.stdout:(58/174): openblas-openmp-0.3.21-2.el9.x86_64.r 21 MB/s | 4.7 MB 00:00 2024-03-18T15:02:44.769 INFO:teuthology.orchestra.run.smithi104.stdout:(59/174): postgresql-contrib-13.11-1.el9.x86_64 3.7 MB/s | 811 kB 00:00 2024-03-18T15:02:44.794 INFO:teuthology.orchestra.run.smithi104.stdout:(60/174): postgresql-private-libs-13.11-1.el9.x 1.1 MB/s | 134 kB 00:00 2024-03-18T15:02:44.797 INFO:teuthology.orchestra.run.smithi019.stdout:(9/174): ceph-selinux-19.0.0-2162.g3d78f5ba.el9 341 kB/s | 25 kB 00:00 2024-03-18T15:02:44.911 INFO:teuthology.orchestra.run.smithi104.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 7.1 MB/s | 1.0 MB 00:00 2024-03-18T15:02:44.986 INFO:teuthology.orchestra.run.smithi104.stdout:(62/174): postgresql-server-13.11-1.el9.x86_64. 26 MB/s | 5.7 MB 00:00 2024-03-18T15:02:45.011 INFO:teuthology.orchestra.run.smithi104.stdout:(63/174): python3-attrs-20.3.0-7.el9.noarch.rpm 361 kB/s | 78 kB 00:00 2024-03-18T15:02:45.128 INFO:teuthology.orchestra.run.smithi104.stdout:(64/174): python3-babel-2.9.1-2.el9.noarch.rpm 27 MB/s | 5.8 MB 00:00 2024-03-18T15:02:45.145 INFO:teuthology.orchestra.run.smithi104.stdout:(65/174): python3-jsonpatch-1.21-16.el9.noarch. 186 kB/s | 24 kB 00:00 2024-03-18T15:02:45.162 INFO:teuthology.orchestra.run.smithi104.stdout:(66/174): python3-jsonpointer-2.0-4.el9.noarch. 122 kB/s | 18 kB 00:00 2024-03-18T15:02:45.187 INFO:teuthology.orchestra.run.smithi104.stdout:(67/174): python3-mako-1.1.4-6.el9.noarch.rpm 2.4 MB/s | 144 kB 00:00 2024-03-18T15:02:45.204 INFO:teuthology.orchestra.run.smithi104.stdout:(68/174): python3-markupsafe-1.1.1-12.el9.x86_6 762 kB/s | 32 kB 00:00 2024-03-18T15:02:45.338 INFO:teuthology.orchestra.run.smithi104.stdout:(69/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 27 MB/s | 4.8 MB 00:00 2024-03-18T15:02:45.363 INFO:teuthology.orchestra.run.smithi104.stdout:(70/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.2 MB/s | 217 kB 00:00 2024-03-18T15:02:45.388 INFO:teuthology.orchestra.run.smithi104.stdout:(71/174): python3-oauthlib-3.1.1-5.el9.noarch.r 958 kB/s | 174 kB 00:00 2024-03-18T15:02:45.414 INFO:teuthology.orchestra.run.smithi104.stdout:(72/174): python3-packaging-20.9-5.el9.noarch.r 920 kB/s | 69 kB 00:00 2024-03-18T15:02:45.430 INFO:teuthology.orchestra.run.smithi104.stdout:(73/174): python3-prettytable-0.7.2-27.el9.noar 989 kB/s | 41 kB 00:00 2024-03-18T15:02:45.455 INFO:teuthology.orchestra.run.smithi104.stdout:(74/174): python3-protobuf-3.14.0-13.el9.noarch 3.5 MB/s | 238 kB 00:00 2024-03-18T15:02:45.481 INFO:teuthology.orchestra.run.smithi104.stdout:(75/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2024-03-18T15:02:45.514 INFO:teuthology.orchestra.run.smithi104.stdout:(76/174): python3-pyasn1-modules-0.4.8-6.el9.no 3.5 MB/s | 211 kB 00:00 2024-03-18T15:02:45.531 INFO:teuthology.orchestra.run.smithi104.stdout:(77/174): python3-pytz-2021.1-5.el9.noarch.rpm 632 kB/s | 47 kB 00:00 2024-03-18T15:02:45.548 INFO:teuthology.orchestra.run.smithi104.stdout:(78/174): python3-requests-oauthlib-1.3.0-12.el 644 kB/s | 43 kB 00:00 2024-03-18T15:02:45.607 INFO:teuthology.orchestra.run.smithi104.stdout:(79/174): python3-toml-0.10.2-6.el9.noarch.rpm 516 kB/s | 39 kB 00:00 2024-03-18T15:02:45.715 INFO:teuthology.orchestra.run.smithi104.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 247 kB/s | 41 kB 00:00 2024-03-18T15:02:45.757 INFO:teuthology.orchestra.run.smithi104.stdout:(81/174): socat-1.7.4.1-5.el9.x86_64.rpm 2.0 MB/s | 300 kB 00:00 2024-03-18T15:02:45.908 INFO:teuthology.orchestra.run.smithi104.stdout:(82/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 42 MB/s | 16 MB 00:00 2024-03-18T15:02:45.933 INFO:teuthology.orchestra.run.smithi104.stdout:(83/174): uuid-1.6.2-55.el9.x86_64.rpm 263 kB/s | 56 kB 00:00 2024-03-18T15:02:45.958 INFO:teuthology.orchestra.run.smithi104.stdout:(84/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 315 kB/s | 63 kB 00:00 2024-03-18T15:02:45.983 INFO:teuthology.orchestra.run.smithi104.stdout:(85/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 883 kB/s | 44 kB 00:00 2024-03-18T15:02:46.008 INFO:teuthology.orchestra.run.smithi104.stdout:(86/174): fuse-2.9.9-15.el9.x86_64.rpm 1.0 MB/s | 79 kB 00:00 2024-03-18T15:02:46.034 INFO:teuthology.orchestra.run.smithi104.stdout:(87/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.1 MB/s | 241 kB 00:00 2024-03-18T15:02:46.067 INFO:teuthology.orchestra.run.smithi104.stdout:(88/174): python3-cryptography-36.0.1-4.el9.x86 14 MB/s | 1.1 MB 00:00 2024-03-18T15:02:46.093 INFO:teuthology.orchestra.run.smithi104.stdout:(89/174): python3-ply-3.11-14.el9.noarch.rpm 1.2 MB/s | 103 kB 00:00 2024-03-18T15:02:46.126 INFO:teuthology.orchestra.run.smithi104.stdout:(90/174): python3-pycparser-2.20-6.el9.noarch.r 1.3 MB/s | 124 kB 00:00 2024-03-18T15:02:46.152 INFO:teuthology.orchestra.run.smithi104.stdout:(91/174): python3-urllib3-1.26.5-5.el9.noarch.r 2.2 MB/s | 187 kB 00:00 2024-03-18T15:02:46.772 INFO:teuthology.orchestra.run.smithi019.stdout:(10/174): ceph-radosgw-19.0.0-2162.g3d78f5ba.el 4.0 MB/s | 9.8 MB 00:02 2024-03-18T15:02:46.830 INFO:teuthology.orchestra.run.smithi019.stdout:(11/174): cephfs-mirror-19.0.0-2162.g3d78f5ba.e 3.8 MB/s | 221 kB 00:00 2024-03-18T15:02:47.110 INFO:teuthology.orchestra.run.smithi104.stdout:(92/174): jq-1.6-16.el9.x86_64.rpm 190 kB/s | 187 kB 00:00 2024-03-18T15:02:47.122 INFO:teuthology.orchestra.run.smithi019.stdout:(12/174): ceph-common-19.0.0-2162.g3d78f5ba.el9 3.3 MB/s | 21 MB 00:06 2024-03-18T15:02:47.135 INFO:teuthology.orchestra.run.smithi104.stdout:(93/174): libquadmath-11.4.1-3.el9.x86_64.rpm 199 kB/s | 195 kB 00:00 2024-03-18T15:02:47.147 INFO:teuthology.orchestra.run.smithi019.stdout:(13/174): cephfs-shell-19.0.0-2162.g3d78f5ba.el 86 kB/s | 27 kB 00:00 2024-03-18T15:02:47.198 INFO:teuthology.orchestra.run.smithi019.stdout:(14/174): libcephfs-devel-19.0.0-2162.g3d78f5ba 652 kB/s | 32 kB 00:00 2024-03-18T15:02:47.290 INFO:teuthology.orchestra.run.smithi019.stdout:(15/174): libcephsqlite-19.0.0-2162.g3d78f5ba.e 1.8 MB/s | 163 kB 00:00 2024-03-18T15:02:47.332 INFO:teuthology.orchestra.run.smithi019.stdout:(16/174): libcephfs2-19.0.0-2162.g3d78f5ba.el9. 4.0 MB/s | 747 kB 00:00 2024-03-18T15:02:47.365 INFO:teuthology.orchestra.run.smithi019.stdout:(17/174): librados-devel-19.0.0-2162.g3d78f5ba. 1.7 MB/s | 127 kB 00:00 2024-03-18T15:02:47.461 INFO:teuthology.orchestra.run.smithi104.stdout:(94/174): elfutils-libelf-devel-0.190-2.el9.x86 66 kB/s | 21 kB 00:00 2024-03-18T15:02:47.569 INFO:teuthology.orchestra.run.smithi104.stdout:(95/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 237 kB/s | 25 kB 00:00 2024-03-18T15:02:47.883 INFO:teuthology.orchestra.run.smithi019.stdout:(18/174): ceph-test-19.0.0-2162.g3d78f5ba.el9.x 15 MB/s | 48 MB 00:03 2024-03-18T15:02:47.924 INFO:teuthology.orchestra.run.smithi019.stdout:(19/174): libradosstriper1-19.0.0-2162.g3d78f5b 847 kB/s | 501 kB 00:00 2024-03-18T15:02:47.945 INFO:teuthology.orchestra.run.smithi104.stdout:(96/174): libgfortran-11.4.1-3.el9.x86_64.rpm 442 kB/s | 803 kB 00:01 2024-03-18T15:02:47.970 INFO:teuthology.orchestra.run.smithi104.stdout:(97/174): python3-requests-2.25.1-8.el9.noarch. 145 kB/s | 125 kB 00:00 2024-03-18T15:02:47.970 INFO:teuthology.orchestra.run.smithi019.stdout:(20/174): python3-ceph-argparse-19.0.0-2162.g3d 525 kB/s | 45 kB 00:00 2024-03-18T15:02:48.000 INFO:teuthology.orchestra.run.smithi019.stdout:(21/174): python3-ceph-common-19.0.0-2162.g3d78 1.7 MB/s | 131 kB 00:00 2024-03-18T15:02:48.033 INFO:teuthology.orchestra.run.smithi019.stdout:(22/174): python3-cephfs-19.0.0-2162.g3d78f5ba. 2.6 MB/s | 162 kB 00:00 2024-03-18T15:02:48.092 INFO:teuthology.orchestra.run.smithi019.stdout:(23/174): python3-rados-19.0.0-2162.g3d78f5ba.e 3.4 MB/s | 320 kB 00:00 2024-03-18T15:02:48.126 INFO:teuthology.orchestra.run.smithi019.stdout:(24/174): python3-rbd-19.0.0-2162.g3d78f5ba.el9 3.2 MB/s | 298 kB 00:00 2024-03-18T15:02:48.159 INFO:teuthology.orchestra.run.smithi019.stdout:(25/174): python3-rgw-19.0.0-2162.g3d78f5ba.el9 1.5 MB/s | 99 kB 00:00 2024-03-18T15:02:48.176 INFO:teuthology.orchestra.run.smithi019.stdout:(26/174): rbd-fuse-19.0.0-2162.g3d78f5ba.el9.x8 1.7 MB/s | 85 kB 00:00 2024-03-18T15:02:48.226 INFO:teuthology.orchestra.run.smithi019.stdout:(27/174): rbd-nbd-19.0.0-2162.g3d78f5ba.el9.x86 3.4 MB/s | 170 kB 00:00 2024-03-18T15:02:48.268 INFO:teuthology.orchestra.run.smithi019.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2162.g 836 kB/s | 34 kB 00:00 2024-03-18T15:02:48.337 INFO:teuthology.orchestra.run.smithi104.stdout:(98/174): openssl-devel-3.0.7-27.el9.x86_64.rpm 5.4 MB/s | 4.1 MB 00:00 2024-03-18T15:02:48.377 INFO:teuthology.orchestra.run.smithi019.stdout:(29/174): librgw2-19.0.0-2162.g3d78f5ba.el9.x86 5.1 MB/s | 5.1 MB 00:01 2024-03-18T15:02:48.431 INFO:teuthology.orchestra.run.smithi019.stdout:(30/174): ceph-mgr-cephadm-19.0.0-2162.g3d78f5b 861 kB/s | 139 kB 00:00 2024-03-18T15:02:48.445 INFO:teuthology.orchestra.run.smithi104.stdout:(99/174): qatlib-23.11.0-1.el9.x86_64.rpm 2.0 MB/s | 216 kB 00:00 2024-03-18T15:02:48.545 INFO:teuthology.orchestra.run.smithi104.stdout:(100/174): python3-devel-3.9.17-2.el9.x86_64.rp 409 kB/s | 245 kB 00:00 2024-03-18T15:02:48.580 INFO:teuthology.orchestra.run.smithi104.stdout:(101/174): qatlib-service-23.11.0-1.el9.x86_64. 259 kB/s | 35 kB 00:00 2024-03-18T15:02:48.605 INFO:teuthology.orchestra.run.smithi104.stdout:(102/174): python3-jinja2-2.11.3-5.el9.noarch.r 390 kB/s | 248 kB 00:00 2024-03-18T15:02:48.672 INFO:teuthology.orchestra.run.smithi104.stdout:(103/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 353 kB/s | 44 kB 00:00 2024-03-18T15:02:48.705 INFO:teuthology.orchestra.run.smithi104.stdout:(104/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 747 kB/s | 94 kB 00:00 2024-03-18T15:02:48.747 INFO:teuthology.orchestra.run.smithi104.stdout:(105/174): lua-devel-5.4.4-4.el9.x86_64.rpm 157 kB/s | 22 kB 00:00 2024-03-18T15:02:48.803 INFO:teuthology.orchestra.run.smithi019.stdout:(31/174): ceph-mgr-dashboard-19.0.0-2162.g3d78f 4.2 MB/s | 1.8 MB 00:00 2024-03-18T15:02:48.900 INFO:teuthology.orchestra.run.smithi019.stdout:(32/174): rbd-mirror-19.0.0-2162.g3d78f5ba.el9. 4.2 MB/s | 3.1 MB 00:00 2024-03-18T15:02:48.945 INFO:teuthology.orchestra.run.smithi019.stdout:(33/174): ceph-mgr-modules-core-19.0.0-2162.g3d 1.7 MB/s | 248 kB 00:00 2024-03-18T15:02:48.962 INFO:teuthology.orchestra.run.smithi019.stdout:(34/174): ceph-mgr-rook-19.0.0-2162.g3d78f5ba.e 813 kB/s | 49 kB 00:00 2024-03-18T15:02:48.987 INFO:teuthology.orchestra.run.smithi019.stdout:(35/174): ceph-prometheus-alerts-19.0.0-2162.g3 399 kB/s | 16 kB 00:00 2024-03-18T15:02:49.023 INFO:teuthology.orchestra.run.smithi104.stdout:(106/174): protobuf-compiler-3.14.0-13.el9.x86_ 2.4 MB/s | 863 kB 00:00 2024-03-18T15:02:49.121 INFO:teuthology.orchestra.run.smithi019.stdout:(36/174): ceph-mgr-diskprediction-local-19.0.0- 11 MB/s | 7.4 MB 00:00 2024-03-18T15:02:49.154 INFO:teuthology.orchestra.run.smithi019.stdout:(37/174): ceph-volume-19.0.0-2162.g3d78f5ba.el9 1.3 MB/s | 264 kB 00:00 2024-03-18T15:02:49.171 INFO:teuthology.orchestra.run.smithi019.stdout:(38/174): cephfs-top-19.0.0-2162.g3d78f5ba.el9. 371 kB/s | 18 kB 00:00 2024-03-18T15:02:49.240 INFO:teuthology.orchestra.run.smithi104.stdout:(107/174): gperftools-libs-2.9.1-2.el9.x86_64.r 628 kB/s | 309 kB 00:00 2024-03-18T15:02:49.265 INFO:teuthology.orchestra.run.smithi104.stdout:(108/174): grpc-data-1.46.7-10.el9.noarch.rpm 80 kB/s | 19 kB 00:00 2024-03-18T15:02:49.290 INFO:teuthology.orchestra.run.smithi104.stdout:(109/174): abseil-cpp-20211102.0-3.el9.x86_64.r 938 kB/s | 548 kB 00:00 2024-03-18T15:02:49.353 INFO:teuthology.orchestra.run.smithi019.stdout:(39/174): cephadm-19.0.0-2162.g3d78f5ba.el9.noa 2.0 MB/s | 748 kB 00:00 2024-03-18T15:02:49.440 INFO:teuthology.orchestra.run.smithi104.stdout:(110/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 148 kB/s | 26 kB 00:00 2024-03-18T15:02:49.455 INFO:teuthology.orchestra.run.smithi019.stdout:(40/174): automake-1.16.2-8.el9.noarch.rpm 2.2 MB/s | 662 kB 00:00 2024-03-18T15:02:49.457 INFO:teuthology.orchestra.run.smithi104.stdout:(111/174): liboath-2.6.7-2.el9.x86_64.rpm 292 kB/s | 49 kB 00:00 2024-03-18T15:02:49.480 INFO:teuthology.orchestra.run.smithi019.stdout:(41/174): boost-program-options-1.75.0-8.el9.x8 852 kB/s | 106 kB 00:00 2024-03-18T15:02:49.514 INFO:teuthology.orchestra.run.smithi019.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 5.4 MB/s | 307 kB 00:00 2024-03-18T15:02:49.516 INFO:teuthology.orchestra.run.smithi104.stdout:(112/174): libarrow-9.0.0-10.el9.x86_64.rpm 16 MB/s | 4.4 MB 00:00 2024-03-18T15:02:49.540 INFO:teuthology.orchestra.run.smithi019.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 593 kB/s | 32 kB 00:00 2024-03-18T15:02:49.541 INFO:teuthology.orchestra.run.smithi104.stdout:(113/174): libunwind-1.6.2-1.el9.x86_64.rpm 670 kB/s | 67 kB 00:00 2024-03-18T15:02:49.566 INFO:teuthology.orchestra.run.smithi104.stdout:(114/174): luarocks-3.9.2-1.el9.noarch.rpm 1.4 MB/s | 151 kB 00:00 2024-03-18T15:02:49.590 INFO:teuthology.orchestra.run.smithi019.stdout:(44/174): bison-3.7.4-5.el9.x86_64.rpm 2.2 MB/s | 920 kB 00:00 2024-03-18T15:02:49.616 INFO:teuthology.orchestra.run.smithi104.stdout:(115/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 8.2 MB/s | 838 kB 00:00 2024-03-18T15:02:49.642 INFO:teuthology.orchestra.run.smithi104.stdout:(116/174): python3-asyncssh-2.13.2-4.el9.noarch 7.1 MB/s | 548 kB 00:00 2024-03-18T15:02:49.667 INFO:teuthology.orchestra.run.smithi104.stdout:(117/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 433 kB/s | 43 kB 00:00 2024-03-18T15:02:49.682 INFO:teuthology.orchestra.run.smithi019.stdout:(45/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 18 MB/s | 3.0 MB 00:00 2024-03-18T15:02:49.692 INFO:teuthology.orchestra.run.smithi104.stdout:(118/174): python3-cachetools-4.2.4-1.el9.noarc 431 kB/s | 32 kB 00:00 2024-03-18T15:02:49.707 INFO:teuthology.orchestra.run.smithi019.stdout:(46/174): flexiblas-openblas-openmp-3.0.4-8.el9 110 kB/s | 17 kB 00:00 2024-03-18T15:02:49.717 INFO:teuthology.orchestra.run.smithi104.stdout:(119/174): python3-certifi-2021.10.8-2.el9.noar 199 kB/s | 15 kB 00:00 2024-03-18T15:02:49.732 INFO:teuthology.orchestra.run.smithi019.stdout:(47/174): libaio-devel-0.3.111-13.el9.x86_64.rp 79 kB/s | 10 kB 00:00 2024-03-18T15:02:49.743 INFO:teuthology.orchestra.run.smithi104.stdout:(120/174): python3-cheroot-8.6.0-4.el9.noarch.r 2.2 MB/s | 172 kB 00:00 2024-03-18T15:02:49.766 INFO:teuthology.orchestra.run.smithi019.stdout:(48/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.7 MB/s | 159 kB 00:00 2024-03-18T15:02:49.768 INFO:teuthology.orchestra.run.smithi104.stdout:(121/174): python3-cherrypy-18.6.1-2.el9.noarch 4.6 MB/s | 358 kB 00:00 2024-03-18T15:02:49.801 INFO:teuthology.orchestra.run.smithi104.stdout:(122/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.6 MB/s | 303 kB 00:00 2024-03-18T15:02:49.824 INFO:teuthology.orchestra.run.smithi019.stdout:(49/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 382 kB/s | 44 kB 00:00 2024-03-18T15:02:49.827 INFO:teuthology.orchestra.run.smithi104.stdout:(123/174): python3-colorama-0.4.4-12.el9.noarch 424 kB/s | 35 kB 00:00 2024-03-18T15:02:49.852 INFO:teuthology.orchestra.run.smithi104.stdout:(124/174): python3-google-auth-2.28.0-1.el9.noa 2.4 MB/s | 203 kB 00:00 2024-03-18T15:02:49.866 INFO:teuthology.orchestra.run.smithi019.stdout:(50/174): librdkafka-1.6.1-102.el9.x86_64.rpm 4.9 MB/s | 661 kB 00:00 2024-03-18T15:02:49.908 INFO:teuthology.orchestra.run.smithi019.stdout:(51/174): libtool-2.4.6-45.el9.x86_64.rpm 7.4 MB/s | 578 kB 00:00 2024-03-18T15:02:49.931 INFO:teuthology.orchestra.run.smithi104.stdout:(125/174): python3-grpcio-1.46.7-10.el9.x86_64. 16 MB/s | 2.0 MB 00:00 2024-03-18T15:02:49.933 INFO:teuthology.orchestra.run.smithi019.stdout:(52/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 456 kB/s | 46 kB 00:00 2024-03-18T15:02:49.961 INFO:teuthology.orchestra.run.smithi104.stdout:(126/174): python3-grpcio-tools-1.46.7-10.el9.x 1.1 MB/s | 144 kB 00:00 2024-03-18T15:02:49.967 INFO:teuthology.orchestra.run.smithi019.stdout:(53/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.9 MB/s | 282 kB 00:00 2024-03-18T15:02:49.986 INFO:teuthology.orchestra.run.smithi104.stdout:(127/174): python3-jaraco-8.2.1-3.el9.noarch.rp 80 kB/s | 11 kB 00:00 2024-03-18T15:02:49.992 INFO:teuthology.orchestra.run.smithi019.stdout:(54/174): lua-5.4.4-4.el9.x86_64.rpm 3.3 MB/s | 187 kB 00:00 2024-03-18T15:02:50.011 INFO:teuthology.orchestra.run.smithi104.stdout:(128/174): python3-jaraco-classes-3.2.1-5.el9.n 222 kB/s | 18 kB 00:00 2024-03-18T15:02:50.017 INFO:teuthology.orchestra.run.smithi019.stdout:(55/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.7 MB/s | 217 kB 00:00 2024-03-18T15:02:50.043 INFO:teuthology.orchestra.run.smithi019.stdout:(56/174): openblas-0.3.21-2.el9.x86_64.rpm 480 kB/s | 34 kB 00:00 2024-03-18T15:02:50.054 INFO:teuthology.orchestra.run.smithi104.stdout:(129/174): python3-jaraco-collections-3.0.0-8.e 348 kB/s | 23 kB 00:00 2024-03-18T15:02:50.079 INFO:teuthology.orchestra.run.smithi104.stdout:(130/174): python3-jaraco-functools-3.5.0-2.el9 212 kB/s | 19 kB 00:00 2024-03-18T15:02:50.104 INFO:teuthology.orchestra.run.smithi104.stdout:(131/174): python3-jaraco-text-3.2.0-6.el9.noar 213 kB/s | 20 kB 00:00 2024-03-18T15:02:50.163 INFO:teuthology.orchestra.run.smithi104.stdout:(132/174): python3-kubernetes-26.1.0-2.el9.noar 9.4 MB/s | 1.0 MB 00:00 2024-03-18T15:02:50.188 INFO:teuthology.orchestra.run.smithi104.stdout:(133/174): python3-logutils-0.3.5-21.el9.noarch 425 kB/s | 46 kB 00:00 2024-03-18T15:02:50.193 INFO:teuthology.orchestra.run.smithi019.stdout:(57/174): postgresql-13.11-1.el9.x86_64.rpm 8.8 MB/s | 1.5 MB 00:00 2024-03-18T15:02:50.213 INFO:teuthology.orchestra.run.smithi104.stdout:(134/174): python3-more-itertools-8.12.0-2.el9. 725 kB/s | 79 kB 00:00 2024-03-18T15:02:50.238 INFO:teuthology.orchestra.run.smithi104.stdout:(135/174): python3-natsort-7.1.1-5.el9.noarch.r 766 kB/s | 58 kB 00:00 2024-03-18T15:02:50.272 INFO:teuthology.orchestra.run.smithi104.stdout:(136/174): python3-pecan-1.4.2-3.el9.noarch.rpm 3.2 MB/s | 272 kB 00:00 2024-03-18T15:02:50.297 INFO:teuthology.orchestra.run.smithi104.stdout:(137/174): python3-portend-3.1.0-2.el9.noarch.r 196 kB/s | 16 kB 00:00 2024-03-18T15:02:50.318 INFO:teuthology.orchestra.run.smithi019.stdout:(58/174): openblas-openmp-0.3.21-2.el9.x86_64.r 16 MB/s | 4.7 MB 00:00 2024-03-18T15:02:50.323 INFO:teuthology.orchestra.run.smithi104.stdout:(138/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.0 MB/s | 90 kB 00:00 2024-03-18T15:02:50.349 INFO:teuthology.orchestra.run.smithi104.stdout:(139/174): python3-pyperclip-1.8.2-2.el9.noarch 374 kB/s | 28 kB 00:00 2024-03-18T15:02:50.360 INFO:teuthology.orchestra.run.smithi019.stdout:(59/174): postgresql-contrib-13.11-1.el9.x86_64 2.6 MB/s | 811 kB 00:00 2024-03-18T15:02:50.373 INFO:teuthology.orchestra.run.smithi104.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 409 kB/s | 31 kB 00:00 2024-03-18T15:02:50.385 INFO:teuthology.orchestra.run.smithi019.stdout:(60/174): postgresql-private-libs-13.11-1.el9.x 759 kB/s | 134 kB 00:00 2024-03-18T15:02:50.398 INFO:teuthology.orchestra.run.smithi104.stdout:(141/174): python3-routes-2.5.1-5.el9.noarch.rp 2.4 MB/s | 188 kB 00:00 2024-03-18T15:02:50.424 INFO:teuthology.orchestra.run.smithi104.stdout:(142/174): python3-rsa-4.9-2.el9.noarch.rpm 782 kB/s | 59 kB 00:00 2024-03-18T15:02:50.457 INFO:teuthology.orchestra.run.smithi104.stdout:(143/174): python3-tempora-5.0.0-2.el9.noarch.r 428 kB/s | 36 kB 00:00 2024-03-18T15:02:50.499 INFO:teuthology.orchestra.run.smithi104.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 508 kB/s | 51 kB 00:00 2024-03-18T15:02:50.519 INFO:teuthology.orchestra.run.smithi019.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 6.5 MB/s | 1.0 MB 00:00 2024-03-18T15:02:50.524 INFO:teuthology.orchestra.run.smithi104.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-03-18T15:02:50.550 INFO:teuthology.orchestra.run.smithi104.stdout:(146/174): python3-websocket-client-1.2.3-2.el9 974 kB/s | 90 kB 00:00 2024-03-18T15:02:50.600 INFO:teuthology.orchestra.run.smithi104.stdout:(147/174): python3-werkzeug-2.0.3-3.el9.1.noarc 4.2 MB/s | 427 kB 00:00 2024-03-18T15:02:50.625 INFO:teuthology.orchestra.run.smithi104.stdout:(148/174): python3-zc-lockfile-2.0-10.el9.noarc 198 kB/s | 20 kB 00:00 2024-03-18T15:02:50.650 INFO:teuthology.orchestra.run.smithi104.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 1.9 MB/s | 192 kB 00:00 2024-03-18T15:02:50.661 INFO:teuthology.orchestra.run.smithi019.stdout:(62/174): postgresql-server-13.11-1.el9.x86_64. 19 MB/s | 5.7 MB 00:00 2024-03-18T15:02:50.711 INFO:teuthology.orchestra.run.smithi019.stdout:(63/174): python3-attrs-20.3.0-7.el9.noarch.rpm 243 kB/s | 78 kB 00:00 2024-03-18T15:02:50.742 INFO:teuthology.orchestra.run.smithi104.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-03-18T15:02:50.809 INFO:teuthology.orchestra.run.smithi104.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 4.7 MB/s | 321 kB 00:00 2024-03-18T15:02:50.870 INFO:teuthology.orchestra.run.smithi019.stdout:(64/174): python3-babel-2.9.1-2.el9.noarch.rpm 17 MB/s | 5.8 MB 00:00 2024-03-18T15:02:50.886 INFO:teuthology.orchestra.run.smithi019.stdout:(65/174): python3-jsonpatch-1.21-16.el9.noarch. 116 kB/s | 24 kB 00:00 2024-03-18T15:02:50.903 INFO:teuthology.orchestra.run.smithi019.stdout:(66/174): python3-jsonpointer-2.0-4.el9.noarch. 102 kB/s | 18 kB 00:00 2024-03-18T15:02:50.937 INFO:teuthology.orchestra.run.smithi019.stdout:(67/174): python3-mako-1.1.4-6.el9.noarch.rpm 2.9 MB/s | 144 kB 00:00 2024-03-18T15:02:50.962 INFO:teuthology.orchestra.run.smithi019.stdout:(68/174): python3-markupsafe-1.1.1-12.el9.x86_6 426 kB/s | 32 kB 00:00 2024-03-18T15:02:50.984 INFO:teuthology.orchestra.run.smithi104.stdout:(152/174): librbd1-19.0.0-2162.g3d78f5ba.el9.x8 9.4 MB/s | 3.1 MB 00:00 2024-03-18T15:02:51.113 INFO:teuthology.orchestra.run.smithi019.stdout:(69/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 23 MB/s | 4.8 MB 00:00 2024-03-18T15:02:51.171 INFO:teuthology.orchestra.run.smithi019.stdout:(70/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 961 kB/s | 217 kB 00:00 2024-03-18T15:02:51.196 INFO:teuthology.orchestra.run.smithi019.stdout:(71/174): python3-oauthlib-3.1.1-5.el9.noarch.r 807 kB/s | 174 kB 00:00 2024-03-18T15:02:51.222 INFO:teuthology.orchestra.run.smithi019.stdout:(72/174): python3-packaging-20.9-5.el9.noarch.r 655 kB/s | 69 kB 00:00 2024-03-18T15:02:51.255 INFO:teuthology.orchestra.run.smithi019.stdout:(73/174): python3-prettytable-0.7.2-27.el9.noar 724 kB/s | 41 kB 00:00 2024-03-18T15:02:51.281 INFO:teuthology.orchestra.run.smithi019.stdout:(74/174): python3-protobuf-3.14.0-13.el9.noarch 2.8 MB/s | 238 kB 00:00 2024-03-18T15:02:51.306 INFO:teuthology.orchestra.run.smithi019.stdout:(75/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.5 MB/s | 132 kB 00:00 2024-03-18T15:02:51.340 INFO:teuthology.orchestra.run.smithi019.stdout:(76/174): python3-pyasn1-modules-0.4.8-6.el9.no 2.6 MB/s | 211 kB 00:00 2024-03-18T15:02:51.365 INFO:teuthology.orchestra.run.smithi019.stdout:(77/174): python3-pytz-2021.1-5.el9.noarch.rpm 571 kB/s | 47 kB 00:00 2024-03-18T15:02:51.390 INFO:teuthology.orchestra.run.smithi019.stdout:(78/174): python3-requests-oauthlib-1.3.0-12.el 533 kB/s | 43 kB 00:00 2024-03-18T15:02:51.586 INFO:teuthology.orchestra.run.smithi019.stdout:(79/174): python3-toml-0.10.2-6.el9.noarch.rpm 176 kB/s | 39 kB 00:00 2024-03-18T15:02:51.726 INFO:teuthology.orchestra.run.smithi104.stdout:(153/174): librados2-19.0.0-2162.g3d78f5ba.el9. 3.1 MB/s | 3.4 MB 00:01 2024-03-18T15:02:51.849 INFO:teuthology.orchestra.run.smithi019.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 89 kB/s | 41 kB 00:00 2024-03-18T15:02:52.133 INFO:teuthology.orchestra.run.smithi019.stdout:(81/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 20 MB/s | 16 MB 00:00 2024-03-18T15:02:52.166 INFO:teuthology.orchestra.run.smithi019.stdout:(82/174): socat-1.7.4.1-5.el9.x86_64.rpm 531 kB/s | 300 kB 00:00 2024-03-18T15:02:52.191 INFO:teuthology.orchestra.run.smithi019.stdout:(83/174): uuid-1.6.2-55.el9.x86_64.rpm 173 kB/s | 56 kB 00:00 2024-03-18T15:02:52.216 INFO:teuthology.orchestra.run.smithi019.stdout:(84/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 789 kB/s | 63 kB 00:00 2024-03-18T15:02:52.267 INFO:teuthology.orchestra.run.smithi019.stdout:(85/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 598 kB/s | 44 kB 00:00 2024-03-18T15:02:52.317 INFO:teuthology.orchestra.run.smithi019.stdout:(86/174): fuse-2.9.9-15.el9.x86_64.rpm 636 kB/s | 79 kB 00:00 2024-03-18T15:02:52.334 INFO:teuthology.orchestra.run.smithi019.stdout:(87/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.0 MB/s | 241 kB 00:00 2024-03-18T15:02:52.384 INFO:teuthology.orchestra.run.smithi019.stdout:(88/174): python3-cryptography-36.0.1-4.el9.x86 9.8 MB/s | 1.1 MB 00:00 2024-03-18T15:02:52.409 INFO:teuthology.orchestra.run.smithi019.stdout:(89/174): python3-ply-3.11-14.el9.noarch.rpm 1.6 MB/s | 103 kB 00:00 2024-03-18T15:02:52.435 INFO:teuthology.orchestra.run.smithi019.stdout:(90/174): python3-pycparser-2.20-6.el9.noarch.r 1.4 MB/s | 124 kB 00:00 2024-03-18T15:02:52.460 INFO:teuthology.orchestra.run.smithi019.stdout:(91/174): python3-urllib3-1.26.5-5.el9.noarch.r 2.5 MB/s | 187 kB 00:00 2024-03-18T15:02:52.551 INFO:teuthology.orchestra.run.smithi104.stdout:(154/174): NetworkManager-team-1.46.0-1.el9.x86 51 kB/s | 42 kB 00:00 2024-03-18T15:02:53.602 INFO:teuthology.orchestra.run.smithi019.stdout:(92/174): jq-1.6-16.el9.x86_64.rpm 161 kB/s | 187 kB 00:01 2024-03-18T15:02:53.636 INFO:teuthology.orchestra.run.smithi019.stdout:(93/174): libquadmath-11.4.1-3.el9.x86_64.rpm 166 kB/s | 195 kB 00:01 2024-03-18T15:02:53.718 INFO:teuthology.orchestra.run.smithi104.stdout:(155/174): NetworkManager-tui-1.46.0-1.el9.x86_ 213 kB/s | 248 kB 00:01 2024-03-18T15:02:53.769 INFO:teuthology.orchestra.run.smithi019.stdout:(94/174): elfutils-libelf-devel-0.190-2.el9.x86 162 kB/s | 21 kB 00:00 2024-03-18T15:02:53.828 INFO:teuthology.orchestra.run.smithi019.stdout:(95/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 441 kB/s | 25 kB 00:00 2024-03-18T15:02:54.380 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS Stream 9 - AppStream 560 kB/s | 19 MB 00:34 2024-03-18T15:02:54.543 INFO:teuthology.orchestra.run.smithi104.stdout:(156/174): acl-2.3.1-4.el9.x86_64.rpm 86 kB/s | 71 kB 00:00 2024-03-18T15:02:54.570 INFO:teuthology.orchestra.run.smithi019.stdout:(96/174): libgfortran-11.4.1-3.el9.x86_64.rpm 377 kB/s | 803 kB 00:02 2024-03-18T15:02:54.670 INFO:teuthology.orchestra.run.smithi019.stdout:(97/174): openssl-devel-3.0.7-27.el9.x86_64.rpm 4.9 MB/s | 4.1 MB 00:00 2024-03-18T15:02:54.704 INFO:teuthology.orchestra.run.smithi019.stdout:(98/174): python3-requests-2.25.1-8.el9.noarch. 114 kB/s | 125 kB 00:01 2024-03-18T15:02:54.737 INFO:teuthology.orchestra.run.smithi019.stdout:(99/174): python3-devel-3.9.17-2.el9.x86_64.rpm 1.4 MB/s | 245 kB 00:00 2024-03-18T15:02:54.804 INFO:teuthology.orchestra.run.smithi019.stdout:(100/174): qatlib-service-23.11.0-1.el9.x86_64. 528 kB/s | 35 kB 00:00 2024-03-18T15:02:54.827 INFO:teuthology.orchestra.run.smithi104.stdout:(157/174): NetworkManager-libnm-1.46.0-1.el9.x8 484 kB/s | 1.8 MB 00:03 2024-03-18T15:02:54.838 INFO:teuthology.orchestra.run.smithi019.stdout:(101/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-18T15:02:54.896 INFO:teuthology.orchestra.run.smithi019.stdout:(102/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.6 MB/s | 94 kB 00:00 2024-03-18T15:02:54.922 INFO:teuthology.orchestra.run.smithi019.stdout:(103/174): qatlib-23.11.0-1.el9.x86_64.rpm 998 kB/s | 216 kB 00:00 2024-03-18T15:02:54.947 INFO:teuthology.orchestra.run.smithi019.stdout:(104/174): lua-devel-5.4.4-4.el9.x86_64.rpm 452 kB/s | 22 kB 00:00 2024-03-18T15:02:54.981 INFO:teuthology.orchestra.run.smithi019.stdout:(105/174): python3-jinja2-2.11.3-5.el9.noarch.r 808 kB/s | 248 kB 00:00 2024-03-18T15:02:55.056 INFO:teuthology.orchestra.run.smithi019.stdout:(106/174): protobuf-compiler-3.14.0-13.el9.x86_ 6.3 MB/s | 863 kB 00:00 2024-03-18T15:02:55.247 INFO:teuthology.orchestra.run.smithi019.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 102 kB/s | 19 kB 00:00 2024-03-18T15:02:55.294 INFO:teuthology.orchestra.run.smithi104.stdout:(158/174): NetworkManager-1.46.0-1.el9.x86_64.r 523 kB/s | 2.3 MB 00:04 2024-03-18T15:02:55.319 INFO:teuthology.orchestra.run.smithi104.stdout:(159/174): elfutils-debuginfod-client-0.190-2.e 47 kB/s | 36 kB 00:00 2024-03-18T15:02:55.465 INFO:teuthology.orchestra.run.smithi019.stdout:(108/174): gperftools-libs-2.9.1-2.el9.x86_64.r 639 kB/s | 309 kB 00:00 2024-03-18T15:02:55.532 INFO:teuthology.orchestra.run.smithi019.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 395 kB/s | 26 kB 00:00 2024-03-18T15:02:55.607 INFO:teuthology.orchestra.run.smithi019.stdout:(110/174): liboath-2.6.7-2.el9.x86_64.rpm 655 kB/s | 49 kB 00:00 2024-03-18T15:02:55.649 INFO:teuthology.orchestra.run.smithi019.stdout:(111/174): abseil-cpp-20211102.0-3.el9.x86_64.r 782 kB/s | 548 kB 00:00 2024-03-18T15:02:55.699 INFO:teuthology.orchestra.run.smithi019.stdout:(112/174): libunwind-1.6.2-1.el9.x86_64.rpm 738 kB/s | 67 kB 00:00 2024-03-18T15:02:55.766 INFO:teuthology.orchestra.run.smithi019.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 1.3 MB/s | 151 kB 00:00 2024-03-18T15:02:55.878 INFO:teuthology.orchestra.run.smithi104.stdout:(160/174): elfutils-libelf-0.190-2.el9.x86_64.r 188 kB/s | 192 kB 00:01 2024-03-18T15:02:56.003 INFO:teuthology.orchestra.run.smithi104.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 34 kB/s | 23 kB 00:00 2024-03-18T15:02:56.153 INFO:teuthology.orchestra.run.smithi019.stdout:(114/174): python3-asyncssh-2.13.2-4.el9.noarch 1.4 MB/s | 548 kB 00:00 2024-03-18T15:02:56.242 INFO:teuthology.orchestra.run.smithi019.stdout:(115/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 496 kB/s | 43 kB 00:00 2024-03-18T15:02:56.300 INFO:teuthology.orchestra.run.smithi019.stdout:(116/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 1.4 MB/s | 838 kB 00:00 2024-03-18T15:02:56.334 INFO:teuthology.orchestra.run.smithi019.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 352 kB/s | 32 kB 00:00 2024-03-18T15:02:56.393 INFO:teuthology.orchestra.run.smithi019.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 164 kB/s | 15 kB 00:00 2024-03-18T15:02:56.485 INFO:teuthology.orchestra.run.smithi019.stdout:(119/174): python3-cheroot-8.6.0-4.el9.noarch.r 1.1 MB/s | 172 kB 00:00 2024-03-18T15:02:56.503 INFO:teuthology.orchestra.run.smithi104.stdout:(162/174): elfutils-libs-0.190-2.el9.x86_64.rpm 216 kB/s | 256 kB 00:01 2024-03-18T15:02:56.643 INFO:teuthology.orchestra.run.smithi019.stdout:(120/174): python3-cherrypy-18.6.1-2.el9.noarch 1.4 MB/s | 358 kB 00:00 2024-03-18T15:02:56.677 INFO:teuthology.orchestra.run.smithi019.stdout:(121/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 1.5 MB/s | 303 kB 00:00 2024-03-18T15:02:56.719 INFO:teuthology.orchestra.run.smithi019.stdout:(122/174): python3-colorama-0.4.4-12.el9.noarch 474 kB/s | 35 kB 00:00 2024-03-18T15:02:56.895 INFO:teuthology.orchestra.run.smithi104.stdout:(163/174): libblkid-2.37.4-18.el9.x86_64.rpm 107 kB/s | 108 kB 00:01 2024-03-18T15:02:56.919 INFO:teuthology.orchestra.run.smithi019.stdout:(123/174): python3-google-auth-2.28.0-1.el9.noa 840 kB/s | 203 kB 00:00 2024-03-18T15:02:57.069 INFO:teuthology.orchestra.run.smithi019.stdout:(124/174): python3-grpcio-tools-1.46.7-10.el9.x 969 kB/s | 144 kB 00:00 2024-03-18T15:02:57.103 INFO:teuthology.orchestra.run.smithi104.stdout:(164/174): libfdisk-2.37.4-18.el9.x86_64.rpm 141 kB/s | 155 kB 00:01 2024-03-18T15:02:57.153 INFO:teuthology.orchestra.run.smithi019.stdout:(125/174): python3-jaraco-8.2.1-3.el9.noarch.rp 129 kB/s | 11 kB 00:00 2024-03-18T15:02:57.253 INFO:teuthology.orchestra.run.smithi019.stdout:(126/174): python3-jaraco-classes-3.2.1-5.el9.n 178 kB/s | 18 kB 00:00 2024-03-18T15:02:57.337 INFO:teuthology.orchestra.run.smithi019.stdout:(127/174): python3-jaraco-collections-3.0.0-8.e 280 kB/s | 23 kB 00:00 2024-03-18T15:02:57.420 INFO:teuthology.orchestra.run.smithi019.stdout:(128/174): python3-jaraco-functools-3.5.0-2.el9 235 kB/s | 19 kB 00:00 2024-03-18T15:02:57.521 INFO:teuthology.orchestra.run.smithi019.stdout:(129/174): python3-jaraco-text-3.2.0-6.el9.noar 198 kB/s | 20 kB 00:00 2024-03-18T15:02:57.662 INFO:teuthology.orchestra.run.smithi104.stdout:(165/174): libmount-2.37.4-18.el9.x86_64.rpm 117 kB/s | 136 kB 00:01 2024-03-18T15:02:57.847 INFO:teuthology.orchestra.run.smithi019.stdout:(130/174): python3-grpcio-1.46.7-10.el9.x86_64. 1.8 MB/s | 2.0 MB 00:01 2024-03-18T15:02:57.937 INFO:teuthology.orchestra.run.smithi104.stdout:(166/174): libuuid-2.37.4-18.el9.x86_64.rpm 34 kB/s | 29 kB 00:00 2024-03-18T15:02:57.963 INFO:teuthology.orchestra.run.smithi019.stdout:(131/174): libarrow-9.0.0-10.el9.x86_64.rpm 1.6 MB/s | 4.4 MB 00:02 2024-03-18T15:02:58.005 INFO:teuthology.orchestra.run.smithi019.stdout:(132/174): python3-logutils-0.3.5-21.el9.noarch 295 kB/s | 46 kB 00:00 2024-03-18T15:02:58.056 INFO:teuthology.orchestra.run.smithi019.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 868 kB/s | 79 kB 00:00 2024-03-18T15:02:58.098 INFO:teuthology.orchestra.run.smithi019.stdout:(134/174): python3-natsort-7.1.1-5.el9.noarch.r 632 kB/s | 58 kB 00:00 2024-03-18T15:02:58.121 INFO:teuthology.orchestra.run.smithi104.stdout:(167/174): libsmartcols-2.37.4-18.el9.x86_64.rp 52 kB/s | 64 kB 00:01 2024-03-18T15:02:58.173 INFO:teuthology.orchestra.run.smithi019.stdout:(135/174): python3-portend-3.1.0-2.el9.noarch.r 222 kB/s | 16 kB 00:00 2024-03-18T15:02:58.214 INFO:teuthology.orchestra.run.smithi019.stdout:(136/174): python3-kubernetes-26.1.0-2.el9.noar 1.5 MB/s | 1.0 MB 00:00 2024-03-18T15:02:58.248 INFO:teuthology.orchestra.run.smithi019.stdout:(137/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.2 MB/s | 90 kB 00:00 2024-03-18T15:02:58.282 INFO:teuthology.orchestra.run.smithi019.stdout:(138/174): python3-pyperclip-1.8.2-2.el9.noarch 426 kB/s | 28 kB 00:00 2024-03-18T15:02:58.315 INFO:teuthology.orchestra.run.smithi019.stdout:(139/174): python3-pecan-1.4.2-3.el9.noarch.rpm 1.0 MB/s | 272 kB 00:00 2024-03-18T15:02:58.341 INFO:teuthology.orchestra.run.smithi019.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 335 kB/s | 31 kB 00:00 2024-03-18T15:02:58.416 INFO:teuthology.orchestra.run.smithi019.stdout:(141/174): python3-rsa-4.9-2.el9.noarch.rpm 594 kB/s | 59 kB 00:00 2024-03-18T15:02:58.441 INFO:teuthology.orchestra.run.smithi019.stdout:(142/174): python3-tempora-5.0.0-2.el9.noarch.r 359 kB/s | 36 kB 00:00 2024-03-18T15:02:58.466 INFO:teuthology.orchestra.run.smithi019.stdout:(143/174): python3-routes-2.5.1-5.el9.noarch.rp 1.0 MB/s | 188 kB 00:00 2024-03-18T15:02:58.500 INFO:teuthology.orchestra.run.smithi019.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 614 kB/s | 51 kB 00:00 2024-03-18T15:02:58.559 INFO:teuthology.orchestra.run.smithi019.stdout:(145/174): python3-websocket-client-1.2.3-2.el9 979 kB/s | 90 kB 00:00 2024-03-18T15:02:58.659 INFO:teuthology.orchestra.run.smithi019.stdout:(146/174): python3-zc-lockfile-2.0-10.el9.noarc 201 kB/s | 20 kB 00:00 2024-03-18T15:02:58.684 INFO:teuthology.orchestra.run.smithi019.stdout:(147/174): python3-webob-1.8.7-6.el9.noarch.rpm 949 kB/s | 230 kB 00:00 2024-03-18T15:02:58.852 INFO:teuthology.orchestra.run.smithi019.stdout:(148/174): re2-20211101-3.el9.x86_64.rpm 1.0 MB/s | 192 kB 00:00 2024-03-18T15:02:58.952 INFO:teuthology.orchestra.run.smithi019.stdout:(149/174): python3-werkzeug-2.0.3-3.el9.1.noarc 949 kB/s | 427 kB 00:00 2024-03-18T15:02:59.160 INFO:teuthology.orchestra.run.smithi019.stdout:(150/174): librados2-19.0.0-2162.g3d78f5ba.el9. 11 MB/s | 3.4 MB 00:00 2024-03-18T15:02:59.210 INFO:teuthology.orchestra.run.smithi019.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 6.4 MB/s | 321 kB 00:00 2024-03-18T15:02:59.469 INFO:teuthology.orchestra.run.smithi019.stdout:(152/174): librbd1-19.0.0-2162.g3d78f5ba.el9.x8 6.1 MB/s | 3.1 MB 00:00 2024-03-18T15:02:59.769 INFO:teuthology.orchestra.run.smithi019.stdout:(153/174): thrift-0.15.0-2.el9.x86_64.rpm 1.5 MB/s | 1.6 MB 00:01 2024-03-18T15:03:00.104 INFO:teuthology.orchestra.run.smithi104.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 508 kB/s | 1.2 MB 00:02 2024-03-18T15:03:00.462 INFO:teuthology.orchestra.run.smithi019.stdout:(154/174): NetworkManager-team-1.46.0-1.el9.x86 61 kB/s | 42 kB 00:00 2024-03-18T15:03:01.453 INFO:teuthology.orchestra.run.smithi019.stdout:(155/174): NetworkManager-tui-1.46.0-1.el9.x86_ 251 kB/s | 248 kB 00:00 2024-03-18T15:03:01.462 INFO:teuthology.orchestra.run.smithi104.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 343 kB/s | 465 kB 00:01 2024-03-18T15:03:01.679 INFO:teuthology.orchestra.run.smithi104.stdout:(170/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 590 kB/s | 2.2 MB 00:03 2024-03-18T15:03:02.029 INFO:teuthology.orchestra.run.smithi104.stdout:(171/174): util-linux-2.37.4-18.el9.x86_64.rpm 598 kB/s | 2.3 MB 00:03 2024-03-18T15:03:02.113 INFO:teuthology.orchestra.run.smithi104.stdout:(172/174): util-linux-user-2.37.4-18.el9.x86_64 48 kB/s | 31 kB 00:00 2024-03-18T15:03:02.270 INFO:teuthology.orchestra.run.smithi019.stdout:(156/174): acl-2.3.1-4.el9.x86_64.rpm 87 kB/s | 71 kB 00:00 2024-03-18T15:03:02.704 INFO:teuthology.orchestra.run.smithi019.stdout:(157/174): NetworkManager-libnm-1.46.0-1.el9.x8 588 kB/s | 1.8 MB 00:03 2024-03-18T15:03:02.954 INFO:teuthology.orchestra.run.smithi019.stdout:(158/174): elfutils-debuginfod-client-0.190-2.e 53 kB/s | 36 kB 00:00 2024-03-18T15:03:03.121 INFO:teuthology.orchestra.run.smithi019.stdout:(159/174): NetworkManager-1.46.0-1.el9.x86_64.r 600 kB/s | 2.3 MB 00:03 2024-03-18T15:03:03.555 INFO:teuthology.orchestra.run.smithi104.stdout:(173/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 49 kB/s | 74 kB 00:01 2024-03-18T15:03:03.680 INFO:teuthology.orchestra.run.smithi019.stdout:(160/174): elfutils-libelf-0.190-2.el9.x86_64.r 197 kB/s | 192 kB 00:00 2024-03-18T15:03:03.738 INFO:teuthology.orchestra.run.smithi019.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 38 kB/s | 23 kB 00:00 2024-03-18T15:03:03.763 INFO:teuthology.orchestra.run.smithi104.stdout:(174/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 528 kB/s | 1.1 MB 00:02 2024-03-18T15:03:03.769 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-03-18T15:03:03.769 INFO:teuthology.orchestra.run.smithi104.stdout:Total 8.6 MB/s | 224 MB 00:26 2024-03-18T15:03:04.072 INFO:teuthology.orchestra.run.smithi019.stdout:(162/174): elfutils-libs-0.190-2.el9.x86_64.rpm 245 kB/s | 256 kB 00:01 2024-03-18T15:03:04.230 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-03-18T15:03:04.499 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-03-18T15:03:04.500 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-03-18T15:03:04.731 INFO:teuthology.orchestra.run.smithi019.stdout:(163/174): libfdisk-2.37.4-18.el9.x86_64.rpm 156 kB/s | 155 kB 00:00 2024-03-18T15:03:04.789 INFO:teuthology.orchestra.run.smithi019.stdout:(164/174): libblkid-2.37.4-18.el9.x86_64.rpm 98 kB/s | 108 kB 00:01 2024-03-18T15:03:05.214 INFO:teuthology.orchestra.run.smithi019.stdout:(165/174): libmount-2.37.4-18.el9.x86_64.rpm 119 kB/s | 136 kB 00:01 2024-03-18T15:03:05.364 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS Stream 9 - CRB 863 kB/s | 6.1 MB 00:07 2024-03-18T15:03:05.573 INFO:teuthology.orchestra.run.smithi019.stdout:(166/174): libuuid-2.37.4-18.el9.x86_64.rpm 37 kB/s | 29 kB 00:00 2024-03-18T15:03:05.632 INFO:teuthology.orchestra.run.smithi019.stdout:(167/174): libsmartcols-2.37.4-18.el9.x86_64.rp 71 kB/s | 64 kB 00:00 2024-03-18T15:03:06.204 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-03-18T15:03:06.204 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-03-18T15:03:07.632 INFO:teuthology.orchestra.run.smithi019.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 513 kB/s | 1.2 MB 00:02 2024-03-18T15:03:07.673 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS Stream 9 - Extras packages 14 kB/s | 16 kB 00:01 2024-03-18T15:03:08.874 INFO:teuthology.orchestra.run.smithi019.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 375 kB/s | 465 kB 00:01 2024-03-18T15:03:09.316 INFO:teuthology.orchestra.run.smithi019.stdout:(170/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 590 kB/s | 2.2 MB 00:03 2024-03-18T15:03:09.382 INFO:teuthology.orchestra.run.smithi019.stdout:(171/174): util-linux-2.37.4-18.el9.x86_64.rpm 624 kB/s | 2.3 MB 00:03 2024-03-18T15:03:09.483 INFO:teuthology.orchestra.run.smithi019.stdout:(172/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 748 kB/s | 74 kB 00:00 2024-03-18T15:03:09.632 INFO:teuthology.orchestra.run.smithi033.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-03-18T15:03:09.691 INFO:teuthology.orchestra.run.smithi019.stdout:(173/174): util-linux-user-2.37.4-18.el9.x86_64 38 kB/s | 31 kB 00:00 2024-03-18T15:03:11.651 INFO:teuthology.orchestra.run.smithi019.stdout:(174/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 472 kB/s | 1.1 MB 00:02 2024-03-18T15:03:11.661 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-03-18T15:03:11.662 INFO:teuthology.orchestra.run.smithi019.stdout:Total 7.1 MB/s | 224 MB 00:31 2024-03-18T15:03:13.049 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-03-18T15:03:13.166 INFO:teuthology.orchestra.run.smithi104.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-18T15:03:13.173 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-03-18T15:03:13.192 INFO:teuthology.orchestra.run.smithi104.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-18T15:03:13.223 INFO:teuthology.orchestra.run.smithi104.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-18T15:03:13.232 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-18T15:03:13.264 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-18T15:03:13.317 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-18T15:03:13.351 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-18T15:03:13.377 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-18T15:03:13.418 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-18T15:03:13.568 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-03-18T15:03:13.569 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-03-18T15:03:13.737 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-18T15:03:13.794 INFO:teuthology.orchestra.run.smithi104.stdout: Upgrading : librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-18T15:03:13.916 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-18T15:03:13.945 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-18T15:03:14.038 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-18T15:03:14.066 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 12/198 2024-03-18T15:03:14.107 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-18T15:03:14.263 INFO:teuthology.orchestra.run.smithi104.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-18T15:03:14.314 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-18T15:03:14.342 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-18T15:03:14.437 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-18T15:03:14.460 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 17/198 2024-03-18T15:03:14.523 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 17/198 2024-03-18T15:03:14.555 INFO:teuthology.orchestra.run.smithi104.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-18T15:03:14.578 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-18T15:03:14.593 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-18T15:03:14.708 INFO:teuthology.orchestra.run.smithi033.stdout:lab-extras 25 kB/s | 1.7 kB 00:00 2024-03-18T15:03:14.786 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-18T15:03:14.817 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-18T15:03:14.857 INFO:teuthology.orchestra.run.smithi104.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-18T15:03:14.956 INFO:teuthology.orchestra.run.smithi104.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-18T15:03:15.045 INFO:teuthology.orchestra.run.smithi104.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-18T15:03:15.105 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-18T15:03:15.170 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-18T15:03:15.222 INFO:teuthology.orchestra.run.smithi104.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-18T15:03:15.428 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-18T15:03:15.534 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-18T15:03:15.630 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-18T15:03:15.669 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-18T15:03:15.704 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-18T15:03:15.739 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-18T15:03:15.813 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-18T15:03:15.876 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-18T15:03:15.914 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-18T15:03:15.978 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-18T15:03:16.013 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-18T15:03:16.046 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-03-18T15:03:16.047 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-03-18T15:03:16.057 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-18T15:03:16.089 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-18T15:03:16.117 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-18T15:03:16.186 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-18T15:03:16.222 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9. 39/198 2024-03-18T15:03:16.265 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el 40/198 2024-03-18T15:03:16.294 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_6 41/198 2024-03-18T15:03:16.362 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-18T15:03:16.408 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-18T15:03:16.482 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-18T15:03:16.615 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-18T15:03:16.669 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-18T15:03:17.187 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-18T15:03:17.272 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-18T15:03:17.279 INFO:teuthology.orchestra.run.smithi033.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-18T15:03:17.280 INFO:teuthology.orchestra.run.smithi033.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-18T15:03:17.295 INFO:teuthology.orchestra.run.smithi033.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-18T15:03:17.296 INFO:teuthology.orchestra.run.smithi033.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-18T15:03:17.297 INFO:teuthology.orchestra.run.smithi033.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-18T15:03:17.298 INFO:teuthology.orchestra.run.smithi033.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-18T15:03:17.300 INFO:teuthology.orchestra.run.smithi033.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-18T15:03:17.303 INFO:teuthology.orchestra.run.smithi033.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-18T15:03:17.304 INFO:teuthology.orchestra.run.smithi033.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-18T15:03:17.353 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-18T15:03:17.389 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-18T15:03:17.401 INFO:teuthology.orchestra.run.smithi104.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-18T15:03:17.407 INFO:teuthology.orchestra.run.smithi033.stdout:============================================================================================ 2024-03-18T15:03:17.407 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repository Size 2024-03-18T15:03:17.407 INFO:teuthology.orchestra.run.smithi033.stdout:============================================================================================ 2024-03-18T15:03:17.407 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2024-03-18T15:03:17.407 INFO:teuthology.orchestra.run.smithi033.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-18T15:03:17.407 INFO:teuthology.orchestra.run.smithi033.stdout: ceph x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 6.6 k 2024-03-18T15:03:17.407 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-base x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 5.4 M 2024-03-18T15:03:17.407 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-fuse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 890 k 2024-03-18T15:03:17.408 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 145 k 2024-03-18T15:03:17.408 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 1.5 M 2024-03-18T15:03:17.408 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 139 k 2024-03-18T15:03:17.408 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 1.8 M 2024-03-18T15:03:17.408 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 7.4 M 2024-03-18T15:03:17.408 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-rook noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 49 k 2024-03-18T15:03:17.408 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-radosgw x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 9.8 M 2024-03-18T15:03:17.408 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-test x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 48 M 2024-03-18T15:03:17.408 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-volume noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 264 k 2024-03-18T15:03:17.409 INFO:teuthology.orchestra.run.smithi033.stdout: cephadm noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 748 k 2024-03-18T15:03:17.409 INFO:teuthology.orchestra.run.smithi033.stdout: cephfs-mirror x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 221 k 2024-03-18T15:03:17.409 INFO:teuthology.orchestra.run.smithi033.stdout: cephfs-shell x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 27 k 2024-03-18T15:03:17.409 INFO:teuthology.orchestra.run.smithi033.stdout: cephfs-top noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 18 k 2024-03-18T15:03:17.409 INFO:teuthology.orchestra.run.smithi033.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-18T15:03:17.409 INFO:teuthology.orchestra.run.smithi033.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-18T15:03:17.409 INFO:teuthology.orchestra.run.smithi033.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-18T15:03:17.409 INFO:teuthology.orchestra.run.smithi033.stdout: libcephfs-devel x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 32 k 2024-03-18T15:03:17.410 INFO:teuthology.orchestra.run.smithi033.stdout: libcephfs2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 747 k 2024-03-18T15:03:17.410 INFO:teuthology.orchestra.run.smithi033.stdout: librados-devel x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 127 k 2024-03-18T15:03:17.410 INFO:teuthology.orchestra.run.smithi033.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-18T15:03:17.410 INFO:teuthology.orchestra.run.smithi033.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-18T15:03:17.410 INFO:teuthology.orchestra.run.smithi033.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-18T15:03:17.410 INFO:teuthology.orchestra.run.smithi033.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-18T15:03:17.410 INFO:teuthology.orchestra.run.smithi033.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-18T15:03:17.411 INFO:teuthology.orchestra.run.smithi033.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-18T15:03:17.411 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cephfs x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 162 k 2024-03-18T15:03:17.411 INFO:teuthology.orchestra.run.smithi033.stdout: python3-rados x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 320 k 2024-03-18T15:03:17.411 INFO:teuthology.orchestra.run.smithi033.stdout: python3-rbd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 298 k 2024-03-18T15:03:17.411 INFO:teuthology.orchestra.run.smithi033.stdout: python3-rgw x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 99 k 2024-03-18T15:03:17.411 INFO:teuthology.orchestra.run.smithi033.stdout: rbd-fuse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 85 k 2024-03-18T15:03:17.411 INFO:teuthology.orchestra.run.smithi033.stdout: rbd-mirror x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.1 M 2024-03-18T15:03:17.411 INFO:teuthology.orchestra.run.smithi033.stdout: rbd-nbd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 170 k 2024-03-18T15:03:17.411 INFO:teuthology.orchestra.run.smithi033.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-18T15:03:17.412 INFO:teuthology.orchestra.run.smithi033.stdout:Upgrading: 2024-03-18T15:03:17.412 INFO:teuthology.orchestra.run.smithi033.stdout: NetworkManager x86_64 1:1.46.0-1.el9 baseos 2.3 M 2024-03-18T15:03:17.412 INFO:teuthology.orchestra.run.smithi033.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 baseos 1.8 M 2024-03-18T15:03:17.412 INFO:teuthology.orchestra.run.smithi033.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 baseos 42 k 2024-03-18T15:03:17.412 INFO:teuthology.orchestra.run.smithi033.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 baseos 248 k 2024-03-18T15:03:17.412 INFO:teuthology.orchestra.run.smithi033.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-18T15:03:17.412 INFO:teuthology.orchestra.run.smithi033.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-18T15:03:17.412 INFO:teuthology.orchestra.run.smithi033.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-18T15:03:17.412 INFO:teuthology.orchestra.run.smithi033.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-18T15:03:17.413 INFO:teuthology.orchestra.run.smithi033.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-18T15:03:17.413 INFO:teuthology.orchestra.run.smithi033.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-18T15:03:17.413 INFO:teuthology.orchestra.run.smithi033.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-18T15:03:17.413 INFO:teuthology.orchestra.run.smithi033.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-18T15:03:17.413 INFO:teuthology.orchestra.run.smithi033.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-18T15:03:17.413 INFO:teuthology.orchestra.run.smithi033.stdout: librados2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.4 M 2024-03-18T15:03:17.413 INFO:teuthology.orchestra.run.smithi033.stdout: librbd1 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.1 M 2024-03-18T15:03:17.413 INFO:teuthology.orchestra.run.smithi033.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-18T15:03:17.413 INFO:teuthology.orchestra.run.smithi033.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-18T15:03:17.414 INFO:teuthology.orchestra.run.smithi033.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-18T15:03:17.414 INFO:teuthology.orchestra.run.smithi033.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-18T15:03:17.414 INFO:teuthology.orchestra.run.smithi033.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-18T15:03:17.414 INFO:teuthology.orchestra.run.smithi033.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-18T15:03:17.414 INFO:teuthology.orchestra.run.smithi033.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-18T15:03:17.414 INFO:teuthology.orchestra.run.smithi033.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-18T15:03:17.414 INFO:teuthology.orchestra.run.smithi033.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-18T15:03:17.414 INFO:teuthology.orchestra.run.smithi033.stdout:Installing dependencies: 2024-03-18T15:03:17.415 INFO:teuthology.orchestra.run.smithi033.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-18T15:03:17.415 INFO:teuthology.orchestra.run.smithi033.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-18T15:03:17.415 INFO:teuthology.orchestra.run.smithi033.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-18T15:03:17.415 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-common x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 21 M 2024-03-18T15:03:17.415 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 34 k 2024-03-18T15:03:17.415 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mds x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 2.4 M 2024-03-18T15:03:17.415 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 248 k 2024-03-18T15:03:17.415 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mon x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 4.7 M 2024-03-18T15:03:17.415 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-osd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 17 M 2024-03-18T15:03:17.416 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 16 k 2024-03-18T15:03:17.416 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-selinux x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 25 k 2024-03-18T15:03:17.416 INFO:teuthology.orchestra.run.smithi033.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-18T15:03:17.416 INFO:teuthology.orchestra.run.smithi033.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-18T15:03:17.416 INFO:teuthology.orchestra.run.smithi033.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-18T15:03:17.416 INFO:teuthology.orchestra.run.smithi033.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-18T15:03:17.416 INFO:teuthology.orchestra.run.smithi033.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-18T15:03:17.416 INFO:teuthology.orchestra.run.smithi033.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-18T15:03:17.416 INFO:teuthology.orchestra.run.smithi033.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-18T15:03:17.416 INFO:teuthology.orchestra.run.smithi033.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-18T15:03:17.417 INFO:teuthology.orchestra.run.smithi033.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-18T15:03:17.417 INFO:teuthology.orchestra.run.smithi033.stdout: libcephsqlite x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 163 k 2024-03-18T15:03:17.417 INFO:teuthology.orchestra.run.smithi033.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-18T15:03:17.417 INFO:teuthology.orchestra.run.smithi033.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-18T15:03:17.417 INFO:teuthology.orchestra.run.smithi033.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-18T15:03:17.417 INFO:teuthology.orchestra.run.smithi033.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-18T15:03:17.417 INFO:teuthology.orchestra.run.smithi033.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-18T15:03:17.417 INFO:teuthology.orchestra.run.smithi033.stdout: libradosstriper1 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 501 k 2024-03-18T15:03:17.418 INFO:teuthology.orchestra.run.smithi033.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-18T15:03:17.418 INFO:teuthology.orchestra.run.smithi033.stdout: librgw2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 5.1 M 2024-03-18T15:03:17.418 INFO:teuthology.orchestra.run.smithi033.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-18T15:03:17.418 INFO:teuthology.orchestra.run.smithi033.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-18T15:03:17.418 INFO:teuthology.orchestra.run.smithi033.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-18T15:03:17.418 INFO:teuthology.orchestra.run.smithi033.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-18T15:03:17.418 INFO:teuthology.orchestra.run.smithi033.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-18T15:03:17.418 INFO:teuthology.orchestra.run.smithi033.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-18T15:03:17.419 INFO:teuthology.orchestra.run.smithi033.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-18T15:03:17.419 INFO:teuthology.orchestra.run.smithi033.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-18T15:03:17.419 INFO:teuthology.orchestra.run.smithi033.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-18T15:03:17.419 INFO:teuthology.orchestra.run.smithi033.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-18T15:03:17.419 INFO:teuthology.orchestra.run.smithi033.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-18T15:03:17.419 INFO:teuthology.orchestra.run.smithi033.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-18T15:03:17.419 INFO:teuthology.orchestra.run.smithi033.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-18T15:03:17.419 INFO:teuthology.orchestra.run.smithi033.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-18T15:03:17.419 INFO:teuthology.orchestra.run.smithi033.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-18T15:03:17.419 INFO:teuthology.orchestra.run.smithi033.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-18T15:03:17.420 INFO:teuthology.orchestra.run.smithi033.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-18T15:03:17.420 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-18T15:03:17.420 INFO:teuthology.orchestra.run.smithi033.stdout: python3-ceph-argparse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 45 k 2024-03-18T15:03:17.420 INFO:teuthology.orchestra.run.smithi033.stdout: python3-ceph-common x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 131 k 2024-03-18T15:03:17.420 INFO:teuthology.orchestra.run.smithi033.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-18T15:03:17.420 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-18T15:03:17.420 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-18T15:03:17.420 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-18T15:03:17.420 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-18T15:03:17.421 INFO:teuthology.orchestra.run.smithi033.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-18T15:03:17.421 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-18T15:03:17.421 INFO:teuthology.orchestra.run.smithi033.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-18T15:03:17.421 INFO:teuthology.orchestra.run.smithi033.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-18T15:03:17.421 INFO:teuthology.orchestra.run.smithi033.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-18T15:03:17.421 INFO:teuthology.orchestra.run.smithi033.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-18T15:03:17.421 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-18T15:03:17.421 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-18T15:03:17.422 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-18T15:03:17.422 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-18T15:03:17.422 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-18T15:03:17.422 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-18T15:03:17.422 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-18T15:03:17.422 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-18T15:03:17.422 INFO:teuthology.orchestra.run.smithi033.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-18T15:03:17.422 INFO:teuthology.orchestra.run.smithi033.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-18T15:03:17.423 INFO:teuthology.orchestra.run.smithi033.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-18T15:03:17.423 INFO:teuthology.orchestra.run.smithi033.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-18T15:03:17.423 INFO:teuthology.orchestra.run.smithi033.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-18T15:03:17.423 INFO:teuthology.orchestra.run.smithi033.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-18T15:03:17.423 INFO:teuthology.orchestra.run.smithi033.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-18T15:03:17.423 INFO:teuthology.orchestra.run.smithi033.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-18T15:03:17.423 INFO:teuthology.orchestra.run.smithi033.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-18T15:03:17.423 INFO:teuthology.orchestra.run.smithi033.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-18T15:03:17.423 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-18T15:03:17.423 INFO:teuthology.orchestra.run.smithi033.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-18T15:03:17.424 INFO:teuthology.orchestra.run.smithi033.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-18T15:03:17.424 INFO:teuthology.orchestra.run.smithi033.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-18T15:03:17.424 INFO:teuthology.orchestra.run.smithi033.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-18T15:03:17.424 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-18T15:03:17.424 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-18T15:03:17.424 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-18T15:03:17.424 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-18T15:03:17.424 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-18T15:03:17.424 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-18T15:03:17.424 INFO:teuthology.orchestra.run.smithi033.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-18T15:03:17.425 INFO:teuthology.orchestra.run.smithi033.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-18T15:03:17.425 INFO:teuthology.orchestra.run.smithi033.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-18T15:03:17.425 INFO:teuthology.orchestra.run.smithi033.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-18T15:03:17.425 INFO:teuthology.orchestra.run.smithi033.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-18T15:03:17.425 INFO:teuthology.orchestra.run.smithi033.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-18T15:03:17.425 INFO:teuthology.orchestra.run.smithi033.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-18T15:03:17.425 INFO:teuthology.orchestra.run.smithi033.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-18T15:03:17.425 INFO:teuthology.orchestra.run.smithi033.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-18T15:03:17.425 INFO:teuthology.orchestra.run.smithi033.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-18T15:03:17.425 INFO:teuthology.orchestra.run.smithi033.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-18T15:03:17.426 INFO:teuthology.orchestra.run.smithi033.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-18T15:03:17.426 INFO:teuthology.orchestra.run.smithi033.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-18T15:03:17.426 INFO:teuthology.orchestra.run.smithi033.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-18T15:03:17.426 INFO:teuthology.orchestra.run.smithi033.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-18T15:03:17.426 INFO:teuthology.orchestra.run.smithi033.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-18T15:03:17.426 INFO:teuthology.orchestra.run.smithi033.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-18T15:03:17.426 INFO:teuthology.orchestra.run.smithi033.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-18T15:03:17.426 INFO:teuthology.orchestra.run.smithi033.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-18T15:03:17.426 INFO:teuthology.orchestra.run.smithi033.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-18T15:03:17.426 INFO:teuthology.orchestra.run.smithi033.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-18T15:03:17.426 INFO:teuthology.orchestra.run.smithi033.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-18T15:03:17.427 INFO:teuthology.orchestra.run.smithi033.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-18T15:03:17.427 INFO:teuthology.orchestra.run.smithi033.stdout:Installing weak dependencies: 2024-03-18T15:03:17.427 INFO:teuthology.orchestra.run.smithi033.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-18T15:03:17.427 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-18T15:03:17.427 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-03-18T15:03:17.427 INFO:teuthology.orchestra.run.smithi033.stdout:============================================================================================ 2024-03-18T15:03:17.427 INFO:teuthology.orchestra.run.smithi033.stdout:Install 150 Packages 2024-03-18T15:03:17.427 INFO:teuthology.orchestra.run.smithi033.stdout:Upgrade 24 Packages 2024-03-18T15:03:17.427 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-18T15:03:17.428 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-18T15:03:17.428 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 224 M 2024-03-18T15:03:17.429 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-03-18T15:03:17.441 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-18T15:03:17.490 INFO:teuthology.orchestra.run.smithi104.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-18T15:03:17.527 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_6 53/198 2024-03-18T15:03:17.573 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-18T15:03:17.787 INFO:teuthology.orchestra.run.smithi104.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-18T15:03:17.945 INFO:teuthology.orchestra.run.smithi104.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-18T15:03:18.018 INFO:teuthology.orchestra.run.smithi104.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-18T15:03:18.061 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-18T15:03:18.197 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-18T15:03:18.239 INFO:teuthology.orchestra.run.smithi104.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-18T15:03:18.293 INFO:teuthology.orchestra.run.smithi033.stdout:(1/174): ceph-19.0.0-2162.g3d78f5ba.el9.x86_64. 28 kB/s | 6.6 kB 00:00 2024-03-18T15:03:18.293 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-18T15:03:18.374 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-18T15:03:18.414 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-18T15:03:18.449 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-18T15:03:18.496 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-18T15:03:18.532 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-18T15:03:18.577 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-18T15:03:18.659 INFO:teuthology.orchestra.run.smithi033.stdout:(2/174): ceph-base-19.0.0-2162.g3d78f5ba.el9.x8 8.9 MB/s | 5.4 MB 00:00 2024-03-18T15:03:18.718 INFO:teuthology.orchestra.run.smithi033.stdout:(3/174): ceph-immutable-object-cache-19.0.0-216 2.4 MB/s | 145 kB 00:00 2024-03-18T15:03:18.737 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-18T15:03:18.960 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-18T15:03:18.960 INFO:teuthology.orchestra.run.smithi033.stdout:(4/174): ceph-mds-19.0.0-2162.g3d78f5ba.el9.x86 9.8 MB/s | 2.4 MB 00:00 2024-03-18T15:03:19.006 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-18T15:03:19.036 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-18T15:03:19.077 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-18T15:03:19.127 INFO:teuthology.orchestra.run.smithi033.stdout:(5/174): ceph-mgr-19.0.0-2162.g3d78f5ba.el9.x86 9.0 MB/s | 1.5 MB 00:00 2024-03-18T15:03:19.160 INFO:teuthology.orchestra.run.smithi033.stdout:(6/174): ceph-fuse-19.0.0-2162.g3d78f5ba.el9.x8 1.0 MB/s | 890 kB 00:00 2024-03-18T15:03:19.180 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-18T15:03:19.211 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 74/198 2024-03-18T15:03:19.251 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-18T15:03:19.276 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-18T15:03:19.301 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-18T15:03:19.373 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-18T15:03:19.501 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-18T15:03:19.608 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-18T15:03:19.650 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-18T15:03:19.699 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-18T15:03:19.754 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-18T15:03:19.886 INFO:teuthology.orchestra.run.smithi033.stdout:(7/174): ceph-common-19.0.0-2162.g3d78f5ba.el9. 11 MB/s | 21 MB 00:01 2024-03-18T15:03:19.948 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-18T15:03:19.996 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-18T15:03:20.124 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-18T15:03:20.161 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 87/198 2024-03-18T15:03:20.193 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-18T15:03:20.224 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-18T15:03:20.252 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-18T15:03:20.286 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-18T15:03:20.326 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-18T15:03:20.356 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-18T15:03:20.386 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-18T15:03:20.462 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-18T15:03:20.494 INFO:teuthology.orchestra.run.smithi033.stdout:(8/174): ceph-osd-19.0.0-2162.g3d78f5ba.el9.x86 13 MB/s | 17 MB 00:01 2024-03-18T15:03:20.497 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-18T15:03:20.530 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-18T15:03:20.536 INFO:teuthology.orchestra.run.smithi033.stdout:(9/174): ceph-selinux-19.0.0-2162.g3d78f5ba.el9 607 kB/s | 25 kB 00:00 2024-03-18T15:03:20.571 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-18T15:03:20.606 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-18T15:03:20.637 INFO:teuthology.orchestra.run.smithi033.stdout:(10/174): ceph-radosgw-19.0.0-2162.g3d78f5ba.el 13 MB/s | 9.8 MB 00:00 2024-03-18T15:03:20.695 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-18T15:03:20.720 INFO:teuthology.orchestra.run.smithi033.stdout:(11/174): cephfs-mirror-19.0.0-2162.g3d78f5ba.e 2.6 MB/s | 221 kB 00:00 2024-03-18T15:03:20.779 INFO:teuthology.orchestra.run.smithi033.stdout:(12/174): cephfs-shell-19.0.0-2162.g3d78f5ba.el 464 kB/s | 27 kB 00:00 2024-03-18T15:03:20.785 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-18T15:03:20.829 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-18T15:03:20.854 INFO:teuthology.orchestra.run.smithi033.stdout:(13/174): ceph-mon-19.0.0-2162.g3d78f5ba.el9.x8 2.7 MB/s | 4.7 MB 00:01 2024-03-18T15:03:20.861 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-18T15:03:20.879 INFO:teuthology.orchestra.run.smithi033.stdout:(14/174): libcephfs-devel-19.0.0-2162.g3d78f5ba 315 kB/s | 32 kB 00:00 2024-03-18T15:03:20.908 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-18T15:03:20.955 INFO:teuthology.orchestra.run.smithi033.stdout:(15/174): libcephsqlite-19.0.0-2162.g3d78f5ba.e 2.1 MB/s | 163 kB 00:00 2024-03-18T15:03:21.004 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-18T15:03:21.022 INFO:teuthology.orchestra.run.smithi033.stdout:(16/174): librados-devel-19.0.0-2162.g3d78f5ba. 1.9 MB/s | 127 kB 00:00 2024-03-18T15:03:21.047 INFO:teuthology.orchestra.run.smithi033.stdout:(17/174): libcephfs2-19.0.0-2162.g3d78f5ba.el9. 3.8 MB/s | 747 kB 00:00 2024-03-18T15:03:21.061 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-18T15:03:21.105 INFO:teuthology.orchestra.run.smithi033.stdout:(18/174): libradosstriper1-19.0.0-2162.g3d78f5b 5.9 MB/s | 501 kB 00:00 2024-03-18T15:03:21.172 INFO:teuthology.orchestra.run.smithi033.stdout:(19/174): python3-ceph-argparse-19.0.0-2162.g3d 677 kB/s | 45 kB 00:00 2024-03-18T15:03:21.239 INFO:teuthology.orchestra.run.smithi033.stdout:(20/174): python3-ceph-common-19.0.0-2162.g3d78 1.9 MB/s | 131 kB 00:00 2024-03-18T15:03:21.315 INFO:teuthology.orchestra.run.smithi033.stdout:(21/174): python3-cephfs-19.0.0-2162.g3d78f5ba. 2.1 MB/s | 162 kB 00:00 2024-03-18T15:03:21.398 INFO:teuthology.orchestra.run.smithi033.stdout:(22/174): python3-rados-19.0.0-2162.g3d78f5ba.e 3.8 MB/s | 320 kB 00:00 2024-03-18T15:03:21.482 INFO:teuthology.orchestra.run.smithi033.stdout:(23/174): python3-rbd-19.0.0-2162.g3d78f5ba.el9 3.5 MB/s | 298 kB 00:00 2024-03-18T15:03:21.565 INFO:teuthology.orchestra.run.smithi033.stdout:(24/174): python3-rgw-19.0.0-2162.g3d78f5ba.el9 1.2 MB/s | 99 kB 00:00 2024-03-18T15:03:21.649 INFO:teuthology.orchestra.run.smithi033.stdout:(25/174): rbd-fuse-19.0.0-2162.g3d78f5ba.el9.x8 1.0 MB/s | 85 kB 00:00 2024-03-18T15:03:21.776 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-18T15:03:21.816 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-18T15:03:21.850 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-18T15:03:21.898 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-18T15:03:21.958 INFO:teuthology.orchestra.run.smithi033.stdout:(26/174): librgw2-19.0.0-2162.g3d78f5ba.el9.x86 5.6 MB/s | 5.1 MB 00:00 2024-03-18T15:03:21.976 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-18T15:03:22.016 INFO:teuthology.orchestra.run.smithi033.stdout:(27/174): rbd-mirror-19.0.0-2162.g3d78f5ba.el9. 8.4 MB/s | 3.1 MB 00:00 2024-03-18T15:03:22.018 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-18T15:03:22.041 INFO:teuthology.orchestra.run.smithi033.stdout:(28/174): rbd-nbd-19.0.0-2162.g3d78f5ba.el9.x86 2.0 MB/s | 170 kB 00:00 2024-03-18T15:03:22.075 INFO:teuthology.orchestra.run.smithi033.stdout:(29/174): ceph-grafana-dashboards-19.0.0-2162.g 581 kB/s | 34 kB 00:00 2024-03-18T15:03:22.117 INFO:teuthology.orchestra.run.smithi033.stdout:(30/174): ceph-mgr-cephadm-19.0.0-2162.g3d78f5b 1.8 MB/s | 139 kB 00:00 2024-03-18T15:03:22.194 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-18T15:03:22.224 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-18T15:03:22.284 INFO:teuthology.orchestra.run.smithi033.stdout:(31/174): ceph-mgr-dashboard-19.0.0-2162.g3d78f 8.6 MB/s | 1.8 MB 00:00 2024-03-18T15:03:22.301 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-18T15:03:22.376 INFO:teuthology.orchestra.run.smithi033.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2162.g3d 2.7 MB/s | 248 kB 00:00 2024-03-18T15:03:22.443 INFO:teuthology.orchestra.run.smithi033.stdout:(33/174): ceph-mgr-rook-19.0.0-2162.g3d78f5ba.e 741 kB/s | 49 kB 00:00 2024-03-18T15:03:22.510 INFO:teuthology.orchestra.run.smithi033.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2162.g3 246 kB/s | 16 kB 00:00 2024-03-18T15:03:22.602 INFO:teuthology.orchestra.run.smithi033.stdout:(35/174): ceph-volume-19.0.0-2162.g3d78f5ba.el9 2.8 MB/s | 264 kB 00:00 2024-03-18T15:03:22.719 INFO:teuthology.orchestra.run.smithi033.stdout:(36/174): cephadm-19.0.0-2162.g3d78f5ba.el9.noa 6.3 MB/s | 748 kB 00:00 2024-03-18T15:03:22.786 INFO:teuthology.orchestra.run.smithi033.stdout:(37/174): cephfs-top-19.0.0-2162.g3d78f5ba.el9. 274 kB/s | 18 kB 00:00 2024-03-18T15:03:22.928 INFO:teuthology.orchestra.run.smithi033.stdout:(38/174): automake-1.16.2-8.el9.noarch.rpm 4.6 MB/s | 662 kB 00:00 2024-03-18T15:03:22.929 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-18T15:03:22.995 INFO:teuthology.orchestra.run.smithi033.stdout:(39/174): bison-3.7.4-5.el9.x86_64.rpm 14 MB/s | 920 kB 00:00 2024-03-18T15:03:23.037 INFO:teuthology.orchestra.run.smithi033.stdout:(40/174): boost-program-options-1.75.0-8.el9.x8 2.5 MB/s | 106 kB 00:00 2024-03-18T15:03:23.037 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-18T15:03:23.082 INFO:teuthology.orchestra.run.smithi033.stdout:(41/174): flex-2.6.4-9.el9.x86_64.rpm 6.7 MB/s | 307 kB 00:00 2024-03-18T15:03:23.524 INFO:teuthology.orchestra.run.smithi033.stdout:(42/174): ceph-test-19.0.0-2162.g3d78f5ba.el9.x 16 MB/s | 48 MB 00:02 2024-03-18T15:03:23.550 INFO:teuthology.orchestra.run.smithi033.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 68 kB/s | 32 kB 00:00 2024-03-18T15:03:23.642 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-03-18T15:03:23.650 INFO:teuthology.orchestra.run.smithi033.stdout:(44/174): ceph-mgr-diskprediction-local-19.0.0- 4.8 MB/s | 7.4 MB 00:01 2024-03-18T15:03:23.709 INFO:teuthology.orchestra.run.smithi033.stdout:(45/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 16 MB/s | 3.0 MB 00:00 2024-03-18T15:03:23.734 INFO:teuthology.orchestra.run.smithi033.stdout:(46/174): flexiblas-openblas-openmp-3.0.4-8.el9 94 kB/s | 17 kB 00:00 2024-03-18T15:03:23.759 INFO:teuthology.orchestra.run.smithi033.stdout:(47/174): libaio-devel-0.3.111-13.el9.x86_64.rp 95 kB/s | 10 kB 00:00 2024-03-18T15:03:23.785 INFO:teuthology.orchestra.run.smithi033.stdout:(48/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-03-18T15:03:23.810 INFO:teuthology.orchestra.run.smithi033.stdout:(49/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 589 kB/s | 44 kB 00:00 2024-03-18T15:03:23.815 INFO:teuthology.orchestra.run.smithi019.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-18T15:03:23.835 INFO:teuthology.orchestra.run.smithi033.stdout:(50/174): librdkafka-1.6.1-102.el9.x86_64.rpm 8.6 MB/s | 661 kB 00:00 2024-03-18T15:03:23.860 INFO:teuthology.orchestra.run.smithi033.stdout:(51/174): libtool-2.4.6-45.el9.x86_64.rpm 12 MB/s | 578 kB 00:00 2024-03-18T15:03:23.873 INFO:teuthology.orchestra.run.smithi019.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-18T15:03:23.885 INFO:teuthology.orchestra.run.smithi033.stdout:(52/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 627 kB/s | 46 kB 00:00 2024-03-18T15:03:23.911 INFO:teuthology.orchestra.run.smithi033.stdout:(53/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.7 MB/s | 282 kB 00:00 2024-03-18T15:03:23.927 INFO:teuthology.orchestra.run.smithi019.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-18T15:03:23.936 INFO:teuthology.orchestra.run.smithi033.stdout:(54/174): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 187 kB 00:00 2024-03-18T15:03:23.961 INFO:teuthology.orchestra.run.smithi033.stdout:(55/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.8 MB/s | 217 kB 00:00 2024-03-18T15:03:23.973 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-18T15:03:23.986 INFO:teuthology.orchestra.run.smithi033.stdout:(56/174): openblas-0.3.21-2.el9.x86_64.rpm 455 kB/s | 34 kB 00:00 2024-03-18T15:03:24.060 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-18T15:03:24.103 INFO:teuthology.orchestra.run.smithi033.stdout:(57/174): openblas-openmp-0.3.21-2.el9.x86_64.r 28 MB/s | 4.7 MB 00:00 2024-03-18T15:03:24.154 INFO:teuthology.orchestra.run.smithi033.stdout:(58/174): postgresql-13.11-1.el9.x86_64.rpm 9.3 MB/s | 1.5 MB 00:00 2024-03-18T15:03:24.155 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-18T15:03:24.187 INFO:teuthology.orchestra.run.smithi033.stdout:(59/174): postgresql-contrib-13.11-1.el9.x86_64 4.0 MB/s | 811 kB 00:00 2024-03-18T15:03:24.213 INFO:teuthology.orchestra.run.smithi033.stdout:(60/174): postgresql-private-libs-13.11-1.el9.x 1.2 MB/s | 134 kB 00:00 2024-03-18T15:03:24.237 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-18T15:03:24.238 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-18T15:03:24.268 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-18T15:03:24.315 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-18T15:03:24.346 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-18T15:03:24.361 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-18T15:03:24.377 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-18T15:03:24.405 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-18T15:03:24.429 INFO:teuthology.orchestra.run.smithi033.stdout:(61/174): postgresql-server-13.11-1.el9.x86_64. 21 MB/s | 5.7 MB 00:00 2024-03-18T15:03:24.463 INFO:teuthology.orchestra.run.smithi033.stdout:(62/174): protobuf-3.14.0-13.el9.x86_64.rpm 4.1 MB/s | 1.0 MB 00:00 2024-03-18T15:03:24.488 INFO:teuthology.orchestra.run.smithi033.stdout:(63/174): python3-attrs-20.3.0-7.el9.noarch.rpm 285 kB/s | 78 kB 00:00 2024-03-18T15:03:24.697 INFO:teuthology.orchestra.run.smithi033.stdout:(64/174): python3-babel-2.9.1-2.el9.noarch.rpm 22 MB/s | 5.8 MB 00:00 2024-03-18T15:03:24.715 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-18T15:03:24.722 INFO:teuthology.orchestra.run.smithi033.stdout:(65/174): python3-jsonpatch-1.21-16.el9.noarch. 106 kB/s | 24 kB 00:00 2024-03-18T15:03:24.743 INFO:teuthology.orchestra.run.smithi104.stdout: Upgrading : librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 124/198 2024-03-18T15:03:24.747 INFO:teuthology.orchestra.run.smithi033.stdout:(66/174): python3-jsonpointer-2.0-4.el9.noarch. 71 kB/s | 18 kB 00:00 2024-03-18T15:03:24.784 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-18T15:03:24.797 INFO:teuthology.orchestra.run.smithi033.stdout:(67/174): python3-mako-1.1.4-6.el9.noarch.rpm 2.8 MB/s | 144 kB 00:00 2024-03-18T15:03:24.827 INFO:teuthology.orchestra.run.smithi019.stdout: Upgrading : librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-18T15:03:24.831 INFO:teuthology.orchestra.run.smithi033.stdout:(68/174): python3-markupsafe-1.1.1-12.el9.x86_6 378 kB/s | 32 kB 00:00 2024-03-18T15:03:24.833 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 124/198 2024-03-18T15:03:24.864 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 125/198 2024-03-18T15:03:24.900 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-18T15:03:24.948 INFO:teuthology.orchestra.run.smithi033.stdout:(69/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 24 MB/s | 4.8 MB 00:00 2024-03-18T15:03:24.973 INFO:teuthology.orchestra.run.smithi033.stdout:(70/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.2 MB/s | 217 kB 00:00 2024-03-18T15:03:24.988 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-18T15:03:24.999 INFO:teuthology.orchestra.run.smithi033.stdout:(71/174): python3-oauthlib-3.1.1-5.el9.noarch.r 1.0 MB/s | 174 kB 00:00 2024-03-18T15:03:25.020 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-18T15:03:25.024 INFO:teuthology.orchestra.run.smithi033.stdout:(72/174): python3-packaging-20.9-5.el9.noarch.r 921 kB/s | 69 kB 00:00 2024-03-18T15:03:25.049 INFO:teuthology.orchestra.run.smithi033.stdout:(73/174): python3-prettytable-0.7.2-27.el9.noar 823 kB/s | 41 kB 00:00 2024-03-18T15:03:25.074 INFO:teuthology.orchestra.run.smithi033.stdout:(74/174): python3-protobuf-3.14.0-13.el9.noarch 3.1 MB/s | 238 kB 00:00 2024-03-18T15:03:25.100 INFO:teuthology.orchestra.run.smithi033.stdout:(75/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-18T15:03:25.125 INFO:teuthology.orchestra.run.smithi033.stdout:(76/174): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 211 kB 00:00 2024-03-18T15:03:25.150 INFO:teuthology.orchestra.run.smithi033.stdout:(77/174): python3-pytz-2021.1-5.el9.noarch.rpm 631 kB/s | 47 kB 00:00 2024-03-18T15:03:25.170 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-18T15:03:25.175 INFO:teuthology.orchestra.run.smithi033.stdout:(78/174): python3-requests-oauthlib-1.3.0-12.el 571 kB/s | 43 kB 00:00 2024-03-18T15:03:25.201 INFO:teuthology.orchestra.run.smithi033.stdout:(79/174): python3-toml-0.10.2-6.el9.noarch.rpm 777 kB/s | 39 kB 00:00 2024-03-18T15:03:25.207 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 12/198 2024-03-18T15:03:25.278 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-18T15:03:25.357 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-18T15:03:25.379 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 128/198 2024-03-18T15:03:25.449 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 128/198 2024-03-18T15:03:25.472 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 129/198 2024-03-18T15:03:25.611 INFO:teuthology.orchestra.run.smithi019.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-18T15:03:25.618 INFO:teuthology.orchestra.run.smithi033.stdout:(80/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 34 MB/s | 16 MB 00:00 2024-03-18T15:03:25.643 INFO:teuthology.orchestra.run.smithi033.stdout:(81/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 88 kB/s | 41 kB 00:00 2024-03-18T15:03:25.670 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-18T15:03:25.671 INFO:teuthology.orchestra.run.smithi033.stdout:(82/174): socat-1.7.4.1-5.el9.x86_64.rpm 646 kB/s | 300 kB 00:00 2024-03-18T15:03:25.691 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-18T15:03:25.693 INFO:teuthology.orchestra.run.smithi033.stdout:(83/174): uuid-1.6.2-55.el9.x86_64.rpm 758 kB/s | 56 kB 00:00 2024-03-18T15:03:25.719 INFO:teuthology.orchestra.run.smithi033.stdout:(84/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-03-18T15:03:25.744 INFO:teuthology.orchestra.run.smithi033.stdout:(85/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 587 kB/s | 44 kB 00:00 2024-03-18T15:03:25.769 INFO:teuthology.orchestra.run.smithi033.stdout:(86/174): fuse-2.9.9-15.el9.x86_64.rpm 1.0 MB/s | 79 kB 00:00 2024-03-18T15:03:25.819 INFO:teuthology.orchestra.run.smithi033.stdout:(87/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.1 MB/s | 241 kB 00:00 2024-03-18T15:03:25.857 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-18T15:03:25.861 INFO:teuthology.orchestra.run.smithi033.stdout:(88/174): python3-cryptography-36.0.1-4.el9.x86 9.8 MB/s | 1.1 MB 00:00 2024-03-18T15:03:25.884 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 17/198 2024-03-18T15:03:25.886 INFO:teuthology.orchestra.run.smithi033.stdout:(89/174): python3-ply-3.11-14.el9.noarch.rpm 887 kB/s | 103 kB 00:00 2024-03-18T15:03:25.912 INFO:teuthology.orchestra.run.smithi033.stdout:(90/174): python3-pycparser-2.20-6.el9.noarch.r 1.3 MB/s | 124 kB 00:00 2024-03-18T15:03:25.937 INFO:teuthology.orchestra.run.smithi033.stdout:(91/174): python3-urllib3-1.26.5-5.el9.noarch.r 3.7 MB/s | 187 kB 00:00 2024-03-18T15:03:26.000 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 17/198 2024-03-18T15:03:26.071 INFO:teuthology.orchestra.run.smithi019.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-18T15:03:26.102 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-18T15:03:26.154 INFO:teuthology.orchestra.run.smithi033.stdout:(92/174): jq-1.6-16.el9.x86_64.rpm 702 kB/s | 187 kB 00:00 2024-03-18T15:03:26.177 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-18T15:03:26.213 INFO:teuthology.orchestra.run.smithi033.stdout:(93/174): libquadmath-11.4.1-3.el9.x86_64.rpm 710 kB/s | 195 kB 00:00 2024-03-18T15:03:26.304 INFO:teuthology.orchestra.run.smithi033.stdout:(94/174): python3-requests-2.25.1-8.el9.noarch. 832 kB/s | 125 kB 00:00 2024-03-18T15:03:26.405 INFO:teuthology.orchestra.run.smithi033.stdout:(95/174): elfutils-libelf-devel-0.190-2.el9.x86 112 kB/s | 21 kB 00:00 2024-03-18T15:03:26.497 INFO:teuthology.orchestra.run.smithi033.stdout:(96/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 132 kB/s | 25 kB 00:00 2024-03-18T15:03:26.499 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-18T15:03:26.522 INFO:teuthology.orchestra.run.smithi033.stdout:(97/174): libgfortran-11.4.1-3.el9.x86_64.rpm 1.3 MB/s | 803 kB 00:00 2024-03-18T15:03:26.544 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-18T15:03:26.611 INFO:teuthology.orchestra.run.smithi019.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-18T15:03:26.722 INFO:teuthology.orchestra.run.smithi033.stdout:(98/174): python3-devel-3.9.17-2.el9.x86_64.rpm 1.1 MB/s | 245 kB 00:00 2024-03-18T15:03:26.789 INFO:teuthology.orchestra.run.smithi033.stdout:(99/174): qatlib-23.11.0-1.el9.x86_64.rpm 3.2 MB/s | 216 kB 00:00 2024-03-18T15:03:26.827 INFO:teuthology.orchestra.run.smithi033.stdout:(100/174): python3-jinja2-2.11.3-5.el9.noarch.r 814 kB/s | 248 kB 00:00 2024-03-18T15:03:26.860 INFO:teuthology.orchestra.run.smithi033.stdout:(101/174): qatlib-service-23.11.0-1.el9.x86_64. 493 kB/s | 35 kB 00:00 2024-03-18T15:03:26.894 INFO:teuthology.orchestra.run.smithi033.stdout:(102/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 666 kB/s | 44 kB 00:00 2024-03-18T15:03:26.936 INFO:teuthology.orchestra.run.smithi033.stdout:(103/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.2 MB/s | 94 kB 00:00 2024-03-18T15:03:26.995 INFO:teuthology.orchestra.run.smithi033.stdout:(104/174): lua-devel-5.4.4-4.el9.x86_64.rpm 222 kB/s | 22 kB 00:00 2024-03-18T15:03:27.011 INFO:teuthology.orchestra.run.smithi019.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-18T15:03:27.168 INFO:teuthology.orchestra.run.smithi019.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-18T15:03:27.170 INFO:teuthology.orchestra.run.smithi033.stdout:(105/174): protobuf-compiler-3.14.0-13.el9.x86_ 3.6 MB/s | 863 kB 00:00 2024-03-18T15:03:27.258 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-18T15:03:27.478 INFO:teuthology.orchestra.run.smithi033.stdout:(106/174): abseil-cpp-20211102.0-3.el9.x86_64.r 1.1 MB/s | 548 kB 00:00 2024-03-18T15:03:27.501 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-18T15:03:27.571 INFO:teuthology.orchestra.run.smithi019.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-18T15:03:27.578 INFO:teuthology.orchestra.run.smithi033.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 195 kB/s | 19 kB 00:00 2024-03-18T15:03:27.637 INFO:teuthology.orchestra.run.smithi033.stdout:(108/174): gperftools-libs-2.9.1-2.el9.x86_64.r 661 kB/s | 309 kB 00:00 2024-03-18T15:03:27.721 INFO:teuthology.orchestra.run.smithi033.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 311 kB/s | 26 kB 00:00 2024-03-18T15:03:27.728 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-18T15:03:27.768 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-18T15:03:27.855 INFO:teuthology.orchestra.run.smithi033.stdout:(110/174): openssl-devel-3.0.7-27.el9.x86_64.rp 2.9 MB/s | 4.1 MB 00:01 2024-03-18T15:03:27.880 INFO:teuthology.orchestra.run.smithi033.stdout:(111/174): liboath-2.6.7-2.el9.x86_64.rpm 306 kB/s | 49 kB 00:00 2024-03-18T15:03:27.972 INFO:teuthology.orchestra.run.smithi033.stdout:(112/174): luarocks-3.9.2-1.el9.noarch.rpm 1.6 MB/s | 151 kB 00:00 2024-03-18T15:03:28.072 INFO:teuthology.orchestra.run.smithi033.stdout:(113/174): libunwind-1.6.2-1.el9.x86_64.rpm 310 kB/s | 67 kB 00:00 2024-03-18T15:03:28.156 INFO:teuthology.orchestra.run.smithi033.stdout:(114/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.5 MB/s | 838 kB 00:00 2024-03-18T15:03:28.214 INFO:teuthology.orchestra.run.smithi033.stdout:(115/174): libarrow-9.0.0-10.el9.x86_64.rpm 7.0 MB/s | 4.4 MB 00:00 2024-03-18T15:03:28.217 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-18T15:03:28.248 INFO:teuthology.orchestra.run.smithi033.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 473 kB/s | 43 kB 00:00 2024-03-18T15:03:28.298 INFO:teuthology.orchestra.run.smithi033.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 385 kB/s | 32 kB 00:00 2024-03-18T15:03:28.304 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-18T15:03:28.323 INFO:teuthology.orchestra.run.smithi033.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 199 kB/s | 15 kB 00:00 2024-03-18T15:03:28.346 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 131/198 2024-03-18T15:03:28.367 INFO:teuthology.orchestra.run.smithi033.stdout:(119/174): python3-cheroot-8.6.0-4.el9.noarch.r 2.4 MB/s | 172 kB 00:00 2024-03-18T15:03:28.401 INFO:teuthology.orchestra.run.smithi033.stdout:(120/174): python3-cherrypy-18.6.1-2.el9.noarch 4.5 MB/s | 358 kB 00:00 2024-03-18T15:03:28.427 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 131/198 2024-03-18T15:03:28.427 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-18T15:03:28.427 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-18T15:03:28.427 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-18T15:03:28.435 INFO:teuthology.orchestra.run.smithi033.stdout:(121/174): python3-asyncssh-2.13.2-4.el9.noarch 1.5 MB/s | 548 kB 00:00 2024-03-18T15:03:28.456 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 132/198 2024-03-18T15:03:28.460 INFO:teuthology.orchestra.run.smithi033.stdout:(122/174): python3-colorama-0.4.4-12.el9.noarch 602 kB/s | 35 kB 00:00 2024-03-18T15:03:28.485 INFO:teuthology.orchestra.run.smithi033.stdout:(123/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.5 MB/s | 303 kB 00:00 2024-03-18T15:03:28.511 INFO:teuthology.orchestra.run.smithi033.stdout:(124/174): python3-google-auth-2.28.0-1.el9.noa 2.6 MB/s | 203 kB 00:00 2024-03-18T15:03:28.545 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-18T15:03:28.561 INFO:teuthology.orchestra.run.smithi033.stdout:(125/174): python3-grpcio-tools-1.46.7-10.el9.x 1.9 MB/s | 144 kB 00:00 2024-03-18T15:03:28.586 INFO:teuthology.orchestra.run.smithi033.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 142 kB/s | 11 kB 00:00 2024-03-18T15:03:28.620 INFO:teuthology.orchestra.run.smithi033.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 301 kB/s | 18 kB 00:00 2024-03-18T15:03:28.670 INFO:teuthology.orchestra.run.smithi033.stdout:(128/174): python3-grpcio-1.46.7-10.el9.x86_64. 9.7 MB/s | 2.0 MB 00:00 2024-03-18T15:03:28.675 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-18T15:03:28.695 INFO:teuthology.orchestra.run.smithi033.stdout:(129/174): python3-jaraco-collections-3.0.0-8.e 213 kB/s | 23 kB 00:00 2024-03-18T15:03:28.718 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-18T15:03:28.720 INFO:teuthology.orchestra.run.smithi033.stdout:(130/174): python3-jaraco-functools-3.5.0-2.el9 194 kB/s | 19 kB 00:00 2024-03-18T15:03:28.746 INFO:teuthology.orchestra.run.smithi033.stdout:(131/174): python3-jaraco-text-3.2.0-6.el9.noar 261 kB/s | 20 kB 00:00 2024-03-18T15:03:28.762 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-18T15:03:28.787 INFO:teuthology.orchestra.run.smithi033.stdout:(132/174): python3-logutils-0.3.5-21.el9.noarch 694 kB/s | 46 kB 00:00 2024-03-18T15:03:28.821 INFO:teuthology.orchestra.run.smithi033.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 1.0 MB/s | 79 kB 00:00 2024-03-18T15:03:28.826 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-18T15:03:28.863 INFO:teuthology.orchestra.run.smithi033.stdout:(134/174): python3-natsort-7.1.1-5.el9.noarch.r 768 kB/s | 58 kB 00:00 2024-03-18T15:03:28.896 INFO:teuthology.orchestra.run.smithi033.stdout:(135/174): python3-kubernetes-26.1.0-2.el9.noar 5.8 MB/s | 1.0 MB 00:00 2024-03-18T15:03:28.939 INFO:teuthology.orchestra.run.smithi033.stdout:(136/174): python3-pecan-1.4.2-3.el9.noarch.rpm 2.3 MB/s | 272 kB 00:00 2024-03-18T15:03:28.953 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-18T15:03:28.964 INFO:teuthology.orchestra.run.smithi033.stdout:(137/174): python3-portend-3.1.0-2.el9.noarch.r 163 kB/s | 16 kB 00:00 2024-03-18T15:03:28.989 INFO:teuthology.orchestra.run.smithi033.stdout:(138/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 975 kB/s | 90 kB 00:00 2024-03-18T15:03:29.014 INFO:teuthology.orchestra.run.smithi033.stdout:(139/174): python3-pyperclip-1.8.2-2.el9.noarch 375 kB/s | 28 kB 00:00 2024-03-18T15:03:29.022 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-18T15:03:29.040 INFO:teuthology.orchestra.run.smithi033.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 407 kB/s | 31 kB 00:00 2024-03-18T15:03:29.065 INFO:teuthology.orchestra.run.smithi033.stdout:(141/174): python3-routes-2.5.1-5.el9.noarch.rp 2.4 MB/s | 188 kB 00:00 2024-03-18T15:03:29.069 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-18T15:03:29.090 INFO:teuthology.orchestra.run.smithi033.stdout:(142/174): python3-rsa-4.9-2.el9.noarch.rpm 782 kB/s | 59 kB 00:00 2024-03-18T15:03:29.116 INFO:teuthology.orchestra.run.smithi033.stdout:(143/174): python3-tempora-5.0.0-2.el9.noarch.r 474 kB/s | 36 kB 00:00 2024-03-18T15:03:29.141 INFO:teuthology.orchestra.run.smithi033.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 677 kB/s | 51 kB 00:00 2024-03-18T15:03:29.166 INFO:teuthology.orchestra.run.smithi033.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-03-18T15:03:29.183 INFO:teuthology.orchestra.run.smithi033.stdout:(146/174): python3-websocket-client-1.2.3-2.el9 1.3 MB/s | 90 kB 00:00 2024-03-18T15:03:29.198 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-18T15:03:29.225 INFO:teuthology.orchestra.run.smithi033.stdout:(147/174): python3-zc-lockfile-2.0-10.el9.noarc 341 kB/s | 20 kB 00:00 2024-03-18T15:03:29.250 INFO:teuthology.orchestra.run.smithi033.stdout:(148/174): python3-werkzeug-2.0.3-3.el9.1.noarc 3.8 MB/s | 427 kB 00:00 2024-03-18T15:03:29.260 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-18T15:03:29.275 INFO:teuthology.orchestra.run.smithi033.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 2.0 MB/s | 192 kB 00:00 2024-03-18T15:03:29.332 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-18T15:03:29.342 INFO:teuthology.orchestra.run.smithi033.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 14 MB/s | 1.6 MB 00:00 2024-03-18T15:03:29.382 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-18T15:03:29.420 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-18T15:03:29.484 INFO:teuthology.orchestra.run.smithi033.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 2.7 MB/s | 321 kB 00:00 2024-03-18T15:03:29.531 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-18T15:03:29.586 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9. 39/198 2024-03-18T15:03:29.666 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el 40/198 2024-03-18T15:03:29.706 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_6 41/198 2024-03-18T15:03:29.840 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-18T15:03:29.904 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-18T15:03:30.019 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-18T15:03:30.034 INFO:teuthology.orchestra.run.smithi033.stdout:(152/174): librados2-19.0.0-2162.g3d78f5ba.el9. 4.3 MB/s | 3.4 MB 00:00 2024-03-18T15:03:30.093 INFO:teuthology.orchestra.run.smithi033.stdout:(153/174): NetworkManager-1.46.0-1.el9.x86_64.r 3.8 MB/s | 2.3 MB 00:00 2024-03-18T15:03:30.143 INFO:teuthology.orchestra.run.smithi033.stdout:(154/174): librbd1-19.0.0-2162.g3d78f5ba.el9.x8 3.7 MB/s | 3.1 MB 00:00 2024-03-18T15:03:30.169 INFO:teuthology.orchestra.run.smithi033.stdout:(155/174): NetworkManager-team-1.46.0-1.el9.x86 564 kB/s | 42 kB 00:00 2024-03-18T15:03:30.211 INFO:teuthology.orchestra.run.smithi033.stdout:(156/174): acl-2.3.1-4.el9.x86_64.rpm 1.7 MB/s | 71 kB 00:00 2024-03-18T15:03:30.217 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-18T15:03:30.261 INFO:teuthology.orchestra.run.smithi033.stdout:(157/174): elfutils-debuginfod-client-0.190-2.e 730 kB/s | 36 kB 00:00 2024-03-18T15:03:30.277 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-18T15:03:30.311 INFO:teuthology.orchestra.run.smithi033.stdout:(158/174): NetworkManager-tui-1.46.0-1.el9.x86_ 1.5 MB/s | 248 kB 00:00 2024-03-18T15:03:30.353 INFO:teuthology.orchestra.run.smithi033.stdout:(159/174): NetworkManager-libnm-1.46.0-1.el9.x8 5.7 MB/s | 1.8 MB 00:00 2024-03-18T15:03:30.378 INFO:teuthology.orchestra.run.smithi033.stdout:(160/174): elfutils-libelf-0.190-2.el9.x86_64.r 1.6 MB/s | 192 kB 00:00 2024-03-18T15:03:30.395 INFO:teuthology.orchestra.run.smithi033.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 550 kB/s | 23 kB 00:00 2024-03-18T15:03:30.420 INFO:teuthology.orchestra.run.smithi033.stdout:(162/174): elfutils-libs-0.190-2.el9.x86_64.rpm 2.3 MB/s | 256 kB 00:00 2024-03-18T15:03:30.446 INFO:teuthology.orchestra.run.smithi033.stdout:(163/174): libblkid-2.37.4-18.el9.x86_64.rpm 1.6 MB/s | 108 kB 00:00 2024-03-18T15:03:30.471 INFO:teuthology.orchestra.run.smithi033.stdout:(164/174): libfdisk-2.37.4-18.el9.x86_64.rpm 2.0 MB/s | 155 kB 00:00 2024-03-18T15:03:30.496 INFO:teuthology.orchestra.run.smithi033.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 1.2 MB/s | 64 kB 00:00 2024-03-18T15:03:30.522 INFO:teuthology.orchestra.run.smithi033.stdout:(166/174): libmount-2.37.4-18.el9.x86_64.rpm 1.3 MB/s | 136 kB 00:00 2024-03-18T15:03:30.546 INFO:teuthology.orchestra.run.smithi033.stdout:(167/174): libuuid-2.37.4-18.el9.x86_64.rpm 382 kB/s | 29 kB 00:00 2024-03-18T15:03:30.664 INFO:teuthology.orchestra.run.smithi033.stdout:(168/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 15 MB/s | 2.2 MB 00:00 2024-03-18T15:03:30.706 INFO:teuthology.orchestra.run.smithi033.stdout:(169/174): util-linux-2.37.4-18.el9.x86_64.rpm 14 MB/s | 2.3 MB 00:00 2024-03-18T15:03:30.739 INFO:teuthology.orchestra.run.smithi033.stdout:(170/174): openssl-3.0.7-27.el9.x86_64.rpm 5.0 MB/s | 1.2 MB 00:00 2024-03-18T15:03:30.764 INFO:teuthology.orchestra.run.smithi033.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 534 kB/s | 31 kB 00:00 2024-03-18T15:03:30.798 INFO:teuthology.orchestra.run.smithi033.stdout:(172/174): util-linux-core-2.37.4-18.el9.x86_64 3.4 MB/s | 465 kB 00:00 2024-03-18T15:03:30.898 INFO:teuthology.orchestra.run.smithi033.stdout:(173/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 6.8 MB/s | 1.1 MB 00:00 2024-03-18T15:03:30.923 INFO:teuthology.orchestra.run.smithi033.stdout:(174/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 468 kB/s | 74 kB 00:00 2024-03-18T15:03:30.927 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-03-18T15:03:30.927 INFO:teuthology.orchestra.run.smithi033.stdout:Total 17 MB/s | 224 MB 00:13 2024-03-18T15:03:31.011 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-18T15:03:31.114 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-18T15:03:31.236 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-18T15:03:31.308 INFO:teuthology.orchestra.run.smithi019.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-18T15:03:31.336 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-18T15:03:31.403 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-03-18T15:03:31.421 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-18T15:03:31.515 INFO:teuthology.orchestra.run.smithi019.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-18T15:03:31.571 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_6 53/198 2024-03-18T15:03:31.619 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-18T15:03:31.701 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-03-18T15:03:31.702 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-03-18T15:03:32.020 INFO:teuthology.orchestra.run.smithi019.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-18T15:03:32.240 INFO:teuthology.orchestra.run.smithi019.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-18T15:03:32.337 INFO:teuthology.orchestra.run.smithi019.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-18T15:03:32.410 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-18T15:03:32.622 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-18T15:03:32.667 INFO:teuthology.orchestra.run.smithi019.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-18T15:03:32.755 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-18T15:03:32.868 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-18T15:03:32.909 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-18T15:03:32.949 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-18T15:03:33.017 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-18T15:03:33.074 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-18T15:03:33.142 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-18T15:03:33.205 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-18T15:03:33.427 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-18T15:03:33.488 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-18T15:03:33.521 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-03-18T15:03:33.521 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-03-18T15:03:33.525 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-18T15:03:33.584 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-18T15:03:33.719 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-18T15:03:33.765 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 74/198 2024-03-18T15:03:33.820 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-18T15:03:33.857 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-18T15:03:33.889 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-18T15:03:34.060 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-18T15:03:34.260 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-18T15:03:34.402 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-18T15:03:34.458 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-18T15:03:34.521 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-18T15:03:34.615 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-18T15:03:34.877 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-18T15:03:34.945 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-18T15:03:35.124 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-18T15:03:35.168 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 87/198 2024-03-18T15:03:35.207 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-18T15:03:35.246 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-18T15:03:35.287 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-18T15:03:35.333 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-18T15:03:35.402 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-18T15:03:35.460 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-18T15:03:35.508 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-18T15:03:35.650 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-18T15:03:35.704 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-18T15:03:35.739 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-18T15:03:35.804 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-18T15:03:35.843 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-18T15:03:35.980 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-18T15:03:36.100 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-18T15:03:36.155 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-18T15:03:36.190 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-18T15:03:36.250 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-18T15:03:36.393 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-18T15:03:36.466 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-18T15:03:37.463 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-18T15:03:37.523 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-18T15:03:37.556 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-18T15:03:37.608 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-18T15:03:37.715 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-18T15:03:37.757 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-18T15:03:37.997 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-18T15:03:38.028 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-18T15:03:38.126 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-18T15:03:38.978 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-18T15:03:39.112 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-18T15:03:39.873 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 132/198 2024-03-18T15:03:39.873 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /sys 2024-03-18T15:03:39.873 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /proc 2024-03-18T15:03:39.873 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /mnt 2024-03-18T15:03:39.873 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /var/tmp 2024-03-18T15:03:39.873 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /home 2024-03-18T15:03:39.873 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /root 2024-03-18T15:03:39.873 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /tmp 2024-03-18T15:03:39.873 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-18T15:03:40.050 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 133/198 2024-03-18T15:03:40.065 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 133/198 2024-03-18T15:03:40.065 INFO:teuthology.orchestra.run.smithi104.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T15:03:40.065 INFO:teuthology.orchestra.run.smithi104.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-18T15:03:40.065 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-18T15:03:40.731 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-18T15:03:40.829 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-03-18T15:03:40.832 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-18T15:03:40.897 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-18T15:03:40.928 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-18T15:03:40.969 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-18T15:03:41.001 INFO:teuthology.orchestra.run.smithi033.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-18T15:03:41.078 INFO:teuthology.orchestra.run.smithi033.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-18T15:03:41.117 INFO:teuthology.orchestra.run.smithi033.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-18T15:03:41.127 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-18T15:03:41.167 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-18T15:03:41.222 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-18T15:03:41.263 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-18T15:03:41.288 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-18T15:03:41.331 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-18T15:03:41.365 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-18T15:03:41.397 INFO:teuthology.orchestra.run.smithi019.stdout: Upgrading : librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 124/198 2024-03-18T15:03:41.512 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 134/198 2024-03-18T15:03:41.518 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 124/198 2024-03-18T15:03:41.536 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 134/198 2024-03-18T15:03:41.536 INFO:teuthology.orchestra.run.smithi104.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T15:03:41.536 INFO:teuthology.orchestra.run.smithi104.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-18T15:03:41.536 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-18T15:03:41.537 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-18T15:03:41.537 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-18T15:03:41.553 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 125/198 2024-03-18T15:03:41.609 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-18T15:03:41.676 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-18T15:03:41.730 INFO:teuthology.orchestra.run.smithi033.stdout: Upgrading : librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-18T15:03:41.762 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 135/198 2024-03-18T15:03:41.784 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 135/198 2024-03-18T15:03:41.784 INFO:teuthology.orchestra.run.smithi104.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T15:03:41.784 INFO:teuthology.orchestra.run.smithi104.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-18T15:03:41.785 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-18T15:03:41.785 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-18T15:03:41.785 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-18T15:03:41.854 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-18T15:03:41.881 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-18T15:03:41.973 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-18T15:03:42.002 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 12/198 2024-03-18T15:03:42.044 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-18T15:03:42.126 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-18T15:03:42.159 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 128/198 2024-03-18T15:03:42.221 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 136/198 2024-03-18T15:03:42.224 INFO:teuthology.orchestra.run.smithi033.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-18T15:03:42.243 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 136/198 2024-03-18T15:03:42.243 INFO:teuthology.orchestra.run.smithi104.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T15:03:42.243 INFO:teuthology.orchestra.run.smithi104.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-18T15:03:42.244 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-18T15:03:42.244 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-18T15:03:42.244 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-18T15:03:42.267 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 128/198 2024-03-18T15:03:42.295 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 129/198 2024-03-18T15:03:42.301 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-18T15:03:42.328 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-18T15:03:42.339 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-18T15:03:42.433 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-18T15:03:42.462 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 17/198 2024-03-18T15:03:42.526 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 17/198 2024-03-18T15:03:42.591 INFO:teuthology.orchestra.run.smithi033.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-18T15:03:42.614 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-18T15:03:42.630 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-18T15:03:42.723 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 138/198 2024-03-18T15:03:42.754 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 138/198 2024-03-18T15:03:42.786 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.e 139/198 2024-03-18T15:03:42.823 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-18T15:03:42.845 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba. 140/198 2024-03-18T15:03:42.853 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-18T15:03:42.869 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 141/198 2024-03-18T15:03:42.893 INFO:teuthology.orchestra.run.smithi033.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-18T15:03:43.001 INFO:teuthology.orchestra.run.smithi033.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-18T15:03:43.102 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 141/198 2024-03-18T15:03:43.114 INFO:teuthology.orchestra.run.smithi033.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-18T15:03:43.136 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 142/198 2024-03-18T15:03:43.181 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-18T15:03:43.259 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-18T15:03:43.307 INFO:teuthology.orchestra.run.smithi033.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-18T15:03:43.520 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-18T15:03:43.632 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-18T15:03:43.726 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-18T15:03:43.761 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-18T15:03:43.788 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-18T15:03:43.824 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-18T15:03:43.899 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-18T15:03:43.953 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-18T15:03:43.990 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-18T15:03:44.056 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-18T15:03:44.089 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-18T15:03:44.092 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 142/198 2024-03-18T15:03:44.120 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 143/198 2024-03-18T15:03:44.133 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-18T15:03:44.165 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-18T15:03:44.194 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-18T15:03:44.220 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 143/198 2024-03-18T15:03:44.265 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-18T15:03:44.298 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9. 39/198 2024-03-18T15:03:44.350 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el 40/198 2024-03-18T15:03:44.378 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_6 41/198 2024-03-18T15:03:44.389 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el 144/198 2024-03-18T15:03:44.422 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 145/198 2024-03-18T15:03:44.445 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 145/198 2024-03-18T15:03:44.445 INFO:teuthology.orchestra.run.smithi104.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T15:03:44.445 INFO:teuthology.orchestra.run.smithi104.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-18T15:03:44.445 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-18T15:03:44.445 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-18T15:03:44.445 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-18T15:03:44.449 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-18T15:03:44.489 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 146/198 2024-03-18T15:03:44.500 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 146/198 2024-03-18T15:03:44.502 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-18T15:03:44.544 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 147/198 2024-03-18T15:03:44.578 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-18T15:03:44.716 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-18T15:03:44.771 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-18T15:03:45.012 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-18T15:03:45.059 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-18T15:03:45.321 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-18T15:03:45.401 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-18T15:03:45.409 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-18T15:03:45.449 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 149/198 2024-03-18T15:03:45.470 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-18T15:03:45.471 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 149/198 2024-03-18T15:03:45.471 INFO:teuthology.orchestra.run.smithi104.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T15:03:45.471 INFO:teuthology.orchestra.run.smithi104.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-18T15:03:45.471 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-18T15:03:45.471 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-18T15:03:45.471 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-18T15:03:45.508 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 150/198 2024-03-18T15:03:45.510 INFO:teuthology.orchestra.run.smithi033.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-18T15:03:45.519 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-18T15:03:45.530 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 150/198 2024-03-18T15:03:45.530 INFO:teuthology.orchestra.run.smithi104.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T15:03:45.530 INFO:teuthology.orchestra.run.smithi104.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-18T15:03:45.530 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-18T15:03:45.550 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-18T15:03:45.576 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-18T15:03:45.591 INFO:teuthology.orchestra.run.smithi033.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-18T15:03:45.598 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-18T15:03:45.598 INFO:teuthology.orchestra.run.smithi104.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T15:03:45.598 INFO:teuthology.orchestra.run.smithi104.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-18T15:03:45.598 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-18T15:03:45.598 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-18T15:03:45.598 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-18T15:03:45.629 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_6 53/198 2024-03-18T15:03:45.659 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-18T15:03:45.720 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-18T15:03:45.771 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 131/198 2024-03-18T15:03:45.882 INFO:teuthology.orchestra.run.smithi033.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-18T15:03:45.910 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 152/198 2024-03-18T15:03:45.924 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 131/198 2024-03-18T15:03:45.924 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-18T15:03:45.924 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-18T15:03:45.925 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-18T15:03:45.931 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 152/198 2024-03-18T15:03:45.932 INFO:teuthology.orchestra.run.smithi104.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T15:03:45.932 INFO:teuthology.orchestra.run.smithi104.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-18T15:03:45.932 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-18T15:03:45.932 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-18T15:03:45.932 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-18T15:03:45.971 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 132/198 2024-03-18T15:03:46.036 INFO:teuthology.orchestra.run.smithi033.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-18T15:03:46.111 INFO:teuthology.orchestra.run.smithi033.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-18T15:03:46.155 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-18T15:03:46.300 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-18T15:03:46.433 INFO:teuthology.orchestra.run.smithi033.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-18T15:03:46.615 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-18T15:03:46.688 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-18T15:03:46.734 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-18T15:03:46.770 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-18T15:03:46.818 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-18T15:03:46.852 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-18T15:03:46.896 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-18T15:03:46.938 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-18T15:03:47.076 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-18T15:03:47.127 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-18T15:03:47.157 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-18T15:03:47.199 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-18T15:03:47.297 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-18T15:03:47.331 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 74/198 2024-03-18T15:03:47.372 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-18T15:03:47.398 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-18T15:03:47.422 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-18T15:03:47.496 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-18T15:03:47.627 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-18T15:03:47.734 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-18T15:03:47.780 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-18T15:03:47.829 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-18T15:03:47.902 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-18T15:03:48.105 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-18T15:03:48.151 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-18T15:03:48.295 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-18T15:03:48.333 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 87/198 2024-03-18T15:03:48.364 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-18T15:03:48.403 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-18T15:03:48.432 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-18T15:03:48.466 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-18T15:03:48.505 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-18T15:03:48.537 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-18T15:03:48.567 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-18T15:03:48.645 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-18T15:03:48.677 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-18T15:03:48.710 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-18T15:03:48.752 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-18T15:03:48.786 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-18T15:03:48.879 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-18T15:03:48.976 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-18T15:03:49.026 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-18T15:03:49.057 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-18T15:03:49.108 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-18T15:03:49.213 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-18T15:03:49.268 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-18T15:03:50.030 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-18T15:03:50.080 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-18T15:03:50.114 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-18T15:03:50.163 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-18T15:03:50.252 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-18T15:03:50.281 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-18T15:03:50.338 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 153/198 2024-03-18T15:03:50.379 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 154/198 2024-03-18T15:03:50.401 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 155/198 2024-03-18T15:03:50.428 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-18T15:03:50.464 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-18T15:03:50.495 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-18T15:03:50.575 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-18T15:03:50.607 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 157/198 2024-03-18T15:03:50.736 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-18T15:03:50.777 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 159/198 2024-03-18T15:03:50.816 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-18T15:03:50.860 INFO:teuthology.orchestra.run.smithi104.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-18T15:03:50.903 INFO:teuthology.orchestra.run.smithi104.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-18T15:03:50.919 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_ 163/198 2024-03-18T15:03:50.951 INFO:teuthology.orchestra.run.smithi104.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-18T15:03:50.981 INFO:teuthology.orchestra.run.smithi104.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 165/198 2024-03-18T15:03:51.027 INFO:teuthology.orchestra.run.smithi104.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 166/198 2024-03-18T15:03:51.080 INFO:teuthology.orchestra.run.smithi104.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-18T15:03:51.104 INFO:teuthology.orchestra.run.smithi104.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-18T15:03:51.249 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-18T15:03:51.379 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-18T15:03:51.663 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-18T15:03:51.704 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-18T15:03:51.723 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-18T15:03:52.527 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 170/198 2024-03-18T15:03:52.632 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-18T15:03:52.658 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-18T15:03:52.672 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-18T15:03:52.737 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-18T15:03:52.761 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-18T15:03:52.768 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-18T15:03:52.800 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-18T15:03:52.801 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-18T15:03:52.826 INFO:teuthology.orchestra.run.smithi104.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-18T15:03:52.827 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-18T15:03:52.852 INFO:teuthology.orchestra.run.smithi104.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-18T15:03:52.878 INFO:teuthology.orchestra.run.smithi104.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-18T15:03:52.878 INFO:teuthology.orchestra.run.smithi104.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-18T15:03:52.958 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-18T15:03:52.988 INFO:teuthology.orchestra.run.smithi104.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-18T15:03:52.988 INFO:teuthology.orchestra.run.smithi104.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-18T15:03:53.056 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-18T15:03:53.081 INFO:teuthology.orchestra.run.smithi104.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-18T15:03:53.115 INFO:teuthology.orchestra.run.smithi104.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-18T15:03:53.140 INFO:teuthology.orchestra.run.smithi104.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-18T15:03:53.156 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-18T15:03:53.165 INFO:teuthology.orchestra.run.smithi104.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-18T15:03:53.190 INFO:teuthology.orchestra.run.smithi033.stdout: Upgrading : librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 124/198 2024-03-18T15:03:53.198 INFO:teuthology.orchestra.run.smithi104.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-18T15:03:53.246 INFO:teuthology.orchestra.run.smithi104.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-18T15:03:53.273 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 124/198 2024-03-18T15:03:53.286 INFO:teuthology.orchestra.run.smithi104.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-18T15:03:53.303 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 125/198 2024-03-18T15:03:53.312 INFO:teuthology.orchestra.run.smithi104.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-18T15:03:53.337 INFO:teuthology.orchestra.run.smithi104.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-18T15:03:53.340 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-18T15:03:53.362 INFO:teuthology.orchestra.run.smithi104.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-18T15:03:53.387 INFO:teuthology.orchestra.run.smithi104.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-18T15:03:53.393 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-18T15:03:53.394 INFO:teuthology.orchestra.run.smithi104.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-18T15:03:53.574 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-18T15:03:53.574 INFO:teuthology.orchestra.run.smithi104.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-18T15:03:53.640 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-18T15:03:53.664 INFO:teuthology.orchestra.run.smithi104.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-18T15:03:53.690 INFO:teuthology.orchestra.run.smithi104.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-18T15:03:53.723 INFO:teuthology.orchestra.run.smithi104.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-18T15:03:53.749 INFO:teuthology.orchestra.run.smithi104.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-18T15:03:53.813 INFO:teuthology.orchestra.run.smithi104.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-18T15:03:53.827 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-18T15:03:53.852 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 128/198 2024-03-18T15:03:53.923 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 128/198 2024-03-18T15:03:53.953 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 129/198 2024-03-18T15:03:56.345 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-18T15:03:56.375 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-18T15:03:56.941 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-18T15:03:56.978 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 131/198 2024-03-18T15:03:57.052 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 131/198 2024-03-18T15:03:57.052 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-18T15:03:57.052 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-18T15:03:57.052 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-18T15:03:57.079 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 132/198 2024-03-18T15:03:58.379 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-18T15:03:58.379 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 1/198 2024-03-18T15:03:58.379 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2/198 2024-03-18T15:03:58.379 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 3/198 2024-03-18T15:03:58.379 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 4/198 2024-03-18T15:03:58.379 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 5/198 2024-03-18T15:03:58.379 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 6/198 2024-03-18T15:03:58.379 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 7/198 2024-03-18T15:03:58.380 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 8/198 2024-03-18T15:03:58.380 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 9/198 2024-03-18T15:03:58.380 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-18T15:03:58.380 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-18T15:03:58.380 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 12/198 2024-03-18T15:03:58.380 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 13/198 2024-03-18T15:03:58.380 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 14/198 2024-03-18T15:03:58.380 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_ 15/198 2024-03-18T15:03:58.380 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-18T15:03:58.380 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 17/198 2024-03-18T15:03:58.380 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_6 18/198 2024-03-18T15:03:58.380 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 19/198 2024-03-18T15:03:58.381 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 20/198 2024-03-18T15:03:58.381 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el 21/198 2024-03-18T15:03:58.381 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9. 22/198 2024-03-18T15:03:58.381 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_6 23/198 2024-03-18T15:03:58.381 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 24/198 2024-03-18T15:03:58.381 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 25/198 2024-03-18T15:03:58.381 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 26/198 2024-03-18T15:03:58.381 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 27/198 2024-03-18T15:03:58.381 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 28/198 2024-03-18T15:03:58.381 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 29/198 2024-03-18T15:03:58.381 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba. 30/198 2024-03-18T15:03:58.381 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 31/198 2024-03-18T15:03:58.381 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 32/198 2024-03-18T15:03:58.382 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 33/198 2024-03-18T15:03:58.382 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el 34/198 2024-03-18T15:03:58.382 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 35/198 2024-03-18T15:03:58.382 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.e 36/198 2024-03-18T15:03:58.382 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 37/198 2024-03-18T15:03:58.382 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 38/198 2024-03-18T15:03:58.382 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 39/198 2024-03-18T15:03:58.382 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-18T15:03:58.382 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-18T15:03:58.382 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-18T15:03:58.382 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-18T15:03:58.382 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-18T15:03:58.382 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-18T15:03:58.383 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-18T15:03:58.383 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-18T15:03:58.383 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-18T15:03:58.383 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-18T15:03:58.383 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-18T15:03:58.383 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-18T15:03:58.383 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-18T15:03:58.383 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-18T15:03:58.383 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-18T15:03:58.383 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-18T15:03:58.383 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-18T15:03:58.383 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-18T15:03:58.383 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-18T15:03:58.383 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-18T15:03:58.383 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-18T15:03:58.384 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-18T15:03:58.384 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-18T15:03:58.384 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-18T15:03:58.384 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-18T15:03:58.384 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-18T15:03:58.384 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-18T15:03:58.384 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-18T15:03:58.384 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-18T15:03:58.384 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-18T15:03:58.384 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-18T15:03:58.384 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-18T15:03:58.384 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-18T15:03:58.384 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-18T15:03:58.384 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-18T15:03:58.385 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-18T15:03:58.385 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-18T15:03:58.385 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-18T15:03:58.385 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-18T15:03:58.385 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-18T15:03:58.385 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-18T15:03:58.385 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-18T15:03:58.385 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-18T15:03:58.385 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-18T15:03:58.385 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-18T15:03:58.385 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-18T15:03:58.385 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-18T15:03:58.385 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-18T15:03:58.386 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-18T15:03:58.386 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-18T15:03:58.386 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-18T15:03:58.386 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-18T15:03:58.386 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-18T15:03:58.386 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-18T15:03:58.386 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-18T15:03:58.386 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-18T15:03:58.386 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-18T15:03:58.386 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-18T15:03:58.386 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-18T15:03:58.386 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-18T15:03:58.386 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-18T15:03:58.387 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-18T15:03:58.387 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-18T15:03:58.387 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-18T15:03:58.387 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-18T15:03:58.387 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-18T15:03:58.387 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-18T15:03:58.387 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-18T15:03:58.387 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-18T15:03:58.387 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-18T15:03:58.387 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-18T15:03:58.387 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-18T15:03:58.387 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-18T15:03:58.387 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-18T15:03:58.387 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-18T15:03:58.388 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-18T15:03:58.388 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/198 2024-03-18T15:03:58.388 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-18T15:03:58.388 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-18T15:03:58.388 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-18T15:03:58.388 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-18T15:03:58.388 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-18T15:03:58.388 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-18T15:03:58.388 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-18T15:03:58.388 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/198 2024-03-18T15:03:58.388 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-18T15:03:58.388 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-18T15:03:58.388 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-18T15:03:58.388 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-18T15:03:58.389 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-18T15:03:58.389 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-18T15:03:58.389 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-18T15:03:58.389 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-18T15:03:58.389 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-18T15:03:58.389 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-18T15:03:58.389 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-18T15:03:58.389 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-18T15:03:58.389 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-18T15:03:58.389 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-18T15:03:58.389 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-18T15:03:58.389 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-18T15:03:58.389 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-18T15:03:58.389 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-18T15:03:58.389 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-18T15:03:58.389 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-18T15:03:58.390 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-18T15:03:58.390 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-18T15:03:58.390 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-18T15:03:58.390 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-18T15:03:58.390 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-18T15:03:58.390 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-18T15:03:58.390 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-18T15:03:58.390 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-18T15:03:58.390 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 153/198 2024-03-18T15:03:58.390 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-18T15:03:58.390 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-18T15:03:58.390 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-18T15:03:58.390 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/198 2024-03-18T15:03:58.391 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-18T15:03:58.391 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/198 2024-03-18T15:03:58.391 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-18T15:03:58.391 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/198 2024-03-18T15:03:58.391 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-18T15:03:58.391 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/198 2024-03-18T15:03:58.391 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-18T15:03:58.391 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-18T15:03:58.391 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-18T15:03:58.391 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-18T15:03:58.391 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-18T15:03:58.391 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-18T15:03:58.391 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-18T15:03:58.391 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-18T15:03:58.392 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-18T15:03:58.392 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-18T15:03:58.392 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-18T15:03:58.392 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-18T15:03:58.392 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-18T15:03:58.392 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-18T15:03:58.392 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-18T15:03:58.392 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-18T15:03:58.392 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-18T15:03:58.392 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-18T15:03:58.392 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-18T15:03:58.392 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-18T15:03:58.392 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-18T15:03:58.392 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-18T15:03:58.393 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-18T15:03:58.393 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-18T15:03:58.393 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-18T15:03:58.393 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-18T15:03:58.393 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-18T15:03:58.393 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-18T15:03:58.393 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-18T15:03:58.393 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-18T15:03:58.393 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-18T15:03:58.393 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-18T15:03:58.393 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-18T15:03:58.393 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-18T15:03:59.801 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 132/198 2024-03-18T15:03:59.801 INFO:teuthology.orchestra.run.smithi019.stdout:skipping the directory /sys 2024-03-18T15:03:59.801 INFO:teuthology.orchestra.run.smithi019.stdout:skipping the directory /proc 2024-03-18T15:03:59.801 INFO:teuthology.orchestra.run.smithi019.stdout:skipping the directory /mnt 2024-03-18T15:03:59.801 INFO:teuthology.orchestra.run.smithi019.stdout:skipping the directory /var/tmp 2024-03-18T15:03:59.801 INFO:teuthology.orchestra.run.smithi019.stdout:skipping the directory /home 2024-03-18T15:03:59.801 INFO:teuthology.orchestra.run.smithi019.stdout:skipping the directory /root 2024-03-18T15:03:59.801 INFO:teuthology.orchestra.run.smithi019.stdout:skipping the directory /tmp 2024-03-18T15:03:59.801 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-18T15:04:00.051 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 133/198 2024-03-18T15:04:00.102 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 133/198 2024-03-18T15:04:00.102 INFO:teuthology.orchestra.run.smithi019.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T15:04:00.102 INFO:teuthology.orchestra.run.smithi019.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-18T15:04:00.103 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-18T15:04:01.735 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-18T15:04:01.735 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-18T15:04:01.736 INFO:teuthology.orchestra.run.smithi104.stdout:Upgraded: 2024-03-18T15:04:01.736 INFO:teuthology.orchestra.run.smithi104.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-18T15:04:01.736 INFO:teuthology.orchestra.run.smithi104.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-18T15:04:01.736 INFO:teuthology.orchestra.run.smithi104.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-18T15:04:01.736 INFO:teuthology.orchestra.run.smithi104.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-18T15:04:01.736 INFO:teuthology.orchestra.run.smithi104.stdout: acl-2.3.1-4.el9.x86_64 2024-03-18T15:04:01.736 INFO:teuthology.orchestra.run.smithi104.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-18T15:04:01.736 INFO:teuthology.orchestra.run.smithi104.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-18T15:04:01.736 INFO:teuthology.orchestra.run.smithi104.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-18T15:04:01.736 INFO:teuthology.orchestra.run.smithi104.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-18T15:04:01.736 INFO:teuthology.orchestra.run.smithi104.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-18T15:04:01.736 INFO:teuthology.orchestra.run.smithi104.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-18T15:04:01.736 INFO:teuthology.orchestra.run.smithi104.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-18T15:04:01.737 INFO:teuthology.orchestra.run.smithi104.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-18T15:04:01.737 INFO:teuthology.orchestra.run.smithi104.stdout: librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:04:01.737 INFO:teuthology.orchestra.run.smithi104.stdout: librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:04:01.737 INFO:teuthology.orchestra.run.smithi104.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-18T15:04:01.737 INFO:teuthology.orchestra.run.smithi104.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-18T15:04:01.737 INFO:teuthology.orchestra.run.smithi104.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-18T15:04:01.737 INFO:teuthology.orchestra.run.smithi104.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-18T15:04:01.737 INFO:teuthology.orchestra.run.smithi104.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-18T15:04:01.737 INFO:teuthology.orchestra.run.smithi104.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-18T15:04:01.737 INFO:teuthology.orchestra.run.smithi104.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-18T15:04:01.737 INFO:teuthology.orchestra.run.smithi104.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-18T15:04:01.737 INFO:teuthology.orchestra.run.smithi104.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-18T15:04:01.737 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-03-18T15:04:01.738 INFO:teuthology.orchestra.run.smithi104.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-18T15:04:01.738 INFO:teuthology.orchestra.run.smithi104.stdout: automake-1.16.2-8.el9.noarch 2024-03-18T15:04:01.738 INFO:teuthology.orchestra.run.smithi104.stdout: bison-3.7.4-5.el9.x86_64 2024-03-18T15:04:01.738 INFO:teuthology.orchestra.run.smithi104.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-18T15:04:01.738 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:04:01.738 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:04:01.738 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:04:01.738 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:04:01.738 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T15:04:01.738 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-immutable-object-cache-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:04:01.738 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:04:01.738 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:04:01.738 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T15:04:01.738 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T15:04:01.739 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T15:04:01.739 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T15:04:01.739 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T15:04:01.739 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:04:01.739 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:04:01.739 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T15:04:01.739 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:04:01.739 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:04:01.739 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:04:01.739 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T15:04:01.739 INFO:teuthology.orchestra.run.smithi104.stdout: cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T15:04:01.739 INFO:teuthology.orchestra.run.smithi104.stdout: cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:04:01.739 INFO:teuthology.orchestra.run.smithi104.stdout: cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:04:01.739 INFO:teuthology.orchestra.run.smithi104.stdout: cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T15:04:01.739 INFO:teuthology.orchestra.run.smithi104.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-18T15:04:01.740 INFO:teuthology.orchestra.run.smithi104.stdout: flex-2.6.4-9.el9.x86_64 2024-03-18T15:04:01.740 INFO:teuthology.orchestra.run.smithi104.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-18T15:04:01.740 INFO:teuthology.orchestra.run.smithi104.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-18T15:04:01.740 INFO:teuthology.orchestra.run.smithi104.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-18T15:04:01.740 INFO:teuthology.orchestra.run.smithi104.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-18T15:04:01.740 INFO:teuthology.orchestra.run.smithi104.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-18T15:04:01.740 INFO:teuthology.orchestra.run.smithi104.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-18T15:04:01.740 INFO:teuthology.orchestra.run.smithi104.stdout: jq-1.6-16.el9.x86_64 2024-03-18T15:04:01.740 INFO:teuthology.orchestra.run.smithi104.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-18T15:04:01.740 INFO:teuthology.orchestra.run.smithi104.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-18T15:04:01.740 INFO:teuthology.orchestra.run.smithi104.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-18T15:04:01.740 INFO:teuthology.orchestra.run.smithi104.stdout: libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:04:01.740 INFO:teuthology.orchestra.run.smithi104.stdout: libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:04:01.740 INFO:teuthology.orchestra.run.smithi104.stdout: libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:04:01.740 INFO:teuthology.orchestra.run.smithi104.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-18T15:04:01.741 INFO:teuthology.orchestra.run.smithi104.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-18T15:04:01.741 INFO:teuthology.orchestra.run.smithi104.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-18T15:04:01.741 INFO:teuthology.orchestra.run.smithi104.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-18T15:04:01.741 INFO:teuthology.orchestra.run.smithi104.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-18T15:04:01.741 INFO:teuthology.orchestra.run.smithi104.stdout: librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:04:01.741 INFO:teuthology.orchestra.run.smithi104.stdout: libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:04:01.741 INFO:teuthology.orchestra.run.smithi104.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-18T15:04:01.741 INFO:teuthology.orchestra.run.smithi104.stdout: librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:04:01.741 INFO:teuthology.orchestra.run.smithi104.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-18T15:04:01.741 INFO:teuthology.orchestra.run.smithi104.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-18T15:04:01.741 INFO:teuthology.orchestra.run.smithi104.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-18T15:04:01.741 INFO:teuthology.orchestra.run.smithi104.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-18T15:04:01.741 INFO:teuthology.orchestra.run.smithi104.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-18T15:04:01.741 INFO:teuthology.orchestra.run.smithi104.stdout: lua-5.4.4-4.el9.x86_64 2024-03-18T15:04:01.741 INFO:teuthology.orchestra.run.smithi104.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-18T15:04:01.741 INFO:teuthology.orchestra.run.smithi104.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-18T15:04:01.742 INFO:teuthology.orchestra.run.smithi104.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-18T15:04:01.742 INFO:teuthology.orchestra.run.smithi104.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-18T15:04:01.742 INFO:teuthology.orchestra.run.smithi104.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-18T15:04:01.742 INFO:teuthology.orchestra.run.smithi104.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-18T15:04:01.742 INFO:teuthology.orchestra.run.smithi104.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-18T15:04:01.742 INFO:teuthology.orchestra.run.smithi104.stdout: postgresql-13.11-1.el9.x86_64 2024-03-18T15:04:01.742 INFO:teuthology.orchestra.run.smithi104.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-18T15:04:01.742 INFO:teuthology.orchestra.run.smithi104.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-18T15:04:01.742 INFO:teuthology.orchestra.run.smithi104.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-18T15:04:01.742 INFO:teuthology.orchestra.run.smithi104.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-18T15:04:01.742 INFO:teuthology.orchestra.run.smithi104.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-18T15:04:01.742 INFO:teuthology.orchestra.run.smithi104.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-18T15:04:01.742 INFO:teuthology.orchestra.run.smithi104.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-18T15:04:01.742 INFO:teuthology.orchestra.run.smithi104.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-18T15:04:01.743 INFO:teuthology.orchestra.run.smithi104.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-18T15:04:01.743 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-18T15:04:01.743 INFO:teuthology.orchestra.run.smithi104.stdout: python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:04:01.743 INFO:teuthology.orchestra.run.smithi104.stdout: python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:04:01.743 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:04:01.743 INFO:teuthology.orchestra.run.smithi104.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-18T15:04:01.743 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-18T15:04:01.743 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-18T15:04:01.743 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-18T15:04:01.743 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-18T15:04:01.743 INFO:teuthology.orchestra.run.smithi104.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-18T15:04:01.743 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-18T15:04:01.743 INFO:teuthology.orchestra.run.smithi104.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-18T15:04:01.743 INFO:teuthology.orchestra.run.smithi104.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-18T15:04:01.743 INFO:teuthology.orchestra.run.smithi104.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-18T15:04:01.743 INFO:teuthology.orchestra.run.smithi104.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-18T15:04:01.744 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-18T15:04:01.744 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-18T15:04:01.744 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-18T15:04:01.744 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-18T15:04:01.744 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-18T15:04:01.744 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-18T15:04:01.744 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-18T15:04:01.744 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-18T15:04:01.744 INFO:teuthology.orchestra.run.smithi104.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-18T15:04:01.744 INFO:teuthology.orchestra.run.smithi104.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-18T15:04:01.744 INFO:teuthology.orchestra.run.smithi104.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-18T15:04:01.744 INFO:teuthology.orchestra.run.smithi104.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-18T15:04:01.744 INFO:teuthology.orchestra.run.smithi104.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-18T15:04:01.744 INFO:teuthology.orchestra.run.smithi104.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-18T15:04:01.745 INFO:teuthology.orchestra.run.smithi104.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-18T15:04:01.745 INFO:teuthology.orchestra.run.smithi104.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-18T15:04:01.745 INFO:teuthology.orchestra.run.smithi104.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-18T15:04:01.745 INFO:teuthology.orchestra.run.smithi104.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-18T15:04:01.745 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-18T15:04:01.745 INFO:teuthology.orchestra.run.smithi104.stdout: python3-ply-3.11-14.el9.noarch 2024-03-18T15:04:01.745 INFO:teuthology.orchestra.run.smithi104.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-18T15:04:01.745 INFO:teuthology.orchestra.run.smithi104.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-18T15:04:01.745 INFO:teuthology.orchestra.run.smithi104.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-18T15:04:01.745 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-18T15:04:01.745 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-18T15:04:01.745 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-18T15:04:01.745 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-18T15:04:01.745 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-18T15:04:01.745 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-18T15:04:01.745 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:04:01.746 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:04:01.746 INFO:teuthology.orchestra.run.smithi104.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-18T15:04:01.746 INFO:teuthology.orchestra.run.smithi104.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-18T15:04:01.746 INFO:teuthology.orchestra.run.smithi104.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-18T15:04:01.746 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:04:01.746 INFO:teuthology.orchestra.run.smithi104.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-18T15:04:01.746 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-18T15:04:01.746 INFO:teuthology.orchestra.run.smithi104.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-18T15:04:01.746 INFO:teuthology.orchestra.run.smithi104.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-18T15:04:01.746 INFO:teuthology.orchestra.run.smithi104.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-18T15:04:01.746 INFO:teuthology.orchestra.run.smithi104.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-18T15:04:01.746 INFO:teuthology.orchestra.run.smithi104.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-18T15:04:01.746 INFO:teuthology.orchestra.run.smithi104.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-18T15:04:01.746 INFO:teuthology.orchestra.run.smithi104.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-18T15:04:01.746 INFO:teuthology.orchestra.run.smithi104.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-18T15:04:01.747 INFO:teuthology.orchestra.run.smithi104.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-18T15:04:01.747 INFO:teuthology.orchestra.run.smithi104.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-18T15:04:01.747 INFO:teuthology.orchestra.run.smithi104.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-18T15:04:01.747 INFO:teuthology.orchestra.run.smithi104.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-18T15:04:01.747 INFO:teuthology.orchestra.run.smithi104.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-18T15:04:01.747 INFO:teuthology.orchestra.run.smithi104.stdout: rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:04:01.747 INFO:teuthology.orchestra.run.smithi104.stdout: rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:04:01.747 INFO:teuthology.orchestra.run.smithi104.stdout: rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:04:01.747 INFO:teuthology.orchestra.run.smithi104.stdout: re2-1:20211101-3.el9.x86_64 2024-03-18T15:04:01.747 INFO:teuthology.orchestra.run.smithi104.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-18T15:04:01.747 INFO:teuthology.orchestra.run.smithi104.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-18T15:04:01.747 INFO:teuthology.orchestra.run.smithi104.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-18T15:04:01.747 INFO:teuthology.orchestra.run.smithi104.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-18T15:04:01.747 INFO:teuthology.orchestra.run.smithi104.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-18T15:04:01.748 INFO:teuthology.orchestra.run.smithi104.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-18T15:04:01.748 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-18T15:04:01.748 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-03-18T15:04:01.780 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 134/198 2024-03-18T15:04:01.862 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 134/198 2024-03-18T15:04:01.863 INFO:teuthology.orchestra.run.smithi019.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T15:04:01.863 INFO:teuthology.orchestra.run.smithi019.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-18T15:04:01.863 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-18T15:04:01.863 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-18T15:04:01.863 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-18T15:04:02.122 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 135/198 2024-03-18T15:04:02.201 DEBUG:teuthology.parallel:result is None 2024-03-18T15:04:02.201 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 135/198 2024-03-18T15:04:02.201 INFO:teuthology.orchestra.run.smithi019.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T15:04:02.201 INFO:teuthology.orchestra.run.smithi019.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-18T15:04:02.202 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-18T15:04:02.202 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-18T15:04:02.202 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-18T15:04:02.718 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 136/198 2024-03-18T15:04:02.795 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 136/198 2024-03-18T15:04:02.795 INFO:teuthology.orchestra.run.smithi019.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T15:04:02.795 INFO:teuthology.orchestra.run.smithi019.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-18T15:04:02.795 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-18T15:04:02.795 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-18T15:04:02.795 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-18T15:04:02.916 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-18T15:04:03.551 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 138/198 2024-03-18T15:04:03.584 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 138/198 2024-03-18T15:04:03.654 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.e 139/198 2024-03-18T15:04:03.760 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba. 140/198 2024-03-18T15:04:03.788 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 141/198 2024-03-18T15:04:04.134 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 141/198 2024-03-18T15:04:04.172 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 142/198 2024-03-18T15:04:05.350 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 142/198 2024-03-18T15:04:05.413 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 143/198 2024-03-18T15:04:05.570 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 143/198 2024-03-18T15:04:05.753 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el 144/198 2024-03-18T15:04:05.782 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 145/198 2024-03-18T15:04:05.859 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 145/198 2024-03-18T15:04:05.859 INFO:teuthology.orchestra.run.smithi019.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T15:04:05.859 INFO:teuthology.orchestra.run.smithi019.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-18T15:04:05.860 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-18T15:04:05.860 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-18T15:04:05.860 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-18T15:04:05.916 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 146/198 2024-03-18T15:04:05.958 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 146/198 2024-03-18T15:04:06.011 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 147/198 2024-03-18T15:04:07.004 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-18T15:04:07.035 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 149/198 2024-03-18T15:04:07.121 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 149/198 2024-03-18T15:04:07.121 INFO:teuthology.orchestra.run.smithi019.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T15:04:07.121 INFO:teuthology.orchestra.run.smithi019.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-18T15:04:07.121 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-18T15:04:07.121 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-18T15:04:07.121 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-18T15:04:07.169 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 150/198 2024-03-18T15:04:07.241 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 150/198 2024-03-18T15:04:07.241 INFO:teuthology.orchestra.run.smithi019.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T15:04:07.241 INFO:teuthology.orchestra.run.smithi019.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-18T15:04:07.241 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-18T15:04:07.303 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-18T15:04:07.381 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-18T15:04:07.381 INFO:teuthology.orchestra.run.smithi019.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T15:04:07.382 INFO:teuthology.orchestra.run.smithi019.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-18T15:04:07.382 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-18T15:04:07.382 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-18T15:04:07.382 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-18T15:04:07.762 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 152/198 2024-03-18T15:04:07.833 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 152/198 2024-03-18T15:04:07.833 INFO:teuthology.orchestra.run.smithi019.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T15:04:07.833 INFO:teuthology.orchestra.run.smithi019.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-18T15:04:07.833 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-18T15:04:07.833 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-18T15:04:07.833 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-18T15:04:09.558 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 132/198 2024-03-18T15:04:09.559 INFO:teuthology.orchestra.run.smithi033.stdout:skipping the directory /sys 2024-03-18T15:04:09.559 INFO:teuthology.orchestra.run.smithi033.stdout:skipping the directory /proc 2024-03-18T15:04:09.559 INFO:teuthology.orchestra.run.smithi033.stdout:skipping the directory /mnt 2024-03-18T15:04:09.559 INFO:teuthology.orchestra.run.smithi033.stdout:skipping the directory /var/tmp 2024-03-18T15:04:09.559 INFO:teuthology.orchestra.run.smithi033.stdout:skipping the directory /home 2024-03-18T15:04:09.559 INFO:teuthology.orchestra.run.smithi033.stdout:skipping the directory /root 2024-03-18T15:04:09.559 INFO:teuthology.orchestra.run.smithi033.stdout:skipping the directory /tmp 2024-03-18T15:04:09.559 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-18T15:04:09.733 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 133/198 2024-03-18T15:04:09.748 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 133/198 2024-03-18T15:04:09.749 INFO:teuthology.orchestra.run.smithi033.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T15:04:09.749 INFO:teuthology.orchestra.run.smithi033.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-18T15:04:09.749 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-18T15:04:11.269 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 134/198 2024-03-18T15:04:11.295 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 134/198 2024-03-18T15:04:11.296 INFO:teuthology.orchestra.run.smithi033.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T15:04:11.296 INFO:teuthology.orchestra.run.smithi033.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-18T15:04:11.296 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-18T15:04:11.296 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-18T15:04:11.296 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-18T15:04:11.528 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 135/198 2024-03-18T15:04:11.552 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 135/198 2024-03-18T15:04:11.552 INFO:teuthology.orchestra.run.smithi033.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T15:04:11.552 INFO:teuthology.orchestra.run.smithi033.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-18T15:04:11.552 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-18T15:04:11.552 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-18T15:04:11.552 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-18T15:04:12.012 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 136/198 2024-03-18T15:04:12.035 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 136/198 2024-03-18T15:04:12.035 INFO:teuthology.orchestra.run.smithi033.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T15:04:12.036 INFO:teuthology.orchestra.run.smithi033.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-18T15:04:12.036 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-18T15:04:12.036 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-18T15:04:12.036 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-18T15:04:12.130 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-18T15:04:12.465 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 138/198 2024-03-18T15:04:12.495 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 138/198 2024-03-18T15:04:12.527 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.e 139/198 2024-03-18T15:04:12.589 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba. 140/198 2024-03-18T15:04:12.620 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 141/198 2024-03-18T15:04:12.866 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 141/198 2024-03-18T15:04:12.894 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 153/198 2024-03-18T15:04:12.896 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 142/198 2024-03-18T15:04:12.944 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 154/198 2024-03-18T15:04:12.983 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 155/198 2024-03-18T15:04:13.041 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-18T15:04:13.349 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 157/198 2024-03-18T15:04:13.484 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-18T15:04:13.535 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 159/198 2024-03-18T15:04:13.594 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-18T15:04:13.642 INFO:teuthology.orchestra.run.smithi019.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-18T15:04:13.687 INFO:teuthology.orchestra.run.smithi019.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-18T15:04:13.723 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_ 163/198 2024-03-18T15:04:13.768 INFO:teuthology.orchestra.run.smithi019.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-18T15:04:13.820 INFO:teuthology.orchestra.run.smithi019.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 165/198 2024-03-18T15:04:13.911 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 142/198 2024-03-18T15:04:13.913 INFO:teuthology.orchestra.run.smithi019.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 166/198 2024-03-18T15:04:13.938 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 143/198 2024-03-18T15:04:14.043 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 143/198 2024-03-18T15:04:14.070 INFO:teuthology.orchestra.run.smithi019.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-18T15:04:14.118 INFO:teuthology.orchestra.run.smithi019.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-18T15:04:14.208 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el 144/198 2024-03-18T15:04:14.240 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 145/198 2024-03-18T15:04:14.265 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 145/198 2024-03-18T15:04:14.265 INFO:teuthology.orchestra.run.smithi033.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T15:04:14.265 INFO:teuthology.orchestra.run.smithi033.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-18T15:04:14.265 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-18T15:04:14.265 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-18T15:04:14.265 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-18T15:04:14.315 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 146/198 2024-03-18T15:04:14.326 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 146/198 2024-03-18T15:04:14.373 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 147/198 2024-03-18T15:04:14.983 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-18T15:04:15.024 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-18T15:04:15.081 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-18T15:04:15.292 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-18T15:04:15.326 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 149/198 2024-03-18T15:04:15.350 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 149/198 2024-03-18T15:04:15.350 INFO:teuthology.orchestra.run.smithi033.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T15:04:15.350 INFO:teuthology.orchestra.run.smithi033.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-18T15:04:15.350 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-18T15:04:15.350 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-18T15:04:15.350 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-18T15:04:15.393 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 150/198 2024-03-18T15:04:15.416 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 150/198 2024-03-18T15:04:15.416 INFO:teuthology.orchestra.run.smithi033.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T15:04:15.416 INFO:teuthology.orchestra.run.smithi033.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-18T15:04:15.416 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-18T15:04:15.460 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-18T15:04:15.484 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-18T15:04:15.484 INFO:teuthology.orchestra.run.smithi033.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T15:04:15.484 INFO:teuthology.orchestra.run.smithi033.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-18T15:04:15.484 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-18T15:04:15.484 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-18T15:04:15.484 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-18T15:04:15.812 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 152/198 2024-03-18T15:04:15.835 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 152/198 2024-03-18T15:04:15.836 INFO:teuthology.orchestra.run.smithi033.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T15:04:15.836 INFO:teuthology.orchestra.run.smithi033.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-18T15:04:15.836 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-18T15:04:15.836 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-18T15:04:15.836 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-18T15:04:16.463 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 170/198 2024-03-18T15:04:16.579 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-18T15:04:16.663 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-18T15:04:16.813 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-18T15:04:16.848 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-18T15:04:16.890 INFO:teuthology.orchestra.run.smithi019.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-18T15:04:16.924 INFO:teuthology.orchestra.run.smithi019.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-18T15:04:16.967 INFO:teuthology.orchestra.run.smithi019.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-18T15:04:16.968 INFO:teuthology.orchestra.run.smithi019.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-18T15:04:17.154 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-18T15:04:17.195 INFO:teuthology.orchestra.run.smithi019.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-18T15:04:17.195 INFO:teuthology.orchestra.run.smithi019.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-18T15:04:17.333 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-18T15:04:17.377 INFO:teuthology.orchestra.run.smithi019.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-18T15:04:17.422 INFO:teuthology.orchestra.run.smithi019.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-18T15:04:17.457 INFO:teuthology.orchestra.run.smithi019.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-18T15:04:17.489 INFO:teuthology.orchestra.run.smithi019.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-18T15:04:17.589 INFO:teuthology.orchestra.run.smithi019.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-18T15:04:17.671 INFO:teuthology.orchestra.run.smithi019.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-18T15:04:17.719 INFO:teuthology.orchestra.run.smithi019.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-18T15:04:17.752 INFO:teuthology.orchestra.run.smithi019.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-18T15:04:17.785 INFO:teuthology.orchestra.run.smithi019.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-18T15:04:17.819 INFO:teuthology.orchestra.run.smithi019.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-18T15:04:17.852 INFO:teuthology.orchestra.run.smithi019.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-18T15:04:17.889 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-18T15:04:17.890 INFO:teuthology.orchestra.run.smithi019.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-18T15:04:18.203 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-18T15:04:18.203 INFO:teuthology.orchestra.run.smithi019.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-18T15:04:18.358 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-18T15:04:18.389 INFO:teuthology.orchestra.run.smithi019.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-18T15:04:18.439 INFO:teuthology.orchestra.run.smithi019.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-18T15:04:18.505 INFO:teuthology.orchestra.run.smithi019.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-18T15:04:18.550 INFO:teuthology.orchestra.run.smithi019.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-18T15:04:18.672 INFO:teuthology.orchestra.run.smithi019.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-18T15:04:18.939 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-18T15:04:18.939 INFO:teuthology.orchestra.run.smithi019.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-03-18T15:04:18.940 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-18T15:04:20.541 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 153/198 2024-03-18T15:04:20.582 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 154/198 2024-03-18T15:04:20.612 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 155/198 2024-03-18T15:04:20.639 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-18T15:04:20.831 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 157/198 2024-03-18T15:04:20.951 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-18T15:04:20.988 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 159/198 2024-03-18T15:04:21.018 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-18T15:04:21.062 INFO:teuthology.orchestra.run.smithi033.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-18T15:04:21.105 INFO:teuthology.orchestra.run.smithi033.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-18T15:04:21.130 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_ 163/198 2024-03-18T15:04:21.162 INFO:teuthology.orchestra.run.smithi033.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-18T15:04:21.192 INFO:teuthology.orchestra.run.smithi033.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 165/198 2024-03-18T15:04:21.246 INFO:teuthology.orchestra.run.smithi033.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 166/198 2024-03-18T15:04:21.290 INFO:teuthology.orchestra.run.smithi033.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-18T15:04:21.315 INFO:teuthology.orchestra.run.smithi033.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-18T15:04:21.888 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-18T15:04:21.929 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-18T15:04:21.950 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-18T15:04:22.830 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 170/198 2024-03-18T15:04:22.916 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-18T15:04:22.957 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-18T15:04:23.040 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-18T15:04:23.078 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-18T15:04:23.103 INFO:teuthology.orchestra.run.smithi033.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-18T15:04:23.128 INFO:teuthology.orchestra.run.smithi033.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-18T15:04:23.155 INFO:teuthology.orchestra.run.smithi033.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-18T15:04:23.155 INFO:teuthology.orchestra.run.smithi033.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-18T15:04:23.226 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-18T15:04:23.255 INFO:teuthology.orchestra.run.smithi033.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-18T15:04:23.255 INFO:teuthology.orchestra.run.smithi033.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-18T15:04:23.332 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-18T15:04:23.358 INFO:teuthology.orchestra.run.smithi033.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-18T15:04:23.391 INFO:teuthology.orchestra.run.smithi033.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-18T15:04:23.417 INFO:teuthology.orchestra.run.smithi033.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-18T15:04:23.441 INFO:teuthology.orchestra.run.smithi033.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-18T15:04:23.475 INFO:teuthology.orchestra.run.smithi033.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-18T15:04:23.522 INFO:teuthology.orchestra.run.smithi033.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-18T15:04:23.554 INFO:teuthology.orchestra.run.smithi033.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-18T15:04:23.580 INFO:teuthology.orchestra.run.smithi033.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-18T15:04:23.605 INFO:teuthology.orchestra.run.smithi033.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-18T15:04:23.630 INFO:teuthology.orchestra.run.smithi033.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-18T15:04:23.647 INFO:teuthology.orchestra.run.smithi033.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-18T15:04:23.654 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-18T15:04:23.654 INFO:teuthology.orchestra.run.smithi033.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-18T15:04:23.842 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-18T15:04:23.843 INFO:teuthology.orchestra.run.smithi033.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-18T15:04:23.925 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-18T15:04:23.939 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 1/198 2024-03-18T15:04:23.939 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2/198 2024-03-18T15:04:23.939 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 3/198 2024-03-18T15:04:23.939 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 4/198 2024-03-18T15:04:23.939 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 5/198 2024-03-18T15:04:23.939 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 6/198 2024-03-18T15:04:23.939 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 7/198 2024-03-18T15:04:23.939 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 8/198 2024-03-18T15:04:23.939 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 9/198 2024-03-18T15:04:23.939 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-18T15:04:23.940 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-18T15:04:23.940 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 12/198 2024-03-18T15:04:23.940 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 13/198 2024-03-18T15:04:23.940 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 14/198 2024-03-18T15:04:23.940 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_ 15/198 2024-03-18T15:04:23.940 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-18T15:04:23.940 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 17/198 2024-03-18T15:04:23.940 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_6 18/198 2024-03-18T15:04:23.940 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 19/198 2024-03-18T15:04:23.940 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 20/198 2024-03-18T15:04:23.940 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el 21/198 2024-03-18T15:04:23.940 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9. 22/198 2024-03-18T15:04:23.940 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_6 23/198 2024-03-18T15:04:23.940 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 24/198 2024-03-18T15:04:23.941 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 25/198 2024-03-18T15:04:23.956 INFO:teuthology.orchestra.run.smithi033.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-18T15:04:23.957 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 26/198 2024-03-18T15:04:23.957 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 27/198 2024-03-18T15:04:23.958 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 28/198 2024-03-18T15:04:23.958 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 29/198 2024-03-18T15:04:23.958 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba. 30/198 2024-03-18T15:04:23.958 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 31/198 2024-03-18T15:04:23.958 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 32/198 2024-03-18T15:04:23.958 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 33/198 2024-03-18T15:04:23.958 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el 34/198 2024-03-18T15:04:23.958 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 35/198 2024-03-18T15:04:23.958 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.e 36/198 2024-03-18T15:04:23.958 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 37/198 2024-03-18T15:04:23.958 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 38/198 2024-03-18T15:04:23.958 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 39/198 2024-03-18T15:04:23.959 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-18T15:04:23.959 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-18T15:04:23.959 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-18T15:04:23.959 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-18T15:04:23.959 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-18T15:04:23.959 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-18T15:04:23.959 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-18T15:04:23.959 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-18T15:04:23.959 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-18T15:04:23.959 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-18T15:04:23.959 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-18T15:04:23.959 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-18T15:04:23.960 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-18T15:04:23.960 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-18T15:04:23.960 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-18T15:04:23.960 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-18T15:04:23.960 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-18T15:04:23.960 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-18T15:04:23.960 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-18T15:04:23.960 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-18T15:04:23.960 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-18T15:04:23.960 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-18T15:04:23.960 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-18T15:04:23.960 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-18T15:04:23.960 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-18T15:04:23.961 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-18T15:04:23.961 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-18T15:04:23.961 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-18T15:04:23.961 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-18T15:04:23.961 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-18T15:04:23.961 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-18T15:04:23.961 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-18T15:04:23.961 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-18T15:04:23.961 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-18T15:04:23.961 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-18T15:04:23.961 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-18T15:04:23.961 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-18T15:04:23.961 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-18T15:04:23.962 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-18T15:04:23.962 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-18T15:04:23.962 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-18T15:04:23.962 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-18T15:04:23.962 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-18T15:04:23.962 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-18T15:04:23.962 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-18T15:04:23.962 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-18T15:04:23.962 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-18T15:04:23.962 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-18T15:04:23.962 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-18T15:04:23.962 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-18T15:04:23.962 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-18T15:04:23.963 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-18T15:04:23.963 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-18T15:04:23.963 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-18T15:04:23.963 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-18T15:04:23.963 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-18T15:04:23.963 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-18T15:04:23.963 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-18T15:04:23.963 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-18T15:04:23.963 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-18T15:04:23.963 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-18T15:04:23.963 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-18T15:04:23.963 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-18T15:04:23.963 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-18T15:04:23.964 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-18T15:04:23.964 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-18T15:04:23.964 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-18T15:04:23.964 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-18T15:04:23.964 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-18T15:04:23.964 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-18T15:04:23.964 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-18T15:04:23.964 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-18T15:04:23.964 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-18T15:04:23.964 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-18T15:04:23.964 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-18T15:04:23.964 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-18T15:04:23.964 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/198 2024-03-18T15:04:23.964 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-18T15:04:23.965 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-18T15:04:23.965 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-18T15:04:23.965 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-18T15:04:23.965 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-18T15:04:23.965 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-18T15:04:23.965 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-18T15:04:23.965 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/198 2024-03-18T15:04:23.965 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-18T15:04:23.965 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-18T15:04:23.965 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-18T15:04:23.965 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-18T15:04:23.965 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-18T15:04:23.965 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-18T15:04:23.966 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-18T15:04:23.966 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-18T15:04:23.966 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-18T15:04:23.966 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-18T15:04:23.966 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-18T15:04:23.966 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-18T15:04:23.966 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-18T15:04:23.966 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-18T15:04:23.966 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-18T15:04:23.966 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-18T15:04:23.966 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-18T15:04:23.966 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-18T15:04:23.966 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-18T15:04:23.967 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-18T15:04:23.967 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-18T15:04:23.967 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-18T15:04:23.967 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-18T15:04:23.967 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-18T15:04:23.967 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-18T15:04:23.967 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-18T15:04:23.967 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-18T15:04:23.967 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-18T15:04:23.967 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 153/198 2024-03-18T15:04:23.967 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-18T15:04:23.967 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-18T15:04:23.967 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-18T15:04:23.967 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/198 2024-03-18T15:04:23.968 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-18T15:04:23.968 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/198 2024-03-18T15:04:23.968 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-18T15:04:23.968 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/198 2024-03-18T15:04:23.968 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-18T15:04:23.968 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/198 2024-03-18T15:04:23.968 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-18T15:04:23.968 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-18T15:04:23.968 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-18T15:04:23.968 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-18T15:04:23.968 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-18T15:04:23.968 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-18T15:04:23.968 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-18T15:04:23.968 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-18T15:04:23.969 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-18T15:04:23.969 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-18T15:04:23.969 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-18T15:04:23.969 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-18T15:04:23.969 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-18T15:04:23.969 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-18T15:04:23.969 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-18T15:04:23.969 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-18T15:04:23.969 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-18T15:04:23.969 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-18T15:04:23.969 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-18T15:04:23.969 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-18T15:04:23.969 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-18T15:04:23.969 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-18T15:04:23.970 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-18T15:04:23.970 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-18T15:04:23.970 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-18T15:04:23.970 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-18T15:04:23.970 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-18T15:04:23.970 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-18T15:04:23.970 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-18T15:04:23.970 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-18T15:04:23.970 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-18T15:04:23.970 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-18T15:04:23.970 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-18T15:04:23.970 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-18T15:04:23.975 INFO:teuthology.orchestra.run.smithi033.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-18T15:04:24.007 INFO:teuthology.orchestra.run.smithi033.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-18T15:04:24.034 INFO:teuthology.orchestra.run.smithi033.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-18T15:04:24.100 INFO:teuthology.orchestra.run.smithi033.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-18T15:04:26.558 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-18T15:04:26.558 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-18T15:04:26.558 INFO:teuthology.orchestra.run.smithi019.stdout:Upgraded: 2024-03-18T15:04:26.558 INFO:teuthology.orchestra.run.smithi019.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-18T15:04:26.558 INFO:teuthology.orchestra.run.smithi019.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-18T15:04:26.558 INFO:teuthology.orchestra.run.smithi019.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-18T15:04:26.558 INFO:teuthology.orchestra.run.smithi019.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-18T15:04:26.558 INFO:teuthology.orchestra.run.smithi019.stdout: acl-2.3.1-4.el9.x86_64 2024-03-18T15:04:26.558 INFO:teuthology.orchestra.run.smithi019.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-18T15:04:26.558 INFO:teuthology.orchestra.run.smithi019.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-18T15:04:26.559 INFO:teuthology.orchestra.run.smithi019.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-18T15:04:26.559 INFO:teuthology.orchestra.run.smithi019.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-18T15:04:26.559 INFO:teuthology.orchestra.run.smithi019.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-18T15:04:26.559 INFO:teuthology.orchestra.run.smithi019.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-18T15:04:26.559 INFO:teuthology.orchestra.run.smithi019.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-18T15:04:26.559 INFO:teuthology.orchestra.run.smithi019.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-18T15:04:26.559 INFO:teuthology.orchestra.run.smithi019.stdout: librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:04:26.559 INFO:teuthology.orchestra.run.smithi019.stdout: librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:04:26.559 INFO:teuthology.orchestra.run.smithi019.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-18T15:04:26.559 INFO:teuthology.orchestra.run.smithi019.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-18T15:04:26.559 INFO:teuthology.orchestra.run.smithi019.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-18T15:04:26.559 INFO:teuthology.orchestra.run.smithi019.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-18T15:04:26.559 INFO:teuthology.orchestra.run.smithi019.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-18T15:04:26.559 INFO:teuthology.orchestra.run.smithi019.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-18T15:04:26.560 INFO:teuthology.orchestra.run.smithi019.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-18T15:04:26.560 INFO:teuthology.orchestra.run.smithi019.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-18T15:04:26.560 INFO:teuthology.orchestra.run.smithi019.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-18T15:04:26.560 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-03-18T15:04:26.560 INFO:teuthology.orchestra.run.smithi019.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-18T15:04:26.560 INFO:teuthology.orchestra.run.smithi019.stdout: automake-1.16.2-8.el9.noarch 2024-03-18T15:04:26.560 INFO:teuthology.orchestra.run.smithi019.stdout: bison-3.7.4-5.el9.x86_64 2024-03-18T15:04:26.560 INFO:teuthology.orchestra.run.smithi019.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-18T15:04:26.560 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:04:26.560 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:04:26.560 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:04:26.560 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:04:26.560 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T15:04:26.561 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-immutable-object-cache-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:04:26.561 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:04:26.561 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:04:26.561 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T15:04:26.561 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T15:04:26.561 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T15:04:26.561 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T15:04:26.561 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T15:04:26.561 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:04:26.561 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:04:26.561 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T15:04:26.561 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:04:26.561 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:04:26.562 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:04:26.562 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T15:04:26.562 INFO:teuthology.orchestra.run.smithi019.stdout: cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T15:04:26.562 INFO:teuthology.orchestra.run.smithi019.stdout: cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:04:26.562 INFO:teuthology.orchestra.run.smithi019.stdout: cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:04:26.562 INFO:teuthology.orchestra.run.smithi019.stdout: cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T15:04:26.562 INFO:teuthology.orchestra.run.smithi019.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-18T15:04:26.562 INFO:teuthology.orchestra.run.smithi019.stdout: flex-2.6.4-9.el9.x86_64 2024-03-18T15:04:26.562 INFO:teuthology.orchestra.run.smithi019.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-18T15:04:26.562 INFO:teuthology.orchestra.run.smithi019.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-18T15:04:26.562 INFO:teuthology.orchestra.run.smithi019.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-18T15:04:26.562 INFO:teuthology.orchestra.run.smithi019.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-18T15:04:26.562 INFO:teuthology.orchestra.run.smithi019.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-18T15:04:26.563 INFO:teuthology.orchestra.run.smithi019.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-18T15:04:26.563 INFO:teuthology.orchestra.run.smithi019.stdout: jq-1.6-16.el9.x86_64 2024-03-18T15:04:26.563 INFO:teuthology.orchestra.run.smithi019.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-18T15:04:26.563 INFO:teuthology.orchestra.run.smithi019.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-18T15:04:26.563 INFO:teuthology.orchestra.run.smithi019.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-18T15:04:26.563 INFO:teuthology.orchestra.run.smithi019.stdout: libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:04:26.563 INFO:teuthology.orchestra.run.smithi019.stdout: libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:04:26.563 INFO:teuthology.orchestra.run.smithi019.stdout: libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:04:26.563 INFO:teuthology.orchestra.run.smithi019.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-18T15:04:26.563 INFO:teuthology.orchestra.run.smithi019.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-18T15:04:26.563 INFO:teuthology.orchestra.run.smithi019.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-18T15:04:26.563 INFO:teuthology.orchestra.run.smithi019.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-18T15:04:26.564 INFO:teuthology.orchestra.run.smithi019.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-18T15:04:26.564 INFO:teuthology.orchestra.run.smithi019.stdout: librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:04:26.564 INFO:teuthology.orchestra.run.smithi019.stdout: libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:04:26.564 INFO:teuthology.orchestra.run.smithi019.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-18T15:04:26.564 INFO:teuthology.orchestra.run.smithi019.stdout: librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:04:26.564 INFO:teuthology.orchestra.run.smithi019.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-18T15:04:26.564 INFO:teuthology.orchestra.run.smithi019.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-18T15:04:26.564 INFO:teuthology.orchestra.run.smithi019.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-18T15:04:26.565 INFO:teuthology.orchestra.run.smithi019.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-18T15:04:26.565 INFO:teuthology.orchestra.run.smithi019.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-18T15:04:26.565 INFO:teuthology.orchestra.run.smithi019.stdout: lua-5.4.4-4.el9.x86_64 2024-03-18T15:04:26.565 INFO:teuthology.orchestra.run.smithi019.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-18T15:04:26.565 INFO:teuthology.orchestra.run.smithi019.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-18T15:04:26.565 INFO:teuthology.orchestra.run.smithi019.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-18T15:04:26.565 INFO:teuthology.orchestra.run.smithi019.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-18T15:04:26.565 INFO:teuthology.orchestra.run.smithi019.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-18T15:04:26.565 INFO:teuthology.orchestra.run.smithi019.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-18T15:04:26.565 INFO:teuthology.orchestra.run.smithi019.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-18T15:04:26.565 INFO:teuthology.orchestra.run.smithi019.stdout: postgresql-13.11-1.el9.x86_64 2024-03-18T15:04:26.565 INFO:teuthology.orchestra.run.smithi019.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-18T15:04:26.565 INFO:teuthology.orchestra.run.smithi019.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-18T15:04:26.565 INFO:teuthology.orchestra.run.smithi019.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-18T15:04:26.566 INFO:teuthology.orchestra.run.smithi019.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-18T15:04:26.566 INFO:teuthology.orchestra.run.smithi019.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-18T15:04:26.566 INFO:teuthology.orchestra.run.smithi019.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-18T15:04:26.566 INFO:teuthology.orchestra.run.smithi019.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-18T15:04:26.566 INFO:teuthology.orchestra.run.smithi019.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-18T15:04:26.566 INFO:teuthology.orchestra.run.smithi019.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-18T15:04:26.566 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-18T15:04:26.566 INFO:teuthology.orchestra.run.smithi019.stdout: python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:04:26.566 INFO:teuthology.orchestra.run.smithi019.stdout: python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:04:26.566 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:04:26.566 INFO:teuthology.orchestra.run.smithi019.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-18T15:04:26.566 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-18T15:04:26.566 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-18T15:04:26.567 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-18T15:04:26.567 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-18T15:04:26.567 INFO:teuthology.orchestra.run.smithi019.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-18T15:04:26.567 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-18T15:04:26.567 INFO:teuthology.orchestra.run.smithi019.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-18T15:04:26.567 INFO:teuthology.orchestra.run.smithi019.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-18T15:04:26.567 INFO:teuthology.orchestra.run.smithi019.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-18T15:04:26.567 INFO:teuthology.orchestra.run.smithi019.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-18T15:04:26.567 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-18T15:04:26.567 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-18T15:04:26.567 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-18T15:04:26.567 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-18T15:04:26.567 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-18T15:04:26.567 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-18T15:04:26.568 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-18T15:04:26.568 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-18T15:04:26.568 INFO:teuthology.orchestra.run.smithi019.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-18T15:04:26.568 INFO:teuthology.orchestra.run.smithi019.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-18T15:04:26.568 INFO:teuthology.orchestra.run.smithi019.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-18T15:04:26.568 INFO:teuthology.orchestra.run.smithi019.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-18T15:04:26.568 INFO:teuthology.orchestra.run.smithi019.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-18T15:04:26.568 INFO:teuthology.orchestra.run.smithi019.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-18T15:04:26.568 INFO:teuthology.orchestra.run.smithi019.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-18T15:04:26.568 INFO:teuthology.orchestra.run.smithi019.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-18T15:04:26.568 INFO:teuthology.orchestra.run.smithi019.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-18T15:04:26.568 INFO:teuthology.orchestra.run.smithi019.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-18T15:04:26.568 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-18T15:04:26.569 INFO:teuthology.orchestra.run.smithi019.stdout: python3-ply-3.11-14.el9.noarch 2024-03-18T15:04:26.569 INFO:teuthology.orchestra.run.smithi019.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-18T15:04:26.569 INFO:teuthology.orchestra.run.smithi019.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-18T15:04:26.569 INFO:teuthology.orchestra.run.smithi019.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-18T15:04:26.569 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-18T15:04:26.569 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-18T15:04:26.569 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-18T15:04:26.569 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-18T15:04:26.569 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-18T15:04:26.569 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-18T15:04:26.569 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:04:26.569 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:04:26.569 INFO:teuthology.orchestra.run.smithi019.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-18T15:04:26.569 INFO:teuthology.orchestra.run.smithi019.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-18T15:04:26.569 INFO:teuthology.orchestra.run.smithi019.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-18T15:04:26.570 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:04:26.570 INFO:teuthology.orchestra.run.smithi019.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-18T15:04:26.570 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-18T15:04:26.570 INFO:teuthology.orchestra.run.smithi019.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-18T15:04:26.570 INFO:teuthology.orchestra.run.smithi019.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-18T15:04:26.570 INFO:teuthology.orchestra.run.smithi019.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-18T15:04:26.570 INFO:teuthology.orchestra.run.smithi019.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-18T15:04:26.570 INFO:teuthology.orchestra.run.smithi019.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-18T15:04:26.570 INFO:teuthology.orchestra.run.smithi019.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-18T15:04:26.570 INFO:teuthology.orchestra.run.smithi019.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-18T15:04:26.570 INFO:teuthology.orchestra.run.smithi019.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-18T15:04:26.570 INFO:teuthology.orchestra.run.smithi019.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-18T15:04:26.570 INFO:teuthology.orchestra.run.smithi019.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-18T15:04:26.571 INFO:teuthology.orchestra.run.smithi019.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-18T15:04:26.571 INFO:teuthology.orchestra.run.smithi019.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-18T15:04:26.571 INFO:teuthology.orchestra.run.smithi019.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-18T15:04:26.571 INFO:teuthology.orchestra.run.smithi019.stdout: rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:04:26.571 INFO:teuthology.orchestra.run.smithi019.stdout: rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:04:26.571 INFO:teuthology.orchestra.run.smithi019.stdout: rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:04:26.571 INFO:teuthology.orchestra.run.smithi019.stdout: re2-1:20211101-3.el9.x86_64 2024-03-18T15:04:26.571 INFO:teuthology.orchestra.run.smithi019.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-18T15:04:26.571 INFO:teuthology.orchestra.run.smithi019.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-18T15:04:26.571 INFO:teuthology.orchestra.run.smithi019.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-18T15:04:26.571 INFO:teuthology.orchestra.run.smithi019.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-18T15:04:26.571 INFO:teuthology.orchestra.run.smithi019.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-18T15:04:26.572 INFO:teuthology.orchestra.run.smithi019.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-18T15:04:26.572 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-18T15:04:26.572 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-03-18T15:04:27.227 DEBUG:teuthology.parallel:result is None 2024-03-18T15:04:28.752 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-18T15:04:28.752 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 1/198 2024-03-18T15:04:28.752 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2/198 2024-03-18T15:04:28.752 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 3/198 2024-03-18T15:04:28.752 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 4/198 2024-03-18T15:04:28.752 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 5/198 2024-03-18T15:04:28.752 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 6/198 2024-03-18T15:04:28.752 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 7/198 2024-03-18T15:04:28.753 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 8/198 2024-03-18T15:04:28.753 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 9/198 2024-03-18T15:04:28.753 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-18T15:04:28.753 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-18T15:04:28.753 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 12/198 2024-03-18T15:04:28.753 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 13/198 2024-03-18T15:04:28.753 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 14/198 2024-03-18T15:04:28.753 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_ 15/198 2024-03-18T15:04:28.753 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-18T15:04:28.753 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 17/198 2024-03-18T15:04:28.753 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_6 18/198 2024-03-18T15:04:28.753 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 19/198 2024-03-18T15:04:28.753 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 20/198 2024-03-18T15:04:28.753 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el 21/198 2024-03-18T15:04:28.754 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9. 22/198 2024-03-18T15:04:28.754 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_6 23/198 2024-03-18T15:04:28.754 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 24/198 2024-03-18T15:04:28.754 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 25/198 2024-03-18T15:04:28.754 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 26/198 2024-03-18T15:04:28.754 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 27/198 2024-03-18T15:04:28.772 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 28/198 2024-03-18T15:04:28.772 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 29/198 2024-03-18T15:04:28.772 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba. 30/198 2024-03-18T15:04:28.772 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 31/198 2024-03-18T15:04:28.772 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 32/198 2024-03-18T15:04:28.772 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 33/198 2024-03-18T15:04:28.772 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el 34/198 2024-03-18T15:04:28.773 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 35/198 2024-03-18T15:04:28.773 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.e 36/198 2024-03-18T15:04:28.773 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 37/198 2024-03-18T15:04:28.773 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 38/198 2024-03-18T15:04:28.773 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 39/198 2024-03-18T15:04:28.773 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-18T15:04:28.773 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-18T15:04:28.773 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-18T15:04:28.773 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-18T15:04:28.773 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-18T15:04:28.773 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-18T15:04:28.773 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-18T15:04:28.773 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-18T15:04:28.774 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-18T15:04:28.774 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-18T15:04:28.774 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-18T15:04:28.774 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-18T15:04:28.774 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-18T15:04:28.774 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-18T15:04:28.774 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-18T15:04:28.774 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-18T15:04:28.774 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-18T15:04:28.774 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-18T15:04:28.774 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-18T15:04:28.774 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-18T15:04:28.775 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-18T15:04:28.775 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-18T15:04:28.775 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-18T15:04:28.775 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-18T15:04:28.775 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-18T15:04:28.775 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-18T15:04:28.775 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-18T15:04:28.775 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-18T15:04:28.775 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-18T15:04:28.775 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-18T15:04:28.775 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-18T15:04:28.775 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-18T15:04:28.775 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-18T15:04:28.775 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-18T15:04:28.776 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-18T15:04:28.776 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-18T15:04:28.776 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-18T15:04:28.776 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-18T15:04:28.776 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-18T15:04:28.776 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-18T15:04:28.776 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-18T15:04:28.776 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-18T15:04:28.776 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-18T15:04:28.776 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-18T15:04:28.776 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-18T15:04:28.776 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-18T15:04:28.776 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-18T15:04:28.776 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-18T15:04:28.776 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-18T15:04:28.777 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-18T15:04:28.777 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-18T15:04:28.777 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-18T15:04:28.777 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-18T15:04:28.777 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-18T15:04:28.777 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-18T15:04:28.777 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-18T15:04:28.777 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-18T15:04:28.777 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-18T15:04:28.777 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-18T15:04:28.777 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-18T15:04:28.777 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-18T15:04:28.777 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-18T15:04:28.778 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-18T15:04:28.778 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-18T15:04:28.778 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-18T15:04:28.778 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-18T15:04:28.778 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-18T15:04:28.778 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-18T15:04:28.778 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-18T15:04:28.778 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-18T15:04:28.778 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-18T15:04:28.778 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-18T15:04:28.778 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-18T15:04:28.778 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-18T15:04:28.778 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-18T15:04:28.778 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-18T15:04:28.779 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/198 2024-03-18T15:04:28.779 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-18T15:04:28.779 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-18T15:04:28.779 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-18T15:04:28.779 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-18T15:04:28.779 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-18T15:04:28.779 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-18T15:04:28.779 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-18T15:04:28.779 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/198 2024-03-18T15:04:28.779 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-18T15:04:28.779 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-18T15:04:28.779 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-18T15:04:28.779 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-18T15:04:28.780 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-18T15:04:28.780 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-18T15:04:28.780 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-18T15:04:28.780 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-18T15:04:28.780 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-18T15:04:28.780 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-18T15:04:28.780 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-18T15:04:28.780 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-18T15:04:28.780 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-18T15:04:28.780 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-18T15:04:28.780 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-18T15:04:28.780 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-18T15:04:28.780 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-18T15:04:28.781 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-18T15:04:28.781 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-18T15:04:28.781 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-18T15:04:28.781 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-18T15:04:28.781 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-18T15:04:28.781 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-18T15:04:28.781 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-18T15:04:28.781 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-18T15:04:28.781 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-18T15:04:28.781 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-18T15:04:28.781 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-18T15:04:28.781 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 153/198 2024-03-18T15:04:28.781 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-18T15:04:28.781 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-18T15:04:28.782 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-18T15:04:28.782 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/198 2024-03-18T15:04:28.782 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-18T15:04:28.782 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/198 2024-03-18T15:04:28.782 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-18T15:04:28.782 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/198 2024-03-18T15:04:28.782 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-18T15:04:28.782 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/198 2024-03-18T15:04:28.782 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-18T15:04:28.782 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-18T15:04:28.782 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-18T15:04:28.782 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-18T15:04:28.782 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-18T15:04:28.782 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-18T15:04:28.783 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-18T15:04:28.783 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-18T15:04:28.783 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-18T15:04:28.783 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-18T15:04:28.783 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-18T15:04:28.783 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-18T15:04:28.783 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-18T15:04:28.783 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-18T15:04:28.783 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-18T15:04:28.783 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-18T15:04:28.783 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-18T15:04:28.783 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-18T15:04:28.783 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-18T15:04:28.783 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-18T15:04:28.784 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-18T15:04:28.784 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-18T15:04:28.784 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-18T15:04:28.784 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-18T15:04:28.784 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-18T15:04:28.784 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-18T15:04:28.784 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-18T15:04:28.784 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-18T15:04:28.784 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-18T15:04:28.784 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-18T15:04:28.784 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-18T15:04:28.784 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-18T15:04:28.784 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-18T15:04:28.784 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-18T15:04:31.909 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-18T15:04:31.909 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-18T15:04:31.909 INFO:teuthology.orchestra.run.smithi033.stdout:Upgraded: 2024-03-18T15:04:31.909 INFO:teuthology.orchestra.run.smithi033.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-18T15:04:31.909 INFO:teuthology.orchestra.run.smithi033.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-18T15:04:31.909 INFO:teuthology.orchestra.run.smithi033.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-18T15:04:31.909 INFO:teuthology.orchestra.run.smithi033.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-18T15:04:31.909 INFO:teuthology.orchestra.run.smithi033.stdout: acl-2.3.1-4.el9.x86_64 2024-03-18T15:04:31.909 INFO:teuthology.orchestra.run.smithi033.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-18T15:04:31.910 INFO:teuthology.orchestra.run.smithi033.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-18T15:04:31.910 INFO:teuthology.orchestra.run.smithi033.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-18T15:04:31.910 INFO:teuthology.orchestra.run.smithi033.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-18T15:04:31.910 INFO:teuthology.orchestra.run.smithi033.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-18T15:04:31.910 INFO:teuthology.orchestra.run.smithi033.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-18T15:04:31.910 INFO:teuthology.orchestra.run.smithi033.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-18T15:04:31.910 INFO:teuthology.orchestra.run.smithi033.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-18T15:04:31.910 INFO:teuthology.orchestra.run.smithi033.stdout: librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:04:31.910 INFO:teuthology.orchestra.run.smithi033.stdout: librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:04:31.910 INFO:teuthology.orchestra.run.smithi033.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-18T15:04:31.910 INFO:teuthology.orchestra.run.smithi033.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-18T15:04:31.910 INFO:teuthology.orchestra.run.smithi033.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-18T15:04:31.911 INFO:teuthology.orchestra.run.smithi033.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-18T15:04:31.911 INFO:teuthology.orchestra.run.smithi033.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-18T15:04:31.911 INFO:teuthology.orchestra.run.smithi033.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-18T15:04:31.911 INFO:teuthology.orchestra.run.smithi033.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-18T15:04:31.911 INFO:teuthology.orchestra.run.smithi033.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-18T15:04:31.911 INFO:teuthology.orchestra.run.smithi033.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-18T15:04:31.911 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-03-18T15:04:31.911 INFO:teuthology.orchestra.run.smithi033.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-18T15:04:31.911 INFO:teuthology.orchestra.run.smithi033.stdout: automake-1.16.2-8.el9.noarch 2024-03-18T15:04:31.911 INFO:teuthology.orchestra.run.smithi033.stdout: bison-3.7.4-5.el9.x86_64 2024-03-18T15:04:31.911 INFO:teuthology.orchestra.run.smithi033.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-18T15:04:31.911 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:04:31.911 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:04:31.912 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:04:31.912 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:04:31.912 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T15:04:31.912 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-immutable-object-cache-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:04:31.912 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:04:31.912 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:04:31.912 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T15:04:31.912 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T15:04:31.912 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T15:04:31.912 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T15:04:31.912 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T15:04:31.912 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:04:31.912 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:04:31.913 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T15:04:31.913 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:04:31.913 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:04:31.913 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:04:31.913 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T15:04:31.913 INFO:teuthology.orchestra.run.smithi033.stdout: cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T15:04:31.913 INFO:teuthology.orchestra.run.smithi033.stdout: cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:04:31.913 INFO:teuthology.orchestra.run.smithi033.stdout: cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:04:31.913 INFO:teuthology.orchestra.run.smithi033.stdout: cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T15:04:31.913 INFO:teuthology.orchestra.run.smithi033.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-18T15:04:31.913 INFO:teuthology.orchestra.run.smithi033.stdout: flex-2.6.4-9.el9.x86_64 2024-03-18T15:04:31.913 INFO:teuthology.orchestra.run.smithi033.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-18T15:04:31.913 INFO:teuthology.orchestra.run.smithi033.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-18T15:04:31.914 INFO:teuthology.orchestra.run.smithi033.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-18T15:04:31.914 INFO:teuthology.orchestra.run.smithi033.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-18T15:04:31.914 INFO:teuthology.orchestra.run.smithi033.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-18T15:04:31.914 INFO:teuthology.orchestra.run.smithi033.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-18T15:04:31.914 INFO:teuthology.orchestra.run.smithi033.stdout: jq-1.6-16.el9.x86_64 2024-03-18T15:04:31.914 INFO:teuthology.orchestra.run.smithi033.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-18T15:04:31.914 INFO:teuthology.orchestra.run.smithi033.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-18T15:04:31.914 INFO:teuthology.orchestra.run.smithi033.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-18T15:04:31.914 INFO:teuthology.orchestra.run.smithi033.stdout: libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:04:31.914 INFO:teuthology.orchestra.run.smithi033.stdout: libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:04:31.914 INFO:teuthology.orchestra.run.smithi033.stdout: libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:04:31.914 INFO:teuthology.orchestra.run.smithi033.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-18T15:04:31.914 INFO:teuthology.orchestra.run.smithi033.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-18T15:04:31.915 INFO:teuthology.orchestra.run.smithi033.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-18T15:04:31.915 INFO:teuthology.orchestra.run.smithi033.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-18T15:04:31.915 INFO:teuthology.orchestra.run.smithi033.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-18T15:04:31.915 INFO:teuthology.orchestra.run.smithi033.stdout: librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:04:31.915 INFO:teuthology.orchestra.run.smithi033.stdout: libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:04:31.915 INFO:teuthology.orchestra.run.smithi033.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-18T15:04:31.915 INFO:teuthology.orchestra.run.smithi033.stdout: librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:04:31.915 INFO:teuthology.orchestra.run.smithi033.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-18T15:04:31.915 INFO:teuthology.orchestra.run.smithi033.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-18T15:04:31.915 INFO:teuthology.orchestra.run.smithi033.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-18T15:04:31.915 INFO:teuthology.orchestra.run.smithi033.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-18T15:04:31.915 INFO:teuthology.orchestra.run.smithi033.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-18T15:04:31.915 INFO:teuthology.orchestra.run.smithi033.stdout: lua-5.4.4-4.el9.x86_64 2024-03-18T15:04:31.916 INFO:teuthology.orchestra.run.smithi033.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-18T15:04:31.916 INFO:teuthology.orchestra.run.smithi033.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-18T15:04:31.916 INFO:teuthology.orchestra.run.smithi033.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-18T15:04:31.916 INFO:teuthology.orchestra.run.smithi033.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-18T15:04:31.916 INFO:teuthology.orchestra.run.smithi033.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-18T15:04:31.916 INFO:teuthology.orchestra.run.smithi033.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-18T15:04:31.916 INFO:teuthology.orchestra.run.smithi033.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-18T15:04:31.916 INFO:teuthology.orchestra.run.smithi033.stdout: postgresql-13.11-1.el9.x86_64 2024-03-18T15:04:31.916 INFO:teuthology.orchestra.run.smithi033.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-18T15:04:31.916 INFO:teuthology.orchestra.run.smithi033.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-18T15:04:31.916 INFO:teuthology.orchestra.run.smithi033.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-18T15:04:31.916 INFO:teuthology.orchestra.run.smithi033.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-18T15:04:31.916 INFO:teuthology.orchestra.run.smithi033.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-18T15:04:31.916 INFO:teuthology.orchestra.run.smithi033.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-18T15:04:31.917 INFO:teuthology.orchestra.run.smithi033.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-18T15:04:31.917 INFO:teuthology.orchestra.run.smithi033.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-18T15:04:31.917 INFO:teuthology.orchestra.run.smithi033.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-18T15:04:31.917 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-18T15:04:31.917 INFO:teuthology.orchestra.run.smithi033.stdout: python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:04:31.917 INFO:teuthology.orchestra.run.smithi033.stdout: python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:04:31.917 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:04:31.917 INFO:teuthology.orchestra.run.smithi033.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-18T15:04:31.917 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-18T15:04:31.917 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-18T15:04:31.917 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-18T15:04:31.917 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-18T15:04:31.917 INFO:teuthology.orchestra.run.smithi033.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-18T15:04:31.917 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-18T15:04:31.918 INFO:teuthology.orchestra.run.smithi033.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-18T15:04:31.918 INFO:teuthology.orchestra.run.smithi033.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-18T15:04:31.918 INFO:teuthology.orchestra.run.smithi033.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-18T15:04:31.918 INFO:teuthology.orchestra.run.smithi033.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-18T15:04:31.918 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-18T15:04:31.918 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-18T15:04:31.918 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-18T15:04:31.918 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-18T15:04:31.918 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-18T15:04:31.918 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-18T15:04:31.918 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-18T15:04:31.918 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-18T15:04:31.918 INFO:teuthology.orchestra.run.smithi033.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-18T15:04:31.919 INFO:teuthology.orchestra.run.smithi033.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-18T15:04:31.919 INFO:teuthology.orchestra.run.smithi033.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-18T15:04:31.919 INFO:teuthology.orchestra.run.smithi033.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-18T15:04:31.919 INFO:teuthology.orchestra.run.smithi033.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-18T15:04:31.919 INFO:teuthology.orchestra.run.smithi033.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-18T15:04:31.919 INFO:teuthology.orchestra.run.smithi033.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-18T15:04:31.919 INFO:teuthology.orchestra.run.smithi033.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-18T15:04:31.919 INFO:teuthology.orchestra.run.smithi033.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-18T15:04:31.919 INFO:teuthology.orchestra.run.smithi033.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-18T15:04:31.919 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-18T15:04:31.919 INFO:teuthology.orchestra.run.smithi033.stdout: python3-ply-3.11-14.el9.noarch 2024-03-18T15:04:31.919 INFO:teuthology.orchestra.run.smithi033.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-18T15:04:31.919 INFO:teuthology.orchestra.run.smithi033.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-18T15:04:31.920 INFO:teuthology.orchestra.run.smithi033.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-18T15:04:31.920 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-18T15:04:31.920 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-18T15:04:31.920 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-18T15:04:31.920 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-18T15:04:31.920 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-18T15:04:31.920 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-18T15:04:31.920 INFO:teuthology.orchestra.run.smithi033.stdout: python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:04:31.920 INFO:teuthology.orchestra.run.smithi033.stdout: python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:04:31.920 INFO:teuthology.orchestra.run.smithi033.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-18T15:04:31.920 INFO:teuthology.orchestra.run.smithi033.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-18T15:04:31.920 INFO:teuthology.orchestra.run.smithi033.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-18T15:04:31.920 INFO:teuthology.orchestra.run.smithi033.stdout: python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:04:31.920 INFO:teuthology.orchestra.run.smithi033.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-18T15:04:31.921 INFO:teuthology.orchestra.run.smithi033.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-18T15:04:31.921 INFO:teuthology.orchestra.run.smithi033.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-18T15:04:31.921 INFO:teuthology.orchestra.run.smithi033.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-18T15:04:31.921 INFO:teuthology.orchestra.run.smithi033.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-18T15:04:31.921 INFO:teuthology.orchestra.run.smithi033.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-18T15:04:31.921 INFO:teuthology.orchestra.run.smithi033.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-18T15:04:31.921 INFO:teuthology.orchestra.run.smithi033.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-18T15:04:31.921 INFO:teuthology.orchestra.run.smithi033.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-18T15:04:31.921 INFO:teuthology.orchestra.run.smithi033.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-18T15:04:31.921 INFO:teuthology.orchestra.run.smithi033.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-18T15:04:31.921 INFO:teuthology.orchestra.run.smithi033.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-18T15:04:31.921 INFO:teuthology.orchestra.run.smithi033.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-18T15:04:31.921 INFO:teuthology.orchestra.run.smithi033.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-18T15:04:31.921 INFO:teuthology.orchestra.run.smithi033.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-18T15:04:31.922 INFO:teuthology.orchestra.run.smithi033.stdout: rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:04:31.922 INFO:teuthology.orchestra.run.smithi033.stdout: rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:04:31.922 INFO:teuthology.orchestra.run.smithi033.stdout: rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:04:31.922 INFO:teuthology.orchestra.run.smithi033.stdout: re2-1:20211101-3.el9.x86_64 2024-03-18T15:04:31.922 INFO:teuthology.orchestra.run.smithi033.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-18T15:04:31.922 INFO:teuthology.orchestra.run.smithi033.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-18T15:04:31.922 INFO:teuthology.orchestra.run.smithi033.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-18T15:04:31.922 INFO:teuthology.orchestra.run.smithi033.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-18T15:04:31.922 INFO:teuthology.orchestra.run.smithi033.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-18T15:04:31.923 INFO:teuthology.orchestra.run.smithi033.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-18T15:04:31.923 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-18T15:04:31.923 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-18T15:04:32.382 DEBUG:teuthology.parallel:result is None 2024-03-18T15:04:32.382 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-18T15:04:32.523 DEBUG:teuthology.orchestra.run.smithi019:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-18T15:04:32.580 INFO:teuthology.orchestra.run.smithi019.stdout:19.0.0-2162.g3d78f5ba.el9 2024-03-18T15:04:32.581 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2162.g3d78f5ba.el9 2024-03-18T15:04:32.581 INFO:teuthology.task.install:The correct ceph version 19.0.0-2162.g3d78f5ba is installed. 2024-03-18T15:04:32.583 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-18T15:04:32.894 DEBUG:teuthology.orchestra.run.smithi033:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-18T15:04:32.918 INFO:teuthology.orchestra.run.smithi033.stdout:19.0.0-2162.g3d78f5ba.el9 2024-03-18T15:04:32.918 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2162.g3d78f5ba.el9 2024-03-18T15:04:32.918 INFO:teuthology.task.install:The correct ceph version 19.0.0-2162.g3d78f5ba is installed. 2024-03-18T15:04:32.920 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-18T15:04:33.138 DEBUG:teuthology.orchestra.run.smithi104:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-18T15:04:33.163 INFO:teuthology.orchestra.run.smithi104.stdout:19.0.0-2162.g3d78f5ba.el9 2024-03-18T15:04:33.163 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2162.g3d78f5ba.el9 2024-03-18T15:04:33.164 INFO:teuthology.task.install:The correct ceph version 19.0.0-2162.g3d78f5ba is installed. 2024-03-18T15:04:33.165 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-18T15:04:33.166 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-03-18T15:04:33.166 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-18T15:04:33.269 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-18T15:04:33.269 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-18T15:04:33.308 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-03-18T15:04:33.308 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-18T15:04:33.342 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-18T15:04:33.342 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-03-18T15:04:33.342 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/usr/bin/daemon-helper 2024-03-18T15:04:33.447 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-18T15:04:33.545 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-18T15:04:33.545 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/usr/bin/daemon-helper 2024-03-18T15:04:33.581 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-18T15:04:33.655 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-03-18T15:04:33.655 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/usr/bin/daemon-helper 2024-03-18T15:04:33.687 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-18T15:04:33.756 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-18T15:04:33.756 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-03-18T15:04:33.757 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-18T15:04:33.860 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-18T15:04:34.003 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-18T15:04:34.003 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-18T15:04:34.038 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-18T15:04:34.112 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-03-18T15:04:34.112 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-18T15:04:34.140 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-18T15:04:34.207 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-18T15:04:34.207 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-03-18T15:04:34.207 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/usr/bin/stdin-killer 2024-03-18T15:04:34.312 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-18T15:04:34.413 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-18T15:04:34.413 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/usr/bin/stdin-killer 2024-03-18T15:04:34.448 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-18T15:04:34.524 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-03-18T15:04:34.524 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/usr/bin/stdin-killer 2024-03-18T15:04:34.550 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-18T15:04:34.616 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-18T15:04:34.718 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_bal_replicate_threshold': 1, 'mds_export_ephemeral_random_max': 0.1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'mons down, quorum', 'out of quorum', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN'], 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af', 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2024-03-18T15:04:34.718 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-18T15:04:34.719 INFO:tasks.cephadm:Cluster fsid is d4dc2ba6-e538-11ee-95c9-87774f69a715 2024-03-18T15:04:34.719 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-18T15:04:34.719 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.19', 'mon.b': '172.21.15.33', 'mon.c': '172.21.15.104'} 2024-03-18T15:04:34.719 INFO:tasks.cephadm:First mon is mon.a on smithi019 2024-03-18T15:04:34.719 INFO:tasks.cephadm:First mgr is x 2024-03-18T15:04:34.719 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-18T15:04:34.720 DEBUG:teuthology.orchestra.run.smithi019:> sudo hostname $(hostname -s) 2024-03-18T15:04:34.827 DEBUG:teuthology.orchestra.run.smithi033:> sudo hostname $(hostname -s) 2024-03-18T15:04:34.862 DEBUG:teuthology.orchestra.run.smithi104:> sudo hostname $(hostname -s) 2024-03-18T15:04:34.890 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-03-18T15:04:34.890 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-18T15:04:35.122 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/77658/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-2162-g3d78f5ba', 'node_name': '172.21.5.34+adami04', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-2162.g3d78f5ba'}, 'url': 'https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-15 01:33:04.210510', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-batrick-testing-20240314.183243', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-03-18T15:04:35.259 INFO:tasks.util.chacra:got chacra host 3.chacra.ceph.com, ref wip-batrick-testing-20240314.183243, sha1 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-18T15:04:35.261 INFO:tasks.cephadm:Discovered cachra url: https://3.chacra.ceph.com/binaries/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/x86_64/flavors/default/cephadm 2024-03-18T15:04:35.261 DEBUG:teuthology.orchestra.run.smithi019:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-18T15:04:36.131 INFO:teuthology.orchestra.run.smithi019.stdout:-rw-r--r--. 1 ubuntu ubuntu 767536 Mar 18 15:04 /home/ubuntu/cephtest/cephadm 2024-03-18T15:04:36.131 DEBUG:teuthology.orchestra.run.smithi033:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-18T15:04:36.763 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-r--r--. 1 ubuntu ubuntu 767536 Mar 18 15:04 /home/ubuntu/cephtest/cephadm 2024-03-18T15:04:36.763 DEBUG:teuthology.orchestra.run.smithi104:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-18T15:04:37.411 INFO:teuthology.orchestra.run.smithi104.stdout:-rw-r--r--. 1 ubuntu ubuntu 767536 Mar 18 15:04 /home/ubuntu/cephtest/cephadm 2024-03-18T15:04:37.412 DEBUG:teuthology.orchestra.run.smithi019:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-18T15:04:37.475 DEBUG:teuthology.orchestra.run.smithi033:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-18T15:04:37.494 DEBUG:teuthology.orchestra.run.smithi104:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-18T15:04:37.529 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af on all hosts... 2024-03-18T15:04:37.530 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af pull 2024-03-18T15:04:37.537 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af pull 2024-03-18T15:04:37.539 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af pull 2024-03-18T15:04:38.307 INFO:teuthology.orchestra.run.smithi104.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af... 2024-03-18T15:04:38.513 INFO:teuthology.orchestra.run.smithi033.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af... 2024-03-18T15:04:38.807 INFO:teuthology.orchestra.run.smithi019.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af... 2024-03-18T15:05:39.160 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-03-18T15:05:39.161 INFO:teuthology.orchestra.run.smithi033.stdout: "ceph_version": "ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev)", 2024-03-18T15:05:39.161 INFO:teuthology.orchestra.run.smithi033.stdout: "image_id": "be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae", 2024-03-18T15:05:39.161 INFO:teuthology.orchestra.run.smithi033.stdout: "repo_digests": [ 2024-03-18T15:05:39.161 INFO:teuthology.orchestra.run.smithi033.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513" 2024-03-18T15:05:39.161 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-03-18T15:05:39.161 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-03-18T15:05:40.488 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-03-18T15:05:40.488 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph_version": "ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev)", 2024-03-18T15:05:40.488 INFO:teuthology.orchestra.run.smithi104.stdout: "image_id": "be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae", 2024-03-18T15:05:40.488 INFO:teuthology.orchestra.run.smithi104.stdout: "repo_digests": [ 2024-03-18T15:05:40.489 INFO:teuthology.orchestra.run.smithi104.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513" 2024-03-18T15:05:40.489 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-03-18T15:05:40.489 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-03-18T15:05:44.926 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-03-18T15:05:44.926 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph_version": "ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev)", 2024-03-18T15:05:44.926 INFO:teuthology.orchestra.run.smithi019.stdout: "image_id": "be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae", 2024-03-18T15:05:44.926 INFO:teuthology.orchestra.run.smithi019.stdout: "repo_digests": [ 2024-03-18T15:05:44.926 INFO:teuthology.orchestra.run.smithi019.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513" 2024-03-18T15:05:44.926 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-03-18T15:05:44.926 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-03-18T15:05:44.988 DEBUG:teuthology.orchestra.run.smithi019:> sudo mkdir -p /etc/ceph 2024-03-18T15:05:45.093 DEBUG:teuthology.orchestra.run.smithi033:> sudo mkdir -p /etc/ceph 2024-03-18T15:05:45.133 DEBUG:teuthology.orchestra.run.smithi104:> sudo mkdir -p /etc/ceph 2024-03-18T15:05:45.168 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod 777 /etc/ceph 2024-03-18T15:05:45.273 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod 777 /etc/ceph 2024-03-18T15:05:45.308 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod 777 /etc/ceph 2024-03-18T15:05:45.340 INFO:tasks.cephadm:Writing seed config... 2024-03-18T15:05:45.341 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-03-18T15:05:45.341 INFO:tasks.cephadm: override: [client] debug client = 20 2024-03-18T15:05:45.341 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-03-18T15:05:45.341 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-03-18T15:05:45.341 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-03-18T15:05:45.342 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-03-18T15:05:45.342 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-03-18T15:05:45.342 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-03-18T15:05:45.342 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-03-18T15:05:45.342 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-03-18T15:05:45.342 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-03-18T15:05:45.342 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-03-18T15:05:45.342 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-03-18T15:05:45.342 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-03-18T15:05:45.342 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-03-18T15:05:45.342 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-03-18T15:05:45.342 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-03-18T15:05:45.342 INFO:tasks.cephadm: override: [mds] mds_bal_replicate_threshold = 1 2024-03-18T15:05:45.342 INFO:tasks.cephadm: override: [mds] mds_export_ephemeral_random_max = 0.1 2024-03-18T15:05:45.343 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-03-18T15:05:45.343 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-03-18T15:05:45.343 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-03-18T15:05:45.343 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-03-18T15:05:45.343 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2024-03-18T15:05:45.343 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-18T15:05:45.343 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-18T15:05:45.343 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-18T15:05:45.343 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-18T15:05:45.343 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-18T15:05:45.343 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-03-18T15:05:45.343 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-03-18T15:05:45.343 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2024-03-18T15:05:45.343 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-03-18T15:05:45.344 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-03-18T15:05:45.344 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-03-18T15:05:45.344 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-18T15:05:45.344 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-18T15:05:45.344 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-03-18T15:05:45.344 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-03-18T15:05:45.344 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-03-18T15:05:45.344 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-03-18T15:05:45.344 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-03-18T15:05:45.344 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-03-18T15:05:45.344 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-03-18T15:05:45.344 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-18T15:05:45.344 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2024-03-18T15:05:45.345 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-03-18T15:05:45.345 DEBUG:teuthology.orchestra.run.smithi019:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-18T15:05:45.405 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = d4dc2ba6-e538-11ee-95c9-87774f69a715 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_bal_replicate_threshold = 1 mds_export_ephemeral_random_max = 0.1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-03-18T15:05:45.405 DEBUG:teuthology.orchestra.run.smithi019:mon.a> sudo journalctl -f -n 0 -u ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@mon.a.service 2024-03-18T15:05:45.412 DEBUG:teuthology.orchestra.run.smithi019:mgr.x> sudo journalctl -f -n 0 -u ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@mgr.x.service 2024-03-18T15:05:45.457 INFO:tasks.cephadm:Bootstrapping... 2024-03-18T15:05:45.457 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af -v bootstrap --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.19 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-18T15:05:45.743 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-03-18T15:05:45.743 INFO:teuthology.orchestra.run.smithi019.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af', '-v', 'bootstrap', '--fsid', 'd4dc2ba6-e538-11ee-95c9-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.19', '--skip-admin-label'] 2024-03-18T15:05:45.802 INFO:teuthology.orchestra.run.smithi019.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-18T15:05:45.802 INFO:teuthology.orchestra.run.smithi019.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-18T15:05:45.803 INFO:teuthology.orchestra.run.smithi019.stdout:Verifying podman|docker is present... 2024-03-18T15:05:45.856 INFO:teuthology.orchestra.run.smithi019.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-18T15:05:45.856 INFO:teuthology.orchestra.run.smithi019.stdout:Verifying lvm2 is present... 2024-03-18T15:05:45.856 INFO:teuthology.orchestra.run.smithi019.stdout:Verifying time synchronization is in place... 2024-03-18T15:05:45.872 INFO:teuthology.orchestra.run.smithi019.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-18T15:05:45.872 INFO:teuthology.orchestra.run.smithi019.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-18T15:05:45.888 INFO:teuthology.orchestra.run.smithi019.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-18T15:05:45.888 INFO:teuthology.orchestra.run.smithi019.stdout:systemctl: stdout inactive 2024-03-18T15:05:45.913 INFO:teuthology.orchestra.run.smithi019.stdout:systemctl: stdout enabled 2024-03-18T15:05:45.933 INFO:teuthology.orchestra.run.smithi019.stdout:systemctl: stdout active 2024-03-18T15:05:45.933 INFO:teuthology.orchestra.run.smithi019.stdout:Unit chronyd.service is enabled and running 2024-03-18T15:05:45.933 INFO:teuthology.orchestra.run.smithi019.stdout:Repeating the final host check... 2024-03-18T15:05:45.995 INFO:teuthology.orchestra.run.smithi019.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-18T15:05:45.996 INFO:teuthology.orchestra.run.smithi019.stdout:podman (/bin/podman) version 4.9.3 is present 2024-03-18T15:05:45.996 INFO:teuthology.orchestra.run.smithi019.stdout:systemctl is present 2024-03-18T15:05:45.996 INFO:teuthology.orchestra.run.smithi019.stdout:lvcreate is present 2024-03-18T15:05:46.014 INFO:teuthology.orchestra.run.smithi019.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-18T15:05:46.014 INFO:teuthology.orchestra.run.smithi019.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-18T15:05:46.036 INFO:teuthology.orchestra.run.smithi019.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-18T15:05:46.036 INFO:teuthology.orchestra.run.smithi019.stdout:systemctl: stdout inactive 2024-03-18T15:05:46.062 INFO:teuthology.orchestra.run.smithi019.stdout:systemctl: stdout enabled 2024-03-18T15:05:46.084 INFO:teuthology.orchestra.run.smithi019.stdout:systemctl: stdout active 2024-03-18T15:05:46.084 INFO:teuthology.orchestra.run.smithi019.stdout:Unit chronyd.service is enabled and running 2024-03-18T15:05:46.084 INFO:teuthology.orchestra.run.smithi019.stdout:Host looks OK 2024-03-18T15:05:46.084 INFO:teuthology.orchestra.run.smithi019.stdout:Cluster fsid: d4dc2ba6-e538-11ee-95c9-87774f69a715 2024-03-18T15:05:46.085 INFO:teuthology.orchestra.run.smithi019.stdout:Acquiring lock 140198339108288 on /run/cephadm/d4dc2ba6-e538-11ee-95c9-87774f69a715.lock 2024-03-18T15:05:46.085 INFO:teuthology.orchestra.run.smithi019.stdout:Lock 140198339108288 acquired on /run/cephadm/d4dc2ba6-e538-11ee-95c9-87774f69a715.lock 2024-03-18T15:05:46.087 INFO:teuthology.orchestra.run.smithi019.stdout:Verifying IP 172.21.15.19 port 3300 ... 2024-03-18T15:05:46.088 INFO:teuthology.orchestra.run.smithi019.stdout:Verifying IP 172.21.15.19 port 6789 ... 2024-03-18T15:05:46.088 INFO:teuthology.orchestra.run.smithi019.stdout:Base mon IP(s) is [172.21.15.19:3300, 172.21.15.19:6789], mon addrv is [v2:172.21.15.19:3300,v1:172.21.15.19:6789] 2024-03-18T15:05:46.104 INFO:teuthology.orchestra.run.smithi019.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.19 metric 100 2024-03-18T15:05:46.104 INFO:teuthology.orchestra.run.smithi019.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.19 metric 100 2024-03-18T15:05:46.116 INFO:teuthology.orchestra.run.smithi019.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-18T15:05:46.116 INFO:teuthology.orchestra.run.smithi019.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-03-18T15:05:46.117 INFO:teuthology.orchestra.run.smithi019.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-03-18T15:05:46.127 INFO:teuthology.orchestra.run.smithi019.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-18T15:05:46.128 INFO:teuthology.orchestra.run.smithi019.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-18T15:05:46.128 INFO:teuthology.orchestra.run.smithi019.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-18T15:05:46.128 INFO:teuthology.orchestra.run.smithi019.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-03-18T15:05:46.128 INFO:teuthology.orchestra.run.smithi019.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:161e/64 scope link noprefixroute 2024-03-18T15:05:46.129 INFO:teuthology.orchestra.run.smithi019.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-18T15:05:46.131 INFO:teuthology.orchestra.run.smithi019.stdout:Mon IP `172.21.15.19` is in CIDR network `172.21.0.0/20` 2024-03-18T15:05:46.131 INFO:teuthology.orchestra.run.smithi019.stdout:Mon IP `172.21.15.19` is in CIDR network `172.21.0.0/20` 2024-03-18T15:05:46.131 INFO:teuthology.orchestra.run.smithi019.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-18T15:05:46.134 INFO:teuthology.orchestra.run.smithi019.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-18T15:05:46.137 INFO:teuthology.orchestra.run.smithi019.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af... 2024-03-18T15:05:46.844 INFO:teuthology.orchestra.run.smithi019.stdout:/bin/podman: stdout be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae 2024-03-18T15:05:46.844 INFO:teuthology.orchestra.run.smithi019.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af... 2024-03-18T15:05:46.845 INFO:teuthology.orchestra.run.smithi019.stdout:/bin/podman: stderr Getting image source signatures 2024-03-18T15:05:46.845 INFO:teuthology.orchestra.run.smithi019.stdout:/bin/podman: stderr Copying blob sha256:66487374cb611c95435850c17b53ebb64c22ed9555f72cd43d678b60262a12b1 2024-03-18T15:05:46.845 INFO:teuthology.orchestra.run.smithi019.stdout:/bin/podman: stderr Copying blob sha256:f58b4670ef4403d11d1c912aa26a4dbcf73447df5302e9dc66f6a7fc0d9d1e47 2024-03-18T15:05:46.845 INFO:teuthology.orchestra.run.smithi019.stdout:/bin/podman: stderr Copying config sha256:be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae 2024-03-18T15:05:46.845 INFO:teuthology.orchestra.run.smithi019.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-18T15:05:48.148 INFO:teuthology.orchestra.run.smithi019.stdout:ceph: stdout ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev) 2024-03-18T15:05:48.148 INFO:teuthology.orchestra.run.smithi019.stdout:Ceph version: ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev) 2024-03-18T15:05:48.148 INFO:teuthology.orchestra.run.smithi019.stdout:Extracting ceph user uid/gid from container image... 2024-03-18T15:05:49.332 INFO:teuthology.orchestra.run.smithi019.stdout:stat: stdout 167 167 2024-03-18T15:05:49.333 INFO:teuthology.orchestra.run.smithi019.stdout:Creating initial keys... 2024-03-18T15:05:50.483 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph-authtool: stdout AQBNWPhlOwTwMBAAuJXMc8qVR19XqUgZx30f/w== 2024-03-18T15:05:51.693 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph-authtool: stdout AQBOWPhl4VetOhAAZYMuNoTvFUO2wagzbNpOxw== 2024-03-18T15:05:52.941 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph-authtool: stdout AQBQWPhlUJArDBAAnzHvvwDV+yjo1BZHEXiGRw== 2024-03-18T15:05:52.942 INFO:teuthology.orchestra.run.smithi019.stdout:Creating initial monmap... 2024-03-18T15:05:54.072 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-18T15:05:54.072 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-18T15:05:54.072 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to d4dc2ba6-e538-11ee-95c9-87774f69a715 2024-03-18T15:05:54.072 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-18T15:05:54.072 INFO:teuthology.orchestra.run.smithi019.stdout:monmaptool for a [v2:172.21.15.19:3300,v1:172.21.15.19:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-18T15:05:54.072 INFO:teuthology.orchestra.run.smithi019.stdout:setting min_mon_release = quincy 2024-03-18T15:05:54.072 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/monmaptool: set fsid to d4dc2ba6-e538-11ee-95c9-87774f69a715 2024-03-18T15:05:54.073 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-18T15:05:54.073 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-18T15:05:54.073 INFO:teuthology.orchestra.run.smithi019.stdout:Creating mon... 2024-03-18T15:05:55.189 INFO:teuthology.orchestra.run.smithi019.stdout:create mon.a on 2024-03-18T15:05:55.552 INFO:teuthology.orchestra.run.smithi019.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-03-18T15:05:55.795 INFO:teuthology.orchestra.run.smithi019.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-18T15:05:56.097 INFO:teuthology.orchestra.run.smithi019.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715.target → /etc/systemd/system/ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715.target. 2024-03-18T15:05:56.097 INFO:teuthology.orchestra.run.smithi019.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715.target → /etc/systemd/system/ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715.target. 2024-03-18T15:05:56.496 INFO:teuthology.orchestra.run.smithi019.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@mon.a 2024-03-18T15:05:56.496 INFO:teuthology.orchestra.run.smithi019.stdout:systemctl: stderr Failed to reset failed state of unit ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@mon.a.service: Unit ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@mon.a.service not loaded. 2024-03-18T15:05:56.789 INFO:teuthology.orchestra.run.smithi019.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715.target.wants/ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@mon.a.service → /etc/systemd/system/ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@.service. 2024-03-18T15:05:57.450 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:05:57 smithi019 podman[31135]: 2024-03-18 15:05:57.179595949 +0000 UTC m=+0.128945338 container create 21b5d75b760ddac43a9d6d13e0e795f441956c7ac703d23639202aa890af5ba9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mon-a, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T15:05:57.451 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:05:57 smithi019 podman[31135]: 2024-03-18 15:05:57.449546318 +0000 UTC m=+0.398895708 container init 21b5d75b760ddac43a9d6d13e0e795f441956c7ac703d23639202aa890af5ba9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mon-a, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T15:05:57.569 INFO:teuthology.orchestra.run.smithi019.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-18T15:05:57.569 INFO:teuthology.orchestra.run.smithi019.stdout:systemctl: stdout disabled 2024-03-18T15:05:57.584 INFO:teuthology.orchestra.run.smithi019.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-18T15:05:57.584 INFO:teuthology.orchestra.run.smithi019.stdout:systemctl: stdout inactive 2024-03-18T15:05:57.584 INFO:teuthology.orchestra.run.smithi019.stdout:firewalld.service is not enabled 2024-03-18T15:05:57.584 INFO:teuthology.orchestra.run.smithi019.stdout:Not possible to enable service . firewalld.service is not available 2024-03-18T15:05:57.584 INFO:teuthology.orchestra.run.smithi019.stdout:Waiting for mon to start... 2024-03-18T15:05:57.585 INFO:teuthology.orchestra.run.smithi019.stdout:Waiting for mon... 2024-03-18T15:05:57.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:05:57 smithi019 podman[31135]: 2024-03-18 15:05:57.460381175 +0000 UTC m=+0.409730565 container start 21b5d75b760ddac43a9d6d13e0e795f441956c7ac703d23639202aa890af5ba9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mon-a, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-03-18T15:05:57.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:05:57 smithi019 bash[31135]: 21b5d75b760ddac43a9d6d13e0e795f441956c7ac703d23639202aa890af5ba9 2024-03-18T15:05:57.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:05:57 smithi019 ceph-mon[31154]: mkfs d4dc2ba6-e538-11ee-95c9-87774f69a715 2024-03-18T15:05:57.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:05:57 smithi019 ceph-mon[31154]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-18T15:05:57.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:05:57 smithi019 systemd[1]: Started Ceph mon.a for d4dc2ba6-e538-11ee-95c9-87774f69a715. 2024-03-18T15:05:58.890 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:05:58 smithi019 ceph-mon[31154]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-18T15:05:58.890 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:05:58 smithi019 ceph-mon[31154]: monmap epoch 1 2024-03-18T15:05:58.890 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:05:58 smithi019 ceph-mon[31154]: fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 2024-03-18T15:05:58.890 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:05:58 smithi019 ceph-mon[31154]: last_changed 2024-03-18T15:05:53.426642+0000 2024-03-18T15:05:58.891 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:05:58 smithi019 ceph-mon[31154]: created 2024-03-18T15:05:53.426642+0000 2024-03-18T15:05:58.891 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:05:58 smithi019 ceph-mon[31154]: min_mon_release 19 (squid) 2024-03-18T15:05:58.891 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:05:58 smithi019 ceph-mon[31154]: election_strategy: 1 2024-03-18T15:05:58.891 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:05:58 smithi019 ceph-mon[31154]: 0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-03-18T15:05:58.891 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:05:58 smithi019 ceph-mon[31154]: fsmap 2024-03-18T15:05:58.891 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:05:58 smithi019 ceph-mon[31154]: osdmap e1: 0 total, 0 up, 0 in 2024-03-18T15:05:58.891 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:05:58 smithi019 ceph-mon[31154]: mgrmap e1: no daemons active 2024-03-18T15:05:58.891 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:05:58 smithi019 ceph-mon[31154]: from='client.? 172.21.15.19:0/2529646634' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-18T15:05:58.960 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout cluster: 2024-03-18T15:05:58.967 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout id: d4dc2ba6-e538-11ee-95c9-87774f69a715 2024-03-18T15:05:58.967 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-18T15:05:58.967 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout 2024-03-18T15:05:58.968 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout services: 2024-03-18T15:05:58.968 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.714446s) 2024-03-18T15:05:58.968 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-18T15:05:58.968 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-18T15:05:58.968 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout 2024-03-18T15:05:58.968 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout data: 2024-03-18T15:05:58.968 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-18T15:05:58.968 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-18T15:05:58.968 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-18T15:05:58.968 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout pgs: 2024-03-18T15:05:58.968 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout 2024-03-18T15:05:58.968 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:05:58.156+0000 7fdfdbe00700 1 Processor -- start 2024-03-18T15:05:58.968 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:05:58.156+0000 7fdfdbe00700 1 -- start start 2024-03-18T15:05:58.969 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:05:58.156+0000 7fdfdbe00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fdfd4106560 0x7fdfd4106980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:05:58.969 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:05:58.157+0000 7fdfdbe00700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fdfd407c6d0 con 0x7fdfd4106560 2024-03-18T15:05:58.969 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:05:58.158+0000 7fdfd9400700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fdfd4106560 0x7fdfd4106980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:05:58.969 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:05:58.158+0000 7fdfd9400700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fdfd4106560 0x7fdfd4106980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:44708/0 (socket says 172.21.15.19:44708) 2024-03-18T15:05:58.969 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:05:58.158+0000 7fdfd9400700 1 -- 172.21.15.19:0/3640757981 learned_addr learned my addr 172.21.15.19:0/3640757981 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:05:58.969 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:05:58.158+0000 7fdfd9400700 1 -- 172.21.15.19:0/3640757981 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfd4106ec0 con 0x7fdfd4106560 2024-03-18T15:05:58.969 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:05:58.159+0000 7fdfd9400700 1 --2- 172.21.15.19:0/3640757981 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fdfd4106560 0x7fdfd4106980 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fdfc8009e80 tx=0x7fdfc800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d01dbf26616d5a98 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:05:58.969 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:05:58.159+0000 7fdfd3e00700 1 -- 172.21.15.19:0/3640757981 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdfc8017070 con 0x7fdfd4106560 2024-03-18T15:05:58.969 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:05:58.159+0000 7fdfd3e00700 1 -- 172.21.15.19:0/3640757981 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdfc8003a20 con 0x7fdfd4106560 2024-03-18T15:05:58.969 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:05:58.159+0000 7fdfd3e00700 1 -- 172.21.15.19:0/3640757981 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdfc8003d20 con 0x7fdfd4106560 2024-03-18T15:05:58.969 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:05:58.160+0000 7fdfdbe00700 1 -- 172.21.15.19:0/3640757981 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fdfd4106560 msgr2=0x7fdfd4106980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:05:58.969 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:05:58.160+0000 7fdfdbe00700 1 --2- 172.21.15.19:0/3640757981 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fdfd4106560 0x7fdfd4106980 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fdfc8009e80 tx=0x7fdfc800b330 comp rx=0 tx=0).stop 2024-03-18T15:05:58.969 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:05:58.160+0000 7fdfdbe00700 1 -- 172.21.15.19:0/3640757981 shutdown_connections 2024-03-18T15:05:58.969 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:05:58.160+0000 7fdfdbe00700 1 --2- 172.21.15.19:0/3640757981 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fdfd4106560 0x7fdfd4106980 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:05:58.969 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:05:58.160+0000 7fdfdbe00700 1 -- 172.21.15.19:0/3640757981 >> 172.21.15.19:0/3640757981 conn(0x7fdfd4102100 msgr2=0x7fdfd4104580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:05:58.970 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:05:58.160+0000 7fdfdbe00700 1 -- 172.21.15.19:0/3640757981 shutdown_connections 2024-03-18T15:05:58.970 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:05:58.160+0000 7fdfdbe00700 1 -- 172.21.15.19:0/3640757981 wait complete. 2024-03-18T15:05:58.970 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:05:58.162+0000 7fdfdbe00700 1 Processor -- start 2024-03-18T15:05:58.970 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:05:58.162+0000 7fdfdbe00700 1 -- start start 2024-03-18T15:05:58.970 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:05:58.162+0000 7fdfdbe00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fdfd4106560 0x7fdfd419d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:05:58.970 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:05:58.162+0000 7fdfdbe00700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fdfd41073f0 con 0x7fdfd4106560 2024-03-18T15:05:58.970 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:05:58.163+0000 7fdfd9400700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fdfd4106560 0x7fdfd419d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:05:58.970 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:05:58.163+0000 7fdfd9400700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fdfd4106560 0x7fdfd419d640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:44718/0 (socket says 172.21.15.19:44718) 2024-03-18T15:05:58.970 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:05:58.163+0000 7fdfd9400700 1 -- 172.21.15.19:0/2529646634 learned_addr learned my addr 172.21.15.19:0/2529646634 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:05:58.970 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:05:58.164+0000 7fdfd9400700 1 -- 172.21.15.19:0/2529646634 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfd419db80 con 0x7fdfd4106560 2024-03-18T15:05:58.970 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:05:58.164+0000 7fdfd9400700 1 --2- 172.21.15.19:0/2529646634 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fdfd4106560 0x7fdfd419d640 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fdfd407bb10 tx=0x7fdfc8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:05:58.970 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:05:58.164+0000 7fdfd2000700 1 -- 172.21.15.19:0/2529646634 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdfc8022020 con 0x7fdfd4106560 2024-03-18T15:05:58.970 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:05:58.164+0000 7fdfd2000700 1 -- 172.21.15.19:0/2529646634 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdfc8018040 con 0x7fdfd4106560 2024-03-18T15:05:58.971 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:05:58.164+0000 7fdfd2000700 1 -- 172.21.15.19:0/2529646634 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdfc8017070 con 0x7fdfd4106560 2024-03-18T15:05:58.971 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:05:58.164+0000 7fdfdbe00700 1 -- 172.21.15.19:0/2529646634 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfd419de10 con 0x7fdfd4106560 2024-03-18T15:05:58.971 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:05:58.165+0000 7fdfdbe00700 1 -- 172.21.15.19:0/2529646634 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfd41a0b00 con 0x7fdfd4106560 2024-03-18T15:05:58.971 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:05:58.165+0000 7fdfd2000700 1 -- 172.21.15.19:0/2529646634 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fdfc8025020 con 0x7fdfd4106560 2024-03-18T15:05:58.971 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:05:58.165+0000 7fdfd2000700 1 -- 172.21.15.19:0/2529646634 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fdfc800f050 con 0x7fdfd4106560 2024-03-18T15:05:58.971 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:05:58.166+0000 7fdfdbe00700 1 -- 172.21.15.19:0/2529646634 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdfd40693d0 con 0x7fdfd4106560 2024-03-18T15:05:58.971 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:05:58.169+0000 7fdfd2000700 1 -- 172.21.15.19:0/2529646634 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fdfc802c030 con 0x7fdfd4106560 2024-03-18T15:05:58.971 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:05:58.250+0000 7fdfdbe00700 1 -- 172.21.15.19:0/2529646634 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7fdfd41a1020 con 0x7fdfd4106560 2024-03-18T15:05:58.971 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:05:58.251+0000 7fdfd2000700 1 -- 172.21.15.19:0/2529646634 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+317 (secure 0 0 0) 0x7fdfc801e480 con 0x7fdfd4106560 2024-03-18T15:05:58.971 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:05:58.252+0000 7fdfdbe00700 1 -- 172.21.15.19:0/2529646634 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fdfd4106560 msgr2=0x7fdfd419d640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:05:58.971 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:05:58.252+0000 7fdfdbe00700 1 --2- 172.21.15.19:0/2529646634 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fdfd4106560 0x7fdfd419d640 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fdfd407bb10 tx=0x7fdfc8004ad0 comp rx=0 tx=0).stop 2024-03-18T15:05:58.971 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:05:58.252+0000 7fdfdbe00700 1 -- 172.21.15.19:0/2529646634 shutdown_connections 2024-03-18T15:05:58.971 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:05:58.252+0000 7fdfdbe00700 1 --2- 172.21.15.19:0/2529646634 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fdfd4106560 0x7fdfd419d640 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:05:58.971 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:05:58.252+0000 7fdfdbe00700 1 -- 172.21.15.19:0/2529646634 >> 172.21.15.19:0/2529646634 conn(0x7fdfd4102100 msgr2=0x7fdfd4193c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:05:58.972 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:05:58.252+0000 7fdfdbe00700 1 -- 172.21.15.19:0/2529646634 shutdown_connections 2024-03-18T15:05:58.972 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:05:58.253+0000 7fdfdbe00700 1 -- 172.21.15.19:0/2529646634 wait complete. 2024-03-18T15:05:58.972 INFO:teuthology.orchestra.run.smithi019.stdout:mon is available 2024-03-18T15:05:58.972 INFO:teuthology.orchestra.run.smithi019.stdout:Assimilating anything we can from ceph.conf... 2024-03-18T15:05:59.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:05:59 smithi019 ceph-mon[31154]: from='client.? 172.21.15.19:0/3244368158' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-18T15:05:59.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:05:59 smithi019 ceph-mon[31154]: from='client.? 172.21.15.19:0/3244368158' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-18T15:06:00.317 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout 2024-03-18T15:06:00.317 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout [global] 2024-03-18T15:06:00.317 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout fsid = d4dc2ba6-e538-11ee-95c9-87774f69a715 2024-03-18T15:06:00.317 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.19:3300,v1:172.21.15.19:6789] 2024-03-18T15:06:00.318 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-18T15:06:00.319 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-18T15:06:00.319 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-18T15:06:00.319 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-18T15:06:00.319 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout 2024-03-18T15:06:00.319 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-18T15:06:00.319 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-18T15:06:00.319 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout 2024-03-18T15:06:00.319 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout [osd] 2024-03-18T15:06:00.321 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-18T15:06:00.321 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-18T15:06:00.321 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-18T15:06:00.321 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-18T15:06:00.321 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-18T15:06:00.321 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:05:59.551+0000 7f3b69a00700 1 Processor -- start 2024-03-18T15:06:00.322 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:05:59.552+0000 7f3b69a00700 1 -- start start 2024-03-18T15:06:00.322 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:05:59.552+0000 7f3b69a00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3b641065d0 0x7f3b641069f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:06:00.322 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:05:59.552+0000 7f3b69a00700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f3b6407be00 con 0x7f3b641065d0 2024-03-18T15:06:00.322 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:05:59.552+0000 7f3b62a00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3b641065d0 0x7f3b641069f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:06:00.322 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:05:59.553+0000 7f3b62a00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3b641065d0 0x7f3b641069f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:44724/0 (socket says 172.21.15.19:44724) 2024-03-18T15:06:00.322 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:05:59.553+0000 7f3b62a00700 1 -- 172.21.15.19:0/3439380308 learned_addr learned my addr 172.21.15.19:0/3439380308 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:06:00.322 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:05:59.553+0000 7f3b62a00700 1 -- 172.21.15.19:0/3439380308 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b64106f30 con 0x7f3b641065d0 2024-03-18T15:06:00.322 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:05:59.553+0000 7f3b62a00700 1 --2- 172.21.15.19:0/3439380308 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3b641065d0 0x7f3b641069f0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f3b540098a0 tx=0x7f3b54009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29a395ff3f0b86b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:06:00.322 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:05:59.554+0000 7f3b61600700 1 -- 172.21.15.19:0/3439380308 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3b54004160 con 0x7f3b641065d0 2024-03-18T15:06:00.322 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:05:59.554+0000 7f3b61600700 1 -- 172.21.15.19:0/3439380308 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b54004300 con 0x7f3b641065d0 2024-03-18T15:06:00.322 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:05:59.554+0000 7f3b61600700 1 -- 172.21.15.19:0/3439380308 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3b540036a0 con 0x7f3b641065d0 2024-03-18T15:06:00.326 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:05:59.554+0000 7f3b69a00700 1 -- 172.21.15.19:0/3439380308 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3b641065d0 msgr2=0x7f3b641069f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:06:00.326 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:05:59.554+0000 7f3b69a00700 1 --2- 172.21.15.19:0/3439380308 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3b641065d0 0x7f3b641069f0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f3b540098a0 tx=0x7f3b54009bb0 comp rx=0 tx=0).stop 2024-03-18T15:06:00.326 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:05:59.555+0000 7f3b69a00700 1 -- 172.21.15.19:0/3439380308 shutdown_connections 2024-03-18T15:06:00.326 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:05:59.555+0000 7f3b69a00700 1 --2- 172.21.15.19:0/3439380308 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3b641065d0 0x7f3b641069f0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:06:00.326 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:05:59.555+0000 7f3b69a00700 1 -- 172.21.15.19:0/3439380308 >> 172.21.15.19:0/3439380308 conn(0x7f3b64102130 msgr2=0x7f3b641045b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:06:00.326 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:05:59.555+0000 7f3b69a00700 1 -- 172.21.15.19:0/3439380308 shutdown_connections 2024-03-18T15:06:00.326 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:05:59.555+0000 7f3b69a00700 1 -- 172.21.15.19:0/3439380308 wait complete. 2024-03-18T15:06:00.327 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:05:59.556+0000 7f3b69a00700 1 Processor -- start 2024-03-18T15:06:00.327 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:05:59.556+0000 7f3b69a00700 1 -- start start 2024-03-18T15:06:00.327 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:05:59.557+0000 7f3b69a00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3b641065d0 0x7f3b6419d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:06:00.327 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:05:59.557+0000 7f3b69a00700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f3b64107460 con 0x7f3b641065d0 2024-03-18T15:06:00.327 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:05:59.557+0000 7f3b62a00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3b641065d0 0x7f3b6419d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:06:00.327 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:05:59.557+0000 7f3b62a00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3b641065d0 0x7f3b6419d5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:44730/0 (socket says 172.21.15.19:44730) 2024-03-18T15:06:00.327 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:05:59.557+0000 7f3b62a00700 1 -- 172.21.15.19:0/3244368158 learned_addr learned my addr 172.21.15.19:0/3244368158 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:06:00.327 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:05:59.557+0000 7f3b62a00700 1 -- 172.21.15.19:0/3244368158 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b6419daf0 con 0x7f3b641065d0 2024-03-18T15:06:00.327 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:05:59.558+0000 7f3b62a00700 1 --2- 172.21.15.19:0/3244368158 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3b641065d0 0x7f3b6419d5b0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f3b54000c00 tx=0x7f3b54004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:06:00.327 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:05:59.558+0000 7f3b5b400700 1 -- 172.21.15.19:0/3244368158 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3b54003db0 con 0x7f3b641065d0 2024-03-18T15:06:00.327 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:05:59.558+0000 7f3b5b400700 1 -- 172.21.15.19:0/3244368158 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b54011630 con 0x7f3b641065d0 2024-03-18T15:06:00.327 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:05:59.558+0000 7f3b69a00700 1 -- 172.21.15.19:0/3244368158 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b6419dd80 con 0x7f3b641065d0 2024-03-18T15:06:00.327 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:05:59.558+0000 7f3b5b400700 1 -- 172.21.15.19:0/3244368158 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3b54011950 con 0x7f3b641065d0 2024-03-18T15:06:00.327 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:05:59.558+0000 7f3b69a00700 1 -- 172.21.15.19:0/3244368158 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b641a0a70 con 0x7f3b641065d0 2024-03-18T15:06:00.328 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:05:59.559+0000 7f3b5b400700 1 -- 172.21.15.19:0/3244368158 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f3b54011af0 con 0x7f3b641065d0 2024-03-18T15:06:00.328 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:05:59.559+0000 7f3b5b400700 1 -- 172.21.15.19:0/3244368158 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f3b5401c450 con 0x7f3b641065d0 2024-03-18T15:06:00.328 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:05:59.559+0000 7f3b69a00700 1 -- 172.21.15.19:0/3244368158 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b640693d0 con 0x7f3b641065d0 2024-03-18T15:06:00.328 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:05:59.562+0000 7f3b5b400700 1 -- 172.21.15.19:0/3244368158 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f3b54032020 con 0x7f3b641065d0 2024-03-18T15:06:00.328 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:05:59.640+0000 7f3b69a00700 1 -- 172.21.15.19:0/3244368158 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f3b641a0c40 con 0x7f3b641065d0 2024-03-18T15:06:00.328 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:05:59.646+0000 7f3b5b400700 1 -- 172.21.15.19:0/3244368158 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+429 (secure 0 0 0) 0x7f3b54004260 con 0x7f3b641065d0 2024-03-18T15:06:00.328 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:05:59.647+0000 7f3b5b400700 1 -- 172.21.15.19:0/3244368158 <== mon.0 v2:172.21.15.19:3300/0 8 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f3b5401cb60 con 0x7f3b641065d0 2024-03-18T15:06:00.328 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:05:59.648+0000 7f3b69a00700 1 -- 172.21.15.19:0/3244368158 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3b641065d0 msgr2=0x7f3b6419d5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:06:00.328 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:05:59.648+0000 7f3b69a00700 1 --2- 172.21.15.19:0/3244368158 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3b641065d0 0x7f3b6419d5b0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f3b54000c00 tx=0x7f3b54004750 comp rx=0 tx=0).stop 2024-03-18T15:06:00.328 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:05:59.649+0000 7f3b69a00700 1 -- 172.21.15.19:0/3244368158 shutdown_connections 2024-03-18T15:06:00.328 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:05:59.649+0000 7f3b69a00700 1 --2- 172.21.15.19:0/3244368158 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3b641065d0 0x7f3b6419d5b0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:06:00.328 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:05:59.649+0000 7f3b69a00700 1 -- 172.21.15.19:0/3244368158 >> 172.21.15.19:0/3244368158 conn(0x7f3b64102130 msgr2=0x7f3b64193bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:06:00.328 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:05:59.649+0000 7f3b69a00700 1 -- 172.21.15.19:0/3244368158 shutdown_connections 2024-03-18T15:06:00.328 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:05:59.649+0000 7f3b69a00700 1 -- 172.21.15.19:0/3244368158 wait complete. 2024-03-18T15:06:00.329 INFO:teuthology.orchestra.run.smithi019.stdout:Generating new minimal ceph.conf... 2024-03-18T15:06:01.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:01 smithi019 ceph-mon[31154]: from='client.? 172.21.15.19:0/3747449241' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:06:01.677 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:00.933+0000 7f4cd6c00700 1 Processor -- start 2024-03-18T15:06:01.677 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:00.933+0000 7f4cd6c00700 1 -- start start 2024-03-18T15:06:01.678 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:00.934+0000 7f4cd6c00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4cd010a140 0x7f4cd010a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:06:01.678 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:00.934+0000 7f4cd6c00700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f4cd010ab30 con 0x7f4cd010a140 2024-03-18T15:06:01.678 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:00.934+0000 7f4cd5800700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4cd010a140 0x7f4cd010a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:06:01.678 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:00.934+0000 7f4cd5800700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4cd010a140 0x7f4cd010a560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:44734/0 (socket says 172.21.15.19:44734) 2024-03-18T15:06:01.678 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:00.934+0000 7f4cd5800700 1 -- 172.21.15.19:0/3744958018 learned_addr learned my addr 172.21.15.19:0/3744958018 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:06:01.679 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:00.935+0000 7f4cd5800700 1 -- 172.21.15.19:0/3744958018 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cd010af00 con 0x7f4cd010a140 2024-03-18T15:06:01.679 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:00.935+0000 7f4cd5800700 1 --2- 172.21.15.19:0/3744958018 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4cd010a140 0x7f4cd010a560 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f4cc40098a0 tx=0x7f4cc4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28d3caa2adea67a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:06:01.679 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:00.935+0000 7f4ccfe00700 1 -- 172.21.15.19:0/3744958018 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4cc4004160 con 0x7f4cd010a140 2024-03-18T15:06:01.679 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:00.935+0000 7f4ccfe00700 1 -- 172.21.15.19:0/3744958018 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f4cc4004300 con 0x7f4cd010a140 2024-03-18T15:06:01.680 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:00.935+0000 7f4ccfe00700 1 -- 172.21.15.19:0/3744958018 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4cc4011b50 con 0x7f4cd010a140 2024-03-18T15:06:01.680 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:00.936+0000 7f4cd6c00700 1 -- 172.21.15.19:0/3744958018 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4cd010a140 msgr2=0x7f4cd010a560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:06:01.680 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:00.936+0000 7f4cd6c00700 1 --2- 172.21.15.19:0/3744958018 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4cd010a140 0x7f4cd010a560 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f4cc40098a0 tx=0x7f4cc4009bb0 comp rx=0 tx=0).stop 2024-03-18T15:06:01.680 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:00.936+0000 7f4cd6c00700 1 -- 172.21.15.19:0/3744958018 shutdown_connections 2024-03-18T15:06:01.681 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:00.936+0000 7f4cd6c00700 1 --2- 172.21.15.19:0/3744958018 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4cd010a140 0x7f4cd010a560 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:06:01.681 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:00.936+0000 7f4cd6c00700 1 -- 172.21.15.19:0/3744958018 >> 172.21.15.19:0/3744958018 conn(0x7f4cd007d0f0 msgr2=0x7f4cd007d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:06:01.681 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:00.936+0000 7f4cd6c00700 1 -- 172.21.15.19:0/3744958018 shutdown_connections 2024-03-18T15:06:01.681 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:00.936+0000 7f4cd6c00700 1 -- 172.21.15.19:0/3744958018 wait complete. 2024-03-18T15:06:01.682 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:00.937+0000 7f4cd6c00700 1 Processor -- start 2024-03-18T15:06:01.682 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:00.937+0000 7f4cd6c00700 1 -- start start 2024-03-18T15:06:01.682 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:00.938+0000 7f4cd6c00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4cd010a140 0x7f4cd019dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:06:01.682 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:00.938+0000 7f4cd6c00700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f4cd010e2b0 con 0x7f4cd010a140 2024-03-18T15:06:01.682 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:00.938+0000 7f4cd5800700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4cd010a140 0x7f4cd019dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:06:01.684 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:00.938+0000 7f4cd5800700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4cd010a140 0x7f4cd019dd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:44746/0 (socket says 172.21.15.19:44746) 2024-03-18T15:06:01.684 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:00.938+0000 7f4cd5800700 1 -- 172.21.15.19:0/3747449241 learned_addr learned my addr 172.21.15.19:0/3747449241 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:06:01.684 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:00.938+0000 7f4cd5800700 1 -- 172.21.15.19:0/3747449241 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cd019e290 con 0x7f4cd010a140 2024-03-18T15:06:01.684 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:00.939+0000 7f4cd5800700 1 --2- 172.21.15.19:0/3747449241 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4cd010a140 0x7f4cd019dd50 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f4cc4011850 tx=0x7f4cc4011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:06:01.684 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:00.939+0000 7f4cce000700 1 -- 172.21.15.19:0/3747449241 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4cc4003af0 con 0x7f4cd010a140 2024-03-18T15:06:01.684 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:00.939+0000 7f4cce000700 1 -- 172.21.15.19:0/3747449241 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f4cc4020070 con 0x7f4cd010a140 2024-03-18T15:06:01.684 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:00.939+0000 7f4cd6c00700 1 -- 172.21.15.19:0/3747449241 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4cd019e520 con 0x7f4cd010a140 2024-03-18T15:06:01.685 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:00.939+0000 7f4cce000700 1 -- 172.21.15.19:0/3747449241 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4cc40195a0 con 0x7f4cd010a140 2024-03-18T15:06:01.685 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:00.939+0000 7f4cd6c00700 1 -- 172.21.15.19:0/3747449241 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4cd01a1210 con 0x7f4cd010a140 2024-03-18T15:06:01.686 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:00.939+0000 7f4cce000700 1 -- 172.21.15.19:0/3747449241 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f4cc4019be0 con 0x7f4cd010a140 2024-03-18T15:06:01.686 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:00.940+0000 7f4cce000700 1 -- 172.21.15.19:0/3747449241 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f4cc4011e00 con 0x7f4cd010a140 2024-03-18T15:06:01.686 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:00.940+0000 7f4cd6c00700 1 -- 172.21.15.19:0/3747449241 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4cd00693d0 con 0x7f4cd010a140 2024-03-18T15:06:01.686 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:00.943+0000 7f4cce000700 1 -- 172.21.15.19:0/3747449241 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f4cc4003c90 con 0x7f4cd010a140 2024-03-18T15:06:01.686 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:01.022+0000 7f4cd6c00700 1 -- 172.21.15.19:0/3747449241 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f4cd01a13e0 con 0x7f4cd010a140 2024-03-18T15:06:01.686 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:01.022+0000 7f4cce000700 1 -- 172.21.15.19:0/3747449241 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+175 (secure 0 0 0) 0x7f4cc4025460 con 0x7f4cd010a140 2024-03-18T15:06:01.686 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:01.024+0000 7f4cd6c00700 1 -- 172.21.15.19:0/3747449241 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4cd010a140 msgr2=0x7f4cd019dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:06:01.687 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:01.024+0000 7f4cd6c00700 1 --2- 172.21.15.19:0/3747449241 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4cd010a140 0x7f4cd019dd50 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f4cc4011850 tx=0x7f4cc4011c10 comp rx=0 tx=0).stop 2024-03-18T15:06:01.687 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:01.024+0000 7f4cd6c00700 1 -- 172.21.15.19:0/3747449241 shutdown_connections 2024-03-18T15:06:01.687 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:01.024+0000 7f4cd6c00700 1 --2- 172.21.15.19:0/3747449241 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4cd010a140 0x7f4cd019dd50 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:06:01.688 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:01.024+0000 7f4cd6c00700 1 -- 172.21.15.19:0/3747449241 >> 172.21.15.19:0/3747449241 conn(0x7f4cd007d0f0 msgr2=0x7f4cd0107e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:06:01.688 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:01.024+0000 7f4cd6c00700 1 -- 172.21.15.19:0/3747449241 shutdown_connections 2024-03-18T15:06:01.688 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:01.024+0000 7f4cd6c00700 1 -- 172.21.15.19:0/3747449241 wait complete. 2024-03-18T15:06:01.689 INFO:teuthology.orchestra.run.smithi019.stdout:Restarting the monitor... 2024-03-18T15:06:01.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:01 smithi019 systemd[1]: Stopping Ceph mon.a for d4dc2ba6-e538-11ee-95c9-87774f69a715... 2024-03-18T15:06:05.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:05 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mon-a[31150]: 2024-03-18T15:06:05.295+0000 7f1717c00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T15:06:05.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:05 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mon-a[31150]: 2024-03-18T15:06:05.295+0000 7f1717c00700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-18T15:06:05.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:05 smithi019 podman[31398]: 2024-03-18 15:06:05.541460172 +0000 UTC m=+3.483621424 container died 21b5d75b760ddac43a9d6d13e0e795f441956c7ac703d23639202aa890af5ba9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mon-a, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True) 2024-03-18T15:06:10.173 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:09 smithi019 podman[31398]: 2024-03-18 15:06:09.921917418 +0000 UTC m=+7.864078666 container cleanup 21b5d75b760ddac43a9d6d13e0e795f441956c7ac703d23639202aa890af5ba9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-18T15:06:10.173 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:09 smithi019 bash[31398]: ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mon-a 2024-03-18T15:06:10.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:10 smithi019 podman[31411]: 2024-03-18 15:06:10.172758852 +0000 UTC m=+4.615308648 container remove 21b5d75b760ddac43a9d6d13e0e795f441956c7ac703d23639202aa890af5ba9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mon-a, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-18T15:06:10.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:10 smithi019 systemd[1]: ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-18T15:06:10.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:10 smithi019 systemd[1]: Stopped Ceph mon.a for d4dc2ba6-e538-11ee-95c9-87774f69a715. 2024-03-18T15:06:10.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:10 smithi019 systemd[1]: ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@mon.a.service: Consumed 2.026s CPU time. 2024-03-18T15:06:10.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:10 smithi019 systemd[1]: Starting Ceph mon.a for d4dc2ba6-e538-11ee-95c9-87774f69a715... 2024-03-18T15:06:11.378 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 podman[31504]: 2024-03-18 15:06:10.987917037 +0000 UTC m=+0.039617205 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-18T15:06:11.378 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 podman[31504]: 2024-03-18 15:06:11.081704287 +0000 UTC m=+0.133404441 container create 21fe4b08ac45f6975b99831acf5267fbbe1580ae6db5a8c7857a30b26b5f9d7b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-18T15:06:11.378 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 podman[31504]: 2024-03-18 15:06:11.318588829 +0000 UTC m=+0.370289066 container init 21fe4b08ac45f6975b99831acf5267fbbe1580ae6db5a8c7857a30b26b5f9d7b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mon-a, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-18T15:06:11.378 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 podman[31504]: 2024-03-18 15:06:11.328843496 +0000 UTC m=+0.380543695 container start 21fe4b08ac45f6975b99831acf5267fbbe1580ae6db5a8c7857a30b26b5f9d7b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mon-a, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T15:06:11.446 INFO:teuthology.orchestra.run.smithi019.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-18T15:06:11.718 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: set uid:gid to 167:167 (ceph:ceph) 2024-03-18T15:06:11.718 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev), process ceph-mon, pid 2 2024-03-18T15:06:11.718 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: pidfile_write: ignore empty --pid-file 2024-03-18T15:06:11.718 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: load: jerasure load: lrc 2024-03-18T15:06:11.718 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: RocksDB version: 7.9.2 2024-03-18T15:06:11.718 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Git sha 0 2024-03-18T15:06:11.719 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Compile date 2024-03-15 00:33:34 2024-03-18T15:06:11.719 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: DB SUMMARY 2024-03-18T15:06:11.719 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: DB Session ID: NXRNCAJF7TPY0OF6YWHQ 2024-03-18T15:06:11.719 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: CURRENT file: CURRENT 2024-03-18T15:06:11.719 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: IDENTITY file: IDENTITY 2024-03-18T15:06:11.719 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-18T15:06:11.719 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-18T15:06:11.719 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 105370 ; 2024-03-18T15:06:11.719 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.error_if_exists: 0 2024-03-18T15:06:11.719 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.create_if_missing: 0 2024-03-18T15:06:11.719 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.paranoid_checks: 1 2024-03-18T15:06:11.719 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-18T15:06:11.719 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-18T15:06:11.719 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-18T15:06:11.720 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.env: 0x559e32d03c60 2024-03-18T15:06:11.720 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.fs: PosixFileSystem 2024-03-18T15:06:11.720 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.info_log: 0x559e338e3b00 2024-03-18T15:06:11.720 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.max_file_opening_threads: 16 2024-03-18T15:06:11.720 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.statistics: (nil) 2024-03-18T15:06:11.720 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.use_fsync: 0 2024-03-18T15:06:11.720 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.max_log_file_size: 0 2024-03-18T15:06:11.720 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-18T15:06:11.720 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-18T15:06:11.720 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.keep_log_file_num: 1000 2024-03-18T15:06:11.720 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.recycle_log_file_num: 0 2024-03-18T15:06:11.720 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.allow_fallocate: 1 2024-03-18T15:06:11.720 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.allow_mmap_reads: 0 2024-03-18T15:06:11.720 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.allow_mmap_writes: 0 2024-03-18T15:06:11.721 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.use_direct_reads: 0 2024-03-18T15:06:11.721 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-18T15:06:11.721 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.create_missing_column_families: 0 2024-03-18T15:06:11.721 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.db_log_dir: 2024-03-18T15:06:11.721 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.wal_dir: 2024-03-18T15:06:11.721 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-18T15:06:11.721 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-18T15:06:11.721 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-18T15:06:11.722 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-18T15:06:11.722 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-18T15:06:11.722 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-18T15:06:11.722 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.advise_random_on_open: 1 2024-03-18T15:06:11.722 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.db_write_buffer_size: 0 2024-03-18T15:06:11.722 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.write_buffer_manager: 0x559e338b61e0 2024-03-18T15:06:11.722 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-18T15:06:11.722 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-18T15:06:11.722 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-18T15:06:11.722 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.rate_limiter: (nil) 2024-03-18T15:06:11.722 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-18T15:06:11.722 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.wal_recovery_mode: 2 2024-03-18T15:06:11.723 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.enable_thread_tracking: 0 2024-03-18T15:06:11.723 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.enable_pipelined_write: 0 2024-03-18T15:06:11.723 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.unordered_write: 0 2024-03-18T15:06:11.723 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-18T15:06:11.723 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-18T15:06:11.723 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-18T15:06:11.723 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-18T15:06:11.723 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.row_cache: None 2024-03-18T15:06:11.723 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.wal_filter: None 2024-03-18T15:06:11.723 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-18T15:06:11.723 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.allow_ingest_behind: 0 2024-03-18T15:06:11.723 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.two_write_queues: 0 2024-03-18T15:06:11.723 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.manual_wal_flush: 0 2024-03-18T15:06:11.723 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.wal_compression: 0 2024-03-18T15:06:11.724 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.atomic_flush: 0 2024-03-18T15:06:11.724 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-18T15:06:11.724 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-18T15:06:11.724 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-18T15:06:11.724 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.log_readahead_size: 0 2024-03-18T15:06:11.724 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-18T15:06:11.724 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.best_efforts_recovery: 0 2024-03-18T15:06:11.724 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-18T15:06:11.724 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-18T15:06:11.724 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.allow_data_in_errors: 0 2024-03-18T15:06:11.724 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.db_host_id: __hostname__ 2024-03-18T15:06:11.724 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-18T15:06:11.725 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.max_background_jobs: 2 2024-03-18T15:06:11.725 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.max_background_compactions: -1 2024-03-18T15:06:11.725 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.max_subcompactions: 1 2024-03-18T15:06:11.725 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-18T15:06:11.725 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-18T15:06:11.725 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-18T15:06:11.726 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.max_total_wal_size: 0 2024-03-18T15:06:11.726 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-18T15:06:11.726 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-18T15:06:11.726 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-18T15:06:11.726 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-18T15:06:11.726 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.max_open_files: -1 2024-03-18T15:06:11.726 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.bytes_per_sync: 0 2024-03-18T15:06:11.726 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-18T15:06:11.726 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-18T15:06:11.726 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.compaction_readahead_size: 0 2024-03-18T15:06:11.726 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.max_background_flushes: -1 2024-03-18T15:06:11.726 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Compression algorithms supported: 2024-03-18T15:06:11.727 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-18T15:06:11.727 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: kZSTD supported: 0 2024-03-18T15:06:11.727 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: kXpressCompression supported: 0 2024-03-18T15:06:11.727 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: kLZ4HCCompression supported: 1 2024-03-18T15:06:11.727 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: kZlibCompression supported: 1 2024-03-18T15:06:11.727 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: kSnappyCompression supported: 1 2024-03-18T15:06:11.727 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: kLZ4Compression supported: 1 2024-03-18T15:06:11.727 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: kBZip2Compression supported: 0 2024-03-18T15:06:11.727 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-18T15:06:11.727 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-18T15:06:11.727 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-18T15:06:11.727 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-18T15:06:11.727 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-18T15:06:11.728 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.merge_operator: 2024-03-18T15:06:11.728 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.compaction_filter: None 2024-03-18T15:06:11.728 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.compaction_filter_factory: None 2024-03-18T15:06:11.728 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.sst_partitioner_factory: None 2024-03-18T15:06:11.728 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-18T15:06:11.728 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-18T15:06:11.728 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x559e338e3c20) 2024-03-18T15:06:11.728 INFO:journalctl@ceph.mon.a.smithi019.stdout: cache_index_and_filter_blocks: 1 2024-03-18T15:06:11.728 INFO:journalctl@ceph.mon.a.smithi019.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-18T15:06:11.728 INFO:journalctl@ceph.mon.a.smithi019.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-18T15:06:11.728 INFO:journalctl@ceph.mon.a.smithi019.stdout: pin_top_level_index_and_filter: 1 2024-03-18T15:06:11.728 INFO:journalctl@ceph.mon.a.smithi019.stdout: index_type: 0 2024-03-18T15:06:11.728 INFO:journalctl@ceph.mon.a.smithi019.stdout: data_block_index_type: 0 2024-03-18T15:06:11.728 INFO:journalctl@ceph.mon.a.smithi019.stdout: index_shortening: 1 2024-03-18T15:06:11.728 INFO:journalctl@ceph.mon.a.smithi019.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-18T15:06:11.729 INFO:journalctl@ceph.mon.a.smithi019.stdout: checksum: 4 2024-03-18T15:06:11.729 INFO:journalctl@ceph.mon.a.smithi019.stdout: no_block_cache: 0 2024-03-18T15:06:11.729 INFO:journalctl@ceph.mon.a.smithi019.stdout: block_cache: 0x559e338e9090 2024-03-18T15:06:11.729 INFO:journalctl@ceph.mon.a.smithi019.stdout: block_cache_name: BinnedLRUCache 2024-03-18T15:06:11.729 INFO:journalctl@ceph.mon.a.smithi019.stdout: block_cache_options: 2024-03-18T15:06:11.729 INFO:journalctl@ceph.mon.a.smithi019.stdout: capacity : 536870912 2024-03-18T15:06:11.729 INFO:journalctl@ceph.mon.a.smithi019.stdout: num_shard_bits : 4 2024-03-18T15:06:11.729 INFO:journalctl@ceph.mon.a.smithi019.stdout: strict_capacity_limit : 0 2024-03-18T15:06:11.729 INFO:journalctl@ceph.mon.a.smithi019.stdout: high_pri_pool_ratio: 0.000 2024-03-18T15:06:11.729 INFO:journalctl@ceph.mon.a.smithi019.stdout: block_cache_compressed: (nil) 2024-03-18T15:06:11.729 INFO:journalctl@ceph.mon.a.smithi019.stdout: persistent_cache: (nil) 2024-03-18T15:06:11.729 INFO:journalctl@ceph.mon.a.smithi019.stdout: block_size: 4096 2024-03-18T15:06:11.729 INFO:journalctl@ceph.mon.a.smithi019.stdout: block_size_deviation: 10 2024-03-18T15:06:11.729 INFO:journalctl@ceph.mon.a.smithi019.stdout: block_restart_interval: 16 2024-03-18T15:06:11.729 INFO:journalctl@ceph.mon.a.smithi019.stdout: index_block_restart_interval: 1 2024-03-18T15:06:11.730 INFO:journalctl@ceph.mon.a.smithi019.stdout: metadata_block_size: 4096 2024-03-18T15:06:11.730 INFO:journalctl@ceph.mon.a.smithi019.stdout: partition_filters: 0 2024-03-18T15:06:11.730 INFO:journalctl@ceph.mon.a.smithi019.stdout: use_delta_encoding: 1 2024-03-18T15:06:11.730 INFO:journalctl@ceph.mon.a.smithi019.stdout: filter_policy: bloomfilter 2024-03-18T15:06:11.730 INFO:journalctl@ceph.mon.a.smithi019.stdout: whole_key_filtering: 1 2024-03-18T15:06:11.730 INFO:journalctl@ceph.mon.a.smithi019.stdout: verify_compression: 0 2024-03-18T15:06:11.730 INFO:journalctl@ceph.mon.a.smithi019.stdout: read_amp_bytes_per_bit: 0 2024-03-18T15:06:11.730 INFO:journalctl@ceph.mon.a.smithi019.stdout: format_version: 5 2024-03-18T15:06:11.730 INFO:journalctl@ceph.mon.a.smithi019.stdout: enable_index_compression: 1 2024-03-18T15:06:11.730 INFO:journalctl@ceph.mon.a.smithi019.stdout: block_align: 0 2024-03-18T15:06:11.730 INFO:journalctl@ceph.mon.a.smithi019.stdout: max_auto_readahead_size: 262144 2024-03-18T15:06:11.730 INFO:journalctl@ceph.mon.a.smithi019.stdout: prepopulate_block_cache: 0 2024-03-18T15:06:11.730 INFO:journalctl@ceph.mon.a.smithi019.stdout: initial_auto_readahead_size: 8192 2024-03-18T15:06:11.730 INFO:journalctl@ceph.mon.a.smithi019.stdout: num_file_reads_for_auto_readahead: 2 2024-03-18T15:06:11.731 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.write_buffer_size: 33554432 2024-03-18T15:06:11.731 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.max_write_buffer_number: 2 2024-03-18T15:06:11.731 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.compression: NoCompression 2024-03-18T15:06:11.731 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.bottommost_compression: Disabled 2024-03-18T15:06:11.731 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.prefix_extractor: nullptr 2024-03-18T15:06:11.731 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-18T15:06:11.731 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.num_levels: 7 2024-03-18T15:06:11.731 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-18T15:06:11.731 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-18T15:06:11.731 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-18T15:06:11.731 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-18T15:06:11.731 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-18T15:06:11.731 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-18T15:06:11.731 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-18T15:06:11.732 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-18T15:06:11.732 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-18T15:06:11.732 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-18T15:06:11.732 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-18T15:06:11.732 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-18T15:06:11.732 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-18T15:06:11.732 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.compression_opts.level: 32767 2024-03-18T15:06:11.732 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.compression_opts.strategy: 0 2024-03-18T15:06:11.732 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-18T15:06:11.732 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-18T15:06:11.732 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-18T15:06:11.732 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-18T15:06:11.732 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.compression_opts.enabled: false 2024-03-18T15:06:11.733 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-18T15:06:11.733 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-18T15:06:11.733 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-18T15:06:11.733 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-18T15:06:11.733 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.target_file_size_base: 67108864 2024-03-18T15:06:11.733 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-18T15:06:11.733 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-18T15:06:11.733 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-18T15:06:11.733 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-18T15:06:11.733 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-18T15:06:11.733 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-18T15:06:11.733 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-18T15:06:11.733 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-18T15:06:11.733 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-18T15:06:11.733 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-18T15:06:11.734 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-18T15:06:11.734 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-18T15:06:11.734 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-18T15:06:11.734 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-18T15:06:11.734 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.arena_block_size: 1048576 2024-03-18T15:06:11.734 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-18T15:06:11.734 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-18T15:06:11.734 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.disable_auto_compactions: 0 2024-03-18T15:06:11.734 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-18T15:06:11.734 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-18T15:06:11.734 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-18T15:06:11.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-18T15:06:11.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-18T15:06:11.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-18T15:06:11.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-18T15:06:11.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-18T15:06:11.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-18T15:06:11.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-18T15:06:11.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.table_properties_collectors: 2024-03-18T15:06:11.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.inplace_update_support: 0 2024-03-18T15:06:11.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-18T15:06:11.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-18T15:06:11.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-18T15:06:11.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-18T15:06:11.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.bloom_locality: 0 2024-03-18T15:06:11.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.max_successive_merges: 0 2024-03-18T15:06:11.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-18T15:06:11.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.paranoid_file_checks: 0 2024-03-18T15:06:11.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.force_consistency_checks: 1 2024-03-18T15:06:11.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.report_bg_io_stats: 0 2024-03-18T15:06:11.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.ttl: 2592000 2024-03-18T15:06:11.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-18T15:06:11.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-18T15:06:11.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-18T15:06:11.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.enable_blob_files: false 2024-03-18T15:06:11.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.min_blob_size: 0 2024-03-18T15:06:11.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.blob_file_size: 268435456 2024-03-18T15:06:11.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-18T15:06:11.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-18T15:06:11.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-18T15:06:11.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-18T15:06:11.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-18T15:06:11.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.blob_file_starting_level: 0 2024-03-18T15:06:11.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-18T15:06:11.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-18T15:06:11.738 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-18T15:06:11.738 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 486a935b-3da3-4559-8705-5186a83f1907 2024-03-18T15:06:11.738 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710774371388344, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-18T15:06:11.738 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-18T15:06:11.738 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710774371389901, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 100481, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 326, "table_properties": {"data_size": 98392, "index_size": 307, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 773, "raw_key_size": 14219, "raw_average_key_size": 49, "raw_value_size": 90656, "raw_average_value_size": 318, "num_data_blocks": 12, "num_entries": 285, "num_filter_entries": 285, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710774371, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "486a935b-3da3-4559-8705-5186a83f1907", "db_session_id": "NXRNCAJF7TPY0OF6YWHQ", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-18T15:06:11.738 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710774371390057, "job": 1, "event": "recovery_finished"} 2024-03-18T15:06:11.738 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-18T15:06:11.738 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-18T15:06:11.738 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x559e339dc000 2024-03-18T15:06:11.738 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: DB pointer 0x559e339c6000 2024-03-18T15:06:11.738 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-18T15:06:11.738 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-18T15:06:11.738 INFO:journalctl@ceph.mon.a.smithi019.stdout: ** DB Stats ** 2024-03-18T15:06:11.738 INFO:journalctl@ceph.mon.a.smithi019.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-18T15:06:11.739 INFO:journalctl@ceph.mon.a.smithi019.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-18T15:06:11.739 INFO:journalctl@ceph.mon.a.smithi019.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-18T15:06:11.739 INFO:journalctl@ceph.mon.a.smithi019.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-18T15:06:11.739 INFO:journalctl@ceph.mon.a.smithi019.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-18T15:06:11.739 INFO:journalctl@ceph.mon.a.smithi019.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-18T15:06:11.739 INFO:journalctl@ceph.mon.a.smithi019.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-18T15:06:11.739 INFO:journalctl@ceph.mon.a.smithi019.stdout: 2024-03-18T15:06:11.739 INFO:journalctl@ceph.mon.a.smithi019.stdout: ** Compaction Stats [default] ** 2024-03-18T15:06:11.739 INFO:journalctl@ceph.mon.a.smithi019.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-18T15:06:11.739 INFO:journalctl@ceph.mon.a.smithi019.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-18T15:06:11.739 INFO:journalctl@ceph.mon.a.smithi019.stdout: L0 2/0 99.96 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 100.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-18T15:06:11.739 INFO:journalctl@ceph.mon.a.smithi019.stdout: Sum 2/0 99.96 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 100.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-18T15:06:11.739 INFO:journalctl@ceph.mon.a.smithi019.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 100.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-18T15:06:11.739 INFO:journalctl@ceph.mon.a.smithi019.stdout: 2024-03-18T15:06:11.740 INFO:journalctl@ceph.mon.a.smithi019.stdout: ** Compaction Stats [default] ** 2024-03-18T15:06:11.740 INFO:journalctl@ceph.mon.a.smithi019.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-18T15:06:11.740 INFO:journalctl@ceph.mon.a.smithi019.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-18T15:06:11.740 INFO:journalctl@ceph.mon.a.smithi019.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 100.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-18T15:06:11.740 INFO:journalctl@ceph.mon.a.smithi019.stdout: 2024-03-18T15:06:11.740 INFO:journalctl@ceph.mon.a.smithi019.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-18T15:06:11.740 INFO:journalctl@ceph.mon.a.smithi019.stdout: 2024-03-18T15:06:11.740 INFO:journalctl@ceph.mon.a.smithi019.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-18T15:06:11.740 INFO:journalctl@ceph.mon.a.smithi019.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-18T15:06:11.740 INFO:journalctl@ceph.mon.a.smithi019.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-18T15:06:11.740 INFO:journalctl@ceph.mon.a.smithi019.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-18T15:06:11.740 INFO:journalctl@ceph.mon.a.smithi019.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-18T15:06:11.740 INFO:journalctl@ceph.mon.a.smithi019.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-18T15:06:11.740 INFO:journalctl@ceph.mon.a.smithi019.stdout: Cumulative compaction: 0.00 GB write, 14.59 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-18T15:06:11.740 INFO:journalctl@ceph.mon.a.smithi019.stdout: Interval compaction: 0.00 GB write, 14.59 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-18T15:06:11.741 INFO:journalctl@ceph.mon.a.smithi019.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-18T15:06:11.741 INFO:journalctl@ceph.mon.a.smithi019.stdout: Block cache BinnedLRUCache@0x559e338e9090#2 capacity: 512.00 MB usage: 1.52 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.2e-05 secs_since: 0 2024-03-18T15:06:11.741 INFO:journalctl@ceph.mon.a.smithi019.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,1.02 KB,0.000193715%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-03-18T15:06:11.741 INFO:journalctl@ceph.mon.a.smithi019.stdout: 2024-03-18T15:06:11.741 INFO:journalctl@ceph.mon.a.smithi019.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-18T15:06:11.741 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: starting mon.a rank 0 at public addrs [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] at bind addrs [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 2024-03-18T15:06:11.741 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: mon.a@-1(???) e1 preinit fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 2024-03-18T15:06:11.741 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: mon.a@-1(???).mds e1 new map 2024-03-18T15:06:11.741 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: mon.a@-1(???).mds e1 print_map 2024-03-18T15:06:11.741 INFO:journalctl@ceph.mon.a.smithi019.stdout: e1 2024-03-18T15:06:11.741 INFO:journalctl@ceph.mon.a.smithi019.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-18T15:06:11.741 INFO:journalctl@ceph.mon.a.smithi019.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-18T15:06:11.741 INFO:journalctl@ceph.mon.a.smithi019.stdout: legacy client fscid: -1 2024-03-18T15:06:11.741 INFO:journalctl@ceph.mon.a.smithi019.stdout: 2024-03-18T15:06:11.742 INFO:journalctl@ceph.mon.a.smithi019.stdout: No filesystems configured 2024-03-18T15:06:11.742 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-18T15:06:11.742 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-18T15:06:11.742 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-18T15:06:11.742 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-18T15:06:11.742 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-18T15:06:11.742 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 bash[31504]: 21fe4b08ac45f6975b99831acf5267fbbe1580ae6db5a8c7857a30b26b5f9d7b 2024-03-18T15:06:11.742 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 systemd[1]: Started Ceph mon.a for d4dc2ba6-e538-11ee-95c9-87774f69a715. 2024-03-18T15:06:11.742 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-18T15:06:11.742 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: monmap epoch 1 2024-03-18T15:06:11.742 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 2024-03-18T15:06:11.742 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: last_changed 2024-03-18T15:05:53.426642+0000 2024-03-18T15:06:11.742 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: created 2024-03-18T15:05:53.426642+0000 2024-03-18T15:06:11.742 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: min_mon_release 19 (squid) 2024-03-18T15:06:11.742 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: election_strategy: 1 2024-03-18T15:06:11.743 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: 0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-03-18T15:06:11.743 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: fsmap 2024-03-18T15:06:11.743 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: osdmap e1: 0 total, 0 up, 0 in 2024-03-18T15:06:11.743 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:11 smithi019 ceph-mon[31522]: mgrmap e1: no daemons active 2024-03-18T15:06:12.914 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:12.069+0000 7fc51a600700 1 Processor -- start 2024-03-18T15:06:12.915 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:12.070+0000 7fc51a600700 1 -- start start 2024-03-18T15:06:12.915 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:12.070+0000 7fc51a600700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc514106580 0x7fc5141069a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:06:12.915 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:12.070+0000 7fc51a600700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fc51407be40 con 0x7fc514106580 2024-03-18T15:06:12.915 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:12.071+0000 7fc513400700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc514106580 0x7fc5141069a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:06:12.915 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:12.071+0000 7fc513400700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc514106580 0x7fc5141069a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:39166/0 (socket says 172.21.15.19:39166) 2024-03-18T15:06:12.916 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:12.071+0000 7fc513400700 1 -- 172.21.15.19:0/3125172188 learned_addr learned my addr 172.21.15.19:0/3125172188 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:06:12.917 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:12.071+0000 7fc513400700 1 -- 172.21.15.19:0/3125172188 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc514106ee0 con 0x7fc514106580 2024-03-18T15:06:12.917 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:12.072+0000 7fc513400700 1 --2- 172.21.15.19:0/3125172188 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc514106580 0x7fc5141069a0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fc5040098a0 tx=0x7fc504009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a90c0a4186bd3606 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:06:12.917 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:12.072+0000 7fc512a00700 1 -- 172.21.15.19:0/3125172188 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc504004160 con 0x7fc514106580 2024-03-18T15:06:12.917 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:12.072+0000 7fc512a00700 1 -- 172.21.15.19:0/3125172188 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fc504004300 con 0x7fc514106580 2024-03-18T15:06:12.918 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:12.072+0000 7fc512a00700 1 -- 172.21.15.19:0/3125172188 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc504011b50 con 0x7fc514106580 2024-03-18T15:06:12.918 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:12.073+0000 7fc51a600700 1 -- 172.21.15.19:0/3125172188 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc514106580 msgr2=0x7fc5141069a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:06:12.918 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:12.073+0000 7fc51a600700 1 --2- 172.21.15.19:0/3125172188 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc514106580 0x7fc5141069a0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fc5040098a0 tx=0x7fc504009bb0 comp rx=0 tx=0).stop 2024-03-18T15:06:12.918 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:12.073+0000 7fc51a600700 1 -- 172.21.15.19:0/3125172188 shutdown_connections 2024-03-18T15:06:12.918 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:12.073+0000 7fc51a600700 1 --2- 172.21.15.19:0/3125172188 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc514106580 0x7fc5141069a0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:06:12.918 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:12.073+0000 7fc51a600700 1 -- 172.21.15.19:0/3125172188 >> 172.21.15.19:0/3125172188 conn(0x7fc514102140 msgr2=0x7fc5141045a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:06:12.918 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:12.073+0000 7fc51a600700 1 -- 172.21.15.19:0/3125172188 shutdown_connections 2024-03-18T15:06:12.918 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:12.073+0000 7fc51a600700 1 -- 172.21.15.19:0/3125172188 wait complete. 2024-03-18T15:06:12.918 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:12.074+0000 7fc51a600700 1 Processor -- start 2024-03-18T15:06:12.918 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:12.074+0000 7fc51a600700 1 -- start start 2024-03-18T15:06:12.918 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:12.075+0000 7fc51a600700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc514106580 0x7fc51419de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:06:12.921 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:12.075+0000 7fc51a600700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fc514107f70 con 0x7fc514106580 2024-03-18T15:06:12.921 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:12.075+0000 7fc513400700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc514106580 0x7fc51419de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:06:12.921 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:12.075+0000 7fc513400700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc514106580 0x7fc51419de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:39168/0 (socket says 172.21.15.19:39168) 2024-03-18T15:06:12.921 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:12.075+0000 7fc513400700 1 -- 172.21.15.19:0/1539828259 learned_addr learned my addr 172.21.15.19:0/1539828259 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:06:12.921 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:12.075+0000 7fc513400700 1 -- 172.21.15.19:0/1539828259 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc51419e3b0 con 0x7fc514106580 2024-03-18T15:06:12.921 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:12.076+0000 7fc513400700 1 --2- 172.21.15.19:0/1539828259 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc514106580 0x7fc51419de70 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fc504011670 tx=0x7fc504011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:06:12.921 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:12.076+0000 7fc510c00700 1 -- 172.21.15.19:0/1539828259 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc5040039c0 con 0x7fc514106580 2024-03-18T15:06:12.921 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:12.076+0000 7fc510c00700 1 -- 172.21.15.19:0/1539828259 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fc504011e50 con 0x7fc514106580 2024-03-18T15:06:12.921 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:12.076+0000 7fc510c00700 1 -- 172.21.15.19:0/1539828259 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc50401a5c0 con 0x7fc514106580 2024-03-18T15:06:12.921 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:12.076+0000 7fc51a600700 1 -- 172.21.15.19:0/1539828259 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc51419e640 con 0x7fc514106580 2024-03-18T15:06:12.921 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:12.076+0000 7fc51a600700 1 -- 172.21.15.19:0/1539828259 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5141a1330 con 0x7fc514106580 2024-03-18T15:06:12.922 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:12.076+0000 7fc510c00700 1 -- 172.21.15.19:0/1539828259 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fc50401ac50 con 0x7fc514106580 2024-03-18T15:06:12.922 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:12.077+0000 7fc510c00700 1 -- 172.21.15.19:0/1539828259 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fc5040193c0 con 0x7fc514106580 2024-03-18T15:06:12.922 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:12.077+0000 7fc51a600700 1 -- 172.21.15.19:0/1539828259 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc514063900 con 0x7fc514106580 2024-03-18T15:06:12.922 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:12.081+0000 7fc510c00700 1 -- 172.21.15.19:0/1539828259 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fc504003b60 con 0x7fc514106580 2024-03-18T15:06:12.922 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:12.160+0000 7fc51a600700 1 -- 172.21.15.19:0/1539828259 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7fc51402db70 con 0x7fc514106580 2024-03-18T15:06:12.922 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:12.164+0000 7fc510c00700 1 -- 172.21.15.19:0/1539828259 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7fc504024770 con 0x7fc514106580 2024-03-18T15:06:12.922 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:12.165+0000 7fc51a600700 1 -- 172.21.15.19:0/1539828259 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc514106580 msgr2=0x7fc51419de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:06:12.922 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:12.165+0000 7fc51a600700 1 --2- 172.21.15.19:0/1539828259 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc514106580 0x7fc51419de70 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fc504011670 tx=0x7fc504011b60 comp rx=0 tx=0).stop 2024-03-18T15:06:12.922 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:12.165+0000 7fc51a600700 1 -- 172.21.15.19:0/1539828259 shutdown_connections 2024-03-18T15:06:12.922 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:12.165+0000 7fc51a600700 1 --2- 172.21.15.19:0/1539828259 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc514106580 0x7fc51419de70 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:06:12.922 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:12.165+0000 7fc51a600700 1 -- 172.21.15.19:0/1539828259 >> 172.21.15.19:0/1539828259 conn(0x7fc514102140 msgr2=0x7fc514194ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:06:12.922 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:12.166+0000 7fc51a600700 1 -- 172.21.15.19:0/1539828259 shutdown_connections 2024-03-18T15:06:12.922 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:12.166+0000 7fc51a600700 1 -- 172.21.15.19:0/1539828259 wait complete. 2024-03-18T15:06:12.923 INFO:teuthology.orchestra.run.smithi019.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-18T15:06:12.948 INFO:teuthology.orchestra.run.smithi019.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-18T15:06:12.948 INFO:teuthology.orchestra.run.smithi019.stdout:Creating mgr... 2024-03-18T15:06:12.949 INFO:teuthology.orchestra.run.smithi019.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-18T15:06:12.951 INFO:teuthology.orchestra.run.smithi019.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-18T15:06:13.165 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:13 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/1539828259' entity='client.admin' 2024-03-18T15:06:13.397 INFO:teuthology.orchestra.run.smithi019.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@mgr.x 2024-03-18T15:06:13.397 INFO:teuthology.orchestra.run.smithi019.stdout:systemctl: stderr Failed to reset failed state of unit ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@mgr.x.service: Unit ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@mgr.x.service not loaded. 2024-03-18T15:06:13.659 INFO:teuthology.orchestra.run.smithi019.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715.target.wants/ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@mgr.x.service → /etc/systemd/system/ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@.service. 2024-03-18T15:06:13.965 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:06:13 smithi019 systemd[1]: Starting Ceph mgr.x for d4dc2ba6-e538-11ee-95c9-87774f69a715... 2024-03-18T15:06:14.466 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:06:14 smithi019 podman[31735]: 2024-03-18 15:06:13.962651603 +0000 UTC m=+0.038083066 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-18T15:06:14.466 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:06:14 smithi019 podman[31735]: 2024-03-18 15:06:14.05908438 +0000 UTC m=+0.134515869 container create 8574ee959649a93587a0fd2a8e544ed301f945e10af3b1244982598b1fa93a50 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True) 2024-03-18T15:06:15.112 INFO:teuthology.orchestra.run.smithi019.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-18T15:06:15.112 INFO:teuthology.orchestra.run.smithi019.stdout:systemctl: stdout disabled 2024-03-18T15:06:15.131 INFO:teuthology.orchestra.run.smithi019.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-18T15:06:15.131 INFO:teuthology.orchestra.run.smithi019.stdout:systemctl: stdout inactive 2024-03-18T15:06:15.131 INFO:teuthology.orchestra.run.smithi019.stdout:firewalld.service is not enabled 2024-03-18T15:06:15.131 INFO:teuthology.orchestra.run.smithi019.stdout:Not possible to enable service . firewalld.service is not available 2024-03-18T15:06:15.156 INFO:teuthology.orchestra.run.smithi019.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-18T15:06:15.156 INFO:teuthology.orchestra.run.smithi019.stdout:systemctl: stdout disabled 2024-03-18T15:06:15.176 INFO:teuthology.orchestra.run.smithi019.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-18T15:06:15.176 INFO:teuthology.orchestra.run.smithi019.stdout:systemctl: stdout inactive 2024-03-18T15:06:15.176 INFO:teuthology.orchestra.run.smithi019.stdout:firewalld.service is not enabled 2024-03-18T15:06:15.176 INFO:teuthology.orchestra.run.smithi019.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-18T15:06:15.177 INFO:teuthology.orchestra.run.smithi019.stdout:Waiting for mgr to start... 2024-03-18T15:06:15.177 INFO:teuthology.orchestra.run.smithi019.stdout:Waiting for mgr... 2024-03-18T15:06:15.215 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:06:14 smithi019 podman[31735]: 2024-03-18 15:06:14.838229293 +0000 UTC m=+0.913660745 container init 8574ee959649a93587a0fd2a8e544ed301f945e10af3b1244982598b1fa93a50 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-18T15:06:15.215 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:06:14 smithi019 podman[31735]: 2024-03-18 15:06:14.853873288 +0000 UTC m=+0.929304740 container start 8574ee959649a93587a0fd2a8e544ed301f945e10af3b1244982598b1fa93a50 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-18T15:06:15.215 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:06:15 smithi019 bash[31735]: 8574ee959649a93587a0fd2a8e544ed301f945e10af3b1244982598b1fa93a50 2024-03-18T15:06:15.215 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:06:15 smithi019 systemd[1]: Started Ceph mgr.x for d4dc2ba6-e538-11ee-95c9-87774f69a715. 2024-03-18T15:06:15.912 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:06:15 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:06:15.677+0000 7f7ef3371200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T15:06:15.913 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:06:15 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:06:15.821+0000 7f7ef3371200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T15:06:16.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:15 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/2819499189' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-18T15:06:16.657 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout 2024-03-18T15:06:16.657 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout { 2024-03-18T15:06:16.657 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "fsid": "d4dc2ba6-e538-11ee-95c9-87774f69a715", 2024-03-18T15:06:16.657 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "health": { 2024-03-18T15:06:16.658 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-18T15:06:16.660 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-18T15:06:16.660 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-18T15:06:16.660 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout }, 2024-03-18T15:06:16.660 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-18T15:06:16.660 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-18T15:06:16.660 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout 0 2024-03-18T15:06:16.660 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout ], 2024-03-18T15:06:16.660 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-18T15:06:16.660 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "a" 2024-03-18T15:06:16.660 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout ], 2024-03-18T15:06:16.660 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "quorum_age": 4, 2024-03-18T15:06:16.660 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-18T15:06:16.661 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T15:06:16.661 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-18T15:06:16.661 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-18T15:06:16.661 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout }, 2024-03-18T15:06:16.661 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-18T15:06:16.661 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T15:06:16.661 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-18T15:06:16.661 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-18T15:06:16.661 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-18T15:06:16.669 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-18T15:06:16.669 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-18T15:06:16.669 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-18T15:06:16.669 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout }, 2024-03-18T15:06:16.670 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-18T15:06:16.670 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-18T15:06:16.670 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-18T15:06:16.670 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-18T15:06:16.670 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-18T15:06:16.670 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-18T15:06:16.670 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-18T15:06:16.670 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-18T15:06:16.670 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-18T15:06:16.670 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout }, 2024-03-18T15:06:16.670 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-18T15:06:16.670 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T15:06:16.670 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-18T15:06:16.671 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-18T15:06:16.671 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout }, 2024-03-18T15:06:16.671 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-18T15:06:16.671 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-18T15:06:16.671 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-18T15:06:16.671 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-18T15:06:16.671 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-18T15:06:16.671 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-18T15:06:16.671 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "restful" 2024-03-18T15:06:16.671 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout ], 2024-03-18T15:06:16.671 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-18T15:06:16.671 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout }, 2024-03-18T15:06:16.671 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-18T15:06:16.671 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T15:06:16.671 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "modified": "2024-03-18T15:05:57.539135+0000", 2024-03-18T15:06:16.672 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-18T15:06:16.672 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout }, 2024-03-18T15:06:16.672 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-18T15:06:16.672 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout } 2024-03-18T15:06:16.672 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:15.794+0000 7f43bb800700 1 Processor -- start 2024-03-18T15:06:16.672 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:15.795+0000 7f43bb800700 1 -- start start 2024-03-18T15:06:16.672 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:15.795+0000 7f43bb800700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f43b4108820 0x7f43b410ac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:06:16.672 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:15.795+0000 7f43bb800700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f43b407bdc0 con 0x7f43b4108820 2024-03-18T15:06:16.672 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:15.796+0000 7f43b8e00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f43b4108820 0x7f43b410ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:06:16.672 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:15.796+0000 7f43b8e00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f43b4108820 0x7f43b410ac10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:39188/0 (socket says 172.21.15.19:39188) 2024-03-18T15:06:16.672 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:15.796+0000 7f43b8e00700 1 -- 172.21.15.19:0/3237153859 learned_addr learned my addr 172.21.15.19:0/3237153859 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:06:16.672 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:15.797+0000 7f43b8e00700 1 -- 172.21.15.19:0/3237153859 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43b410b150 con 0x7f43b4108820 2024-03-18T15:06:16.672 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:15.797+0000 7f43b8e00700 1 --2- 172.21.15.19:0/3237153859 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f43b4108820 0x7f43b410ac10 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f43a80098a0 tx=0x7f43a8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1933912aac6c0c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:06:16.673 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:15.797+0000 7f43b3400700 1 -- 172.21.15.19:0/3237153859 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f43a8004160 con 0x7f43b4108820 2024-03-18T15:06:16.673 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:15.797+0000 7f43b3400700 1 -- 172.21.15.19:0/3237153859 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f43a8004300 con 0x7f43b4108820 2024-03-18T15:06:16.673 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:15.797+0000 7f43b3400700 1 -- 172.21.15.19:0/3237153859 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f43a8011b50 con 0x7f43b4108820 2024-03-18T15:06:16.673 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:15.798+0000 7f43bb800700 1 -- 172.21.15.19:0/3237153859 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f43b4108820 msgr2=0x7f43b410ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:06:16.673 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:15.798+0000 7f43bb800700 1 --2- 172.21.15.19:0/3237153859 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f43b4108820 0x7f43b410ac10 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f43a80098a0 tx=0x7f43a8009bb0 comp rx=0 tx=0).stop 2024-03-18T15:06:16.673 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:15.799+0000 7f43bb800700 1 -- 172.21.15.19:0/3237153859 shutdown_connections 2024-03-18T15:06:16.673 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:15.799+0000 7f43bb800700 1 --2- 172.21.15.19:0/3237153859 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f43b4108820 0x7f43b410ac10 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:06:16.673 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:15.799+0000 7f43bb800700 1 -- 172.21.15.19:0/3237153859 >> 172.21.15.19:0/3237153859 conn(0x7f43b4102130 msgr2=0x7f43b41045b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:06:16.673 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:15.799+0000 7f43bb800700 1 -- 172.21.15.19:0/3237153859 shutdown_connections 2024-03-18T15:06:16.673 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:15.799+0000 7f43bb800700 1 -- 172.21.15.19:0/3237153859 wait complete. 2024-03-18T15:06:16.673 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:15.800+0000 7f43bb800700 1 Processor -- start 2024-03-18T15:06:16.673 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:15.800+0000 7f43bb800700 1 -- start start 2024-03-18T15:06:16.673 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:15.801+0000 7f43bb800700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f43b4108820 0x7f43b419df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:06:16.673 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:15.801+0000 7f43bb800700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f43b410c0b0 con 0x7f43b4108820 2024-03-18T15:06:16.674 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:15.801+0000 7f43b8e00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f43b4108820 0x7f43b419df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:06:16.674 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:15.801+0000 7f43b8e00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f43b4108820 0x7f43b419df20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:39196/0 (socket says 172.21.15.19:39196) 2024-03-18T15:06:16.674 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:15.801+0000 7f43b8e00700 1 -- 172.21.15.19:0/2819499189 learned_addr learned my addr 172.21.15.19:0/2819499189 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:06:16.674 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:15.801+0000 7f43b8e00700 1 -- 172.21.15.19:0/2819499189 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43b419e460 con 0x7f43b4108820 2024-03-18T15:06:16.674 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:15.802+0000 7f43b8e00700 1 --2- 172.21.15.19:0/2819499189 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f43b4108820 0x7f43b419df20 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f43a8011b60 tx=0x7f43a8011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:06:16.674 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:15.802+0000 7f43b1600700 1 -- 172.21.15.19:0/2819499189 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f43a8003970 con 0x7f43b4108820 2024-03-18T15:06:16.674 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:15.802+0000 7f43b1600700 1 -- 172.21.15.19:0/2819499189 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f43a8003b10 con 0x7f43b4108820 2024-03-18T15:06:16.674 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:15.802+0000 7f43bb800700 1 -- 172.21.15.19:0/2819499189 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43b419e6f0 con 0x7f43b4108820 2024-03-18T15:06:16.674 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:15.802+0000 7f43b1600700 1 -- 172.21.15.19:0/2819499189 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f43a8024610 con 0x7f43b4108820 2024-03-18T15:06:16.674 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:15.802+0000 7f43bb800700 1 -- 172.21.15.19:0/2819499189 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43b41a13e0 con 0x7f43b4108820 2024-03-18T15:06:16.674 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:15.803+0000 7f43b1600700 1 -- 172.21.15.19:0/2819499189 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f43a8024c50 con 0x7f43b4108820 2024-03-18T15:06:16.674 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:15.804+0000 7f43b1600700 1 -- 172.21.15.19:0/2819499189 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f43a8011e00 con 0x7f43b4108820 2024-03-18T15:06:16.674 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:15.804+0000 7f43bb800700 1 -- 172.21.15.19:0/2819499189 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43b40693d0 con 0x7f43b4108820 2024-03-18T15:06:16.674 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:15.807+0000 7f43b1600700 1 -- 172.21.15.19:0/2819499189 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f43a801a770 con 0x7f43b4108820 2024-03-18T15:06:16.675 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:15.910+0000 7f43bb800700 1 -- 172.21.15.19:0/2819499189 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f43b41a15b0 con 0x7f43b4108820 2024-03-18T15:06:16.675 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:15.911+0000 7f43b1600700 1 -- 172.21.15.19:0/2819499189 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f43a801a9f0 con 0x7f43b4108820 2024-03-18T15:06:16.675 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:15.913+0000 7f43bb800700 1 -- 172.21.15.19:0/2819499189 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f43b4108820 msgr2=0x7f43b419df20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:06:16.675 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:15.913+0000 7f43bb800700 1 --2- 172.21.15.19:0/2819499189 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f43b4108820 0x7f43b419df20 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f43a8011b60 tx=0x7f43a8011c40 comp rx=0 tx=0).stop 2024-03-18T15:06:16.675 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:15.913+0000 7f43bb800700 1 -- 172.21.15.19:0/2819499189 shutdown_connections 2024-03-18T15:06:16.675 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:15.913+0000 7f43bb800700 1 --2- 172.21.15.19:0/2819499189 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f43b4108820 0x7f43b419df20 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:06:16.675 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:15.913+0000 7f43bb800700 1 -- 172.21.15.19:0/2819499189 >> 172.21.15.19:0/2819499189 conn(0x7f43b4102130 msgr2=0x7f43b4102cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:06:16.675 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:15.913+0000 7f43bb800700 1 -- 172.21.15.19:0/2819499189 shutdown_connections 2024-03-18T15:06:16.675 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:15.913+0000 7f43bb800700 1 -- 172.21.15.19:0/2819499189 wait complete. 2024-03-18T15:06:16.675 INFO:teuthology.orchestra.run.smithi019.stdout:mgr not available, waiting (1/15)... 2024-03-18T15:06:16.965 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:06:16 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:06:16.695+0000 7f7ef3371200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T15:06:16.965 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:06:16 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:06:16.830+0000 7f7ef3371200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T15:06:17.465 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:06:17 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:06:17.025+0000 7f7ef3371200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T15:06:17.465 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:06:17 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:06:17.159+0000 7f7ef3371200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T15:06:17.465 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:06:17 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:06:17.272+0000 7f7ef3371200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T15:06:17.878 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:06:17 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:06:17.615+0000 7f7ef3371200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T15:06:17.878 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:06:17 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:06:17.761+0000 7f7ef3371200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T15:06:17.879 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:06:17 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:06:17.876+0000 7f7ef3371200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T15:06:18.215 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:06:18 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:06:18.005+0000 7f7ef3371200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T15:06:18.715 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:06:18 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:06:18.227+0000 7f7ef3371200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T15:06:19.605 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:19 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/2203502300' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-18T15:06:19.855 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:06:19 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:06:19.602+0000 7f7ef3371200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T15:06:19.855 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:06:19 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:06:19.779+0000 7f7ef3371200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T15:06:20.039 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout 2024-03-18T15:06:20.051 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout { 2024-03-18T15:06:20.051 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "fsid": "d4dc2ba6-e538-11ee-95c9-87774f69a715", 2024-03-18T15:06:20.051 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "health": { 2024-03-18T15:06:20.051 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-18T15:06:20.051 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-18T15:06:20.051 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-18T15:06:20.051 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout }, 2024-03-18T15:06:20.052 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-18T15:06:20.052 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-18T15:06:20.052 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout 0 2024-03-18T15:06:20.052 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout ], 2024-03-18T15:06:20.052 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-18T15:06:20.052 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "a" 2024-03-18T15:06:20.052 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout ], 2024-03-18T15:06:20.052 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "quorum_age": 7, 2024-03-18T15:06:20.052 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-18T15:06:20.052 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T15:06:20.052 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-18T15:06:20.052 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-18T15:06:20.052 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout }, 2024-03-18T15:06:20.053 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-18T15:06:20.053 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T15:06:20.053 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-18T15:06:20.053 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-18T15:06:20.053 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-18T15:06:20.053 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-18T15:06:20.053 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-18T15:06:20.053 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-18T15:06:20.053 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout }, 2024-03-18T15:06:20.053 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-18T15:06:20.053 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-18T15:06:20.053 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-18T15:06:20.053 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-18T15:06:20.054 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-18T15:06:20.054 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-18T15:06:20.054 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-18T15:06:20.054 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-18T15:06:20.054 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-18T15:06:20.054 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout }, 2024-03-18T15:06:20.054 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-18T15:06:20.054 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T15:06:20.054 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-18T15:06:20.054 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-18T15:06:20.054 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout }, 2024-03-18T15:06:20.054 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-18T15:06:20.054 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-18T15:06:20.055 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-18T15:06:20.055 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-18T15:06:20.055 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-18T15:06:20.055 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-18T15:06:20.055 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "restful" 2024-03-18T15:06:20.055 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout ], 2024-03-18T15:06:20.055 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-18T15:06:20.055 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout }, 2024-03-18T15:06:20.055 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-18T15:06:20.055 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T15:06:20.055 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "modified": "2024-03-18T15:05:57.539135+0000", 2024-03-18T15:06:20.055 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-18T15:06:20.055 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout }, 2024-03-18T15:06:20.056 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-18T15:06:20.056 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout } 2024-03-18T15:06:20.056 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:19.236+0000 7f49b4200700 1 Processor -- start 2024-03-18T15:06:20.056 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:19.237+0000 7f49b4200700 1 -- start start 2024-03-18T15:06:20.056 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:19.237+0000 7f49b4200700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f49ac109f70 0x7f49ac10a390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:06:20.056 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:19.237+0000 7f49b4200700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f49ac10a960 con 0x7f49ac109f70 2024-03-18T15:06:20.056 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:19.238+0000 7f49b1800700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f49ac109f70 0x7f49ac10a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:06:20.056 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:19.238+0000 7f49b1800700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f49ac109f70 0x7f49ac10a390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:39210/0 (socket says 172.21.15.19:39210) 2024-03-18T15:06:20.056 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:19.238+0000 7f49b1800700 1 -- 172.21.15.19:0/2885910128 learned_addr learned my addr 172.21.15.19:0/2885910128 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:06:20.056 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:19.238+0000 7f49b1800700 1 -- 172.21.15.19:0/2885910128 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49ac10b170 con 0x7f49ac109f70 2024-03-18T15:06:20.056 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:19.238+0000 7f49b1800700 1 --2- 172.21.15.19:0/2885910128 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f49ac109f70 0x7f49ac10a390 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f49a00098a0 tx=0x7f49a0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=647e428d288f1315 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:06:20.056 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:19.239+0000 7f49abe00700 1 -- 172.21.15.19:0/2885910128 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f49a0004160 con 0x7f49ac109f70 2024-03-18T15:06:20.056 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:19.239+0000 7f49abe00700 1 -- 172.21.15.19:0/2885910128 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f49a0004300 con 0x7f49ac109f70 2024-03-18T15:06:20.056 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:19.239+0000 7f49abe00700 1 -- 172.21.15.19:0/2885910128 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f49a0011b50 con 0x7f49ac109f70 2024-03-18T15:06:20.057 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:19.240+0000 7f49b4200700 1 -- 172.21.15.19:0/2885910128 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f49ac109f70 msgr2=0x7f49ac10a390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:06:20.057 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:19.240+0000 7f49b4200700 1 --2- 172.21.15.19:0/2885910128 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f49ac109f70 0x7f49ac10a390 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f49a00098a0 tx=0x7f49a0009bb0 comp rx=0 tx=0).stop 2024-03-18T15:06:20.057 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:19.240+0000 7f49b4200700 1 -- 172.21.15.19:0/2885910128 shutdown_connections 2024-03-18T15:06:20.057 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:19.240+0000 7f49b4200700 1 --2- 172.21.15.19:0/2885910128 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f49ac109f70 0x7f49ac10a390 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:06:20.057 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:19.240+0000 7f49b4200700 1 -- 172.21.15.19:0/2885910128 >> 172.21.15.19:0/2885910128 conn(0x7f49ac07d360 msgr2=0x7f49ac07d770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:06:20.057 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:19.240+0000 7f49b4200700 1 -- 172.21.15.19:0/2885910128 shutdown_connections 2024-03-18T15:06:20.057 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:19.240+0000 7f49b4200700 1 -- 172.21.15.19:0/2885910128 wait complete. 2024-03-18T15:06:20.057 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:19.241+0000 7f49b4200700 1 Processor -- start 2024-03-18T15:06:20.057 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:19.241+0000 7f49b4200700 1 -- start start 2024-03-18T15:06:20.057 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:19.241+0000 7f49b4200700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f49ac109f70 0x7f49ac19df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:06:20.057 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:19.241+0000 7f49b4200700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f49ac10e520 con 0x7f49ac109f70 2024-03-18T15:06:20.057 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:19.242+0000 7f49b1800700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f49ac109f70 0x7f49ac19df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:06:20.057 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:19.242+0000 7f49b1800700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f49ac109f70 0x7f49ac19df20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:39212/0 (socket says 172.21.15.19:39212) 2024-03-18T15:06:20.057 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:19.242+0000 7f49b1800700 1 -- 172.21.15.19:0/2203502300 learned_addr learned my addr 172.21.15.19:0/2203502300 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:06:20.057 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:19.242+0000 7f49b1800700 1 -- 172.21.15.19:0/2203502300 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49ac19e460 con 0x7f49ac109f70 2024-03-18T15:06:20.058 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:19.242+0000 7f49b1800700 1 --2- 172.21.15.19:0/2203502300 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f49ac109f70 0x7f49ac19df20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f49a0009fb0 tx=0x7f49a0011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:06:20.058 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:19.243+0000 7f49aa000700 1 -- 172.21.15.19:0/2203502300 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f49a00038b0 con 0x7f49ac109f70 2024-03-18T15:06:20.058 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:19.243+0000 7f49aa000700 1 -- 172.21.15.19:0/2203502300 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f49a0011e50 con 0x7f49ac109f70 2024-03-18T15:06:20.058 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:19.243+0000 7f49aa000700 1 -- 172.21.15.19:0/2203502300 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f49a0019550 con 0x7f49ac109f70 2024-03-18T15:06:20.058 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:19.243+0000 7f49b4200700 1 -- 172.21.15.19:0/2203502300 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49ac19e6f0 con 0x7f49ac109f70 2024-03-18T15:06:20.058 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:19.243+0000 7f49b4200700 1 -- 172.21.15.19:0/2203502300 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49ac1a13e0 con 0x7f49ac109f70 2024-03-18T15:06:20.058 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:19.243+0000 7f49aa000700 1 -- 172.21.15.19:0/2203502300 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f49a0019b90 con 0x7f49ac109f70 2024-03-18T15:06:20.058 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:19.243+0000 7f49aa000700 1 -- 172.21.15.19:0/2203502300 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f49a0024730 con 0x7f49ac109f70 2024-03-18T15:06:20.058 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:19.244+0000 7f49b4200700 1 -- 172.21.15.19:0/2203502300 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49ac0693d0 con 0x7f49ac109f70 2024-03-18T15:06:20.058 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:19.247+0000 7f49aa000700 1 -- 172.21.15.19:0/2203502300 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f49a002dda0 con 0x7f49ac109f70 2024-03-18T15:06:20.058 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:19.327+0000 7f49b4200700 1 -- 172.21.15.19:0/2203502300 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f49ac1a15b0 con 0x7f49ac109f70 2024-03-18T15:06:20.058 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:19.328+0000 7f49aa000700 1 -- 172.21.15.19:0/2203502300 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f49a0032030 con 0x7f49ac109f70 2024-03-18T15:06:20.058 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:19.329+0000 7f49b4200700 1 -- 172.21.15.19:0/2203502300 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f49ac109f70 msgr2=0x7f49ac19df20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:06:20.058 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:19.329+0000 7f49b4200700 1 --2- 172.21.15.19:0/2203502300 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f49ac109f70 0x7f49ac19df20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f49a0009fb0 tx=0x7f49a0011bf0 comp rx=0 tx=0).stop 2024-03-18T15:06:20.059 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:19.329+0000 7f49b4200700 1 -- 172.21.15.19:0/2203502300 shutdown_connections 2024-03-18T15:06:20.059 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:19.329+0000 7f49b4200700 1 --2- 172.21.15.19:0/2203502300 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f49ac109f70 0x7f49ac19df20 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:06:20.059 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:19.329+0000 7f49b4200700 1 -- 172.21.15.19:0/2203502300 >> 172.21.15.19:0/2203502300 conn(0x7f49ac07d360 msgr2=0x7f49ac107560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:06:20.059 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:19.330+0000 7f49b4200700 1 -- 172.21.15.19:0/2203502300 shutdown_connections 2024-03-18T15:06:20.059 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:19.330+0000 7f49b4200700 1 -- 172.21.15.19:0/2203502300 wait complete. 2024-03-18T15:06:20.059 INFO:teuthology.orchestra.run.smithi019.stdout:mgr not available, waiting (2/15)... 2024-03-18T15:06:20.621 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:06:20 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:06:20.222+0000 7f7ef3371200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T15:06:20.622 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:06:20 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:06:20.350+0000 7f7ef3371200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T15:06:20.870 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:06:20 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:06:20.618+0000 7f7ef3371200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T15:06:21.215 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:06:20 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:06:20.869+0000 7f7ef3371200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T15:06:21.576 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:06:21 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:06:21.253+0000 7f7ef3371200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T15:06:21.577 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:06:21 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:06:21.575+0000 7f7ef3371200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T15:06:22.210 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:06:21 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:06:21.890+0000 7f7ef3371200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T15:06:22.988 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:22 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/2716888169' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-18T15:06:23.400 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:06:22 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:06:22.986+0000 7f7ef3371200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T15:06:23.421 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout 2024-03-18T15:06:23.421 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout { 2024-03-18T15:06:23.421 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "fsid": "d4dc2ba6-e538-11ee-95c9-87774f69a715", 2024-03-18T15:06:23.422 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "health": { 2024-03-18T15:06:23.422 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-18T15:06:23.422 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-18T15:06:23.422 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-18T15:06:23.422 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout }, 2024-03-18T15:06:23.422 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-18T15:06:23.422 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-18T15:06:23.422 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout 0 2024-03-18T15:06:23.422 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout ], 2024-03-18T15:06:23.422 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-18T15:06:23.422 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "a" 2024-03-18T15:06:23.422 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout ], 2024-03-18T15:06:23.422 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "quorum_age": 11, 2024-03-18T15:06:23.423 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-18T15:06:23.423 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T15:06:23.423 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-18T15:06:23.423 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-18T15:06:23.423 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout }, 2024-03-18T15:06:23.423 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-18T15:06:23.423 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T15:06:23.423 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-18T15:06:23.423 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-18T15:06:23.423 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-18T15:06:23.423 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-18T15:06:23.423 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-18T15:06:23.423 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-18T15:06:23.424 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout }, 2024-03-18T15:06:23.424 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-18T15:06:23.424 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-18T15:06:23.424 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-18T15:06:23.424 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-18T15:06:23.424 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-18T15:06:23.424 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-18T15:06:23.424 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-18T15:06:23.424 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-18T15:06:23.424 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-18T15:06:23.424 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout }, 2024-03-18T15:06:23.424 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-18T15:06:23.424 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T15:06:23.424 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-18T15:06:23.425 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-18T15:06:23.425 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout }, 2024-03-18T15:06:23.425 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-18T15:06:23.425 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-18T15:06:23.425 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-18T15:06:23.425 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-18T15:06:23.425 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-18T15:06:23.425 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-18T15:06:23.425 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "restful" 2024-03-18T15:06:23.425 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout ], 2024-03-18T15:06:23.425 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-18T15:06:23.425 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout }, 2024-03-18T15:06:23.425 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-18T15:06:23.425 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T15:06:23.425 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "modified": "2024-03-18T15:05:57.539135+0000", 2024-03-18T15:06:23.426 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-18T15:06:23.426 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout }, 2024-03-18T15:06:23.426 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-18T15:06:23.426 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout } 2024-03-18T15:06:23.426 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:22.607+0000 7f1b4f200700 1 Processor -- start 2024-03-18T15:06:23.426 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:22.608+0000 7f1b4f200700 1 -- start start 2024-03-18T15:06:23.426 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:22.608+0000 7f1b4f200700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1b50107b00 0x7f1b50107f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:06:23.426 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:22.608+0000 7f1b4f200700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f1b501084f0 con 0x7f1b50107b00 2024-03-18T15:06:23.426 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:22.608+0000 7f1b4de00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1b50107b00 0x7f1b50107f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:06:23.426 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:22.608+0000 7f1b4de00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1b50107b00 0x7f1b50107f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:38086/0 (socket says 172.21.15.19:38086) 2024-03-18T15:06:23.426 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:22.608+0000 7f1b4de00700 1 -- 172.21.15.19:0/3635477300 learned_addr learned my addr 172.21.15.19:0/3635477300 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:06:23.426 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:22.609+0000 7f1b4de00700 1 -- 172.21.15.19:0/3635477300 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b50108d00 con 0x7f1b50107b00 2024-03-18T15:06:23.426 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:22.609+0000 7f1b4de00700 1 --2- 172.21.15.19:0/3635477300 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1b50107b00 0x7f1b50107f20 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f1b400098a0 tx=0x7f1b40009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf82ac933a182ec5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:06:23.427 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:22.609+0000 7f1b4ca00700 1 -- 172.21.15.19:0/3635477300 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1b40004160 con 0x7f1b50107b00 2024-03-18T15:06:23.427 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:22.610+0000 7f1b4ca00700 1 -- 172.21.15.19:0/3635477300 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f1b40004300 con 0x7f1b50107b00 2024-03-18T15:06:23.427 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:22.610+0000 7f1b4ca00700 1 -- 172.21.15.19:0/3635477300 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1b40011b50 con 0x7f1b50107b00 2024-03-18T15:06:23.427 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:22.610+0000 7f1b4f200700 1 -- 172.21.15.19:0/3635477300 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1b50107b00 msgr2=0x7f1b50107f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:06:23.427 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:22.610+0000 7f1b4f200700 1 --2- 172.21.15.19:0/3635477300 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1b50107b00 0x7f1b50107f20 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f1b400098a0 tx=0x7f1b40009bb0 comp rx=0 tx=0).stop 2024-03-18T15:06:23.427 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:22.610+0000 7f1b4f200700 1 -- 172.21.15.19:0/3635477300 shutdown_connections 2024-03-18T15:06:23.427 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:22.610+0000 7f1b4f200700 1 --2- 172.21.15.19:0/3635477300 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1b50107b00 0x7f1b50107f20 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:06:23.427 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:22.610+0000 7f1b4f200700 1 -- 172.21.15.19:0/3635477300 >> 172.21.15.19:0/3635477300 conn(0x7f1b50103080 msgr2=0x7f1b501054e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:06:23.427 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:22.610+0000 7f1b4f200700 1 -- 172.21.15.19:0/3635477300 shutdown_connections 2024-03-18T15:06:23.427 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:22.611+0000 7f1b4f200700 1 -- 172.21.15.19:0/3635477300 wait complete. 2024-03-18T15:06:23.427 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:22.612+0000 7f1b4f200700 1 Processor -- start 2024-03-18T15:06:23.427 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:22.612+0000 7f1b4f200700 1 -- start start 2024-03-18T15:06:23.427 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:22.613+0000 7f1b4f200700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1b50107b00 0x7f1b5007db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:06:23.427 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:22.613+0000 7f1b4f200700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f1b5010c0b0 con 0x7f1b50107b00 2024-03-18T15:06:23.428 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:22.613+0000 7f1b4de00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1b50107b00 0x7f1b5007db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:06:23.428 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:22.613+0000 7f1b4de00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1b50107b00 0x7f1b5007db60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:38094/0 (socket says 172.21.15.19:38094) 2024-03-18T15:06:23.428 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:22.613+0000 7f1b4de00700 1 -- 172.21.15.19:0/2716888169 learned_addr learned my addr 172.21.15.19:0/2716888169 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:06:23.428 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:22.613+0000 7f1b4de00700 1 -- 172.21.15.19:0/2716888169 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b5007e0a0 con 0x7f1b50107b00 2024-03-18T15:06:23.428 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:22.614+0000 7f1b4de00700 1 --2- 172.21.15.19:0/2716888169 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1b50107b00 0x7f1b5007db60 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f1b40011850 tx=0x7f1b40011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:06:23.428 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:22.614+0000 7f1b46a00700 1 -- 172.21.15.19:0/2716888169 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1b40003af0 con 0x7f1b50107b00 2024-03-18T15:06:23.428 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:22.614+0000 7f1b46a00700 1 -- 172.21.15.19:0/2716888169 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f1b40020070 con 0x7f1b50107b00 2024-03-18T15:06:23.428 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:22.614+0000 7f1b46a00700 1 -- 172.21.15.19:0/2716888169 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1b400195a0 con 0x7f1b50107b00 2024-03-18T15:06:23.428 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:22.614+0000 7f1b4f200700 1 -- 172.21.15.19:0/2716888169 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b5007c290 con 0x7f1b50107b00 2024-03-18T15:06:23.428 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:22.614+0000 7f1b4f200700 1 -- 172.21.15.19:0/2716888169 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b5007c7d0 con 0x7f1b50107b00 2024-03-18T15:06:23.428 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:22.614+0000 7f1b46a00700 1 -- 172.21.15.19:0/2716888169 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f1b40019be0 con 0x7f1b50107b00 2024-03-18T15:06:23.428 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:22.614+0000 7f1b46a00700 1 -- 172.21.15.19:0/2716888169 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f1b40011e00 con 0x7f1b50107b00 2024-03-18T15:06:23.428 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:22.615+0000 7f1b4f200700 1 -- 172.21.15.19:0/2716888169 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b500693d0 con 0x7f1b50107b00 2024-03-18T15:06:23.429 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:22.619+0000 7f1b46a00700 1 -- 172.21.15.19:0/2716888169 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f1b40003c90 con 0x7f1b50107b00 2024-03-18T15:06:23.429 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:22.699+0000 7f1b4f200700 1 -- 172.21.15.19:0/2716888169 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f1b5007cb10 con 0x7f1b50107b00 2024-03-18T15:06:23.429 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:22.699+0000 7f1b46a00700 1 -- 172.21.15.19:0/2716888169 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1240 (secure 0 0 0) 0x7f1b40025460 con 0x7f1b50107b00 2024-03-18T15:06:23.429 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:22.701+0000 7f1b4f200700 1 -- 172.21.15.19:0/2716888169 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1b50107b00 msgr2=0x7f1b5007db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:06:23.429 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:22.701+0000 7f1b4f200700 1 --2- 172.21.15.19:0/2716888169 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1b50107b00 0x7f1b5007db60 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f1b40011850 tx=0x7f1b40011c10 comp rx=0 tx=0).stop 2024-03-18T15:06:23.429 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:22.701+0000 7f1b4f200700 1 -- 172.21.15.19:0/2716888169 shutdown_connections 2024-03-18T15:06:23.429 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:22.701+0000 7f1b4f200700 1 --2- 172.21.15.19:0/2716888169 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1b50107b00 0x7f1b5007db60 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:06:23.429 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:22.701+0000 7f1b4f200700 1 -- 172.21.15.19:0/2716888169 >> 172.21.15.19:0/2716888169 conn(0x7f1b50103080 msgr2=0x7f1b50103c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:06:23.429 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:22.701+0000 7f1b4f200700 1 -- 172.21.15.19:0/2716888169 shutdown_connections 2024-03-18T15:06:23.429 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:22.701+0000 7f1b4f200700 1 -- 172.21.15.19:0/2716888169 wait complete. 2024-03-18T15:06:23.429 INFO:teuthology.orchestra.run.smithi019.stdout:mgr not available, waiting (3/15)... 2024-03-18T15:06:23.965 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:06:23 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:06:23.709+0000 7f7ef3371200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T15:06:23.966 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:06:23 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:06:23.844+0000 7f7ef3371200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T15:06:24.465 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:06:24 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:06:24.136+0000 7f7ef3371200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T15:06:24.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:24 smithi019 ceph-mon[31522]: Activating manager daemon x 2024-03-18T15:06:24.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:24 smithi019 ceph-mon[31522]: mgrmap e2: x(active, starting, since 0.0130516s) 2024-03-18T15:06:24.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:24 smithi019 ceph-mon[31522]: from='mgr.14100 172.21.15.19:0/4203075054' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-18T15:06:24.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:24 smithi019 ceph-mon[31522]: from='mgr.14100 172.21.15.19:0/4203075054' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-18T15:06:24.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:24 smithi019 ceph-mon[31522]: from='mgr.14100 172.21.15.19:0/4203075054' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-18T15:06:24.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:24 smithi019 ceph-mon[31522]: from='mgr.14100 172.21.15.19:0/4203075054' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T15:06:24.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:24 smithi019 ceph-mon[31522]: from='mgr.14100 172.21.15.19:0/4203075054' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-18T15:06:24.467 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:24 smithi019 ceph-mon[31522]: Manager daemon x is now available 2024-03-18T15:06:24.467 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:24 smithi019 ceph-mon[31522]: from='mgr.14100 172.21.15.19:0/4203075054' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-18T15:06:24.467 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:24 smithi019 ceph-mon[31522]: from='mgr.14100 172.21.15.19:0/4203075054' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-18T15:06:24.467 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:24 smithi019 ceph-mon[31522]: from='mgr.14100 172.21.15.19:0/4203075054' entity='mgr.x' 2024-03-18T15:06:25.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:25 smithi019 ceph-mon[31522]: from='mgr.14100 172.21.15.19:0/4203075054' entity='mgr.x' 2024-03-18T15:06:25.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:25 smithi019 ceph-mon[31522]: from='mgr.14100 172.21.15.19:0/4203075054' entity='mgr.x' 2024-03-18T15:06:25.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:25 smithi019 ceph-mon[31522]: mgrmap e3: x(active, since 1.02083s) 2024-03-18T15:06:27.045 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout 2024-03-18T15:06:27.045 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout { 2024-03-18T15:06:27.045 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "fsid": "d4dc2ba6-e538-11ee-95c9-87774f69a715", 2024-03-18T15:06:27.045 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "health": { 2024-03-18T15:06:27.045 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-18T15:06:27.045 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-18T15:06:27.046 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-18T15:06:27.046 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout }, 2024-03-18T15:06:27.046 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-18T15:06:27.046 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-18T15:06:27.046 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout 0 2024-03-18T15:06:27.046 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout ], 2024-03-18T15:06:27.046 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-18T15:06:27.046 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "a" 2024-03-18T15:06:27.046 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout ], 2024-03-18T15:06:27.046 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "quorum_age": 14, 2024-03-18T15:06:27.046 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-18T15:06:27.046 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T15:06:27.046 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-18T15:06:27.047 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-18T15:06:27.047 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout }, 2024-03-18T15:06:27.047 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-18T15:06:27.047 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T15:06:27.047 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-18T15:06:27.047 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-18T15:06:27.047 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-18T15:06:27.056 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-18T15:06:27.056 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-18T15:06:27.056 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-18T15:06:27.057 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout }, 2024-03-18T15:06:27.057 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-18T15:06:27.057 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-18T15:06:27.057 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-18T15:06:27.057 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-18T15:06:27.057 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-18T15:06:27.057 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-18T15:06:27.057 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-18T15:06:27.057 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-18T15:06:27.057 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-18T15:06:27.057 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout }, 2024-03-18T15:06:27.057 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-18T15:06:27.058 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T15:06:27.058 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-18T15:06:27.058 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-18T15:06:27.058 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout }, 2024-03-18T15:06:27.058 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-18T15:06:27.058 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-18T15:06:27.058 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-18T15:06:27.058 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-18T15:06:27.058 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-18T15:06:27.058 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-18T15:06:27.058 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "restful" 2024-03-18T15:06:27.058 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout ], 2024-03-18T15:06:27.059 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-18T15:06:27.059 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout }, 2024-03-18T15:06:27.059 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-18T15:06:27.059 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T15:06:27.059 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "modified": "2024-03-18T15:05:57.539135+0000", 2024-03-18T15:06:27.059 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-18T15:06:27.059 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout }, 2024-03-18T15:06:27.059 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-18T15:06:27.059 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout } 2024-03-18T15:06:27.059 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:25.995+0000 7ff38a400700 1 Processor -- start 2024-03-18T15:06:27.059 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:25.995+0000 7ff38a400700 1 -- start start 2024-03-18T15:06:27.059 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:25.996+0000 7ff38a400700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff3841087a0 0x7ff38410ab90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:06:27.059 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:25.996+0000 7ff38a400700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7ff38407be40 con 0x7ff3841087a0 2024-03-18T15:06:27.060 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:25.996+0000 7ff383400700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff3841087a0 0x7ff38410ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:06:27.060 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:25.996+0000 7ff383400700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff3841087a0 0x7ff38410ab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:38174/0 (socket says 172.21.15.19:38174) 2024-03-18T15:06:27.060 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:25.996+0000 7ff383400700 1 -- 172.21.15.19:0/1345390983 learned_addr learned my addr 172.21.15.19:0/1345390983 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:06:27.060 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:25.997+0000 7ff383400700 1 -- 172.21.15.19:0/1345390983 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff38410b0d0 con 0x7ff3841087a0 2024-03-18T15:06:27.060 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:25.997+0000 7ff383400700 1 --2- 172.21.15.19:0/1345390983 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff3841087a0 0x7ff38410ab90 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7ff3740098a0 tx=0x7ff374009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b3da12f694d4a6b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:06:27.060 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:25.998+0000 7ff382000700 1 -- 172.21.15.19:0/1345390983 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff374004160 con 0x7ff3841087a0 2024-03-18T15:06:27.060 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:25.998+0000 7ff382000700 1 -- 172.21.15.19:0/1345390983 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7ff374004300 con 0x7ff3841087a0 2024-03-18T15:06:27.060 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:25.998+0000 7ff382000700 1 -- 172.21.15.19:0/1345390983 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff374011b50 con 0x7ff3841087a0 2024-03-18T15:06:27.060 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:25.998+0000 7ff38a400700 1 -- 172.21.15.19:0/1345390983 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff3841087a0 msgr2=0x7ff38410ab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:06:27.060 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:25.998+0000 7ff38a400700 1 --2- 172.21.15.19:0/1345390983 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff3841087a0 0x7ff38410ab90 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7ff3740098a0 tx=0x7ff374009bb0 comp rx=0 tx=0).stop 2024-03-18T15:06:27.060 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:25.998+0000 7ff38a400700 1 -- 172.21.15.19:0/1345390983 shutdown_connections 2024-03-18T15:06:27.060 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:25.998+0000 7ff38a400700 1 --2- 172.21.15.19:0/1345390983 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff3841087a0 0x7ff38410ab90 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:06:27.061 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:25.998+0000 7ff38a400700 1 -- 172.21.15.19:0/1345390983 >> 172.21.15.19:0/1345390983 conn(0x7ff384102130 msgr2=0x7ff3841045b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:06:27.061 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:25.999+0000 7ff38a400700 1 -- 172.21.15.19:0/1345390983 shutdown_connections 2024-03-18T15:06:27.061 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:25.999+0000 7ff38a400700 1 -- 172.21.15.19:0/1345390983 wait complete. 2024-03-18T15:06:27.061 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:26.000+0000 7ff38a400700 1 Processor -- start 2024-03-18T15:06:27.061 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:26.000+0000 7ff38a400700 1 -- start start 2024-03-18T15:06:27.061 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:26.000+0000 7ff38a400700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff3841087a0 0x7ff38419df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:06:27.061 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:26.000+0000 7ff38a400700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7ff38410c0a0 con 0x7ff3841087a0 2024-03-18T15:06:27.061 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:26.000+0000 7ff383400700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff3841087a0 0x7ff38419df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:06:27.061 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:26.000+0000 7ff383400700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff3841087a0 0x7ff38419df60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:38190/0 (socket says 172.21.15.19:38190) 2024-03-18T15:06:27.061 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:26.000+0000 7ff383400700 1 -- 172.21.15.19:0/1453410129 learned_addr learned my addr 172.21.15.19:0/1453410129 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:06:27.061 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:26.001+0000 7ff383400700 1 -- 172.21.15.19:0/1453410129 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff38419e4a0 con 0x7ff3841087a0 2024-03-18T15:06:27.061 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:26.001+0000 7ff383400700 1 --2- 172.21.15.19:0/1453410129 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff3841087a0 0x7ff38419df60 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ff374009fb0 tx=0x7ff374011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:06:27.061 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:26.001+0000 7ff37be00700 1 -- 172.21.15.19:0/1453410129 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff3740038b0 con 0x7ff3841087a0 2024-03-18T15:06:27.061 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:26.001+0000 7ff37be00700 1 -- 172.21.15.19:0/1453410129 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7ff374011e50 con 0x7ff3841087a0 2024-03-18T15:06:27.062 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:26.001+0000 7ff37be00700 1 -- 172.21.15.19:0/1453410129 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff374019420 con 0x7ff3841087a0 2024-03-18T15:06:27.062 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:26.001+0000 7ff38a400700 1 -- 172.21.15.19:0/1453410129 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff38419e730 con 0x7ff3841087a0 2024-03-18T15:06:27.062 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:26.002+0000 7ff38a400700 1 -- 172.21.15.19:0/1453410129 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3841a1420 con 0x7ff3841087a0 2024-03-18T15:06:27.062 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:26.003+0000 7ff37be00700 1 -- 172.21.15.19:0/1453410129 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 3) ==== 45989+0+0 (secure 0 0 0) 0x7ff374003a50 con 0x7ff3841087a0 2024-03-18T15:06:27.062 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:26.003+0000 7ff38a400700 1 -- 172.21.15.19:0/1453410129 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3840693d0 con 0x7ff3841087a0 2024-03-18T15:06:27.062 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:26.003+0000 7ff37be00700 1 --2- 172.21.15.19:0/1453410129 >> [v2:172.21.15.19:6800/3264337803,v1:172.21.15.19:6801/3264337803] conn(0x7ff368038dc0 0x7ff36803b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:06:27.062 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:26.003+0000 7ff37be00700 1 -- 172.21.15.19:0/1453410129 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7ff37404ce40 con 0x7ff3841087a0 2024-03-18T15:06:27.062 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:26.004+0000 7ff382a00700 1 --2- 172.21.15.19:0/1453410129 >> [v2:172.21.15.19:6800/3264337803,v1:172.21.15.19:6801/3264337803] conn(0x7ff368038dc0 0x7ff36803b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:06:27.062 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:26.004+0000 7ff382a00700 1 --2- 172.21.15.19:0/1453410129 >> [v2:172.21.15.19:6800/3264337803,v1:172.21.15.19:6801/3264337803] conn(0x7ff368038dc0 0x7ff36803b280 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7ff370006fd0 tx=0x7ff370006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:06:27.062 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:26.010+0000 7ff37be00700 1 -- 172.21.15.19:0/1453410129 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff374023390 con 0x7ff3841087a0 2024-03-18T15:06:27.062 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:26.209+0000 7ff37be00700 1 -- 172.21.15.19:0/1453410129 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mgrmap(e 4) ==== 46095+0+0 (secure 0 0 0) 0x7ff374019660 con 0x7ff3841087a0 2024-03-18T15:06:27.062 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:26.311+0000 7ff38a400700 1 -- 172.21.15.19:0/1453410129 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7ff3841a15f0 con 0x7ff3841087a0 2024-03-18T15:06:27.063 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:26.312+0000 7ff37be00700 1 -- 172.21.15.19:0/1453410129 <== mon.0 v2:172.21.15.19:3300/0 8 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7ff3841a15f0 con 0x7ff3841087a0 2024-03-18T15:06:27.063 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:26.315+0000 7ff38a400700 1 -- 172.21.15.19:0/1453410129 >> [v2:172.21.15.19:6800/3264337803,v1:172.21.15.19:6801/3264337803] conn(0x7ff368038dc0 msgr2=0x7ff36803b280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:06:27.063 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:26.315+0000 7ff38a400700 1 --2- 172.21.15.19:0/1453410129 >> [v2:172.21.15.19:6800/3264337803,v1:172.21.15.19:6801/3264337803] conn(0x7ff368038dc0 0x7ff36803b280 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7ff370006fd0 tx=0x7ff370006e40 comp rx=0 tx=0).stop 2024-03-18T15:06:27.063 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:26.315+0000 7ff38a400700 1 -- 172.21.15.19:0/1453410129 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff3841087a0 msgr2=0x7ff38419df60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:06:27.063 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:26.315+0000 7ff38a400700 1 --2- 172.21.15.19:0/1453410129 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff3841087a0 0x7ff38419df60 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ff374009fb0 tx=0x7ff374011bf0 comp rx=0 tx=0).stop 2024-03-18T15:06:27.063 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:26.315+0000 7ff38a400700 1 -- 172.21.15.19:0/1453410129 shutdown_connections 2024-03-18T15:06:27.063 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:26.315+0000 7ff38a400700 1 --2- 172.21.15.19:0/1453410129 >> [v2:172.21.15.19:6800/3264337803,v1:172.21.15.19:6801/3264337803] conn(0x7ff368038dc0 0x7ff36803b280 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:06:27.063 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:26.315+0000 7ff38a400700 1 --2- 172.21.15.19:0/1453410129 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff3841087a0 0x7ff38419df60 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:06:27.063 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:26.315+0000 7ff38a400700 1 -- 172.21.15.19:0/1453410129 >> 172.21.15.19:0/1453410129 conn(0x7ff384102130 msgr2=0x7ff384102cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:06:27.063 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:26.315+0000 7ff38a400700 1 -- 172.21.15.19:0/1453410129 shutdown_connections 2024-03-18T15:06:27.063 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:26.315+0000 7ff38a400700 1 -- 172.21.15.19:0/1453410129 wait complete. 2024-03-18T15:06:27.063 INFO:teuthology.orchestra.run.smithi019.stdout:mgr is available 2024-03-18T15:06:27.378 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:27 smithi019 ceph-mon[31522]: mgrmap e4: x(active, since 2s) 2024-03-18T15:06:27.378 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:27 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/1453410129' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-18T15:06:28.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:28 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/2843780134' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-18T15:06:28.551 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout 2024-03-18T15:06:28.552 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout [global] 2024-03-18T15:06:28.552 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout fsid = d4dc2ba6-e538-11ee-95c9-87774f69a715 2024-03-18T15:06:28.552 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-18T15:06:28.552 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-18T15:06:28.553 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-18T15:06:28.554 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-18T15:06:28.554 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout 2024-03-18T15:06:28.554 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-18T15:06:28.554 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-18T15:06:28.554 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout 2024-03-18T15:06:28.554 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout [osd] 2024-03-18T15:06:28.554 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-18T15:06:28.554 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-18T15:06:28.554 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-18T15:06:28.554 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-18T15:06:28.554 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-18T15:06:28.557 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:27.667+0000 7f78ade00700 1 Processor -- start 2024-03-18T15:06:28.558 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:27.668+0000 7f78ade00700 1 -- start start 2024-03-18T15:06:28.558 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:27.668+0000 7f78ade00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f78a8107cd0 0x7f78a81080f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:06:28.558 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:27.668+0000 7f78ade00700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f78a81086c0 con 0x7f78a8107cd0 2024-03-18T15:06:28.558 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:27.669+0000 7f78a6a00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f78a8107cd0 0x7f78a81080f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:06:28.558 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:27.669+0000 7f78a6a00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f78a8107cd0 0x7f78a81080f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:38192/0 (socket says 172.21.15.19:38192) 2024-03-18T15:06:28.558 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:27.669+0000 7f78a6a00700 1 -- 172.21.15.19:0/625166511 learned_addr learned my addr 172.21.15.19:0/625166511 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:06:28.558 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:27.669+0000 7f78a6a00700 1 -- 172.21.15.19:0/625166511 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78a8108e80 con 0x7f78a8107cd0 2024-03-18T15:06:28.558 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:27.670+0000 7f78a6a00700 1 --2- 172.21.15.19:0/625166511 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f78a8107cd0 0x7f78a81080f0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f78980098a0 tx=0x7f7898009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a392c3b862967029 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:06:28.558 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:27.670+0000 7f78a5600700 1 -- 172.21.15.19:0/625166511 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7898004160 con 0x7f78a8107cd0 2024-03-18T15:06:28.558 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:27.670+0000 7f78a5600700 1 -- 172.21.15.19:0/625166511 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f7898004300 con 0x7f78a8107cd0 2024-03-18T15:06:28.558 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:27.670+0000 7f78a5600700 1 -- 172.21.15.19:0/625166511 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7898011b50 con 0x7f78a8107cd0 2024-03-18T15:06:28.558 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:27.671+0000 7f78ade00700 1 -- 172.21.15.19:0/625166511 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f78a8107cd0 msgr2=0x7f78a81080f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:06:28.559 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:27.671+0000 7f78ade00700 1 --2- 172.21.15.19:0/625166511 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f78a8107cd0 0x7f78a81080f0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f78980098a0 tx=0x7f7898009bb0 comp rx=0 tx=0).stop 2024-03-18T15:06:28.559 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:27.671+0000 7f78ade00700 1 -- 172.21.15.19:0/625166511 shutdown_connections 2024-03-18T15:06:28.559 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:27.671+0000 7f78ade00700 1 --2- 172.21.15.19:0/625166511 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f78a8107cd0 0x7f78a81080f0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:06:28.559 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:27.671+0000 7f78ade00700 1 -- 172.21.15.19:0/625166511 >> 172.21.15.19:0/625166511 conn(0x7f78a8103230 msgr2=0x7f78a81056b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:06:28.559 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:27.671+0000 7f78ade00700 1 -- 172.21.15.19:0/625166511 shutdown_connections 2024-03-18T15:06:28.559 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:27.671+0000 7f78ade00700 1 -- 172.21.15.19:0/625166511 wait complete. 2024-03-18T15:06:28.559 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:27.672+0000 7f78ade00700 1 Processor -- start 2024-03-18T15:06:28.559 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:27.672+0000 7f78ade00700 1 -- start start 2024-03-18T15:06:28.559 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:27.673+0000 7f78ade00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f78a8107cd0 0x7f78a819ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:06:28.559 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:27.673+0000 7f78ade00700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f78a807c250 con 0x7f78a8107cd0 2024-03-18T15:06:28.559 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:27.673+0000 7f78a6a00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f78a8107cd0 0x7f78a819ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:06:28.559 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:27.673+0000 7f78a6a00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f78a8107cd0 0x7f78a819ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:38196/0 (socket says 172.21.15.19:38196) 2024-03-18T15:06:28.559 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:27.673+0000 7f78a6a00700 1 -- 172.21.15.19:0/2843780134 learned_addr learned my addr 172.21.15.19:0/2843780134 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:06:28.559 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:27.674+0000 7f78a6a00700 1 -- 172.21.15.19:0/2843780134 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78a819e410 con 0x7f78a8107cd0 2024-03-18T15:06:28.560 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:27.674+0000 7f78a6a00700 1 --2- 172.21.15.19:0/2843780134 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f78a8107cd0 0x7f78a819ded0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f7898011850 tx=0x7f7898011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:06:28.562 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:27.674+0000 7f78aca00700 1 -- 172.21.15.19:0/2843780134 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7898003af0 con 0x7f78a8107cd0 2024-03-18T15:06:28.562 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:27.674+0000 7f78aca00700 1 -- 172.21.15.19:0/2843780134 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f7898020070 con 0x7f78a8107cd0 2024-03-18T15:06:28.562 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:27.674+0000 7f78aca00700 1 -- 172.21.15.19:0/2843780134 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f78980195a0 con 0x7f78a8107cd0 2024-03-18T15:06:28.562 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:27.675+0000 7f78ade00700 1 -- 172.21.15.19:0/2843780134 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78a819e6a0 con 0x7f78a8107cd0 2024-03-18T15:06:28.563 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:27.675+0000 7f78ade00700 1 -- 172.21.15.19:0/2843780134 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78a81a1390 con 0x7f78a8107cd0 2024-03-18T15:06:28.563 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:27.676+0000 7f78ade00700 1 -- 172.21.15.19:0/2843780134 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78a80693d0 con 0x7f78a8107cd0 2024-03-18T15:06:28.563 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:27.677+0000 7f78aca00700 1 -- 172.21.15.19:0/2843780134 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 4) ==== 46095+0+0 (secure 0 0 0) 0x7f789801a430 con 0x7f78a8107cd0 2024-03-18T15:06:28.563 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:27.677+0000 7f78aca00700 1 --2- 172.21.15.19:0/2843780134 >> [v2:172.21.15.19:6800/3264337803,v1:172.21.15.19:6801/3264337803] conn(0x7f788c038f20 0x7f788c03b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:06:28.563 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:27.677+0000 7f78aca00700 1 -- 172.21.15.19:0/2843780134 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f7898011e00 con 0x7f78a8107cd0 2024-03-18T15:06:28.563 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:27.678+0000 7f78a6000700 1 --2- 172.21.15.19:0/2843780134 >> [v2:172.21.15.19:6800/3264337803,v1:172.21.15.19:6801/3264337803] conn(0x7f788c038f20 0x7f788c03b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:06:28.563 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:27.678+0000 7f78a6000700 1 --2- 172.21.15.19:0/2843780134 >> [v2:172.21.15.19:6800/3264337803,v1:172.21.15.19:6801/3264337803] conn(0x7f788c038f20 0x7f788c03b3e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f7894006fd0 tx=0x7f7894006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:06:28.563 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:27.689+0000 7f78aca00700 1 -- 172.21.15.19:0/2843780134 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7898021270 con 0x7f78a8107cd0 2024-03-18T15:06:28.563 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:27.918+0000 7f78ade00700 1 -- 172.21.15.19:0/2843780134 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f78a802de40 con 0x7f78a8107cd0 2024-03-18T15:06:28.563 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:27.920+0000 7f78aca00700 1 -- 172.21.15.19:0/2843780134 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+373 (secure 0 0 0) 0x7f789801a720 con 0x7f78a8107cd0 2024-03-18T15:06:28.563 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:27.923+0000 7f78ade00700 1 -- 172.21.15.19:0/2843780134 >> [v2:172.21.15.19:6800/3264337803,v1:172.21.15.19:6801/3264337803] conn(0x7f788c038f20 msgr2=0x7f788c03b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:06:28.563 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:27.923+0000 7f78ade00700 1 --2- 172.21.15.19:0/2843780134 >> [v2:172.21.15.19:6800/3264337803,v1:172.21.15.19:6801/3264337803] conn(0x7f788c038f20 0x7f788c03b3e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f7894006fd0 tx=0x7f7894006e40 comp rx=0 tx=0).stop 2024-03-18T15:06:28.563 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:27.923+0000 7f78ade00700 1 -- 172.21.15.19:0/2843780134 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f78a8107cd0 msgr2=0x7f78a819ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:06:28.563 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:27.923+0000 7f78ade00700 1 --2- 172.21.15.19:0/2843780134 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f78a8107cd0 0x7f78a819ded0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f7898011850 tx=0x7f7898011c10 comp rx=0 tx=0).stop 2024-03-18T15:06:28.564 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:27.923+0000 7f78ade00700 1 -- 172.21.15.19:0/2843780134 shutdown_connections 2024-03-18T15:06:28.564 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:27.923+0000 7f78ade00700 1 --2- 172.21.15.19:0/2843780134 >> [v2:172.21.15.19:6800/3264337803,v1:172.21.15.19:6801/3264337803] conn(0x7f788c038f20 0x7f788c03b3e0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:06:28.564 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:27.923+0000 7f78ade00700 1 --2- 172.21.15.19:0/2843780134 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f78a8107cd0 0x7f78a819ded0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:06:28.564 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:27.923+0000 7f78ade00700 1 -- 172.21.15.19:0/2843780134 >> 172.21.15.19:0/2843780134 conn(0x7f78a8103230 msgr2=0x7f78a81951d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:06:28.564 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:27.924+0000 7f78ade00700 1 -- 172.21.15.19:0/2843780134 shutdown_connections 2024-03-18T15:06:28.564 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:27.924+0000 7f78ade00700 1 -- 172.21.15.19:0/2843780134 wait complete. 2024-03-18T15:06:28.564 INFO:teuthology.orchestra.run.smithi019.stdout:Enabling cephadm module... 2024-03-18T15:06:29.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:29 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/2064227166' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-18T15:06:29.715 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:06:29 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: ignoring --setuser ceph since I am not root 2024-03-18T15:06:29.716 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:06:29 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: ignoring --setgroup ceph since I am not root 2024-03-18T15:06:30.229 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:29.134+0000 7fdac4a00700 1 Processor -- start 2024-03-18T15:06:30.229 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:29.135+0000 7fdac4a00700 1 -- start start 2024-03-18T15:06:30.229 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:29.135+0000 7fdac4a00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fdac010a0f0 0x7fdac010a510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:06:30.229 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:29.135+0000 7fdac4a00700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fdac010aae0 con 0x7fdac010a0f0 2024-03-18T15:06:30.229 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:29.136+0000 7fdabde00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fdac010a0f0 0x7fdac010a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:06:30.229 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:29.136+0000 7fdabde00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fdac010a0f0 0x7fdac010a510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:38198/0 (socket says 172.21.15.19:38198) 2024-03-18T15:06:30.229 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:29.136+0000 7fdabde00700 1 -- 172.21.15.19:0/2883207769 learned_addr learned my addr 172.21.15.19:0/2883207769 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:06:30.229 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:29.136+0000 7fdabde00700 1 -- 172.21.15.19:0/2883207769 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdac010ac60 con 0x7fdac010a0f0 2024-03-18T15:06:30.229 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:29.136+0000 7fdabde00700 1 --2- 172.21.15.19:0/2883207769 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fdac010a0f0 0x7fdac010a510 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fdab00098a0 tx=0x7fdab0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ec9cbd449a35c38 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:06:30.229 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:29.137+0000 7fdabca00700 1 -- 172.21.15.19:0/2883207769 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdab0004160 con 0x7fdac010a0f0 2024-03-18T15:06:30.230 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:29.137+0000 7fdabca00700 1 -- 172.21.15.19:0/2883207769 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fdab0004300 con 0x7fdac010a0f0 2024-03-18T15:06:30.230 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:29.137+0000 7fdabca00700 1 -- 172.21.15.19:0/2883207769 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdab0011b50 con 0x7fdac010a0f0 2024-03-18T15:06:30.230 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:29.137+0000 7fdac4a00700 1 -- 172.21.15.19:0/2883207769 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fdac010a0f0 msgr2=0x7fdac010a510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:06:30.230 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:29.137+0000 7fdac4a00700 1 --2- 172.21.15.19:0/2883207769 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fdac010a0f0 0x7fdac010a510 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fdab00098a0 tx=0x7fdab0009bb0 comp rx=0 tx=0).stop 2024-03-18T15:06:30.230 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:29.138+0000 7fdac4a00700 1 -- 172.21.15.19:0/2883207769 shutdown_connections 2024-03-18T15:06:30.230 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:29.138+0000 7fdac4a00700 1 --2- 172.21.15.19:0/2883207769 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fdac010a0f0 0x7fdac010a510 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:06:30.230 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:29.138+0000 7fdac4a00700 1 -- 172.21.15.19:0/2883207769 >> 172.21.15.19:0/2883207769 conn(0x7fdac007d2c0 msgr2=0x7fdac007d6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:06:30.230 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:29.138+0000 7fdac4a00700 1 -- 172.21.15.19:0/2883207769 shutdown_connections 2024-03-18T15:06:30.230 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:29.138+0000 7fdac4a00700 1 -- 172.21.15.19:0/2883207769 wait complete. 2024-03-18T15:06:30.230 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:29.139+0000 7fdac4a00700 1 Processor -- start 2024-03-18T15:06:30.230 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:29.139+0000 7fdac4a00700 1 -- start start 2024-03-18T15:06:30.230 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:29.139+0000 7fdac4a00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fdac010a0f0 0x7fdac019dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:06:30.230 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:29.139+0000 7fdac4a00700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fdac010e300 con 0x7fdac010a0f0 2024-03-18T15:06:30.231 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:29.139+0000 7fdabde00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fdac010a0f0 0x7fdac019dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:06:30.231 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:29.140+0000 7fdabde00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fdac010a0f0 0x7fdac019dea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:38202/0 (socket says 172.21.15.19:38202) 2024-03-18T15:06:30.231 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:29.140+0000 7fdabde00700 1 -- 172.21.15.19:0/2064227166 learned_addr learned my addr 172.21.15.19:0/2064227166 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:06:30.231 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:29.140+0000 7fdabde00700 1 -- 172.21.15.19:0/2064227166 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdac019e3e0 con 0x7fdac010a0f0 2024-03-18T15:06:30.231 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:29.140+0000 7fdabde00700 1 --2- 172.21.15.19:0/2064227166 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fdac010a0f0 0x7fdac019dea0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fdab0011b60 tx=0x7fdab0011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:06:30.231 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:29.141+0000 7fdab6a00700 1 -- 172.21.15.19:0/2064227166 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdab0003970 con 0x7fdac010a0f0 2024-03-18T15:06:30.231 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:29.141+0000 7fdab6a00700 1 -- 172.21.15.19:0/2064227166 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fdab0003b10 con 0x7fdac010a0f0 2024-03-18T15:06:30.231 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:29.141+0000 7fdac4a00700 1 -- 172.21.15.19:0/2064227166 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdac019e670 con 0x7fdac010a0f0 2024-03-18T15:06:30.231 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:29.141+0000 7fdab6a00700 1 -- 172.21.15.19:0/2064227166 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdab0024610 con 0x7fdac010a0f0 2024-03-18T15:06:30.231 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:29.141+0000 7fdac4a00700 1 -- 172.21.15.19:0/2064227166 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdac01a1360 con 0x7fdac010a0f0 2024-03-18T15:06:30.231 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:29.142+0000 7fdab6a00700 1 -- 172.21.15.19:0/2064227166 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 4) ==== 46095+0+0 (secure 0 0 0) 0x7fdab001a430 con 0x7fdac010a0f0 2024-03-18T15:06:30.231 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:29.142+0000 7fdac4a00700 1 -- 172.21.15.19:0/2064227166 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdac010e480 con 0x7fdac010a0f0 2024-03-18T15:06:30.231 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:29.143+0000 7fdab6a00700 1 --2- 172.21.15.19:0/2064227166 >> [v2:172.21.15.19:6800/3264337803,v1:172.21.15.19:6801/3264337803] conn(0x7fdaa4038f20 0x7fdaa403b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:06:30.232 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:29.143+0000 7fdab6a00700 1 -- 172.21.15.19:0/2064227166 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fdab0011e00 con 0x7fdac010a0f0 2024-03-18T15:06:30.232 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:29.143+0000 7fdabd400700 1 --2- 172.21.15.19:0/2064227166 >> [v2:172.21.15.19:6800/3264337803,v1:172.21.15.19:6801/3264337803] conn(0x7fdaa4038f20 0x7fdaa403b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:06:30.232 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:29.144+0000 7fdabd400700 1 --2- 172.21.15.19:0/2064227166 >> [v2:172.21.15.19:6800/3264337803,v1:172.21.15.19:6801/3264337803] conn(0x7fdaa4038f20 0x7fdaa403b3e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fdaac006fd0 tx=0x7fdaac006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:06:30.232 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:29.148+0000 7fdab6a00700 1 -- 172.21.15.19:0/2064227166 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fdab000e260 con 0x7fdac010a0f0 2024-03-18T15:06:30.232 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:29.435+0000 7fdac4a00700 1 -- 172.21.15.19:0/2064227166 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7fdac01a19b0 con 0x7fdac010a0f0 2024-03-18T15:06:30.232 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:29.488+0000 7fdab6a00700 1 -- 172.21.15.19:0/2064227166 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7fdab0022db0 con 0x7fdac010a0f0 2024-03-18T15:06:30.232 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:29.492+0000 7fdac4a00700 1 -- 172.21.15.19:0/2064227166 >> [v2:172.21.15.19:6800/3264337803,v1:172.21.15.19:6801/3264337803] conn(0x7fdaa4038f20 msgr2=0x7fdaa403b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:06:30.232 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:29.492+0000 7fdac4a00700 1 --2- 172.21.15.19:0/2064227166 >> [v2:172.21.15.19:6800/3264337803,v1:172.21.15.19:6801/3264337803] conn(0x7fdaa4038f20 0x7fdaa403b3e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fdaac006fd0 tx=0x7fdaac006e40 comp rx=0 tx=0).stop 2024-03-18T15:06:30.232 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:29.492+0000 7fdac4a00700 1 -- 172.21.15.19:0/2064227166 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fdac010a0f0 msgr2=0x7fdac019dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:06:30.232 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:29.492+0000 7fdac4a00700 1 --2- 172.21.15.19:0/2064227166 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fdac010a0f0 0x7fdac019dea0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fdab0011b60 tx=0x7fdab0011c40 comp rx=0 tx=0).stop 2024-03-18T15:06:30.232 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:29.492+0000 7fdab6a00700 1 -- 172.21.15.19:0/2064227166 <== mon.0 v2:172.21.15.19:3300/0 8 ==== mgrmap(e 5) ==== 46106+0+0 (secure 0 0 0) 0x7fdab001a9f0 con 0x7fdac010a0f0 2024-03-18T15:06:30.232 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:29.493+0000 7fdac4a00700 1 -- 172.21.15.19:0/2064227166 shutdown_connections 2024-03-18T15:06:30.232 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:29.493+0000 7fdac4a00700 1 --2- 172.21.15.19:0/2064227166 >> [v2:172.21.15.19:6800/3264337803,v1:172.21.15.19:6801/3264337803] conn(0x7fdaa4038f20 0x7fdaa403b3e0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:06:30.232 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:29.493+0000 7fdac4a00700 1 --2- 172.21.15.19:0/2064227166 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fdac010a0f0 0x7fdac019dea0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:06:30.233 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:29.493+0000 7fdac4a00700 1 -- 172.21.15.19:0/2064227166 >> 172.21.15.19:0/2064227166 conn(0x7fdac007d2c0 msgr2=0x7fdac0107d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:06:30.233 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:29.493+0000 7fdac4a00700 1 -- 172.21.15.19:0/2064227166 shutdown_connections 2024-03-18T15:06:30.233 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:29.494+0000 7fdac4a00700 1 -- 172.21.15.19:0/2064227166 wait complete. 2024-03-18T15:06:30.463 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:06:30 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:06:30.335+0000 7fba941a3200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T15:06:30.464 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:06:30 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:06:30.462+0000 7fba941a3200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T15:06:30.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:30 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/2064227166' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-18T15:06:30.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:30 smithi019 ceph-mon[31522]: mgrmap e5: x(active, since 5s) 2024-03-18T15:06:31.552 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:31 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/2839139736' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-18T15:06:31.553 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:06:31 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:06:31.299+0000 7fba941a3200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T15:06:31.553 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:06:31 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:06:31.428+0000 7fba941a3200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T15:06:31.870 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:06:31 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:06:31.613+0000 7fba941a3200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T15:06:31.870 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:06:31 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:06:31.744+0000 7fba941a3200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T15:06:31.870 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:06:31 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:06:31.860+0000 7fba941a3200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T15:06:31.884 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout { 2024-03-18T15:06:31.885 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-18T15:06:31.885 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-18T15:06:31.885 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-18T15:06:31.885 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-18T15:06:31.885 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout } 2024-03-18T15:06:31.885 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:30.832+0000 7f2ba5800700 1 Processor -- start 2024-03-18T15:06:31.885 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:30.832+0000 7f2ba5800700 1 -- start start 2024-03-18T15:06:31.885 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:30.833+0000 7f2ba5800700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2ba007de70 0x7f2ba007c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:06:31.885 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:30.833+0000 7f2ba5800700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f2ba007c8a0 con 0x7f2ba007de70 2024-03-18T15:06:31.885 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:30.833+0000 7f2b9ea00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2ba007de70 0x7f2ba007c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:06:31.885 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:30.833+0000 7f2b9ea00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2ba007de70 0x7f2ba007c2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:38232/0 (socket says 172.21.15.19:38232) 2024-03-18T15:06:31.885 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:30.833+0000 7f2b9ea00700 1 -- 172.21.15.19:0/2938911415 learned_addr learned my addr 172.21.15.19:0/2938911415 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:06:31.886 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:30.834+0000 7f2b9ea00700 1 -- 172.21.15.19:0/2938911415 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ba007ca20 con 0x7f2ba007de70 2024-03-18T15:06:31.886 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:30.834+0000 7f2b9ea00700 1 --2- 172.21.15.19:0/2938911415 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2ba007de70 0x7f2ba007c2d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2b900098a0 tx=0x7f2b90009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bef1e032f7b384a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:06:31.886 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:30.834+0000 7f2b9d600700 1 -- 172.21.15.19:0/2938911415 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2b90004160 con 0x7f2ba007de70 2024-03-18T15:06:31.886 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:30.834+0000 7f2b9d600700 1 -- 172.21.15.19:0/2938911415 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f2b90004300 con 0x7f2ba007de70 2024-03-18T15:06:31.886 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:30.834+0000 7f2b9d600700 1 -- 172.21.15.19:0/2938911415 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2b90011b50 con 0x7f2ba007de70 2024-03-18T15:06:31.886 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:30.835+0000 7f2ba5800700 1 -- 172.21.15.19:0/2938911415 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2ba007de70 msgr2=0x7f2ba007c2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:06:31.886 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:30.835+0000 7f2ba5800700 1 --2- 172.21.15.19:0/2938911415 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2ba007de70 0x7f2ba007c2d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2b900098a0 tx=0x7f2b90009bb0 comp rx=0 tx=0).stop 2024-03-18T15:06:31.886 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:30.835+0000 7f2ba5800700 1 -- 172.21.15.19:0/2938911415 shutdown_connections 2024-03-18T15:06:31.886 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:30.835+0000 7f2ba5800700 1 --2- 172.21.15.19:0/2938911415 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2ba007de70 0x7f2ba007c2d0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:06:31.886 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:30.835+0000 7f2ba5800700 1 -- 172.21.15.19:0/2938911415 >> 172.21.15.19:0/2938911415 conn(0x7f2ba0103250 msgr2=0x7f2ba01056b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:06:31.886 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:30.835+0000 7f2ba5800700 1 -- 172.21.15.19:0/2938911415 shutdown_connections 2024-03-18T15:06:31.886 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:30.835+0000 7f2ba5800700 1 -- 172.21.15.19:0/2938911415 wait complete. 2024-03-18T15:06:31.886 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:30.836+0000 7f2ba5800700 1 Processor -- start 2024-03-18T15:06:31.887 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:30.836+0000 7f2ba5800700 1 -- start start 2024-03-18T15:06:31.887 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:30.837+0000 7f2ba5800700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2ba007de70 0x7f2ba019df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:06:31.887 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:30.837+0000 7f2ba5800700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f2ba010e300 con 0x7f2ba007de70 2024-03-18T15:06:31.887 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:30.837+0000 7f2b9ea00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2ba007de70 0x7f2ba019df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:06:31.887 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:30.837+0000 7f2b9ea00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2ba007de70 0x7f2ba019df00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:38236/0 (socket says 172.21.15.19:38236) 2024-03-18T15:06:31.887 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:30.837+0000 7f2b9ea00700 1 -- 172.21.15.19:0/2839139736 learned_addr learned my addr 172.21.15.19:0/2839139736 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:06:31.887 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:30.837+0000 7f2b9ea00700 1 -- 172.21.15.19:0/2839139736 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ba019e440 con 0x7f2ba007de70 2024-03-18T15:06:31.887 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:30.838+0000 7f2b9ea00700 1 --2- 172.21.15.19:0/2839139736 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2ba007de70 0x7f2ba019df00 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2b90009fb0 tx=0x7f2b90011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:06:31.887 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:30.838+0000 7f2b97400700 1 -- 172.21.15.19:0/2839139736 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2b900038b0 con 0x7f2ba007de70 2024-03-18T15:06:31.887 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:30.838+0000 7f2b97400700 1 -- 172.21.15.19:0/2839139736 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f2b90011e50 con 0x7f2ba007de70 2024-03-18T15:06:31.887 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:30.838+0000 7f2b97400700 1 -- 172.21.15.19:0/2839139736 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2b90019420 con 0x7f2ba007de70 2024-03-18T15:06:31.887 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:30.838+0000 7f2ba5800700 1 -- 172.21.15.19:0/2839139736 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ba019e6d0 con 0x7f2ba007de70 2024-03-18T15:06:31.888 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:30.838+0000 7f2ba5800700 1 -- 172.21.15.19:0/2839139736 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ba01a13c0 con 0x7f2ba007de70 2024-03-18T15:06:31.888 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:30.839+0000 7f2ba5800700 1 -- 172.21.15.19:0/2839139736 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ba00693d0 con 0x7f2ba007de70 2024-03-18T15:06:31.888 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:30.840+0000 7f2b97400700 1 -- 172.21.15.19:0/2839139736 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 46106+0+0 (secure 0 0 0) 0x7f2b90003a50 con 0x7f2ba007de70 2024-03-18T15:06:31.888 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:30.840+0000 7f2b97400700 1 --2- 172.21.15.19:0/2839139736 >> [v2:172.21.15.19:6800/3264337803,v1:172.21.15.19:6801/3264337803] conn(0x7f2b84038f20 0x7f2b8403b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:06:31.888 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:30.840+0000 7f2b97400700 1 -- 172.21.15.19:0/2839139736 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f2b9004d120 con 0x7f2ba007de70 2024-03-18T15:06:31.888 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:30.840+0000 7f2b9e000700 1 -- 172.21.15.19:0/2839139736 >> [v2:172.21.15.19:6800/3264337803,v1:172.21.15.19:6801/3264337803] conn(0x7f2b84038f20 msgr2=0x7f2b8403b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.19:6800/3264337803 2024-03-18T15:06:31.888 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:30.840+0000 7f2b9e000700 1 --2- 172.21.15.19:0/2839139736 >> [v2:172.21.15.19:6800/3264337803,v1:172.21.15.19:6801/3264337803] conn(0x7f2b84038f20 0x7f2b8403b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T15:06:31.888 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:30.850+0000 7f2b97400700 1 -- 172.21.15.19:0/2839139736 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2b9004cb40 con 0x7f2ba007de70 2024-03-18T15:06:31.888 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:31.041+0000 7f2b9e000700 1 -- 172.21.15.19:0/2839139736 >> [v2:172.21.15.19:6800/3264337803,v1:172.21.15.19:6801/3264337803] conn(0x7f2b84038f20 msgr2=0x7f2b8403b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.19:6800/3264337803 2024-03-18T15:06:31.888 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:31.041+0000 7f2b9e000700 1 --2- 172.21.15.19:0/2839139736 >> [v2:172.21.15.19:6800/3264337803,v1:172.21.15.19:6801/3264337803] conn(0x7f2b84038f20 0x7f2b8403b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T15:06:31.888 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:31.134+0000 7f2ba5800700 1 -- 172.21.15.19:0/2839139736 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f2ba01a1630 con 0x7f2ba007de70 2024-03-18T15:06:31.888 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:31.134+0000 7f2b97400700 1 -- 172.21.15.19:0/2839139736 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7f2b900195c0 con 0x7f2ba007de70 2024-03-18T15:06:31.888 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:31.138+0000 7f2ba5800700 1 -- 172.21.15.19:0/2839139736 >> [v2:172.21.15.19:6800/3264337803,v1:172.21.15.19:6801/3264337803] conn(0x7f2b84038f20 msgr2=0x7f2b8403b3e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T15:06:31.889 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:31.138+0000 7f2ba5800700 1 --2- 172.21.15.19:0/2839139736 >> [v2:172.21.15.19:6800/3264337803,v1:172.21.15.19:6801/3264337803] conn(0x7f2b84038f20 0x7f2b8403b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:06:31.889 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:31.138+0000 7f2ba5800700 1 -- 172.21.15.19:0/2839139736 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2ba007de70 msgr2=0x7f2ba019df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:06:31.889 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:31.138+0000 7f2ba5800700 1 --2- 172.21.15.19:0/2839139736 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2ba007de70 0x7f2ba019df00 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2b90009fb0 tx=0x7f2b90011bf0 comp rx=0 tx=0).stop 2024-03-18T15:06:31.889 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:31.138+0000 7f2ba5800700 1 -- 172.21.15.19:0/2839139736 shutdown_connections 2024-03-18T15:06:31.889 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:31.138+0000 7f2ba5800700 1 --2- 172.21.15.19:0/2839139736 >> [v2:172.21.15.19:6800/3264337803,v1:172.21.15.19:6801/3264337803] conn(0x7f2b84038f20 0x7f2b8403b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:06:31.889 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:31.138+0000 7f2ba5800700 1 --2- 172.21.15.19:0/2839139736 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2ba007de70 0x7f2ba019df00 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:06:31.889 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:31.138+0000 7f2ba5800700 1 -- 172.21.15.19:0/2839139736 >> 172.21.15.19:0/2839139736 conn(0x7f2ba0103250 msgr2=0x7f2ba0103e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:06:31.889 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:31.138+0000 7f2ba5800700 1 -- 172.21.15.19:0/2839139736 shutdown_connections 2024-03-18T15:06:31.889 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:31.138+0000 7f2ba5800700 1 -- 172.21.15.19:0/2839139736 wait complete. 2024-03-18T15:06:31.889 INFO:teuthology.orchestra.run.smithi019.stdout:Waiting for the mgr to restart... 2024-03-18T15:06:31.889 INFO:teuthology.orchestra.run.smithi019.stdout:Waiting for mgr epoch 5... 2024-03-18T15:06:32.454 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:06:32 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:06:32.221+0000 7fba941a3200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T15:06:32.455 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:06:32 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:06:32.370+0000 7fba941a3200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T15:06:32.715 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:06:32 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:06:32.490+0000 7fba941a3200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T15:06:32.715 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:06:32 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:06:32.617+0000 7fba941a3200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T15:06:33.215 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:06:32 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:06:32.841+0000 7fba941a3200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T15:06:34.465 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:06:34 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:06:34.171+0000 7fba941a3200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T15:06:34.465 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:06:34 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:06:34.350+0000 7fba941a3200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T15:06:35.170 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:06:34 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:06:34.780+0000 7fba941a3200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T15:06:35.170 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:06:34 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:06:34.903+0000 7fba941a3200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T15:06:35.170 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:06:35 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:06:35.168+0000 7fba941a3200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T15:06:35.715 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:06:35 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:06:35.424+0000 7fba941a3200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T15:06:36.142 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:06:35 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:06:35.815+0000 7fba941a3200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T15:06:36.460 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:06:36 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:06:36.141+0000 7fba941a3200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T15:06:36.715 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:06:36 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:06:36.458+0000 7fba941a3200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T15:06:37.965 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:06:37 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:06:37.531+0000 7fba941a3200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T15:06:38.669 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:06:38 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:06:38.254+0000 7fba941a3200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T15:06:38.670 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:06:38 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:06:38.390+0000 7fba941a3200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T15:06:38.965 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:06:38 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:06:38.683+0000 7fba941a3200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T15:06:38.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:38 smithi019 ceph-mon[31522]: Active manager daemon x restarted 2024-03-18T15:06:38.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:38 smithi019 ceph-mon[31522]: Activating manager daemon x 2024-03-18T15:06:38.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:38 smithi019 ceph-mon[31522]: osdmap e2: 0 total, 0 up, 0 in 2024-03-18T15:06:38.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:38 smithi019 ceph-mon[31522]: mgrmap e6: x(active, starting, since 0.00783132s) 2024-03-18T15:06:38.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:38 smithi019 ceph-mon[31522]: from='mgr.14120 172.21.15.19:0/2858628912' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T15:06:38.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:38 smithi019 ceph-mon[31522]: from='mgr.14120 172.21.15.19:0/2858628912' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-18T15:06:38.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:38 smithi019 ceph-mon[31522]: from='mgr.14120 172.21.15.19:0/2858628912' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-18T15:06:38.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:38 smithi019 ceph-mon[31522]: from='mgr.14120 172.21.15.19:0/2858628912' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-18T15:06:38.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:38 smithi019 ceph-mon[31522]: from='mgr.14120 172.21.15.19:0/2858628912' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-18T15:06:38.967 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:38 smithi019 ceph-mon[31522]: Manager daemon x is now available 2024-03-18T15:06:38.967 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:38 smithi019 ceph-mon[31522]: from='mgr.14120 172.21.15.19:0/2858628912' entity='mgr.x' 2024-03-18T15:06:39.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:39 smithi019 ceph-mon[31522]: Found migration_current of "None". Setting to last migration. 2024-03-18T15:06:39.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:39 smithi019 ceph-mon[31522]: from='mgr.14120 172.21.15.19:0/2858628912' entity='mgr.x' 2024-03-18T15:06:39.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:39 smithi019 ceph-mon[31522]: from='mgr.14120 172.21.15.19:0/2858628912' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:06:39.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:39 smithi019 ceph-mon[31522]: from='mgr.14120 172.21.15.19:0/2858628912' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:06:39.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:39 smithi019 ceph-mon[31522]: from='mgr.14120 172.21.15.19:0/2858628912' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:06:39.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:39 smithi019 ceph-mon[31522]: from='mgr.14120 172.21.15.19:0/2858628912' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-18T15:06:39.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:39 smithi019 ceph-mon[31522]: from='mgr.14120 172.21.15.19:0/2858628912' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-18T15:06:39.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:39 smithi019 ceph-mon[31522]: from='mgr.14120 172.21.15.19:0/2858628912' entity='mgr.x' 2024-03-18T15:06:39.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:39 smithi019 ceph-mon[31522]: from='mgr.14120 172.21.15.19:0/2858628912' entity='mgr.x' 2024-03-18T15:06:39.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:39 smithi019 ceph-mon[31522]: mgrmap e7: x(active, since 1.01674s) 2024-03-18T15:06:40.520 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout { 2024-03-18T15:06:40.520 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-18T15:06:40.520 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-18T15:06:40.520 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout } 2024-03-18T15:06:40.520 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:32.453+0000 7fb827200700 1 Processor -- start 2024-03-18T15:06:40.520 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:32.453+0000 7fb827200700 1 -- start start 2024-03-18T15:06:40.520 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:32.453+0000 7fb827200700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb8281085f0 0x7fb82810a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:06:40.520 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:32.453+0000 7fb827200700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fb82807c680 con 0x7fb8281085f0 2024-03-18T15:06:40.521 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:32.454+0000 7fb825e00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb8281085f0 0x7fb82810a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:06:40.521 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:32.454+0000 7fb825e00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb8281085f0 0x7fb82810a9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:39644/0 (socket says 172.21.15.19:39644) 2024-03-18T15:06:40.521 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:32.454+0000 7fb825e00700 1 -- 172.21.15.19:0/1864262887 learned_addr learned my addr 172.21.15.19:0/1864262887 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:06:40.521 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:32.454+0000 7fb825e00700 1 -- 172.21.15.19:0/1864262887 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb82810af20 con 0x7fb8281085f0 2024-03-18T15:06:40.521 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:32.455+0000 7fb825e00700 1 --2- 172.21.15.19:0/1864262887 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb8281085f0 0x7fb82810a9e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb8180098a0 tx=0x7fb818009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de972f27b4d88e60 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:06:40.521 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:32.455+0000 7fb824a00700 1 -- 172.21.15.19:0/1864262887 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb818004160 con 0x7fb8281085f0 2024-03-18T15:06:40.521 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:32.455+0000 7fb824a00700 1 -- 172.21.15.19:0/1864262887 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fb818004300 con 0x7fb8281085f0 2024-03-18T15:06:40.521 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:32.455+0000 7fb824a00700 1 -- 172.21.15.19:0/1864262887 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb818011b50 con 0x7fb8281085f0 2024-03-18T15:06:40.521 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:32.456+0000 7fb827200700 1 -- 172.21.15.19:0/1864262887 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb8281085f0 msgr2=0x7fb82810a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:06:40.521 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:32.456+0000 7fb827200700 1 --2- 172.21.15.19:0/1864262887 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb8281085f0 0x7fb82810a9e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb8180098a0 tx=0x7fb818009bb0 comp rx=0 tx=0).stop 2024-03-18T15:06:40.521 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:32.456+0000 7fb827200700 1 -- 172.21.15.19:0/1864262887 shutdown_connections 2024-03-18T15:06:40.522 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:32.456+0000 7fb827200700 1 --2- 172.21.15.19:0/1864262887 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb8281085f0 0x7fb82810a9e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:06:40.522 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:32.456+0000 7fb827200700 1 -- 172.21.15.19:0/1864262887 >> 172.21.15.19:0/1864262887 conn(0x7fb828101fc0 msgr2=0x7fb828104400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:06:40.522 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:32.456+0000 7fb827200700 1 -- 172.21.15.19:0/1864262887 shutdown_connections 2024-03-18T15:06:40.522 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:32.456+0000 7fb827200700 1 -- 172.21.15.19:0/1864262887 wait complete. 2024-03-18T15:06:40.522 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:32.458+0000 7fb827200700 1 Processor -- start 2024-03-18T15:06:40.522 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:32.458+0000 7fb827200700 1 -- start start 2024-03-18T15:06:40.522 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:32.458+0000 7fb827200700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb8281085f0 0x7fb828107e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:06:40.522 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:32.458+0000 7fb827200700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fb82810bee0 con 0x7fb8281085f0 2024-03-18T15:06:40.522 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:32.458+0000 7fb825e00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb8281085f0 0x7fb828107e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:06:40.522 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:32.458+0000 7fb825e00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb8281085f0 0x7fb828107e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:39652/0 (socket says 172.21.15.19:39652) 2024-03-18T15:06:40.522 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:32.459+0000 7fb825e00700 1 -- 172.21.15.19:0/3885664032 learned_addr learned my addr 172.21.15.19:0/3885664032 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:06:40.522 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:32.459+0000 7fb825e00700 1 -- 172.21.15.19:0/3885664032 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8281083b0 con 0x7fb8281085f0 2024-03-18T15:06:40.522 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:32.459+0000 7fb825e00700 1 --2- 172.21.15.19:0/3885664032 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb8281085f0 0x7fb828107e70 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb818011b60 tx=0x7fb818011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:06:40.523 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:32.459+0000 7fb81ea00700 1 -- 172.21.15.19:0/3885664032 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb818003970 con 0x7fb8281085f0 2024-03-18T15:06:40.523 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:32.459+0000 7fb81ea00700 1 -- 172.21.15.19:0/3885664032 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fb818003b10 con 0x7fb8281085f0 2024-03-18T15:06:40.523 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:32.460+0000 7fb81ea00700 1 -- 172.21.15.19:0/3885664032 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb818024610 con 0x7fb8281085f0 2024-03-18T15:06:40.523 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:32.460+0000 7fb827200700 1 -- 172.21.15.19:0/3885664032 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8281065a0 con 0x7fb8281085f0 2024-03-18T15:06:40.523 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:32.460+0000 7fb827200700 1 -- 172.21.15.19:0/3885664032 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb828106ae0 con 0x7fb8281085f0 2024-03-18T15:06:40.523 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:32.461+0000 7fb81ea00700 1 -- 172.21.15.19:0/3885664032 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 46106+0+0 (secure 0 0 0) 0x7fb81801a430 con 0x7fb8281085f0 2024-03-18T15:06:40.523 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:32.461+0000 7fb81ea00700 1 --2- 172.21.15.19:0/3885664032 >> [v2:172.21.15.19:6800/3264337803,v1:172.21.15.19:6801/3264337803] conn(0x7fb80c038fd0 0x7fb80c03b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:06:40.523 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:32.461+0000 7fb81ea00700 1 -- 172.21.15.19:0/3885664032 --> [v2:172.21.15.19:6800/3264337803,v1:172.21.15.19:6801/3264337803] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fb80c03bba0 con 0x7fb80c038fd0 2024-03-18T15:06:40.523 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:32.461+0000 7fb81ea00700 1 -- 172.21.15.19:0/3885664032 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fb818011e00 con 0x7fb8281085f0 2024-03-18T15:06:40.523 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:32.461+0000 7fb825400700 1 -- 172.21.15.19:0/3885664032 >> [v2:172.21.15.19:6800/3264337803,v1:172.21.15.19:6801/3264337803] conn(0x7fb80c038fd0 msgr2=0x7fb80c03b490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.19:6800/3264337803 2024-03-18T15:06:40.523 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:32.461+0000 7fb825400700 1 --2- 172.21.15.19:0/3885664032 >> [v2:172.21.15.19:6800/3264337803,v1:172.21.15.19:6801/3264337803] conn(0x7fb80c038fd0 0x7fb80c03b490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T15:06:40.523 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:32.662+0000 7fb825400700 1 -- 172.21.15.19:0/3885664032 >> [v2:172.21.15.19:6800/3264337803,v1:172.21.15.19:6801/3264337803] conn(0x7fb80c038fd0 msgr2=0x7fb80c03b490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.19:6800/3264337803 2024-03-18T15:06:40.524 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:32.662+0000 7fb825400700 1 --2- 172.21.15.19:0/3885664032 >> [v2:172.21.15.19:6800/3264337803,v1:172.21.15.19:6801/3264337803] conn(0x7fb80c038fd0 0x7fb80c03b490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T15:06:40.524 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:33.063+0000 7fb825400700 1 -- 172.21.15.19:0/3885664032 >> [v2:172.21.15.19:6800/3264337803,v1:172.21.15.19:6801/3264337803] conn(0x7fb80c038fd0 msgr2=0x7fb80c03b490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.19:6800/3264337803 2024-03-18T15:06:40.524 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:33.063+0000 7fb825400700 1 --2- 172.21.15.19:0/3885664032 >> [v2:172.21.15.19:6800/3264337803,v1:172.21.15.19:6801/3264337803] conn(0x7fb80c038fd0 0x7fb80c03b490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-18T15:06:40.524 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:33.864+0000 7fb825400700 1 -- 172.21.15.19:0/3885664032 >> [v2:172.21.15.19:6800/3264337803,v1:172.21.15.19:6801/3264337803] conn(0x7fb80c038fd0 msgr2=0x7fb80c03b490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.19:6800/3264337803 2024-03-18T15:06:40.524 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:33.864+0000 7fb825400700 1 --2- 172.21.15.19:0/3885664032 >> [v2:172.21.15.19:6800/3264337803,v1:172.21.15.19:6801/3264337803] conn(0x7fb80c038fd0 0x7fb80c03b490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-18T15:06:40.524 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:35.466+0000 7fb825400700 1 -- 172.21.15.19:0/3885664032 >> [v2:172.21.15.19:6800/3264337803,v1:172.21.15.19:6801/3264337803] conn(0x7fb80c038fd0 msgr2=0x7fb80c03b490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.19:6800/3264337803 2024-03-18T15:06:40.524 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:35.466+0000 7fb825400700 1 --2- 172.21.15.19:0/3885664032 >> [v2:172.21.15.19:6800/3264337803,v1:172.21.15.19:6801/3264337803] conn(0x7fb80c038fd0 0x7fb80c03b490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-18T15:06:40.524 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:38.668+0000 7fb825400700 1 -- 172.21.15.19:0/3885664032 >> [v2:172.21.15.19:6800/3264337803,v1:172.21.15.19:6801/3264337803] conn(0x7fb80c038fd0 msgr2=0x7fb80c03b490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.19:6800/3264337803 2024-03-18T15:06:40.524 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:38.668+0000 7fb825400700 1 --2- 172.21.15.19:0/3885664032 >> [v2:172.21.15.19:6800/3264337803,v1:172.21.15.19:6801/3264337803] conn(0x7fb80c038fd0 0x7fb80c03b490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-03-18T15:06:40.524 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:38.692+0000 7fb81ea00700 1 -- 172.21.15.19:0/3885664032 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mgrmap(e 6) ==== 45873+0+0 (secure 0 0 0) 0x7fb81801a9f0 con 0x7fb8281085f0 2024-03-18T15:06:40.524 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:38.692+0000 7fb81ea00700 1 -- 172.21.15.19:0/3885664032 >> [v2:172.21.15.19:6800/3264337803,v1:172.21.15.19:6801/3264337803] conn(0x7fb80c038fd0 msgr2=0x7fb80c03b490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T15:06:40.524 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:38.693+0000 7fb81ea00700 1 --2- 172.21.15.19:0/3885664032 >> [v2:172.21.15.19:6800/3264337803,v1:172.21.15.19:6801/3264337803] conn(0x7fb80c038fd0 0x7fb80c03b490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:06:40.525 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:39.697+0000 7fb81ea00700 1 -- 172.21.15.19:0/3885664032 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mgrmap(e 7) ==== 46000+0+0 (secure 0 0 0) 0x7fb81804e7b0 con 0x7fb8281085f0 2024-03-18T15:06:40.525 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:39.697+0000 7fb81ea00700 1 --2- 172.21.15.19:0/3885664032 >> [v2:172.21.15.19:6800/2776008962,v1:172.21.15.19:6801/2776008962] conn(0x7fb80c03c8f0 0x7fb80c03ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:06:40.525 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:39.697+0000 7fb81ea00700 1 -- 172.21.15.19:0/3885664032 --> [v2:172.21.15.19:6800/2776008962,v1:172.21.15.19:6801/2776008962] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fb818001d30 con 0x7fb80c03c8f0 2024-03-18T15:06:40.525 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:39.700+0000 7fb825400700 1 --2- 172.21.15.19:0/3885664032 >> [v2:172.21.15.19:6800/2776008962,v1:172.21.15.19:6801/2776008962] conn(0x7fb80c03c8f0 0x7fb80c03ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:06:40.525 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:39.703+0000 7fb825400700 1 --2- 172.21.15.19:0/3885664032 >> [v2:172.21.15.19:6800/2776008962,v1:172.21.15.19:6801/2776008962] conn(0x7fb80c03c8f0 0x7fb80c03ece0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fb814003a10 tx=0x7fb8140092b0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:06:40.525 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:39.705+0000 7fb81ea00700 1 -- 172.21.15.19:0/3885664032 <== mgr.14120 v2:172.21.15.19:6800/2776008962 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7fb818001d30 con 0x7fb80c03c8f0 2024-03-18T15:06:40.525 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:39.713+0000 7fb827200700 1 -- 172.21.15.19:0/3885664032 --> [v2:172.21.15.19:6800/2776008962,v1:172.21.15.19:6801/2776008962] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7fb8280693d0 con 0x7fb80c03c8f0 2024-03-18T15:06:40.525 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:39.714+0000 7fb81ea00700 1 -- 172.21.15.19:0/3885664032 <== mgr.14120 v2:172.21.15.19:6800/2776008962 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7fb8280693d0 con 0x7fb80c03c8f0 2024-03-18T15:06:40.525 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:39.714+0000 7fb827200700 1 -- 172.21.15.19:0/3885664032 >> [v2:172.21.15.19:6800/2776008962,v1:172.21.15.19:6801/2776008962] conn(0x7fb80c03c8f0 msgr2=0x7fb80c03ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:06:40.525 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:39.714+0000 7fb827200700 1 --2- 172.21.15.19:0/3885664032 >> [v2:172.21.15.19:6800/2776008962,v1:172.21.15.19:6801/2776008962] conn(0x7fb80c03c8f0 0x7fb80c03ece0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fb814003a10 tx=0x7fb8140092b0 comp rx=0 tx=0).stop 2024-03-18T15:06:40.525 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:39.714+0000 7fb827200700 1 -- 172.21.15.19:0/3885664032 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb8281085f0 msgr2=0x7fb828107e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:06:40.526 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:39.714+0000 7fb827200700 1 --2- 172.21.15.19:0/3885664032 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb8281085f0 0x7fb828107e70 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb818011b60 tx=0x7fb818011c40 comp rx=0 tx=0).stop 2024-03-18T15:06:40.526 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:39.714+0000 7fb827200700 1 -- 172.21.15.19:0/3885664032 shutdown_connections 2024-03-18T15:06:40.526 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:39.714+0000 7fb827200700 1 --2- 172.21.15.19:0/3885664032 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb8281085f0 0x7fb828107e70 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:06:40.526 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:39.714+0000 7fb827200700 1 --2- 172.21.15.19:0/3885664032 >> [v2:172.21.15.19:6800/2776008962,v1:172.21.15.19:6801/2776008962] conn(0x7fb80c03c8f0 0x7fb80c03ece0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:06:40.526 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:39.714+0000 7fb827200700 1 --2- 172.21.15.19:0/3885664032 >> [v2:172.21.15.19:6800/3264337803,v1:172.21.15.19:6801/3264337803] conn(0x7fb80c038fd0 0x7fb80c03b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:06:40.526 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:39.714+0000 7fb827200700 1 -- 172.21.15.19:0/3885664032 >> 172.21.15.19:0/3885664032 conn(0x7fb828101fc0 msgr2=0x7fb828102b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:06:40.526 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:39.715+0000 7fb827200700 1 -- 172.21.15.19:0/3885664032 shutdown_connections 2024-03-18T15:06:40.526 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:39.715+0000 7fb827200700 1 -- 172.21.15.19:0/3885664032 wait complete. 2024-03-18T15:06:40.526 INFO:teuthology.orchestra.run.smithi019.stdout:mgr epoch 5 is available 2024-03-18T15:06:40.526 INFO:teuthology.orchestra.run.smithi019.stdout:Setting orchestrator backend to cephadm... 2024-03-18T15:06:41.064 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:40 smithi019 ceph-mon[31522]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-18T15:06:41.064 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:40 smithi019 ceph-mon[31522]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-18T15:06:41.064 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:40 smithi019 ceph-mon[31522]: [18/Mar/2024:15:06:40] ENGINE Bus STARTING 2024-03-18T15:06:41.064 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:40 smithi019 ceph-mon[31522]: [18/Mar/2024:15:06:40] ENGINE Serving on https://172.21.15.19:7150 2024-03-18T15:06:41.064 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:40 smithi019 ceph-mon[31522]: [18/Mar/2024:15:06:40] ENGINE Serving on http://172.21.15.19:8765 2024-03-18T15:06:41.064 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:40 smithi019 ceph-mon[31522]: [18/Mar/2024:15:06:40] ENGINE Bus STARTED 2024-03-18T15:06:41.064 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:40 smithi019 ceph-mon[31522]: from='mgr.14120 172.21.15.19:0/2858628912' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:06:42.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:41 smithi019 ceph-mon[31522]: mgrmap e8: x(active, since 2s) 2024-03-18T15:06:42.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:41 smithi019 ceph-mon[31522]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:06:42.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:41 smithi019 ceph-mon[31522]: from='mgr.14120 172.21.15.19:0/2858628912' entity='mgr.x' 2024-03-18T15:06:42.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:41 smithi019 ceph-mon[31522]: from='mgr.14120 172.21.15.19:0/2858628912' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:06:42.396 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:41.148+0000 7f659bc00700 1 Processor -- start 2024-03-18T15:06:42.396 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:41.148+0000 7f659bc00700 1 -- start start 2024-03-18T15:06:42.396 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:41.148+0000 7f659bc00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f659410a0f0 0x7f659410a510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:06:42.396 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:41.149+0000 7f659bc00700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f659410aae0 con 0x7f659410a0f0 2024-03-18T15:06:42.396 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:41.149+0000 7f6599200700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f659410a0f0 0x7f659410a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:06:42.396 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:41.149+0000 7f6599200700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f659410a0f0 0x7f659410a510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:39726/0 (socket says 172.21.15.19:39726) 2024-03-18T15:06:42.396 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:41.149+0000 7f6599200700 1 -- 172.21.15.19:0/2419000062 learned_addr learned my addr 172.21.15.19:0/2419000062 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:06:42.396 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:41.149+0000 7f6599200700 1 -- 172.21.15.19:0/2419000062 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f659410ac60 con 0x7f659410a0f0 2024-03-18T15:06:42.396 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:41.150+0000 7f6599200700 1 --2- 172.21.15.19:0/2419000062 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f659410a0f0 0x7f659410a510 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f65880098a0 tx=0x7f6588009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee146ae6c49ea079 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:06:42.397 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:41.150+0000 7f6593400700 1 -- 172.21.15.19:0/2419000062 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6588004160 con 0x7f659410a0f0 2024-03-18T15:06:42.397 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:41.150+0000 7f6593400700 1 -- 172.21.15.19:0/2419000062 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f6588004300 con 0x7f659410a0f0 2024-03-18T15:06:42.397 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:41.150+0000 7f6593400700 1 -- 172.21.15.19:0/2419000062 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6588011b50 con 0x7f659410a0f0 2024-03-18T15:06:42.397 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:41.151+0000 7f659bc00700 1 -- 172.21.15.19:0/2419000062 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f659410a0f0 msgr2=0x7f659410a510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:06:42.397 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:41.151+0000 7f659bc00700 1 --2- 172.21.15.19:0/2419000062 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f659410a0f0 0x7f659410a510 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f65880098a0 tx=0x7f6588009bb0 comp rx=0 tx=0).stop 2024-03-18T15:06:42.397 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:41.151+0000 7f659bc00700 1 -- 172.21.15.19:0/2419000062 shutdown_connections 2024-03-18T15:06:42.397 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:41.151+0000 7f659bc00700 1 --2- 172.21.15.19:0/2419000062 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f659410a0f0 0x7f659410a510 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:06:42.397 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:41.151+0000 7f659bc00700 1 -- 172.21.15.19:0/2419000062 >> 172.21.15.19:0/2419000062 conn(0x7f659407d2c0 msgr2=0x7f659407d6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:06:42.397 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:41.151+0000 7f659bc00700 1 -- 172.21.15.19:0/2419000062 shutdown_connections 2024-03-18T15:06:42.397 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:41.151+0000 7f659bc00700 1 -- 172.21.15.19:0/2419000062 wait complete. 2024-03-18T15:06:42.397 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:41.153+0000 7f659bc00700 1 Processor -- start 2024-03-18T15:06:42.397 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:41.153+0000 7f659bc00700 1 -- start start 2024-03-18T15:06:42.397 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:41.153+0000 7f659bc00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f659410a0f0 0x7f659419ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:06:42.397 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:41.153+0000 7f659bc00700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f659410e300 con 0x7f659410a0f0 2024-03-18T15:06:42.398 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:41.154+0000 7f6599200700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f659410a0f0 0x7f659419ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:06:42.398 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:41.154+0000 7f6599200700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f659410a0f0 0x7f659419ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:39740/0 (socket says 172.21.15.19:39740) 2024-03-18T15:06:42.398 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:41.154+0000 7f6599200700 1 -- 172.21.15.19:0/3562333593 learned_addr learned my addr 172.21.15.19:0/3562333593 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:06:42.398 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:41.154+0000 7f6599200700 1 -- 172.21.15.19:0/3562333593 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f659419e410 con 0x7f659410a0f0 2024-03-18T15:06:42.398 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:41.155+0000 7f6599200700 1 --2- 172.21.15.19:0/3562333593 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f659410a0f0 0x7f659419ded0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f6588011b60 tx=0x7f6588011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:06:42.398 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:41.155+0000 7f6591600700 1 -- 172.21.15.19:0/3562333593 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6588003970 con 0x7f659410a0f0 2024-03-18T15:06:42.398 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:41.155+0000 7f6591600700 1 -- 172.21.15.19:0/3562333593 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f6588003b10 con 0x7f659410a0f0 2024-03-18T15:06:42.398 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:41.155+0000 7f6591600700 1 -- 172.21.15.19:0/3562333593 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6588024610 con 0x7f659410a0f0 2024-03-18T15:06:42.398 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:41.155+0000 7f659bc00700 1 -- 172.21.15.19:0/3562333593 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f659419e6a0 con 0x7f659410a0f0 2024-03-18T15:06:42.398 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:41.155+0000 7f659bc00700 1 -- 172.21.15.19:0/3562333593 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65941a1390 con 0x7f659410a0f0 2024-03-18T15:06:42.398 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:41.156+0000 7f6591600700 1 -- 172.21.15.19:0/3562333593 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f658801a9f0 con 0x7f659410a0f0 2024-03-18T15:06:42.398 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:41.156+0000 7f659bc00700 1 -- 172.21.15.19:0/3562333593 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f659410e480 con 0x7f659410a0f0 2024-03-18T15:06:42.398 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:41.157+0000 7f6591600700 1 --2- 172.21.15.19:0/3562333593 >> [v2:172.21.15.19:6800/2776008962,v1:172.21.15.19:6801/2776008962] conn(0x7f657c038f70 0x7f657c03b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:06:42.398 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:41.157+0000 7f6591600700 1 -- 172.21.15.19:0/3562333593 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f6588011e00 con 0x7f659410a0f0 2024-03-18T15:06:42.399 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:41.157+0000 7f6593e00700 1 --2- 172.21.15.19:0/3562333593 >> [v2:172.21.15.19:6800/2776008962,v1:172.21.15.19:6801/2776008962] conn(0x7f657c038f70 0x7f657c03b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:06:42.399 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:41.158+0000 7f6593e00700 1 --2- 172.21.15.19:0/3562333593 >> [v2:172.21.15.19:6800/2776008962,v1:172.21.15.19:6801/2776008962] conn(0x7f657c038f70 0x7f657c03b430 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6580006fd0 tx=0x7f6580006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:06:42.399 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:41.163+0000 7f6591600700 1 -- 172.21.15.19:0/3562333593 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6588019e10 con 0x7f659410a0f0 2024-03-18T15:06:42.399 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:41.407+0000 7f659bc00700 1 -- 172.21.15.19:0/3562333593 --> [v2:172.21.15.19:6800/2776008962,v1:172.21.15.19:6801/2776008962] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7f65941084d0 con 0x7f657c038f70 2024-03-18T15:06:42.399 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:41.417+0000 7f6591600700 1 -- 172.21.15.19:0/3562333593 <== mgr.14120 v2:172.21.15.19:6800/2776008962 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f65941084d0 con 0x7f657c038f70 2024-03-18T15:06:42.399 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:41.420+0000 7f659bc00700 1 -- 172.21.15.19:0/3562333593 >> [v2:172.21.15.19:6800/2776008962,v1:172.21.15.19:6801/2776008962] conn(0x7f657c038f70 msgr2=0x7f657c03b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:06:42.399 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:41.420+0000 7f659bc00700 1 --2- 172.21.15.19:0/3562333593 >> [v2:172.21.15.19:6800/2776008962,v1:172.21.15.19:6801/2776008962] conn(0x7f657c038f70 0x7f657c03b430 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6580006fd0 tx=0x7f6580006e40 comp rx=0 tx=0).stop 2024-03-18T15:06:42.399 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:41.420+0000 7f659bc00700 1 -- 172.21.15.19:0/3562333593 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f659410a0f0 msgr2=0x7f659419ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:06:42.399 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:41.420+0000 7f659bc00700 1 --2- 172.21.15.19:0/3562333593 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f659410a0f0 0x7f659419ded0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f6588011b60 tx=0x7f6588011c40 comp rx=0 tx=0).stop 2024-03-18T15:06:42.399 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:41.421+0000 7f659bc00700 1 -- 172.21.15.19:0/3562333593 shutdown_connections 2024-03-18T15:06:42.399 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:41.421+0000 7f659bc00700 1 --2- 172.21.15.19:0/3562333593 >> [v2:172.21.15.19:6800/2776008962,v1:172.21.15.19:6801/2776008962] conn(0x7f657c038f70 0x7f657c03b430 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:06:42.399 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:41.421+0000 7f659bc00700 1 --2- 172.21.15.19:0/3562333593 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f659410a0f0 0x7f659419ded0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:06:42.399 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:41.421+0000 7f659bc00700 1 -- 172.21.15.19:0/3562333593 >> 172.21.15.19:0/3562333593 conn(0x7f659407d2c0 msgr2=0x7f6594107dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:06:42.400 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:41.421+0000 7f659bc00700 1 -- 172.21.15.19:0/3562333593 shutdown_connections 2024-03-18T15:06:42.400 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:41.422+0000 7f659bc00700 1 -- 172.21.15.19:0/3562333593 wait complete. 2024-03-18T15:06:43.853 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:43 smithi019 ceph-mon[31522]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:06:43.857 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-18T15:06:43.857 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:42.993+0000 7f2ef2200700 1 Processor -- start 2024-03-18T15:06:43.857 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:42.993+0000 7f2ef2200700 1 -- start start 2024-03-18T15:06:43.857 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:42.994+0000 7f2ef2200700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2eec10a0f0 0x7f2eec10a510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:06:43.859 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:42.994+0000 7f2ef2200700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f2eec10aae0 con 0x7f2eec10a0f0 2024-03-18T15:06:43.859 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:42.994+0000 7f2eeb400700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2eec10a0f0 0x7f2eec10a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:06:43.859 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:42.994+0000 7f2eeb400700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2eec10a0f0 0x7f2eec10a510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:55820/0 (socket says 172.21.15.19:55820) 2024-03-18T15:06:43.859 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:42.994+0000 7f2eeb400700 1 -- 172.21.15.19:0/3324535663 learned_addr learned my addr 172.21.15.19:0/3324535663 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:06:43.859 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:42.995+0000 7f2eeb400700 1 -- 172.21.15.19:0/3324535663 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2eec10ac60 con 0x7f2eec10a0f0 2024-03-18T15:06:43.859 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:42.995+0000 7f2eeb400700 1 --2- 172.21.15.19:0/3324535663 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2eec10a0f0 0x7f2eec10a510 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f2edc009ab0 tx=0x7f2edc009dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa4fe57bbd8a9cbd server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:06:43.859 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:42.995+0000 7f2eeaa00700 1 -- 172.21.15.19:0/3324535663 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2edc004160 con 0x7f2eec10a0f0 2024-03-18T15:06:43.860 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:42.995+0000 7f2eeaa00700 1 -- 172.21.15.19:0/3324535663 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f2edc00bca0 con 0x7f2eec10a0f0 2024-03-18T15:06:43.860 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:42.995+0000 7f2eeaa00700 1 -- 172.21.15.19:0/3324535663 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2edc003d60 con 0x7f2eec10a0f0 2024-03-18T15:06:43.860 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:42.996+0000 7f2ef2200700 1 -- 172.21.15.19:0/3324535663 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2eec10a0f0 msgr2=0x7f2eec10a510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:06:43.860 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:42.996+0000 7f2ef2200700 1 --2- 172.21.15.19:0/3324535663 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2eec10a0f0 0x7f2eec10a510 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f2edc009ab0 tx=0x7f2edc009dc0 comp rx=0 tx=0).stop 2024-03-18T15:06:43.863 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:42.996+0000 7f2ef2200700 1 -- 172.21.15.19:0/3324535663 shutdown_connections 2024-03-18T15:06:43.863 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:42.996+0000 7f2ef2200700 1 --2- 172.21.15.19:0/3324535663 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2eec10a0f0 0x7f2eec10a510 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:06:43.863 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:42.996+0000 7f2ef2200700 1 -- 172.21.15.19:0/3324535663 >> 172.21.15.19:0/3324535663 conn(0x7f2eec07d2d0 msgr2=0x7f2eec07d6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:06:43.863 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:42.996+0000 7f2ef2200700 1 -- 172.21.15.19:0/3324535663 shutdown_connections 2024-03-18T15:06:43.864 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:42.996+0000 7f2ef2200700 1 -- 172.21.15.19:0/3324535663 wait complete. 2024-03-18T15:06:43.864 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:42.997+0000 7f2ef2200700 1 Processor -- start 2024-03-18T15:06:43.864 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:42.997+0000 7f2ef2200700 1 -- start start 2024-03-18T15:06:43.864 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:42.998+0000 7f2ef2200700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2eec10a0f0 0x7f2eec19df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:06:43.864 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:42.998+0000 7f2ef2200700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f2eec10e340 con 0x7f2eec10a0f0 2024-03-18T15:06:43.864 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:42.998+0000 7f2eeb400700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2eec10a0f0 0x7f2eec19df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:06:43.864 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:42.998+0000 7f2eeb400700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2eec10a0f0 0x7f2eec19df30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:55832/0 (socket says 172.21.15.19:55832) 2024-03-18T15:06:43.864 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:42.998+0000 7f2eeb400700 1 -- 172.21.15.19:0/3699533516 learned_addr learned my addr 172.21.15.19:0/3699533516 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:06:43.864 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:42.998+0000 7f2eeb400700 1 -- 172.21.15.19:0/3699533516 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2eec19e470 con 0x7f2eec10a0f0 2024-03-18T15:06:43.864 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:42.999+0000 7f2eeb400700 1 --2- 172.21.15.19:0/3699533516 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2eec10a0f0 0x7f2eec19df30 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f2edc003e00 tx=0x7f2edc003ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:06:43.864 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:42.999+0000 7f2ee8c00700 1 -- 172.21.15.19:0/3699533516 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2edc0036a0 con 0x7f2eec10a0f0 2024-03-18T15:06:43.864 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:42.999+0000 7f2ee8c00700 1 -- 172.21.15.19:0/3699533516 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f2edc00be50 con 0x7f2eec10a0f0 2024-03-18T15:06:43.864 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:42.999+0000 7f2ef2200700 1 -- 172.21.15.19:0/3699533516 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2eec19e700 con 0x7f2eec10a0f0 2024-03-18T15:06:43.865 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:42.999+0000 7f2ee8c00700 1 -- 172.21.15.19:0/3699533516 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2edc0115b0 con 0x7f2eec10a0f0 2024-03-18T15:06:43.865 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:42.999+0000 7f2ef2200700 1 -- 172.21.15.19:0/3699533516 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2eec1a13f0 con 0x7f2eec10a0f0 2024-03-18T15:06:43.865 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:43.000+0000 7f2ee8c00700 1 -- 172.21.15.19:0/3699533516 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f2edc01a430 con 0x7f2eec10a0f0 2024-03-18T15:06:43.865 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:43.000+0000 7f2ef2200700 1 -- 172.21.15.19:0/3699533516 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2eec10e4c0 con 0x7f2eec10a0f0 2024-03-18T15:06:43.865 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:43.001+0000 7f2ee8c00700 1 --2- 172.21.15.19:0/3699533516 >> [v2:172.21.15.19:6800/2776008962,v1:172.21.15.19:6801/2776008962] conn(0x7f2ed0038f20 0x7f2ed003b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:06:43.865 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:43.001+0000 7f2ee8c00700 1 -- 172.21.15.19:0/3699533516 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f2edc04d0a0 con 0x7f2eec10a0f0 2024-03-18T15:06:43.865 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:43.001+0000 7f2ee3e00700 1 --2- 172.21.15.19:0/3699533516 >> [v2:172.21.15.19:6800/2776008962,v1:172.21.15.19:6801/2776008962] conn(0x7f2ed0038f20 0x7f2ed003b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:06:43.865 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:43.002+0000 7f2ee3e00700 1 --2- 172.21.15.19:0/3699533516 >> [v2:172.21.15.19:6800/2776008962,v1:172.21.15.19:6801/2776008962] conn(0x7f2ed0038f20 0x7f2ed003b3e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2ed4006fd0 tx=0x7f2ed4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:06:43.865 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:43.006+0000 7f2ee8c00700 1 -- 172.21.15.19:0/3699533516 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2edc00bc60 con 0x7f2eec10a0f0 2024-03-18T15:06:43.865 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:43.232+0000 7f2ef2200700 1 -- 172.21.15.19:0/3699533516 --> [v2:172.21.15.19:6800/2776008962,v1:172.21.15.19:6801/2776008962] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7f2eec108500 con 0x7f2ed0038f20 2024-03-18T15:06:43.865 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:43.233+0000 7f2ee8c00700 1 -- 172.21.15.19:0/3699533516 <== mgr.14120 v2:172.21.15.19:6800/2776008962 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7f2eec108500 con 0x7f2ed0038f20 2024-03-18T15:06:43.865 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:43.236+0000 7f2ef2200700 1 -- 172.21.15.19:0/3699533516 >> [v2:172.21.15.19:6800/2776008962,v1:172.21.15.19:6801/2776008962] conn(0x7f2ed0038f20 msgr2=0x7f2ed003b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:06:43.865 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:43.236+0000 7f2ef2200700 1 --2- 172.21.15.19:0/3699533516 >> [v2:172.21.15.19:6800/2776008962,v1:172.21.15.19:6801/2776008962] conn(0x7f2ed0038f20 0x7f2ed003b3e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2ed4006fd0 tx=0x7f2ed4006e40 comp rx=0 tx=0).stop 2024-03-18T15:06:43.865 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:43.236+0000 7f2ef2200700 1 -- 172.21.15.19:0/3699533516 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2eec10a0f0 msgr2=0x7f2eec19df30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:06:43.867 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:43.236+0000 7f2ef2200700 1 --2- 172.21.15.19:0/3699533516 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2eec10a0f0 0x7f2eec19df30 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f2edc003e00 tx=0x7f2edc003ee0 comp rx=0 tx=0).stop 2024-03-18T15:06:43.867 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:43.236+0000 7f2ef2200700 1 -- 172.21.15.19:0/3699533516 shutdown_connections 2024-03-18T15:06:43.867 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:43.236+0000 7f2ef2200700 1 --2- 172.21.15.19:0/3699533516 >> [v2:172.21.15.19:6800/2776008962,v1:172.21.15.19:6801/2776008962] conn(0x7f2ed0038f20 0x7f2ed003b3e0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:06:43.867 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:43.236+0000 7f2ef2200700 1 --2- 172.21.15.19:0/3699533516 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2eec10a0f0 0x7f2eec19df30 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:06:43.867 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:43.236+0000 7f2ef2200700 1 -- 172.21.15.19:0/3699533516 >> 172.21.15.19:0/3699533516 conn(0x7f2eec07d2d0 msgr2=0x7f2eec107df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:06:43.867 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:43.237+0000 7f2ef2200700 1 -- 172.21.15.19:0/3699533516 shutdown_connections 2024-03-18T15:06:43.867 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:43.237+0000 7f2ef2200700 1 -- 172.21.15.19:0/3699533516 wait complete. 2024-03-18T15:06:43.867 INFO:teuthology.orchestra.run.smithi019.stdout:Generating ssh key... 2024-03-18T15:06:45.003 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:06:44 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: Generating public/private rsa key pair. 2024-03-18T15:06:45.003 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:06:44 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: Your identification has been saved in /tmp/tmp0gh0pfkm/key. 2024-03-18T15:06:45.003 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:06:44 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: Your public key has been saved in /tmp/tmp0gh0pfkm/key.pub. 2024-03-18T15:06:45.003 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:06:44 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: The key fingerprint is: 2024-03-18T15:06:45.003 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:06:44 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: SHA256:mpJIqCKE490pgKVqUQZCZAK64yYMQXmwfWTSpgJ3GQ0 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715 2024-03-18T15:06:45.003 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:06:44 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: The key's randomart image is: 2024-03-18T15:06:45.003 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:06:44 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: +---[RSA 3072]----+ 2024-03-18T15:06:45.003 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:06:44 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: |BBo.EB | 2024-03-18T15:06:45.003 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:06:44 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: |B.=.*o. | 2024-03-18T15:06:45.003 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:06:44 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: |+oo=o. | 2024-03-18T15:06:45.003 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:06:44 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: |oBo.. | 2024-03-18T15:06:45.004 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:06:44 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: |X+o S | 2024-03-18T15:06:45.004 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:06:44 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: |O++....o | 2024-03-18T15:06:45.004 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:06:44 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: |**.oooo | 2024-03-18T15:06:45.004 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:06:44 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: |* .. | 2024-03-18T15:06:45.004 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:06:44 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: | | 2024-03-18T15:06:45.004 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:06:44 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: +----[SHA256]-----+ 2024-03-18T15:06:45.575 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:44.507+0000 7fa580c00700 1 Processor -- start 2024-03-18T15:06:45.575 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:44.508+0000 7fa580c00700 1 -- start start 2024-03-18T15:06:45.575 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:44.508+0000 7fa580c00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa57c108760 0x7fa57c10ab50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:06:45.576 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:44.508+0000 7fa580c00700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fa57c07be80 con 0x7fa57c108760 2024-03-18T15:06:45.576 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:44.509+0000 7fa579e00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa57c108760 0x7fa57c10ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:06:45.576 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:44.509+0000 7fa579e00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa57c108760 0x7fa57c10ab50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:55846/0 (socket says 172.21.15.19:55846) 2024-03-18T15:06:45.576 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:44.509+0000 7fa579e00700 1 -- 172.21.15.19:0/1356897858 learned_addr learned my addr 172.21.15.19:0/1356897858 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:06:45.576 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:44.509+0000 7fa579e00700 1 -- 172.21.15.19:0/1356897858 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa57c10b090 con 0x7fa57c108760 2024-03-18T15:06:45.577 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:44.509+0000 7fa579e00700 1 --2- 172.21.15.19:0/1356897858 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa57c108760 0x7fa57c10ab50 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa56c0098a0 tx=0x7fa56c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb7bcbb56b0a86a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:06:45.577 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:44.510+0000 7fa579400700 1 -- 172.21.15.19:0/1356897858 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa56c004160 con 0x7fa57c108760 2024-03-18T15:06:45.578 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:44.510+0000 7fa579400700 1 -- 172.21.15.19:0/1356897858 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fa56c004300 con 0x7fa57c108760 2024-03-18T15:06:45.578 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:44.510+0000 7fa579400700 1 -- 172.21.15.19:0/1356897858 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa56c011b50 con 0x7fa57c108760 2024-03-18T15:06:45.578 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:44.510+0000 7fa580c00700 1 -- 172.21.15.19:0/1356897858 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa57c108760 msgr2=0x7fa57c10ab50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:06:45.578 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:44.510+0000 7fa580c00700 1 --2- 172.21.15.19:0/1356897858 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa57c108760 0x7fa57c10ab50 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa56c0098a0 tx=0x7fa56c009bb0 comp rx=0 tx=0).stop 2024-03-18T15:06:45.578 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:44.511+0000 7fa580c00700 1 -- 172.21.15.19:0/1356897858 shutdown_connections 2024-03-18T15:06:45.578 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:44.511+0000 7fa580c00700 1 --2- 172.21.15.19:0/1356897858 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa57c108760 0x7fa57c10ab50 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:06:45.578 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:44.511+0000 7fa580c00700 1 -- 172.21.15.19:0/1356897858 >> 172.21.15.19:0/1356897858 conn(0x7fa57c102130 msgr2=0x7fa57c1045b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:06:45.578 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:44.511+0000 7fa580c00700 1 -- 172.21.15.19:0/1356897858 shutdown_connections 2024-03-18T15:06:45.581 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:44.511+0000 7fa580c00700 1 -- 172.21.15.19:0/1356897858 wait complete. 2024-03-18T15:06:45.581 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:44.512+0000 7fa580c00700 1 Processor -- start 2024-03-18T15:06:45.581 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:44.512+0000 7fa580c00700 1 -- start start 2024-03-18T15:06:45.581 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:44.512+0000 7fa580c00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa57c108760 0x7fa57c19bc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:06:45.581 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:44.512+0000 7fa580c00700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fa57c10c040 con 0x7fa57c108760 2024-03-18T15:06:45.581 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:44.513+0000 7fa579e00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa57c108760 0x7fa57c19bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:06:45.581 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:44.513+0000 7fa579e00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa57c108760 0x7fa57c19bc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:55856/0 (socket says 172.21.15.19:55856) 2024-03-18T15:06:45.581 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:44.513+0000 7fa579e00700 1 -- 172.21.15.19:0/3438907158 learned_addr learned my addr 172.21.15.19:0/3438907158 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:06:45.581 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:44.513+0000 7fa579e00700 1 -- 172.21.15.19:0/3438907158 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa57c19c1d0 con 0x7fa57c108760 2024-03-18T15:06:45.581 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:44.513+0000 7fa579e00700 1 --2- 172.21.15.19:0/3438907158 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa57c108760 0x7fa57c19bc90 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa56c011b60 tx=0x7fa56c011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:06:45.582 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:44.513+0000 7fa573400700 1 -- 172.21.15.19:0/3438907158 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa56c003970 con 0x7fa57c108760 2024-03-18T15:06:45.582 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:44.514+0000 7fa573400700 1 -- 172.21.15.19:0/3438907158 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fa56c003b10 con 0x7fa57c108760 2024-03-18T15:06:45.582 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:44.514+0000 7fa580c00700 1 -- 172.21.15.19:0/3438907158 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa57c19c4c0 con 0x7fa57c108760 2024-03-18T15:06:45.582 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:44.514+0000 7fa573400700 1 -- 172.21.15.19:0/3438907158 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa56c024610 con 0x7fa57c108760 2024-03-18T15:06:45.582 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:44.514+0000 7fa580c00700 1 -- 172.21.15.19:0/3438907158 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa57c19f150 con 0x7fa57c108760 2024-03-18T15:06:45.582 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:44.515+0000 7fa580c00700 1 -- 172.21.15.19:0/3438907158 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa57c0693d0 con 0x7fa57c108760 2024-03-18T15:06:45.582 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:44.515+0000 7fa573400700 1 -- 172.21.15.19:0/3438907158 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7fa56c01a9f0 con 0x7fa57c108760 2024-03-18T15:06:45.582 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:44.515+0000 7fa573400700 1 --2- 172.21.15.19:0/3438907158 >> [v2:172.21.15.19:6800/2776008962,v1:172.21.15.19:6801/2776008962] conn(0x7fa560038f70 0x7fa56003b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:06:45.582 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:44.515+0000 7fa573400700 1 -- 172.21.15.19:0/3438907158 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fa56c011e00 con 0x7fa57c108760 2024-03-18T15:06:45.584 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:44.516+0000 7fa571400700 1 --2- 172.21.15.19:0/3438907158 >> [v2:172.21.15.19:6800/2776008962,v1:172.21.15.19:6801/2776008962] conn(0x7fa560038f70 0x7fa56003b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:06:45.584 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:44.516+0000 7fa571400700 1 --2- 172.21.15.19:0/3438907158 >> [v2:172.21.15.19:6800/2776008962,v1:172.21.15.19:6801/2776008962] conn(0x7fa560038f70 0x7fa56003b430 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fa564006fd0 tx=0x7fa564006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:06:45.584 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:44.521+0000 7fa573400700 1 -- 172.21.15.19:0/3438907158 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa56c02ce10 con 0x7fa57c108760 2024-03-18T15:06:45.584 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:44.748+0000 7fa580c00700 1 -- 172.21.15.19:0/3438907158 --> [v2:172.21.15.19:6800/2776008962,v1:172.21.15.19:6801/2776008962] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7fa57c19f320 con 0x7fa560038f70 2024-03-18T15:06:45.584 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:44.996+0000 7fa573400700 1 -- 172.21.15.19:0/3438907158 <== mgr.14120 v2:172.21.15.19:6800/2776008962 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fa57c19f320 con 0x7fa560038f70 2024-03-18T15:06:45.584 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:44.999+0000 7fa580c00700 1 -- 172.21.15.19:0/3438907158 >> [v2:172.21.15.19:6800/2776008962,v1:172.21.15.19:6801/2776008962] conn(0x7fa560038f70 msgr2=0x7fa56003b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:06:45.584 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:44.999+0000 7fa580c00700 1 --2- 172.21.15.19:0/3438907158 >> [v2:172.21.15.19:6800/2776008962,v1:172.21.15.19:6801/2776008962] conn(0x7fa560038f70 0x7fa56003b430 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fa564006fd0 tx=0x7fa564006e40 comp rx=0 tx=0).stop 2024-03-18T15:06:45.584 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:44.999+0000 7fa580c00700 1 -- 172.21.15.19:0/3438907158 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa57c108760 msgr2=0x7fa57c19bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:06:45.584 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:44.999+0000 7fa580c00700 1 --2- 172.21.15.19:0/3438907158 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa57c108760 0x7fa57c19bc90 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa56c011b60 tx=0x7fa56c011c40 comp rx=0 tx=0).stop 2024-03-18T15:06:45.585 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:44.999+0000 7fa580c00700 1 -- 172.21.15.19:0/3438907158 shutdown_connections 2024-03-18T15:06:45.585 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:44.999+0000 7fa580c00700 1 --2- 172.21.15.19:0/3438907158 >> [v2:172.21.15.19:6800/2776008962,v1:172.21.15.19:6801/2776008962] conn(0x7fa560038f70 0x7fa56003b430 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:06:45.585 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:44.999+0000 7fa580c00700 1 --2- 172.21.15.19:0/3438907158 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa57c108760 0x7fa57c19bc90 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:06:45.585 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:44.999+0000 7fa580c00700 1 -- 172.21.15.19:0/3438907158 >> 172.21.15.19:0/3438907158 conn(0x7fa57c102130 msgr2=0x7fa57c102d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:06:45.585 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:45.000+0000 7fa580c00700 1 -- 172.21.15.19:0/3438907158 shutdown_connections 2024-03-18T15:06:45.585 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:45.000+0000 7fa580c00700 1 -- 172.21.15.19:0/3438907158 wait complete. 2024-03-18T15:06:46.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:45 smithi019 ceph-mon[31522]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:06:46.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:45 smithi019 ceph-mon[31522]: Generating ssh key... 2024-03-18T15:06:46.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:45 smithi019 ceph-mon[31522]: from='mgr.14120 172.21.15.19:0/2858628912' entity='mgr.x' 2024-03-18T15:06:46.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:45 smithi019 ceph-mon[31522]: from='mgr.14120 172.21.15.19:0/2858628912' entity='mgr.x' 2024-03-18T15:06:47.168 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:46 smithi019 ceph-mon[31522]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:06:47.182 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpvnxZ8ElXr5YJcsABYs3kNNr8JrxvGWZ0cwwo6ZISbL3BG3qVGSQ91ZY1Ke7RvL7DotwbdBASeuMZdl5Ib4WpEA+Z6sBKzjnULIWjYR+obO9epR2boWPqzQeb2Gq5P5t3cvNLjvPMbKcqsbL83FrRdRedXjOSOGZTmfKd6hN7XO8JlOp14rBmwj09gQ5WkWq4WKL+8S5S90mM99A1MauQQ/aeaGfNtFEplnm5uq9ygbKLWH6kGDXmcnISZqTmjUvQ4Z4ctJa+RtLl7w4e62oieSKDYZ8JJ3KBJv+jsU1YVyJZ5UMZ+kiconP/IBgXhPDgFnkoRnMNT569N5LckirlnZri046uGnvGj7yEY7GIP6nDKTcbeLehP8GH+S6WTeFyl5+p34ygU6VI3uJw8GzBHVqsvTvea8kVA0wlHNWx4dutGafWfqzjp1z3hEz0hdlxnKaS+6/3LoQY7fhrcS6uSqN7DiLuMn+3onzVjcsRndIJvYeLlBaFmoTSNkwI9Bs= ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715 2024-03-18T15:06:47.182 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:46.231+0000 7f6bad200700 1 Processor -- start 2024-03-18T15:06:47.182 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:46.232+0000 7f6bad200700 1 -- start start 2024-03-18T15:06:47.182 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:46.232+0000 7f6bad200700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6ba8109f30 0x7f6ba810a350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:06:47.182 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:46.232+0000 7f6bad200700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f6ba810a890 con 0x7f6ba8109f30 2024-03-18T15:06:47.182 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:46.233+0000 7f6ba5e00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6ba8109f30 0x7f6ba810a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:06:47.182 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:46.233+0000 7f6ba5e00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6ba8109f30 0x7f6ba810a350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:55858/0 (socket says 172.21.15.19:55858) 2024-03-18T15:06:47.183 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:46.233+0000 7f6ba5e00700 1 -- 172.21.15.19:0/237021510 learned_addr learned my addr 172.21.15.19:0/237021510 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:06:47.183 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:46.233+0000 7f6ba5e00700 1 -- 172.21.15.19:0/237021510 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ba810b090 con 0x7f6ba8109f30 2024-03-18T15:06:47.183 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:46.234+0000 7f6ba5e00700 1 --2- 172.21.15.19:0/237021510 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6ba8109f30 0x7f6ba810a350 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f6b98009e80 tx=0x7f6b9800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a6f4d5fd8c64395 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:06:47.183 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:46.234+0000 7f6ba5400700 1 -- 172.21.15.19:0/237021510 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6b98017070 con 0x7f6ba8109f30 2024-03-18T15:06:47.183 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:46.234+0000 7f6ba5400700 1 -- 172.21.15.19:0/237021510 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f6b9800b8c0 con 0x7f6ba8109f30 2024-03-18T15:06:47.183 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:46.234+0000 7f6ba5400700 1 -- 172.21.15.19:0/237021510 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6b980045c0 con 0x7f6ba8109f30 2024-03-18T15:06:47.183 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:46.234+0000 7f6bad200700 1 -- 172.21.15.19:0/237021510 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6ba8109f30 msgr2=0x7f6ba810a350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:06:47.183 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:46.234+0000 7f6bad200700 1 --2- 172.21.15.19:0/237021510 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6ba8109f30 0x7f6ba810a350 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f6b98009e80 tx=0x7f6b9800b330 comp rx=0 tx=0).stop 2024-03-18T15:06:47.183 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:46.235+0000 7f6bad200700 1 -- 172.21.15.19:0/237021510 shutdown_connections 2024-03-18T15:06:47.183 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:46.235+0000 7f6bad200700 1 --2- 172.21.15.19:0/237021510 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6ba8109f30 0x7f6ba810a350 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:06:47.183 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:46.235+0000 7f6bad200700 1 -- 172.21.15.19:0/237021510 >> 172.21.15.19:0/237021510 conn(0x7f6ba807d320 msgr2=0x7f6ba807d730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:06:47.183 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:46.235+0000 7f6bad200700 1 -- 172.21.15.19:0/237021510 shutdown_connections 2024-03-18T15:06:47.183 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:46.235+0000 7f6bad200700 1 -- 172.21.15.19:0/237021510 wait complete. 2024-03-18T15:06:47.184 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:46.236+0000 7f6bad200700 1 Processor -- start 2024-03-18T15:06:47.184 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:46.236+0000 7f6bad200700 1 -- start start 2024-03-18T15:06:47.184 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:46.237+0000 7f6bad200700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6ba8109f30 0x7f6ba819de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:06:47.184 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:46.237+0000 7f6bad200700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f6ba810e350 con 0x7f6ba8109f30 2024-03-18T15:06:47.184 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:46.237+0000 7f6ba5e00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6ba8109f30 0x7f6ba819de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:06:47.184 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:46.237+0000 7f6ba5e00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6ba8109f30 0x7f6ba819de60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:55868/0 (socket says 172.21.15.19:55868) 2024-03-18T15:06:47.184 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:46.237+0000 7f6ba5e00700 1 -- 172.21.15.19:0/4065954105 learned_addr learned my addr 172.21.15.19:0/4065954105 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:06:47.184 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:46.237+0000 7f6ba5e00700 1 -- 172.21.15.19:0/4065954105 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ba819e3a0 con 0x7f6ba8109f30 2024-03-18T15:06:47.184 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:46.238+0000 7f6ba5e00700 1 --2- 172.21.15.19:0/4065954105 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6ba8109f30 0x7f6ba819de60 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f6b98009e50 tx=0x7f6b9800bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:06:47.184 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:46.238+0000 7f6b9f400700 1 -- 172.21.15.19:0/4065954105 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6b98020020 con 0x7f6ba8109f30 2024-03-18T15:06:47.184 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:46.238+0000 7f6b9f400700 1 -- 172.21.15.19:0/4065954105 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f6b98003b50 con 0x7f6ba8109f30 2024-03-18T15:06:47.184 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:46.238+0000 7f6b9f400700 1 -- 172.21.15.19:0/4065954105 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6b98017040 con 0x7f6ba8109f30 2024-03-18T15:06:47.184 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:46.238+0000 7f6bad200700 1 -- 172.21.15.19:0/4065954105 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ba819e630 con 0x7f6ba8109f30 2024-03-18T15:06:47.185 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:46.238+0000 7f6bad200700 1 -- 172.21.15.19:0/4065954105 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ba81a1320 con 0x7f6ba8109f30 2024-03-18T15:06:47.185 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:46.239+0000 7f6b9f400700 1 -- 172.21.15.19:0/4065954105 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f6b98023020 con 0x7f6ba8109f30 2024-03-18T15:06:47.185 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:46.239+0000 7f6bad200700 1 -- 172.21.15.19:0/4065954105 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ba810e4d0 con 0x7f6ba8109f30 2024-03-18T15:06:47.185 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:46.240+0000 7f6b9f400700 1 --2- 172.21.15.19:0/4065954105 >> [v2:172.21.15.19:6800/2776008962,v1:172.21.15.19:6801/2776008962] conn(0x7f6b8c038f20 0x7f6b8c03b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:06:47.185 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:46.240+0000 7f6b9f400700 1 -- 172.21.15.19:0/4065954105 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f6b9800f050 con 0x7f6ba8109f30 2024-03-18T15:06:47.185 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:46.240+0000 7f6b9d400700 1 --2- 172.21.15.19:0/4065954105 >> [v2:172.21.15.19:6800/2776008962,v1:172.21.15.19:6801/2776008962] conn(0x7f6b8c038f20 0x7f6b8c03b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:06:47.185 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:46.241+0000 7f6b9d400700 1 --2- 172.21.15.19:0/4065954105 >> [v2:172.21.15.19:6800/2776008962,v1:172.21.15.19:6801/2776008962] conn(0x7f6b8c038f20 0x7f6b8c03b3e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6b90006fd0 tx=0x7f6b90006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:06:47.185 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:46.245+0000 7f6b9f400700 1 -- 172.21.15.19:0/4065954105 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6b9802a020 con 0x7f6ba8109f30 2024-03-18T15:06:47.185 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:46.470+0000 7f6bad200700 1 -- 172.21.15.19:0/4065954105 --> [v2:172.21.15.19:6800/2776008962,v1:172.21.15.19:6801/2776008962] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7f6ba8107ba0 con 0x7f6b8c038f20 2024-03-18T15:06:47.185 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:46.472+0000 7f6b9f400700 1 -- 172.21.15.19:0/4065954105 <== mgr.14120 v2:172.21.15.19:6800/2776008962 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7f6ba8107ba0 con 0x7f6b8c038f20 2024-03-18T15:06:47.185 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:46.475+0000 7f6bad200700 1 -- 172.21.15.19:0/4065954105 >> [v2:172.21.15.19:6800/2776008962,v1:172.21.15.19:6801/2776008962] conn(0x7f6b8c038f20 msgr2=0x7f6b8c03b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:06:47.186 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:46.475+0000 7f6bad200700 1 --2- 172.21.15.19:0/4065954105 >> [v2:172.21.15.19:6800/2776008962,v1:172.21.15.19:6801/2776008962] conn(0x7f6b8c038f20 0x7f6b8c03b3e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6b90006fd0 tx=0x7f6b90006e40 comp rx=0 tx=0).stop 2024-03-18T15:06:47.186 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:46.475+0000 7f6bad200700 1 -- 172.21.15.19:0/4065954105 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6ba8109f30 msgr2=0x7f6ba819de60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:06:47.186 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:46.475+0000 7f6bad200700 1 --2- 172.21.15.19:0/4065954105 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6ba8109f30 0x7f6ba819de60 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f6b98009e50 tx=0x7f6b9800bd30 comp rx=0 tx=0).stop 2024-03-18T15:06:47.186 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:46.475+0000 7f6bad200700 1 -- 172.21.15.19:0/4065954105 shutdown_connections 2024-03-18T15:06:47.186 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:46.475+0000 7f6bad200700 1 --2- 172.21.15.19:0/4065954105 >> [v2:172.21.15.19:6800/2776008962,v1:172.21.15.19:6801/2776008962] conn(0x7f6b8c038f20 0x7f6b8c03b3e0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:06:47.186 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:46.475+0000 7f6bad200700 1 --2- 172.21.15.19:0/4065954105 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6ba8109f30 0x7f6ba819de60 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:06:47.186 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:46.475+0000 7f6bad200700 1 -- 172.21.15.19:0/4065954105 >> 172.21.15.19:0/4065954105 conn(0x7f6ba807d320 msgr2=0x7f6ba8107490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:06:47.186 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:46.475+0000 7f6bad200700 1 -- 172.21.15.19:0/4065954105 shutdown_connections 2024-03-18T15:06:47.186 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:46.475+0000 7f6bad200700 1 -- 172.21.15.19:0/4065954105 wait complete. 2024-03-18T15:06:47.186 INFO:teuthology.orchestra.run.smithi019.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-18T15:06:47.186 INFO:teuthology.orchestra.run.smithi019.stdout:Adding key to root@localhost authorized_keys... 2024-03-18T15:06:47.187 INFO:teuthology.orchestra.run.smithi019.stdout:Adding host smithi019... 2024-03-18T15:06:48.927 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:48 smithi019 ceph-mon[31522]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi019", "addr": "172.21.15.19", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:06:49.905 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:49 smithi019 ceph-mon[31522]: Deploying cephadm binary to smithi019 2024-03-18T15:06:55.457 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout Added host 'smithi019' with addr '172.21.15.19' 2024-03-18T15:06:55.457 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:47.750+0000 7f6a97e00700 1 Processor -- start 2024-03-18T15:06:55.457 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:47.750+0000 7f6a97e00700 1 -- start start 2024-03-18T15:06:55.457 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:47.751+0000 7f6a97e00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6a98107af0 0x7f6a98107f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:06:55.457 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:47.751+0000 7f6a97e00700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f6a981084e0 con 0x7f6a98107af0 2024-03-18T15:06:55.457 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:47.751+0000 7f6a96a00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6a98107af0 0x7f6a98107f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:06:55.457 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:47.751+0000 7f6a96a00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6a98107af0 0x7f6a98107f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:55884/0 (socket says 172.21.15.19:55884) 2024-03-18T15:06:55.457 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:47.751+0000 7f6a96a00700 1 -- 172.21.15.19:0/1080048804 learned_addr learned my addr 172.21.15.19:0/1080048804 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:06:55.457 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:47.752+0000 7f6a96a00700 1 -- 172.21.15.19:0/1080048804 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a98108cf0 con 0x7f6a98107af0 2024-03-18T15:06:55.457 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:47.753+0000 7f6a96a00700 1 --2- 172.21.15.19:0/1080048804 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6a98107af0 0x7f6a98107f10 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f6a880098a0 tx=0x7f6a88009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce3b775a8b4733b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:06:55.458 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:47.753+0000 7f6a95600700 1 -- 172.21.15.19:0/1080048804 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6a88004160 con 0x7f6a98107af0 2024-03-18T15:06:55.458 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:47.753+0000 7f6a95600700 1 -- 172.21.15.19:0/1080048804 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f6a88004300 con 0x7f6a98107af0 2024-03-18T15:06:55.458 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:47.753+0000 7f6a95600700 1 -- 172.21.15.19:0/1080048804 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6a88011b50 con 0x7f6a98107af0 2024-03-18T15:06:55.458 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:47.754+0000 7f6a97e00700 1 -- 172.21.15.19:0/1080048804 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6a98107af0 msgr2=0x7f6a98107f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:06:55.458 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:47.754+0000 7f6a97e00700 1 --2- 172.21.15.19:0/1080048804 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6a98107af0 0x7f6a98107f10 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f6a880098a0 tx=0x7f6a88009bb0 comp rx=0 tx=0).stop 2024-03-18T15:06:55.458 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:47.754+0000 7f6a97e00700 1 -- 172.21.15.19:0/1080048804 shutdown_connections 2024-03-18T15:06:55.458 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:47.754+0000 7f6a97e00700 1 --2- 172.21.15.19:0/1080048804 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6a98107af0 0x7f6a98107f10 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:06:55.458 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:47.754+0000 7f6a97e00700 1 -- 172.21.15.19:0/1080048804 >> 172.21.15.19:0/1080048804 conn(0x7f6a98103090 msgr2=0x7f6a981054d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:06:55.458 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:47.754+0000 7f6a97e00700 1 -- 172.21.15.19:0/1080048804 shutdown_connections 2024-03-18T15:06:55.458 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:47.754+0000 7f6a97e00700 1 -- 172.21.15.19:0/1080048804 wait complete. 2024-03-18T15:06:55.458 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:47.755+0000 7f6a97e00700 1 Processor -- start 2024-03-18T15:06:55.458 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:47.756+0000 7f6a97e00700 1 -- start start 2024-03-18T15:06:55.458 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:47.756+0000 7f6a97e00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6a98107af0 0x7f6a9819dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:06:55.459 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:47.756+0000 7f6a97e00700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f6a9810c0a0 con 0x7f6a98107af0 2024-03-18T15:06:55.459 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:47.756+0000 7f6a96a00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6a98107af0 0x7f6a9819dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:06:55.459 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:47.756+0000 7f6a96a00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6a98107af0 0x7f6a9819dd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:55896/0 (socket says 172.21.15.19:55896) 2024-03-18T15:06:55.459 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:47.756+0000 7f6a96a00700 1 -- 172.21.15.19:0/2097505420 learned_addr learned my addr 172.21.15.19:0/2097505420 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:06:55.459 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:47.757+0000 7f6a96a00700 1 -- 172.21.15.19:0/2097505420 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a9819e2d0 con 0x7f6a98107af0 2024-03-18T15:06:55.459 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:47.757+0000 7f6a96a00700 1 --2- 172.21.15.19:0/2097505420 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6a98107af0 0x7f6a9819dd90 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6a88011850 tx=0x7f6a88011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:06:55.459 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:47.757+0000 7f6a8f400700 1 -- 172.21.15.19:0/2097505420 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6a88003af0 con 0x7f6a98107af0 2024-03-18T15:06:55.459 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:47.757+0000 7f6a8f400700 1 -- 172.21.15.19:0/2097505420 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f6a88020070 con 0x7f6a98107af0 2024-03-18T15:06:55.459 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:47.757+0000 7f6a97e00700 1 -- 172.21.15.19:0/2097505420 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a9819e560 con 0x7f6a98107af0 2024-03-18T15:06:55.459 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:47.757+0000 7f6a8f400700 1 -- 172.21.15.19:0/2097505420 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6a880195a0 con 0x7f6a98107af0 2024-03-18T15:06:55.459 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:47.758+0000 7f6a97e00700 1 -- 172.21.15.19:0/2097505420 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a981a1210 con 0x7f6a98107af0 2024-03-18T15:06:55.459 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:47.759+0000 7f6a8f400700 1 -- 172.21.15.19:0/2097505420 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f6a8801a430 con 0x7f6a98107af0 2024-03-18T15:06:55.459 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:47.759+0000 7f6a97e00700 1 -- 172.21.15.19:0/2097505420 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a980693d0 con 0x7f6a98107af0 2024-03-18T15:06:55.459 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:47.759+0000 7f6a8f400700 1 --2- 172.21.15.19:0/2097505420 >> [v2:172.21.15.19:6800/2776008962,v1:172.21.15.19:6801/2776008962] conn(0x7f6a7c038f70 0x7f6a7c03b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:06:55.460 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:47.759+0000 7f6a8f400700 1 -- 172.21.15.19:0/2097505420 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f6a88011e00 con 0x7f6a98107af0 2024-03-18T15:06:55.460 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:47.760+0000 7f6a96000700 1 --2- 172.21.15.19:0/2097505420 >> [v2:172.21.15.19:6800/2776008962,v1:172.21.15.19:6801/2776008962] conn(0x7f6a7c038f70 0x7f6a7c03b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:06:55.460 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:47.760+0000 7f6a96000700 1 --2- 172.21.15.19:0/2097505420 >> [v2:172.21.15.19:6800/2776008962,v1:172.21.15.19:6801/2776008962] conn(0x7f6a7c038f70 0x7f6a7c03b430 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f6a84006fd0 tx=0x7f6a84006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:06:55.460 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:47.765+0000 7f6a8f400700 1 -- 172.21.15.19:0/2097505420 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6a88021270 con 0x7f6a98107af0 2024-03-18T15:06:55.460 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:47.996+0000 7f6a97e00700 1 -- 172.21.15.19:0/2097505420 --> [v2:172.21.15.19:6800/2776008962,v1:172.21.15.19:6801/2776008962] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi019", "addr": "172.21.15.19", "target": ["mon-mgr", ""]}) -- 0x7f6a981042c0 con 0x7f6a7c038f70 2024-03-18T15:06:55.460 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:54.711+0000 7f6a8f400700 1 -- 172.21.15.19:0/2097505420 <== mgr.14120 v2:172.21.15.19:6800/2776008962 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7f6a981042c0 con 0x7f6a7c038f70 2024-03-18T15:06:55.460 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:54.715+0000 7f6a97e00700 1 -- 172.21.15.19:0/2097505420 >> [v2:172.21.15.19:6800/2776008962,v1:172.21.15.19:6801/2776008962] conn(0x7f6a7c038f70 msgr2=0x7f6a7c03b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:06:55.460 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:54.715+0000 7f6a97e00700 1 --2- 172.21.15.19:0/2097505420 >> [v2:172.21.15.19:6800/2776008962,v1:172.21.15.19:6801/2776008962] conn(0x7f6a7c038f70 0x7f6a7c03b430 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f6a84006fd0 tx=0x7f6a84006e40 comp rx=0 tx=0).stop 2024-03-18T15:06:55.460 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:54.715+0000 7f6a97e00700 1 -- 172.21.15.19:0/2097505420 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6a98107af0 msgr2=0x7f6a9819dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:06:55.460 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:54.715+0000 7f6a97e00700 1 --2- 172.21.15.19:0/2097505420 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6a98107af0 0x7f6a9819dd90 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6a88011850 tx=0x7f6a88011c10 comp rx=0 tx=0).stop 2024-03-18T15:06:55.460 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:54.715+0000 7f6a97e00700 1 -- 172.21.15.19:0/2097505420 shutdown_connections 2024-03-18T15:06:55.460 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:54.715+0000 7f6a97e00700 1 --2- 172.21.15.19:0/2097505420 >> [v2:172.21.15.19:6800/2776008962,v1:172.21.15.19:6801/2776008962] conn(0x7f6a7c038f70 0x7f6a7c03b430 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:06:55.461 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:54.715+0000 7f6a97e00700 1 --2- 172.21.15.19:0/2097505420 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6a98107af0 0x7f6a9819dd90 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:06:55.461 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:54.715+0000 7f6a97e00700 1 -- 172.21.15.19:0/2097505420 >> 172.21.15.19:0/2097505420 conn(0x7f6a98103090 msgr2=0x7f6a98103bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:06:55.461 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:54.716+0000 7f6a97e00700 1 -- 172.21.15.19:0/2097505420 shutdown_connections 2024-03-18T15:06:55.461 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:54.716+0000 7f6a97e00700 1 -- 172.21.15.19:0/2097505420 wait complete. 2024-03-18T15:06:55.461 INFO:teuthology.orchestra.run.smithi019.stdout:Deploying unmanaged mon service... 2024-03-18T15:06:55.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:55 smithi019 ceph-mon[31522]: from='mgr.14120 172.21.15.19:0/2858628912' entity='mgr.x' 2024-03-18T15:06:55.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:55 smithi019 ceph-mon[31522]: Added host smithi019 2024-03-18T15:06:55.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:55 smithi019 ceph-mon[31522]: from='mgr.14120 172.21.15.19:0/2858628912' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:06:57.426 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:57 smithi019 ceph-mon[31522]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:06:57.426 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:57 smithi019 ceph-mon[31522]: Saving service mon spec with placement count:5 2024-03-18T15:06:57.426 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:57 smithi019 ceph-mon[31522]: from='mgr.14120 172.21.15.19:0/2858628912' entity='mgr.x' 2024-03-18T15:06:58.149 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-18T15:06:58.149 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:56.042+0000 7f3206a00700 1 Processor -- start 2024-03-18T15:06:58.149 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:56.042+0000 7f3206a00700 1 -- start start 2024-03-18T15:06:58.150 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:56.043+0000 7f3206a00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3200106bc0 0x7f3200106fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:06:58.150 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:56.043+0000 7f3206a00700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f32001075b0 con 0x7f3200106bc0 2024-03-18T15:06:58.150 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:56.043+0000 7f31ffe00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3200106bc0 0x7f3200106fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:06:58.150 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:56.043+0000 7f31ffe00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3200106bc0 0x7f3200106fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:40318/0 (socket says 172.21.15.19:40318) 2024-03-18T15:06:58.150 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:56.043+0000 7f31ffe00700 1 -- 172.21.15.19:0/1196134934 learned_addr learned my addr 172.21.15.19:0/1196134934 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:06:58.150 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:56.044+0000 7f31ffe00700 1 -- 172.21.15.19:0/1196134934 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3200107dc0 con 0x7f3200106bc0 2024-03-18T15:06:58.150 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:56.044+0000 7f31ffe00700 1 --2- 172.21.15.19:0/1196134934 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3200106bc0 0x7f3200106fe0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f31f00098a0 tx=0x7f31f0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0ec88aa2e524867 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:06:58.150 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:56.044+0000 7f31ff400700 1 -- 172.21.15.19:0/1196134934 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f31f0004160 con 0x7f3200106bc0 2024-03-18T15:06:58.150 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:56.045+0000 7f31ff400700 1 -- 172.21.15.19:0/1196134934 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f31f0004300 con 0x7f3200106bc0 2024-03-18T15:06:58.150 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:56.045+0000 7f31ff400700 1 -- 172.21.15.19:0/1196134934 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f31f0011b50 con 0x7f3200106bc0 2024-03-18T15:06:58.150 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:56.045+0000 7f3206a00700 1 -- 172.21.15.19:0/1196134934 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3200106bc0 msgr2=0x7f3200106fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:06:58.150 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:56.045+0000 7f3206a00700 1 --2- 172.21.15.19:0/1196134934 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3200106bc0 0x7f3200106fe0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f31f00098a0 tx=0x7f31f0009bb0 comp rx=0 tx=0).stop 2024-03-18T15:06:58.150 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:56.045+0000 7f3206a00700 1 -- 172.21.15.19:0/1196134934 shutdown_connections 2024-03-18T15:06:58.155 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:56.045+0000 7f3206a00700 1 --2- 172.21.15.19:0/1196134934 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3200106bc0 0x7f3200106fe0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:06:58.155 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:56.045+0000 7f3206a00700 1 -- 172.21.15.19:0/1196134934 >> 172.21.15.19:0/1196134934 conn(0x7f3200102140 msgr2=0x7f32001045a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:06:58.155 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:56.046+0000 7f3206a00700 1 -- 172.21.15.19:0/1196134934 shutdown_connections 2024-03-18T15:06:58.155 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:56.046+0000 7f3206a00700 1 -- 172.21.15.19:0/1196134934 wait complete. 2024-03-18T15:06:58.155 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:56.047+0000 7f3206a00700 1 Processor -- start 2024-03-18T15:06:58.155 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:56.047+0000 7f3206a00700 1 -- start start 2024-03-18T15:06:58.155 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:56.047+0000 7f3206a00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3200106bc0 0x7f320019dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:06:58.155 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:56.047+0000 7f3206a00700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f320010b170 con 0x7f3200106bc0 2024-03-18T15:06:58.155 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:56.048+0000 7f31ffe00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3200106bc0 0x7f320019dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:06:58.155 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:56.048+0000 7f31ffe00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3200106bc0 0x7f320019dec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:40334/0 (socket says 172.21.15.19:40334) 2024-03-18T15:06:58.155 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:56.048+0000 7f31ffe00700 1 -- 172.21.15.19:0/941026224 learned_addr learned my addr 172.21.15.19:0/941026224 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:06:58.155 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:56.048+0000 7f31ffe00700 1 -- 172.21.15.19:0/941026224 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f320019e400 con 0x7f3200106bc0 2024-03-18T15:06:58.156 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:56.048+0000 7f31ffe00700 1 --2- 172.21.15.19:0/941026224 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3200106bc0 0x7f320019dec0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f31f0011670 tx=0x7f31f0011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:06:58.156 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:56.049+0000 7f31fd600700 1 -- 172.21.15.19:0/941026224 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f31f00039c0 con 0x7f3200106bc0 2024-03-18T15:06:58.156 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:56.049+0000 7f31fd600700 1 -- 172.21.15.19:0/941026224 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f31f0011e50 con 0x7f3200106bc0 2024-03-18T15:06:58.156 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:56.049+0000 7f31fd600700 1 -- 172.21.15.19:0/941026224 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f31f001a5c0 con 0x7f3200106bc0 2024-03-18T15:06:58.156 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:56.049+0000 7f3206a00700 1 -- 172.21.15.19:0/941026224 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f320019e690 con 0x7f3200106bc0 2024-03-18T15:06:58.156 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:56.049+0000 7f3206a00700 1 -- 172.21.15.19:0/941026224 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32001a1380 con 0x7f3200106bc0 2024-03-18T15:06:58.156 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:56.050+0000 7f31fd600700 1 -- 172.21.15.19:0/941026224 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f31f0024430 con 0x7f3200106bc0 2024-03-18T15:06:58.156 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:56.050+0000 7f3206a00700 1 -- 172.21.15.19:0/941026224 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3200063900 con 0x7f3200106bc0 2024-03-18T15:06:58.156 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:56.050+0000 7f31fd600700 1 --2- 172.21.15.19:0/941026224 >> [v2:172.21.15.19:6800/2776008962,v1:172.21.15.19:6801/2776008962] conn(0x7f31e4038f20 0x7f31e403b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:06:58.156 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:56.050+0000 7f31fd600700 1 -- 172.21.15.19:0/941026224 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f31f004d210 con 0x7f3200106bc0 2024-03-18T15:06:58.156 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:56.051+0000 7f31f7400700 1 --2- 172.21.15.19:0/941026224 >> [v2:172.21.15.19:6800/2776008962,v1:172.21.15.19:6801/2776008962] conn(0x7f31e4038f20 0x7f31e403b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:06:58.156 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:56.052+0000 7f31f7400700 1 --2- 172.21.15.19:0/941026224 >> [v2:172.21.15.19:6800/2776008962,v1:172.21.15.19:6801/2776008962] conn(0x7f31e4038f20 0x7f31e403b3e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f31e800ad30 tx=0x7f31e80093f0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:06:58.156 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:56.056+0000 7f31fd600700 1 -- 172.21.15.19:0/941026224 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f31f0017ca0 con 0x7f3200106bc0 2024-03-18T15:06:58.156 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:56.307+0000 7f3206a00700 1 -- 172.21.15.19:0/941026224 --> [v2:172.21.15.19:6800/2776008962,v1:172.21.15.19:6801/2776008962] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f32000008d0 con 0x7f31e4038f20 2024-03-18T15:06:58.157 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:56.315+0000 7f31fd600700 1 -- 172.21.15.19:0/941026224 <== mgr.14120 v2:172.21.15.19:6800/2776008962 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f32000008d0 con 0x7f31e4038f20 2024-03-18T15:06:58.157 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:56.318+0000 7f3206a00700 1 -- 172.21.15.19:0/941026224 >> [v2:172.21.15.19:6800/2776008962,v1:172.21.15.19:6801/2776008962] conn(0x7f31e4038f20 msgr2=0x7f31e403b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:06:58.157 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:56.318+0000 7f3206a00700 1 --2- 172.21.15.19:0/941026224 >> [v2:172.21.15.19:6800/2776008962,v1:172.21.15.19:6801/2776008962] conn(0x7f31e4038f20 0x7f31e403b3e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f31e800ad30 tx=0x7f31e80093f0 comp rx=0 tx=0).stop 2024-03-18T15:06:58.157 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:56.318+0000 7f3206a00700 1 -- 172.21.15.19:0/941026224 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3200106bc0 msgr2=0x7f320019dec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:06:58.157 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:56.318+0000 7f3206a00700 1 --2- 172.21.15.19:0/941026224 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3200106bc0 0x7f320019dec0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f31f0011670 tx=0x7f31f0011b60 comp rx=0 tx=0).stop 2024-03-18T15:06:58.157 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:56.319+0000 7f3206a00700 1 -- 172.21.15.19:0/941026224 shutdown_connections 2024-03-18T15:06:58.157 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:56.319+0000 7f3206a00700 1 --2- 172.21.15.19:0/941026224 >> [v2:172.21.15.19:6800/2776008962,v1:172.21.15.19:6801/2776008962] conn(0x7f31e4038f20 0x7f31e403b3e0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:06:58.157 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:56.319+0000 7f3206a00700 1 --2- 172.21.15.19:0/941026224 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3200106bc0 0x7f320019dec0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:06:58.157 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:56.319+0000 7f3206a00700 1 -- 172.21.15.19:0/941026224 >> 172.21.15.19:0/941026224 conn(0x7f3200102140 msgr2=0x7f3200103050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:06:58.157 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:56.319+0000 7f3206a00700 1 -- 172.21.15.19:0/941026224 shutdown_connections 2024-03-18T15:06:58.157 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:56.319+0000 7f3206a00700 1 -- 172.21.15.19:0/941026224 wait complete. 2024-03-18T15:06:58.157 INFO:teuthology.orchestra.run.smithi019.stdout:Deploying unmanaged mgr service... 2024-03-18T15:06:58.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:58 smithi019 ceph-mon[31522]: from='mgr.14120 172.21.15.19:0/2858628912' entity='mgr.x' 2024-03-18T15:06:58.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:58 smithi019 ceph-mon[31522]: from='mgr.14120 172.21.15.19:0/2858628912' entity='mgr.x' 2024-03-18T15:06:59.627 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-18T15:06:59.627 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:58.701+0000 7f5a35000700 1 Processor -- start 2024-03-18T15:06:59.627 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:58.702+0000 7f5a35000700 1 -- start start 2024-03-18T15:06:59.628 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:58.702+0000 7f5a35000700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5a3007de70 0x7f5a3007c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:06:59.628 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:58.702+0000 7f5a35000700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f5a3007c8a0 con 0x7f5a3007de70 2024-03-18T15:06:59.628 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:58.703+0000 7f5a2de00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5a3007de70 0x7f5a3007c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:06:59.628 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:58.703+0000 7f5a2de00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5a3007de70 0x7f5a3007c2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:40346/0 (socket says 172.21.15.19:40346) 2024-03-18T15:06:59.628 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:58.703+0000 7f5a2de00700 1 -- 172.21.15.19:0/2648177425 learned_addr learned my addr 172.21.15.19:0/2648177425 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:06:59.628 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:58.704+0000 7f5a2de00700 1 -- 172.21.15.19:0/2648177425 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a3007ca20 con 0x7f5a3007de70 2024-03-18T15:06:59.628 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:58.704+0000 7f5a2de00700 1 --2- 172.21.15.19:0/2648177425 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5a3007de70 0x7f5a3007c2d0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f5a200098a0 tx=0x7f5a20009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7aed01f6d0c0764 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:06:59.628 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:58.705+0000 7f5a2ca00700 1 -- 172.21.15.19:0/2648177425 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5a20004160 con 0x7f5a3007de70 2024-03-18T15:06:59.628 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:58.705+0000 7f5a2ca00700 1 -- 172.21.15.19:0/2648177425 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5a20004300 con 0x7f5a3007de70 2024-03-18T15:06:59.628 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:58.705+0000 7f5a2ca00700 1 -- 172.21.15.19:0/2648177425 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5a20011b20 con 0x7f5a3007de70 2024-03-18T15:06:59.628 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:58.706+0000 7f5a35000700 1 -- 172.21.15.19:0/2648177425 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5a3007de70 msgr2=0x7f5a3007c2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:06:59.628 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:58.706+0000 7f5a35000700 1 --2- 172.21.15.19:0/2648177425 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5a3007de70 0x7f5a3007c2d0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f5a200098a0 tx=0x7f5a20009bb0 comp rx=0 tx=0).stop 2024-03-18T15:06:59.628 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:58.706+0000 7f5a35000700 1 -- 172.21.15.19:0/2648177425 shutdown_connections 2024-03-18T15:06:59.629 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:58.706+0000 7f5a35000700 1 --2- 172.21.15.19:0/2648177425 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5a3007de70 0x7f5a3007c2d0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:06:59.629 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:58.706+0000 7f5a35000700 1 -- 172.21.15.19:0/2648177425 >> 172.21.15.19:0/2648177425 conn(0x7f5a30103250 msgr2=0x7f5a301056b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:06:59.629 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:58.706+0000 7f5a35000700 1 -- 172.21.15.19:0/2648177425 shutdown_connections 2024-03-18T15:06:59.629 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:58.706+0000 7f5a35000700 1 -- 172.21.15.19:0/2648177425 wait complete. 2024-03-18T15:06:59.629 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:58.707+0000 7f5a35000700 1 Processor -- start 2024-03-18T15:06:59.629 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:58.707+0000 7f5a35000700 1 -- start start 2024-03-18T15:06:59.629 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:58.707+0000 7f5a35000700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5a3007de70 0x7f5a3019de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:06:59.629 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:58.707+0000 7f5a35000700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f5a3010e350 con 0x7f5a3007de70 2024-03-18T15:06:59.629 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:58.708+0000 7f5a2de00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5a3007de70 0x7f5a3019de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:06:59.629 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:58.708+0000 7f5a2de00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5a3007de70 0x7f5a3019de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:40348/0 (socket says 172.21.15.19:40348) 2024-03-18T15:06:59.629 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:58.708+0000 7f5a2de00700 1 -- 172.21.15.19:0/4086690141 learned_addr learned my addr 172.21.15.19:0/4086690141 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:06:59.629 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:58.708+0000 7f5a2de00700 1 -- 172.21.15.19:0/4086690141 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a3019e340 con 0x7f5a3007de70 2024-03-18T15:06:59.629 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:58.708+0000 7f5a2de00700 1 --2- 172.21.15.19:0/4086690141 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5a3007de70 0x7f5a3019de00 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f5a200117e0 tx=0x7f5a20011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:06:59.630 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:58.709+0000 7f5a26a00700 1 -- 172.21.15.19:0/4086690141 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5a20003ba0 con 0x7f5a3007de70 2024-03-18T15:06:59.630 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:58.709+0000 7f5a26a00700 1 -- 172.21.15.19:0/4086690141 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5a2001a910 con 0x7f5a3007de70 2024-03-18T15:06:59.630 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:58.709+0000 7f5a26a00700 1 -- 172.21.15.19:0/4086690141 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5a200248f0 con 0x7f5a3007de70 2024-03-18T15:06:59.630 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:58.709+0000 7f5a35000700 1 -- 172.21.15.19:0/4086690141 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a3019e5d0 con 0x7f5a3007de70 2024-03-18T15:06:59.630 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:58.709+0000 7f5a35000700 1 -- 172.21.15.19:0/4086690141 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a3019ea30 con 0x7f5a3007de70 2024-03-18T15:06:59.630 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:58.710+0000 7f5a26a00700 1 -- 172.21.15.19:0/4086690141 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f5a20022020 con 0x7f5a3007de70 2024-03-18T15:06:59.630 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:58.710+0000 7f5a35000700 1 -- 172.21.15.19:0/4086690141 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a30063900 con 0x7f5a3007de70 2024-03-18T15:06:59.630 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:58.711+0000 7f5a26a00700 1 --2- 172.21.15.19:0/4086690141 >> [v2:172.21.15.19:6800/2776008962,v1:172.21.15.19:6801/2776008962] conn(0x7f5a14038f70 0x7f5a1403b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:06:59.630 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:58.711+0000 7f5a26a00700 1 -- 172.21.15.19:0/4086690141 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f5a2004d330 con 0x7f5a3007de70 2024-03-18T15:06:59.630 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:58.713+0000 7f5a2d400700 1 --2- 172.21.15.19:0/4086690141 >> [v2:172.21.15.19:6800/2776008962,v1:172.21.15.19:6801/2776008962] conn(0x7f5a14038f70 0x7f5a1403b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:06:59.630 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:58.713+0000 7f5a2d400700 1 --2- 172.21.15.19:0/4086690141 >> [v2:172.21.15.19:6800/2776008962,v1:172.21.15.19:6801/2776008962] conn(0x7f5a14038f70 0x7f5a1403b430 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f5a1c006fd0 tx=0x7f5a1c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:06:59.630 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:58.717+0000 7f5a26a00700 1 -- 172.21.15.19:0/4086690141 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5a2001ab70 con 0x7f5a3007de70 2024-03-18T15:06:59.630 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:58.966+0000 7f5a35000700 1 -- 172.21.15.19:0/4086690141 --> [v2:172.21.15.19:6800/2776008962,v1:172.21.15.19:6801/2776008962] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f5a30104430 con 0x7f5a14038f70 2024-03-18T15:06:59.630 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:58.974+0000 7f5a26a00700 1 -- 172.21.15.19:0/4086690141 <== mgr.14120 v2:172.21.15.19:6800/2776008962 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f5a30104430 con 0x7f5a14038f70 2024-03-18T15:06:59.631 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:58.976+0000 7f5a35000700 1 -- 172.21.15.19:0/4086690141 >> [v2:172.21.15.19:6800/2776008962,v1:172.21.15.19:6801/2776008962] conn(0x7f5a14038f70 msgr2=0x7f5a1403b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:06:59.631 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:58.976+0000 7f5a35000700 1 --2- 172.21.15.19:0/4086690141 >> [v2:172.21.15.19:6800/2776008962,v1:172.21.15.19:6801/2776008962] conn(0x7f5a14038f70 0x7f5a1403b430 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f5a1c006fd0 tx=0x7f5a1c006e40 comp rx=0 tx=0).stop 2024-03-18T15:06:59.631 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:58.976+0000 7f5a35000700 1 -- 172.21.15.19:0/4086690141 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5a3007de70 msgr2=0x7f5a3019de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:06:59.631 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:58.976+0000 7f5a35000700 1 --2- 172.21.15.19:0/4086690141 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5a3007de70 0x7f5a3019de00 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f5a200117e0 tx=0x7f5a20011ba0 comp rx=0 tx=0).stop 2024-03-18T15:06:59.631 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:58.977+0000 7f5a35000700 1 -- 172.21.15.19:0/4086690141 shutdown_connections 2024-03-18T15:06:59.631 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:58.977+0000 7f5a35000700 1 --2- 172.21.15.19:0/4086690141 >> [v2:172.21.15.19:6800/2776008962,v1:172.21.15.19:6801/2776008962] conn(0x7f5a14038f70 0x7f5a1403b430 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:06:59.631 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:58.977+0000 7f5a35000700 1 --2- 172.21.15.19:0/4086690141 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5a3007de70 0x7f5a3019de00 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:06:59.631 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:58.977+0000 7f5a35000700 1 -- 172.21.15.19:0/4086690141 >> 172.21.15.19:0/4086690141 conn(0x7f5a30103250 msgr2=0x7f5a30103d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:06:59.631 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:58.977+0000 7f5a35000700 1 -- 172.21.15.19:0/4086690141 shutdown_connections 2024-03-18T15:06:59.631 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:06:58.977+0000 7f5a35000700 1 -- 172.21.15.19:0/4086690141 wait complete. 2024-03-18T15:07:00.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:59 smithi019 ceph-mon[31522]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:07:00.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:59 smithi019 ceph-mon[31522]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:07:00.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:59 smithi019 ceph-mon[31522]: Saving service mgr spec with placement count:2 2024-03-18T15:07:00.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:06:59 smithi019 ceph-mon[31522]: from='mgr.14120 172.21.15.19:0/2858628912' entity='mgr.x' 2024-03-18T15:07:01.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:01 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/2462113085' entity='client.admin' 2024-03-18T15:07:01.735 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:00.249+0000 7f878a400700 1 Processor -- start 2024-03-18T15:07:01.737 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:00.249+0000 7f878a400700 1 -- start start 2024-03-18T15:07:01.737 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:00.250+0000 7f878a400700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8784108770 0x7f878410ab60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:07:01.737 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:00.250+0000 7f878a400700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f878407c6f0 con 0x7f8784108770 2024-03-18T15:07:01.737 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:00.250+0000 7f8783400700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8784108770 0x7f878410ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:07:01.738 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:00.250+0000 7f8783400700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8784108770 0x7f878410ab60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:40352/0 (socket says 172.21.15.19:40352) 2024-03-18T15:07:01.738 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:00.250+0000 7f8783400700 1 -- 172.21.15.19:0/4182715212 learned_addr learned my addr 172.21.15.19:0/4182715212 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:07:01.738 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:00.250+0000 7f8783400700 1 -- 172.21.15.19:0/4182715212 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f878410b0a0 con 0x7f8784108770 2024-03-18T15:07:01.738 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:00.251+0000 7f8783400700 1 --2- 172.21.15.19:0/4182715212 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8784108770 0x7f878410ab60 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f8774009e80 tx=0x7f877400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=562fcdb9e01904c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:07:01.738 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:00.251+0000 7f8782000700 1 -- 172.21.15.19:0/4182715212 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8774017070 con 0x7f8784108770 2024-03-18T15:07:01.738 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:00.251+0000 7f8782000700 1 -- 172.21.15.19:0/4182715212 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f877400b8c0 con 0x7f8784108770 2024-03-18T15:07:01.738 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:00.251+0000 7f8782000700 1 -- 172.21.15.19:0/4182715212 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f87740045c0 con 0x7f8784108770 2024-03-18T15:07:01.738 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:00.252+0000 7f878a400700 1 -- 172.21.15.19:0/4182715212 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8784108770 msgr2=0x7f878410ab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:07:01.742 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:00.252+0000 7f878a400700 1 --2- 172.21.15.19:0/4182715212 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8784108770 0x7f878410ab60 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f8774009e80 tx=0x7f877400b330 comp rx=0 tx=0).stop 2024-03-18T15:07:01.742 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:00.252+0000 7f878a400700 1 -- 172.21.15.19:0/4182715212 shutdown_connections 2024-03-18T15:07:01.743 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:00.252+0000 7f878a400700 1 --2- 172.21.15.19:0/4182715212 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8784108770 0x7f878410ab60 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:07:01.743 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:00.252+0000 7f878a400700 1 -- 172.21.15.19:0/4182715212 >> 172.21.15.19:0/4182715212 conn(0x7f8784102160 msgr2=0x7f87841045a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:07:01.743 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:00.252+0000 7f878a400700 1 -- 172.21.15.19:0/4182715212 shutdown_connections 2024-03-18T15:07:01.743 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:00.252+0000 7f878a400700 1 -- 172.21.15.19:0/4182715212 wait complete. 2024-03-18T15:07:01.743 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:00.254+0000 7f878a400700 1 Processor -- start 2024-03-18T15:07:01.743 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:00.254+0000 7f878a400700 1 -- start start 2024-03-18T15:07:01.743 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:00.254+0000 7f878a400700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8784108770 0x7f878419deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:07:01.743 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:00.254+0000 7f878a400700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f878410c100 con 0x7f8784108770 2024-03-18T15:07:01.743 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:00.254+0000 7f8783400700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8784108770 0x7f878419deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:07:01.743 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:00.254+0000 7f8783400700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8784108770 0x7f878419deb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:40358/0 (socket says 172.21.15.19:40358) 2024-03-18T15:07:01.743 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:00.254+0000 7f8783400700 1 -- 172.21.15.19:0/2462113085 learned_addr learned my addr 172.21.15.19:0/2462113085 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:07:01.743 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:00.255+0000 7f8783400700 1 -- 172.21.15.19:0/2462113085 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f878419e3f0 con 0x7f8784108770 2024-03-18T15:07:01.743 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:00.255+0000 7f8783400700 1 --2- 172.21.15.19:0/2462113085 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8784108770 0x7f878419deb0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8774009530 tx=0x7f8774003d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:07:01.744 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:00.255+0000 7f877be00700 1 -- 172.21.15.19:0/2462113085 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8774020020 con 0x7f8784108770 2024-03-18T15:07:01.744 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:00.255+0000 7f877be00700 1 -- 172.21.15.19:0/2462113085 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f877400bba0 con 0x7f8784108770 2024-03-18T15:07:01.744 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:00.255+0000 7f878a400700 1 -- 172.21.15.19:0/2462113085 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f878419e680 con 0x7f8784108770 2024-03-18T15:07:01.744 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:00.255+0000 7f877be00700 1 -- 172.21.15.19:0/2462113085 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8774017040 con 0x7f8784108770 2024-03-18T15:07:01.744 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:00.256+0000 7f878a400700 1 -- 172.21.15.19:0/2462113085 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87841a1370 con 0x7f8784108770 2024-03-18T15:07:01.744 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:00.257+0000 7f877be00700 1 -- 172.21.15.19:0/2462113085 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f8774023020 con 0x7f8784108770 2024-03-18T15:07:01.744 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:00.257+0000 7f878a400700 1 -- 172.21.15.19:0/2462113085 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87840693d0 con 0x7f8784108770 2024-03-18T15:07:01.744 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:00.257+0000 7f877be00700 1 --2- 172.21.15.19:0/2462113085 >> [v2:172.21.15.19:6800/2776008962,v1:172.21.15.19:6801/2776008962] conn(0x7f8768038fd0 0x7f876803b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:07:01.744 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:00.258+0000 7f877be00700 1 -- 172.21.15.19:0/2462113085 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f877400f050 con 0x7f8784108770 2024-03-18T15:07:01.744 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:00.258+0000 7f8782a00700 1 --2- 172.21.15.19:0/2462113085 >> [v2:172.21.15.19:6800/2776008962,v1:172.21.15.19:6801/2776008962] conn(0x7f8768038fd0 0x7f876803b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:07:01.744 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:00.259+0000 7f8782a00700 1 --2- 172.21.15.19:0/2462113085 >> [v2:172.21.15.19:6800/2776008962,v1:172.21.15.19:6801/2776008962] conn(0x7f8768038fd0 0x7f876803b490 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f876c006fd0 tx=0x7f876c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:07:01.744 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:00.263+0000 7f877be00700 1 -- 172.21.15.19:0/2462113085 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f877402a020 con 0x7f8784108770 2024-03-18T15:07:01.744 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:00.506+0000 7f878a400700 1 -- 172.21.15.19:0/2462113085 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7f87841a1ad0 con 0x7f8784108770 2024-03-18T15:07:01.744 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:00.512+0000 7f877be00700 1 -- 172.21.15.19:0/2462113085 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) ==== 142+0+0 (secure 0 0 0) 0x7f877402c300 con 0x7f8784108770 2024-03-18T15:07:01.745 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:00.515+0000 7f878a400700 1 -- 172.21.15.19:0/2462113085 >> [v2:172.21.15.19:6800/2776008962,v1:172.21.15.19:6801/2776008962] conn(0x7f8768038fd0 msgr2=0x7f876803b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:07:01.745 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:00.515+0000 7f878a400700 1 --2- 172.21.15.19:0/2462113085 >> [v2:172.21.15.19:6800/2776008962,v1:172.21.15.19:6801/2776008962] conn(0x7f8768038fd0 0x7f876803b490 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f876c006fd0 tx=0x7f876c006e40 comp rx=0 tx=0).stop 2024-03-18T15:07:01.745 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:00.515+0000 7f878a400700 1 -- 172.21.15.19:0/2462113085 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8784108770 msgr2=0x7f878419deb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:07:01.745 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:00.515+0000 7f878a400700 1 --2- 172.21.15.19:0/2462113085 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8784108770 0x7f878419deb0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8774009530 tx=0x7f8774003d00 comp rx=0 tx=0).stop 2024-03-18T15:07:01.745 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:00.515+0000 7f878a400700 1 -- 172.21.15.19:0/2462113085 shutdown_connections 2024-03-18T15:07:01.745 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:00.515+0000 7f878a400700 1 --2- 172.21.15.19:0/2462113085 >> [v2:172.21.15.19:6800/2776008962,v1:172.21.15.19:6801/2776008962] conn(0x7f8768038fd0 0x7f876803b490 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:07:01.745 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:00.515+0000 7f878a400700 1 --2- 172.21.15.19:0/2462113085 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8784108770 0x7f878419deb0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:07:01.745 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:00.515+0000 7f878a400700 1 -- 172.21.15.19:0/2462113085 >> 172.21.15.19:0/2462113085 conn(0x7f8784102160 msgr2=0x7f8784102c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:07:01.745 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:00.516+0000 7f878a400700 1 -- 172.21.15.19:0/2462113085 shutdown_connections 2024-03-18T15:07:01.745 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:00.516+0000 7f878a400700 1 -- 172.21.15.19:0/2462113085 wait complete. 2024-03-18T15:07:02.555 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:02 smithi019 ceph-mon[31522]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:07:02.556 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:02 smithi019 ceph-mon[31522]: from='mgr.14120 172.21.15.19:0/2858628912' entity='mgr.x' 2024-03-18T15:07:03.217 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:02.303+0000 7f3f14e00700 1 Processor -- start 2024-03-18T15:07:03.217 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:02.303+0000 7f3f14e00700 1 -- start start 2024-03-18T15:07:03.217 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:02.304+0000 7f3f14e00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3f10108810 0x7f3f1010ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:07:03.217 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:02.304+0000 7f3f14e00700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f3f1007bdc0 con 0x7f3f10108810 2024-03-18T15:07:03.217 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:02.304+0000 7f3f0de00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3f10108810 0x7f3f1010ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:07:03.217 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:02.304+0000 7f3f0de00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3f10108810 0x7f3f1010ac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:34024/0 (socket says 172.21.15.19:34024) 2024-03-18T15:07:03.217 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:02.304+0000 7f3f0de00700 1 -- 172.21.15.19:0/3935221615 learned_addr learned my addr 172.21.15.19:0/3935221615 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:07:03.217 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:02.305+0000 7f3f0de00700 1 -- 172.21.15.19:0/3935221615 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f1010b140 con 0x7f3f10108810 2024-03-18T15:07:03.217 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:02.305+0000 7f3f0de00700 1 --2- 172.21.15.19:0/3935221615 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3f10108810 0x7f3f1010ac00 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3f00009e80 tx=0x7f3f0000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d172dab6d122453b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:07:03.217 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:02.306+0000 7f3f0d400700 1 -- 172.21.15.19:0/3935221615 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3f00017070 con 0x7f3f10108810 2024-03-18T15:07:03.217 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:02.306+0000 7f3f0d400700 1 -- 172.21.15.19:0/3935221615 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f0000b8c0 con 0x7f3f10108810 2024-03-18T15:07:03.218 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:02.306+0000 7f3f0d400700 1 -- 172.21.15.19:0/3935221615 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3f000045c0 con 0x7f3f10108810 2024-03-18T15:07:03.218 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:02.306+0000 7f3f14e00700 1 -- 172.21.15.19:0/3935221615 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3f10108810 msgr2=0x7f3f1010ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:07:03.218 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:02.306+0000 7f3f14e00700 1 --2- 172.21.15.19:0/3935221615 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3f10108810 0x7f3f1010ac00 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3f00009e80 tx=0x7f3f0000b330 comp rx=0 tx=0).stop 2024-03-18T15:07:03.218 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:02.307+0000 7f3f14e00700 1 -- 172.21.15.19:0/3935221615 shutdown_connections 2024-03-18T15:07:03.218 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:02.307+0000 7f3f14e00700 1 --2- 172.21.15.19:0/3935221615 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3f10108810 0x7f3f1010ac00 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:07:03.218 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:02.307+0000 7f3f14e00700 1 -- 172.21.15.19:0/3935221615 >> 172.21.15.19:0/3935221615 conn(0x7f3f10102140 msgr2=0x7f3f101045a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:07:03.218 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:02.307+0000 7f3f14e00700 1 -- 172.21.15.19:0/3935221615 shutdown_connections 2024-03-18T15:07:03.218 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:02.307+0000 7f3f14e00700 1 -- 172.21.15.19:0/3935221615 wait complete. 2024-03-18T15:07:03.218 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:02.308+0000 7f3f14e00700 1 Processor -- start 2024-03-18T15:07:03.218 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:02.308+0000 7f3f14e00700 1 -- start start 2024-03-18T15:07:03.218 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:02.308+0000 7f3f14e00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3f10108810 0x7f3f1019de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:07:03.218 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:02.308+0000 7f3f14e00700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f3f1010c0e0 con 0x7f3f10108810 2024-03-18T15:07:03.219 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:02.308+0000 7f3f0de00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3f10108810 0x7f3f1019de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:07:03.219 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:02.309+0000 7f3f0de00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3f10108810 0x7f3f1019de90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:34026/0 (socket says 172.21.15.19:34026) 2024-03-18T15:07:03.219 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:02.309+0000 7f3f0de00700 1 -- 172.21.15.19:0/2317423708 learned_addr learned my addr 172.21.15.19:0/2317423708 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:07:03.219 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:02.309+0000 7f3f0de00700 1 -- 172.21.15.19:0/2317423708 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f1019e3d0 con 0x7f3f10108810 2024-03-18T15:07:03.219 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:02.309+0000 7f3f0de00700 1 --2- 172.21.15.19:0/2317423708 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3f10108810 0x7f3f1019de90 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3f0000b7b0 tx=0x7f3f0000bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:07:03.219 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:02.310+0000 7f3f07400700 1 -- 172.21.15.19:0/2317423708 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3f00020020 con 0x7f3f10108810 2024-03-18T15:07:03.219 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:02.310+0000 7f3f07400700 1 -- 172.21.15.19:0/2317423708 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f00003d90 con 0x7f3f10108810 2024-03-18T15:07:03.219 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:02.310+0000 7f3f07400700 1 -- 172.21.15.19:0/2317423708 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3f00017040 con 0x7f3f10108810 2024-03-18T15:07:03.219 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:02.310+0000 7f3f14e00700 1 -- 172.21.15.19:0/2317423708 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f1019e660 con 0x7f3f10108810 2024-03-18T15:07:03.219 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:02.310+0000 7f3f14e00700 1 -- 172.21.15.19:0/2317423708 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f101a1350 con 0x7f3f10108810 2024-03-18T15:07:03.219 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:02.311+0000 7f3f14e00700 1 -- 172.21.15.19:0/2317423708 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f10063900 con 0x7f3f10108810 2024-03-18T15:07:03.219 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:02.311+0000 7f3f07400700 1 -- 172.21.15.19:0/2317423708 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f3f00023020 con 0x7f3f10108810 2024-03-18T15:07:03.219 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:02.312+0000 7f3f07400700 1 --2- 172.21.15.19:0/2317423708 >> [v2:172.21.15.19:6800/2776008962,v1:172.21.15.19:6801/2776008962] conn(0x7f3ef4038f20 0x7f3ef403b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:07:03.220 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:02.312+0000 7f3f07400700 1 -- 172.21.15.19:0/2317423708 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f3f0000f050 con 0x7f3f10108810 2024-03-18T15:07:03.220 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:02.312+0000 7f3f05400700 1 --2- 172.21.15.19:0/2317423708 >> [v2:172.21.15.19:6800/2776008962,v1:172.21.15.19:6801/2776008962] conn(0x7f3ef4038f20 0x7f3ef403b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:07:03.220 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:02.313+0000 7f3f05400700 1 --2- 172.21.15.19:0/2317423708 >> [v2:172.21.15.19:6800/2776008962,v1:172.21.15.19:6801/2776008962] conn(0x7f3ef4038f20 0x7f3ef403b3e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f3ef8006fd0 tx=0x7f3ef8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:07:03.220 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:02.319+0000 7f3f07400700 1 -- 172.21.15.19:0/2317423708 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3f0002a020 con 0x7f3f10108810 2024-03-18T15:07:03.221 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:02.549+0000 7f3f14e00700 1 -- 172.21.15.19:0/2317423708 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7f3f101a1990 con 0x7f3f10108810 2024-03-18T15:07:03.221 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:02.554+0000 7f3f07400700 1 -- 172.21.15.19:0/2317423708 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7f3f0004c920 con 0x7f3f10108810 2024-03-18T15:07:03.221 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:02.557+0000 7f3f14e00700 1 -- 172.21.15.19:0/2317423708 >> [v2:172.21.15.19:6800/2776008962,v1:172.21.15.19:6801/2776008962] conn(0x7f3ef4038f20 msgr2=0x7f3ef403b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:07:03.221 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:02.557+0000 7f3f14e00700 1 --2- 172.21.15.19:0/2317423708 >> [v2:172.21.15.19:6800/2776008962,v1:172.21.15.19:6801/2776008962] conn(0x7f3ef4038f20 0x7f3ef403b3e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f3ef8006fd0 tx=0x7f3ef8006e40 comp rx=0 tx=0).stop 2024-03-18T15:07:03.221 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:02.557+0000 7f3f14e00700 1 -- 172.21.15.19:0/2317423708 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3f10108810 msgr2=0x7f3f1019de90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:07:03.221 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:02.557+0000 7f3f14e00700 1 --2- 172.21.15.19:0/2317423708 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3f10108810 0x7f3f1019de90 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3f0000b7b0 tx=0x7f3f0000bd50 comp rx=0 tx=0).stop 2024-03-18T15:07:03.221 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:02.558+0000 7f3f14e00700 1 -- 172.21.15.19:0/2317423708 shutdown_connections 2024-03-18T15:07:03.221 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:02.558+0000 7f3f14e00700 1 --2- 172.21.15.19:0/2317423708 >> [v2:172.21.15.19:6800/2776008962,v1:172.21.15.19:6801/2776008962] conn(0x7f3ef4038f20 0x7f3ef403b3e0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:07:03.221 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:02.558+0000 7f3f14e00700 1 --2- 172.21.15.19:0/2317423708 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3f10108810 0x7f3f1019de90 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:07:03.221 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:02.558+0000 7f3f14e00700 1 -- 172.21.15.19:0/2317423708 >> 172.21.15.19:0/2317423708 conn(0x7f3f10102140 msgr2=0x7f3f10102bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:07:03.221 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:02.558+0000 7f3f14e00700 1 -- 172.21.15.19:0/2317423708 shutdown_connections 2024-03-18T15:07:03.221 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:02.558+0000 7f3f14e00700 1 -- 172.21.15.19:0/2317423708 wait complete. 2024-03-18T15:07:03.221 INFO:teuthology.orchestra.run.smithi019.stdout:Enabling the dashboard module... 2024-03-18T15:07:03.806 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:03 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/2317423708' entity='client.admin' 2024-03-18T15:07:04.824 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:04 smithi019 ceph-mon[31522]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:07:04.824 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:04 smithi019 ceph-mon[31522]: from='mgr.14120 172.21.15.19:0/2858628912' entity='mgr.x' 2024-03-18T15:07:04.824 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:04 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/3223584247' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-18T15:07:05.149 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:07:04 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: ignoring --setuser ceph since I am not root 2024-03-18T15:07:05.149 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:07:04 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: ignoring --setgroup ceph since I am not root 2024-03-18T15:07:05.934 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:05 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/3223584247' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-18T15:07:05.934 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:05 smithi019 ceph-mon[31522]: mgrmap e9: x(active, since 26s) 2024-03-18T15:07:05.934 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:07:05 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:07:05.697+0000 7f4530e9b200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T15:07:05.934 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:07:05 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:07:05.820+0000 7f4530e9b200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T15:07:05.950 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:03.805+0000 7f1de0600700 1 Processor -- start 2024-03-18T15:07:05.950 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:03.805+0000 7f1de0600700 1 -- start start 2024-03-18T15:07:05.950 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:03.805+0000 7f1de0600700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1dd8106bc0 0x7f1dd8106fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:07:05.950 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:03.805+0000 7f1de0600700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f1dd81075b0 con 0x7f1dd8106bc0 2024-03-18T15:07:05.950 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:03.806+0000 7f1dddc00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1dd8106bc0 0x7f1dd8106fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:07:05.950 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:03.806+0000 7f1dddc00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1dd8106bc0 0x7f1dd8106fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:34030/0 (socket says 172.21.15.19:34030) 2024-03-18T15:07:05.950 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:03.806+0000 7f1dddc00700 1 -- 172.21.15.19:0/1782802297 learned_addr learned my addr 172.21.15.19:0/1782802297 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:07:05.950 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:03.806+0000 7f1dddc00700 1 -- 172.21.15.19:0/1782802297 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1dd8107dc0 con 0x7f1dd8106bc0 2024-03-18T15:07:05.951 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:03.806+0000 7f1dddc00700 1 --2- 172.21.15.19:0/1782802297 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1dd8106bc0 0x7f1dd8106fe0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f1dcc009e80 tx=0x7f1dcc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac40b3e8e6c6fa3b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:07:05.951 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:03.807+0000 7f1dd7e00700 1 -- 172.21.15.19:0/1782802297 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1dcc017070 con 0x7f1dd8106bc0 2024-03-18T15:07:05.951 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:03.807+0000 7f1dd7e00700 1 -- 172.21.15.19:0/1782802297 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1dcc00b8c0 con 0x7f1dd8106bc0 2024-03-18T15:07:05.951 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:03.807+0000 7f1dd7e00700 1 -- 172.21.15.19:0/1782802297 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1dcc0045c0 con 0x7f1dd8106bc0 2024-03-18T15:07:05.951 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:03.808+0000 7f1de0600700 1 -- 172.21.15.19:0/1782802297 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1dd8106bc0 msgr2=0x7f1dd8106fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:07:05.951 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:03.808+0000 7f1de0600700 1 --2- 172.21.15.19:0/1782802297 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1dd8106bc0 0x7f1dd8106fe0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f1dcc009e80 tx=0x7f1dcc00b330 comp rx=0 tx=0).stop 2024-03-18T15:07:05.951 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:03.808+0000 7f1de0600700 1 -- 172.21.15.19:0/1782802297 shutdown_connections 2024-03-18T15:07:05.951 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:03.808+0000 7f1de0600700 1 --2- 172.21.15.19:0/1782802297 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1dd8106bc0 0x7f1dd8106fe0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:07:05.951 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:03.808+0000 7f1de0600700 1 -- 172.21.15.19:0/1782802297 >> 172.21.15.19:0/1782802297 conn(0x7f1dd8102140 msgr2=0x7f1dd81045a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:07:05.951 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:03.808+0000 7f1de0600700 1 -- 172.21.15.19:0/1782802297 shutdown_connections 2024-03-18T15:07:05.951 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:03.808+0000 7f1de0600700 1 -- 172.21.15.19:0/1782802297 wait complete. 2024-03-18T15:07:05.951 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:03.809+0000 7f1de0600700 1 Processor -- start 2024-03-18T15:07:05.951 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:03.810+0000 7f1de0600700 1 -- start start 2024-03-18T15:07:05.951 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:03.810+0000 7f1de0600700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1dd8106bc0 0x7f1dd819de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:07:05.952 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:03.810+0000 7f1de0600700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f1dd810b170 con 0x7f1dd8106bc0 2024-03-18T15:07:05.952 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:03.810+0000 7f1dddc00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1dd8106bc0 0x7f1dd819de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:07:05.952 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:03.810+0000 7f1dddc00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1dd8106bc0 0x7f1dd819de90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:34036/0 (socket says 172.21.15.19:34036) 2024-03-18T15:07:05.952 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:03.810+0000 7f1dddc00700 1 -- 172.21.15.19:0/3223584247 learned_addr learned my addr 172.21.15.19:0/3223584247 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:07:05.952 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:03.811+0000 7f1dddc00700 1 -- 172.21.15.19:0/3223584247 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1dd819e3d0 con 0x7f1dd8106bc0 2024-03-18T15:07:05.952 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:03.811+0000 7f1dddc00700 1 --2- 172.21.15.19:0/3223584247 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1dd8106bc0 0x7f1dd819de90 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1dcc00bca0 tx=0x7f1dcc00bd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:07:05.952 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:03.811+0000 7f1dd6000700 1 -- 172.21.15.19:0/3223584247 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1dcc020020 con 0x7f1dd8106bc0 2024-03-18T15:07:05.952 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:03.811+0000 7f1dd6000700 1 -- 172.21.15.19:0/3223584247 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1dcc003d90 con 0x7f1dd8106bc0 2024-03-18T15:07:05.952 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:03.811+0000 7f1de0600700 1 -- 172.21.15.19:0/3223584247 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1dd819e660 con 0x7f1dd8106bc0 2024-03-18T15:07:05.952 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:03.811+0000 7f1de0600700 1 -- 172.21.15.19:0/3223584247 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1dd81a1350 con 0x7f1dd8106bc0 2024-03-18T15:07:05.952 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:03.811+0000 7f1dd6000700 1 -- 172.21.15.19:0/3223584247 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1dcc017040 con 0x7f1dd8106bc0 2024-03-18T15:07:05.952 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:03.813+0000 7f1de0600700 1 -- 172.21.15.19:0/3223584247 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1dd8063900 con 0x7f1dd8106bc0 2024-03-18T15:07:05.952 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:03.813+0000 7f1dd6000700 1 -- 172.21.15.19:0/3223584247 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f1dcc023020 con 0x7f1dd8106bc0 2024-03-18T15:07:05.952 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:03.813+0000 7f1dd6000700 1 --2- 172.21.15.19:0/3223584247 >> [v2:172.21.15.19:6800/2776008962,v1:172.21.15.19:6801/2776008962] conn(0x7f1dc0038ed0 0x7f1dc003b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:07:05.953 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:03.813+0000 7f1dd6000700 1 -- 172.21.15.19:0/3223584247 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f1dcc00f050 con 0x7f1dd8106bc0 2024-03-18T15:07:05.953 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:03.814+0000 7f1ddd200700 1 --2- 172.21.15.19:0/3223584247 >> [v2:172.21.15.19:6800/2776008962,v1:172.21.15.19:6801/2776008962] conn(0x7f1dc0038ed0 0x7f1dc003b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:07:05.953 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:03.814+0000 7f1ddd200700 1 --2- 172.21.15.19:0/3223584247 >> [v2:172.21.15.19:6800/2776008962,v1:172.21.15.19:6801/2776008962] conn(0x7f1dc0038ed0 0x7f1dc003b390 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f1dc4006fd0 tx=0x7f1dc4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:07:05.953 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:03.819+0000 7f1dd6000700 1 -- 172.21.15.19:0/3223584247 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1dcc02a020 con 0x7f1dd8106bc0 2024-03-18T15:07:05.953 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:04.106+0000 7f1de0600700 1 -- 172.21.15.19:0/3223584247 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7f1dd802deb0 con 0x7f1dd8106bc0 2024-03-18T15:07:05.953 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:04.821+0000 7f1dd6000700 1 -- 172.21.15.19:0/3223584247 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7f1dcc02d350 con 0x7f1dd8106bc0 2024-03-18T15:07:05.953 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:04.826+0000 7f1dd6000700 1 -- 172.21.15.19:0/3223584247 <== mon.0 v2:172.21.15.19:3300/0 8 ==== mgrmap(e 9) ==== 46119+0+0 (secure 0 0 0) 0x7f1dcc012430 con 0x7f1dd8106bc0 2024-03-18T15:07:05.953 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:04.827+0000 7f1de0600700 1 -- 172.21.15.19:0/3223584247 >> [v2:172.21.15.19:6800/2776008962,v1:172.21.15.19:6801/2776008962] conn(0x7f1dc0038ed0 msgr2=0x7f1dc003b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:07:05.954 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:04.827+0000 7f1de0600700 1 --2- 172.21.15.19:0/3223584247 >> [v2:172.21.15.19:6800/2776008962,v1:172.21.15.19:6801/2776008962] conn(0x7f1dc0038ed0 0x7f1dc003b390 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f1dc4006fd0 tx=0x7f1dc4006e40 comp rx=0 tx=0).stop 2024-03-18T15:07:05.954 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:04.827+0000 7f1de0600700 1 -- 172.21.15.19:0/3223584247 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1dd8106bc0 msgr2=0x7f1dd819de90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:07:05.954 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:04.827+0000 7f1de0600700 1 --2- 172.21.15.19:0/3223584247 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1dd8106bc0 0x7f1dd819de90 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1dcc00bca0 tx=0x7f1dcc00bd80 comp rx=0 tx=0).stop 2024-03-18T15:07:05.954 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:04.829+0000 7f1de0600700 1 -- 172.21.15.19:0/3223584247 shutdown_connections 2024-03-18T15:07:05.954 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:04.829+0000 7f1de0600700 1 --2- 172.21.15.19:0/3223584247 >> [v2:172.21.15.19:6800/2776008962,v1:172.21.15.19:6801/2776008962] conn(0x7f1dc0038ed0 0x7f1dc003b390 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:07:05.954 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:04.829+0000 7f1de0600700 1 --2- 172.21.15.19:0/3223584247 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1dd8106bc0 0x7f1dd819de90 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:07:05.954 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:04.829+0000 7f1de0600700 1 -- 172.21.15.19:0/3223584247 >> 172.21.15.19:0/3223584247 conn(0x7f1dd8102140 msgr2=0x7f1dd8103000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:07:05.954 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:04.829+0000 7f1de0600700 1 -- 172.21.15.19:0/3223584247 shutdown_connections 2024-03-18T15:07:05.954 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:04.829+0000 7f1de0600700 1 -- 172.21.15.19:0/3223584247 wait complete. 2024-03-18T15:07:06.803 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:07:06 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:07:06.700+0000 7f4530e9b200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T15:07:07.160 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:06 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/3267432305' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-18T15:07:07.161 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:07:06 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:07:06.830+0000 7f4530e9b200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T15:07:07.161 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:07:07 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:07:07.030+0000 7f4530e9b200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T15:07:07.465 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:07:07 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:07:07.160+0000 7f4530e9b200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T15:07:07.465 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:07:07 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:07:07.276+0000 7f4530e9b200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T15:07:07.589 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout { 2024-03-18T15:07:07.589 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-18T15:07:07.589 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-18T15:07:07.589 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-18T15:07:07.589 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-18T15:07:07.589 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout } 2024-03-18T15:07:07.589 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:06.546+0000 7f883a200700 1 Processor -- start 2024-03-18T15:07:07.589 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:06.547+0000 7f883a200700 1 -- start start 2024-03-18T15:07:07.589 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:06.547+0000 7f883a200700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8834108740 0x7f883410ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:07:07.590 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:06.547+0000 7f883a200700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f883407bea0 con 0x7f8834108740 2024-03-18T15:07:07.590 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:06.547+0000 7f8833400700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8834108740 0x7f883410ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:07:07.590 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:06.547+0000 7f8833400700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8834108740 0x7f883410ab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:34072/0 (socket says 172.21.15.19:34072) 2024-03-18T15:07:07.590 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:06.548+0000 7f8833400700 1 -- 172.21.15.19:0/1784943116 learned_addr learned my addr 172.21.15.19:0/1784943116 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:07:07.590 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:06.548+0000 7f8833400700 1 -- 172.21.15.19:0/1784943116 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f883410b070 con 0x7f8834108740 2024-03-18T15:07:07.590 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:06.548+0000 7f8833400700 1 --2- 172.21.15.19:0/1784943116 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8834108740 0x7f883410ab30 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f88240098a0 tx=0x7f8824009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55468e57413d49dd server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:07:07.590 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:06.549+0000 7f8832000700 1 -- 172.21.15.19:0/1784943116 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8824004160 con 0x7f8834108740 2024-03-18T15:07:07.590 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:06.549+0000 7f8832000700 1 -- 172.21.15.19:0/1784943116 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8824004300 con 0x7f8834108740 2024-03-18T15:07:07.590 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:06.549+0000 7f8832000700 1 -- 172.21.15.19:0/1784943116 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8824011b20 con 0x7f8834108740 2024-03-18T15:07:07.590 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:06.549+0000 7f883a200700 1 -- 172.21.15.19:0/1784943116 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8834108740 msgr2=0x7f883410ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:07:07.590 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:06.549+0000 7f883a200700 1 --2- 172.21.15.19:0/1784943116 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8834108740 0x7f883410ab30 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f88240098a0 tx=0x7f8824009bb0 comp rx=0 tx=0).stop 2024-03-18T15:07:07.590 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:06.549+0000 7f883a200700 1 -- 172.21.15.19:0/1784943116 shutdown_connections 2024-03-18T15:07:07.591 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:06.549+0000 7f883a200700 1 --2- 172.21.15.19:0/1784943116 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8834108740 0x7f883410ab30 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:07:07.591 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:06.549+0000 7f883a200700 1 -- 172.21.15.19:0/1784943116 >> 172.21.15.19:0/1784943116 conn(0x7f8834102130 msgr2=0x7f88341045b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:07:07.591 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:06.549+0000 7f883a200700 1 -- 172.21.15.19:0/1784943116 shutdown_connections 2024-03-18T15:07:07.591 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:06.549+0000 7f883a200700 1 -- 172.21.15.19:0/1784943116 wait complete. 2024-03-18T15:07:07.591 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:06.551+0000 7f883a200700 1 Processor -- start 2024-03-18T15:07:07.591 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:06.551+0000 7f883a200700 1 -- start start 2024-03-18T15:07:07.591 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:06.552+0000 7f883a200700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8834108740 0x7f8834107fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:07:07.591 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:06.552+0000 7f883a200700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f883410c0e0 con 0x7f8834108740 2024-03-18T15:07:07.591 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:06.552+0000 7f8833400700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8834108740 0x7f8834107fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:07:07.591 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:06.552+0000 7f8833400700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8834108740 0x7f8834107fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:34078/0 (socket says 172.21.15.19:34078) 2024-03-18T15:07:07.591 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:06.552+0000 7f8833400700 1 -- 172.21.15.19:0/3267432305 learned_addr learned my addr 172.21.15.19:0/3267432305 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:07:07.591 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:06.552+0000 7f8833400700 1 -- 172.21.15.19:0/3267432305 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8834108500 con 0x7f8834108740 2024-03-18T15:07:07.591 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:06.552+0000 7f8833400700 1 --2- 172.21.15.19:0/3267432305 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8834108740 0x7f8834107fc0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8824011b60 tx=0x7f8824011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:07:07.592 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:06.553+0000 7f882be00700 1 -- 172.21.15.19:0/3267432305 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8824003b60 con 0x7f8834108740 2024-03-18T15:07:07.592 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:06.553+0000 7f882be00700 1 -- 172.21.15.19:0/3267432305 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f882401a910 con 0x7f8834108740 2024-03-18T15:07:07.592 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:06.553+0000 7f882be00700 1 -- 172.21.15.19:0/3267432305 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8824024930 con 0x7f8834108740 2024-03-18T15:07:07.592 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:06.553+0000 7f883a200700 1 -- 172.21.15.19:0/3267432305 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88341066f0 con 0x7f8834108740 2024-03-18T15:07:07.592 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:06.553+0000 7f883a200700 1 -- 172.21.15.19:0/3267432305 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8834106bb0 con 0x7f8834108740 2024-03-18T15:07:07.592 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:06.554+0000 7f882be00700 1 -- 172.21.15.19:0/3267432305 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 9) ==== 46119+0+0 (secure 0 0 0) 0x7f8824022020 con 0x7f8834108740 2024-03-18T15:07:07.592 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:06.554+0000 7f883a200700 1 -- 172.21.15.19:0/3267432305 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88340693d0 con 0x7f8834108740 2024-03-18T15:07:07.592 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:06.554+0000 7f882be00700 1 --2- 172.21.15.19:0/3267432305 >> [v2:172.21.15.19:6800/2776008962,v1:172.21.15.19:6801/2776008962] conn(0x7f8818039020 0x7f881803b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:07:07.592 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:06.555+0000 7f882be00700 1 -- 172.21.15.19:0/3267432305 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f882404d160 con 0x7f8834108740 2024-03-18T15:07:07.592 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:06.555+0000 7f8832a00700 1 -- 172.21.15.19:0/3267432305 >> [v2:172.21.15.19:6800/2776008962,v1:172.21.15.19:6801/2776008962] conn(0x7f8818039020 msgr2=0x7f881803b4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.19:6800/2776008962 2024-03-18T15:07:07.592 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:06.555+0000 7f8832a00700 1 --2- 172.21.15.19:0/3267432305 >> [v2:172.21.15.19:6800/2776008962,v1:172.21.15.19:6801/2776008962] conn(0x7f8818039020 0x7f881803b4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T15:07:07.592 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:06.560+0000 7f882be00700 1 -- 172.21.15.19:0/3267432305 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f882401a4e0 con 0x7f8834108740 2024-03-18T15:07:07.592 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:06.756+0000 7f8832a00700 1 -- 172.21.15.19:0/3267432305 >> [v2:172.21.15.19:6800/2776008962,v1:172.21.15.19:6801/2776008962] conn(0x7f8818039020 msgr2=0x7f881803b4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.19:6800/2776008962 2024-03-18T15:07:07.593 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:06.756+0000 7f8832a00700 1 --2- 172.21.15.19:0/3267432305 >> [v2:172.21.15.19:6800/2776008962,v1:172.21.15.19:6801/2776008962] conn(0x7f8818039020 0x7f881803b4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T15:07:07.593 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:06.898+0000 7f883a200700 1 -- 172.21.15.19:0/3267432305 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f8834107450 con 0x7f8834108740 2024-03-18T15:07:07.593 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:06.898+0000 7f882be00700 1 -- 172.21.15.19:0/3267432305 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7f882402d3f0 con 0x7f8834108740 2024-03-18T15:07:07.593 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:06.901+0000 7f883a200700 1 -- 172.21.15.19:0/3267432305 >> [v2:172.21.15.19:6800/2776008962,v1:172.21.15.19:6801/2776008962] conn(0x7f8818039020 msgr2=0x7f881803b4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T15:07:07.593 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:06.901+0000 7f883a200700 1 --2- 172.21.15.19:0/3267432305 >> [v2:172.21.15.19:6800/2776008962,v1:172.21.15.19:6801/2776008962] conn(0x7f8818039020 0x7f881803b4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:07:07.593 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:06.901+0000 7f883a200700 1 -- 172.21.15.19:0/3267432305 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8834108740 msgr2=0x7f8834107fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:07:07.593 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:06.901+0000 7f883a200700 1 --2- 172.21.15.19:0/3267432305 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8834108740 0x7f8834107fc0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8824011b60 tx=0x7f8824011c40 comp rx=0 tx=0).stop 2024-03-18T15:07:07.593 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:06.901+0000 7f883a200700 1 -- 172.21.15.19:0/3267432305 shutdown_connections 2024-03-18T15:07:07.593 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:06.901+0000 7f883a200700 1 --2- 172.21.15.19:0/3267432305 >> [v2:172.21.15.19:6800/2776008962,v1:172.21.15.19:6801/2776008962] conn(0x7f8818039020 0x7f881803b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:07:07.593 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:06.901+0000 7f883a200700 1 --2- 172.21.15.19:0/3267432305 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8834108740 0x7f8834107fc0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:07:07.593 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:06.901+0000 7f883a200700 1 -- 172.21.15.19:0/3267432305 >> 172.21.15.19:0/3267432305 conn(0x7f8834102130 msgr2=0x7f8834102cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:07:07.593 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:06.901+0000 7f883a200700 1 -- 172.21.15.19:0/3267432305 shutdown_connections 2024-03-18T15:07:07.593 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:06.901+0000 7f883a200700 1 -- 172.21.15.19:0/3267432305 wait complete. 2024-03-18T15:07:07.593 INFO:teuthology.orchestra.run.smithi019.stdout:Waiting for the mgr to restart... 2024-03-18T15:07:07.594 INFO:teuthology.orchestra.run.smithi019.stdout:Waiting for mgr epoch 9... 2024-03-18T15:07:07.871 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:07:07 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:07:07.627+0000 7f4530e9b200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T15:07:07.871 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:07:07 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:07:07.778+0000 7f4530e9b200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T15:07:08.136 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:07:07 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:07:07.898+0000 7f4530e9b200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T15:07:08.136 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:07:08 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:07:08.028+0000 7f4530e9b200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T15:07:08.405 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:07:08 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:07:08.256+0000 7f4530e9b200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T15:07:09.965 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:07:09 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:07:09.589+0000 7f4530e9b200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T15:07:09.965 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:07:09 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:07:09.770+0000 7f4530e9b200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T15:07:10.465 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:07:10 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:07:10.202+0000 7f4530e9b200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T15:07:10.465 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:07:10 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:07:10.327+0000 7f4530e9b200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T15:07:10.850 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:07:10 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:07:10.593+0000 7f4530e9b200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T15:07:10.851 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:07:10 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:07:10.849+0000 7f4530e9b200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T15:07:11.465 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:07:11 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:07:11.241+0000 7f4530e9b200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T15:07:11.884 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:07:11 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:07:11.568+0000 7f4530e9b200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T15:07:12.215 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:07:11 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:07:11.883+0000 7f4530e9b200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T15:07:13.215 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:07:12 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:07:12.959+0000 7f4530e9b200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T15:07:13.790 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:07:13 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:07:13.697+0000 7f4530e9b200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T15:07:14.148 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:07:13 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:07:13.849+0000 7f4530e9b200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T15:07:14.465 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:07:14 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:07:14.147+0000 7f4530e9b200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T15:07:14.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:14 smithi019 ceph-mon[31522]: Active manager daemon x restarted 2024-03-18T15:07:14.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:14 smithi019 ceph-mon[31522]: Activating manager daemon x 2024-03-18T15:07:14.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:14 smithi019 ceph-mon[31522]: osdmap e3: 0 total, 0 up, 0 in 2024-03-18T15:07:14.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:14 smithi019 ceph-mon[31522]: mgrmap e10: x(active, starting, since 0.00735253s) 2024-03-18T15:07:14.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:14 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T15:07:14.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:14 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-18T15:07:14.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:14 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-18T15:07:14.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:14 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-18T15:07:14.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:14 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-18T15:07:14.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:14 smithi019 ceph-mon[31522]: Manager daemon x is now available 2024-03-18T15:07:15.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:15 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:07:15.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:15 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:07:15.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:15 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-18T15:07:15.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:15 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-18T15:07:15.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:15 smithi019 ceph-mon[31522]: [18/Mar/2024:15:07:14] ENGINE Bus STARTING 2024-03-18T15:07:15.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:15 smithi019 ceph-mon[31522]: [18/Mar/2024:15:07:14] ENGINE Serving on http://172.21.15.19:8765 2024-03-18T15:07:15.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:15 smithi019 ceph-mon[31522]: [18/Mar/2024:15:07:14] ENGINE Serving on https://172.21.15.19:7150 2024-03-18T15:07:15.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:15 smithi019 ceph-mon[31522]: [18/Mar/2024:15:07:14] ENGINE Bus STARTED 2024-03-18T15:07:15.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:15 smithi019 ceph-mon[31522]: mgrmap e11: x(active, since 1.0086s) 2024-03-18T15:07:16.007 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout { 2024-03-18T15:07:16.008 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-18T15:07:16.008 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-18T15:07:16.008 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout } 2024-03-18T15:07:16.008 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:08.195+0000 7f6cea000700 1 Processor -- start 2024-03-18T15:07:16.008 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:08.196+0000 7f6cea000700 1 -- start start 2024-03-18T15:07:16.008 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:08.196+0000 7f6cea000700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6ce407deb0 0x7f6ce407c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:07:16.008 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:08.196+0000 7f6cea000700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f6ce407c8e0 con 0x7f6ce407deb0 2024-03-18T15:07:16.008 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:08.196+0000 7f6ce3400700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6ce407deb0 0x7f6ce407c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:07:16.008 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:08.196+0000 7f6ce3400700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6ce407deb0 0x7f6ce407c310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:34080/0 (socket says 172.21.15.19:34080) 2024-03-18T15:07:16.008 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:08.197+0000 7f6ce3400700 1 -- 172.21.15.19:0/955609450 learned_addr learned my addr 172.21.15.19:0/955609450 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:07:16.008 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:08.197+0000 7f6ce3400700 1 -- 172.21.15.19:0/955609450 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ce407ca60 con 0x7f6ce407deb0 2024-03-18T15:07:16.008 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:08.197+0000 7f6ce3400700 1 --2- 172.21.15.19:0/955609450 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6ce407deb0 0x7f6ce407c310 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f6cd40098a0 tx=0x7f6cd4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa3b1c7774aa1cd9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:07:16.008 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:08.198+0000 7f6ce2000700 1 -- 172.21.15.19:0/955609450 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6cd4004160 con 0x7f6ce407deb0 2024-03-18T15:07:16.008 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:08.198+0000 7f6ce2000700 1 -- 172.21.15.19:0/955609450 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6cd4004300 con 0x7f6ce407deb0 2024-03-18T15:07:16.009 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:08.198+0000 7f6ce2000700 1 -- 172.21.15.19:0/955609450 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6cd4011b20 con 0x7f6ce407deb0 2024-03-18T15:07:16.009 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:08.198+0000 7f6cea000700 1 -- 172.21.15.19:0/955609450 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6ce407deb0 msgr2=0x7f6ce407c310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:07:16.009 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:08.198+0000 7f6cea000700 1 --2- 172.21.15.19:0/955609450 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6ce407deb0 0x7f6ce407c310 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f6cd40098a0 tx=0x7f6cd4009bb0 comp rx=0 tx=0).stop 2024-03-18T15:07:16.009 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:08.198+0000 7f6cea000700 1 -- 172.21.15.19:0/955609450 shutdown_connections 2024-03-18T15:07:16.009 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:08.198+0000 7f6cea000700 1 --2- 172.21.15.19:0/955609450 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6ce407deb0 0x7f6ce407c310 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:07:16.009 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:08.198+0000 7f6cea000700 1 -- 172.21.15.19:0/955609450 >> 172.21.15.19:0/955609450 conn(0x7f6ce4103250 msgr2=0x7f6ce41056b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:07:16.009 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:08.199+0000 7f6cea000700 1 -- 172.21.15.19:0/955609450 shutdown_connections 2024-03-18T15:07:16.009 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:08.199+0000 7f6cea000700 1 -- 172.21.15.19:0/955609450 wait complete. 2024-03-18T15:07:16.009 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:08.200+0000 7f6cea000700 1 Processor -- start 2024-03-18T15:07:16.009 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:08.200+0000 7f6cea000700 1 -- start start 2024-03-18T15:07:16.009 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:08.200+0000 7f6cea000700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6ce407deb0 0x7f6ce419dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:07:16.009 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:08.200+0000 7f6cea000700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f6ce410e310 con 0x7f6ce407deb0 2024-03-18T15:07:16.009 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:08.201+0000 7f6ce3400700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6ce407deb0 0x7f6ce419dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:07:16.010 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:08.201+0000 7f6ce3400700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6ce407deb0 0x7f6ce419dec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:34096/0 (socket says 172.21.15.19:34096) 2024-03-18T15:07:16.010 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:08.201+0000 7f6ce3400700 1 -- 172.21.15.19:0/845657763 learned_addr learned my addr 172.21.15.19:0/845657763 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:07:16.010 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:08.201+0000 7f6ce3400700 1 -- 172.21.15.19:0/845657763 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ce419e400 con 0x7f6ce407deb0 2024-03-18T15:07:16.010 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:08.201+0000 7f6ce3400700 1 --2- 172.21.15.19:0/845657763 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6ce407deb0 0x7f6ce419dec0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f6cd4000c00 tx=0x7f6cd4011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:07:16.010 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:08.202+0000 7f6cdbe00700 1 -- 172.21.15.19:0/845657763 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6cd4003b60 con 0x7f6ce407deb0 2024-03-18T15:07:16.010 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:08.202+0000 7f6cdbe00700 1 -- 172.21.15.19:0/845657763 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6cd401a910 con 0x7f6ce407deb0 2024-03-18T15:07:16.010 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:08.202+0000 7f6cdbe00700 1 -- 172.21.15.19:0/845657763 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6cd4024930 con 0x7f6ce407deb0 2024-03-18T15:07:16.010 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:08.202+0000 7f6cea000700 1 -- 172.21.15.19:0/845657763 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ce419e690 con 0x7f6ce407deb0 2024-03-18T15:07:16.010 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:08.202+0000 7f6cea000700 1 -- 172.21.15.19:0/845657763 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ce419eaf0 con 0x7f6ce407deb0 2024-03-18T15:07:16.010 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:08.203+0000 7f6cdbe00700 1 -- 172.21.15.19:0/845657763 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 9) ==== 46119+0+0 (secure 0 0 0) 0x7f6cd4022020 con 0x7f6ce407deb0 2024-03-18T15:07:16.010 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:08.204+0000 7f6cdbe00700 1 --2- 172.21.15.19:0/845657763 >> [v2:172.21.15.19:6800/2776008962,v1:172.21.15.19:6801/2776008962] conn(0x7f6cc8038fd0 0x7f6cc803b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:07:16.010 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:08.204+0000 7f6cdbe00700 1 -- 172.21.15.19:0/845657763 --> [v2:172.21.15.19:6800/2776008962,v1:172.21.15.19:6801/2776008962] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f6cc803bba0 con 0x7f6cc8038fd0 2024-03-18T15:07:16.010 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:08.204+0000 7f6cdbe00700 1 -- 172.21.15.19:0/845657763 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f6cd404d1b0 con 0x7f6ce407deb0 2024-03-18T15:07:16.011 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:08.204+0000 7f6ce2a00700 1 -- 172.21.15.19:0/845657763 >> [v2:172.21.15.19:6800/2776008962,v1:172.21.15.19:6801/2776008962] conn(0x7f6cc8038fd0 msgr2=0x7f6cc803b490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.19:6800/2776008962 2024-03-18T15:07:16.011 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:08.204+0000 7f6ce2a00700 1 --2- 172.21.15.19:0/845657763 >> [v2:172.21.15.19:6800/2776008962,v1:172.21.15.19:6801/2776008962] conn(0x7f6cc8038fd0 0x7f6cc803b490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T15:07:16.011 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:08.405+0000 7f6ce2a00700 1 -- 172.21.15.19:0/845657763 >> [v2:172.21.15.19:6800/2776008962,v1:172.21.15.19:6801/2776008962] conn(0x7f6cc8038fd0 msgr2=0x7f6cc803b490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.19:6800/2776008962 2024-03-18T15:07:16.011 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:08.405+0000 7f6ce2a00700 1 --2- 172.21.15.19:0/845657763 >> [v2:172.21.15.19:6800/2776008962,v1:172.21.15.19:6801/2776008962] conn(0x7f6cc8038fd0 0x7f6cc803b490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T15:07:16.011 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:08.806+0000 7f6ce2a00700 1 -- 172.21.15.19:0/845657763 >> [v2:172.21.15.19:6800/2776008962,v1:172.21.15.19:6801/2776008962] conn(0x7f6cc8038fd0 msgr2=0x7f6cc803b490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.19:6800/2776008962 2024-03-18T15:07:16.011 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:08.806+0000 7f6ce2a00700 1 --2- 172.21.15.19:0/845657763 >> [v2:172.21.15.19:6800/2776008962,v1:172.21.15.19:6801/2776008962] conn(0x7f6cc8038fd0 0x7f6cc803b490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-18T15:07:16.011 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:09.607+0000 7f6ce2a00700 1 -- 172.21.15.19:0/845657763 >> [v2:172.21.15.19:6800/2776008962,v1:172.21.15.19:6801/2776008962] conn(0x7f6cc8038fd0 msgr2=0x7f6cc803b490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.19:6800/2776008962 2024-03-18T15:07:16.011 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:09.607+0000 7f6ce2a00700 1 --2- 172.21.15.19:0/845657763 >> [v2:172.21.15.19:6800/2776008962,v1:172.21.15.19:6801/2776008962] conn(0x7f6cc8038fd0 0x7f6cc803b490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-18T15:07:16.011 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:11.209+0000 7f6ce2a00700 1 -- 172.21.15.19:0/845657763 >> [v2:172.21.15.19:6800/2776008962,v1:172.21.15.19:6801/2776008962] conn(0x7f6cc8038fd0 msgr2=0x7f6cc803b490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.19:6800/2776008962 2024-03-18T15:07:16.011 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:11.209+0000 7f6ce2a00700 1 --2- 172.21.15.19:0/845657763 >> [v2:172.21.15.19:6800/2776008962,v1:172.21.15.19:6801/2776008962] conn(0x7f6cc8038fd0 0x7f6cc803b490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-18T15:07:16.011 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:14.155+0000 7f6cdbe00700 1 -- 172.21.15.19:0/845657763 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mgrmap(e 10) ==== 45886+0+0 (secure 0 0 0) 0x7f6cd402d3f0 con 0x7f6ce407deb0 2024-03-18T15:07:16.011 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:14.156+0000 7f6cdbe00700 1 -- 172.21.15.19:0/845657763 >> [v2:172.21.15.19:6800/2776008962,v1:172.21.15.19:6801/2776008962] conn(0x7f6cc8038fd0 msgr2=0x7f6cc803b490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T15:07:16.011 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:14.156+0000 7f6cdbe00700 1 --2- 172.21.15.19:0/845657763 >> [v2:172.21.15.19:6800/2776008962,v1:172.21.15.19:6801/2776008962] conn(0x7f6cc8038fd0 0x7f6cc803b490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:07:16.011 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:15.157+0000 7f6cdbe00700 1 -- 172.21.15.19:0/845657763 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mgrmap(e 11) ==== 46013+0+0 (secure 0 0 0) 0x7f6cd4017930 con 0x7f6ce407deb0 2024-03-18T15:07:16.012 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:15.157+0000 7f6cdbe00700 1 --2- 172.21.15.19:0/845657763 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f6cc803ca80 0x7f6cc803ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:07:16.012 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:15.157+0000 7f6cdbe00700 1 -- 172.21.15.19:0/845657763 --> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f6cd401ac60 con 0x7f6cc803ca80 2024-03-18T15:07:16.012 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:15.158+0000 7f6ce2a00700 1 --2- 172.21.15.19:0/845657763 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f6cc803ca80 0x7f6cc803ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:07:16.012 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:15.161+0000 7f6ce2a00700 1 --2- 172.21.15.19:0/845657763 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f6cc803ca80 0x7f6cc803ee70 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f6cd0003a10 tx=0x7f6cd0009270 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:07:16.012 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:15.162+0000 7f6cdbe00700 1 -- 172.21.15.19:0/845657763 <== mgr.14152 v2:172.21.15.19:6800/2017714236 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7f6cd401ac60 con 0x7f6cc803ca80 2024-03-18T15:07:16.012 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:15.170+0000 7f6cea000700 1 -- 172.21.15.19:0/845657763 --> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f6ce40693d0 con 0x7f6cc803ca80 2024-03-18T15:07:16.012 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:15.171+0000 7f6cdbe00700 1 -- 172.21.15.19:0/845657763 <== mgr.14152 v2:172.21.15.19:6800/2017714236 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7f6ce40693d0 con 0x7f6cc803ca80 2024-03-18T15:07:16.012 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:15.171+0000 7f6cea000700 1 -- 172.21.15.19:0/845657763 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f6cc803ca80 msgr2=0x7f6cc803ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:07:16.012 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:15.171+0000 7f6cea000700 1 --2- 172.21.15.19:0/845657763 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f6cc803ca80 0x7f6cc803ee70 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f6cd0003a10 tx=0x7f6cd0009270 comp rx=0 tx=0).stop 2024-03-18T15:07:16.012 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:15.171+0000 7f6cea000700 1 -- 172.21.15.19:0/845657763 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6ce407deb0 msgr2=0x7f6ce419dec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:07:16.012 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:15.171+0000 7f6cea000700 1 --2- 172.21.15.19:0/845657763 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6ce407deb0 0x7f6ce419dec0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f6cd4000c00 tx=0x7f6cd4011c10 comp rx=0 tx=0).stop 2024-03-18T15:07:16.012 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:15.171+0000 7f6cea000700 1 -- 172.21.15.19:0/845657763 shutdown_connections 2024-03-18T15:07:16.012 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:15.171+0000 7f6cea000700 1 --2- 172.21.15.19:0/845657763 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6ce407deb0 0x7f6ce419dec0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:07:16.013 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:15.171+0000 7f6cea000700 1 --2- 172.21.15.19:0/845657763 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f6cc803ca80 0x7f6cc803ee70 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:07:16.013 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:15.171+0000 7f6cea000700 1 --2- 172.21.15.19:0/845657763 >> [v2:172.21.15.19:6800/2776008962,v1:172.21.15.19:6801/2776008962] conn(0x7f6cc8038fd0 0x7f6cc803b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:07:16.013 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:15.171+0000 7f6cea000700 1 -- 172.21.15.19:0/845657763 >> 172.21.15.19:0/845657763 conn(0x7f6ce4103250 msgr2=0x7f6ce4103df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:07:16.013 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:15.172+0000 7f6cea000700 1 -- 172.21.15.19:0/845657763 shutdown_connections 2024-03-18T15:07:16.013 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:15.172+0000 7f6cea000700 1 -- 172.21.15.19:0/845657763 wait complete. 2024-03-18T15:07:16.013 INFO:teuthology.orchestra.run.smithi019.stdout:mgr epoch 9 is available 2024-03-18T15:07:16.013 INFO:teuthology.orchestra.run.smithi019.stdout:Generating a dashboard self-signed certificate... 2024-03-18T15:07:16.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:16 smithi019 ceph-mon[31522]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-18T15:07:16.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:16 smithi019 ceph-mon[31522]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-18T15:07:17.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:17 smithi019 ceph-mon[31522]: mgrmap e12: x(active, since 2s) 2024-03-18T15:07:17.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:17 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:07:18.114 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-18T15:07:18.114 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:17.028+0000 7f4455a00700 1 Processor -- start 2024-03-18T15:07:18.114 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:17.028+0000 7f4455a00700 1 -- start start 2024-03-18T15:07:18.114 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:17.029+0000 7f4455a00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4450108740 0x7f445010ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:07:18.114 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:17.029+0000 7f4455a00700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f445007bea0 con 0x7f4450108740 2024-03-18T15:07:18.114 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:17.029+0000 7f444ea00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4450108740 0x7f445010ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:07:18.114 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:17.029+0000 7f444ea00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4450108740 0x7f445010ab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:42440/0 (socket says 172.21.15.19:42440) 2024-03-18T15:07:18.114 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:17.029+0000 7f444ea00700 1 -- 172.21.15.19:0/1571983567 learned_addr learned my addr 172.21.15.19:0/1571983567 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:07:18.114 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:17.030+0000 7f444ea00700 1 -- 172.21.15.19:0/1571983567 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f445010b070 con 0x7f4450108740 2024-03-18T15:07:18.114 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:17.030+0000 7f444ea00700 1 --2- 172.21.15.19:0/1571983567 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4450108740 0x7f445010ab30 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f44400098a0 tx=0x7f4440009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84ee737c72086b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:07:18.114 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:17.030+0000 7f444d600700 1 -- 172.21.15.19:0/1571983567 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4440004160 con 0x7f4450108740 2024-03-18T15:07:18.115 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:17.030+0000 7f444d600700 1 -- 172.21.15.19:0/1571983567 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4440004300 con 0x7f4450108740 2024-03-18T15:07:18.115 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:17.030+0000 7f444d600700 1 -- 172.21.15.19:0/1571983567 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4440011b20 con 0x7f4450108740 2024-03-18T15:07:18.115 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:17.031+0000 7f4455a00700 1 -- 172.21.15.19:0/1571983567 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4450108740 msgr2=0x7f445010ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:07:18.115 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:17.031+0000 7f4455a00700 1 --2- 172.21.15.19:0/1571983567 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4450108740 0x7f445010ab30 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f44400098a0 tx=0x7f4440009bb0 comp rx=0 tx=0).stop 2024-03-18T15:07:18.115 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:17.031+0000 7f4455a00700 1 -- 172.21.15.19:0/1571983567 shutdown_connections 2024-03-18T15:07:18.115 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:17.031+0000 7f4455a00700 1 --2- 172.21.15.19:0/1571983567 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4450108740 0x7f445010ab30 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:07:18.115 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:17.031+0000 7f4455a00700 1 -- 172.21.15.19:0/1571983567 >> 172.21.15.19:0/1571983567 conn(0x7f4450102130 msgr2=0x7f44501045b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:07:18.115 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:17.031+0000 7f4455a00700 1 -- 172.21.15.19:0/1571983567 shutdown_connections 2024-03-18T15:07:18.115 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:17.032+0000 7f4455a00700 1 -- 172.21.15.19:0/1571983567 wait complete. 2024-03-18T15:07:18.115 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:17.033+0000 7f4455a00700 1 Processor -- start 2024-03-18T15:07:18.115 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:17.034+0000 7f4455a00700 1 -- start start 2024-03-18T15:07:18.115 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:17.034+0000 7f4455a00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4450108740 0x7f4450107fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:07:18.115 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:17.034+0000 7f4455a00700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f445010c0e0 con 0x7f4450108740 2024-03-18T15:07:18.116 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:17.034+0000 7f444ea00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4450108740 0x7f4450107fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:07:18.116 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:17.034+0000 7f444ea00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4450108740 0x7f4450107fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:42442/0 (socket says 172.21.15.19:42442) 2024-03-18T15:07:18.116 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:17.034+0000 7f444ea00700 1 -- 172.21.15.19:0/3973501865 learned_addr learned my addr 172.21.15.19:0/3973501865 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:07:18.116 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:17.035+0000 7f444ea00700 1 -- 172.21.15.19:0/3973501865 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4450108500 con 0x7f4450108740 2024-03-18T15:07:18.116 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:17.035+0000 7f444ea00700 1 --2- 172.21.15.19:0/3973501865 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4450108740 0x7f4450107fc0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f4440011b60 tx=0x7f4440011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:07:18.116 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:17.035+0000 7f4447400700 1 -- 172.21.15.19:0/3973501865 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4440003b60 con 0x7f4450108740 2024-03-18T15:07:18.116 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:17.036+0000 7f4455a00700 1 -- 172.21.15.19:0/3973501865 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44501066f0 con 0x7f4450108740 2024-03-18T15:07:18.116 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:17.036+0000 7f4447400700 1 -- 172.21.15.19:0/3973501865 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f444001a910 con 0x7f4450108740 2024-03-18T15:07:18.116 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:17.036+0000 7f4447400700 1 -- 172.21.15.19:0/3973501865 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4440024930 con 0x7f4450108740 2024-03-18T15:07:18.116 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:17.036+0000 7f4455a00700 1 -- 172.21.15.19:0/3973501865 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4450106bb0 con 0x7f4450108740 2024-03-18T15:07:18.116 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:17.037+0000 7f4447400700 1 -- 172.21.15.19:0/3973501865 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 12) ==== 46119+0+0 (secure 0 0 0) 0x7f4440022020 con 0x7f4450108740 2024-03-18T15:07:18.116 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:17.037+0000 7f4447400700 1 --2- 172.21.15.19:0/3973501865 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f4434038fd0 0x7f443403b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:07:18.116 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:17.037+0000 7f4447400700 1 -- 172.21.15.19:0/3973501865 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f444004d240 con 0x7f4450108740 2024-03-18T15:07:18.117 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:17.037+0000 7f4455a00700 1 -- 172.21.15.19:0/3973501865 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44500693d0 con 0x7f4450108740 2024-03-18T15:07:18.117 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:17.038+0000 7f444e000700 1 --2- 172.21.15.19:0/3973501865 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f4434038fd0 0x7f443403b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:07:18.117 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:17.039+0000 7f444e000700 1 --2- 172.21.15.19:0/3973501865 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f4434038fd0 0x7f443403b490 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f443c006fd0 tx=0x7f443c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:07:18.117 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:17.044+0000 7f4447400700 1 -- 172.21.15.19:0/3973501865 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f444001a430 con 0x7f4450108740 2024-03-18T15:07:18.117 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:17.299+0000 7f4455a00700 1 -- 172.21.15.19:0/3973501865 --> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7f44501074e0 con 0x7f4434038fd0 2024-03-18T15:07:18.117 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:17.411+0000 7f4447400700 1 -- 172.21.15.19:0/3973501865 <== mgr.14152 v2:172.21.15.19:6800/2017714236 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7f44501074e0 con 0x7f4434038fd0 2024-03-18T15:07:18.117 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:17.414+0000 7f4455a00700 1 -- 172.21.15.19:0/3973501865 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f4434038fd0 msgr2=0x7f443403b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:07:18.117 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:17.414+0000 7f4455a00700 1 --2- 172.21.15.19:0/3973501865 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f4434038fd0 0x7f443403b490 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f443c006fd0 tx=0x7f443c006e40 comp rx=0 tx=0).stop 2024-03-18T15:07:18.117 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:17.414+0000 7f4455a00700 1 -- 172.21.15.19:0/3973501865 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4450108740 msgr2=0x7f4450107fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:07:18.117 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:17.414+0000 7f4455a00700 1 --2- 172.21.15.19:0/3973501865 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4450108740 0x7f4450107fc0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f4440011b60 tx=0x7f4440011c40 comp rx=0 tx=0).stop 2024-03-18T15:07:18.117 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:17.414+0000 7f4455a00700 1 -- 172.21.15.19:0/3973501865 shutdown_connections 2024-03-18T15:07:18.117 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:17.414+0000 7f4455a00700 1 --2- 172.21.15.19:0/3973501865 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f4434038fd0 0x7f443403b490 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:07:18.117 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:17.414+0000 7f4455a00700 1 --2- 172.21.15.19:0/3973501865 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4450108740 0x7f4450107fc0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:07:18.118 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:17.414+0000 7f4455a00700 1 -- 172.21.15.19:0/3973501865 >> 172.21.15.19:0/3973501865 conn(0x7f4450102130 msgr2=0x7f4450102cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:07:18.118 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:17.414+0000 7f4455a00700 1 -- 172.21.15.19:0/3973501865 shutdown_connections 2024-03-18T15:07:18.118 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:17.414+0000 7f4455a00700 1 -- 172.21.15.19:0/3973501865 wait complete. 2024-03-18T15:07:18.118 INFO:teuthology.orchestra.run.smithi019.stdout:Creating initial admin user... 2024-03-18T15:07:18.698 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:18 smithi019 ceph-mon[31522]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:07:18.699 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:18 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:07:18.699 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:18 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:07:19.953 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$SF4Ysi/oppSeDwsC2lEzgO5vAXJb.HkWKlJCVu4miavnjTQKSyts.", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1710774439, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-18T15:07:19.954 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:18.697+0000 7f2bfca00700 1 Processor -- start 2024-03-18T15:07:19.954 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:18.698+0000 7f2bfca00700 1 -- start start 2024-03-18T15:07:19.954 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:18.698+0000 7f2bfca00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2bf810a0f0 0x7f2bf810a510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:07:19.954 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:18.698+0000 7f2bfca00700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f2bf810aae0 con 0x7f2bf810a0f0 2024-03-18T15:07:19.955 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:18.699+0000 7f2bf5e00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2bf810a0f0 0x7f2bf810a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:07:19.955 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:18.699+0000 7f2bf5e00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2bf810a0f0 0x7f2bf810a510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:42444/0 (socket says 172.21.15.19:42444) 2024-03-18T15:07:19.955 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:18.699+0000 7f2bf5e00700 1 -- 172.21.15.19:0/3425111643 learned_addr learned my addr 172.21.15.19:0/3425111643 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:07:19.955 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:18.699+0000 7f2bf5e00700 1 -- 172.21.15.19:0/3425111643 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bf810ac60 con 0x7f2bf810a0f0 2024-03-18T15:07:19.955 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:18.700+0000 7f2bf5e00700 1 --2- 172.21.15.19:0/3425111643 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2bf810a0f0 0x7f2bf810a510 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f2be8009ab0 tx=0x7f2be8009dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56a713570b327ba1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:07:19.955 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:18.700+0000 7f2bf5400700 1 -- 172.21.15.19:0/3425111643 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2be8004160 con 0x7f2bf810a0f0 2024-03-18T15:07:19.955 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:18.700+0000 7f2bf5400700 1 -- 172.21.15.19:0/3425111643 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2be800bcc0 con 0x7f2bf810a0f0 2024-03-18T15:07:19.956 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:18.700+0000 7f2bf5400700 1 -- 172.21.15.19:0/3425111643 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2be8003d00 con 0x7f2bf810a0f0 2024-03-18T15:07:19.958 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:18.701+0000 7f2bfca00700 1 -- 172.21.15.19:0/3425111643 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2bf810a0f0 msgr2=0x7f2bf810a510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:07:19.958 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:18.701+0000 7f2bfca00700 1 --2- 172.21.15.19:0/3425111643 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2bf810a0f0 0x7f2bf810a510 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f2be8009ab0 tx=0x7f2be8009dc0 comp rx=0 tx=0).stop 2024-03-18T15:07:19.958 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:18.701+0000 7f2bfca00700 1 -- 172.21.15.19:0/3425111643 shutdown_connections 2024-03-18T15:07:19.958 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:18.701+0000 7f2bfca00700 1 --2- 172.21.15.19:0/3425111643 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2bf810a0f0 0x7f2bf810a510 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:07:19.958 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:18.701+0000 7f2bfca00700 1 -- 172.21.15.19:0/3425111643 >> 172.21.15.19:0/3425111643 conn(0x7f2bf807d2d0 msgr2=0x7f2bf807d6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:07:19.958 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:18.701+0000 7f2bfca00700 1 -- 172.21.15.19:0/3425111643 shutdown_connections 2024-03-18T15:07:19.958 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:18.701+0000 7f2bfca00700 1 -- 172.21.15.19:0/3425111643 wait complete. 2024-03-18T15:07:19.958 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:18.702+0000 7f2bfca00700 1 Processor -- start 2024-03-18T15:07:19.958 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:18.702+0000 7f2bfca00700 1 -- start start 2024-03-18T15:07:19.958 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:18.703+0000 7f2bfca00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2bf810a0f0 0x7f2bf819df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:07:19.958 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:18.703+0000 7f2bfca00700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f2bf810e340 con 0x7f2bf810a0f0 2024-03-18T15:07:19.958 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:18.703+0000 7f2bf5e00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2bf810a0f0 0x7f2bf819df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:07:19.959 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:18.703+0000 7f2bf5e00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2bf810a0f0 0x7f2bf819df30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:42458/0 (socket says 172.21.15.19:42458) 2024-03-18T15:07:19.959 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:18.703+0000 7f2bf5e00700 1 -- 172.21.15.19:0/1022543973 learned_addr learned my addr 172.21.15.19:0/1022543973 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:07:19.959 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:18.703+0000 7f2bf5e00700 1 -- 172.21.15.19:0/1022543973 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bf819e470 con 0x7f2bf810a0f0 2024-03-18T15:07:19.959 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:18.704+0000 7f2bf5e00700 1 --2- 172.21.15.19:0/1022543973 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2bf810a0f0 0x7f2bf819df30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f2be800ba10 tx=0x7f2be800baf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:07:19.959 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:18.704+0000 7f2bef400700 1 -- 172.21.15.19:0/1022543973 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2be800be70 con 0x7f2bf810a0f0 2024-03-18T15:07:19.959 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:18.704+0000 7f2bef400700 1 -- 172.21.15.19:0/1022543973 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2be801a430 con 0x7f2bf810a0f0 2024-03-18T15:07:19.959 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:18.704+0000 7f2bef400700 1 -- 172.21.15.19:0/1022543973 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2be80115b0 con 0x7f2bf810a0f0 2024-03-18T15:07:19.961 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:18.704+0000 7f2bfca00700 1 -- 172.21.15.19:0/1022543973 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2bf819e700 con 0x7f2bf810a0f0 2024-03-18T15:07:19.961 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:18.704+0000 7f2bfca00700 1 -- 172.21.15.19:0/1022543973 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2bf81a13f0 con 0x7f2bf810a0f0 2024-03-18T15:07:19.961 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:18.706+0000 7f2bfca00700 1 -- 172.21.15.19:0/1022543973 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2bf80693d0 con 0x7f2bf810a0f0 2024-03-18T15:07:19.962 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:18.706+0000 7f2bef400700 1 -- 172.21.15.19:0/1022543973 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 12) ==== 46119+0+0 (secure 0 0 0) 0x7f2be801a5e0 con 0x7f2bf810a0f0 2024-03-18T15:07:19.962 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:18.706+0000 7f2bef400700 1 --2- 172.21.15.19:0/1022543973 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f2bdc038fc0 0x7f2bdc03b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:07:19.962 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:18.706+0000 7f2bef400700 1 -- 172.21.15.19:0/1022543973 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f2be804d1a0 con 0x7f2bf810a0f0 2024-03-18T15:07:19.962 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:18.707+0000 7f2bed400700 1 --2- 172.21.15.19:0/1022543973 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f2bdc038fc0 0x7f2bdc03b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:07:19.962 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:18.707+0000 7f2bed400700 1 --2- 172.21.15.19:0/1022543973 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f2bdc038fc0 0x7f2bdc03b480 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2be0006fd0 tx=0x7f2be0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:07:19.962 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:18.711+0000 7f2bef400700 1 -- 172.21.15.19:0/1022543973 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2be8004500 con 0x7f2bf810a0f0 2024-03-18T15:07:19.962 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:18.967+0000 7f2bfca00700 1 -- 172.21.15.19:0/1022543973 --> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7f2bf819ebe0 con 0x7f2bdc038fc0 2024-03-18T15:07:19.962 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:19.229+0000 7f2bef400700 1 -- 172.21.15.19:0/1022543973 <== mgr.14152 v2:172.21.15.19:6800/2017714236 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7f2bf819ebe0 con 0x7f2bdc038fc0 2024-03-18T15:07:19.962 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:19.233+0000 7f2bfca00700 1 -- 172.21.15.19:0/1022543973 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f2bdc038fc0 msgr2=0x7f2bdc03b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:07:19.962 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:19.233+0000 7f2bfca00700 1 --2- 172.21.15.19:0/1022543973 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f2bdc038fc0 0x7f2bdc03b480 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2be0006fd0 tx=0x7f2be0006e40 comp rx=0 tx=0).stop 2024-03-18T15:07:19.962 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:19.233+0000 7f2bfca00700 1 -- 172.21.15.19:0/1022543973 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2bf810a0f0 msgr2=0x7f2bf819df30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:07:19.962 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:19.233+0000 7f2bfca00700 1 --2- 172.21.15.19:0/1022543973 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2bf810a0f0 0x7f2bf819df30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f2be800ba10 tx=0x7f2be800baf0 comp rx=0 tx=0).stop 2024-03-18T15:07:19.963 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:19.233+0000 7f2bfca00700 1 -- 172.21.15.19:0/1022543973 shutdown_connections 2024-03-18T15:07:19.963 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:19.233+0000 7f2bfca00700 1 --2- 172.21.15.19:0/1022543973 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f2bdc038fc0 0x7f2bdc03b480 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:07:19.963 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:19.233+0000 7f2bfca00700 1 --2- 172.21.15.19:0/1022543973 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2bf810a0f0 0x7f2bf819df30 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:07:19.963 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:19.233+0000 7f2bfca00700 1 -- 172.21.15.19:0/1022543973 >> 172.21.15.19:0/1022543973 conn(0x7f2bf807d2d0 msgr2=0x7f2bf8107d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:07:19.963 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:19.233+0000 7f2bfca00700 1 -- 172.21.15.19:0/1022543973 shutdown_connections 2024-03-18T15:07:19.963 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:19.233+0000 7f2bfca00700 1 -- 172.21.15.19:0/1022543973 wait complete. 2024-03-18T15:07:19.963 INFO:teuthology.orchestra.run.smithi019.stdout:Fetching dashboard port number... 2024-03-18T15:07:20.156 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:19 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:07:20.156 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:19 smithi019 ceph-mon[31522]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:07:20.157 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:19 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:07:20.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:20 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/1842135812' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-18T15:07:21.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:21 smithi019 ceph-mon[31522]: mgrmap e13: x(active, since 6s) 2024-03-18T15:07:22.270 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout 8443 2024-03-18T15:07:22.271 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:20.615+0000 7fe9f8200700 1 Processor -- start 2024-03-18T15:07:22.271 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:20.615+0000 7fe9f8200700 1 -- start start 2024-03-18T15:07:22.271 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:20.616+0000 7fe9f8200700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe9f007deb0 0x7fe9f007c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:07:22.271 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:20.616+0000 7fe9f8200700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fe9f007c8e0 con 0x7fe9f007deb0 2024-03-18T15:07:22.271 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:20.616+0000 7fe9f5800700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe9f007deb0 0x7fe9f007c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:07:22.279 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:20.616+0000 7fe9f5800700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe9f007deb0 0x7fe9f007c310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:42466/0 (socket says 172.21.15.19:42466) 2024-03-18T15:07:22.279 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:20.616+0000 7fe9f5800700 1 -- 172.21.15.19:0/1848499611 learned_addr learned my addr 172.21.15.19:0/1848499611 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:07:22.279 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:20.617+0000 7fe9f5800700 1 -- 172.21.15.19:0/1848499611 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9f007ca60 con 0x7fe9f007deb0 2024-03-18T15:07:22.279 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:20.617+0000 7fe9f5800700 1 --2- 172.21.15.19:0/1848499611 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe9f007deb0 0x7fe9f007c310 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe9e40098a0 tx=0x7fe9e4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e3298871eafdb7c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:07:22.279 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:20.617+0000 7fe9efe00700 1 -- 172.21.15.19:0/1848499611 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe9e4004160 con 0x7fe9f007deb0 2024-03-18T15:07:22.279 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:20.617+0000 7fe9efe00700 1 -- 172.21.15.19:0/1848499611 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9e4004300 con 0x7fe9f007deb0 2024-03-18T15:07:22.279 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:20.617+0000 7fe9efe00700 1 -- 172.21.15.19:0/1848499611 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe9e4011b20 con 0x7fe9f007deb0 2024-03-18T15:07:22.279 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:20.618+0000 7fe9f8200700 1 -- 172.21.15.19:0/1848499611 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe9f007deb0 msgr2=0x7fe9f007c310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:07:22.279 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:20.618+0000 7fe9f8200700 1 --2- 172.21.15.19:0/1848499611 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe9f007deb0 0x7fe9f007c310 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe9e40098a0 tx=0x7fe9e4009bb0 comp rx=0 tx=0).stop 2024-03-18T15:07:22.279 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:20.618+0000 7fe9f8200700 1 -- 172.21.15.19:0/1848499611 shutdown_connections 2024-03-18T15:07:22.279 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:20.618+0000 7fe9f8200700 1 --2- 172.21.15.19:0/1848499611 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe9f007deb0 0x7fe9f007c310 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:07:22.279 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:20.618+0000 7fe9f8200700 1 -- 172.21.15.19:0/1848499611 >> 172.21.15.19:0/1848499611 conn(0x7fe9f0103250 msgr2=0x7fe9f01056b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:07:22.280 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:20.618+0000 7fe9f8200700 1 -- 172.21.15.19:0/1848499611 shutdown_connections 2024-03-18T15:07:22.280 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:20.618+0000 7fe9f8200700 1 -- 172.21.15.19:0/1848499611 wait complete. 2024-03-18T15:07:22.280 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:20.619+0000 7fe9f8200700 1 Processor -- start 2024-03-18T15:07:22.280 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:20.619+0000 7fe9f8200700 1 -- start start 2024-03-18T15:07:22.280 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:20.620+0000 7fe9f8200700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe9f007deb0 0x7fe9f019de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:07:22.280 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:20.620+0000 7fe9f8200700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fe9f010e310 con 0x7fe9f007deb0 2024-03-18T15:07:22.280 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:20.620+0000 7fe9f5800700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe9f007deb0 0x7fe9f019de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:07:22.280 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:20.620+0000 7fe9f5800700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe9f007deb0 0x7fe9f019de90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:42470/0 (socket says 172.21.15.19:42470) 2024-03-18T15:07:22.280 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:20.620+0000 7fe9f5800700 1 -- 172.21.15.19:0/1842135812 learned_addr learned my addr 172.21.15.19:0/1842135812 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:07:22.280 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:20.620+0000 7fe9f5800700 1 -- 172.21.15.19:0/1842135812 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9f019e3d0 con 0x7fe9f007deb0 2024-03-18T15:07:22.280 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:20.621+0000 7fe9f5800700 1 --2- 172.21.15.19:0/1842135812 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe9f007deb0 0x7fe9f019de90 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe9e4000c00 tx=0x7fe9e4011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:07:22.280 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:20.621+0000 7fe9ee000700 1 -- 172.21.15.19:0/1842135812 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe9e4003b60 con 0x7fe9f007deb0 2024-03-18T15:07:22.280 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:20.621+0000 7fe9ee000700 1 -- 172.21.15.19:0/1842135812 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9e401a910 con 0x7fe9f007deb0 2024-03-18T15:07:22.280 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:20.621+0000 7fe9ee000700 1 -- 172.21.15.19:0/1842135812 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe9e4024930 con 0x7fe9f007deb0 2024-03-18T15:07:22.280 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:20.621+0000 7fe9f8200700 1 -- 172.21.15.19:0/1842135812 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9f019e660 con 0x7fe9f007deb0 2024-03-18T15:07:22.281 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:20.621+0000 7fe9f8200700 1 -- 172.21.15.19:0/1842135812 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9f019eac0 con 0x7fe9f007deb0 2024-03-18T15:07:22.281 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:20.622+0000 7fe9ee000700 1 -- 172.21.15.19:0/1842135812 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 12) ==== 46119+0+0 (secure 0 0 0) 0x7fe9e4022020 con 0x7fe9f007deb0 2024-03-18T15:07:22.281 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:20.622+0000 7fe9f8200700 1 -- 172.21.15.19:0/1842135812 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9f0063900 con 0x7fe9f007deb0 2024-03-18T15:07:22.281 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:20.623+0000 7fe9ee000700 1 --2- 172.21.15.19:0/1842135812 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fe9d8038fd0 0x7fe9d803b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:07:22.281 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:20.623+0000 7fe9ee000700 1 -- 172.21.15.19:0/1842135812 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fe9e404d120 con 0x7fe9f007deb0 2024-03-18T15:07:22.281 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:20.623+0000 7fe9f4e00700 1 --2- 172.21.15.19:0/1842135812 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fe9d8038fd0 0x7fe9d803b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:07:22.281 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:20.624+0000 7fe9f4e00700 1 --2- 172.21.15.19:0/1842135812 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fe9d8038fd0 0x7fe9d803b490 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fe9dc006fd0 tx=0x7fe9dc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:07:22.281 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:20.628+0000 7fe9ee000700 1 -- 172.21.15.19:0/1842135812 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe9e400b3e0 con 0x7fe9f007deb0 2024-03-18T15:07:22.281 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:20.866+0000 7fe9f8200700 1 -- 172.21.15.19:0/1842135812 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7fe9f01a1550 con 0x7fe9f007deb0 2024-03-18T15:07:22.281 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:20.866+0000 7fe9ee000700 1 -- 172.21.15.19:0/1842135812 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7fe9e4016260 con 0x7fe9f007deb0 2024-03-18T15:07:22.281 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:20.869+0000 7fe9f8200700 1 -- 172.21.15.19:0/1842135812 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fe9d8038fd0 msgr2=0x7fe9d803b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:07:22.281 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:20.869+0000 7fe9f8200700 1 --2- 172.21.15.19:0/1842135812 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fe9d8038fd0 0x7fe9d803b490 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fe9dc006fd0 tx=0x7fe9dc006e40 comp rx=0 tx=0).stop 2024-03-18T15:07:22.281 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:20.869+0000 7fe9f8200700 1 -- 172.21.15.19:0/1842135812 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe9f007deb0 msgr2=0x7fe9f019de90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:07:22.281 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:20.869+0000 7fe9f8200700 1 --2- 172.21.15.19:0/1842135812 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe9f007deb0 0x7fe9f019de90 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe9e4000c00 tx=0x7fe9e4011c10 comp rx=0 tx=0).stop 2024-03-18T15:07:22.281 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:20.869+0000 7fe9f8200700 1 -- 172.21.15.19:0/1842135812 shutdown_connections 2024-03-18T15:07:22.282 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:20.869+0000 7fe9f8200700 1 --2- 172.21.15.19:0/1842135812 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fe9d8038fd0 0x7fe9d803b490 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:07:22.282 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:20.869+0000 7fe9f8200700 1 --2- 172.21.15.19:0/1842135812 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe9f007deb0 0x7fe9f019de90 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:07:22.282 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:20.869+0000 7fe9f8200700 1 -- 172.21.15.19:0/1842135812 >> 172.21.15.19:0/1842135812 conn(0x7fe9f0103250 msgr2=0x7fe9f0103dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:07:22.282 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:20.869+0000 7fe9f8200700 1 -- 172.21.15.19:0/1842135812 shutdown_connections 2024-03-18T15:07:22.282 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:20.869+0000 7fe9f8200700 1 -- 172.21.15.19:0/1842135812 wait complete. 2024-03-18T15:07:22.305 INFO:teuthology.orchestra.run.smithi019.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-18T15:07:22.306 INFO:teuthology.orchestra.run.smithi019.stdout:systemctl: stdout disabled 2024-03-18T15:07:22.329 INFO:teuthology.orchestra.run.smithi019.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-18T15:07:22.330 INFO:teuthology.orchestra.run.smithi019.stdout:systemctl: stdout inactive 2024-03-18T15:07:22.330 INFO:teuthology.orchestra.run.smithi019.stdout:firewalld.service is not enabled 2024-03-18T15:07:22.330 INFO:teuthology.orchestra.run.smithi019.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-18T15:07:22.337 INFO:teuthology.orchestra.run.smithi019.stdout:Ceph Dashboard is now available at: 2024-03-18T15:07:22.337 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-18T15:07:22.337 INFO:teuthology.orchestra.run.smithi019.stdout: URL: https://smithi019.front.sepia.ceph.com:8443/ 2024-03-18T15:07:22.337 INFO:teuthology.orchestra.run.smithi019.stdout: User: admin 2024-03-18T15:07:22.337 INFO:teuthology.orchestra.run.smithi019.stdout: Password: 74lgql0kwu 2024-03-18T15:07:22.337 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-18T15:07:22.338 INFO:teuthology.orchestra.run.smithi019.stdout:Saving cluster configuration to /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/config directory 2024-03-18T15:07:22.340 INFO:teuthology.orchestra.run.smithi019.stdout:Enabling autotune for osd_memory_target 2024-03-18T15:07:23.810 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:22.918+0000 7f7aaaa00700 1 Processor -- start 2024-03-18T15:07:23.810 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:22.918+0000 7f7aaaa00700 1 -- start start 2024-03-18T15:07:23.810 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:22.918+0000 7f7aaaa00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7aa410a0f0 0x7f7aa410a510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:07:23.810 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:22.919+0000 7f7aaaa00700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f7aa410aae0 con 0x7f7aa410a0f0 2024-03-18T15:07:23.810 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:22.919+0000 7f7aa3e00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7aa410a0f0 0x7f7aa410a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:07:23.810 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:22.919+0000 7f7aa3e00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7aa410a0f0 0x7f7aa410a510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:36666/0 (socket says 172.21.15.19:36666) 2024-03-18T15:07:23.810 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:22.919+0000 7f7aa3e00700 1 -- 172.21.15.19:0/663758475 learned_addr learned my addr 172.21.15.19:0/663758475 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:07:23.810 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:22.920+0000 7f7aa3e00700 1 -- 172.21.15.19:0/663758475 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7aa410ac60 con 0x7f7aa410a0f0 2024-03-18T15:07:23.810 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:22.920+0000 7f7aa3e00700 1 --2- 172.21.15.19:0/663758475 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7aa410a0f0 0x7f7aa410a510 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f7a940098a0 tx=0x7f7a94009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38b6c145e9bbf11 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:07:23.810 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:22.921+0000 7f7aa3400700 1 -- 172.21.15.19:0/663758475 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7a94004160 con 0x7f7aa410a0f0 2024-03-18T15:07:23.810 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:22.921+0000 7f7aa3400700 1 -- 172.21.15.19:0/663758475 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a94004300 con 0x7f7aa410a0f0 2024-03-18T15:07:23.811 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:22.921+0000 7f7aa3400700 1 -- 172.21.15.19:0/663758475 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7a94011b20 con 0x7f7aa410a0f0 2024-03-18T15:07:23.811 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:22.921+0000 7f7aaaa00700 1 -- 172.21.15.19:0/663758475 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7aa410a0f0 msgr2=0x7f7aa410a510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:07:23.811 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:22.921+0000 7f7aaaa00700 1 --2- 172.21.15.19:0/663758475 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7aa410a0f0 0x7f7aa410a510 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f7a940098a0 tx=0x7f7a94009bb0 comp rx=0 tx=0).stop 2024-03-18T15:07:23.811 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:22.922+0000 7f7aaaa00700 1 -- 172.21.15.19:0/663758475 shutdown_connections 2024-03-18T15:07:23.811 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:22.922+0000 7f7aaaa00700 1 --2- 172.21.15.19:0/663758475 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7aa410a0f0 0x7f7aa410a510 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:07:23.811 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:22.922+0000 7f7aaaa00700 1 -- 172.21.15.19:0/663758475 >> 172.21.15.19:0/663758475 conn(0x7f7aa407d2c0 msgr2=0x7f7aa407d6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:07:23.811 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:22.922+0000 7f7aaaa00700 1 -- 172.21.15.19:0/663758475 shutdown_connections 2024-03-18T15:07:23.811 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:22.922+0000 7f7aaaa00700 1 -- 172.21.15.19:0/663758475 wait complete. 2024-03-18T15:07:23.811 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:22.923+0000 7f7aaaa00700 1 Processor -- start 2024-03-18T15:07:23.811 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:22.923+0000 7f7aaaa00700 1 -- start start 2024-03-18T15:07:23.811 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:22.923+0000 7f7aaaa00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7aa410a0f0 0x7f7aa419de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:07:23.811 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:22.923+0000 7f7aaaa00700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f7aa410e300 con 0x7f7aa410a0f0 2024-03-18T15:07:23.811 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:22.924+0000 7f7aa3e00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7aa410a0f0 0x7f7aa419de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:07:23.812 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:22.924+0000 7f7aa3e00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7aa410a0f0 0x7f7aa419de50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:36672/0 (socket says 172.21.15.19:36672) 2024-03-18T15:07:23.812 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:22.924+0000 7f7aa3e00700 1 -- 172.21.15.19:0/2207568258 learned_addr learned my addr 172.21.15.19:0/2207568258 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:07:23.812 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:22.924+0000 7f7aa3e00700 1 -- 172.21.15.19:0/2207568258 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7aa419e390 con 0x7f7aa410a0f0 2024-03-18T15:07:23.812 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:22.924+0000 7f7aa3e00700 1 --2- 172.21.15.19:0/2207568258 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7aa410a0f0 0x7f7aa419de50 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f7a94011b60 tx=0x7f7a94011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:07:23.812 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:22.925+0000 7f7aa1600700 1 -- 172.21.15.19:0/2207568258 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7a94003b60 con 0x7f7aa410a0f0 2024-03-18T15:07:23.812 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:22.925+0000 7f7aa1600700 1 -- 172.21.15.19:0/2207568258 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a9401a910 con 0x7f7aa410a0f0 2024-03-18T15:07:23.812 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:22.925+0000 7f7aa1600700 1 -- 172.21.15.19:0/2207568258 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7a94024930 con 0x7f7aa410a0f0 2024-03-18T15:07:23.812 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:22.925+0000 7f7aaaa00700 1 -- 172.21.15.19:0/2207568258 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7aa419e620 con 0x7f7aa410a0f0 2024-03-18T15:07:23.812 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:22.925+0000 7f7aaaa00700 1 -- 172.21.15.19:0/2207568258 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7aa419ea80 con 0x7f7aa410a0f0 2024-03-18T15:07:23.812 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:22.926+0000 7f7aaaa00700 1 -- 172.21.15.19:0/2207568258 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7aa4063900 con 0x7f7aa410a0f0 2024-03-18T15:07:23.812 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:22.926+0000 7f7aa1600700 1 -- 172.21.15.19:0/2207568258 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f7a94022020 con 0x7f7aa410a0f0 2024-03-18T15:07:23.812 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:22.926+0000 7f7aa1600700 1 --2- 172.21.15.19:0/2207568258 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f7a88039070 0x7f7a8803b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:07:23.812 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:22.926+0000 7f7aa1600700 1 -- 172.21.15.19:0/2207568258 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f7a9404d200 con 0x7f7aa410a0f0 2024-03-18T15:07:23.813 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:22.927+0000 7f7a9be00700 1 --2- 172.21.15.19:0/2207568258 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f7a88039070 0x7f7a8803b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:07:23.813 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:22.927+0000 7f7a9be00700 1 --2- 172.21.15.19:0/2207568258 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f7a88039070 0x7f7a8803b530 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f7a8c006fd0 tx=0x7f7a8c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:07:23.813 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:22.932+0000 7f7aa1600700 1 -- 172.21.15.19:0/2207568258 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7a9400b3e0 con 0x7f7aa410a0f0 2024-03-18T15:07:23.813 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:23.160+0000 7f7aaaa00700 1 -- 172.21.15.19:0/2207568258 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7f7aa41a1450 con 0x7f7aa410a0f0 2024-03-18T15:07:23.813 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:23.161+0000 7f7aa1600700 1 -- 172.21.15.19:0/2207568258 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7f7a9402de10 con 0x7f7aa410a0f0 2024-03-18T15:07:23.813 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:23.164+0000 7f7aaaa00700 1 -- 172.21.15.19:0/2207568258 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f7a88039070 msgr2=0x7f7a8803b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:07:23.813 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:23.164+0000 7f7aaaa00700 1 --2- 172.21.15.19:0/2207568258 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f7a88039070 0x7f7a8803b530 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f7a8c006fd0 tx=0x7f7a8c006e40 comp rx=0 tx=0).stop 2024-03-18T15:07:23.813 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:23.164+0000 7f7aaaa00700 1 -- 172.21.15.19:0/2207568258 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7aa410a0f0 msgr2=0x7f7aa419de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:07:23.813 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:23.164+0000 7f7aaaa00700 1 --2- 172.21.15.19:0/2207568258 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7aa410a0f0 0x7f7aa419de50 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f7a94011b60 tx=0x7f7a94011c40 comp rx=0 tx=0).stop 2024-03-18T15:07:23.813 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:23.164+0000 7f7aaaa00700 1 -- 172.21.15.19:0/2207568258 shutdown_connections 2024-03-18T15:07:23.813 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:23.164+0000 7f7aaaa00700 1 --2- 172.21.15.19:0/2207568258 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f7a88039070 0x7f7a8803b530 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:07:23.813 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:23.164+0000 7f7aaaa00700 1 --2- 172.21.15.19:0/2207568258 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7aa410a0f0 0x7f7aa419de50 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:07:23.813 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:23.164+0000 7f7aaaa00700 1 -- 172.21.15.19:0/2207568258 >> 172.21.15.19:0/2207568258 conn(0x7f7aa407d2c0 msgr2=0x7f7aa4107d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:07:23.813 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:23.165+0000 7f7aaaa00700 1 -- 172.21.15.19:0/2207568258 shutdown_connections 2024-03-18T15:07:23.814 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:23.165+0000 7f7aaaa00700 1 -- 172.21.15.19:0/2207568258 wait complete. 2024-03-18T15:07:25.444 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:24.429+0000 7fc312400700 1 Processor -- start 2024-03-18T15:07:25.444 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:24.429+0000 7fc312400700 1 -- start start 2024-03-18T15:07:25.445 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:24.430+0000 7fc312400700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc30c10a0f0 0x7fc30c10a510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:07:25.446 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:24.430+0000 7fc312400700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fc30c10aae0 con 0x7fc30c10a0f0 2024-03-18T15:07:25.446 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:24.430+0000 7fc30b400700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc30c10a0f0 0x7fc30c10a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:07:25.446 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:24.430+0000 7fc30b400700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc30c10a0f0 0x7fc30c10a510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:36688/0 (socket says 172.21.15.19:36688) 2024-03-18T15:07:25.446 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:24.430+0000 7fc30b400700 1 -- 172.21.15.19:0/600068722 learned_addr learned my addr 172.21.15.19:0/600068722 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:07:25.446 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:24.430+0000 7fc30b400700 1 -- 172.21.15.19:0/600068722 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc30c10ac60 con 0x7fc30c10a0f0 2024-03-18T15:07:25.446 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:24.431+0000 7fc30b400700 1 --2- 172.21.15.19:0/600068722 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc30c10a0f0 0x7fc30c10a510 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc2fc0098a0 tx=0x7fc2fc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15638e202085a5a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:07:25.446 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:24.431+0000 7fc30aa00700 1 -- 172.21.15.19:0/600068722 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc2fc004160 con 0x7fc30c10a0f0 2024-03-18T15:07:25.448 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:24.431+0000 7fc30aa00700 1 -- 172.21.15.19:0/600068722 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc2fc004300 con 0x7fc30c10a0f0 2024-03-18T15:07:25.449 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:24.431+0000 7fc30aa00700 1 -- 172.21.15.19:0/600068722 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc2fc011b20 con 0x7fc30c10a0f0 2024-03-18T15:07:25.449 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:24.432+0000 7fc312400700 1 -- 172.21.15.19:0/600068722 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc30c10a0f0 msgr2=0x7fc30c10a510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:07:25.449 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:24.432+0000 7fc312400700 1 --2- 172.21.15.19:0/600068722 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc30c10a0f0 0x7fc30c10a510 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc2fc0098a0 tx=0x7fc2fc009bb0 comp rx=0 tx=0).stop 2024-03-18T15:07:25.449 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:24.432+0000 7fc312400700 1 -- 172.21.15.19:0/600068722 shutdown_connections 2024-03-18T15:07:25.449 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:24.432+0000 7fc312400700 1 --2- 172.21.15.19:0/600068722 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc30c10a0f0 0x7fc30c10a510 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:07:25.449 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:24.432+0000 7fc312400700 1 -- 172.21.15.19:0/600068722 >> 172.21.15.19:0/600068722 conn(0x7fc30c07d2c0 msgr2=0x7fc30c07d6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:07:25.449 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:24.432+0000 7fc312400700 1 -- 172.21.15.19:0/600068722 shutdown_connections 2024-03-18T15:07:25.449 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:24.432+0000 7fc312400700 1 -- 172.21.15.19:0/600068722 wait complete. 2024-03-18T15:07:25.449 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:24.433+0000 7fc312400700 1 Processor -- start 2024-03-18T15:07:25.449 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:24.434+0000 7fc312400700 1 -- start start 2024-03-18T15:07:25.449 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:24.434+0000 7fc312400700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc30c10a0f0 0x7fc30c19de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:07:25.449 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:24.434+0000 7fc312400700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fc30c10e300 con 0x7fc30c10a0f0 2024-03-18T15:07:25.449 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:24.434+0000 7fc30b400700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc30c10a0f0 0x7fc30c19de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:07:25.450 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:24.434+0000 7fc30b400700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc30c10a0f0 0x7fc30c19de50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:36698/0 (socket says 172.21.15.19:36698) 2024-03-18T15:07:25.450 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:24.434+0000 7fc30b400700 1 -- 172.21.15.19:0/1582125452 learned_addr learned my addr 172.21.15.19:0/1582125452 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:07:25.450 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:24.435+0000 7fc30b400700 1 -- 172.21.15.19:0/1582125452 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc30c19e390 con 0x7fc30c10a0f0 2024-03-18T15:07:25.450 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:24.435+0000 7fc30b400700 1 --2- 172.21.15.19:0/1582125452 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc30c10a0f0 0x7fc30c19de50 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fc2fc011b60 tx=0x7fc2fc011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:07:25.450 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:24.435+0000 7fc308c00700 1 -- 172.21.15.19:0/1582125452 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc2fc003b60 con 0x7fc30c10a0f0 2024-03-18T15:07:25.453 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:24.435+0000 7fc312400700 1 -- 172.21.15.19:0/1582125452 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc30c19e620 con 0x7fc30c10a0f0 2024-03-18T15:07:25.453 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:24.435+0000 7fc312400700 1 -- 172.21.15.19:0/1582125452 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc30c19ea80 con 0x7fc30c10a0f0 2024-03-18T15:07:25.453 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:24.435+0000 7fc308c00700 1 -- 172.21.15.19:0/1582125452 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc2fc01a910 con 0x7fc30c10a0f0 2024-03-18T15:07:25.453 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:24.435+0000 7fc308c00700 1 -- 172.21.15.19:0/1582125452 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc2fc0248c0 con 0x7fc30c10a0f0 2024-03-18T15:07:25.453 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:24.436+0000 7fc312400700 1 -- 172.21.15.19:0/1582125452 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2e8005360 con 0x7fc30c10a0f0 2024-03-18T15:07:25.453 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:24.436+0000 7fc308c00700 1 -- 172.21.15.19:0/1582125452 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7fc2fc022020 con 0x7fc30c10a0f0 2024-03-18T15:07:25.453 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:24.437+0000 7fc308c00700 1 --2- 172.21.15.19:0/1582125452 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fc2f0039070 0x7fc2f003b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:07:25.453 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:24.437+0000 7fc308c00700 1 -- 172.21.15.19:0/1582125452 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fc2fc04d1d0 con 0x7fc30c10a0f0 2024-03-18T15:07:25.453 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:24.437+0000 7fc302a00700 1 --2- 172.21.15.19:0/1582125452 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fc2f0039070 0x7fc2f003b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:07:25.454 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:24.438+0000 7fc302a00700 1 --2- 172.21.15.19:0/1582125452 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fc2f0039070 0x7fc2f003b530 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fc2f4006fd0 tx=0x7fc2f4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:07:25.454 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:24.443+0000 7fc308c00700 1 -- 172.21.15.19:0/1582125452 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc2fc00b3e0 con 0x7fc30c10a0f0 2024-03-18T15:07:25.454 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:24.787+0000 7fc312400700 1 -- 172.21.15.19:0/1582125452 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7fc2e8005650 con 0x7fc30c10a0f0 2024-03-18T15:07:25.454 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:24.791+0000 7fc308c00700 1 -- 172.21.15.19:0/1582125452 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) ==== 153+0+0 (secure 0 0 0) 0x7fc2fc024a60 con 0x7fc30c10a0f0 2024-03-18T15:07:25.454 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:24.794+0000 7fc312400700 1 -- 172.21.15.19:0/1582125452 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fc2f0039070 msgr2=0x7fc2f003b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:07:25.454 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:24.794+0000 7fc312400700 1 --2- 172.21.15.19:0/1582125452 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fc2f0039070 0x7fc2f003b530 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fc2f4006fd0 tx=0x7fc2f4006e40 comp rx=0 tx=0).stop 2024-03-18T15:07:25.454 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:24.794+0000 7fc312400700 1 -- 172.21.15.19:0/1582125452 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc30c10a0f0 msgr2=0x7fc30c19de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:07:25.454 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:24.794+0000 7fc312400700 1 --2- 172.21.15.19:0/1582125452 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc30c10a0f0 0x7fc30c19de50 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fc2fc011b60 tx=0x7fc2fc011c40 comp rx=0 tx=0).stop 2024-03-18T15:07:25.454 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:24.794+0000 7fc312400700 1 -- 172.21.15.19:0/1582125452 shutdown_connections 2024-03-18T15:07:25.454 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:24.794+0000 7fc312400700 1 --2- 172.21.15.19:0/1582125452 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fc2f0039070 0x7fc2f003b530 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:07:25.454 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:24.794+0000 7fc312400700 1 --2- 172.21.15.19:0/1582125452 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc30c10a0f0 0x7fc30c19de50 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:07:25.454 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:24.794+0000 7fc312400700 1 -- 172.21.15.19:0/1582125452 >> 172.21.15.19:0/1582125452 conn(0x7fc30c07d2c0 msgr2=0x7fc30c107d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:07:25.454 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:24.794+0000 7fc312400700 1 -- 172.21.15.19:0/1582125452 shutdown_connections 2024-03-18T15:07:25.455 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-03-18T15:07:24.794+0000 7fc312400700 1 -- 172.21.15.19:0/1582125452 wait complete. 2024-03-18T15:07:25.455 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-18T15:07:25.455 INFO:teuthology.orchestra.run.smithi019.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-18T15:07:25.455 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-18T15:07:25.455 INFO:teuthology.orchestra.run.smithi019.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-18T15:07:25.455 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-18T15:07:25.455 INFO:teuthology.orchestra.run.smithi019.stdout:Or, if you are only running a single cluster on this host: 2024-03-18T15:07:25.455 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-18T15:07:25.455 INFO:teuthology.orchestra.run.smithi019.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-18T15:07:25.455 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-18T15:07:25.455 INFO:teuthology.orchestra.run.smithi019.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-18T15:07:25.455 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-18T15:07:25.455 INFO:teuthology.orchestra.run.smithi019.stdout: ceph telemetry on 2024-03-18T15:07:25.455 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-18T15:07:25.455 INFO:teuthology.orchestra.run.smithi019.stdout:For more information see: 2024-03-18T15:07:25.456 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-18T15:07:25.456 INFO:teuthology.orchestra.run.smithi019.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-18T15:07:25.456 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-18T15:07:25.456 INFO:teuthology.orchestra.run.smithi019.stdout:Bootstrap complete. 2024-03-18T15:07:25.574 INFO:tasks.cephadm:Fetching config... 2024-03-18T15:07:25.574 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-03-18T15:07:25.574 DEBUG:teuthology.orchestra.run.smithi019:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-18T15:07:25.623 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-18T15:07:25.623 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-03-18T15:07:25.623 DEBUG:teuthology.orchestra.run.smithi019:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-18T15:07:25.703 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-18T15:07:25.703 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-03-18T15:07:25.704 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-18T15:07:25.838 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-18T15:07:25.838 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-03-18T15:07:25.838 DEBUG:teuthology.orchestra.run.smithi019:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-18T15:07:25.887 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-18T15:07:25.888 DEBUG:teuthology.orchestra.run.smithi019:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpvnxZ8ElXr5YJcsABYs3kNNr8JrxvGWZ0cwwo6ZISbL3BG3qVGSQ91ZY1Ke7RvL7DotwbdBASeuMZdl5Ib4WpEA+Z6sBKzjnULIWjYR+obO9epR2boWPqzQeb2Gq5P5t3cvNLjvPMbKcqsbL83FrRdRedXjOSOGZTmfKd6hN7XO8JlOp14rBmwj09gQ5WkWq4WKL+8S5S90mM99A1MauQQ/aeaGfNtFEplnm5uq9ygbKLWH6kGDXmcnISZqTmjUvQ4Z4ctJa+RtLl7w4e62oieSKDYZ8JJ3KBJv+jsU1YVyJZ5UMZ+kiconP/IBgXhPDgFnkoRnMNT569N5LckirlnZri046uGnvGj7yEY7GIP6nDKTcbeLehP8GH+S6WTeFyl5+p34ygU6VI3uJw8GzBHVqsvTvea8kVA0wlHNWx4dutGafWfqzjp1z3hEz0hdlxnKaS+6/3LoQY7fhrcS6uSqN7DiLuMn+3onzVjcsRndIJvYeLlBaFmoTSNkwI9Bs= ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-18T15:07:26.027 INFO:teuthology.orchestra.run.smithi019.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpvnxZ8ElXr5YJcsABYs3kNNr8JrxvGWZ0cwwo6ZISbL3BG3qVGSQ91ZY1Ke7RvL7DotwbdBASeuMZdl5Ib4WpEA+Z6sBKzjnULIWjYR+obO9epR2boWPqzQeb2Gq5P5t3cvNLjvPMbKcqsbL83FrRdRedXjOSOGZTmfKd6hN7XO8JlOp14rBmwj09gQ5WkWq4WKL+8S5S90mM99A1MauQQ/aeaGfNtFEplnm5uq9ygbKLWH6kGDXmcnISZqTmjUvQ4Z4ctJa+RtLl7w4e62oieSKDYZ8JJ3KBJv+jsU1YVyJZ5UMZ+kiconP/IBgXhPDgFnkoRnMNT569N5LckirlnZri046uGnvGj7yEY7GIP6nDKTcbeLehP8GH+S6WTeFyl5+p34ygU6VI3uJw8GzBHVqsvTvea8kVA0wlHNWx4dutGafWfqzjp1z3hEz0hdlxnKaS+6/3LoQY7fhrcS6uSqN7DiLuMn+3onzVjcsRndIJvYeLlBaFmoTSNkwI9Bs= ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715 2024-03-18T15:07:26.028 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:25 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/1582125452' entity='client.admin' 2024-03-18T15:07:26.072 DEBUG:teuthology.orchestra.run.smithi033:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpvnxZ8ElXr5YJcsABYs3kNNr8JrxvGWZ0cwwo6ZISbL3BG3qVGSQ91ZY1Ke7RvL7DotwbdBASeuMZdl5Ib4WpEA+Z6sBKzjnULIWjYR+obO9epR2boWPqzQeb2Gq5P5t3cvNLjvPMbKcqsbL83FrRdRedXjOSOGZTmfKd6hN7XO8JlOp14rBmwj09gQ5WkWq4WKL+8S5S90mM99A1MauQQ/aeaGfNtFEplnm5uq9ygbKLWH6kGDXmcnISZqTmjUvQ4Z4ctJa+RtLl7w4e62oieSKDYZ8JJ3KBJv+jsU1YVyJZ5UMZ+kiconP/IBgXhPDgFnkoRnMNT569N5LckirlnZri046uGnvGj7yEY7GIP6nDKTcbeLehP8GH+S6WTeFyl5+p34ygU6VI3uJw8GzBHVqsvTvea8kVA0wlHNWx4dutGafWfqzjp1z3hEz0hdlxnKaS+6/3LoQY7fhrcS6uSqN7DiLuMn+3onzVjcsRndIJvYeLlBaFmoTSNkwI9Bs= ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-18T15:07:26.127 INFO:teuthology.orchestra.run.smithi033.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpvnxZ8ElXr5YJcsABYs3kNNr8JrxvGWZ0cwwo6ZISbL3BG3qVGSQ91ZY1Ke7RvL7DotwbdBASeuMZdl5Ib4WpEA+Z6sBKzjnULIWjYR+obO9epR2boWPqzQeb2Gq5P5t3cvNLjvPMbKcqsbL83FrRdRedXjOSOGZTmfKd6hN7XO8JlOp14rBmwj09gQ5WkWq4WKL+8S5S90mM99A1MauQQ/aeaGfNtFEplnm5uq9ygbKLWH6kGDXmcnISZqTmjUvQ4Z4ctJa+RtLl7w4e62oieSKDYZ8JJ3KBJv+jsU1YVyJZ5UMZ+kiconP/IBgXhPDgFnkoRnMNT569N5LckirlnZri046uGnvGj7yEY7GIP6nDKTcbeLehP8GH+S6WTeFyl5+p34ygU6VI3uJw8GzBHVqsvTvea8kVA0wlHNWx4dutGafWfqzjp1z3hEz0hdlxnKaS+6/3LoQY7fhrcS6uSqN7DiLuMn+3onzVjcsRndIJvYeLlBaFmoTSNkwI9Bs= ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715 2024-03-18T15:07:26.145 DEBUG:teuthology.orchestra.run.smithi104:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpvnxZ8ElXr5YJcsABYs3kNNr8JrxvGWZ0cwwo6ZISbL3BG3qVGSQ91ZY1Ke7RvL7DotwbdBASeuMZdl5Ib4WpEA+Z6sBKzjnULIWjYR+obO9epR2boWPqzQeb2Gq5P5t3cvNLjvPMbKcqsbL83FrRdRedXjOSOGZTmfKd6hN7XO8JlOp14rBmwj09gQ5WkWq4WKL+8S5S90mM99A1MauQQ/aeaGfNtFEplnm5uq9ygbKLWH6kGDXmcnISZqTmjUvQ4Z4ctJa+RtLl7w4e62oieSKDYZ8JJ3KBJv+jsU1YVyJZ5UMZ+kiconP/IBgXhPDgFnkoRnMNT569N5LckirlnZri046uGnvGj7yEY7GIP6nDKTcbeLehP8GH+S6WTeFyl5+p34ygU6VI3uJw8GzBHVqsvTvea8kVA0wlHNWx4dutGafWfqzjp1z3hEz0hdlxnKaS+6/3LoQY7fhrcS6uSqN7DiLuMn+3onzVjcsRndIJvYeLlBaFmoTSNkwI9Bs= ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-18T15:07:26.193 INFO:teuthology.orchestra.run.smithi104.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpvnxZ8ElXr5YJcsABYs3kNNr8JrxvGWZ0cwwo6ZISbL3BG3qVGSQ91ZY1Ke7RvL7DotwbdBASeuMZdl5Ib4WpEA+Z6sBKzjnULIWjYR+obO9epR2boWPqzQeb2Gq5P5t3cvNLjvPMbKcqsbL83FrRdRedXjOSOGZTmfKd6hN7XO8JlOp14rBmwj09gQ5WkWq4WKL+8S5S90mM99A1MauQQ/aeaGfNtFEplnm5uq9ygbKLWH6kGDXmcnISZqTmjUvQ4Z4ctJa+RtLl7w4e62oieSKDYZ8JJ3KBJv+jsU1YVyJZ5UMZ+kiconP/IBgXhPDgFnkoRnMNT569N5LckirlnZri046uGnvGj7yEY7GIP6nDKTcbeLehP8GH+S6WTeFyl5+p34ygU6VI3uJw8GzBHVqsvTvea8kVA0wlHNWx4dutGafWfqzjp1z3hEz0hdlxnKaS+6/3LoQY7fhrcS6uSqN7DiLuMn+3onzVjcsRndIJvYeLlBaFmoTSNkwI9Bs= ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715 2024-03-18T15:07:26.209 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-18T15:07:26.641 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:07:29.384 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:29.384+0000 7fe144400700 1 -- 172.21.15.19:0/3498034850 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe13c076300 msgr2=0x7fe13c074760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:07:29.385 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:29.384+0000 7fe144400700 1 --2- 172.21.15.19:0/3498034850 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe13c076300 0x7fe13c074760 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe130009910 tx=0x7fe130009c20 comp rx=0 tx=0).stop 2024-03-18T15:07:29.385 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:29.384+0000 7fe144400700 1 -- 172.21.15.19:0/3498034850 shutdown_connections 2024-03-18T15:07:29.385 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:29.384+0000 7fe144400700 1 --2- 172.21.15.19:0/3498034850 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe13c076300 0x7fe13c074760 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:07:29.385 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:29.384+0000 7fe144400700 1 -- 172.21.15.19:0/3498034850 >> 172.21.15.19:0/3498034850 conn(0x7fe13c0fd2a0 msgr2=0x7fe13c0ff6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:07:29.385 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:29.384+0000 7fe144400700 1 -- 172.21.15.19:0/3498034850 shutdown_connections 2024-03-18T15:07:29.385 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:29.384+0000 7fe144400700 1 -- 172.21.15.19:0/3498034850 wait complete. 2024-03-18T15:07:29.385 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:29.385+0000 7fe144400700 1 Processor -- start 2024-03-18T15:07:29.386 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:29.386+0000 7fe144400700 1 -- start start 2024-03-18T15:07:29.386 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:29.386+0000 7fe144400700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe13c076300 0x7fe13c199600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:07:29.386 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:29.386+0000 7fe144400700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fe13c109f90 con 0x7fe13c076300 2024-03-18T15:07:29.387 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:29.386+0000 7fe141a00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe13c076300 0x7fe13c199600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:07:29.387 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:29.387+0000 7fe141a00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe13c076300 0x7fe13c199600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:36724/0 (socket says 172.21.15.19:36724) 2024-03-18T15:07:29.387 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:29.387+0000 7fe141a00700 1 -- 172.21.15.19:0/3424092821 learned_addr learned my addr 172.21.15.19:0/3424092821 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:07:29.387 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:29.387+0000 7fe141a00700 1 -- 172.21.15.19:0/3424092821 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe13c199b40 con 0x7fe13c076300 2024-03-18T15:07:29.387 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:29.387+0000 7fe141a00700 1 --2- 172.21.15.19:0/3424092821 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe13c076300 0x7fe13c199600 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe130006010 tx=0x7fe130005050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:07:29.388 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:29.388+0000 7fe13a000700 1 -- 172.21.15.19:0/3424092821 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe130027070 con 0x7fe13c076300 2024-03-18T15:07:29.388 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:29.388+0000 7fe13a000700 1 -- 172.21.15.19:0/3424092821 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe13000be40 con 0x7fe13c076300 2024-03-18T15:07:29.388 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:29.388+0000 7fe144400700 1 -- 172.21.15.19:0/3424092821 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe13c199dd0 con 0x7fe13c076300 2024-03-18T15:07:29.388 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:29.388+0000 7fe13a000700 1 -- 172.21.15.19:0/3424092821 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe13000f460 con 0x7fe13c076300 2024-03-18T15:07:29.388 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:29.388+0000 7fe144400700 1 -- 172.21.15.19:0/3424092821 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe13c19cac0 con 0x7fe13c076300 2024-03-18T15:07:29.389 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:29.389+0000 7fe144400700 1 -- 172.21.15.19:0/3424092821 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe13c10a710 con 0x7fe13c076300 2024-03-18T15:07:29.389 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:29.389+0000 7fe13a000700 1 -- 172.21.15.19:0/3424092821 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7fe13002a070 con 0x7fe13c076300 2024-03-18T15:07:29.390 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:29.390+0000 7fe13a000700 1 --2- 172.21.15.19:0/3424092821 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fe124038fc0 0x7fe12403b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:07:29.390 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:29.390+0000 7fe13a000700 1 -- 172.21.15.19:0/3424092821 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fe13001e030 con 0x7fe13c076300 2024-03-18T15:07:29.390 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:29.390+0000 7fe141000700 1 --2- 172.21.15.19:0/3424092821 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fe124038fc0 0x7fe12403b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:07:29.391 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:29.391+0000 7fe141000700 1 --2- 172.21.15.19:0/3424092821 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fe124038fc0 0x7fe12403b480 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fe128006fd0 tx=0x7fe128006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:07:29.396 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:29.396+0000 7fe13a000700 1 -- 172.21.15.19:0/3424092821 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe130013070 con 0x7fe13c076300 2024-03-18T15:07:29.626 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:29.625+0000 7fe144400700 1 -- 172.21.15.19:0/3424092821 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7fe13c02dc80 con 0x7fe13c076300 2024-03-18T15:07:29.627 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:29 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:07:29.627 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:29 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:07:29.627 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:29 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi019", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:07:29.627 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:29 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:07:29.627 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:29 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:07:29.627 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:29 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:07:29.630 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:29.630+0000 7fe13a000700 1 -- 172.21.15.19:0/3424092821 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7fe1300310c0 con 0x7fe13c076300 2024-03-18T15:07:29.633 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:29.633+0000 7fe144400700 1 -- 172.21.15.19:0/3424092821 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fe124038fc0 msgr2=0x7fe12403b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:07:29.633 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:29.633+0000 7fe144400700 1 --2- 172.21.15.19:0/3424092821 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fe124038fc0 0x7fe12403b480 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fe128006fd0 tx=0x7fe128006e40 comp rx=0 tx=0).stop 2024-03-18T15:07:29.633 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:29.633+0000 7fe144400700 1 -- 172.21.15.19:0/3424092821 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe13c076300 msgr2=0x7fe13c199600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:07:29.633 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:29.633+0000 7fe144400700 1 --2- 172.21.15.19:0/3424092821 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe13c076300 0x7fe13c199600 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe130006010 tx=0x7fe130005050 comp rx=0 tx=0).stop 2024-03-18T15:07:29.634 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:29.633+0000 7fe144400700 1 -- 172.21.15.19:0/3424092821 shutdown_connections 2024-03-18T15:07:29.634 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:29.633+0000 7fe144400700 1 --2- 172.21.15.19:0/3424092821 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fe124038fc0 0x7fe12403b480 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:07:29.634 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:29.633+0000 7fe144400700 1 --2- 172.21.15.19:0/3424092821 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe13c076300 0x7fe13c199600 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:07:29.634 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:29.633+0000 7fe144400700 1 -- 172.21.15.19:0/3424092821 >> 172.21.15.19:0/3424092821 conn(0x7fe13c0fd2a0 msgr2=0x7fe13c0fdd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:07:29.634 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:29.634+0000 7fe144400700 1 -- 172.21.15.19:0/3424092821 shutdown_connections 2024-03-18T15:07:29.634 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:29.634+0000 7fe144400700 1 -- 172.21.15.19:0/3424092821 wait complete. 2024-03-18T15:07:30.345 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-18T15:07:30.345 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-18T15:07:30.895 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:30 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/3424092821' entity='client.admin' 2024-03-18T15:07:30.895 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:30 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:07:30.895 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:30 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:07:31.279 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:07:33.235 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:33.234+0000 7fade9800700 1 -- 172.21.15.19:0/591536624 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fade4102790 msgr2=0x7fade4104b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:07:33.236 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:33.234+0000 7fade9800700 1 --2- 172.21.15.19:0/591536624 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fade4102790 0x7fade4104b80 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fadd4009910 tx=0x7fadd4009c20 comp rx=0 tx=0).stop 2024-03-18T15:07:33.236 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:33.234+0000 7fade9800700 1 -- 172.21.15.19:0/591536624 shutdown_connections 2024-03-18T15:07:33.236 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:33.234+0000 7fade9800700 1 --2- 172.21.15.19:0/591536624 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fade4102790 0x7fade4104b80 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:07:33.236 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:33.234+0000 7fade9800700 1 -- 172.21.15.19:0/591536624 >> 172.21.15.19:0/591536624 conn(0x7fade40fc1a0 msgr2=0x7fade40fe5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:07:33.236 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:33.235+0000 7fade9800700 1 -- 172.21.15.19:0/591536624 shutdown_connections 2024-03-18T15:07:33.236 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:33.235+0000 7fade9800700 1 -- 172.21.15.19:0/591536624 wait complete. 2024-03-18T15:07:33.236 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:33.236+0000 7fade9800700 1 Processor -- start 2024-03-18T15:07:33.237 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:33.236+0000 7fade9800700 1 -- start start 2024-03-18T15:07:33.237 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:33.237+0000 7fade9800700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fade4102790 0x7fade4199590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:07:33.237 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:33.237+0000 7fade9800700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fade4107d30 con 0x7fade4102790 2024-03-18T15:07:33.237 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:33.237+0000 7fade2a00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fade4102790 0x7fade4199590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:07:33.237 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:33.237+0000 7fade2a00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fade4102790 0x7fade4199590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:56222/0 (socket says 172.21.15.19:56222) 2024-03-18T15:07:33.237 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:33.237+0000 7fade2a00700 1 -- 172.21.15.19:0/3413759438 learned_addr learned my addr 172.21.15.19:0/3413759438 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:07:33.238 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:33.238+0000 7fade2a00700 1 -- 172.21.15.19:0/3413759438 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fade4199ad0 con 0x7fade4102790 2024-03-18T15:07:33.238 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:33.238+0000 7fade2a00700 1 --2- 172.21.15.19:0/3413759438 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fade4102790 0x7fade4199590 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fadd4005950 tx=0x7fadd4005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:07:33.239 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:33.238+0000 7faddb400700 1 -- 172.21.15.19:0/3413759438 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fadd4026070 con 0x7fade4102790 2024-03-18T15:07:33.239 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:33.238+0000 7faddb400700 1 -- 172.21.15.19:0/3413759438 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fadd400bbd0 con 0x7fade4102790 2024-03-18T15:07:33.239 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:33.238+0000 7faddb400700 1 -- 172.21.15.19:0/3413759438 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fadd4003ad0 con 0x7fade4102790 2024-03-18T15:07:33.239 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:33.238+0000 7fade9800700 1 -- 172.21.15.19:0/3413759438 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fade4199dc0 con 0x7fade4102790 2024-03-18T15:07:33.239 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:33.238+0000 7fade9800700 1 -- 172.21.15.19:0/3413759438 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fade419ca50 con 0x7fade4102790 2024-03-18T15:07:33.239 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:33.240+0000 7faddb400700 1 -- 172.21.15.19:0/3413759438 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7fadd4029070 con 0x7fade4102790 2024-03-18T15:07:33.240 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:33.240+0000 7fade9800700 1 -- 172.21.15.19:0/3413759438 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fade41084b0 con 0x7fade4102790 2024-03-18T15:07:33.240 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:33.240+0000 7faddb400700 1 --2- 172.21.15.19:0/3413759438 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fadc8039020 0x7fadc803b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:07:33.240 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:33.240+0000 7faddb400700 1 -- 172.21.15.19:0/3413759438 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fadd401e070 con 0x7fade4102790 2024-03-18T15:07:33.241 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:33.241+0000 7fade2000700 1 --2- 172.21.15.19:0/3413759438 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fadc8039020 0x7fadc803b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:07:33.241 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:33.241+0000 7fade2000700 1 --2- 172.21.15.19:0/3413759438 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fadc8039020 0x7fadc803b4e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fadd0006fd0 tx=0x7fadd0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:07:33.247 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:33.247+0000 7faddb400700 1 -- 172.21.15.19:0/3413759438 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fadd4013070 con 0x7fade4102790 2024-03-18T15:07:33.482 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:33.482+0000 7fade9800700 1 -- 172.21.15.19:0/3413759438 --> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7fade402dd60 con 0x7fadc8039020 2024-03-18T15:07:33.491 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:33.491+0000 7faddb400700 1 -- 172.21.15.19:0/3413759438 <== mgr.14152 v2:172.21.15.19:6800/2017714236 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fade402dd60 con 0x7fadc8039020 2024-03-18T15:07:33.494 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:33.494+0000 7fade9800700 1 -- 172.21.15.19:0/3413759438 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fadc8039020 msgr2=0x7fadc803b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:07:33.495 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:33.494+0000 7fade9800700 1 --2- 172.21.15.19:0/3413759438 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fadc8039020 0x7fadc803b4e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fadd0006fd0 tx=0x7fadd0006e40 comp rx=0 tx=0).stop 2024-03-18T15:07:33.495 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:33.494+0000 7fade9800700 1 -- 172.21.15.19:0/3413759438 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fade4102790 msgr2=0x7fade4199590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:07:33.495 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:33.494+0000 7fade9800700 1 --2- 172.21.15.19:0/3413759438 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fade4102790 0x7fade4199590 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fadd4005950 tx=0x7fadd4005030 comp rx=0 tx=0).stop 2024-03-18T15:07:33.495 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:33.494+0000 7fade9800700 1 -- 172.21.15.19:0/3413759438 shutdown_connections 2024-03-18T15:07:33.495 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:33.494+0000 7fade9800700 1 --2- 172.21.15.19:0/3413759438 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fadc8039020 0x7fadc803b4e0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:07:33.495 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:33.494+0000 7fade9800700 1 --2- 172.21.15.19:0/3413759438 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fade4102790 0x7fade4199590 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:07:33.495 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:33.494+0000 7fade9800700 1 -- 172.21.15.19:0/3413759438 >> 172.21.15.19:0/3413759438 conn(0x7fade40fc1a0 msgr2=0x7fade40fcbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:07:33.495 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:33.495+0000 7fade9800700 1 -- 172.21.15.19:0/3413759438 shutdown_connections 2024-03-18T15:07:33.495 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:33.495+0000 7fade9800700 1 -- 172.21.15.19:0/3413759438 wait complete. 2024-03-18T15:07:34.274 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi033 2024-03-18T15:07:34.274 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-18T15:07:34.274 DEBUG:teuthology.orchestra.run.smithi033:> dd of=/etc/ceph/ceph.conf 2024-03-18T15:07:34.293 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-18T15:07:34.293 DEBUG:teuthology.orchestra.run.smithi033:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-18T15:07:34.352 INFO:tasks.cephadm:Adding host smithi033 to orchestrator... 2024-03-18T15:07:34.352 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 -- ceph orch host add smithi033 2024-03-18T15:07:34.777 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:07:34.917 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:34 smithi019 ceph-mon[31522]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:07:34.918 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:34 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:07:34.918 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:34 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:07:34.918 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:34 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:07:34.918 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:34 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:07:34.918 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:34 smithi019 ceph-mon[31522]: Updating smithi019:/etc/ceph/ceph.conf 2024-03-18T15:07:35.653 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:35 smithi019 ceph-mon[31522]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:07:35.654 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:35 smithi019 ceph-mon[31522]: Updating smithi019:/var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T15:07:35.654 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:35 smithi019 ceph-mon[31522]: Updating smithi019:/etc/ceph/ceph.client.admin.keyring 2024-03-18T15:07:36.507 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:36 smithi019 ceph-mon[31522]: Updating smithi019:/var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-18T15:07:36.507 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:36 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:07:36.508 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:36 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:07:36.508 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:36 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:07:36.668 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:36.668+0000 7fe9da600700 1 -- 172.21.15.19:0/3896597499 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe9d4102e60 msgr2=0x7fe9d4103240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:07:36.668 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:36.668+0000 7fe9da600700 1 --2- 172.21.15.19:0/3896597499 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe9d4102e60 0x7fe9d4103240 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fe9c400b0a0 tx=0x7fe9c400b3b0 comp rx=0 tx=0).stop 2024-03-18T15:07:36.668 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:36.668+0000 7fe9da600700 1 -- 172.21.15.19:0/3896597499 shutdown_connections 2024-03-18T15:07:36.668 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:36.668+0000 7fe9da600700 1 --2- 172.21.15.19:0/3896597499 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe9d4102e60 0x7fe9d4103240 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:07:36.669 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:36.668+0000 7fe9da600700 1 -- 172.21.15.19:0/3896597499 >> 172.21.15.19:0/3896597499 conn(0x7fe9d40fc690 msgr2=0x7fe9d40feab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:07:36.669 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:36.668+0000 7fe9da600700 1 -- 172.21.15.19:0/3896597499 shutdown_connections 2024-03-18T15:07:36.669 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:36.668+0000 7fe9da600700 1 -- 172.21.15.19:0/3896597499 wait complete. 2024-03-18T15:07:36.669 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:36.669+0000 7fe9da600700 1 Processor -- start 2024-03-18T15:07:36.669 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:36.669+0000 7fe9da600700 1 -- start start 2024-03-18T15:07:36.669 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:36.670+0000 7fe9da600700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe9d4102e60 0x7fe9d419f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:07:36.670 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:36.670+0000 7fe9da600700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fe9d410c330 con 0x7fe9d4102e60 2024-03-18T15:07:36.670 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:36.670+0000 7fe9d3400700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe9d4102e60 0x7fe9d419f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:07:36.670 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:36.670+0000 7fe9d3400700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe9d4102e60 0x7fe9d419f700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:56248/0 (socket says 172.21.15.19:56248) 2024-03-18T15:07:36.670 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:36.670+0000 7fe9d3400700 1 -- 172.21.15.19:0/922214732 learned_addr learned my addr 172.21.15.19:0/922214732 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:07:36.670 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:36.670+0000 7fe9d3400700 1 -- 172.21.15.19:0/922214732 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9d419fc40 con 0x7fe9d4102e60 2024-03-18T15:07:36.671 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:36.671+0000 7fe9d3400700 1 --2- 172.21.15.19:0/922214732 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe9d4102e60 0x7fe9d419f700 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fe9c4000f80 tx=0x7fe9c400bd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:07:36.671 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:36.671+0000 7fe9d0c00700 1 -- 172.21.15.19:0/922214732 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe9c4020020 con 0x7fe9d4102e60 2024-03-18T15:07:36.671 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:36.671+0000 7fe9d0c00700 1 -- 172.21.15.19:0/922214732 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9c4018040 con 0x7fe9d4102e60 2024-03-18T15:07:36.672 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:36.671+0000 7fe9d0c00700 1 -- 172.21.15.19:0/922214732 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe9c4017070 con 0x7fe9d4102e60 2024-03-18T15:07:36.672 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:36.671+0000 7fe9da600700 1 -- 172.21.15.19:0/922214732 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9d419fe70 con 0x7fe9d4102e60 2024-03-18T15:07:36.672 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:36.671+0000 7fe9da600700 1 -- 172.21.15.19:0/922214732 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9d41998b0 con 0x7fe9d4102e60 2024-03-18T15:07:36.672 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:36.672+0000 7fe9d0c00700 1 -- 172.21.15.19:0/922214732 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7fe9c4023020 con 0x7fe9d4102e60 2024-03-18T15:07:36.673 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:36.672+0000 7fe9da600700 1 -- 172.21.15.19:0/922214732 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9d406a990 con 0x7fe9d4102e60 2024-03-18T15:07:36.673 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:36.673+0000 7fe9d0c00700 1 --2- 172.21.15.19:0/922214732 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fe9b8039060 0x7fe9b803b520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:07:36.673 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:36.673+0000 7fe9d0c00700 1 -- 172.21.15.19:0/922214732 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fe9c400f050 con 0x7fe9d4102e60 2024-03-18T15:07:36.673 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:36.673+0000 7fe9caa00700 1 --2- 172.21.15.19:0/922214732 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fe9b8039060 0x7fe9b803b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:07:36.674 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:36.674+0000 7fe9caa00700 1 --2- 172.21.15.19:0/922214732 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fe9b8039060 0x7fe9b803b520 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc006fd0 tx=0x7fe9bc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:07:36.681 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:36.681+0000 7fe9d0c00700 1 -- 172.21.15.19:0/922214732 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe9c402a020 con 0x7fe9d4102e60 2024-03-18T15:07:36.910 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:36.909+0000 7fe9da600700 1 -- 172.21.15.19:0/922214732 --> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi033", "target": ["mon-mgr", ""]}) -- 0x7fe9d4063900 con 0x7fe9b8039060 2024-03-18T15:07:37.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:37 smithi019 ceph-mon[31522]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:07:37.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:37 smithi019 ceph-mon[31522]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi033", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:07:38.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:38 smithi019 ceph-mon[31522]: Deploying cephadm binary to smithi033 2024-03-18T15:07:39.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:39 smithi019 ceph-mon[31522]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:07:41.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:41 smithi019 ceph-mon[31522]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:07:41.983 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:41.983+0000 7fe9d0c00700 1 -- 172.21.15.19:0/922214732 <== mgr.14152 v2:172.21.15.19:6800/2017714236 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7fe9d4063900 con 0x7fe9b8039060 2024-03-18T15:07:41.983 INFO:teuthology.orchestra.run.smithi019.stdout:Added host 'smithi033' with addr '172.21.15.33' 2024-03-18T15:07:41.987 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:41.987+0000 7fe9da600700 1 -- 172.21.15.19:0/922214732 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fe9b8039060 msgr2=0x7fe9b803b520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:07:41.987 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:41.987+0000 7fe9da600700 1 --2- 172.21.15.19:0/922214732 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fe9b8039060 0x7fe9b803b520 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc006fd0 tx=0x7fe9bc006e40 comp rx=0 tx=0).stop 2024-03-18T15:07:41.988 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:41.987+0000 7fe9da600700 1 -- 172.21.15.19:0/922214732 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe9d4102e60 msgr2=0x7fe9d419f700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:07:41.988 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:41.987+0000 7fe9da600700 1 --2- 172.21.15.19:0/922214732 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe9d4102e60 0x7fe9d419f700 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fe9c4000f80 tx=0x7fe9c400bd20 comp rx=0 tx=0).stop 2024-03-18T15:07:41.988 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:41.987+0000 7fe9da600700 1 -- 172.21.15.19:0/922214732 shutdown_connections 2024-03-18T15:07:41.988 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:41.987+0000 7fe9da600700 1 --2- 172.21.15.19:0/922214732 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fe9b8039060 0x7fe9b803b520 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:07:41.988 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:41.987+0000 7fe9da600700 1 --2- 172.21.15.19:0/922214732 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe9d4102e60 0x7fe9d419f700 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:07:41.988 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:41.987+0000 7fe9da600700 1 -- 172.21.15.19:0/922214732 >> 172.21.15.19:0/922214732 conn(0x7fe9d40fc690 msgr2=0x7fe9d4108000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:07:41.988 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:41.988+0000 7fe9da600700 1 -- 172.21.15.19:0/922214732 shutdown_connections 2024-03-18T15:07:41.988 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:41.988+0000 7fe9da600700 1 -- 172.21.15.19:0/922214732 wait complete. 2024-03-18T15:07:42.672 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 -- ceph orch host ls --format=json 2024-03-18T15:07:43.118 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:07:43.264 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:42 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:07:43.264 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:42 smithi019 ceph-mon[31522]: Added host smithi033 2024-03-18T15:07:43.264 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:42 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:07:43.265 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:42 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:07:44.243 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:43 smithi019 ceph-mon[31522]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:07:44.244 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:43 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:07:44.994 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:44.994+0000 7f0bc4400700 1 -- 172.21.15.19:0/2561587037 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0bbc100df0 msgr2=0x7f0bbc1011d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:07:44.994 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:44.994+0000 7f0bc4400700 1 --2- 172.21.15.19:0/2561587037 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0bbc100df0 0x7f0bbc1011d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f0bb0009910 tx=0x7f0bb0009c20 comp rx=0 tx=0).stop 2024-03-18T15:07:44.995 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:44.994+0000 7f0bc4400700 1 -- 172.21.15.19:0/2561587037 shutdown_connections 2024-03-18T15:07:44.995 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:44.994+0000 7f0bc4400700 1 --2- 172.21.15.19:0/2561587037 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0bbc100df0 0x7f0bbc1011d0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:07:44.995 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:44.994+0000 7f0bc4400700 1 -- 172.21.15.19:0/2561587037 >> 172.21.15.19:0/2561587037 conn(0x7f0bbc0fc680 msgr2=0x7f0bbc0feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:07:44.995 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:44.994+0000 7f0bc4400700 1 -- 172.21.15.19:0/2561587037 shutdown_connections 2024-03-18T15:07:44.995 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:44.994+0000 7f0bc4400700 1 -- 172.21.15.19:0/2561587037 wait complete. 2024-03-18T15:07:44.995 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:44.995+0000 7f0bc4400700 1 Processor -- start 2024-03-18T15:07:44.996 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:44.995+0000 7f0bc4400700 1 -- start start 2024-03-18T15:07:44.996 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:44.996+0000 7f0bc4400700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0bbc100df0 0x7f0bbc19f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:07:44.996 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:44.996+0000 7f0bc4400700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f0bbc06a650 con 0x7f0bbc100df0 2024-03-18T15:07:44.996 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:44.996+0000 7f0bc1a00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0bbc100df0 0x7f0bbc19f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:07:44.996 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:44.996+0000 7f0bc1a00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0bbc100df0 0x7f0bbc19f680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:52422/0 (socket says 172.21.15.19:52422) 2024-03-18T15:07:44.996 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:44.996+0000 7f0bc1a00700 1 -- 172.21.15.19:0/1155726092 learned_addr learned my addr 172.21.15.19:0/1155726092 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:07:44.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:44.996+0000 7f0bc1a00700 1 -- 172.21.15.19:0/1155726092 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bbc19fbc0 con 0x7f0bbc100df0 2024-03-18T15:07:44.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:44.997+0000 7f0bc1a00700 1 --2- 172.21.15.19:0/1155726092 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0bbc100df0 0x7f0bbc19f680 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f0bb0006010 tx=0x7f0bb0005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:07:44.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:44.997+0000 7f0bba000700 1 -- 172.21.15.19:0/1155726092 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0bb0026070 con 0x7f0bbc100df0 2024-03-18T15:07:44.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:44.997+0000 7f0bba000700 1 -- 172.21.15.19:0/1155726092 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0bb000bbd0 con 0x7f0bbc100df0 2024-03-18T15:07:44.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:44.997+0000 7f0bc4400700 1 -- 172.21.15.19:0/1155726092 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0bbc19feb0 con 0x7f0bbc100df0 2024-03-18T15:07:44.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:44.997+0000 7f0bba000700 1 -- 172.21.15.19:0/1155726092 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0bb0003910 con 0x7f0bbc100df0 2024-03-18T15:07:44.998 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:44.997+0000 7f0bc4400700 1 -- 172.21.15.19:0/1155726092 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0bbc199800 con 0x7f0bbc100df0 2024-03-18T15:07:44.999 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:44.999+0000 7f0bba000700 1 -- 172.21.15.19:0/1155726092 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f0bb0029070 con 0x7f0bbc100df0 2024-03-18T15:07:44.999 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:44.999+0000 7f0bc4400700 1 -- 172.21.15.19:0/1155726092 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0bbc1022d0 con 0x7f0bbc100df0 2024-03-18T15:07:44.999 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:44.999+0000 7f0bba000700 1 --2- 172.21.15.19:0/1155726092 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f0ba4039020 0x7f0ba403b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:07:44.999 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:44.999+0000 7f0bba000700 1 -- 172.21.15.19:0/1155726092 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f0bb001e070 con 0x7f0bbc100df0 2024-03-18T15:07:45.000 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:45.000+0000 7f0bc1000700 1 --2- 172.21.15.19:0/1155726092 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f0ba4039020 0x7f0ba403b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:07:45.000 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:45.000+0000 7f0bc1000700 1 --2- 172.21.15.19:0/1155726092 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f0ba4039020 0x7f0ba403b4e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f0ba8006fd0 tx=0x7f0ba8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:07:45.005 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:45.005+0000 7f0bba000700 1 -- 172.21.15.19:0/1155726092 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0bb0013070 con 0x7f0bbc100df0 2024-03-18T15:07:45.242 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:45.241+0000 7f0bc4400700 1 -- 172.21.15.19:0/1155726092 --> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0bbc19ad30 con 0x7f0ba4039020 2024-03-18T15:07:45.244 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:45.244+0000 7f0bba000700 1 -- 172.21.15.19:0/1155726092 <== mgr.14152 v2:172.21.15.19:6800/2017714236 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+159 (secure 0 0 0) 0x7f0bbc19ad30 con 0x7f0ba4039020 2024-03-18T15:07:45.245 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-18T15:07:45.245 INFO:teuthology.orchestra.run.smithi019.stdout:[{"addr": "172.21.15.19", "hostname": "smithi019", "labels": [], "status": ""}, {"addr": "172.21.15.33", "hostname": "smithi033", "labels": [], "status": ""}] 2024-03-18T15:07:45.248 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:45.247+0000 7f0bc4400700 1 -- 172.21.15.19:0/1155726092 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f0ba4039020 msgr2=0x7f0ba403b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:07:45.248 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:45.247+0000 7f0bc4400700 1 --2- 172.21.15.19:0/1155726092 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f0ba4039020 0x7f0ba403b4e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f0ba8006fd0 tx=0x7f0ba8006e40 comp rx=0 tx=0).stop 2024-03-18T15:07:45.248 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:45.247+0000 7f0bc4400700 1 -- 172.21.15.19:0/1155726092 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0bbc100df0 msgr2=0x7f0bbc19f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:07:45.248 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:45.247+0000 7f0bc4400700 1 --2- 172.21.15.19:0/1155726092 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0bbc100df0 0x7f0bbc19f680 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f0bb0006010 tx=0x7f0bb0005030 comp rx=0 tx=0).stop 2024-03-18T15:07:45.248 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:45.248+0000 7f0bc4400700 1 -- 172.21.15.19:0/1155726092 shutdown_connections 2024-03-18T15:07:45.248 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:45.248+0000 7f0bc4400700 1 --2- 172.21.15.19:0/1155726092 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f0ba4039020 0x7f0ba403b4e0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:07:45.248 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:45.248+0000 7f0bc4400700 1 --2- 172.21.15.19:0/1155726092 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0bbc100df0 0x7f0bbc19f680 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:07:45.248 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:45.248+0000 7f0bc4400700 1 -- 172.21.15.19:0/1155726092 >> 172.21.15.19:0/1155726092 conn(0x7f0bbc0fc680 msgr2=0x7f0bbc10b000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:07:45.248 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:45.248+0000 7f0bc4400700 1 -- 172.21.15.19:0/1155726092 shutdown_connections 2024-03-18T15:07:45.249 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:45.248+0000 7f0bc4400700 1 -- 172.21.15.19:0/1155726092 wait complete. 2024-03-18T15:07:45.891 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:45 smithi019 ceph-mon[31522]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:07:45.892 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:45 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:07:45.961 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi104 2024-03-18T15:07:45.962 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-03-18T15:07:45.962 DEBUG:teuthology.orchestra.run.smithi104:> dd of=/etc/ceph/ceph.conf 2024-03-18T15:07:45.980 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-03-18T15:07:45.980 DEBUG:teuthology.orchestra.run.smithi104:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-18T15:07:46.037 INFO:tasks.cephadm:Adding host smithi104 to orchestrator... 2024-03-18T15:07:46.037 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 -- ceph orch host add smithi104 2024-03-18T15:07:46.470 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:07:46.911 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:46 smithi019 ceph-mon[31522]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T15:07:47.964 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:47 smithi019 ceph-mon[31522]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:07:47.964 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:47 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:07:47.964 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:47 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:07:47.964 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:47 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:07:47.964 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:47 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:07:47.964 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:47 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi033", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:07:47.964 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:47 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:07:48.458 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:48.458+0000 7f69e2600700 1 -- 172.21.15.19:0/3002290663 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f69dc102e60 msgr2=0x7f69dc103240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:07:48.458 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:48.458+0000 7f69e2600700 1 --2- 172.21.15.19:0/3002290663 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f69dc102e60 0x7f69dc103240 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f69cc009910 tx=0x7f69cc009c20 comp rx=0 tx=0).stop 2024-03-18T15:07:48.459 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:48.458+0000 7f69e2600700 1 -- 172.21.15.19:0/3002290663 shutdown_connections 2024-03-18T15:07:48.459 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:48.458+0000 7f69e2600700 1 --2- 172.21.15.19:0/3002290663 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f69dc102e60 0x7f69dc103240 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:07:48.459 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:48.458+0000 7f69e2600700 1 -- 172.21.15.19:0/3002290663 >> 172.21.15.19:0/3002290663 conn(0x7f69dc077810 msgr2=0x7f69dc0feae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:07:48.459 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:48.458+0000 7f69e2600700 1 -- 172.21.15.19:0/3002290663 shutdown_connections 2024-03-18T15:07:48.459 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:48.458+0000 7f69e2600700 1 -- 172.21.15.19:0/3002290663 wait complete. 2024-03-18T15:07:48.459 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:48.459+0000 7f69e2600700 1 Processor -- start 2024-03-18T15:07:48.460 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:48.460+0000 7f69e2600700 1 -- start start 2024-03-18T15:07:48.460 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:48.460+0000 7f69e2600700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f69dc102e60 0x7f69dc100ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:07:48.460 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:48.460+0000 7f69e2600700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f69dc10c350 con 0x7f69dc102e60 2024-03-18T15:07:48.461 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:48.461+0000 7f69db400700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f69dc102e60 0x7f69dc100ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:07:48.461 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:48.461+0000 7f69db400700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f69dc102e60 0x7f69dc100ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:52438/0 (socket says 172.21.15.19:52438) 2024-03-18T15:07:48.461 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:48.461+0000 7f69db400700 1 -- 172.21.15.19:0/393422659 learned_addr learned my addr 172.21.15.19:0/393422659 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:07:48.462 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:48.462+0000 7f69db400700 1 -- 172.21.15.19:0/393422659 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69dc1029b0 con 0x7f69dc102e60 2024-03-18T15:07:48.462 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:48.462+0000 7f69db400700 1 --2- 172.21.15.19:0/393422659 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f69dc102e60 0x7f69dc100ac0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f69cc006010 tx=0x7f69cc005050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:07:48.463 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:48.463+0000 7f69d3e00700 1 -- 172.21.15.19:0/393422659 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f69cc026070 con 0x7f69dc102e60 2024-03-18T15:07:48.463 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:48.463+0000 7f69e2600700 1 -- 172.21.15.19:0/393422659 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69dc101000 con 0x7f69dc102e60 2024-03-18T15:07:48.463 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:48.463+0000 7f69d3e00700 1 -- 172.21.15.19:0/393422659 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f69cc00bc30 con 0x7f69dc102e60 2024-03-18T15:07:48.463 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:48.463+0000 7f69d3e00700 1 -- 172.21.15.19:0/393422659 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f69cc003ab0 con 0x7f69dc102e60 2024-03-18T15:07:48.463 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:48.463+0000 7f69e2600700 1 -- 172.21.15.19:0/393422659 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69dc1014c0 con 0x7f69dc102e60 2024-03-18T15:07:48.464 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:48.464+0000 7f69e2600700 1 -- 172.21.15.19:0/393422659 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69dc069fa0 con 0x7f69dc102e60 2024-03-18T15:07:48.465 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:48.465+0000 7f69d3e00700 1 -- 172.21.15.19:0/393422659 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f69cc029070 con 0x7f69dc102e60 2024-03-18T15:07:48.465 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:48.465+0000 7f69d3e00700 1 --2- 172.21.15.19:0/393422659 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f69c0039020 0x7f69c003b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:07:48.465 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:48.465+0000 7f69d3e00700 1 -- 172.21.15.19:0/393422659 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f69cc01e070 con 0x7f69dc102e60 2024-03-18T15:07:48.466 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:48.466+0000 7f69daa00700 1 --2- 172.21.15.19:0/393422659 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f69c0039020 0x7f69c003b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:07:48.466 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:48.466+0000 7f69daa00700 1 --2- 172.21.15.19:0/393422659 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f69c0039020 0x7f69c003b4e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f69c8006fd0 tx=0x7f69c8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:07:48.471 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:48.471+0000 7f69d3e00700 1 -- 172.21.15.19:0/393422659 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f69cc030020 con 0x7f69dc102e60 2024-03-18T15:07:48.702 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:48.702+0000 7f69e2600700 1 -- 172.21.15.19:0/393422659 --> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi104", "target": ["mon-mgr", ""]}) -- 0x7f69dc101ff0 con 0x7f69c0039020 2024-03-18T15:07:48.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:48 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:07:48.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:48 smithi019 ceph-mon[31522]: Updating smithi033:/etc/ceph/ceph.conf 2024-03-18T15:07:50.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:49 smithi019 ceph-mon[31522]: Updating smithi033:/var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T15:07:50.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:49 smithi019 ceph-mon[31522]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:07:50.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:49 smithi019 ceph-mon[31522]: Updating smithi033:/etc/ceph/ceph.client.admin.keyring 2024-03-18T15:07:50.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:49 smithi019 ceph-mon[31522]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi104", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:07:50.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:49 smithi019 ceph-mon[31522]: Updating smithi033:/var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-18T15:07:50.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:49 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:07:50.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:49 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:07:50.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:49 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:07:51.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:50 smithi019 ceph-mon[31522]: Deploying cephadm binary to smithi104 2024-03-18T15:07:52.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:51 smithi019 ceph-mon[31522]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:07:53.623 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:53.621+0000 7f69d3e00700 1 -- 172.21.15.19:0/393422659 <== mgr.14152 v2:172.21.15.19:6800/2017714236 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f69dc101ff0 con 0x7f69c0039020 2024-03-18T15:07:53.623 INFO:teuthology.orchestra.run.smithi019.stdout:Added host 'smithi104' with addr '172.21.15.104' 2024-03-18T15:07:53.627 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:53.625+0000 7f69e2600700 1 -- 172.21.15.19:0/393422659 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f69c0039020 msgr2=0x7f69c003b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:07:53.627 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:53.625+0000 7f69e2600700 1 --2- 172.21.15.19:0/393422659 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f69c0039020 0x7f69c003b4e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f69c8006fd0 tx=0x7f69c8006e40 comp rx=0 tx=0).stop 2024-03-18T15:07:53.627 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:53.625+0000 7f69e2600700 1 -- 172.21.15.19:0/393422659 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f69dc102e60 msgr2=0x7f69dc100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:07:53.627 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:53.625+0000 7f69e2600700 1 --2- 172.21.15.19:0/393422659 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f69dc102e60 0x7f69dc100ac0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f69cc006010 tx=0x7f69cc005050 comp rx=0 tx=0).stop 2024-03-18T15:07:53.627 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:53.625+0000 7f69e2600700 1 -- 172.21.15.19:0/393422659 shutdown_connections 2024-03-18T15:07:53.627 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:53.625+0000 7f69e2600700 1 --2- 172.21.15.19:0/393422659 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f69c0039020 0x7f69c003b4e0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:07:53.628 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:53.626+0000 7f69e2600700 1 --2- 172.21.15.19:0/393422659 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f69dc102e60 0x7f69dc100ac0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:07:53.628 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:53.626+0000 7f69e2600700 1 -- 172.21.15.19:0/393422659 >> 172.21.15.19:0/393422659 conn(0x7f69dc077810 msgr2=0x7f69dc103c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:07:53.628 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:53.626+0000 7f69e2600700 1 -- 172.21.15.19:0/393422659 shutdown_connections 2024-03-18T15:07:53.628 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:53.626+0000 7f69e2600700 1 -- 172.21.15.19:0/393422659 wait complete. 2024-03-18T15:07:53.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:53 smithi019 ceph-mon[31522]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:07:53.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:53 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:07:53.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:53 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:07:54.322 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 -- ceph orch host ls --format=json 2024-03-18T15:07:54.776 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:07:55.110 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:54 smithi019 ceph-mon[31522]: Added host smithi104 2024-03-18T15:07:55.110 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:54 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:07:55.110 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:54 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:07:55.859 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:55 smithi019 ceph-mon[31522]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:07:56.703 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:56.701+0000 7f07f9200700 1 -- 172.21.15.19:0/3022844081 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f07f40746a0 msgr2=0x7f07f4074a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:07:56.703 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:56.701+0000 7f07f9200700 1 --2- 172.21.15.19:0/3022844081 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f07f40746a0 0x7f07f4074a80 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f07e4009910 tx=0x7f07e4009c20 comp rx=0 tx=0).stop 2024-03-18T15:07:56.703 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:56.701+0000 7f07f9200700 1 -- 172.21.15.19:0/3022844081 shutdown_connections 2024-03-18T15:07:56.703 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:56.701+0000 7f07f9200700 1 --2- 172.21.15.19:0/3022844081 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f07f40746a0 0x7f07f4074a80 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:07:56.703 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:56.701+0000 7f07f9200700 1 -- 172.21.15.19:0/3022844081 >> 172.21.15.19:0/3022844081 conn(0x7f07f40fd7e0 msgr2=0x7f07f40ffc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:07:56.703 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:56.701+0000 7f07f9200700 1 -- 172.21.15.19:0/3022844081 shutdown_connections 2024-03-18T15:07:56.704 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:56.702+0000 7f07f9200700 1 -- 172.21.15.19:0/3022844081 wait complete. 2024-03-18T15:07:56.705 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:56.703+0000 7f07f9200700 1 Processor -- start 2024-03-18T15:07:56.705 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:56.703+0000 7f07f9200700 1 -- start start 2024-03-18T15:07:56.705 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:56.703+0000 7f07f9200700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f07f40746a0 0x7f07f419f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:07:56.705 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:56.703+0000 7f07f9200700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f07f410e550 con 0x7f07f40746a0 2024-03-18T15:07:56.705 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:56.703+0000 7f07f1e00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f07f40746a0 0x7f07f419f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:07:56.706 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:56.704+0000 7f07f1e00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f07f40746a0 0x7f07f419f6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:56486/0 (socket says 172.21.15.19:56486) 2024-03-18T15:07:56.706 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:56.704+0000 7f07f1e00700 1 -- 172.21.15.19:0/929273788 learned_addr learned my addr 172.21.15.19:0/929273788 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:07:56.706 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:56.704+0000 7f07f1e00700 1 -- 172.21.15.19:0/929273788 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07f419fc30 con 0x7f07f40746a0 2024-03-18T15:07:56.706 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:56.704+0000 7f07f1e00700 1 --2- 172.21.15.19:0/929273788 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f07f40746a0 0x7f07f419f6f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f07e4006010 tx=0x7f07e4005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:07:56.706 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:56.705+0000 7f07eaa00700 1 -- 172.21.15.19:0/929273788 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f07e4026070 con 0x7f07f40746a0 2024-03-18T15:07:56.706 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:56.705+0000 7f07eaa00700 1 -- 172.21.15.19:0/929273788 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f07e400bc30 con 0x7f07f40746a0 2024-03-18T15:07:56.707 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:56.705+0000 7f07f9200700 1 -- 172.21.15.19:0/929273788 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07f419fec0 con 0x7f07f40746a0 2024-03-18T15:07:56.707 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:56.705+0000 7f07eaa00700 1 -- 172.21.15.19:0/929273788 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f07e40039a0 con 0x7f07f40746a0 2024-03-18T15:07:56.707 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:56.705+0000 7f07f9200700 1 -- 172.21.15.19:0/929273788 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07f41998b0 con 0x7f07f40746a0 2024-03-18T15:07:56.708 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:56.706+0000 7f07eaa00700 1 -- 172.21.15.19:0/929273788 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f07e4029070 con 0x7f07f40746a0 2024-03-18T15:07:56.708 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:56.706+0000 7f07f9200700 1 -- 172.21.15.19:0/929273788 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07f4075d50 con 0x7f07f40746a0 2024-03-18T15:07:56.708 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:56.706+0000 7f07eaa00700 1 --2- 172.21.15.19:0/929273788 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f07d8039020 0x7f07d803b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:07:56.708 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:56.706+0000 7f07eaa00700 1 -- 172.21.15.19:0/929273788 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f07e401e070 con 0x7f07f40746a0 2024-03-18T15:07:56.708 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:56.707+0000 7f07f1400700 1 --2- 172.21.15.19:0/929273788 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f07d8039020 0x7f07d803b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:07:56.708 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:56.707+0000 7f07f1400700 1 --2- 172.21.15.19:0/929273788 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f07d8039020 0x7f07d803b4e0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f07e0006fd0 tx=0x7f07e0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:07:56.713 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:56.712+0000 7f07eaa00700 1 -- 172.21.15.19:0/929273788 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f07e4013070 con 0x7f07f40746a0 2024-03-18T15:07:56.948 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:56.947+0000 7f07f9200700 1 -- 172.21.15.19:0/929273788 --> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f07f419ade0 con 0x7f07d8039020 2024-03-18T15:07:56.951 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:56.950+0000 7f07eaa00700 1 -- 172.21.15.19:0/929273788 <== mgr.14152 v2:172.21.15.19:6800/2017714236 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+239 (secure 0 0 0) 0x7f07f419ade0 con 0x7f07d8039020 2024-03-18T15:07:56.951 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-18T15:07:56.951 INFO:teuthology.orchestra.run.smithi019.stdout:[{"addr": "172.21.15.19", "hostname": "smithi019", "labels": [], "status": ""}, {"addr": "172.21.15.33", "hostname": "smithi033", "labels": [], "status": ""}, {"addr": "172.21.15.104", "hostname": "smithi104", "labels": [], "status": ""}] 2024-03-18T15:07:56.954 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:56.952+0000 7f07f9200700 1 -- 172.21.15.19:0/929273788 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f07d8039020 msgr2=0x7f07d803b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:07:56.954 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:56.953+0000 7f07f9200700 1 --2- 172.21.15.19:0/929273788 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f07d8039020 0x7f07d803b4e0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f07e0006fd0 tx=0x7f07e0006e40 comp rx=0 tx=0).stop 2024-03-18T15:07:56.954 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:56.953+0000 7f07f9200700 1 -- 172.21.15.19:0/929273788 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f07f40746a0 msgr2=0x7f07f419f6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:07:56.954 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:56.953+0000 7f07f9200700 1 --2- 172.21.15.19:0/929273788 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f07f40746a0 0x7f07f419f6f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f07e4006010 tx=0x7f07e4005030 comp rx=0 tx=0).stop 2024-03-18T15:07:56.954 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:56.953+0000 7f07f9200700 1 -- 172.21.15.19:0/929273788 shutdown_connections 2024-03-18T15:07:56.954 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:56.953+0000 7f07f9200700 1 --2- 172.21.15.19:0/929273788 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f07d8039020 0x7f07d803b4e0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:07:56.955 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:56.953+0000 7f07f9200700 1 --2- 172.21.15.19:0/929273788 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f07f40746a0 0x7f07f419f6f0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:07:56.955 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:56.953+0000 7f07f9200700 1 -- 172.21.15.19:0/929273788 >> 172.21.15.19:0/929273788 conn(0x7f07f40fd7e0 msgr2=0x7f07f4105fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:07:56.955 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:56.953+0000 7f07f9200700 1 -- 172.21.15.19:0/929273788 shutdown_connections 2024-03-18T15:07:56.955 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:07:56.953+0000 7f07f9200700 1 -- 172.21.15.19:0/929273788 wait complete. 2024-03-18T15:07:57.622 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:57 smithi019 ceph-mon[31522]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:07:57.622 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:57 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:07:57.622 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:57 smithi019 ceph-mon[31522]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T15:07:57.696 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-18T15:07:57.696 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 -- ceph osd crush tunables default 2024-03-18T15:07:58.152 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:07:59.566 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:07:59 smithi019 ceph-mon[31522]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:00.078 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:00.076+0000 7fc71de00700 1 -- 172.21.15.19:0/3635598505 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc718102e60 msgr2=0x7fc718103240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:08:00.078 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:00.076+0000 7fc71de00700 1 --2- 172.21.15.19:0/3635598505 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc718102e60 0x7fc718103240 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fc708009910 tx=0x7fc708009c20 comp rx=0 tx=0).stop 2024-03-18T15:08:00.078 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:00.076+0000 7fc71de00700 1 -- 172.21.15.19:0/3635598505 shutdown_connections 2024-03-18T15:08:00.079 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:00.076+0000 7fc71de00700 1 --2- 172.21.15.19:0/3635598505 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc718102e60 0x7fc718103240 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:08:00.079 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:00.076+0000 7fc71de00700 1 -- 172.21.15.19:0/3635598505 >> 172.21.15.19:0/3635598505 conn(0x7fc718077810 msgr2=0x7fc7180feae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:08:00.079 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:00.077+0000 7fc71de00700 1 -- 172.21.15.19:0/3635598505 shutdown_connections 2024-03-18T15:08:00.079 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:00.077+0000 7fc71de00700 1 -- 172.21.15.19:0/3635598505 wait complete. 2024-03-18T15:08:00.080 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:00.079+0000 7fc71de00700 1 Processor -- start 2024-03-18T15:08:00.080 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:00.079+0000 7fc71de00700 1 -- start start 2024-03-18T15:08:00.081 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:00.080+0000 7fc71de00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc718102e60 0x7fc71819d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:08:00.081 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:00.080+0000 7fc71de00700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fc71810c350 con 0x7fc718102e60 2024-03-18T15:08:00.082 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:00.081+0000 7fc716a00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc718102e60 0x7fc71819d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:08:00.082 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:00.081+0000 7fc716a00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc718102e60 0x7fc71819d570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:56510/0 (socket says 172.21.15.19:56510) 2024-03-18T15:08:00.082 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:00.081+0000 7fc716a00700 1 -- 172.21.15.19:0/256177249 learned_addr learned my addr 172.21.15.19:0/256177249 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:08:00.082 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:00.082+0000 7fc716a00700 1 -- 172.21.15.19:0/256177249 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc71819dab0 con 0x7fc718102e60 2024-03-18T15:08:00.083 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:00.082+0000 7fc716a00700 1 --2- 172.21.15.19:0/256177249 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc718102e60 0x7fc71819d570 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fc708006010 tx=0x7fc708005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:08:00.084 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:00.083+0000 7fc71ca00700 1 -- 172.21.15.19:0/256177249 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc708026070 con 0x7fc718102e60 2024-03-18T15:08:00.084 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:00.083+0000 7fc71de00700 1 -- 172.21.15.19:0/256177249 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc71819dd40 con 0x7fc718102e60 2024-03-18T15:08:00.084 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:00.083+0000 7fc71ca00700 1 -- 172.21.15.19:0/256177249 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc70800bc30 con 0x7fc718102e60 2024-03-18T15:08:00.084 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:00.083+0000 7fc71ca00700 1 -- 172.21.15.19:0/256177249 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc7080039a0 con 0x7fc718102e60 2024-03-18T15:08:00.084 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:00.083+0000 7fc71de00700 1 -- 172.21.15.19:0/256177249 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc718197640 con 0x7fc718102e60 2024-03-18T15:08:00.085 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:00.084+0000 7fc71de00700 1 -- 172.21.15.19:0/256177249 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc718069fa0 con 0x7fc718102e60 2024-03-18T15:08:00.088 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:00.087+0000 7fc71ca00700 1 -- 172.21.15.19:0/256177249 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7fc708029070 con 0x7fc718102e60 2024-03-18T15:08:00.088 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:00.087+0000 7fc71ca00700 1 --2- 172.21.15.19:0/256177249 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fc6fc038fc0 0x7fc6fc03b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:08:00.089 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:00.087+0000 7fc71ca00700 1 -- 172.21.15.19:0/256177249 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fc70801e070 con 0x7fc718102e60 2024-03-18T15:08:00.089 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:00.088+0000 7fc716000700 1 --2- 172.21.15.19:0/256177249 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fc6fc038fc0 0x7fc6fc03b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:08:00.089 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:00.088+0000 7fc716000700 1 --2- 172.21.15.19:0/256177249 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fc6fc038fc0 0x7fc6fc03b480 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fc704006fd0 tx=0x7fc704006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:08:00.095 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:00.094+0000 7fc71ca00700 1 -- 172.21.15.19:0/256177249 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc708013070 con 0x7fc718102e60 2024-03-18T15:08:00.331 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:00.329+0000 7fc71de00700 1 -- 172.21.15.19:0/256177249 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7fc718063900 con 0x7fc718102e60 2024-03-18T15:08:00.611 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:00.610+0000 7fc71ca00700 1 -- 172.21.15.19:0/256177249 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7fc708030090 con 0x7fc718102e60 2024-03-18T15:08:00.616 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:00.614+0000 7fc71de00700 1 -- 172.21.15.19:0/256177249 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fc6fc038fc0 msgr2=0x7fc6fc03b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:08:00.616 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:00.614+0000 7fc71de00700 1 --2- 172.21.15.19:0/256177249 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fc6fc038fc0 0x7fc6fc03b480 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fc704006fd0 tx=0x7fc704006e40 comp rx=0 tx=0).stop 2024-03-18T15:08:00.616 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:00.614+0000 7fc71de00700 1 -- 172.21.15.19:0/256177249 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc718102e60 msgr2=0x7fc71819d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:08:00.616 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:00.614+0000 7fc71de00700 1 --2- 172.21.15.19:0/256177249 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc718102e60 0x7fc71819d570 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fc708006010 tx=0x7fc708005030 comp rx=0 tx=0).stop 2024-03-18T15:08:00.616 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:00.615+0000 7fc71de00700 1 -- 172.21.15.19:0/256177249 shutdown_connections 2024-03-18T15:08:00.616 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:00.615+0000 7fc71de00700 1 --2- 172.21.15.19:0/256177249 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fc6fc038fc0 0x7fc6fc03b480 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:08:00.616 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:00.615+0000 7fc71de00700 1 --2- 172.21.15.19:0/256177249 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc718102e60 0x7fc71819d570 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:08:00.617 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:00.615+0000 7fc71de00700 1 -- 172.21.15.19:0/256177249 >> 172.21.15.19:0/256177249 conn(0x7fc718077810 msgr2=0x7fc718103c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:08:00.617 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:00.615+0000 7fc71de00700 1 -- 172.21.15.19:0/256177249 shutdown_connections 2024-03-18T15:08:00.617 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:00.615+0000 7fc71de00700 1 -- 172.21.15.19:0/256177249 wait complete. 2024-03-18T15:08:00.618 INFO:teuthology.orchestra.run.smithi019.stderr:adjusted tunables profile to default 2024-03-18T15:08:00.927 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:00 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:00.928 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:00 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:00.928 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:00 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:00.928 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:00 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:00.928 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:00 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi104", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:08:00.928 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:00 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:08:00.928 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:00 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:08:00.928 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:00 smithi019 ceph-mon[31522]: Updating smithi104:/etc/ceph/ceph.conf 2024-03-18T15:08:00.928 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:00 smithi019 ceph-mon[31522]: Updating smithi104:/var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T15:08:00.928 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:00 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/256177249' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-18T15:08:01.341 INFO:tasks.cephadm:Adding mon.a on smithi019 2024-03-18T15:08:01.341 INFO:tasks.cephadm:Adding mon.b on smithi033 2024-03-18T15:08:01.342 INFO:tasks.cephadm:Adding mon.c on smithi104 2024-03-18T15:08:01.342 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 -- ceph orch apply mon '3;smithi019:172.21.15.19=a;smithi033:172.21.15.33=b;smithi104:172.21.15.104=c' 2024-03-18T15:08:01.568 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T15:08:01.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:01 smithi019 ceph-mon[31522]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:01.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:01 smithi019 ceph-mon[31522]: Updating smithi104:/etc/ceph/ceph.client.admin.keyring 2024-03-18T15:08:01.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:01 smithi019 ceph-mon[31522]: Updating smithi104:/var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-18T15:08:01.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:01 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/256177249' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-18T15:08:01.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:01 smithi019 ceph-mon[31522]: osdmap e4: 0 total, 0 up, 0 in 2024-03-18T15:08:01.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:01 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:01.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:01 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:01.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:01 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:03.024 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:03.006+0000 7f9bb9c73700 1 -- 172.21.15.104:0/368226215 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9bb4101f50 msgr2=0x7f9bb4102330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:08:03.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:03.006+0000 7f9bb9c73700 1 --2- 172.21.15.104:0/368226215 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9bb4101f50 0x7f9bb4102330 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f9ba4009910 tx=0x7f9ba4009c20 comp rx=0 tx=0).stop 2024-03-18T15:08:03.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:03.006+0000 7f9bb9c73700 1 -- 172.21.15.104:0/368226215 shutdown_connections 2024-03-18T15:08:03.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:03.006+0000 7f9bb9c73700 1 --2- 172.21.15.104:0/368226215 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9bb4101f50 0x7f9bb4102330 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:08:03.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:03.006+0000 7f9bb9c73700 1 -- 172.21.15.104:0/368226215 >> 172.21.15.104:0/368226215 conn(0x7f9bb40fd800 msgr2=0x7f9bb40ffc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:08:03.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:03.006+0000 7f9bb9c73700 1 -- 172.21.15.104:0/368226215 shutdown_connections 2024-03-18T15:08:03.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:03.006+0000 7f9bb9c73700 1 -- 172.21.15.104:0/368226215 wait complete. 2024-03-18T15:08:03.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:03.007+0000 7f9bb9c73700 1 Processor -- start 2024-03-18T15:08:03.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:03.007+0000 7f9bb9c73700 1 -- start start 2024-03-18T15:08:03.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:03.007+0000 7f9bb9c73700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9bb4101f50 0x7f9bb419d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:08:03.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:03.007+0000 7f9bb9c73700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f9bb410c320 con 0x7f9bb4101f50 2024-03-18T15:08:03.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:03.008+0000 7f9bb37fe700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9bb4101f50 0x7f9bb419d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:08:03.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:03.008+0000 7f9bb37fe700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9bb4101f50 0x7f9bb419d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.104:45110/0 (socket says 172.21.15.104:45110) 2024-03-18T15:08:03.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:03.008+0000 7f9bb37fe700 1 -- 172.21.15.104:0/1465700890 learned_addr learned my addr 172.21.15.104:0/1465700890 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-18T15:08:03.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:03.008+0000 7f9bb37fe700 1 -- 172.21.15.104:0/1465700890 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bb419d990 con 0x7f9bb4101f50 2024-03-18T15:08:03.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:03.009+0000 7f9bb37fe700 1 --2- 172.21.15.104:0/1465700890 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9bb4101f50 0x7f9bb419d450 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f9ba4005850 tx=0x7f9ba4005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:08:03.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:03.009+0000 7f9bb0ff9700 1 -- 172.21.15.104:0/1465700890 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9ba4026070 con 0x7f9bb4101f50 2024-03-18T15:08:03.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:03.009+0000 7f9bb9c73700 1 -- 172.21.15.104:0/1465700890 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9bb419dc80 con 0x7f9bb4101f50 2024-03-18T15:08:03.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:03.009+0000 7f9bb9c73700 1 -- 172.21.15.104:0/1465700890 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9bb4197610 con 0x7f9bb4101f50 2024-03-18T15:08:03.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:03.009+0000 7f9bb0ff9700 1 -- 172.21.15.104:0/1465700890 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ba400b6f0 con 0x7f9bb4101f50 2024-03-18T15:08:03.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:03.009+0000 7f9bb0ff9700 1 -- 172.21.15.104:0/1465700890 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9ba4030060 con 0x7f9bb4101f50 2024-03-18T15:08:03.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:03.010+0000 7f9bb9c73700 1 -- 172.21.15.104:0/1465700890 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ba0005330 con 0x7f9bb4101f50 2024-03-18T15:08:03.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:03.011+0000 7f9bb0ff9700 1 -- 172.21.15.104:0/1465700890 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f9ba4029070 con 0x7f9bb4101f50 2024-03-18T15:08:03.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:03.011+0000 7f9bb0ff9700 1 --2- 172.21.15.104:0/1465700890 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f9b9c038fc0 0x7f9b9c03b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:08:03.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:03.011+0000 7f9bb0ff9700 1 -- 172.21.15.104:0/1465700890 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f9ba401e070 con 0x7f9bb4101f50 2024-03-18T15:08:03.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:03.011+0000 7f9bb2ffd700 1 --2- 172.21.15.104:0/1465700890 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f9b9c038fc0 0x7f9b9c03b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:08:03.027 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:03.012+0000 7f9bb2ffd700 1 --2- 172.21.15.104:0/1465700890 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f9b9c038fc0 0x7f9b9c03b480 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f9ba8006fd0 tx=0x7f9ba8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:08:03.027 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:03.026+0000 7f9bb0ff9700 1 -- 172.21.15.104:0/1465700890 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9ba407b740 con 0x7f9bb4101f50 2024-03-18T15:08:03.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:03.232+0000 7f9bb9c73700 1 -- 172.21.15.104:0/1465700890 --> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi019:172.21.15.19=a;smithi033:172.21.15.33=b;smithi104:172.21.15.104=c", "target": ["mon-mgr", ""]}) -- 0x7f9ba0000c80 con 0x7f9b9c038fc0 2024-03-18T15:08:03.247 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:03.248+0000 7f9bb0ff9700 1 -- 172.21.15.104:0/1465700890 <== mgr.14152 v2:172.21.15.19:6800/2017714236 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f9ba0000c80 con 0x7f9b9c038fc0 2024-03-18T15:08:03.247 INFO:teuthology.orchestra.run.smithi104.stdout:Scheduled mon update... 2024-03-18T15:08:03.249 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:03.249+0000 7f9bb9c73700 1 -- 172.21.15.104:0/1465700890 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f9b9c038fc0 msgr2=0x7f9b9c03b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:08:03.250 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:03.249+0000 7f9bb9c73700 1 --2- 172.21.15.104:0/1465700890 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f9b9c038fc0 0x7f9b9c03b480 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f9ba8006fd0 tx=0x7f9ba8006e40 comp rx=0 tx=0).stop 2024-03-18T15:08:03.250 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:03.249+0000 7f9bb9c73700 1 -- 172.21.15.104:0/1465700890 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9bb4101f50 msgr2=0x7f9bb419d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:08:03.250 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:03.249+0000 7f9bb9c73700 1 --2- 172.21.15.104:0/1465700890 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9bb4101f50 0x7f9bb419d450 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f9ba4005850 tx=0x7f9ba4005030 comp rx=0 tx=0).stop 2024-03-18T15:08:03.250 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:03.249+0000 7f9bb9c73700 1 -- 172.21.15.104:0/1465700890 shutdown_connections 2024-03-18T15:08:03.250 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:03.249+0000 7f9bb9c73700 1 --2- 172.21.15.104:0/1465700890 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f9b9c038fc0 0x7f9b9c03b480 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:08:03.250 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:03.249+0000 7f9bb9c73700 1 --2- 172.21.15.104:0/1465700890 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9bb4101f50 0x7f9bb419d450 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:08:03.250 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:03.249+0000 7f9bb9c73700 1 -- 172.21.15.104:0/1465700890 >> 172.21.15.104:0/1465700890 conn(0x7f9bb40fd800 msgr2=0x7f9bb4103920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:08:03.250 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:03.249+0000 7f9bb9c73700 1 -- 172.21.15.104:0/1465700890 shutdown_connections 2024-03-18T15:08:03.250 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:03.249+0000 7f9bb9c73700 1 -- 172.21.15.104:0/1465700890 wait complete. 2024-03-18T15:08:03.806 DEBUG:teuthology.orchestra.run.smithi033:mon.b> sudo journalctl -f -n 0 -u ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@mon.b.service 2024-03-18T15:08:03.809 DEBUG:teuthology.orchestra.run.smithi104:mon.c> sudo journalctl -f -n 0 -u ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@mon.c.service 2024-03-18T15:08:03.811 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-18T15:08:03.811 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-18T15:08:03.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:03 smithi019 ceph-mon[31522]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:03.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:03 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:03.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:03 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:08:03.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:03 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:08:03.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:03 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:08:03.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:03 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:03.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:03 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T15:08:03.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:03 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:08:04.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:04 smithi019 ceph-mon[31522]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi019:172.21.15.19=a;smithi033:172.21.15.33=b;smithi104:172.21.15.104=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:08:04.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:04 smithi019 ceph-mon[31522]: Saving service mon spec with placement smithi019:172.21.15.19=a;smithi033:172.21.15.33=b;smithi104:172.21.15.104=c;count:3 2024-03-18T15:08:04.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:04 smithi019 ceph-mon[31522]: Deploying daemon mon.c on smithi104 2024-03-18T15:08:05.925 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T15:08:05.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:05 smithi019 ceph-mon[31522]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:07.567 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 podman[36012]: 2024-03-18 15:08:07.393299946 +0000 UTC m=+0.197128364 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T15:08:07.585 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:07.584+0000 7f7ba6a15700 1 -- 172.21.15.104:0/143853486 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7ba0102fa0 msgr2=0x7f7ba0103380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:08:07.585 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:07.584+0000 7f7ba6a15700 1 --2- 172.21.15.104:0/143853486 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7ba0102fa0 0x7f7ba0103380 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f7b90009910 tx=0x7f7b90009c20 comp rx=0 tx=0).stop 2024-03-18T15:08:07.585 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:07.584+0000 7f7ba6a15700 1 -- 172.21.15.104:0/143853486 shutdown_connections 2024-03-18T15:08:07.586 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:07.584+0000 7f7ba6a15700 1 --2- 172.21.15.104:0/143853486 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7ba0102fa0 0x7f7ba0103380 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:08:07.586 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:07.584+0000 7f7ba6a15700 1 -- 172.21.15.104:0/143853486 >> 172.21.15.104:0/143853486 conn(0x7f7ba00fc7f0 msgr2=0x7f7ba00fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:08:07.586 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:07.584+0000 7f7ba6a15700 1 -- 172.21.15.104:0/143853486 shutdown_connections 2024-03-18T15:08:07.586 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:07.584+0000 7f7ba6a15700 1 -- 172.21.15.104:0/143853486 wait complete. 2024-03-18T15:08:07.586 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:07.584+0000 7f7ba6a15700 1 Processor -- start 2024-03-18T15:08:07.586 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:07.584+0000 7f7ba6a15700 1 -- start start 2024-03-18T15:08:07.586 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:07.585+0000 7f7ba6a15700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7ba0102fa0 0x7f7ba019d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:08:07.586 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:07.585+0000 7f7ba6a15700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f7ba010c490 con 0x7f7ba0102fa0 2024-03-18T15:08:07.587 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:07.585+0000 7f7b9ffff700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7ba0102fa0 0x7f7ba019d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:08:07.587 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:07.585+0000 7f7b9ffff700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7ba0102fa0 0x7f7ba019d670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.104:45136/0 (socket says 172.21.15.104:45136) 2024-03-18T15:08:07.587 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:07.585+0000 7f7b9ffff700 1 -- 172.21.15.104:0/127633963 learned_addr learned my addr 172.21.15.104:0/127633963 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-18T15:08:07.587 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:07.586+0000 7f7b9ffff700 1 -- 172.21.15.104:0/127633963 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ba019dbb0 con 0x7f7ba0102fa0 2024-03-18T15:08:07.587 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:07.586+0000 7f7b9ffff700 1 --2- 172.21.15.104:0/127633963 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7ba0102fa0 0x7f7ba019d670 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f7b90006010 tx=0x7f7b90005050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:08:07.588 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:07.587+0000 7f7b9d7fa700 1 -- 172.21.15.104:0/127633963 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7b90026070 con 0x7f7ba0102fa0 2024-03-18T15:08:07.588 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:07.587+0000 7f7ba6a15700 1 -- 172.21.15.104:0/127633963 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ba019de40 con 0x7f7ba0102fa0 2024-03-18T15:08:07.588 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:07.587+0000 7f7ba6a15700 1 -- 172.21.15.104:0/127633963 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ba01977f0 con 0x7f7ba0102fa0 2024-03-18T15:08:07.588 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:07.587+0000 7f7b9d7fa700 1 -- 172.21.15.104:0/127633963 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b90004830 con 0x7f7ba0102fa0 2024-03-18T15:08:07.588 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:07.587+0000 7f7b9d7fa700 1 -- 172.21.15.104:0/127633963 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7b90030060 con 0x7f7ba0102fa0 2024-03-18T15:08:07.588 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:07.588+0000 7f7ba6a15700 1 -- 172.21.15.104:0/127633963 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b8c005330 con 0x7f7ba0102fa0 2024-03-18T15:08:07.589 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:07.588+0000 7f7b9d7fa700 1 -- 172.21.15.104:0/127633963 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f7b90029070 con 0x7f7ba0102fa0 2024-03-18T15:08:07.589 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:07.589+0000 7f7b9d7fa700 1 --2- 172.21.15.104:0/127633963 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f7b88038fc0 0x7f7b8803b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:08:07.589 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:07.589+0000 7f7b9d7fa700 1 -- 172.21.15.104:0/127633963 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f7b9001e070 con 0x7f7ba0102fa0 2024-03-18T15:08:07.590 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:07.589+0000 7f7b9f7fe700 1 --2- 172.21.15.104:0/127633963 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f7b88038fc0 0x7f7b8803b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:08:07.590 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:07.590+0000 7f7b9f7fe700 1 --2- 172.21.15.104:0/127633963 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f7b88038fc0 0x7f7b8803b480 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f7b94006fd0 tx=0x7f7b94006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:08:07.604 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:07.604+0000 7f7b9d7fa700 1 -- 172.21.15.104:0/127633963 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7b90013070 con 0x7f7ba0102fa0 2024-03-18T15:08:07.886 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 podman[36012]: 2024-03-18 15:08:07.667787163 +0000 UTC m=+0.471615600 container create b6b4027b63821f2b12853abc725550da8b7f89ecf8e4ce4543d33e802404ed2f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mon-c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T15:08:07.887 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:07.885+0000 7f7ba6a15700 1 -- 172.21.15.104:0/127633963 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f7b8c0058f0 con 0x7f7ba0102fa0 2024-03-18T15:08:07.887 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:07.886+0000 7f7b9d7fa700 1 -- 172.21.15.104:0/127633963 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+745 (secure 0 0 0) 0x7f7b9004b920 con 0x7f7ba0102fa0 2024-03-18T15:08:07.887 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-18T15:08:07.887 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":1,"fsid":"d4dc2ba6-e538-11ee-95c9-87774f69a715","modified":"2024-03-18T15:05:53.426642Z","created":"2024-03-18T15:05:53.426642Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-18T15:08:07.889 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:07.888+0000 7f7ba6a15700 1 -- 172.21.15.104:0/127633963 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f7b88038fc0 msgr2=0x7f7b8803b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:08:07.889 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:07.888+0000 7f7ba6a15700 1 --2- 172.21.15.104:0/127633963 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f7b88038fc0 0x7f7b8803b480 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f7b94006fd0 tx=0x7f7b94006e40 comp rx=0 tx=0).stop 2024-03-18T15:08:07.890 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:07.888+0000 7f7ba6a15700 1 -- 172.21.15.104:0/127633963 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7ba0102fa0 msgr2=0x7f7ba019d670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:08:07.890 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:07.888+0000 7f7ba6a15700 1 --2- 172.21.15.104:0/127633963 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7ba0102fa0 0x7f7ba019d670 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f7b90006010 tx=0x7f7b90005050 comp rx=0 tx=0).stop 2024-03-18T15:08:07.890 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:07.889+0000 7f7ba6a15700 1 -- 172.21.15.104:0/127633963 shutdown_connections 2024-03-18T15:08:07.890 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:07.889+0000 7f7ba6a15700 1 --2- 172.21.15.104:0/127633963 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f7b88038fc0 0x7f7b8803b480 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:08:07.890 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:07.889+0000 7f7ba6a15700 1 --2- 172.21.15.104:0/127633963 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7ba0102fa0 0x7f7ba019d670 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:08:07.890 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:07.889+0000 7f7ba6a15700 1 -- 172.21.15.104:0/127633963 >> 172.21.15.104:0/127633963 conn(0x7f7ba00fc7f0 msgr2=0x7f7ba0103d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:08:07.890 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:07.889+0000 7f7ba6a15700 1 -- 172.21.15.104:0/127633963 shutdown_connections 2024-03-18T15:08:07.890 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:07.889+0000 7f7ba6a15700 1 -- 172.21.15.104:0/127633963 wait complete. 2024-03-18T15:08:07.891 INFO:teuthology.orchestra.run.smithi104.stderr:dumped monmap epoch 1 2024-03-18T15:08:07.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:07 smithi019 ceph-mon[31522]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:08.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 podman[36012]: 2024-03-18 15:08:07.945472787 +0000 UTC m=+0.749301190 container init b6b4027b63821f2b12853abc725550da8b7f89ecf8e4ce4543d33e802404ed2f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mon-c, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-18T15:08:08.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 podman[36012]: 2024-03-18 15:08:07.948613915 +0000 UTC m=+0.752442312 container start b6b4027b63821f2b12853abc725550da8b7f89ecf8e4ce4543d33e802404ed2f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mon-c, org.label-schema.build-date=20240307, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T15:08:08.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: set uid:gid to 167:167 (ceph:ceph) 2024-03-18T15:08:08.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev), process ceph-mon, pid 2 2024-03-18T15:08:08.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: pidfile_write: ignore empty --pid-file 2024-03-18T15:08:08.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: load: jerasure load: lrc 2024-03-18T15:08:08.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: RocksDB version: 7.9.2 2024-03-18T15:08:08.196 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Git sha 0 2024-03-18T15:08:08.196 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Compile date 2024-03-15 00:33:34 2024-03-18T15:08:08.196 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: DB SUMMARY 2024-03-18T15:08:08.196 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: DB Session ID: X772115GRS04JQ3XZ0I0 2024-03-18T15:08:08.196 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: CURRENT file: CURRENT 2024-03-18T15:08:08.196 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: IDENTITY file: IDENTITY 2024-03-18T15:08:08.196 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-18T15:08:08.196 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-03-18T15:08:08.196 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-03-18T15:08:08.196 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.error_if_exists: 0 2024-03-18T15:08:08.196 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.create_if_missing: 0 2024-03-18T15:08:08.196 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.paranoid_checks: 1 2024-03-18T15:08:08.196 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-18T15:08:08.197 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-18T15:08:08.197 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-18T15:08:08.197 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.env: 0x5570a1bddc60 2024-03-18T15:08:08.197 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.fs: PosixFileSystem 2024-03-18T15:08:08.197 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.info_log: 0x5570a2fc7ac0 2024-03-18T15:08:08.197 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.max_file_opening_threads: 16 2024-03-18T15:08:08.199 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.statistics: (nil) 2024-03-18T15:08:08.199 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.use_fsync: 0 2024-03-18T15:08:08.199 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.max_log_file_size: 0 2024-03-18T15:08:08.199 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-18T15:08:08.199 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-18T15:08:08.199 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.keep_log_file_num: 1000 2024-03-18T15:08:08.199 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.recycle_log_file_num: 0 2024-03-18T15:08:08.199 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.allow_fallocate: 1 2024-03-18T15:08:08.200 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.allow_mmap_reads: 0 2024-03-18T15:08:08.200 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.allow_mmap_writes: 0 2024-03-18T15:08:08.200 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.use_direct_reads: 0 2024-03-18T15:08:08.200 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-18T15:08:08.200 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.create_missing_column_families: 0 2024-03-18T15:08:08.200 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.db_log_dir: 2024-03-18T15:08:08.200 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.wal_dir: 2024-03-18T15:08:08.200 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-18T15:08:08.200 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-18T15:08:08.200 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-18T15:08:08.200 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-18T15:08:08.200 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-18T15:08:08.200 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-18T15:08:08.200 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.advise_random_on_open: 1 2024-03-18T15:08:08.201 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.db_write_buffer_size: 0 2024-03-18T15:08:08.201 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.write_buffer_manager: 0x5570a2f9a1e0 2024-03-18T15:08:08.201 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-18T15:08:08.201 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-18T15:08:08.201 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-18T15:08:08.201 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.rate_limiter: (nil) 2024-03-18T15:08:08.201 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-18T15:08:08.201 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.wal_recovery_mode: 2 2024-03-18T15:08:08.201 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.enable_thread_tracking: 0 2024-03-18T15:08:08.201 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.enable_pipelined_write: 0 2024-03-18T15:08:08.201 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.unordered_write: 0 2024-03-18T15:08:08.201 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-18T15:08:08.201 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-18T15:08:08.202 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-18T15:08:08.202 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-18T15:08:08.202 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.row_cache: None 2024-03-18T15:08:08.202 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.wal_filter: None 2024-03-18T15:08:08.202 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-18T15:08:08.202 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.allow_ingest_behind: 0 2024-03-18T15:08:08.202 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.two_write_queues: 0 2024-03-18T15:08:08.202 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.manual_wal_flush: 0 2024-03-18T15:08:08.202 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.wal_compression: 0 2024-03-18T15:08:08.202 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.atomic_flush: 0 2024-03-18T15:08:08.202 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-18T15:08:08.202 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-18T15:08:08.202 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-18T15:08:08.203 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.log_readahead_size: 0 2024-03-18T15:08:08.203 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-18T15:08:08.203 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.best_efforts_recovery: 0 2024-03-18T15:08:08.203 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-18T15:08:08.203 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-18T15:08:08.203 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.allow_data_in_errors: 0 2024-03-18T15:08:08.203 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.db_host_id: __hostname__ 2024-03-18T15:08:08.203 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-18T15:08:08.203 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.max_background_jobs: 2 2024-03-18T15:08:08.203 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.max_background_compactions: -1 2024-03-18T15:08:08.203 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.max_subcompactions: 1 2024-03-18T15:08:08.203 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-18T15:08:08.203 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-18T15:08:08.203 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-18T15:08:08.204 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.max_total_wal_size: 0 2024-03-18T15:08:08.204 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-18T15:08:08.204 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-18T15:08:08.204 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-18T15:08:08.204 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-18T15:08:08.204 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.max_open_files: -1 2024-03-18T15:08:08.204 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.bytes_per_sync: 0 2024-03-18T15:08:08.204 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-18T15:08:08.204 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-18T15:08:08.204 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.compaction_readahead_size: 0 2024-03-18T15:08:08.204 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.max_background_flushes: -1 2024-03-18T15:08:08.204 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Compression algorithms supported: 2024-03-18T15:08:08.204 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-18T15:08:08.204 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: kZSTD supported: 0 2024-03-18T15:08:08.205 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: kXpressCompression supported: 0 2024-03-18T15:08:08.205 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: kLZ4HCCompression supported: 1 2024-03-18T15:08:08.205 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: kZlibCompression supported: 1 2024-03-18T15:08:08.205 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: kSnappyCompression supported: 1 2024-03-18T15:08:08.205 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: kLZ4Compression supported: 1 2024-03-18T15:08:08.205 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: kBZip2Compression supported: 0 2024-03-18T15:08:08.205 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-18T15:08:08.205 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-18T15:08:08.205 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-03-18T15:08:08.205 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-18T15:08:08.205 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-18T15:08:08.205 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.merge_operator: 2024-03-18T15:08:08.205 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.compaction_filter: None 2024-03-18T15:08:08.205 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.compaction_filter_factory: None 2024-03-18T15:08:08.206 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.sst_partitioner_factory: None 2024-03-18T15:08:08.206 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-18T15:08:08.206 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-18T15:08:08.206 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5570a2fc7bc0) 2024-03-18T15:08:08.206 INFO:journalctl@ceph.mon.c.smithi104.stdout: cache_index_and_filter_blocks: 1 2024-03-18T15:08:08.206 INFO:journalctl@ceph.mon.c.smithi104.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-18T15:08:08.206 INFO:journalctl@ceph.mon.c.smithi104.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-18T15:08:08.206 INFO:journalctl@ceph.mon.c.smithi104.stdout: pin_top_level_index_and_filter: 1 2024-03-18T15:08:08.206 INFO:journalctl@ceph.mon.c.smithi104.stdout: index_type: 0 2024-03-18T15:08:08.206 INFO:journalctl@ceph.mon.c.smithi104.stdout: data_block_index_type: 0 2024-03-18T15:08:08.206 INFO:journalctl@ceph.mon.c.smithi104.stdout: index_shortening: 1 2024-03-18T15:08:08.206 INFO:journalctl@ceph.mon.c.smithi104.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-18T15:08:08.206 INFO:journalctl@ceph.mon.c.smithi104.stdout: checksum: 4 2024-03-18T15:08:08.206 INFO:journalctl@ceph.mon.c.smithi104.stdout: no_block_cache: 0 2024-03-18T15:08:08.206 INFO:journalctl@ceph.mon.c.smithi104.stdout: block_cache: 0x5570a2fcd090 2024-03-18T15:08:08.207 INFO:journalctl@ceph.mon.c.smithi104.stdout: block_cache_name: BinnedLRUCache 2024-03-18T15:08:08.207 INFO:journalctl@ceph.mon.c.smithi104.stdout: block_cache_options: 2024-03-18T15:08:08.207 INFO:journalctl@ceph.mon.c.smithi104.stdout: capacity : 536870912 2024-03-18T15:08:08.207 INFO:journalctl@ceph.mon.c.smithi104.stdout: num_shard_bits : 4 2024-03-18T15:08:08.207 INFO:journalctl@ceph.mon.c.smithi104.stdout: strict_capacity_limit : 0 2024-03-18T15:08:08.207 INFO:journalctl@ceph.mon.c.smithi104.stdout: high_pri_pool_ratio: 0.000 2024-03-18T15:08:08.207 INFO:journalctl@ceph.mon.c.smithi104.stdout: block_cache_compressed: (nil) 2024-03-18T15:08:08.207 INFO:journalctl@ceph.mon.c.smithi104.stdout: persistent_cache: (nil) 2024-03-18T15:08:08.207 INFO:journalctl@ceph.mon.c.smithi104.stdout: block_size: 4096 2024-03-18T15:08:08.207 INFO:journalctl@ceph.mon.c.smithi104.stdout: block_size_deviation: 10 2024-03-18T15:08:08.207 INFO:journalctl@ceph.mon.c.smithi104.stdout: block_restart_interval: 16 2024-03-18T15:08:08.207 INFO:journalctl@ceph.mon.c.smithi104.stdout: index_block_restart_interval: 1 2024-03-18T15:08:08.207 INFO:journalctl@ceph.mon.c.smithi104.stdout: metadata_block_size: 4096 2024-03-18T15:08:08.207 INFO:journalctl@ceph.mon.c.smithi104.stdout: partition_filters: 0 2024-03-18T15:08:08.208 INFO:journalctl@ceph.mon.c.smithi104.stdout: use_delta_encoding: 1 2024-03-18T15:08:08.208 INFO:journalctl@ceph.mon.c.smithi104.stdout: filter_policy: bloomfilter 2024-03-18T15:08:08.208 INFO:journalctl@ceph.mon.c.smithi104.stdout: whole_key_filtering: 1 2024-03-18T15:08:08.208 INFO:journalctl@ceph.mon.c.smithi104.stdout: verify_compression: 0 2024-03-18T15:08:08.208 INFO:journalctl@ceph.mon.c.smithi104.stdout: read_amp_bytes_per_bit: 0 2024-03-18T15:08:08.208 INFO:journalctl@ceph.mon.c.smithi104.stdout: format_version: 5 2024-03-18T15:08:08.208 INFO:journalctl@ceph.mon.c.smithi104.stdout: enable_index_compression: 1 2024-03-18T15:08:08.208 INFO:journalctl@ceph.mon.c.smithi104.stdout: block_align: 0 2024-03-18T15:08:08.208 INFO:journalctl@ceph.mon.c.smithi104.stdout: max_auto_readahead_size: 262144 2024-03-18T15:08:08.208 INFO:journalctl@ceph.mon.c.smithi104.stdout: prepopulate_block_cache: 0 2024-03-18T15:08:08.208 INFO:journalctl@ceph.mon.c.smithi104.stdout: initial_auto_readahead_size: 8192 2024-03-18T15:08:08.208 INFO:journalctl@ceph.mon.c.smithi104.stdout: num_file_reads_for_auto_readahead: 2 2024-03-18T15:08:08.208 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.write_buffer_size: 33554432 2024-03-18T15:08:08.208 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.max_write_buffer_number: 2 2024-03-18T15:08:08.208 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.compression: NoCompression 2024-03-18T15:08:08.209 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.bottommost_compression: Disabled 2024-03-18T15:08:08.209 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.prefix_extractor: nullptr 2024-03-18T15:08:08.209 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-18T15:08:08.209 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.num_levels: 7 2024-03-18T15:08:08.209 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-18T15:08:08.209 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-18T15:08:08.209 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-18T15:08:08.209 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-18T15:08:08.209 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-18T15:08:08.209 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-18T15:08:08.209 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-18T15:08:08.209 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-18T15:08:08.209 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-18T15:08:08.209 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-18T15:08:08.209 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-18T15:08:08.209 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-18T15:08:08.210 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-18T15:08:08.210 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.compression_opts.level: 32767 2024-03-18T15:08:08.210 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.compression_opts.strategy: 0 2024-03-18T15:08:08.210 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-18T15:08:08.210 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-18T15:08:08.210 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-18T15:08:08.210 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-18T15:08:08.210 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.compression_opts.enabled: false 2024-03-18T15:08:08.210 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-18T15:08:08.210 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-18T15:08:08.210 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-18T15:08:08.210 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-18T15:08:08.210 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.target_file_size_base: 67108864 2024-03-18T15:08:08.210 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-18T15:08:08.211 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-18T15:08:08.211 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-18T15:08:08.211 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-18T15:08:08.211 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-18T15:08:08.211 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-18T15:08:08.211 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-18T15:08:08.211 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-18T15:08:08.211 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-18T15:08:08.211 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-18T15:08:08.211 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-18T15:08:08.211 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-18T15:08:08.211 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-18T15:08:08.211 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-18T15:08:08.211 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.arena_block_size: 1048576 2024-03-18T15:08:08.211 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-18T15:08:08.212 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-18T15:08:08.212 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.disable_auto_compactions: 0 2024-03-18T15:08:08.212 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-18T15:08:08.212 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-18T15:08:08.212 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-18T15:08:08.212 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-18T15:08:08.212 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-18T15:08:08.212 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-18T15:08:08.212 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-18T15:08:08.212 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-18T15:08:08.212 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-18T15:08:08.212 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-18T15:08:08.212 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.table_properties_collectors: 2024-03-18T15:08:08.212 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.inplace_update_support: 0 2024-03-18T15:08:08.212 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-18T15:08:08.213 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-18T15:08:08.213 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-18T15:08:08.213 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-18T15:08:08.213 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.bloom_locality: 0 2024-03-18T15:08:08.213 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.max_successive_merges: 0 2024-03-18T15:08:08.213 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-18T15:08:08.213 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.paranoid_file_checks: 0 2024-03-18T15:08:08.213 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.force_consistency_checks: 1 2024-03-18T15:08:08.213 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.report_bg_io_stats: 0 2024-03-18T15:08:08.213 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.ttl: 2592000 2024-03-18T15:08:08.213 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-18T15:08:08.213 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-18T15:08:08.213 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-18T15:08:08.213 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.enable_blob_files: false 2024-03-18T15:08:08.213 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.min_blob_size: 0 2024-03-18T15:08:08.214 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.blob_file_size: 268435456 2024-03-18T15:08:08.214 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-18T15:08:08.214 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-18T15:08:08.214 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-18T15:08:08.214 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-18T15:08:08.214 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-18T15:08:08.214 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.blob_file_starting_level: 0 2024-03-18T15:08:08.214 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-18T15:08:08.214 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-18T15:08:08.214 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-18T15:08:08.214 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: d0cf6b4b-3573-49f9-8ce4-b02b760a3af9 2024-03-18T15:08:08.214 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710774487977722, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-18T15:08:08.214 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-18T15:08:08.214 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710774487978144, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710774487, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d0cf6b4b-3573-49f9-8ce4-b02b760a3af9", "db_session_id": "X772115GRS04JQ3XZ0I0", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-18T15:08:08.214 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710774487978222, "job": 1, "event": "recovery_finished"} 2024-03-18T15:08:08.215 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-18T15:08:08.215 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-18T15:08:08.215 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5570a30c0000 2024-03-18T15:08:08.215 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: DB pointer 0x5570a30aa000 2024-03-18T15:08:08.215 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-18T15:08:08.215 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-18T15:08:08.215 INFO:journalctl@ceph.mon.c.smithi104.stdout: ** DB Stats ** 2024-03-18T15:08:08.215 INFO:journalctl@ceph.mon.c.smithi104.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-18T15:08:08.215 INFO:journalctl@ceph.mon.c.smithi104.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-18T15:08:08.215 INFO:journalctl@ceph.mon.c.smithi104.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-18T15:08:08.215 INFO:journalctl@ceph.mon.c.smithi104.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-18T15:08:08.215 INFO:journalctl@ceph.mon.c.smithi104.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-18T15:08:08.215 INFO:journalctl@ceph.mon.c.smithi104.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-18T15:08:08.215 INFO:journalctl@ceph.mon.c.smithi104.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-18T15:08:08.215 INFO:journalctl@ceph.mon.c.smithi104.stdout: 2024-03-18T15:08:08.216 INFO:journalctl@ceph.mon.c.smithi104.stdout: ** Compaction Stats [default] ** 2024-03-18T15:08:08.216 INFO:journalctl@ceph.mon.c.smithi104.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-18T15:08:08.216 INFO:journalctl@ceph.mon.c.smithi104.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-18T15:08:08.216 INFO:journalctl@ceph.mon.c.smithi104.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-18T15:08:08.216 INFO:journalctl@ceph.mon.c.smithi104.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-18T15:08:08.216 INFO:journalctl@ceph.mon.c.smithi104.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-18T15:08:08.216 INFO:journalctl@ceph.mon.c.smithi104.stdout: 2024-03-18T15:08:08.216 INFO:journalctl@ceph.mon.c.smithi104.stdout: ** Compaction Stats [default] ** 2024-03-18T15:08:08.216 INFO:journalctl@ceph.mon.c.smithi104.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-18T15:08:08.216 INFO:journalctl@ceph.mon.c.smithi104.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-18T15:08:08.216 INFO:journalctl@ceph.mon.c.smithi104.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-18T15:08:08.216 INFO:journalctl@ceph.mon.c.smithi104.stdout: 2024-03-18T15:08:08.216 INFO:journalctl@ceph.mon.c.smithi104.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-18T15:08:08.216 INFO:journalctl@ceph.mon.c.smithi104.stdout: 2024-03-18T15:08:08.216 INFO:journalctl@ceph.mon.c.smithi104.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-18T15:08:08.216 INFO:journalctl@ceph.mon.c.smithi104.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-18T15:08:08.217 INFO:journalctl@ceph.mon.c.smithi104.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-18T15:08:08.217 INFO:journalctl@ceph.mon.c.smithi104.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-18T15:08:08.217 INFO:journalctl@ceph.mon.c.smithi104.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-18T15:08:08.217 INFO:journalctl@ceph.mon.c.smithi104.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-18T15:08:08.217 INFO:journalctl@ceph.mon.c.smithi104.stdout: Cumulative compaction: 0.00 GB write, 0.65 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-18T15:08:08.217 INFO:journalctl@ceph.mon.c.smithi104.stdout: Interval compaction: 0.00 GB write, 0.65 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-18T15:08:08.217 INFO:journalctl@ceph.mon.c.smithi104.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-18T15:08:08.217 INFO:journalctl@ceph.mon.c.smithi104.stdout: Block cache BinnedLRUCache@0x5570a2fcd090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-18T15:08:08.217 INFO:journalctl@ceph.mon.c.smithi104.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-18T15:08:08.217 INFO:journalctl@ceph.mon.c.smithi104.stdout: 2024-03-18T15:08:08.217 INFO:journalctl@ceph.mon.c.smithi104.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-18T15:08:08.217 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-03-18T15:08:08.217 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: using public_addr v2:172.21.15.104:0/0 -> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-03-18T15:08:08.217 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: starting mon.c rank -1 at public addrs [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] at bind addrs [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 2024-03-18T15:08:08.218 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:07 smithi104 ceph-mon[36070]: mon.c@-1(???) e0 preinit fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 2024-03-18T15:08:08.218 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:08 smithi104 ceph-mon[36070]: mon.c@-1(synchronizing).mds e1 new map 2024-03-18T15:08:08.218 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:08 smithi104 ceph-mon[36070]: mon.c@-1(synchronizing).mds e1 print_map 2024-03-18T15:08:08.218 INFO:journalctl@ceph.mon.c.smithi104.stdout: e1 2024-03-18T15:08:08.218 INFO:journalctl@ceph.mon.c.smithi104.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-18T15:08:08.218 INFO:journalctl@ceph.mon.c.smithi104.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-18T15:08:08.218 INFO:journalctl@ceph.mon.c.smithi104.stdout: legacy client fscid: -1 2024-03-18T15:08:08.218 INFO:journalctl@ceph.mon.c.smithi104.stdout: 2024-03-18T15:08:08.218 INFO:journalctl@ceph.mon.c.smithi104.stdout: No filesystems configured 2024-03-18T15:08:08.218 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:08 smithi104 ceph-mon[36070]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-18T15:08:08.218 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:08 smithi104 ceph-mon[36070]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-18T15:08:08.218 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:08 smithi104 ceph-mon[36070]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-18T15:08:08.218 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:08 smithi104 ceph-mon[36070]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-18T15:08:08.219 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:08 smithi104 ceph-mon[36070]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-18T15:08:08.219 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:08 smithi104 ceph-mon[36070]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-18T15:08:08.219 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:08 smithi104 ceph-mon[36070]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-18T15:08:08.219 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:08 smithi104 ceph-mon[36070]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-18T15:08:08.219 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:08 smithi104 ceph-mon[36070]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-18T15:08:08.219 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:08 smithi104 ceph-mon[36070]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-18T15:08:08.219 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:08 smithi104 ceph-mon[36070]: Deploying cephadm binary to smithi033 2024-03-18T15:08:08.219 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:08 smithi104 ceph-mon[36070]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:08.219 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:08 smithi104 ceph-mon[36070]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:08.219 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:08 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:08.219 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:08 smithi104 ceph-mon[36070]: Added host smithi033 2024-03-18T15:08:08.219 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:08 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:08:08.219 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:08 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:08.219 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:08 smithi104 ceph-mon[36070]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:08.220 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:08 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:08.220 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:08 smithi104 ceph-mon[36070]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:08.220 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:08 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:08.220 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:08 smithi104 ceph-mon[36070]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T15:08:08.220 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:08 smithi104 ceph-mon[36070]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:08.220 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:08 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:08.220 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:08 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:08.220 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:08 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:08.220 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:08 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:08.220 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:08 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi033", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:08:08.220 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:08 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:08:08.220 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:08 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:08:08.220 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:08 smithi104 ceph-mon[36070]: Updating smithi033:/etc/ceph/ceph.conf 2024-03-18T15:08:08.220 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:08 smithi104 ceph-mon[36070]: Updating smithi033:/var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T15:08:08.221 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:08 smithi104 ceph-mon[36070]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:08.221 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:08 smithi104 ceph-mon[36070]: Updating smithi033:/etc/ceph/ceph.client.admin.keyring 2024-03-18T15:08:08.221 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:08 smithi104 ceph-mon[36070]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi104", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:08:08.221 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:08 smithi104 ceph-mon[36070]: Updating smithi033:/var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-18T15:08:08.221 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:08 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:08.221 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:08 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:08.221 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:08 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:08.221 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:08 smithi104 ceph-mon[36070]: Deploying cephadm binary to smithi104 2024-03-18T15:08:08.221 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:08 smithi104 ceph-mon[36070]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:08.221 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:08 smithi104 ceph-mon[36070]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:08.221 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:08 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:08.221 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:08 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:08:08.221 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:08 smithi104 ceph-mon[36070]: Added host smithi104 2024-03-18T15:08:08.221 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:08 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:08.221 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:08 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:08.222 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:08 smithi104 ceph-mon[36070]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:08.222 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:08 smithi104 ceph-mon[36070]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:08.222 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:08 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:08.222 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:08 smithi104 ceph-mon[36070]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T15:08:08.222 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:08 smithi104 ceph-mon[36070]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:08.222 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:08 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:08.222 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:08 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:08.222 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:08 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:08.222 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:08 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:08.222 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:08 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi104", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:08:08.222 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:08 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:08:08.222 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:08 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:08:08.222 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:08 smithi104 ceph-mon[36070]: Updating smithi104:/etc/ceph/ceph.conf 2024-03-18T15:08:08.223 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:08 smithi104 ceph-mon[36070]: Updating smithi104:/var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T15:08:08.223 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:08 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/256177249' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-18T15:08:08.223 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:08 smithi104 ceph-mon[36070]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:08.223 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:08 smithi104 ceph-mon[36070]: Updating smithi104:/etc/ceph/ceph.client.admin.keyring 2024-03-18T15:08:08.223 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:08 smithi104 ceph-mon[36070]: Updating smithi104:/var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-18T15:08:08.223 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:08 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/256177249' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-18T15:08:08.223 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:08 smithi104 ceph-mon[36070]: osdmap e4: 0 total, 0 up, 0 in 2024-03-18T15:08:08.223 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:08 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:08.223 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:08 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:08.223 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:08 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:08.223 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:08 smithi104 ceph-mon[36070]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:08.223 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:08 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:08.223 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:08 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:08:08.223 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:08 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:08:08.223 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:08 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:08:08.223 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:08 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:08.224 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:08 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T15:08:08.224 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:08 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:08:08.224 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:08 smithi104 ceph-mon[36070]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi019:172.21.15.19=a;smithi033:172.21.15.33=b;smithi104:172.21.15.104=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:08:08.224 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:08 smithi104 ceph-mon[36070]: Saving service mon spec with placement smithi019:172.21.15.19=a;smithi033:172.21.15.33=b;smithi104:172.21.15.104=c;count:3 2024-03-18T15:08:08.224 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:08 smithi104 ceph-mon[36070]: Deploying daemon mon.c on smithi104 2024-03-18T15:08:08.224 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:08 smithi104 ceph-mon[36070]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:08.224 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:08 smithi104 ceph-mon[36070]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:08.224 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:08 smithi104 ceph-mon[36070]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-18T15:08:08.224 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:08 smithi104 bash[36012]: b6b4027b63821f2b12853abc725550da8b7f89ecf8e4ce4543d33e802404ed2f 2024-03-18T15:08:08.224 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:08 smithi104 systemd[1]: Started Ceph mon.c for d4dc2ba6-e538-11ee-95c9-87774f69a715. 2024-03-18T15:08:09.658 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-18T15:08:09.658 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-18T15:08:09.901 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T15:08:13.373 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:13 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T15:08:13.373 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:13 smithi019 ceph-mon[31522]: mon.a calling monitor election 2024-03-18T15:08:13.373 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:13 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T15:08:13.373 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:13 smithi019 ceph-mon[31522]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:13.373 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:13 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T15:08:13.373 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:13 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T15:08:13.373 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:13 smithi019 ceph-mon[31522]: mon.c calling monitor election 2024-03-18T15:08:13.374 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:13 smithi019 ceph-mon[31522]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:13.374 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:13 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T15:08:13.374 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:13 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T15:08:13.374 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:13 smithi019 ceph-mon[31522]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:13.374 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:13 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T15:08:13.374 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:13 smithi019 ceph-mon[31522]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-18T15:08:13.374 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:13 smithi019 ceph-mon[31522]: monmap epoch 2 2024-03-18T15:08:13.374 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:13 smithi019 ceph-mon[31522]: fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 2024-03-18T15:08:13.374 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:13 smithi019 ceph-mon[31522]: last_changed 2024-03-18T15:08:08.010552+0000 2024-03-18T15:08:13.374 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:13 smithi019 ceph-mon[31522]: created 2024-03-18T15:05:53.426642+0000 2024-03-18T15:08:13.374 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:13 smithi019 ceph-mon[31522]: min_mon_release 19 (squid) 2024-03-18T15:08:13.374 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:13 smithi019 ceph-mon[31522]: election_strategy: 1 2024-03-18T15:08:13.374 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:13 smithi019 ceph-mon[31522]: 0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-03-18T15:08:13.375 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:13 smithi019 ceph-mon[31522]: 1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.c 2024-03-18T15:08:13.375 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:13 smithi019 ceph-mon[31522]: fsmap 2024-03-18T15:08:13.375 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:13 smithi019 ceph-mon[31522]: osdmap e4: 0 total, 0 up, 0 in 2024-03-18T15:08:13.375 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:13 smithi019 ceph-mon[31522]: mgrmap e13: x(active, since 58s) 2024-03-18T15:08:13.375 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:13 smithi019 ceph-mon[31522]: overall HEALTH_OK 2024-03-18T15:08:13.375 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:13 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:13.375 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:13 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:13.375 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:13 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:13.375 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:13 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T15:08:13.375 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:13 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:08:13.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:13 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T15:08:13.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:13 smithi104 ceph-mon[36070]: mon.a calling monitor election 2024-03-18T15:08:13.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:13 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T15:08:13.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:13 smithi104 ceph-mon[36070]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:13.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:13 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T15:08:13.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:13 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T15:08:13.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:13 smithi104 ceph-mon[36070]: mon.c calling monitor election 2024-03-18T15:08:13.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:13 smithi104 ceph-mon[36070]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:13.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:13 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T15:08:13.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:13 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T15:08:13.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:13 smithi104 ceph-mon[36070]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:13.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:13 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T15:08:13.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:13 smithi104 ceph-mon[36070]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-18T15:08:13.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:13 smithi104 ceph-mon[36070]: monmap epoch 2 2024-03-18T15:08:13.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:13 smithi104 ceph-mon[36070]: fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 2024-03-18T15:08:13.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:13 smithi104 ceph-mon[36070]: last_changed 2024-03-18T15:08:08.010552+0000 2024-03-18T15:08:13.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:13 smithi104 ceph-mon[36070]: created 2024-03-18T15:05:53.426642+0000 2024-03-18T15:08:13.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:13 smithi104 ceph-mon[36070]: min_mon_release 19 (squid) 2024-03-18T15:08:13.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:13 smithi104 ceph-mon[36070]: election_strategy: 1 2024-03-18T15:08:13.446 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:13 smithi104 ceph-mon[36070]: 0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-03-18T15:08:13.446 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:13 smithi104 ceph-mon[36070]: 1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.c 2024-03-18T15:08:13.446 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:13 smithi104 ceph-mon[36070]: fsmap 2024-03-18T15:08:13.446 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:13 smithi104 ceph-mon[36070]: osdmap e4: 0 total, 0 up, 0 in 2024-03-18T15:08:13.446 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:13 smithi104 ceph-mon[36070]: mgrmap e13: x(active, since 58s) 2024-03-18T15:08:13.446 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:13 smithi104 ceph-mon[36070]: overall HEALTH_OK 2024-03-18T15:08:13.446 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:13 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:13.446 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:13 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:13.446 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:13 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:13.446 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:13 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T15:08:13.446 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:13 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:08:14.324 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:14.323+0000 7f7be0a71700 1 -- 172.21.15.104:0/3824009298 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7bc80035c0 msgr2=0x7f7bc8005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:08:14.325 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:14.323+0000 7f7be0a71700 1 --2- 172.21.15.104:0/3824009298 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7bc80035c0 0x7f7bc8005a50 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f7bdc100be0 tx=0x7f7bcc00feb0 comp rx=0 tx=0).stop 2024-03-18T15:08:14.325 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:14.323+0000 7f7be0a71700 1 -- 172.21.15.104:0/3824009298 shutdown_connections 2024-03-18T15:08:14.325 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:14.323+0000 7f7be0a71700 1 --2- 172.21.15.104:0/3824009298 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7bc80035c0 0x7f7bc8005a50 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:08:14.325 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:14.323+0000 7f7be0a71700 1 -- 172.21.15.104:0/3824009298 >> 172.21.15.104:0/3824009298 conn(0x7f7bdc0fc7c0 msgr2=0x7f7bdc0febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:08:14.325 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:14.323+0000 7f7be0a71700 1 -- 172.21.15.104:0/3824009298 shutdown_connections 2024-03-18T15:08:14.325 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:14.323+0000 7f7be0a71700 1 -- 172.21.15.104:0/3824009298 wait complete. 2024-03-18T15:08:14.326 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:14.324+0000 7f7be0a71700 1 Processor -- start 2024-03-18T15:08:14.326 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:14.325+0000 7f7be0a71700 1 -- start start 2024-03-18T15:08:14.326 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:14.325+0000 7f7be0a71700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7bc80035c0 0x7f7bdc196610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:08:14.326 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:14.325+0000 7f7be0a71700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7bdc196b50 0x7f7bdc196fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:08:14.326 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:14.325+0000 7f7be0a71700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f7bdc108d40 con 0x7f7bc80035c0 2024-03-18T15:08:14.326 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:14.325+0000 7f7be0a71700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f7bdc108bc0 con 0x7f7bdc196b50 2024-03-18T15:08:14.327 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:14.325+0000 7f7bd1d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7bdc196b50 0x7f7bdc196fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:08:14.327 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:14.325+0000 7f7bd1d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7bdc196b50 0x7f7bdc196fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36820/0 (socket says 172.21.15.104:36820) 2024-03-18T15:08:14.327 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:14.325+0000 7f7bd1d9b700 1 -- 172.21.15.104:0/865125760 learned_addr learned my addr 172.21.15.104:0/865125760 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-18T15:08:14.327 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:14.325+0000 7f7bd1d9b700 1 -- 172.21.15.104:0/865125760 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7bdc196b50 msgr2=0x7f7bdc196fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-18T15:08:14.327 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:14.325+0000 7f7bd1d9b700 1 -- 172.21.15.104:0/865125760 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7bdc196b50 msgr2=0x7f7bdc196fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-18T15:08:14.327 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:14.325+0000 7f7bd1d9b700 1 --2- 172.21.15.104:0/865125760 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7bdc196b50 0x7f7bdc196fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-18T15:08:14.327 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:14.325+0000 7f7bd1d9b700 1 --2- 172.21.15.104:0/865125760 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7bdc196b50 0x7f7bdc196fd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T15:08:14.327 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:14.325+0000 7f7bda59c700 1 --2- 172.21.15.104:0/865125760 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7bc80035c0 0x7f7bdc196610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:08:14.328 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:14.326+0000 7f7bda59c700 1 -- 172.21.15.104:0/865125760 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7bdc196b50 msgr2=0x7f7bdc196fd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T15:08:14.328 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:14.326+0000 7f7bda59c700 1 --2- 172.21.15.104:0/865125760 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7bdc196b50 0x7f7bdc196fd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:08:14.328 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:14.326+0000 7f7bda59c700 1 -- 172.21.15.104:0/865125760 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bdc06b260 con 0x7f7bc80035c0 2024-03-18T15:08:14.328 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:14.327+0000 7f7bda59c700 1 --2- 172.21.15.104:0/865125760 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7bc80035c0 0x7f7bdc196610 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7bcc010010 tx=0x7f7bcc0055a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:08:14.328 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:14.327+0000 7f7bd3fff700 1 -- 172.21.15.104:0/865125760 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f7bcc004b20 con 0x7f7bc80035c0 2024-03-18T15:08:14.328 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:14.327+0000 7f7be0a71700 1 -- 172.21.15.104:0/865125760 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7bdc1a3c80 con 0x7f7bc80035c0 2024-03-18T15:08:14.328 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:14.327+0000 7f7be0a71700 1 -- 172.21.15.104:0/865125760 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7bdc1a40e0 con 0x7f7bc80035c0 2024-03-18T15:08:14.328 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:14.327+0000 7f7bd3fff700 1 -- 172.21.15.104:0/865125760 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7bcc013e40 con 0x7f7bc80035c0 2024-03-18T15:08:14.328 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:14.327+0000 7f7bd3fff700 1 -- 172.21.15.104:0/865125760 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f7bcc011980 con 0x7f7bc80035c0 2024-03-18T15:08:14.329 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:14.328+0000 7f7be0a71700 1 -- 172.21.15.104:0/865125760 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7bc8002140 con 0x7f7bc80035c0 2024-03-18T15:08:14.329 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:14.328+0000 7f7bd3fff700 1 -- 172.21.15.104:0/865125760 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f7bcc023070 con 0x7f7bc80035c0 2024-03-18T15:08:14.329 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:14.329+0000 7f7bd3fff700 1 --2- 172.21.15.104:0/865125760 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f7bbc038d60 0x7f7bbc03b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:08:14.329 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:14.329+0000 7f7bd3fff700 1 -- 172.21.15.104:0/865125760 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f7bcc020050 con 0x7f7bc80035c0 2024-03-18T15:08:14.330 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:14.329+0000 7f7bd1d9b700 1 --2- 172.21.15.104:0/865125760 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f7bbc038d60 0x7f7bbc03b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:08:14.331 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:14.330+0000 7f7bd1d9b700 1 --2- 172.21.15.104:0/865125760 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f7bbc038d60 0x7f7bbc03b220 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f7bc4009180 tx=0x7f7bc4008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:08:14.344 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:14.343+0000 7f7bd3fff700 1 -- 172.21.15.104:0/865125760 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7bcc028020 con 0x7f7bc80035c0 2024-03-18T15:08:14.397 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:14 smithi019 ceph-mon[31522]: Deploying daemon mon.b on smithi033 2024-03-18T15:08:14.398 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:14 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T15:08:14.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:14 smithi104 ceph-mon[36070]: Deploying daemon mon.b on smithi033 2024-03-18T15:08:14.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:14 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T15:08:14.621 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:14.621+0000 7f7be0a71700 1 -- 172.21.15.104:0/865125760 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f7bc8002c30 con 0x7f7bc80035c0 2024-03-18T15:08:14.622 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:14.622+0000 7f7bd3fff700 1 -- 172.21.15.104:0/865125760 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1015 (secure 0 0 0) 0x7f7bcc00b2c0 con 0x7f7bc80035c0 2024-03-18T15:08:14.623 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-18T15:08:14.623 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":2,"fsid":"d4dc2ba6-e538-11ee-95c9-87774f69a715","modified":"2024-03-18T15:08:08.010552Z","created":"2024-03-18T15:05:53.426642Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-18T15:08:14.625 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:14.624+0000 7f7be0a71700 1 -- 172.21.15.104:0/865125760 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f7bbc038d60 msgr2=0x7f7bbc03b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:08:14.625 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:14.624+0000 7f7be0a71700 1 --2- 172.21.15.104:0/865125760 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f7bbc038d60 0x7f7bbc03b220 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f7bc4009180 tx=0x7f7bc4008040 comp rx=0 tx=0).stop 2024-03-18T15:08:14.625 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:14.624+0000 7f7be0a71700 1 -- 172.21.15.104:0/865125760 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7bc80035c0 msgr2=0x7f7bdc196610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:08:14.625 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:14.624+0000 7f7be0a71700 1 --2- 172.21.15.104:0/865125760 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7bc80035c0 0x7f7bdc196610 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7bcc010010 tx=0x7f7bcc0055a0 comp rx=0 tx=0).stop 2024-03-18T15:08:14.625 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:14.624+0000 7f7be0a71700 1 -- 172.21.15.104:0/865125760 shutdown_connections 2024-03-18T15:08:14.625 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:14.624+0000 7f7be0a71700 1 --2- 172.21.15.104:0/865125760 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f7bbc038d60 0x7f7bbc03b220 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:08:14.625 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:14.624+0000 7f7be0a71700 1 --2- 172.21.15.104:0/865125760 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7bc80035c0 0x7f7bdc196610 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:08:14.625 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:14.624+0000 7f7be0a71700 1 --2- 172.21.15.104:0/865125760 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7bdc196b50 0x7f7bdc196fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:08:14.625 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:14.624+0000 7f7be0a71700 1 -- 172.21.15.104:0/865125760 >> 172.21.15.104:0/865125760 conn(0x7f7bdc0fc7c0 msgr2=0x7f7bdc0fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:08:14.625 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:14.624+0000 7f7be0a71700 1 -- 172.21.15.104:0/865125760 shutdown_connections 2024-03-18T15:08:14.625 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:14.624+0000 7f7be0a71700 1 -- 172.21.15.104:0/865125760 wait complete. 2024-03-18T15:08:14.626 INFO:teuthology.orchestra.run.smithi104.stderr:dumped monmap epoch 2 2024-03-18T15:08:15.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:15 smithi104 ceph-mon[36070]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:15.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:15 smithi104 ceph-mon[36070]: from='client.? 172.21.15.104:0/865125760' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-18T15:08:15.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:15 smithi019 ceph-mon[31522]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:15.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:15 smithi019 ceph-mon[31522]: from='client.? 172.21.15.104:0/865125760' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-18T15:08:16.240 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-18T15:08:16.240 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-18T15:08:16.481 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T15:08:16.917 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 podman[36628]: 2024-03-18 15:08:16.893076108 +0000 UTC m=+0.338303574 container start 5d7d00c66554d26cf7bbeb62dc6eedbbb10125147a831a90d65f4722c6488094 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mon-b, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-18T15:08:17.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: set uid:gid to 167:167 (ceph:ceph) 2024-03-18T15:08:17.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev), process ceph-mon, pid 2 2024-03-18T15:08:17.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: pidfile_write: ignore empty --pid-file 2024-03-18T15:08:17.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: load: jerasure load: lrc 2024-03-18T15:08:17.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: RocksDB version: 7.9.2 2024-03-18T15:08:17.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Git sha 0 2024-03-18T15:08:17.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Compile date 2024-03-15 00:33:34 2024-03-18T15:08:17.258 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: DB SUMMARY 2024-03-18T15:08:17.258 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: DB Session ID: JYSLA0MCSU5YBFN9JZEP 2024-03-18T15:08:17.258 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: CURRENT file: CURRENT 2024-03-18T15:08:17.258 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: IDENTITY file: IDENTITY 2024-03-18T15:08:17.258 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-18T15:08:17.258 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-03-18T15:08:17.258 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2024-03-18T15:08:17.258 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.error_if_exists: 0 2024-03-18T15:08:17.258 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.create_if_missing: 0 2024-03-18T15:08:17.258 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.paranoid_checks: 1 2024-03-18T15:08:17.258 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-18T15:08:17.258 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-18T15:08:17.258 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-18T15:08:17.259 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.env: 0x55865179ec60 2024-03-18T15:08:17.259 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.fs: PosixFileSystem 2024-03-18T15:08:17.259 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.info_log: 0x558652a81ac0 2024-03-18T15:08:17.259 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.max_file_opening_threads: 16 2024-03-18T15:08:17.259 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.statistics: (nil) 2024-03-18T15:08:17.259 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.use_fsync: 0 2024-03-18T15:08:17.259 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.max_log_file_size: 0 2024-03-18T15:08:17.259 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-18T15:08:17.259 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-18T15:08:17.259 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.keep_log_file_num: 1000 2024-03-18T15:08:17.259 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.recycle_log_file_num: 0 2024-03-18T15:08:17.259 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.allow_fallocate: 1 2024-03-18T15:08:17.259 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.allow_mmap_reads: 0 2024-03-18T15:08:17.260 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.allow_mmap_writes: 0 2024-03-18T15:08:17.260 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.use_direct_reads: 0 2024-03-18T15:08:17.260 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-18T15:08:17.260 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.create_missing_column_families: 0 2024-03-18T15:08:17.260 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.db_log_dir: 2024-03-18T15:08:17.260 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.wal_dir: 2024-03-18T15:08:17.260 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-18T15:08:17.262 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-18T15:08:17.262 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-18T15:08:17.262 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-18T15:08:17.262 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-18T15:08:17.262 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-18T15:08:17.262 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.advise_random_on_open: 1 2024-03-18T15:08:17.262 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.db_write_buffer_size: 0 2024-03-18T15:08:17.263 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.write_buffer_manager: 0x558652a541e0 2024-03-18T15:08:17.263 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-18T15:08:17.263 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-18T15:08:17.263 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-18T15:08:17.263 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.rate_limiter: (nil) 2024-03-18T15:08:17.263 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-18T15:08:17.263 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.wal_recovery_mode: 2 2024-03-18T15:08:17.263 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.enable_thread_tracking: 0 2024-03-18T15:08:17.263 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.enable_pipelined_write: 0 2024-03-18T15:08:17.263 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.unordered_write: 0 2024-03-18T15:08:17.263 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-18T15:08:17.263 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-18T15:08:17.263 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-18T15:08:17.264 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-18T15:08:17.264 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.row_cache: None 2024-03-18T15:08:17.264 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.wal_filter: None 2024-03-18T15:08:17.264 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-18T15:08:17.264 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.allow_ingest_behind: 0 2024-03-18T15:08:17.264 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.two_write_queues: 0 2024-03-18T15:08:17.264 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.manual_wal_flush: 0 2024-03-18T15:08:17.264 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.wal_compression: 0 2024-03-18T15:08:17.264 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.atomic_flush: 0 2024-03-18T15:08:17.264 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-18T15:08:17.264 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-18T15:08:17.264 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-18T15:08:17.264 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.log_readahead_size: 0 2024-03-18T15:08:17.264 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-18T15:08:17.265 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.best_efforts_recovery: 0 2024-03-18T15:08:17.265 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-18T15:08:17.265 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-18T15:08:17.265 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.allow_data_in_errors: 0 2024-03-18T15:08:17.265 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.db_host_id: __hostname__ 2024-03-18T15:08:17.265 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-18T15:08:17.265 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.max_background_jobs: 2 2024-03-18T15:08:17.265 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.max_background_compactions: -1 2024-03-18T15:08:17.265 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.max_subcompactions: 1 2024-03-18T15:08:17.265 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-18T15:08:17.265 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-18T15:08:17.265 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-18T15:08:17.265 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.max_total_wal_size: 0 2024-03-18T15:08:17.265 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-18T15:08:17.266 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-18T15:08:17.266 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-18T15:08:17.266 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-18T15:08:17.266 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.max_open_files: -1 2024-03-18T15:08:17.266 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.bytes_per_sync: 0 2024-03-18T15:08:17.266 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-18T15:08:17.266 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-18T15:08:17.266 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.compaction_readahead_size: 0 2024-03-18T15:08:17.266 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.max_background_flushes: -1 2024-03-18T15:08:17.266 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Compression algorithms supported: 2024-03-18T15:08:17.266 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-18T15:08:17.266 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: kZSTD supported: 0 2024-03-18T15:08:17.266 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: kXpressCompression supported: 0 2024-03-18T15:08:17.266 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: kLZ4HCCompression supported: 1 2024-03-18T15:08:17.267 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: kZlibCompression supported: 1 2024-03-18T15:08:17.267 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: kSnappyCompression supported: 1 2024-03-18T15:08:17.267 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: kLZ4Compression supported: 1 2024-03-18T15:08:17.267 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: kBZip2Compression supported: 0 2024-03-18T15:08:17.267 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-18T15:08:17.267 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-18T15:08:17.267 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-03-18T15:08:17.267 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-18T15:08:17.267 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-18T15:08:17.267 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.merge_operator: 2024-03-18T15:08:17.267 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.compaction_filter: None 2024-03-18T15:08:17.267 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.compaction_filter_factory: None 2024-03-18T15:08:17.267 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.sst_partitioner_factory: None 2024-03-18T15:08:17.267 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-18T15:08:17.268 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-18T15:08:17.268 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x558652a81bc0) 2024-03-18T15:08:17.268 INFO:journalctl@ceph.mon.b.smithi033.stdout: cache_index_and_filter_blocks: 1 2024-03-18T15:08:17.268 INFO:journalctl@ceph.mon.b.smithi033.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-18T15:08:17.268 INFO:journalctl@ceph.mon.b.smithi033.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-18T15:08:17.268 INFO:journalctl@ceph.mon.b.smithi033.stdout: pin_top_level_index_and_filter: 1 2024-03-18T15:08:17.268 INFO:journalctl@ceph.mon.b.smithi033.stdout: index_type: 0 2024-03-18T15:08:17.268 INFO:journalctl@ceph.mon.b.smithi033.stdout: data_block_index_type: 0 2024-03-18T15:08:17.268 INFO:journalctl@ceph.mon.b.smithi033.stdout: index_shortening: 1 2024-03-18T15:08:17.268 INFO:journalctl@ceph.mon.b.smithi033.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-18T15:08:17.268 INFO:journalctl@ceph.mon.b.smithi033.stdout: checksum: 4 2024-03-18T15:08:17.268 INFO:journalctl@ceph.mon.b.smithi033.stdout: no_block_cache: 0 2024-03-18T15:08:17.269 INFO:journalctl@ceph.mon.b.smithi033.stdout: block_cache: 0x558652a87090 2024-03-18T15:08:17.269 INFO:journalctl@ceph.mon.b.smithi033.stdout: block_cache_name: BinnedLRUCache 2024-03-18T15:08:17.269 INFO:journalctl@ceph.mon.b.smithi033.stdout: block_cache_options: 2024-03-18T15:08:17.269 INFO:journalctl@ceph.mon.b.smithi033.stdout: capacity : 536870912 2024-03-18T15:08:17.269 INFO:journalctl@ceph.mon.b.smithi033.stdout: num_shard_bits : 4 2024-03-18T15:08:17.269 INFO:journalctl@ceph.mon.b.smithi033.stdout: strict_capacity_limit : 0 2024-03-18T15:08:17.269 INFO:journalctl@ceph.mon.b.smithi033.stdout: high_pri_pool_ratio: 0.000 2024-03-18T15:08:17.269 INFO:journalctl@ceph.mon.b.smithi033.stdout: block_cache_compressed: (nil) 2024-03-18T15:08:17.269 INFO:journalctl@ceph.mon.b.smithi033.stdout: persistent_cache: (nil) 2024-03-18T15:08:17.269 INFO:journalctl@ceph.mon.b.smithi033.stdout: block_size: 4096 2024-03-18T15:08:17.269 INFO:journalctl@ceph.mon.b.smithi033.stdout: block_size_deviation: 10 2024-03-18T15:08:17.269 INFO:journalctl@ceph.mon.b.smithi033.stdout: block_restart_interval: 16 2024-03-18T15:08:17.269 INFO:journalctl@ceph.mon.b.smithi033.stdout: index_block_restart_interval: 1 2024-03-18T15:08:17.269 INFO:journalctl@ceph.mon.b.smithi033.stdout: metadata_block_size: 4096 2024-03-18T15:08:17.270 INFO:journalctl@ceph.mon.b.smithi033.stdout: partition_filters: 0 2024-03-18T15:08:17.270 INFO:journalctl@ceph.mon.b.smithi033.stdout: use_delta_encoding: 1 2024-03-18T15:08:17.270 INFO:journalctl@ceph.mon.b.smithi033.stdout: filter_policy: bloomfilter 2024-03-18T15:08:17.270 INFO:journalctl@ceph.mon.b.smithi033.stdout: whole_key_filtering: 1 2024-03-18T15:08:17.270 INFO:journalctl@ceph.mon.b.smithi033.stdout: verify_compression: 0 2024-03-18T15:08:17.270 INFO:journalctl@ceph.mon.b.smithi033.stdout: read_amp_bytes_per_bit: 0 2024-03-18T15:08:17.270 INFO:journalctl@ceph.mon.b.smithi033.stdout: format_version: 5 2024-03-18T15:08:17.270 INFO:journalctl@ceph.mon.b.smithi033.stdout: enable_index_compression: 1 2024-03-18T15:08:17.270 INFO:journalctl@ceph.mon.b.smithi033.stdout: block_align: 0 2024-03-18T15:08:17.270 INFO:journalctl@ceph.mon.b.smithi033.stdout: max_auto_readahead_size: 262144 2024-03-18T15:08:17.270 INFO:journalctl@ceph.mon.b.smithi033.stdout: prepopulate_block_cache: 0 2024-03-18T15:08:17.270 INFO:journalctl@ceph.mon.b.smithi033.stdout: initial_auto_readahead_size: 8192 2024-03-18T15:08:17.270 INFO:journalctl@ceph.mon.b.smithi033.stdout: num_file_reads_for_auto_readahead: 2 2024-03-18T15:08:17.271 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.write_buffer_size: 33554432 2024-03-18T15:08:17.271 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.max_write_buffer_number: 2 2024-03-18T15:08:17.271 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.compression: NoCompression 2024-03-18T15:08:17.271 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.bottommost_compression: Disabled 2024-03-18T15:08:17.271 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.prefix_extractor: nullptr 2024-03-18T15:08:17.271 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-18T15:08:17.271 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.num_levels: 7 2024-03-18T15:08:17.271 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-18T15:08:17.271 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-18T15:08:17.271 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-18T15:08:17.271 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-18T15:08:17.271 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-18T15:08:17.271 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-18T15:08:17.272 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-18T15:08:17.272 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-18T15:08:17.272 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-18T15:08:17.272 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-18T15:08:17.272 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-18T15:08:17.272 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-18T15:08:17.272 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-18T15:08:17.272 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.compression_opts.level: 32767 2024-03-18T15:08:17.272 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.compression_opts.strategy: 0 2024-03-18T15:08:17.272 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-18T15:08:17.272 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-18T15:08:17.272 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-18T15:08:17.272 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-18T15:08:17.272 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.compression_opts.enabled: false 2024-03-18T15:08:17.273 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-18T15:08:17.273 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-18T15:08:17.273 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-18T15:08:17.273 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-18T15:08:17.273 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.target_file_size_base: 67108864 2024-03-18T15:08:17.273 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-18T15:08:17.273 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-18T15:08:17.273 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-18T15:08:17.273 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-18T15:08:17.273 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-18T15:08:17.273 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-18T15:08:17.273 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-18T15:08:17.273 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-18T15:08:17.273 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-18T15:08:17.274 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-18T15:08:17.274 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-18T15:08:17.274 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-18T15:08:17.274 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-18T15:08:17.274 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-18T15:08:17.274 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.arena_block_size: 1048576 2024-03-18T15:08:17.274 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-18T15:08:17.274 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-18T15:08:17.274 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.disable_auto_compactions: 0 2024-03-18T15:08:17.274 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-18T15:08:17.274 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-18T15:08:17.274 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-18T15:08:17.274 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-18T15:08:17.275 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-18T15:08:17.275 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-18T15:08:17.275 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-18T15:08:17.275 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-18T15:08:17.275 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-18T15:08:17.275 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-18T15:08:17.275 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.table_properties_collectors: 2024-03-18T15:08:17.275 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.inplace_update_support: 0 2024-03-18T15:08:17.275 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-18T15:08:17.275 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-18T15:08:17.275 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-18T15:08:17.275 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-18T15:08:17.275 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.bloom_locality: 0 2024-03-18T15:08:17.275 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.max_successive_merges: 0 2024-03-18T15:08:17.276 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-18T15:08:17.276 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.paranoid_file_checks: 0 2024-03-18T15:08:17.276 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.force_consistency_checks: 1 2024-03-18T15:08:17.276 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.report_bg_io_stats: 0 2024-03-18T15:08:17.276 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.ttl: 2592000 2024-03-18T15:08:17.276 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-18T15:08:17.276 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-18T15:08:17.276 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-18T15:08:17.276 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.enable_blob_files: false 2024-03-18T15:08:17.276 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.min_blob_size: 0 2024-03-18T15:08:17.276 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.blob_file_size: 268435456 2024-03-18T15:08:17.276 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-18T15:08:17.276 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-18T15:08:17.276 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-18T15:08:17.277 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-18T15:08:17.277 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-18T15:08:17.277 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.blob_file_starting_level: 0 2024-03-18T15:08:17.277 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-18T15:08:17.277 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-18T15:08:17.277 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-18T15:08:17.277 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: b59e6a98-c2a4-4913-84b3-2fdc62bf85de 2024-03-18T15:08:17.277 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710774496923384, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-18T15:08:17.277 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-18T15:08:17.277 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710774496923795, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710774496, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "b59e6a98-c2a4-4913-84b3-2fdc62bf85de", "db_session_id": "JYSLA0MCSU5YBFN9JZEP", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-18T15:08:17.277 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710774496923877, "job": 1, "event": "recovery_finished"} 2024-03-18T15:08:17.277 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-18T15:08:17.277 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-18T15:08:17.278 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x558652b7a000 2024-03-18T15:08:17.278 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: DB pointer 0x558652b64000 2024-03-18T15:08:17.278 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-18T15:08:17.278 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-18T15:08:17.278 INFO:journalctl@ceph.mon.b.smithi033.stdout: ** DB Stats ** 2024-03-18T15:08:17.278 INFO:journalctl@ceph.mon.b.smithi033.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-18T15:08:17.278 INFO:journalctl@ceph.mon.b.smithi033.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-18T15:08:17.278 INFO:journalctl@ceph.mon.b.smithi033.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-18T15:08:17.278 INFO:journalctl@ceph.mon.b.smithi033.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-18T15:08:17.278 INFO:journalctl@ceph.mon.b.smithi033.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-18T15:08:17.278 INFO:journalctl@ceph.mon.b.smithi033.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-18T15:08:17.278 INFO:journalctl@ceph.mon.b.smithi033.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-18T15:08:17.278 INFO:journalctl@ceph.mon.b.smithi033.stdout: 2024-03-18T15:08:17.278 INFO:journalctl@ceph.mon.b.smithi033.stdout: ** Compaction Stats [default] ** 2024-03-18T15:08:17.279 INFO:journalctl@ceph.mon.b.smithi033.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-18T15:08:17.279 INFO:journalctl@ceph.mon.b.smithi033.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-18T15:08:17.279 INFO:journalctl@ceph.mon.b.smithi033.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-18T15:08:17.279 INFO:journalctl@ceph.mon.b.smithi033.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-18T15:08:17.279 INFO:journalctl@ceph.mon.b.smithi033.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-18T15:08:17.279 INFO:journalctl@ceph.mon.b.smithi033.stdout: 2024-03-18T15:08:17.279 INFO:journalctl@ceph.mon.b.smithi033.stdout: ** Compaction Stats [default] ** 2024-03-18T15:08:17.279 INFO:journalctl@ceph.mon.b.smithi033.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-18T15:08:17.279 INFO:journalctl@ceph.mon.b.smithi033.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-18T15:08:17.279 INFO:journalctl@ceph.mon.b.smithi033.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-18T15:08:17.279 INFO:journalctl@ceph.mon.b.smithi033.stdout: 2024-03-18T15:08:17.279 INFO:journalctl@ceph.mon.b.smithi033.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-18T15:08:17.279 INFO:journalctl@ceph.mon.b.smithi033.stdout: 2024-03-18T15:08:17.280 INFO:journalctl@ceph.mon.b.smithi033.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-18T15:08:17.280 INFO:journalctl@ceph.mon.b.smithi033.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-18T15:08:17.280 INFO:journalctl@ceph.mon.b.smithi033.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-18T15:08:17.280 INFO:journalctl@ceph.mon.b.smithi033.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-18T15:08:17.280 INFO:journalctl@ceph.mon.b.smithi033.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-18T15:08:17.280 INFO:journalctl@ceph.mon.b.smithi033.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-18T15:08:17.280 INFO:journalctl@ceph.mon.b.smithi033.stdout: Cumulative compaction: 0.00 GB write, 0.67 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-18T15:08:17.280 INFO:journalctl@ceph.mon.b.smithi033.stdout: Interval compaction: 0.00 GB write, 0.67 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-18T15:08:17.280 INFO:journalctl@ceph.mon.b.smithi033.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-18T15:08:17.280 INFO:journalctl@ceph.mon.b.smithi033.stdout: Block cache BinnedLRUCache@0x558652a87090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-18T15:08:17.280 INFO:journalctl@ceph.mon.b.smithi033.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-18T15:08:17.280 INFO:journalctl@ceph.mon.b.smithi033.stdout: 2024-03-18T15:08:17.280 INFO:journalctl@ceph.mon.b.smithi033.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-18T15:08:17.280 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-03-18T15:08:17.281 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: using public_addr v2:172.21.15.33:0/0 -> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] 2024-03-18T15:08:17.281 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: starting mon.b rank -1 at public addrs [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] at bind addrs [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 2024-03-18T15:08:17.281 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: mon.b@-1(???) e0 preinit fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 2024-03-18T15:08:17.281 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: mon.b@-1(synchronizing).mds e1 new map 2024-03-18T15:08:17.281 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: mon.b@-1(synchronizing).mds e1 print_map 2024-03-18T15:08:17.281 INFO:journalctl@ceph.mon.b.smithi033.stdout: e1 2024-03-18T15:08:17.281 INFO:journalctl@ceph.mon.b.smithi033.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-18T15:08:17.281 INFO:journalctl@ceph.mon.b.smithi033.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-18T15:08:17.281 INFO:journalctl@ceph.mon.b.smithi033.stdout: legacy client fscid: -1 2024-03-18T15:08:17.281 INFO:journalctl@ceph.mon.b.smithi033.stdout: 2024-03-18T15:08:17.281 INFO:journalctl@ceph.mon.b.smithi033.stdout: No filesystems configured 2024-03-18T15:08:17.281 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-18T15:08:17.281 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-18T15:08:17.282 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-18T15:08:17.282 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-18T15:08:17.282 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-18T15:08:17.282 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-18T15:08:17.282 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-18T15:08:17.282 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-18T15:08:17.282 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-18T15:08:17.282 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-18T15:08:17.282 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: Deploying cephadm binary to smithi033 2024-03-18T15:08:17.282 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:17.282 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:17.282 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:17.282 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: Added host smithi033 2024-03-18T15:08:17.283 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:08:17.283 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:17.283 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:17.283 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:17.283 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:17.283 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:17.283 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T15:08:17.283 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:17.283 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:17.283 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:17.283 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:17.283 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:17.283 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi033", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:08:17.284 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:08:17.284 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:08:17.284 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: Updating smithi033:/etc/ceph/ceph.conf 2024-03-18T15:08:17.284 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: Updating smithi033:/var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T15:08:17.284 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:17.284 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: Updating smithi033:/etc/ceph/ceph.client.admin.keyring 2024-03-18T15:08:17.284 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi104", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:08:17.284 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: Updating smithi033:/var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-18T15:08:17.284 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:17.284 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:17.284 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:17.284 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: Deploying cephadm binary to smithi104 2024-03-18T15:08:17.284 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:17.285 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:17.285 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:17.285 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:08:17.285 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: Added host smithi104 2024-03-18T15:08:17.285 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:17.285 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:17.285 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:17.285 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:17.285 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:17.285 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T15:08:17.285 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:17.285 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:17.285 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:17.285 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:17.286 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:17.286 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi104", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:08:17.286 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:08:17.286 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:08:17.286 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: Updating smithi104:/etc/ceph/ceph.conf 2024-03-18T15:08:17.286 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: Updating smithi104:/var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T15:08:17.286 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/256177249' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-18T15:08:17.286 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:17.286 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: Updating smithi104:/etc/ceph/ceph.client.admin.keyring 2024-03-18T15:08:17.286 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: Updating smithi104:/var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-18T15:08:17.286 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/256177249' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-18T15:08:17.286 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: osdmap e4: 0 total, 0 up, 0 in 2024-03-18T15:08:17.286 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:17.287 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:17.287 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:17.287 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:17.287 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:17.287 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:08:17.287 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:08:17.287 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:08:17.287 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:17.287 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T15:08:17.287 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:08:17.287 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi019:172.21.15.19=a;smithi033:172.21.15.33=b;smithi104:172.21.15.104=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:08:17.287 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: Saving service mon spec with placement smithi019:172.21.15.19=a;smithi033:172.21.15.33=b;smithi104:172.21.15.104=c;count:3 2024-03-18T15:08:17.287 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: Deploying daemon mon.c on smithi104 2024-03-18T15:08:17.287 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:17.288 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:17.288 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T15:08:17.288 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: mon.a calling monitor election 2024-03-18T15:08:17.288 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T15:08:17.288 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:17.288 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T15:08:17.288 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T15:08:17.288 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: mon.c calling monitor election 2024-03-18T15:08:17.288 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:17.288 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T15:08:17.288 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T15:08:17.288 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:17.288 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T15:08:17.288 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-18T15:08:17.288 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: monmap epoch 2 2024-03-18T15:08:17.289 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 2024-03-18T15:08:17.289 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: last_changed 2024-03-18T15:08:08.010552+0000 2024-03-18T15:08:17.289 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: created 2024-03-18T15:05:53.426642+0000 2024-03-18T15:08:17.289 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: min_mon_release 19 (squid) 2024-03-18T15:08:17.289 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: election_strategy: 1 2024-03-18T15:08:17.289 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: 0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-03-18T15:08:17.289 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: 1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.c 2024-03-18T15:08:17.289 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: fsmap 2024-03-18T15:08:17.289 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: osdmap e4: 0 total, 0 up, 0 in 2024-03-18T15:08:17.289 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: mgrmap e13: x(active, since 58s) 2024-03-18T15:08:17.289 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: overall HEALTH_OK 2024-03-18T15:08:17.289 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:17.289 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:17.289 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:17.289 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T15:08:17.290 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:08:17.290 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: Deploying daemon mon.b on smithi033 2024-03-18T15:08:17.290 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T15:08:17.290 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:17.290 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: from='client.? 172.21.15.104:0/865125760' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-18T15:08:17.290 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 ceph-mon[36648]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-18T15:08:17.290 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 bash[36628]: 5d7d00c66554d26cf7bbeb62dc6eedbbb10125147a831a90d65f4722c6488094 2024-03-18T15:08:17.290 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:16 smithi033 systemd[1]: Started Ceph mon.b for d4dc2ba6-e538-11ee-95c9-87774f69a715. 2024-03-18T15:08:22.342 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:22.341+0000 7f62fbb52700 1 -- 172.21.15.104:0/2626076013 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f62e00035c0 msgr2=0x7f62e0005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:08:22.342 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:22.341+0000 7f62fbb52700 1 --2- 172.21.15.104:0/2626076013 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f62e00035c0 0x7f62e0005a50 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f62f406a530 tx=0x7f62f000fe60 comp rx=0 tx=0).stop 2024-03-18T15:08:22.342 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:22.341+0000 7f62fbb52700 1 -- 172.21.15.104:0/2626076013 shutdown_connections 2024-03-18T15:08:22.342 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:22.341+0000 7f62fbb52700 1 --2- 172.21.15.104:0/2626076013 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f62e00035c0 0x7f62e0005a50 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:08:22.342 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:22.341+0000 7f62fbb52700 1 -- 172.21.15.104:0/2626076013 >> 172.21.15.104:0/2626076013 conn(0x7f62f40fc7e0 msgr2=0x7f62f40fec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:08:22.343 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:22.341+0000 7f62fbb52700 1 -- 172.21.15.104:0/2626076013 shutdown_connections 2024-03-18T15:08:22.343 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:22.341+0000 7f62fbb52700 1 -- 172.21.15.104:0/2626076013 wait complete. 2024-03-18T15:08:22.343 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:22 smithi104 ceph-mon[36070]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:22.343 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:22 smithi104 ceph-mon[36070]: mon.c calling monitor election 2024-03-18T15:08:22.343 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:22 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T15:08:22.343 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:22 smithi104 ceph-mon[36070]: mon.a calling monitor election 2024-03-18T15:08:22.343 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:22 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T15:08:22.343 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:22 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T15:08:22.343 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:22 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T15:08:22.343 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:22 smithi104 ceph-mon[36070]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:22.344 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:22 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T15:08:22.344 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:22 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T15:08:22.344 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:22 smithi104 ceph-mon[36070]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:22.344 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:22 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T15:08:22.344 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:22 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T15:08:22.344 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:22 smithi104 ceph-mon[36070]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-18T15:08:22.344 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:22 smithi104 ceph-mon[36070]: monmap epoch 3 2024-03-18T15:08:22.344 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:22 smithi104 ceph-mon[36070]: fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 2024-03-18T15:08:22.344 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:22 smithi104 ceph-mon[36070]: last_changed 2024-03-18T15:08:16.947306+0000 2024-03-18T15:08:22.344 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:22 smithi104 ceph-mon[36070]: created 2024-03-18T15:05:53.426642+0000 2024-03-18T15:08:22.344 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:22 smithi104 ceph-mon[36070]: min_mon_release 19 (squid) 2024-03-18T15:08:22.344 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:22 smithi104 ceph-mon[36070]: election_strategy: 1 2024-03-18T15:08:22.344 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:22 smithi104 ceph-mon[36070]: 0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-03-18T15:08:22.344 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:22 smithi104 ceph-mon[36070]: 1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.c 2024-03-18T15:08:22.345 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:22 smithi104 ceph-mon[36070]: 2: [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] mon.b 2024-03-18T15:08:22.345 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:22 smithi104 ceph-mon[36070]: fsmap 2024-03-18T15:08:22.345 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:22 smithi104 ceph-mon[36070]: osdmap e4: 0 total, 0 up, 0 in 2024-03-18T15:08:22.345 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:22 smithi104 ceph-mon[36070]: mgrmap e13: x(active, since 67s) 2024-03-18T15:08:22.345 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:22 smithi104 ceph-mon[36070]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-18T15:08:22.345 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:22 smithi104 ceph-mon[36070]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-18T15:08:22.345 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:22 smithi104 ceph-mon[36070]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-18T15:08:22.345 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:22 smithi104 ceph-mon[36070]: mon.b (rank 2) addr [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] is down (out of quorum) 2024-03-18T15:08:22.345 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:22 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:22.345 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:22 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:22.348 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:22.342+0000 7f62fbb52700 1 Processor -- start 2024-03-18T15:08:22.348 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:22.342+0000 7f62fbb52700 1 -- start start 2024-03-18T15:08:22.348 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:22.342+0000 7f62fbb52700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f62e00035c0 0x7f62f4100c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:08:22.348 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:22.342+0000 7f62fbb52700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f62f4101190 0x7f62f4101610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:08:22.348 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:22.342+0000 7f62fbb52700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62f41a3c80 0x7f62f41a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:08:22.348 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:22.342+0000 7f62fbb52700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f62f4109e10 con 0x7f62e00035c0 2024-03-18T15:08:22.348 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:22.342+0000 7f62fbb52700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f62f4109c90 con 0x7f62f4101190 2024-03-18T15:08:22.348 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:22.342+0000 7f62fbb52700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f62f4109f90 con 0x7f62f41a3c80 2024-03-18T15:08:22.348 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:22.343+0000 7f62fa0ef700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62f41a3c80 0x7f62f41a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:08:22.348 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:22.343+0000 7f62fa0ef700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62f41a3c80 0x7f62f41a6070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54326/0 (socket says 172.21.15.104:54326) 2024-03-18T15:08:22.348 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:22.343+0000 7f62fa0ef700 1 -- 172.21.15.104:0/3531960556 learned_addr learned my addr 172.21.15.104:0/3531960556 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-18T15:08:22.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:22.343+0000 7f62fa0ef700 1 -- 172.21.15.104:0/3531960556 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62f41a3c80 msgr2=0x7f62f41a6070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-03-18T15:08:22.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:22.343+0000 7f62fa0ef700 1 -- 172.21.15.104:0/3531960556 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62f41a3c80 msgr2=0x7f62f41a6070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-18T15:08:22.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:22.343+0000 7f62fa0ef700 1 --2- 172.21.15.104:0/3531960556 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62f41a3c80 0x7f62f41a6070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-18T15:08:22.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:22.343+0000 7f62f90ed700 1 --2- 172.21.15.104:0/3531960556 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f62f4101190 0x7f62f4101610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:08:22.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:22.343+0000 7f62fa0ef700 1 --2- 172.21.15.104:0/3531960556 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62f41a3c80 0x7f62f41a6070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T15:08:22.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:22.343+0000 7f62f98ee700 1 --2- 172.21.15.104:0/3531960556 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f62e00035c0 0x7f62f4100c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:08:22.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:22.343+0000 7f62f90ed700 1 -- 172.21.15.104:0/3531960556 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f62f4101190 msgr2=0x7f62f4101610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-18T15:08:22.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:22.343+0000 7f62f90ed700 1 -- 172.21.15.104:0/3531960556 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f62f4101190 msgr2=0x7f62f4101610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-18T15:08:22.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:22.343+0000 7f62f90ed700 1 --2- 172.21.15.104:0/3531960556 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f62f4101190 0x7f62f4101610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-18T15:08:22.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:22.343+0000 7f62f90ed700 1 --2- 172.21.15.104:0/3531960556 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f62f4101190 0x7f62f4101610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T15:08:22.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:22.343+0000 7f62f98ee700 1 -- 172.21.15.104:0/3531960556 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62f41a3c80 msgr2=0x7f62f41a6070 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T15:08:22.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:22.343+0000 7f62f98ee700 1 --2- 172.21.15.104:0/3531960556 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62f41a3c80 0x7f62f41a6070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:08:22.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:22.343+0000 7f62f98ee700 1 -- 172.21.15.104:0/3531960556 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f62f4101190 msgr2=0x7f62f4101610 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T15:08:22.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:22.343+0000 7f62f98ee700 1 --2- 172.21.15.104:0/3531960556 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f62f4101190 0x7f62f4101610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:08:22.350 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:22.344+0000 7f62f98ee700 1 -- 172.21.15.104:0/3531960556 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62f41a67a0 con 0x7f62e00035c0 2024-03-18T15:08:22.350 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:22.344+0000 7f62f98ee700 1 --2- 172.21.15.104:0/3531960556 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f62e00035c0 0x7f62f4100c50 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f62f406a530 tx=0x7f62f0005720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:08:22.350 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:22.344+0000 7f62eaffd700 1 -- 172.21.15.104:0/3531960556 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62f002b020 con 0x7f62e00035c0 2024-03-18T15:08:22.350 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:22.344+0000 7f62fbb52700 1 -- 172.21.15.104:0/3531960556 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62f41a6a30 con 0x7f62e00035c0 2024-03-18T15:08:22.350 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:22.344+0000 7f62eaffd700 1 -- 172.21.15.104:0/3531960556 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f62f0004110 con 0x7f62e00035c0 2024-03-18T15:08:22.350 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:22.344+0000 7f62eaffd700 1 -- 172.21.15.104:0/3531960556 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62f0022040 con 0x7f62e00035c0 2024-03-18T15:08:22.350 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:22.344+0000 7f62fbb52700 1 -- 172.21.15.104:0/3531960556 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62f41a6f70 con 0x7f62e00035c0 2024-03-18T15:08:22.350 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:22.346+0000 7f62fbb52700 1 -- 172.21.15.104:0/3531960556 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62f4063900 con 0x7f62e00035c0 2024-03-18T15:08:22.350 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:22.346+0000 7f62eaffd700 1 -- 172.21.15.104:0/3531960556 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f62f002e020 con 0x7f62e00035c0 2024-03-18T15:08:22.350 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:22.346+0000 7f62eaffd700 1 --2- 172.21.15.104:0/3531960556 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f62d8039450 0x7f62d803b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:08:22.350 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:22.346+0000 7f62eaffd700 1 -- 172.21.15.104:0/3531960556 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f62f004ea20 con 0x7f62e00035c0 2024-03-18T15:08:22.350 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:22.347+0000 7f62f90ed700 1 --2- 172.21.15.104:0/3531960556 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f62d8039450 0x7f62d803b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:08:22.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:22.348+0000 7f62f90ed700 1 --2- 172.21.15.104:0/3531960556 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f62d8039450 0x7f62d803b910 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f62e4009850 tx=0x7f62e4008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:08:22.364 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:22.363+0000 7f62eaffd700 1 -- 172.21.15.104:0/3531960556 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f62f0035020 con 0x7f62e00035c0 2024-03-18T15:08:22.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:22 smithi019 ceph-mon[31522]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:22.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:22 smithi019 ceph-mon[31522]: mon.c calling monitor election 2024-03-18T15:08:22.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:22 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T15:08:22.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:22 smithi019 ceph-mon[31522]: mon.a calling monitor election 2024-03-18T15:08:22.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:22 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T15:08:22.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:22 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T15:08:22.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:22 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T15:08:22.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:22 smithi019 ceph-mon[31522]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:22.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:22 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T15:08:22.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:22 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T15:08:22.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:22 smithi019 ceph-mon[31522]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:22.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:22 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T15:08:22.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:22 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T15:08:22.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:22 smithi019 ceph-mon[31522]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-18T15:08:22.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:22 smithi019 ceph-mon[31522]: monmap epoch 3 2024-03-18T15:08:22.467 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:22 smithi019 ceph-mon[31522]: fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 2024-03-18T15:08:22.467 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:22 smithi019 ceph-mon[31522]: last_changed 2024-03-18T15:08:16.947306+0000 2024-03-18T15:08:22.467 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:22 smithi019 ceph-mon[31522]: created 2024-03-18T15:05:53.426642+0000 2024-03-18T15:08:22.467 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:22 smithi019 ceph-mon[31522]: min_mon_release 19 (squid) 2024-03-18T15:08:22.467 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:22 smithi019 ceph-mon[31522]: election_strategy: 1 2024-03-18T15:08:22.467 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:22 smithi019 ceph-mon[31522]: 0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-03-18T15:08:22.467 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:22 smithi019 ceph-mon[31522]: 1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.c 2024-03-18T15:08:22.467 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:22 smithi019 ceph-mon[31522]: 2: [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] mon.b 2024-03-18T15:08:22.467 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:22 smithi019 ceph-mon[31522]: fsmap 2024-03-18T15:08:22.467 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:22 smithi019 ceph-mon[31522]: osdmap e4: 0 total, 0 up, 0 in 2024-03-18T15:08:22.467 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:22 smithi019 ceph-mon[31522]: mgrmap e13: x(active, since 67s) 2024-03-18T15:08:22.467 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:22 smithi019 ceph-mon[31522]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-18T15:08:22.467 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:22 smithi019 ceph-mon[31522]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-18T15:08:22.467 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:22 smithi019 ceph-mon[31522]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-18T15:08:22.467 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:22 smithi019 ceph-mon[31522]: mon.b (rank 2) addr [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] is down (out of quorum) 2024-03-18T15:08:22.468 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:22 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:22.468 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:22 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:22.651 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:22.651+0000 7f62fbb52700 1 -- 172.21.15.104:0/3531960556 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f62f41a7240 con 0x7f62e00035c0 2024-03-18T15:08:22.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:22.652+0000 7f62eaffd700 1 -- 172.21.15.104:0/3531960556 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1279 (secure 0 0 0) 0x7f62f0010c70 con 0x7f62e00035c0 2024-03-18T15:08:22.653 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-18T15:08:22.653 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":3,"fsid":"d4dc2ba6-e538-11ee-95c9-87774f69a715","modified":"2024-03-18T15:08:16.947306Z","created":"2024-03-18T15:05:53.426642Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:3300","nonce":0},{"type":"v1","addr":"172.21.15.33:6789","nonce":0}]},"addr":"172.21.15.33:6789/0","public_addr":"172.21.15.33:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-18T15:08:22.655 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:22.654+0000 7f62fbb52700 1 -- 172.21.15.104:0/3531960556 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f62d8039450 msgr2=0x7f62d803b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:08:22.655 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:22.654+0000 7f62fbb52700 1 --2- 172.21.15.104:0/3531960556 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f62d8039450 0x7f62d803b910 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f62e4009850 tx=0x7f62e4008040 comp rx=0 tx=0).stop 2024-03-18T15:08:22.655 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:22.654+0000 7f62fbb52700 1 -- 172.21.15.104:0/3531960556 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f62e00035c0 msgr2=0x7f62f4100c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:08:22.655 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:22.654+0000 7f62fbb52700 1 --2- 172.21.15.104:0/3531960556 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f62e00035c0 0x7f62f4100c50 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f62f406a530 tx=0x7f62f0005720 comp rx=0 tx=0).stop 2024-03-18T15:08:22.655 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:22.654+0000 7f62fbb52700 1 -- 172.21.15.104:0/3531960556 shutdown_connections 2024-03-18T15:08:22.655 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:22.654+0000 7f62fbb52700 1 --2- 172.21.15.104:0/3531960556 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f62d8039450 0x7f62d803b910 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:08:22.655 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:22.654+0000 7f62fbb52700 1 --2- 172.21.15.104:0/3531960556 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62f41a3c80 0x7f62f41a6070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:08:22.655 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:22.654+0000 7f62fbb52700 1 --2- 172.21.15.104:0/3531960556 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f62e00035c0 0x7f62f4100c50 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:08:22.655 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:22.654+0000 7f62fbb52700 1 --2- 172.21.15.104:0/3531960556 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f62f4101190 0x7f62f4101610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:08:22.656 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:22.654+0000 7f62fbb52700 1 -- 172.21.15.104:0/3531960556 >> 172.21.15.104:0/3531960556 conn(0x7f62f40fc7e0 msgr2=0x7f62f4103da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:08:22.656 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:22.654+0000 7f62fbb52700 1 -- 172.21.15.104:0/3531960556 shutdown_connections 2024-03-18T15:08:22.656 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:22.654+0000 7f62fbb52700 1 -- 172.21.15.104:0/3531960556 wait complete. 2024-03-18T15:08:22.656 INFO:teuthology.orchestra.run.smithi104.stderr:dumped monmap epoch 3 2024-03-18T15:08:23.229 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:23 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:23.229 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:23 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:23.229 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:23 smithi104 ceph-mon[36070]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:23.230 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:23 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:08:23.230 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:23 smithi104 ceph-mon[36070]: from='client.? 172.21.15.104:0/3531960556' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-18T15:08:23.230 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:23 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T15:08:23.254 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-18T15:08:23.254 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 -- ceph config generate-minimal-conf 2024-03-18T15:08:23.341 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:23 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:23.341 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:23 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:23.341 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:23 smithi019 ceph-mon[31522]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:23.341 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:23 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:08:23.341 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:23 smithi019 ceph-mon[31522]: from='client.? 172.21.15.104:0/3531960556' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-18T15:08:23.341 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:23 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T15:08:23.702 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:08:24.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:23 smithi033 ceph-mon[36648]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:24.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:23 smithi033 ceph-mon[36648]: mon.c calling monitor election 2024-03-18T15:08:24.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:23 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T15:08:24.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:23 smithi033 ceph-mon[36648]: mon.a calling monitor election 2024-03-18T15:08:24.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:23 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T15:08:24.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:23 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T15:08:24.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:23 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T15:08:24.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:23 smithi033 ceph-mon[36648]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:24.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:23 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T15:08:24.258 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:23 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T15:08:24.258 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:23 smithi033 ceph-mon[36648]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:24.258 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:23 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T15:08:24.258 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:23 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T15:08:24.258 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:23 smithi033 ceph-mon[36648]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-18T15:08:24.258 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:23 smithi033 ceph-mon[36648]: monmap epoch 3 2024-03-18T15:08:24.258 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:23 smithi033 ceph-mon[36648]: fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 2024-03-18T15:08:24.258 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:23 smithi033 ceph-mon[36648]: last_changed 2024-03-18T15:08:16.947306+0000 2024-03-18T15:08:24.258 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:23 smithi033 ceph-mon[36648]: created 2024-03-18T15:05:53.426642+0000 2024-03-18T15:08:24.258 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:23 smithi033 ceph-mon[36648]: min_mon_release 19 (squid) 2024-03-18T15:08:24.258 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:23 smithi033 ceph-mon[36648]: election_strategy: 1 2024-03-18T15:08:24.258 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:23 smithi033 ceph-mon[36648]: 0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-03-18T15:08:24.258 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:23 smithi033 ceph-mon[36648]: 1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.c 2024-03-18T15:08:24.259 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:23 smithi033 ceph-mon[36648]: 2: [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] mon.b 2024-03-18T15:08:24.259 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:23 smithi033 ceph-mon[36648]: fsmap 2024-03-18T15:08:24.259 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:23 smithi033 ceph-mon[36648]: osdmap e4: 0 total, 0 up, 0 in 2024-03-18T15:08:24.259 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:23 smithi033 ceph-mon[36648]: mgrmap e13: x(active, since 67s) 2024-03-18T15:08:24.259 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:23 smithi033 ceph-mon[36648]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-18T15:08:24.259 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:23 smithi033 ceph-mon[36648]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-18T15:08:24.259 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:23 smithi033 ceph-mon[36648]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-18T15:08:24.259 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:23 smithi033 ceph-mon[36648]: mon.b (rank 2) addr [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] is down (out of quorum) 2024-03-18T15:08:24.259 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:23 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:24.259 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:23 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:24.259 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:23 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:24.259 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:23 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:24.259 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:23 smithi033 ceph-mon[36648]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:24.259 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:23 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:08:24.259 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:23 smithi033 ceph-mon[36648]: from='client.? 172.21.15.104:0/3531960556' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-18T15:08:24.259 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:23 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T15:08:24.845 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:24 smithi019 ceph-mon[31522]: mon.b calling monitor election 2024-03-18T15:08:24.846 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:24 smithi019 ceph-mon[31522]: mon.b calling monitor election 2024-03-18T15:08:24.846 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:24 smithi019 ceph-mon[31522]: mon.c calling monitor election 2024-03-18T15:08:24.846 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:24 smithi019 ceph-mon[31522]: mon.a calling monitor election 2024-03-18T15:08:24.846 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:24 smithi019 ceph-mon[31522]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-18T15:08:24.846 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:24 smithi019 ceph-mon[31522]: monmap epoch 3 2024-03-18T15:08:24.846 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:24 smithi019 ceph-mon[31522]: fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 2024-03-18T15:08:24.846 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:24 smithi019 ceph-mon[31522]: last_changed 2024-03-18T15:08:16.947306+0000 2024-03-18T15:08:24.846 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:24 smithi019 ceph-mon[31522]: created 2024-03-18T15:05:53.426642+0000 2024-03-18T15:08:24.846 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:24 smithi019 ceph-mon[31522]: min_mon_release 19 (squid) 2024-03-18T15:08:24.846 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:24 smithi019 ceph-mon[31522]: election_strategy: 1 2024-03-18T15:08:24.846 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:24 smithi019 ceph-mon[31522]: 0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-03-18T15:08:24.846 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:24 smithi019 ceph-mon[31522]: 1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.c 2024-03-18T15:08:24.846 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:24 smithi019 ceph-mon[31522]: 2: [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] mon.b 2024-03-18T15:08:24.847 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:24 smithi019 ceph-mon[31522]: fsmap 2024-03-18T15:08:24.847 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:24 smithi019 ceph-mon[31522]: osdmap e4: 0 total, 0 up, 0 in 2024-03-18T15:08:24.847 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:24 smithi019 ceph-mon[31522]: mgrmap e13: x(active, since 69s) 2024-03-18T15:08:24.847 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:24 smithi019 ceph-mon[31522]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-18T15:08:24.847 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:24 smithi019 ceph-mon[31522]: Cluster is now healthy 2024-03-18T15:08:24.847 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:24 smithi019 ceph-mon[31522]: overall HEALTH_OK 2024-03-18T15:08:24.847 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:24 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:24.847 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:24 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:24.847 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:24 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:08:24.847 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:24 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:08:24.847 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:24 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:24.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:24 smithi104 ceph-mon[36070]: mon.b calling monitor election 2024-03-18T15:08:24.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:24 smithi104 ceph-mon[36070]: mon.b calling monitor election 2024-03-18T15:08:24.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:24 smithi104 ceph-mon[36070]: mon.c calling monitor election 2024-03-18T15:08:24.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:24 smithi104 ceph-mon[36070]: mon.a calling monitor election 2024-03-18T15:08:24.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:24 smithi104 ceph-mon[36070]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-18T15:08:24.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:24 smithi104 ceph-mon[36070]: monmap epoch 3 2024-03-18T15:08:24.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:24 smithi104 ceph-mon[36070]: fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 2024-03-18T15:08:24.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:24 smithi104 ceph-mon[36070]: last_changed 2024-03-18T15:08:16.947306+0000 2024-03-18T15:08:24.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:24 smithi104 ceph-mon[36070]: created 2024-03-18T15:05:53.426642+0000 2024-03-18T15:08:24.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:24 smithi104 ceph-mon[36070]: min_mon_release 19 (squid) 2024-03-18T15:08:24.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:24 smithi104 ceph-mon[36070]: election_strategy: 1 2024-03-18T15:08:24.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:24 smithi104 ceph-mon[36070]: 0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-03-18T15:08:24.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:24 smithi104 ceph-mon[36070]: 1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.c 2024-03-18T15:08:24.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:24 smithi104 ceph-mon[36070]: 2: [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] mon.b 2024-03-18T15:08:24.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:24 smithi104 ceph-mon[36070]: fsmap 2024-03-18T15:08:24.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:24 smithi104 ceph-mon[36070]: osdmap e4: 0 total, 0 up, 0 in 2024-03-18T15:08:24.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:24 smithi104 ceph-mon[36070]: mgrmap e13: x(active, since 69s) 2024-03-18T15:08:24.946 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:24 smithi104 ceph-mon[36070]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-18T15:08:24.946 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:24 smithi104 ceph-mon[36070]: Cluster is now healthy 2024-03-18T15:08:24.946 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:24 smithi104 ceph-mon[36070]: overall HEALTH_OK 2024-03-18T15:08:24.946 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:24 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:24.946 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:24 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:24.946 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:24 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:08:24.946 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:24 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:08:24.946 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:24 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:25.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:24 smithi033 ceph-mon[36648]: mon.b calling monitor election 2024-03-18T15:08:25.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:24 smithi033 ceph-mon[36648]: mon.b calling monitor election 2024-03-18T15:08:25.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:24 smithi033 ceph-mon[36648]: mon.c calling monitor election 2024-03-18T15:08:25.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:24 smithi033 ceph-mon[36648]: mon.a calling monitor election 2024-03-18T15:08:25.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:24 smithi033 ceph-mon[36648]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-18T15:08:25.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:24 smithi033 ceph-mon[36648]: monmap epoch 3 2024-03-18T15:08:25.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:24 smithi033 ceph-mon[36648]: fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 2024-03-18T15:08:25.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:24 smithi033 ceph-mon[36648]: last_changed 2024-03-18T15:08:16.947306+0000 2024-03-18T15:08:25.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:24 smithi033 ceph-mon[36648]: created 2024-03-18T15:05:53.426642+0000 2024-03-18T15:08:25.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:24 smithi033 ceph-mon[36648]: min_mon_release 19 (squid) 2024-03-18T15:08:25.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:24 smithi033 ceph-mon[36648]: election_strategy: 1 2024-03-18T15:08:25.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:24 smithi033 ceph-mon[36648]: 0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-03-18T15:08:25.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:24 smithi033 ceph-mon[36648]: 1: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.c 2024-03-18T15:08:25.008 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:24 smithi033 ceph-mon[36648]: 2: [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] mon.b 2024-03-18T15:08:25.008 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:24 smithi033 ceph-mon[36648]: fsmap 2024-03-18T15:08:25.008 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:24 smithi033 ceph-mon[36648]: osdmap e4: 0 total, 0 up, 0 in 2024-03-18T15:08:25.008 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:24 smithi033 ceph-mon[36648]: mgrmap e13: x(active, since 69s) 2024-03-18T15:08:25.008 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:24 smithi033 ceph-mon[36648]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-18T15:08:25.008 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:24 smithi033 ceph-mon[36648]: Cluster is now healthy 2024-03-18T15:08:25.008 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:24 smithi033 ceph-mon[36648]: overall HEALTH_OK 2024-03-18T15:08:25.008 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:24 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:25.008 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:24 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:25.008 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:24 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:08:25.008 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:24 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:08:25.008 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:24 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:25.627 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:25.626+0000 7f179ce00700 1 -- 172.21.15.19:0/74659839 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1798104120 msgr2=0x7f1798104500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:08:25.627 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:25.626+0000 7f179ce00700 1 --2- 172.21.15.19:0/74659839 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1798104120 0x7f1798104500 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f178800b0a0 tx=0x7f178800b3b0 comp rx=0 tx=0).stop 2024-03-18T15:08:25.628 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:25.626+0000 7f179ce00700 1 -- 172.21.15.19:0/74659839 shutdown_connections 2024-03-18T15:08:25.628 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:25.626+0000 7f179ce00700 1 --2- 172.21.15.19:0/74659839 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1798104120 0x7f1798104500 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:08:25.628 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:25.626+0000 7f179ce00700 1 -- 172.21.15.19:0/74659839 >> 172.21.15.19:0/74659839 conn(0x7f17980ff9d0 msgr2=0x7f1798101df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:08:25.628 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:25.626+0000 7f179ce00700 1 -- 172.21.15.19:0/74659839 shutdown_connections 2024-03-18T15:08:25.628 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:25.626+0000 7f179ce00700 1 -- 172.21.15.19:0/74659839 wait complete. 2024-03-18T15:08:25.628 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:25.627+0000 7f179ce00700 1 Processor -- start 2024-03-18T15:08:25.628 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:25.627+0000 7f179ce00700 1 -- start start 2024-03-18T15:08:25.629 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:25.628+0000 7f179ce00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1798104120 0x7f179819f890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:08:25.629 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:25.628+0000 7f179ce00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f179819fdd0 0x7f1798199a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:08:25.629 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:25.628+0000 7f179ce00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1798199f90 0x7f179819a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:08:25.629 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:25.628+0000 7f179ce00700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f179810e4b0 con 0x7f1798199f90 2024-03-18T15:08:25.629 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:25.628+0000 7f179ce00700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f179810e330 con 0x7f1798104120 2024-03-18T15:08:25.629 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:25.628+0000 7f179ce00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f179810e630 con 0x7f179819fdd0 2024-03-18T15:08:25.629 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:25.628+0000 7f1796800700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1798199f90 0x7f179819a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:08:25.629 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:25.628+0000 7f178d400700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f179819fdd0 0x7f1798199a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:08:25.630 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:25.628+0000 7f1795e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1798104120 0x7f179819f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:08:25.630 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:25.628+0000 7f178d400700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f179819fdd0 0x7f1798199a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.19:41396/0 (socket says 172.21.15.19:41396) 2024-03-18T15:08:25.630 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:25.628+0000 7f178d400700 1 -- 172.21.15.19:0/4181191443 learned_addr learned my addr 172.21.15.19:0/4181191443 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:08:25.631 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:25.628+0000 7f1795e00700 1 -- 172.21.15.19:0/4181191443 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f179819fdd0 msgr2=0x7f1798199a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:08:25.631 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:25.628+0000 7f1795e00700 1 --2- 172.21.15.19:0/4181191443 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f179819fdd0 0x7f1798199a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:08:25.631 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:25.628+0000 7f1795e00700 1 -- 172.21.15.19:0/4181191443 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1798199f90 msgr2=0x7f179819a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:08:25.631 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:25.628+0000 7f1795e00700 1 --2- 172.21.15.19:0/4181191443 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1798199f90 0x7f179819a440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:08:25.631 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:25.628+0000 7f1795e00700 1 -- 172.21.15.19:0/4181191443 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17981a6760 con 0x7f1798104120 2024-03-18T15:08:25.631 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:25.629+0000 7f178d400700 1 --2- 172.21.15.19:0/4181191443 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f179819fdd0 0x7f1798199a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:08:25.631 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:25.629+0000 7f1795e00700 1 --2- 172.21.15.19:0/4181191443 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1798104120 0x7f179819f890 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f1788007600 tx=0x7f1788003f70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:08:25.631 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:25.629+0000 7f178f400700 1 -- 172.21.15.19:0/4181191443 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1788022070 con 0x7f1798104120 2024-03-18T15:08:25.631 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:25.629+0000 7f178f400700 1 -- 172.21.15.19:0/4181191443 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1788005580 con 0x7f1798104120 2024-03-18T15:08:25.631 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:25.629+0000 7f178f400700 1 -- 172.21.15.19:0/4181191443 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1788017070 con 0x7f1798104120 2024-03-18T15:08:25.631 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:25.629+0000 7f179ce00700 1 -- 172.21.15.19:0/4181191443 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17981a6930 con 0x7f1798104120 2024-03-18T15:08:25.632 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:25.629+0000 7f179ce00700 1 -- 172.21.15.19:0/4181191443 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17981a6df0 con 0x7f1798104120 2024-03-18T15:08:25.632 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:25.631+0000 7f179ce00700 1 -- 172.21.15.19:0/4181191443 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1798063900 con 0x7f1798104120 2024-03-18T15:08:25.632 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:25.631+0000 7f178f400700 1 -- 172.21.15.19:0/4181191443 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f1788005030 con 0x7f1798104120 2024-03-18T15:08:25.632 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:25.631+0000 7f178f400700 1 --2- 172.21.15.19:0/4181191443 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f177c039450 0x7f177c03b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:08:25.633 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:25.631+0000 7f178f400700 1 -- 172.21.15.19:0/4181191443 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f178800f020 con 0x7f1798104120 2024-03-18T15:08:25.633 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:25.631+0000 7f178d400700 1 --2- 172.21.15.19:0/4181191443 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f177c039450 0x7f177c03b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:08:25.633 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:25.632+0000 7f178d400700 1 --2- 172.21.15.19:0/4181191443 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f177c039450 0x7f177c03b910 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1780009f70 tx=0x7f17800092c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:08:25.638 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:25.637+0000 7f178f400700 1 -- 172.21.15.19:0/4181191443 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1788005320 con 0x7f1798104120 2024-03-18T15:08:25.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:25 smithi019 ceph-mon[31522]: Updating smithi019:/etc/ceph/ceph.conf 2024-03-18T15:08:25.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:25 smithi019 ceph-mon[31522]: Updating smithi033:/etc/ceph/ceph.conf 2024-03-18T15:08:25.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:25 smithi019 ceph-mon[31522]: Updating smithi104:/etc/ceph/ceph.conf 2024-03-18T15:08:25.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:25 smithi019 ceph-mon[31522]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:25.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:25 smithi019 ceph-mon[31522]: Updating smithi033:/var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T15:08:25.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:25 smithi019 ceph-mon[31522]: Updating smithi104:/var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T15:08:25.717 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:25 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:25.717 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:25 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:25.717 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:25 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:25.717 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:25 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:25.717 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:25 smithi019 ceph-mon[31522]: Updating smithi019:/var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T15:08:25.717 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:25 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T15:08:25.717 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:25 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:25.717 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:25 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:25.717 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:25 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:25.874 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:25.873+0000 7f179ce00700 1 -- 172.21.15.19:0/4181191443 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f17981a7120 con 0x7f1798104120 2024-03-18T15:08:25.875 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:25.874+0000 7f178f400700 1 -- 172.21.15.19:0/4181191443 <== mon.2 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+273 (secure 0 0 0) 0x7f1788022da0 con 0x7f1798104120 2024-03-18T15:08:25.875 INFO:teuthology.orchestra.run.smithi019.stdout:# minimal ceph.conf for d4dc2ba6-e538-11ee-95c9-87774f69a715 2024-03-18T15:08:25.875 INFO:teuthology.orchestra.run.smithi019.stdout:[global] 2024-03-18T15:08:25.875 INFO:teuthology.orchestra.run.smithi019.stdout: fsid = d4dc2ba6-e538-11ee-95c9-87774f69a715 2024-03-18T15:08:25.875 INFO:teuthology.orchestra.run.smithi019.stdout: mon_host = [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-03-18T15:08:25.878 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:25.877+0000 7f179ce00700 1 -- 172.21.15.19:0/4181191443 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f177c039450 msgr2=0x7f177c03b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:08:25.878 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:25.877+0000 7f179ce00700 1 --2- 172.21.15.19:0/4181191443 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f177c039450 0x7f177c03b910 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1780009f70 tx=0x7f17800092c0 comp rx=0 tx=0).stop 2024-03-18T15:08:25.878 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:25.877+0000 7f179ce00700 1 -- 172.21.15.19:0/4181191443 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1798104120 msgr2=0x7f179819f890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:08:25.878 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:25.877+0000 7f179ce00700 1 --2- 172.21.15.19:0/4181191443 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1798104120 0x7f179819f890 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f1788007600 tx=0x7f1788003f70 comp rx=0 tx=0).stop 2024-03-18T15:08:25.878 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:25.877+0000 7f179ce00700 1 -- 172.21.15.19:0/4181191443 shutdown_connections 2024-03-18T15:08:25.878 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:25.877+0000 7f179ce00700 1 --2- 172.21.15.19:0/4181191443 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f177c039450 0x7f177c03b910 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:08:25.878 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:25.877+0000 7f179ce00700 1 --2- 172.21.15.19:0/4181191443 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1798199f90 0x7f179819a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:08:25.878 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:25.877+0000 7f179ce00700 1 --2- 172.21.15.19:0/4181191443 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1798104120 0x7f179819f890 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:08:25.878 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:25.877+0000 7f179ce00700 1 --2- 172.21.15.19:0/4181191443 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f179819fdd0 0x7f1798199a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:08:25.879 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:25.877+0000 7f179ce00700 1 -- 172.21.15.19:0/4181191443 >> 172.21.15.19:0/4181191443 conn(0x7f17980ff9d0 msgr2=0x7f1798105af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:08:25.879 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:25.877+0000 7f179ce00700 1 -- 172.21.15.19:0/4181191443 shutdown_connections 2024-03-18T15:08:25.879 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:25.877+0000 7f179ce00700 1 -- 172.21.15.19:0/4181191443 wait complete. 2024-03-18T15:08:25.946 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:25 smithi104 ceph-mon[36070]: Updating smithi019:/etc/ceph/ceph.conf 2024-03-18T15:08:25.947 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:25 smithi104 ceph-mon[36070]: Updating smithi033:/etc/ceph/ceph.conf 2024-03-18T15:08:25.947 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:25 smithi104 ceph-mon[36070]: Updating smithi104:/etc/ceph/ceph.conf 2024-03-18T15:08:25.947 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:25 smithi104 ceph-mon[36070]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:25.947 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:25 smithi104 ceph-mon[36070]: Updating smithi033:/var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T15:08:25.947 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:25 smithi104 ceph-mon[36070]: Updating smithi104:/var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T15:08:25.947 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:25 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:25.947 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:25 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:25.947 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:25 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:25.947 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:25 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:25.947 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:25 smithi104 ceph-mon[36070]: Updating smithi019:/var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T15:08:25.947 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:25 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T15:08:25.947 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:25 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:25.947 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:25 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:25.948 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:25 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:26.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:25 smithi033 ceph-mon[36648]: Updating smithi019:/etc/ceph/ceph.conf 2024-03-18T15:08:26.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:25 smithi033 ceph-mon[36648]: Updating smithi033:/etc/ceph/ceph.conf 2024-03-18T15:08:26.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:25 smithi033 ceph-mon[36648]: Updating smithi104:/etc/ceph/ceph.conf 2024-03-18T15:08:26.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:25 smithi033 ceph-mon[36648]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:26.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:25 smithi033 ceph-mon[36648]: Updating smithi033:/var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T15:08:26.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:25 smithi033 ceph-mon[36648]: Updating smithi104:/var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T15:08:26.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:25 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:26.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:25 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:26.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:25 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:26.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:25 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:26.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:25 smithi033 ceph-mon[36648]: Updating smithi019:/var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T15:08:26.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:25 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T15:08:26.008 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:25 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:26.008 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:25 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:26.008 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:25 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:26.126 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:08:25 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:08:25.947+0000 7f44e6000700 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-18T15:08:26.585 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-18T15:08:26.585 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-03-18T15:08:26.585 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/etc/ceph/ceph.conf 2024-03-18T15:08:26.670 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-03-18T15:08:26.671 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-18T15:08:26.753 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:26 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:26.754 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:26 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:26.754 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:26 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:26.754 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:26 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:26.754 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:26 smithi019 ceph-mon[31522]: Reconfiguring mon.a (unknown last config time)... 2024-03-18T15:08:26.754 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:26 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T15:08:26.754 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:26 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-18T15:08:26.754 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:26 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:08:26.754 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:26 smithi019 ceph-mon[31522]: Reconfiguring daemon mon.a on smithi019 2024-03-18T15:08:26.754 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:26 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/4181191443' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:08:26.785 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-18T15:08:26.785 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/etc/ceph/ceph.conf 2024-03-18T15:08:26.819 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-18T15:08:26.819 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-18T15:08:26.890 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-03-18T15:08:26.891 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/etc/ceph/ceph.conf 2024-03-18T15:08:26.923 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-03-18T15:08:26.924 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-18T15:08:26.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:26 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:26.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:26 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:26.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:26 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:26.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:26 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:26.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:26 smithi104 ceph-mon[36070]: Reconfiguring mon.a (unknown last config time)... 2024-03-18T15:08:26.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:26 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T15:08:26.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:26 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-18T15:08:26.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:26 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:08:26.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:26 smithi104 ceph-mon[36070]: Reconfiguring daemon mon.a on smithi019 2024-03-18T15:08:26.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:26 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/4181191443' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:08:26.972 INFO:tasks.cephadm:Adding mgr.x on smithi019 2024-03-18T15:08:26.972 INFO:tasks.cephadm:Adding mgr.y on smithi033 2024-03-18T15:08:26.972 INFO:tasks.cephadm:Adding mgr.z on smithi104 2024-03-18T15:08:26.972 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 -- ceph orch apply mgr '3;smithi019=x;smithi033=y;smithi104=z' 2024-03-18T15:08:27.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:26 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:27.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:26 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:27.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:26 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:27.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:26 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:27.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:26 smithi033 ceph-mon[36648]: Reconfiguring mon.a (unknown last config time)... 2024-03-18T15:08:27.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:26 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T15:08:27.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:26 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-18T15:08:27.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:26 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:08:27.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:26 smithi033 ceph-mon[36648]: Reconfiguring daemon mon.a on smithi019 2024-03-18T15:08:27.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:26 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/4181191443' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:08:27.248 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T15:08:27.775 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:27 smithi019 ceph-mon[31522]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:27.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:27 smithi104 ceph-mon[36070]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:28.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:27 smithi033 ceph-mon[36648]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:28.713 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:28.709+0000 7fcd01518700 1 -- 172.21.15.104:0/2630308466 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fccfc100cf0 msgr2=0x7fccfc101110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:08:28.713 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:28.709+0000 7fcd01518700 1 --2- 172.21.15.104:0/2630308466 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fccfc100cf0 0x7fccfc101110 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fccec009910 tx=0x7fccec009c20 comp rx=0 tx=0).stop 2024-03-18T15:08:28.713 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:28.709+0000 7fcd01518700 1 -- 172.21.15.104:0/2630308466 shutdown_connections 2024-03-18T15:08:28.713 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:28.709+0000 7fcd01518700 1 --2- 172.21.15.104:0/2630308466 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fccfc100cf0 0x7fccfc101110 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:08:28.713 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:28.709+0000 7fcd01518700 1 -- 172.21.15.104:0/2630308466 >> 172.21.15.104:0/2630308466 conn(0x7fccfc0fc2b0 msgr2=0x7fccfc0fe6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:08:28.713 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:28.709+0000 7fcd01518700 1 -- 172.21.15.104:0/2630308466 shutdown_connections 2024-03-18T15:08:28.713 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:28.709+0000 7fcd01518700 1 -- 172.21.15.104:0/2630308466 wait complete. 2024-03-18T15:08:28.713 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:28.709+0000 7fcd01518700 1 Processor -- start 2024-03-18T15:08:28.713 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:28.710+0000 7fcd01518700 1 -- start start 2024-03-18T15:08:28.713 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:28.710+0000 7fcd01518700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fccfc100cf0 0x7fccfc199800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:08:28.714 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:28.710+0000 7fcd01518700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fccfc199d40 0x7fccfc19edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:08:28.714 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:28.710+0000 7fcd01518700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fccfc19a1c0 0x7fccfc1a1340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:08:28.714 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:28.710+0000 7fcd01518700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fccfc106d30 con 0x7fccfc19a1c0 2024-03-18T15:08:28.714 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:28.710+0000 7fcd01518700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fccfc106bb0 con 0x7fccfc100cf0 2024-03-18T15:08:28.714 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:28.710+0000 7fcd01518700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fccfc106eb0 con 0x7fccfc199d40 2024-03-18T15:08:28.714 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:28.710+0000 7fccfaffd700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fccfc100cf0 0x7fccfc199800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:08:28.714 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:28.710+0000 7fccfa7fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fccfc199d40 0x7fccfc19edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:08:28.714 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:28.710+0000 7fccfaffd700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fccfc100cf0 0x7fccfc199800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.104:38242/0 (socket says 172.21.15.104:38242) 2024-03-18T15:08:28.714 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:28.710+0000 7fccfaffd700 1 -- 172.21.15.104:0/3559320223 learned_addr learned my addr 172.21.15.104:0/3559320223 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-18T15:08:28.714 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:28.710+0000 7fccfb7fe700 1 --2- 172.21.15.104:0/3559320223 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fccfc19a1c0 0x7fccfc1a1340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:08:28.714 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:28.710+0000 7fccfa7fc700 1 -- 172.21.15.104:0/3559320223 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fccfc100cf0 msgr2=0x7fccfc199800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:08:28.715 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:28.710+0000 7fccfa7fc700 1 --2- 172.21.15.104:0/3559320223 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fccfc100cf0 0x7fccfc199800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:08:28.715 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:28.710+0000 7fccfa7fc700 1 -- 172.21.15.104:0/3559320223 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fccfc19a1c0 msgr2=0x7fccfc1a1340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:08:28.715 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:28.710+0000 7fccfa7fc700 1 --2- 172.21.15.104:0/3559320223 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fccfc19a1c0 0x7fccfc1a1340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:08:28.715 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:28.710+0000 7fccfa7fc700 1 -- 172.21.15.104:0/3559320223 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccfc1a1c10 con 0x7fccfc199d40 2024-03-18T15:08:28.715 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:28.711+0000 7fccfa7fc700 1 --2- 172.21.15.104:0/3559320223 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fccfc199d40 0x7fccfc19edb0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fccf000b550 tx=0x7fccf000b860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:08:28.715 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:28.711+0000 7fcce3fff700 1 -- 172.21.15.104:0/3559320223 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccf0004230 con 0x7fccfc199d40 2024-03-18T15:08:28.715 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:28.711+0000 7fcce3fff700 1 -- 172.21.15.104:0/3559320223 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fccf00043d0 con 0x7fccfc199d40 2024-03-18T15:08:28.715 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:28.711+0000 7fcd01518700 1 -- 172.21.15.104:0/3559320223 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fccfc1a1f00 con 0x7fccfc199d40 2024-03-18T15:08:28.715 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:28.711+0000 7fcce3fff700 1 -- 172.21.15.104:0/3559320223 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccf0011c20 con 0x7fccfc199d40 2024-03-18T15:08:28.715 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:28.711+0000 7fcd01518700 1 -- 172.21.15.104:0/3559320223 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fccfc1a2430 con 0x7fccfc199d40 2024-03-18T15:08:28.715 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:28.712+0000 7fcce3fff700 1 -- 172.21.15.104:0/3559320223 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7fccf001a400 con 0x7fccfc199d40 2024-03-18T15:08:28.715 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:28.712+0000 7fcd01518700 1 -- 172.21.15.104:0/3559320223 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fccfc063900 con 0x7fccfc199d40 2024-03-18T15:08:28.715 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:28.712+0000 7fcce3fff700 1 --2- 172.21.15.104:0/3559320223 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fcce4039450 0x7fcce403b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:08:28.716 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:28.712+0000 7fcce3fff700 1 -- 172.21.15.104:0/3559320223 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fccf004bae0 con 0x7fccfc199d40 2024-03-18T15:08:28.716 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:28.712+0000 7fccfaffd700 1 --2- 172.21.15.104:0/3559320223 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fcce4039450 0x7fcce403b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:08:28.716 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:28.713+0000 7fccfaffd700 1 --2- 172.21.15.104:0/3559320223 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fcce4039450 0x7fcce403b910 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fccec005200 tx=0x7fccec01a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:08:28.717 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:28.717+0000 7fcce3fff700 1 -- 172.21.15.104:0/3559320223 <== mon.1 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fccf0020500 con 0x7fccfc199d40 2024-03-18T15:08:28.919 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:28.918+0000 7fcd01518700 1 -- 172.21.15.104:0/3559320223 --> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi019=x;smithi033=y;smithi104=z", "target": ["mon-mgr", ""]}) -- 0x7fccfc0627a0 con 0x7fcce4039450 2024-03-18T15:08:28.932 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:28.932+0000 7fcce3fff700 1 -- 172.21.15.104:0/3559320223 <== mgr.14152 v2:172.21.15.19:6800/2017714236 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fccfc0627a0 con 0x7fcce4039450 2024-03-18T15:08:28.933 INFO:teuthology.orchestra.run.smithi104.stdout:Scheduled mgr update... 2024-03-18T15:08:28.935 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:28.935+0000 7fcd01518700 1 -- 172.21.15.104:0/3559320223 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fcce4039450 msgr2=0x7fcce403b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:08:28.935 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:28.935+0000 7fcd01518700 1 --2- 172.21.15.104:0/3559320223 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fcce4039450 0x7fcce403b910 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fccec005200 tx=0x7fccec01a040 comp rx=0 tx=0).stop 2024-03-18T15:08:28.936 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:28.935+0000 7fcd01518700 1 -- 172.21.15.104:0/3559320223 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fccfc199d40 msgr2=0x7fccfc19edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:08:28.936 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:28.935+0000 7fcd01518700 1 --2- 172.21.15.104:0/3559320223 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fccfc199d40 0x7fccfc19edb0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fccf000b550 tx=0x7fccf000b860 comp rx=0 tx=0).stop 2024-03-18T15:08:28.936 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:28.935+0000 7fcd01518700 1 -- 172.21.15.104:0/3559320223 shutdown_connections 2024-03-18T15:08:28.936 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:28.935+0000 7fcd01518700 1 --2- 172.21.15.104:0/3559320223 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fcce4039450 0x7fcce403b910 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:08:28.936 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:28.935+0000 7fcd01518700 1 --2- 172.21.15.104:0/3559320223 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fccfc19a1c0 0x7fccfc1a1340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:08:28.936 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:28.935+0000 7fcd01518700 1 --2- 172.21.15.104:0/3559320223 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fccfc100cf0 0x7fccfc199800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:08:28.936 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:28.935+0000 7fcd01518700 1 --2- 172.21.15.104:0/3559320223 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fccfc199d40 0x7fccfc19edb0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:08:28.936 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:28.935+0000 7fcd01518700 1 -- 172.21.15.104:0/3559320223 >> 172.21.15.104:0/3559320223 conn(0x7fccfc0fc2b0 msgr2=0x7fccfc1907e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:08:28.936 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:28.935+0000 7fcd01518700 1 -- 172.21.15.104:0/3559320223 shutdown_connections 2024-03-18T15:08:28.936 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:08:28.935+0000 7fcd01518700 1 -- 172.21.15.104:0/3559320223 wait complete. 2024-03-18T15:08:29.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:28 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:29.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:28 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:29.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:28 smithi104 ceph-mon[36070]: Reconfiguring mon.b (monmap changed)... 2024-03-18T15:08:29.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:28 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T15:08:29.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:28 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-18T15:08:29.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:28 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:08:29.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:28 smithi104 ceph-mon[36070]: Reconfiguring daemon mon.b on smithi033 2024-03-18T15:08:29.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:28 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:29.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:28 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:29.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:28 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:29.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:28 smithi019 ceph-mon[31522]: Reconfiguring mon.b (monmap changed)... 2024-03-18T15:08:29.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:28 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T15:08:29.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:28 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-18T15:08:29.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:28 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:08:29.217 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:28 smithi019 ceph-mon[31522]: Reconfiguring daemon mon.b on smithi033 2024-03-18T15:08:29.217 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:28 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:29.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:28 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:29.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:28 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:29.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:28 smithi033 ceph-mon[36648]: Reconfiguring mon.b (monmap changed)... 2024-03-18T15:08:29.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:28 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T15:08:29.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:28 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-18T15:08:29.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:28 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:08:29.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:28 smithi033 ceph-mon[36648]: Reconfiguring daemon mon.b on smithi033 2024-03-18T15:08:29.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:28 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:29.518 DEBUG:teuthology.orchestra.run.smithi033:mgr.y> sudo journalctl -f -n 0 -u ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@mgr.y.service 2024-03-18T15:08:29.521 DEBUG:teuthology.orchestra.run.smithi104:mgr.z> sudo journalctl -f -n 0 -u ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@mgr.z.service 2024-03-18T15:08:29.523 INFO:tasks.cephadm:Deploying OSDs... 2024-03-18T15:08:29.523 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-03-18T15:08:29.523 DEBUG:teuthology.orchestra.run.smithi019:> dd if=/scratch_devs of=/dev/stdout 2024-03-18T15:08:29.583 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-18T15:08:29.584 DEBUG:teuthology.orchestra.run.smithi019:> stat /dev/vg_nvme/lv_1 2024-03-18T15:08:29.639 INFO:teuthology.orchestra.run.smithi019.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-18T15:08:29.639 INFO:teuthology.orchestra.run.smithi019.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T15:08:29.639 INFO:teuthology.orchestra.run.smithi019.stdout:Device: 5h/5d Inode: 631 Links: 1 2024-03-18T15:08:29.639 INFO:teuthology.orchestra.run.smithi019.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T15:08:29.639 INFO:teuthology.orchestra.run.smithi019.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T15:08:29.639 INFO:teuthology.orchestra.run.smithi019.stdout:Access: 2024-03-18 15:07:30.055207758 +0000 2024-03-18T15:08:29.639 INFO:teuthology.orchestra.run.smithi019.stdout:Modify: 2024-03-18 15:01:37.810156641 +0000 2024-03-18T15:08:29.639 INFO:teuthology.orchestra.run.smithi019.stdout:Change: 2024-03-18 15:01:37.810156641 +0000 2024-03-18T15:08:29.639 INFO:teuthology.orchestra.run.smithi019.stdout: Birth: 2024-03-18 15:01:37.810156641 +0000 2024-03-18T15:08:29.640 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-18T15:08:29.740 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records in 2024-03-18T15:08:29.740 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records out 2024-03-18T15:08:29.740 INFO:teuthology.orchestra.run.smithi019.stderr:512 bytes copied, 0.000395963 s, 1.3 MB/s 2024-03-18T15:08:29.744 DEBUG:teuthology.orchestra.run.smithi019:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-18T15:08:29.802 DEBUG:teuthology.orchestra.run.smithi019:> stat /dev/vg_nvme/lv_2 2024-03-18T15:08:29.856 INFO:teuthology.orchestra.run.smithi019.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-18T15:08:29.856 INFO:teuthology.orchestra.run.smithi019.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T15:08:29.856 INFO:teuthology.orchestra.run.smithi019.stdout:Device: 5h/5d Inode: 652 Links: 1 2024-03-18T15:08:29.856 INFO:teuthology.orchestra.run.smithi019.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T15:08:29.856 INFO:teuthology.orchestra.run.smithi019.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T15:08:29.856 INFO:teuthology.orchestra.run.smithi019.stdout:Access: 2024-03-18 15:07:30.142206865 +0000 2024-03-18T15:08:29.856 INFO:teuthology.orchestra.run.smithi019.stdout:Modify: 2024-03-18 15:01:38.412150543 +0000 2024-03-18T15:08:29.856 INFO:teuthology.orchestra.run.smithi019.stdout:Change: 2024-03-18 15:01:38.412150543 +0000 2024-03-18T15:08:29.856 INFO:teuthology.orchestra.run.smithi019.stdout: Birth: 2024-03-18 15:01:38.412150543 +0000 2024-03-18T15:08:29.857 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-18T15:08:29.948 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records in 2024-03-18T15:08:29.948 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records out 2024-03-18T15:08:29.948 INFO:teuthology.orchestra.run.smithi019.stderr:512 bytes copied, 0.000377401 s, 1.4 MB/s 2024-03-18T15:08:29.954 DEBUG:teuthology.orchestra.run.smithi019:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-18T15:08:30.010 DEBUG:teuthology.orchestra.run.smithi019:> stat /dev/vg_nvme/lv_3 2024-03-18T15:08:30.062 INFO:teuthology.orchestra.run.smithi019.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-18T15:08:30.062 INFO:teuthology.orchestra.run.smithi019.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T15:08:30.062 INFO:teuthology.orchestra.run.smithi019.stdout:Device: 5h/5d Inode: 669 Links: 1 2024-03-18T15:08:30.063 INFO:teuthology.orchestra.run.smithi019.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T15:08:30.063 INFO:teuthology.orchestra.run.smithi019.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T15:08:30.063 INFO:teuthology.orchestra.run.smithi019.stdout:Access: 2024-03-18 15:07:30.233205931 +0000 2024-03-18T15:08:30.063 INFO:teuthology.orchestra.run.smithi019.stdout:Modify: 2024-03-18 15:01:38.991144671 +0000 2024-03-18T15:08:30.063 INFO:teuthology.orchestra.run.smithi019.stdout:Change: 2024-03-18 15:01:38.991144671 +0000 2024-03-18T15:08:30.063 INFO:teuthology.orchestra.run.smithi019.stdout: Birth: 2024-03-18 15:01:38.991144671 +0000 2024-03-18T15:08:30.063 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-18T15:08:30.145 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:29 smithi019 ceph-mon[31522]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:30.145 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:29 smithi019 ceph-mon[31522]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi019=x;smithi033=y;smithi104=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:08:30.145 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:29 smithi019 ceph-mon[31522]: Saving service mgr spec with placement smithi019=x;smithi033=y;smithi104=z;count:3 2024-03-18T15:08:30.145 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:29 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:30.146 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:29 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:30.146 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:29 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T15:08:30.146 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:29 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-18T15:08:30.146 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:29 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:08:30.156 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records in 2024-03-18T15:08:30.156 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records out 2024-03-18T15:08:30.156 INFO:teuthology.orchestra.run.smithi019.stderr:512 bytes copied, 0.000369457 s, 1.4 MB/s 2024-03-18T15:08:30.160 DEBUG:teuthology.orchestra.run.smithi019:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-18T15:08:30.221 DEBUG:teuthology.orchestra.run.smithi019:> stat /dev/vg_nvme/lv_4 2024-03-18T15:08:30.236 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:29 smithi104 ceph-mon[36070]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:30.236 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:29 smithi104 ceph-mon[36070]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi019=x;smithi033=y;smithi104=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:08:30.236 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:29 smithi104 ceph-mon[36070]: Saving service mgr spec with placement smithi019=x;smithi033=y;smithi104=z;count:3 2024-03-18T15:08:30.236 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:29 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:30.236 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:29 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:30.236 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:29 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T15:08:30.236 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:29 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-18T15:08:30.236 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:29 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:08:30.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:29 smithi033 ceph-mon[36648]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:30.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:29 smithi033 ceph-mon[36648]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi019=x;smithi033=y;smithi104=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:08:30.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:29 smithi033 ceph-mon[36648]: Saving service mgr spec with placement smithi019=x;smithi033=y;smithi104=z;count:3 2024-03-18T15:08:30.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:29 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:30.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:29 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:30.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:29 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T15:08:30.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:29 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-18T15:08:30.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:29 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:08:30.274 INFO:teuthology.orchestra.run.smithi019.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-18T15:08:30.274 INFO:teuthology.orchestra.run.smithi019.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T15:08:30.274 INFO:teuthology.orchestra.run.smithi019.stdout:Device: 5h/5d Inode: 688 Links: 1 2024-03-18T15:08:30.274 INFO:teuthology.orchestra.run.smithi019.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T15:08:30.274 INFO:teuthology.orchestra.run.smithi019.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T15:08:30.274 INFO:teuthology.orchestra.run.smithi019.stdout:Access: 2024-03-18 15:07:30.318205059 +0000 2024-03-18T15:08:30.274 INFO:teuthology.orchestra.run.smithi019.stdout:Modify: 2024-03-18 15:01:39.606138435 +0000 2024-03-18T15:08:30.274 INFO:teuthology.orchestra.run.smithi019.stdout:Change: 2024-03-18 15:01:39.606138435 +0000 2024-03-18T15:08:30.274 INFO:teuthology.orchestra.run.smithi019.stdout: Birth: 2024-03-18 15:01:39.606138435 +0000 2024-03-18T15:08:30.275 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-18T15:08:30.370 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records in 2024-03-18T15:08:30.370 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records out 2024-03-18T15:08:30.370 INFO:teuthology.orchestra.run.smithi019.stderr:512 bytes copied, 0.000464619 s, 1.1 MB/s 2024-03-18T15:08:30.375 DEBUG:teuthology.orchestra.run.smithi019:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-18T15:08:30.437 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-18T15:08:30.437 DEBUG:teuthology.orchestra.run.smithi033:> dd if=/scratch_devs of=/dev/stdout 2024-03-18T15:08:30.457 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-18T15:08:30.457 DEBUG:teuthology.orchestra.run.smithi033:> stat /dev/vg_nvme/lv_1 2024-03-18T15:08:30.516 INFO:teuthology.orchestra.run.smithi033.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-18T15:08:30.516 INFO:teuthology.orchestra.run.smithi033.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T15:08:30.516 INFO:teuthology.orchestra.run.smithi033.stdout:Device: 5h/5d Inode: 663 Links: 1 2024-03-18T15:08:30.516 INFO:teuthology.orchestra.run.smithi033.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T15:08:30.516 INFO:teuthology.orchestra.run.smithi033.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T15:08:30.516 INFO:teuthology.orchestra.run.smithi033.stdout:Access: 2024-03-18 15:08:16.375672053 +0000 2024-03-18T15:08:30.516 INFO:teuthology.orchestra.run.smithi033.stdout:Modify: 2024-03-18 15:00:29.282623230 +0000 2024-03-18T15:08:30.516 INFO:teuthology.orchestra.run.smithi033.stdout:Change: 2024-03-18 15:00:29.282623230 +0000 2024-03-18T15:08:30.516 INFO:teuthology.orchestra.run.smithi033.stdout: Birth: 2024-03-18 15:00:29.282623230 +0000 2024-03-18T15:08:30.516 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-18T15:08:30.590 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records in 2024-03-18T15:08:30.590 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records out 2024-03-18T15:08:30.590 INFO:teuthology.orchestra.run.smithi033.stderr:512 bytes copied, 0.00027314 s, 1.9 MB/s 2024-03-18T15:08:30.592 DEBUG:teuthology.orchestra.run.smithi033:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-18T15:08:30.651 DEBUG:teuthology.orchestra.run.smithi033:> stat /dev/vg_nvme/lv_2 2024-03-18T15:08:30.708 INFO:teuthology.orchestra.run.smithi033.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-18T15:08:30.708 INFO:teuthology.orchestra.run.smithi033.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T15:08:30.709 INFO:teuthology.orchestra.run.smithi033.stdout:Device: 5h/5d Inode: 682 Links: 1 2024-03-18T15:08:30.709 INFO:teuthology.orchestra.run.smithi033.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T15:08:30.709 INFO:teuthology.orchestra.run.smithi033.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T15:08:30.709 INFO:teuthology.orchestra.run.smithi033.stdout:Access: 2024-03-18 15:08:16.376672037 +0000 2024-03-18T15:08:30.709 INFO:teuthology.orchestra.run.smithi033.stdout:Modify: 2024-03-18 15:00:29.644617040 +0000 2024-03-18T15:08:30.709 INFO:teuthology.orchestra.run.smithi033.stdout:Change: 2024-03-18 15:00:29.644617040 +0000 2024-03-18T15:08:30.709 INFO:teuthology.orchestra.run.smithi033.stdout: Birth: 2024-03-18 15:00:29.644617040 +0000 2024-03-18T15:08:30.709 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-18T15:08:30.780 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records in 2024-03-18T15:08:30.780 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records out 2024-03-18T15:08:30.780 INFO:teuthology.orchestra.run.smithi033.stderr:512 bytes copied, 0.000191819 s, 2.7 MB/s 2024-03-18T15:08:30.781 DEBUG:teuthology.orchestra.run.smithi033:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-18T15:08:30.838 DEBUG:teuthology.orchestra.run.smithi033:> stat /dev/vg_nvme/lv_3 2024-03-18T15:08:30.894 INFO:teuthology.orchestra.run.smithi033.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-18T15:08:30.894 INFO:teuthology.orchestra.run.smithi033.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T15:08:30.895 INFO:teuthology.orchestra.run.smithi033.stdout:Device: 5h/5d Inode: 699 Links: 1 2024-03-18T15:08:30.895 INFO:teuthology.orchestra.run.smithi033.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T15:08:30.895 INFO:teuthology.orchestra.run.smithi033.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T15:08:30.895 INFO:teuthology.orchestra.run.smithi033.stdout:Access: 2024-03-18 15:08:16.376672037 +0000 2024-03-18T15:08:30.895 INFO:teuthology.orchestra.run.smithi033.stdout:Modify: 2024-03-18 15:00:30.001610935 +0000 2024-03-18T15:08:30.895 INFO:teuthology.orchestra.run.smithi033.stdout:Change: 2024-03-18 15:00:30.001610935 +0000 2024-03-18T15:08:30.895 INFO:teuthology.orchestra.run.smithi033.stdout: Birth: 2024-03-18 15:00:30.001610935 +0000 2024-03-18T15:08:30.895 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-18T15:08:30.965 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records in 2024-03-18T15:08:30.966 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records out 2024-03-18T15:08:30.966 INFO:teuthology.orchestra.run.smithi033.stderr:512 bytes copied, 0.000215814 s, 2.4 MB/s 2024-03-18T15:08:30.967 DEBUG:teuthology.orchestra.run.smithi033:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-18T15:08:31.024 DEBUG:teuthology.orchestra.run.smithi033:> stat /dev/vg_nvme/lv_4 2024-03-18T15:08:31.080 INFO:teuthology.orchestra.run.smithi033.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-18T15:08:31.080 INFO:teuthology.orchestra.run.smithi033.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T15:08:31.080 INFO:teuthology.orchestra.run.smithi033.stdout:Device: 5h/5d Inode: 721 Links: 1 2024-03-18T15:08:31.080 INFO:teuthology.orchestra.run.smithi033.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T15:08:31.080 INFO:teuthology.orchestra.run.smithi033.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T15:08:31.080 INFO:teuthology.orchestra.run.smithi033.stdout:Access: 2024-03-18 15:08:16.377672020 +0000 2024-03-18T15:08:31.080 INFO:teuthology.orchestra.run.smithi033.stdout:Modify: 2024-03-18 15:00:30.362604762 +0000 2024-03-18T15:08:31.080 INFO:teuthology.orchestra.run.smithi033.stdout:Change: 2024-03-18 15:00:30.362604762 +0000 2024-03-18T15:08:31.080 INFO:teuthology.orchestra.run.smithi033.stdout: Birth: 2024-03-18 15:00:30.362604762 +0000 2024-03-18T15:08:31.081 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-18T15:08:31.150 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records in 2024-03-18T15:08:31.151 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records out 2024-03-18T15:08:31.151 INFO:teuthology.orchestra.run.smithi033.stderr:512 bytes copied, 0.000193523 s, 2.6 MB/s 2024-03-18T15:08:31.152 DEBUG:teuthology.orchestra.run.smithi033:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-18T15:08:31.209 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-03-18T15:08:31.209 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/scratch_devs of=/dev/stdout 2024-03-18T15:08:31.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:30 smithi019 ceph-mon[31522]: Reconfiguring mon.c (monmap changed)... 2024-03-18T15:08:31.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:30 smithi019 ceph-mon[31522]: Reconfiguring daemon mon.c on smithi104 2024-03-18T15:08:31.223 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-18T15:08:31.223 DEBUG:teuthology.orchestra.run.smithi104:> stat /dev/vg_nvme/lv_1 2024-03-18T15:08:31.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:30 smithi033 ceph-mon[36648]: Reconfiguring mon.c (monmap changed)... 2024-03-18T15:08:31.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:30 smithi033 ceph-mon[36648]: Reconfiguring daemon mon.c on smithi104 2024-03-18T15:08:31.278 INFO:teuthology.orchestra.run.smithi104.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-18T15:08:31.278 INFO:teuthology.orchestra.run.smithi104.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T15:08:31.278 INFO:teuthology.orchestra.run.smithi104.stdout:Device: 5h/5d Inode: 664 Links: 1 2024-03-18T15:08:31.278 INFO:teuthology.orchestra.run.smithi104.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T15:08:31.278 INFO:teuthology.orchestra.run.smithi104.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T15:08:31.278 INFO:teuthology.orchestra.run.smithi104.stdout:Access: 2024-03-18 15:08:06.890986615 +0000 2024-03-18T15:08:31.279 INFO:teuthology.orchestra.run.smithi104.stdout:Modify: 2024-03-18 15:00:15.118552845 +0000 2024-03-18T15:08:31.279 INFO:teuthology.orchestra.run.smithi104.stdout:Change: 2024-03-18 15:00:15.118552845 +0000 2024-03-18T15:08:31.279 INFO:teuthology.orchestra.run.smithi104.stdout: Birth: 2024-03-18 15:00:15.118552845 +0000 2024-03-18T15:08:31.279 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-18T15:08:31.342 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:30 smithi104 ceph-mon[36070]: Reconfiguring mon.c (monmap changed)... 2024-03-18T15:08:31.342 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:30 smithi104 ceph-mon[36070]: Reconfiguring daemon mon.c on smithi104 2024-03-18T15:08:31.345 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records in 2024-03-18T15:08:31.345 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records out 2024-03-18T15:08:31.345 INFO:teuthology.orchestra.run.smithi104.stderr:512 bytes copied, 0.000197062 s, 2.6 MB/s 2024-03-18T15:08:31.346 DEBUG:teuthology.orchestra.run.smithi104:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-18T15:08:31.402 DEBUG:teuthology.orchestra.run.smithi104:> stat /dev/vg_nvme/lv_2 2024-03-18T15:08:31.456 INFO:teuthology.orchestra.run.smithi104.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-18T15:08:31.456 INFO:teuthology.orchestra.run.smithi104.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T15:08:31.456 INFO:teuthology.orchestra.run.smithi104.stdout:Device: 5h/5d Inode: 680 Links: 1 2024-03-18T15:08:31.456 INFO:teuthology.orchestra.run.smithi104.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T15:08:31.456 INFO:teuthology.orchestra.run.smithi104.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T15:08:31.456 INFO:teuthology.orchestra.run.smithi104.stdout:Access: 2024-03-18 15:08:06.890986615 +0000 2024-03-18T15:08:31.456 INFO:teuthology.orchestra.run.smithi104.stdout:Modify: 2024-03-18 15:00:15.455548750 +0000 2024-03-18T15:08:31.457 INFO:teuthology.orchestra.run.smithi104.stdout:Change: 2024-03-18 15:00:15.455548750 +0000 2024-03-18T15:08:31.457 INFO:teuthology.orchestra.run.smithi104.stdout: Birth: 2024-03-18 15:00:15.455548750 +0000 2024-03-18T15:08:31.457 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-18T15:08:31.521 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records in 2024-03-18T15:08:31.521 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records out 2024-03-18T15:08:31.521 INFO:teuthology.orchestra.run.smithi104.stderr:512 bytes copied, 0.000174319 s, 2.9 MB/s 2024-03-18T15:08:31.523 DEBUG:teuthology.orchestra.run.smithi104:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-18T15:08:31.577 DEBUG:teuthology.orchestra.run.smithi104:> stat /dev/vg_nvme/lv_3 2024-03-18T15:08:31.634 INFO:teuthology.orchestra.run.smithi104.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-18T15:08:31.634 INFO:teuthology.orchestra.run.smithi104.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T15:08:31.634 INFO:teuthology.orchestra.run.smithi104.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-03-18T15:08:31.634 INFO:teuthology.orchestra.run.smithi104.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T15:08:31.634 INFO:teuthology.orchestra.run.smithi104.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T15:08:31.634 INFO:teuthology.orchestra.run.smithi104.stdout:Access: 2024-03-18 15:08:06.891986603 +0000 2024-03-18T15:08:31.634 INFO:teuthology.orchestra.run.smithi104.stdout:Modify: 2024-03-18 15:00:15.804544510 +0000 2024-03-18T15:08:31.634 INFO:teuthology.orchestra.run.smithi104.stdout:Change: 2024-03-18 15:00:15.804544510 +0000 2024-03-18T15:08:31.634 INFO:teuthology.orchestra.run.smithi104.stdout: Birth: 2024-03-18 15:00:15.804544510 +0000 2024-03-18T15:08:31.635 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-18T15:08:31.700 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records in 2024-03-18T15:08:31.700 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records out 2024-03-18T15:08:31.701 INFO:teuthology.orchestra.run.smithi104.stderr:512 bytes copied, 0.000210816 s, 2.4 MB/s 2024-03-18T15:08:31.702 DEBUG:teuthology.orchestra.run.smithi104:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-18T15:08:31.757 DEBUG:teuthology.orchestra.run.smithi104:> stat /dev/vg_nvme/lv_4 2024-03-18T15:08:31.812 INFO:teuthology.orchestra.run.smithi104.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-18T15:08:31.812 INFO:teuthology.orchestra.run.smithi104.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T15:08:31.812 INFO:teuthology.orchestra.run.smithi104.stdout:Device: 5h/5d Inode: 720 Links: 1 2024-03-18T15:08:31.812 INFO:teuthology.orchestra.run.smithi104.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T15:08:31.812 INFO:teuthology.orchestra.run.smithi104.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T15:08:31.812 INFO:teuthology.orchestra.run.smithi104.stdout:Access: 2024-03-18 15:08:06.891986603 +0000 2024-03-18T15:08:31.812 INFO:teuthology.orchestra.run.smithi104.stdout:Modify: 2024-03-18 15:00:16.139540440 +0000 2024-03-18T15:08:31.812 INFO:teuthology.orchestra.run.smithi104.stdout:Change: 2024-03-18 15:00:16.139540440 +0000 2024-03-18T15:08:31.812 INFO:teuthology.orchestra.run.smithi104.stdout: Birth: 2024-03-18 15:00:16.139540440 +0000 2024-03-18T15:08:31.813 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-18T15:08:31.877 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records in 2024-03-18T15:08:31.877 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records out 2024-03-18T15:08:31.877 INFO:teuthology.orchestra.run.smithi104.stderr:512 bytes copied, 0.000183787 s, 2.8 MB/s 2024-03-18T15:08:31.878 DEBUG:teuthology.orchestra.run.smithi104:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-18T15:08:31.934 INFO:tasks.cephadm:Deploying osd.0 on smithi019 with /dev/vg_nvme/lv_4... 2024-03-18T15:08:31.934 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-18T15:08:32.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:31 smithi019 ceph-mon[31522]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:32.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:31 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:32.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:31 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:32.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:31 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:08:32.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:31 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:08:32.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:31 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:08:32.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:31 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:32.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:31 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-18T15:08:32.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:31 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-18T15:08:32.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:31 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-18T15:08:32.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:31 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:08:32.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:31 smithi033 ceph-mon[36648]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:32.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:31 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:32.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:31 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:32.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:31 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:08:32.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:31 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:08:32.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:31 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:08:32.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:31 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:32.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:31 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-18T15:08:32.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:31 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-18T15:08:32.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:31 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-18T15:08:32.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:31 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:08:32.393 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:08:32.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:31 smithi104 ceph-mon[36070]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:32.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:31 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:32.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:31 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:32.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:31 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:08:32.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:31 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:08:32.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:31 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:08:32.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:31 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:32.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:31 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-18T15:08:32.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:31 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-18T15:08:32.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:31 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-18T15:08:32.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:31 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:08:33.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:32 smithi104 ceph-mon[36070]: Deploying daemon mgr.z on smithi104 2024-03-18T15:08:33.195 INFO:journalctl@ceph.mgr.z.smithi104.stdout:Mar 18 15:08:33 smithi104 systemd[1]: Starting Ceph mgr.z for d4dc2ba6-e538-11ee-95c9-87774f69a715... 2024-03-18T15:08:33.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:32 smithi019 ceph-mon[31522]: Deploying daemon mgr.z on smithi104 2024-03-18T15:08:33.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:32 smithi033 ceph-mon[36648]: Deploying daemon mgr.z on smithi104 2024-03-18T15:08:33.570 INFO:journalctl@ceph.mgr.z.smithi104.stdout:Mar 18 15:08:33 smithi104 podman[37921]: 2024-03-18 15:08:33.168926634 +0000 UTC m=+0.017320181 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T15:08:33.570 INFO:journalctl@ceph.mgr.z.smithi104.stdout:Mar 18 15:08:33 smithi104 podman[37921]: 2024-03-18 15:08:33.275876385 +0000 UTC m=+0.124269930 container create 363a7311517323a28c3c4befeea29ecdaf790735c0ccef1a57c1f5964b89a784 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-z, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-03-18T15:08:33.570 INFO:journalctl@ceph.mgr.z.smithi104.stdout:Mar 18 15:08:33 smithi104 podman[37921]: 2024-03-18 15:08:33.486691028 +0000 UTC m=+0.335084568 container init 363a7311517323a28c3c4befeea29ecdaf790735c0ccef1a57c1f5964b89a784 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-z, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T15:08:33.570 INFO:journalctl@ceph.mgr.z.smithi104.stdout:Mar 18 15:08:33 smithi104 podman[37921]: 2024-03-18 15:08:33.489265171 +0000 UTC m=+0.337658710 container start 363a7311517323a28c3c4befeea29ecdaf790735c0ccef1a57c1f5964b89a784 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-z, GIT_CLEAN=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T15:08:33.570 INFO:journalctl@ceph.mgr.z.smithi104.stdout:Mar 18 15:08:33 smithi104 ceph-mgr[37940]: -- 172.21.15.104:0/3614897059 <== mon.1 v2:172.21.15.104:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d93df625a0 con 0x55d93df37000 2024-03-18T15:08:33.570 INFO:journalctl@ceph.mgr.z.smithi104.stdout:Mar 18 15:08:33 smithi104 bash[37921]: 363a7311517323a28c3c4befeea29ecdaf790735c0ccef1a57c1f5964b89a784 2024-03-18T15:08:33.944 INFO:journalctl@ceph.mgr.z.smithi104.stdout:Mar 18 15:08:33 smithi104 systemd[1]: Started Ceph mgr.z for d4dc2ba6-e538-11ee-95c9-87774f69a715. 2024-03-18T15:08:34.238 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:33 smithi019 ceph-mon[31522]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:34.238 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:33 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:34.238 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:33 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:34.238 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:33 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:34.238 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:33 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-18T15:08:34.238 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:33 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-18T15:08:34.238 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:33 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-18T15:08:34.238 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:33 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:08:34.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:33 smithi033 ceph-mon[36648]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:34.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:33 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:34.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:33 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:34.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:33 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:34.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:33 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-18T15:08:34.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:33 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-18T15:08:34.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:33 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-18T15:08:34.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:33 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:08:34.444 INFO:journalctl@ceph.mgr.z.smithi104.stdout:Mar 18 15:08:34 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-z[37936]: 2024-03-18T15:08:34.020+0000 7fa823e7e200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T15:08:34.444 INFO:journalctl@ceph.mgr.z.smithi104.stdout:Mar 18 15:08:34 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-z[37936]: 2024-03-18T15:08:34.110+0000 7fa823e7e200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T15:08:34.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:33 smithi104 ceph-mon[36070]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:34.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:33 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:34.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:33 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:34.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:33 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:34.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:33 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-18T15:08:34.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:33 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-18T15:08:34.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:33 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-18T15:08:34.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:33 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:08:35.051 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:34 smithi104 ceph-mon[36070]: Deploying daemon mgr.y on smithi033 2024-03-18T15:08:35.052 INFO:journalctl@ceph.mgr.z.smithi104.stdout:Mar 18 15:08:34 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-z[37936]: 2024-03-18T15:08:34.733+0000 7fa823e7e200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T15:08:35.052 INFO:journalctl@ceph.mgr.z.smithi104.stdout:Mar 18 15:08:34 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-z[37936]: 2024-03-18T15:08:34.821+0000 7fa823e7e200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T15:08:35.052 INFO:journalctl@ceph.mgr.z.smithi104.stdout:Mar 18 15:08:34 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-z[37936]: 2024-03-18T15:08:34.958+0000 7fa823e7e200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T15:08:35.239 INFO:teuthology.orchestra.run.smithi019.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-18T15:08:35.239 INFO:teuthology.orchestra.run.smithi019.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-18T15:08:35.239 INFO:teuthology.orchestra.run.smithi019.stderr: stderr: 10+0 records in 2024-03-18T15:08:35.239 INFO:teuthology.orchestra.run.smithi019.stderr:10+0 records out 2024-03-18T15:08:35.239 INFO:teuthology.orchestra.run.smithi019.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0369392 s, 284 MB/s 2024-03-18T15:08:35.239 INFO:teuthology.orchestra.run.smithi019.stderr:--> Zapping successful for: 2024-03-18T15:08:35.239 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:34 smithi019 ceph-mon[31522]: Deploying daemon mgr.y on smithi033 2024-03-18T15:08:35.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:34 smithi033 ceph-mon[36648]: Deploying daemon mgr.y on smithi033 2024-03-18T15:08:35.386 INFO:journalctl@ceph.mgr.z.smithi104.stdout:Mar 18 15:08:35 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-z[37936]: 2024-03-18T15:08:35.051+0000 7fa823e7e200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T15:08:35.386 INFO:journalctl@ceph.mgr.z.smithi104.stdout:Mar 18 15:08:35 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-z[37936]: 2024-03-18T15:08:35.131+0000 7fa823e7e200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T15:08:35.386 INFO:journalctl@ceph.mgr.z.smithi104.stdout:Mar 18 15:08:35 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-z[37936]: 2024-03-18T15:08:35.385+0000 7fa823e7e200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T15:08:35.645 INFO:journalctl@ceph.mgr.z.smithi104.stdout:Mar 18 15:08:35 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-z[37936]: 2024-03-18T15:08:35.492+0000 7fa823e7e200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T15:08:35.645 INFO:journalctl@ceph.mgr.z.smithi104.stdout:Mar 18 15:08:35 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-z[37936]: 2024-03-18T15:08:35.573+0000 7fa823e7e200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T15:08:35.944 INFO:journalctl@ceph.mgr.z.smithi104.stdout:Mar 18 15:08:35 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-z[37936]: 2024-03-18T15:08:35.666+0000 7fa823e7e200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T15:08:35.944 INFO:journalctl@ceph.mgr.z.smithi104.stdout:Mar 18 15:08:35 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-z[37936]: 2024-03-18T15:08:35.836+0000 7fa823e7e200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T15:08:35.958 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Mar 18 15:08:35 smithi033 systemd[1]: Starting Ceph mgr.y for d4dc2ba6-e538-11ee-95c9-87774f69a715... 2024-03-18T15:08:35.969 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi019:vg_nvme/lv_4 2024-03-18T15:08:36.212 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:35 smithi033 ceph-mon[36648]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:36.213 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Mar 18 15:08:35 smithi033 podman[38019]: 2024-03-18 15:08:35.95847903 +0000 UTC m=+0.105096141 container create 1946eb032c4c4d1671275b5e16e8d207923313bb123046370624ba414b31bdaf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-y, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307) 2024-03-18T15:08:36.213 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Mar 18 15:08:35 smithi033 podman[38019]: 2024-03-18 15:08:35.871419877 +0000 UTC m=+0.018036961 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T15:08:36.213 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Mar 18 15:08:36 smithi033 podman[38019]: 2024-03-18 15:08:36.185761635 +0000 UTC m=+0.332378711 container init 1946eb032c4c4d1671275b5e16e8d207923313bb123046370624ba414b31bdaf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-y, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2) 2024-03-18T15:08:36.213 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Mar 18 15:08:36 smithi033 podman[38019]: 2024-03-18 15:08:36.18836988 +0000 UTC m=+0.334986952 container start 1946eb032c4c4d1671275b5e16e8d207923313bb123046370624ba414b31bdaf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-y, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-18T15:08:36.213 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Mar 18 15:08:36 smithi033 ceph-mgr[38038]: -- 172.21.15.33:0/3495922822 <== mon.2 v2:172.21.15.33:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e9705445a0 con 0x55e970518800 2024-03-18T15:08:36.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:35 smithi019 ceph-mon[31522]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:36.421 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:08:36.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:35 smithi104 ceph-mon[36070]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:36.506 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Mar 18 15:08:36 smithi033 bash[38019]: 1946eb032c4c4d1671275b5e16e8d207923313bb123046370624ba414b31bdaf 2024-03-18T15:08:36.506 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Mar 18 15:08:36 smithi033 systemd[1]: Started Ceph mgr.y for d4dc2ba6-e538-11ee-95c9-87774f69a715. 2024-03-18T15:08:37.006 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Mar 18 15:08:36 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-y[38034]: 2024-03-18T15:08:36.751+0000 7f263983b200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T15:08:37.007 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Mar 18 15:08:36 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-y[38034]: 2024-03-18T15:08:36.851+0000 7f263983b200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T15:08:37.194 INFO:journalctl@ceph.mgr.z.smithi104.stdout:Mar 18 15:08:36 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-z[37936]: 2024-03-18T15:08:36.797+0000 7fa823e7e200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T15:08:37.194 INFO:journalctl@ceph.mgr.z.smithi104.stdout:Mar 18 15:08:36 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-z[37936]: 2024-03-18T15:08:36.928+0000 7fa823e7e200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T15:08:37.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:37 smithi019 ceph-mon[31522]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:37.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:37 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:37.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:37 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:37.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:37 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:37.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:37 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:37.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:37 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:08:37.570 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:37 smithi104 ceph-mon[36070]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:37.570 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:37 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:37.570 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:37 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:37.570 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:37 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:37.571 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:37 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:37.571 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:37 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:08:37.571 INFO:journalctl@ceph.mgr.z.smithi104.stdout:Mar 18 15:08:37 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-z[37936]: 2024-03-18T15:08:37.288+0000 7fa823e7e200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T15:08:37.571 INFO:journalctl@ceph.mgr.z.smithi104.stdout:Mar 18 15:08:37 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-z[37936]: 2024-03-18T15:08:37.375+0000 7fa823e7e200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T15:08:37.571 INFO:journalctl@ceph.mgr.z.smithi104.stdout:Mar 18 15:08:37 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-z[37936]: 2024-03-18T15:08:37.569+0000 7fa823e7e200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T15:08:37.596 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Mar 18 15:08:37 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-y[38034]: 2024-03-18T15:08:37.545+0000 7f263983b200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T15:08:37.596 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:37 smithi033 ceph-mon[36648]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:37.597 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:37 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:37.597 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:37 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:37.597 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:37 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:37.597 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:37 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:37.597 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:37 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:08:37.896 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Mar 18 15:08:37 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-y[38034]: 2024-03-18T15:08:37.643+0000 7f263983b200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T15:08:37.897 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Mar 18 15:08:37 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-y[38034]: 2024-03-18T15:08:37.797+0000 7f263983b200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T15:08:37.897 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Mar 18 15:08:37 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-y[38034]: 2024-03-18T15:08:37.896+0000 7f263983b200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T15:08:37.944 INFO:journalctl@ceph.mgr.z.smithi104.stdout:Mar 18 15:08:37 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-z[37936]: 2024-03-18T15:08:37.758+0000 7fa823e7e200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T15:08:38.256 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Mar 18 15:08:37 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-y[38034]: 2024-03-18T15:08:37.989+0000 7f263983b200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T15:08:38.301 INFO:journalctl@ceph.mgr.z.smithi104.stdout:Mar 18 15:08:38 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-z[37936]: 2024-03-18T15:08:38.043+0000 7fa823e7e200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T15:08:38.327 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:38.326+0000 7f4507e00700 1 -- 172.21.15.19:0/1241955558 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4500101ea0 msgr2=0x7f4500102320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:08:38.328 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:38.326+0000 7f4507e00700 1 --2- 172.21.15.19:0/1241955558 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4500101ea0 0x7f4500102320 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f44f800b0a0 tx=0x7f44f800b3b0 comp rx=0 tx=0).stop 2024-03-18T15:08:38.328 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:38.326+0000 7f4507e00700 1 -- 172.21.15.19:0/1241955558 shutdown_connections 2024-03-18T15:08:38.328 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:38.326+0000 7f4507e00700 1 --2- 172.21.15.19:0/1241955558 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4500100b80 0x7f4500100fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:08:38.328 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:38.326+0000 7f4507e00700 1 --2- 172.21.15.19:0/1241955558 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4500101ea0 0x7f4500102320 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:08:38.328 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:38.326+0000 7f4507e00700 1 --2- 172.21.15.19:0/1241955558 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f45001014e0 0x7f4500101960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:08:38.328 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:38.326+0000 7f4507e00700 1 -- 172.21.15.19:0/1241955558 >> 172.21.15.19:0/1241955558 conn(0x7f45000fc7a0 msgr2=0x7f45000febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:08:38.328 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:38.326+0000 7f4507e00700 1 -- 172.21.15.19:0/1241955558 shutdown_connections 2024-03-18T15:08:38.328 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:38.326+0000 7f4507e00700 1 -- 172.21.15.19:0/1241955558 wait complete. 2024-03-18T15:08:38.329 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:38.327+0000 7f4507e00700 1 Processor -- start 2024-03-18T15:08:38.329 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:38.327+0000 7f4507e00700 1 -- start start 2024-03-18T15:08:38.329 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:38.327+0000 7f4507e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4500100b80 0x7f450019a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:08:38.329 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:38.327+0000 7f4507e00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f45001014e0 0x7f450019ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:08:38.329 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:38.328+0000 7f4507e00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4500101ea0 0x7f45001a1cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:08:38.329 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:38.328+0000 7f4507e00700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f4500110100 con 0x7f45001014e0 2024-03-18T15:08:38.330 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:38.328+0000 7f4507e00700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f450010ff80 con 0x7f4500100b80 2024-03-18T15:08:38.330 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:38.328+0000 7f4507e00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f4500110280 con 0x7f4500101ea0 2024-03-18T15:08:38.330 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:38.328+0000 7f4505400700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4500100b80 0x7f450019a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:08:38.330 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:38.328+0000 7f4504a00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f45001014e0 0x7f450019ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:08:38.330 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:38.328+0000 7f4505400700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4500100b80 0x7f450019a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:35048/0 (socket says 172.21.15.19:35048) 2024-03-18T15:08:38.330 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:38.328+0000 7f4505400700 1 -- 172.21.15.19:0/3766803104 learned_addr learned my addr 172.21.15.19:0/3766803104 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:08:38.330 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:38.328+0000 7f4505e00700 1 --2- 172.21.15.19:0/3766803104 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4500101ea0 0x7f45001a1cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:08:38.331 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:38.328+0000 7f4505400700 1 -- 172.21.15.19:0/3766803104 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4500101ea0 msgr2=0x7f45001a1cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:08:38.331 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:38.328+0000 7f4505400700 1 --2- 172.21.15.19:0/3766803104 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4500101ea0 0x7f45001a1cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:08:38.331 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:38.328+0000 7f4505400700 1 -- 172.21.15.19:0/3766803104 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f45001014e0 msgr2=0x7f450019ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:08:38.331 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:38.328+0000 7f4505400700 1 --2- 172.21.15.19:0/3766803104 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f45001014e0 0x7f450019ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:08:38.331 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:38.328+0000 7f4505400700 1 -- 172.21.15.19:0/3766803104 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45001a2370 con 0x7f4500100b80 2024-03-18T15:08:38.331 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:38.328+0000 7f4505e00700 1 --2- 172.21.15.19:0/3766803104 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4500101ea0 0x7f45001a1cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:08:38.331 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:38.329+0000 7f4505400700 1 --2- 172.21.15.19:0/3766803104 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4500100b80 0x7f450019a710 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f44f400eb30 tx=0x7f44f400ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:08:38.331 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:38.329+0000 7f44fe000700 1 -- 172.21.15.19:0/3766803104 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44f400cbd0 con 0x7f4500100b80 2024-03-18T15:08:38.332 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:38.329+0000 7f44fe000700 1 -- 172.21.15.19:0/3766803104 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f44f4004500 con 0x7f4500100b80 2024-03-18T15:08:38.332 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:38.329+0000 7f4507e00700 1 -- 172.21.15.19:0/3766803104 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45001a2660 con 0x7f4500100b80 2024-03-18T15:08:38.332 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:38.329+0000 7f44fe000700 1 -- 172.21.15.19:0/3766803104 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44f4010430 con 0x7f4500100b80 2024-03-18T15:08:38.332 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:38.329+0000 7f4507e00700 1 -- 172.21.15.19:0/3766803104 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45001a2bf0 con 0x7f4500100b80 2024-03-18T15:08:38.333 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:38.330+0000 7f4507e00700 1 -- 172.21.15.19:0/3766803104 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4500068f90 con 0x7f4500100b80 2024-03-18T15:08:38.333 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:38.331+0000 7f44fe000700 1 -- 172.21.15.19:0/3766803104 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f44f40105d0 con 0x7f4500100b80 2024-03-18T15:08:38.333 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:38.331+0000 7f44fe000700 1 --2- 172.21.15.19:0/3766803104 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f44e8039200 0x7f44e803b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:08:38.333 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:38.331+0000 7f44fe000700 1 -- 172.21.15.19:0/3766803104 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f44f404c470 con 0x7f4500100b80 2024-03-18T15:08:38.333 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:38.332+0000 7f4504a00700 1 --2- 172.21.15.19:0/3766803104 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f44e8039200 0x7f44e803b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:08:38.333 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:38.332+0000 7f4504a00700 1 --2- 172.21.15.19:0/3766803104 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f44e8039200 0x7f44e803b6c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f44ec009c60 tx=0x7f44ec009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:08:38.338 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:38.337+0000 7f44fe000700 1 -- 172.21.15.19:0/3766803104 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f44f401fba0 con 0x7f4500100b80 2024-03-18T15:08:38.562 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Mar 18 15:08:38 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-y[38034]: 2024-03-18T15:08:38.263+0000 7f263983b200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T15:08:38.562 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Mar 18 15:08:38 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-y[38034]: 2024-03-18T15:08:38.377+0000 7f263983b200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T15:08:38.563 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Mar 18 15:08:38 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-y[38034]: 2024-03-18T15:08:38.464+0000 7f263983b200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T15:08:38.563 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Mar 18 15:08:38 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-y[38034]: 2024-03-18T15:08:38.561+0000 7f263983b200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T15:08:38.585 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:38.584+0000 7f4507e00700 1 -- 172.21.15.19:0/3766803104 --> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi019:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f4500062710 con 0x7f44e8039200 2024-03-18T15:08:38.694 INFO:journalctl@ceph.mgr.z.smithi104.stdout:Mar 18 15:08:38 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-z[37936]: 2024-03-18T15:08:38.300+0000 7fa823e7e200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T15:08:38.694 INFO:journalctl@ceph.mgr.z.smithi104.stdout:Mar 18 15:08:38 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-z[37936]: 2024-03-18T15:08:38.534+0000 7fa823e7e200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T15:08:39.006 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Mar 18 15:08:38 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-y[38034]: 2024-03-18T15:08:38.743+0000 7f263983b200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T15:08:39.267 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:39 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:39.267 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:39 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:39.268 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:39 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:39.268 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:39 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:39.268 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:39 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:08:39.268 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:39 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:08:39.268 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:39 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:39.268 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:39 smithi019 ceph-mon[31522]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:39.268 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:39 smithi019 ceph-mon[31522]: Reconfiguring mgr.x (unknown last config time)... 2024-03-18T15:08:39.268 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:39 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-18T15:08:39.268 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:39 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-18T15:08:39.268 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:39 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:08:39.268 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:39 smithi019 ceph-mon[31522]: Reconfiguring daemon mgr.x on smithi019 2024-03-18T15:08:39.268 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:39 smithi019 ceph-mon[31522]: from='client.24116 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi019:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:08:39.268 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:39 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T15:08:39.268 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:39 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T15:08:39.269 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:39 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:08:39.327 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:39 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:39.327 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:39 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:39.327 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:39 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:39.327 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:39 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:39.327 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:39 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:08:39.327 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:39 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:08:39.327 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:39 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:39.327 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:39 smithi104 ceph-mon[36070]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:39.327 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:39 smithi104 ceph-mon[36070]: Reconfiguring mgr.x (unknown last config time)... 2024-03-18T15:08:39.328 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:39 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-18T15:08:39.328 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:39 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-18T15:08:39.328 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:39 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:08:39.328 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:39 smithi104 ceph-mon[36070]: Reconfiguring daemon mgr.x on smithi019 2024-03-18T15:08:39.328 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:39 smithi104 ceph-mon[36070]: from='client.24116 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi019:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:08:39.328 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:39 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T15:08:39.328 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:39 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T15:08:39.328 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:39 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:08:39.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:39 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:39.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:39 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:39.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:39 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:39.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:39 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:39.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:39 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:08:39.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:39 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:08:39.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:39 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:39.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:39 smithi033 ceph-mon[36648]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:39.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:39 smithi033 ceph-mon[36648]: Reconfiguring mgr.x (unknown last config time)... 2024-03-18T15:08:39.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:39 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-18T15:08:39.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:39 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-18T15:08:39.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:39 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:08:39.508 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:39 smithi033 ceph-mon[36648]: Reconfiguring daemon mgr.x on smithi019 2024-03-18T15:08:39.508 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:39 smithi033 ceph-mon[36648]: from='client.24116 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi019:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:08:39.508 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:39 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T15:08:39.508 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:39 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T15:08:39.508 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:39 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:08:39.694 INFO:journalctl@ceph.mgr.z.smithi104.stdout:Mar 18 15:08:39 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-z[37936]: 2024-03-18T15:08:39.326+0000 7fa823e7e200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T15:08:40.194 INFO:journalctl@ceph.mgr.z.smithi104.stdout:Mar 18 15:08:39 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-z[37936]: 2024-03-18T15:08:39.915+0000 7fa823e7e200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T15:08:40.194 INFO:journalctl@ceph.mgr.z.smithi104.stdout:Mar 18 15:08:40 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-z[37936]: 2024-03-18T15:08:40.011+0000 7fa823e7e200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T15:08:40.235 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Mar 18 15:08:39 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-y[38034]: 2024-03-18T15:08:39.765+0000 7f263983b200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T15:08:40.235 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Mar 18 15:08:39 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-y[38034]: 2024-03-18T15:08:39.900+0000 7f263983b200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T15:08:40.405 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:40 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:40.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:40 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:40.507 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Mar 18 15:08:40 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-y[38034]: 2024-03-18T15:08:40.245+0000 7f263983b200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T15:08:40.507 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Mar 18 15:08:40 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-y[38034]: 2024-03-18T15:08:40.337+0000 7f263983b200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T15:08:40.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:40 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:40.694 INFO:journalctl@ceph.mgr.z.smithi104.stdout:Mar 18 15:08:40 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-z[37936]: 2024-03-18T15:08:40.223+0000 7fa823e7e200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T15:08:41.006 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Mar 18 15:08:40 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-y[38034]: 2024-03-18T15:08:40.538+0000 7f263983b200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T15:08:41.006 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Mar 18 15:08:40 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-y[38034]: 2024-03-18T15:08:40.727+0000 7f263983b200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T15:08:41.256 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:41.254+0000 7f44fe000700 1 -- 172.21.15.19:0/3766803104 <== mon.2 v2:172.21.15.33:3300/0 7 ==== mgrmap(e 14) ==== 91698+0+0 (secure 0 0 0) 0x7f44f4017cd0 con 0x7f4500100b80 2024-03-18T15:08:41.288 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Mar 18 15:08:41 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-y[38034]: 2024-03-18T15:08:41.018+0000 7f263983b200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T15:08:41.288 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Mar 18 15:08:41 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-y[38034]: 2024-03-18T15:08:41.287+0000 7f263983b200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T15:08:41.288 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:41 smithi033 ceph-mon[36648]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:41.289 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:41 smithi033 ceph-mon[36648]: from='mgr.? 172.21.15.104:0/931881435' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-18T15:08:41.289 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:41 smithi033 ceph-mon[36648]: Standby manager daemon z started 2024-03-18T15:08:41.289 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:41 smithi033 ceph-mon[36648]: from='mgr.? 172.21.15.104:0/931881435' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-18T15:08:41.289 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:41 smithi033 ceph-mon[36648]: from='mgr.? 172.21.15.104:0/931881435' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-18T15:08:41.289 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:41 smithi033 ceph-mon[36648]: from='mgr.? 172.21.15.104:0/931881435' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-18T15:08:41.289 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:41 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:41.289 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:41 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:41.289 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:41 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:08:41.289 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:41 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:08:41.289 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:41 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:08:41.289 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:41 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:41.428 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:41 smithi019 ceph-mon[31522]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:41.428 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:41 smithi019 ceph-mon[31522]: from='mgr.? 172.21.15.104:0/931881435' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-18T15:08:41.429 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:41 smithi019 ceph-mon[31522]: Standby manager daemon z started 2024-03-18T15:08:41.429 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:41 smithi019 ceph-mon[31522]: from='mgr.? 172.21.15.104:0/931881435' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-18T15:08:41.429 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:41 smithi019 ceph-mon[31522]: from='mgr.? 172.21.15.104:0/931881435' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-18T15:08:41.429 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:41 smithi019 ceph-mon[31522]: from='mgr.? 172.21.15.104:0/931881435' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-18T15:08:41.429 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:41 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:41.429 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:41 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:41.429 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:41 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:08:41.429 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:41 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:08:41.429 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:41 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:08:41.429 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:41 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:41.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:41 smithi104 ceph-mon[36070]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:41.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:41 smithi104 ceph-mon[36070]: from='mgr.? 172.21.15.104:0/931881435' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-18T15:08:41.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:41 smithi104 ceph-mon[36070]: Standby manager daemon z started 2024-03-18T15:08:41.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:41 smithi104 ceph-mon[36070]: from='mgr.? 172.21.15.104:0/931881435' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-18T15:08:41.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:41 smithi104 ceph-mon[36070]: from='mgr.? 172.21.15.104:0/931881435' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-18T15:08:41.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:41 smithi104 ceph-mon[36070]: from='mgr.? 172.21.15.104:0/931881435' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-18T15:08:41.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:41 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:41.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:41 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:41.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:41 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:08:41.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:41 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:08:41.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:41 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:08:41.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:41 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:08:41.756 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Mar 18 15:08:41 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-y[38034]: 2024-03-18T15:08:41.530+0000 7f263983b200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T15:08:42.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:42 smithi019 ceph-mon[31522]: mgrmap e14: x(active, since 87s), standbys: z 2024-03-18T15:08:42.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:42 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-18T15:08:42.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:42 smithi104 ceph-mon[36070]: mgrmap e14: x(active, since 87s), standbys: z 2024-03-18T15:08:42.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:42 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-18T15:08:42.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:42 smithi033 ceph-mon[36648]: mgrmap e14: x(active, since 87s), standbys: z 2024-03-18T15:08:42.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:42 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-18T15:08:42.757 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Mar 18 15:08:42 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-y[38034]: 2024-03-18T15:08:42.361+0000 7f263983b200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T15:08:43.256 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Mar 18 15:08:42 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-y[38034]: 2024-03-18T15:08:42.954+0000 7f263983b200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T15:08:43.257 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Mar 18 15:08:43 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-y[38034]: 2024-03-18T15:08:43.054+0000 7f263983b200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T15:08:43.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:43 smithi104 ceph-mon[36070]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:43.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:43 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/416386632' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "37b5a5cd-27ae-4e77-9b05-774b817b2ee9"}]: dispatch 2024-03-18T15:08:43.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:43 smithi104 ceph-mon[36070]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "37b5a5cd-27ae-4e77-9b05-774b817b2ee9"}]: dispatch 2024-03-18T15:08:43.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:43 smithi104 ceph-mon[36070]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "37b5a5cd-27ae-4e77-9b05-774b817b2ee9"}]': finished 2024-03-18T15:08:43.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:43 smithi104 ceph-mon[36070]: osdmap e5: 1 total, 0 up, 1 in 2024-03-18T15:08:43.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:43 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T15:08:43.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:43 smithi019 ceph-mon[31522]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:43.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:43 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/416386632' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "37b5a5cd-27ae-4e77-9b05-774b817b2ee9"}]: dispatch 2024-03-18T15:08:43.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:43 smithi019 ceph-mon[31522]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "37b5a5cd-27ae-4e77-9b05-774b817b2ee9"}]: dispatch 2024-03-18T15:08:43.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:43 smithi019 ceph-mon[31522]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "37b5a5cd-27ae-4e77-9b05-774b817b2ee9"}]': finished 2024-03-18T15:08:43.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:43 smithi019 ceph-mon[31522]: osdmap e5: 1 total, 0 up, 1 in 2024-03-18T15:08:43.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:43 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T15:08:43.756 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Mar 18 15:08:43 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-y[38034]: 2024-03-18T15:08:43.276+0000 7f263983b200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T15:08:43.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:43 smithi033 ceph-mon[36648]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:43.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:43 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/416386632' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "37b5a5cd-27ae-4e77-9b05-774b817b2ee9"}]: dispatch 2024-03-18T15:08:43.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:43 smithi033 ceph-mon[36648]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "37b5a5cd-27ae-4e77-9b05-774b817b2ee9"}]: dispatch 2024-03-18T15:08:43.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:43 smithi033 ceph-mon[36648]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "37b5a5cd-27ae-4e77-9b05-774b817b2ee9"}]': finished 2024-03-18T15:08:43.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:43 smithi033 ceph-mon[36648]: osdmap e5: 1 total, 0 up, 1 in 2024-03-18T15:08:43.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:43 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T15:08:44.289 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:08:44.287+0000 7f44fe000700 1 -- 172.21.15.19:0/3766803104 <== mon.2 v2:172.21.15.33:3300/0 8 ==== mgrmap(e 15) ==== 137234+0+0 (secure 0 0 0) 0x7f44f400f3c0 con 0x7f4500100b80 2024-03-18T15:08:44.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:44 smithi104 ceph-mon[36070]: Standby manager daemon y started 2024-03-18T15:08:44.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:44 smithi104 ceph-mon[36070]: from='mgr.? 172.21.15.33:0/2569667435' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-18T15:08:44.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:44 smithi104 ceph-mon[36070]: from='mgr.? 172.21.15.33:0/2569667435' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-18T15:08:44.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:44 smithi104 ceph-mon[36070]: from='mgr.? 172.21.15.33:0/2569667435' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-18T15:08:44.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:44 smithi104 ceph-mon[36070]: from='mgr.? 172.21.15.33:0/2569667435' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-18T15:08:44.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:44 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/3427490044' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T15:08:44.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:44 smithi019 ceph-mon[31522]: Standby manager daemon y started 2024-03-18T15:08:44.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:44 smithi019 ceph-mon[31522]: from='mgr.? 172.21.15.33:0/2569667435' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-18T15:08:44.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:44 smithi019 ceph-mon[31522]: from='mgr.? 172.21.15.33:0/2569667435' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-18T15:08:44.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:44 smithi019 ceph-mon[31522]: from='mgr.? 172.21.15.33:0/2569667435' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-18T15:08:44.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:44 smithi019 ceph-mon[31522]: from='mgr.? 172.21.15.33:0/2569667435' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-18T15:08:44.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:44 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/3427490044' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T15:08:44.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:44 smithi033 ceph-mon[36648]: Standby manager daemon y started 2024-03-18T15:08:44.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:44 smithi033 ceph-mon[36648]: from='mgr.? 172.21.15.33:0/2569667435' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-18T15:08:44.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:44 smithi033 ceph-mon[36648]: from='mgr.? 172.21.15.33:0/2569667435' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-18T15:08:44.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:44 smithi033 ceph-mon[36648]: from='mgr.? 172.21.15.33:0/2569667435' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-18T15:08:44.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:44 smithi033 ceph-mon[36648]: from='mgr.? 172.21.15.33:0/2569667435' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-18T15:08:44.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:44 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/3427490044' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T15:08:45.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:45 smithi019 ceph-mon[31522]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:45.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:45 smithi019 ceph-mon[31522]: mgrmap e15: x(active, since 90s), standbys: z, y 2024-03-18T15:08:45.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:45 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-18T15:08:45.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:45 smithi104 ceph-mon[36070]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:45.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:45 smithi104 ceph-mon[36070]: mgrmap e15: x(active, since 90s), standbys: z, y 2024-03-18T15:08:45.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:45 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-18T15:08:45.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:45 smithi033 ceph-mon[36648]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:45.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:45 smithi033 ceph-mon[36648]: mgrmap e15: x(active, since 90s), standbys: z, y 2024-03-18T15:08:45.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:45 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-18T15:08:47.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:47 smithi104 ceph-mon[36070]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:47.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:47 smithi019 ceph-mon[31522]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:47.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:47 smithi033 ceph-mon[36648]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:49.612 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:49 smithi019 ceph-mon[31522]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:49.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:49 smithi104 ceph-mon[36070]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:49.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:49 smithi033 ceph-mon[36648]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:51.467 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:51 smithi019 ceph-mon[31522]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:51.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:51 smithi104 ceph-mon[36070]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:51.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:51 smithi033 ceph-mon[36648]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:53.562 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:53 smithi019 ceph-mon[31522]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:53.562 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:53 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-18T15:08:53.562 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:53 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:08:53.562 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:53 smithi019 ceph-mon[31522]: Deploying daemon osd.0 on smithi019 2024-03-18T15:08:53.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:53 smithi104 ceph-mon[36070]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:53.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:53 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-18T15:08:53.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:53 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:08:53.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:53 smithi104 ceph-mon[36070]: Deploying daemon osd.0 on smithi019 2024-03-18T15:08:53.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:53 smithi033 ceph-mon[36648]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:53.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:53 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-18T15:08:53.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:53 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:08:53.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:53 smithi033 ceph-mon[36648]: Deploying daemon osd.0 on smithi019 2024-03-18T15:08:55.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:55 smithi104 ceph-mon[36070]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:55.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:55 smithi019 ceph-mon[31522]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:55.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:55 smithi033 ceph-mon[36648]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:57.580 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:57 smithi019 ceph-mon[31522]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:57.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:57 smithi104 ceph-mon[36070]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:57.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:57 smithi033 ceph-mon[36648]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:59.518 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:08:59 smithi019 ceph-mon[31522]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:59.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:08:59 smithi104 ceph-mon[36070]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:08:59.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:08:59 smithi033 ceph-mon[36648]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:09:01.606 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:01 smithi019 ceph-mon[31522]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:09:01.607 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:01 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:09:01.607 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:01 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:09:01.607 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:01 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:09:01.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:01 smithi104 ceph-mon[36070]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:09:01.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:01 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:09:01.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:01 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:09:01.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:01 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:09:01.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:01 smithi033 ceph-mon[36648]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:09:01.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:01 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:09:01.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:01 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:09:01.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:01 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:09:03.359 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:03 smithi019 ceph-mon[31522]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:09:03.359 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:03 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:09:03.359 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:03 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:09:03.359 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:03 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:09:03.360 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:03 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:09:03.360 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:03 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:09:03.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:03 smithi104 ceph-mon[36070]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:09:03.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:03 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:09:03.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:03 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:09:03.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:03 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:09:03.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:03 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:09:03.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:03 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:09:03.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:03 smithi033 ceph-mon[36648]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:09:03.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:03 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:09:03.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:03 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:09:03.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:03 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:09:03.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:03 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:09:03.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:03 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:09:04.658 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:04 smithi033 ceph-mon[36648]: from='osd.0 [v2:172.21.15.19:6802/402333948,v1:172.21.15.19:6803/402333948]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-18T15:09:04.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:04 smithi104 ceph-mon[36070]: from='osd.0 [v2:172.21.15.19:6802/402333948,v1:172.21.15.19:6803/402333948]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-18T15:09:04.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:04 smithi019 ceph-mon[31522]: from='osd.0 [v2:172.21.15.19:6802/402333948,v1:172.21.15.19:6803/402333948]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-18T15:09:05.240 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:09:05.240+0000 7f44fe000700 1 -- 172.21.15.19:0/3766803104 <== mgr.14152 v2:172.21.15.19:6800/2017714236 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f4500062710 con 0x7f44e8039200 2024-03-18T15:09:05.240 INFO:teuthology.orchestra.run.smithi019.stdout:Created osd(s) 0 on host 'smithi019' 2024-03-18T15:09:05.244 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:09:05.244+0000 7f4507e00700 1 -- 172.21.15.19:0/3766803104 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f44e8039200 msgr2=0x7f44e803b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:09:05.245 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:09:05.244+0000 7f4507e00700 1 --2- 172.21.15.19:0/3766803104 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f44e8039200 0x7f44e803b6c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f44ec009c60 tx=0x7f44ec009260 comp rx=0 tx=0).stop 2024-03-18T15:09:05.245 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:09:05.244+0000 7f4507e00700 1 -- 172.21.15.19:0/3766803104 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4500100b80 msgr2=0x7f450019a710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:09:05.245 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:09:05.244+0000 7f4507e00700 1 --2- 172.21.15.19:0/3766803104 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4500100b80 0x7f450019a710 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f44f400eb30 tx=0x7f44f400ee40 comp rx=0 tx=0).stop 2024-03-18T15:09:05.245 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:09:05.244+0000 7f4507e00700 1 -- 172.21.15.19:0/3766803104 shutdown_connections 2024-03-18T15:09:05.245 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:09:05.244+0000 7f4507e00700 1 --2- 172.21.15.19:0/3766803104 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f44e8039200 0x7f44e803b6c0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:09:05.245 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:09:05.244+0000 7f4507e00700 1 --2- 172.21.15.19:0/3766803104 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4500101ea0 0x7f45001a1cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:09:05.245 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:09:05.244+0000 7f4507e00700 1 --2- 172.21.15.19:0/3766803104 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4500100b80 0x7f450019a710 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:09:05.245 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:09:05.244+0000 7f4507e00700 1 --2- 172.21.15.19:0/3766803104 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f45001014e0 0x7f450019ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:09:05.245 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:09:05.244+0000 7f4507e00700 1 -- 172.21.15.19:0/3766803104 >> 172.21.15.19:0/3766803104 conn(0x7f45000fc7a0 msgr2=0x7f45000fe2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:09:05.245 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:09:05.244+0000 7f4507e00700 1 -- 172.21.15.19:0/3766803104 shutdown_connections 2024-03-18T15:09:05.245 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:09:05.245+0000 7f4507e00700 1 -- 172.21.15.19:0/3766803104 wait complete. 2024-03-18T15:09:05.352 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:05 smithi019 ceph-mon[31522]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:09:05.352 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:05 smithi019 ceph-mon[31522]: from='osd.0 [v2:172.21.15.19:6802/402333948,v1:172.21.15.19:6803/402333948]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-18T15:09:05.352 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:05 smithi019 ceph-mon[31522]: osdmap e6: 1 total, 0 up, 1 in 2024-03-18T15:09:05.352 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:05 smithi019 ceph-mon[31522]: from='osd.0 [v2:172.21.15.19:6802/402333948,v1:172.21.15.19:6803/402333948]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]: dispatch 2024-03-18T15:09:05.352 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:05 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T15:09:05.352 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:05 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:09:05.353 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:05 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:09:05.353 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:05 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:09:05.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:05 smithi104 ceph-mon[36070]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:09:05.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:05 smithi104 ceph-mon[36070]: from='osd.0 [v2:172.21.15.19:6802/402333948,v1:172.21.15.19:6803/402333948]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-18T15:09:05.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:05 smithi104 ceph-mon[36070]: osdmap e6: 1 total, 0 up, 1 in 2024-03-18T15:09:05.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:05 smithi104 ceph-mon[36070]: from='osd.0 [v2:172.21.15.19:6802/402333948,v1:172.21.15.19:6803/402333948]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]: dispatch 2024-03-18T15:09:05.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:05 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T15:09:05.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:05 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:09:05.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:05 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:09:05.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:05 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:09:05.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:05 smithi033 ceph-mon[36648]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:09:05.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:05 smithi033 ceph-mon[36648]: from='osd.0 [v2:172.21.15.19:6802/402333948,v1:172.21.15.19:6803/402333948]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-18T15:09:05.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:05 smithi033 ceph-mon[36648]: osdmap e6: 1 total, 0 up, 1 in 2024-03-18T15:09:05.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:05 smithi033 ceph-mon[36648]: from='osd.0 [v2:172.21.15.19:6802/402333948,v1:172.21.15.19:6803/402333948]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]: dispatch 2024-03-18T15:09:05.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:05 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T15:09:05.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:05 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:09:05.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:05 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:09:05.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:05 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:09:06.007 DEBUG:teuthology.orchestra.run.smithi019:osd.0> sudo journalctl -f -n 0 -u ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@osd.0.service 2024-03-18T15:09:06.013 INFO:tasks.cephadm:Deploying osd.1 on smithi033 with /dev/vg_nvme/lv_4... 2024-03-18T15:09:06.013 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-18T15:09:06.258 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T15:09:06.619 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:06 smithi019 ceph-mon[31522]: from='osd.0 [v2:172.21.15.19:6802/402333948,v1:172.21.15.19:6803/402333948]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]': finished 2024-03-18T15:09:06.620 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:06 smithi019 ceph-mon[31522]: osdmap e7: 1 total, 0 up, 1 in 2024-03-18T15:09:06.620 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:06 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T15:09:06.620 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:06 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T15:09:06.690 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:06 smithi033 ceph-mon[36648]: from='osd.0 [v2:172.21.15.19:6802/402333948,v1:172.21.15.19:6803/402333948]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]': finished 2024-03-18T15:09:06.690 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:06 smithi033 ceph-mon[36648]: osdmap e7: 1 total, 0 up, 1 in 2024-03-18T15:09:06.691 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:06 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T15:09:06.691 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:06 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T15:09:06.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:06 smithi104 ceph-mon[36070]: from='osd.0 [v2:172.21.15.19:6802/402333948,v1:172.21.15.19:6803/402333948]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]': finished 2024-03-18T15:09:06.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:06 smithi104 ceph-mon[36070]: osdmap e7: 1 total, 0 up, 1 in 2024-03-18T15:09:06.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:06 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T15:09:06.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:06 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T15:09:07.544 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:07 smithi033 ceph-mon[36648]: purged_snaps scrub starts 2024-03-18T15:09:07.545 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:07 smithi033 ceph-mon[36648]: purged_snaps scrub ok 2024-03-18T15:09:07.545 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:07 smithi033 ceph-mon[36648]: pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:09:07.545 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:07 smithi033 ceph-mon[36648]: osd.0 [v2:172.21.15.19:6802/402333948,v1:172.21.15.19:6803/402333948] boot 2024-03-18T15:09:07.545 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:07 smithi033 ceph-mon[36648]: osdmap e8: 1 total, 1 up, 1 in 2024-03-18T15:09:07.545 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:07 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T15:09:07.685 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:07 smithi019 ceph-mon[31522]: purged_snaps scrub starts 2024-03-18T15:09:07.685 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:07 smithi019 ceph-mon[31522]: purged_snaps scrub ok 2024-03-18T15:09:07.685 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:07 smithi019 ceph-mon[31522]: pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:09:07.686 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:07 smithi019 ceph-mon[31522]: osd.0 [v2:172.21.15.19:6802/402333948,v1:172.21.15.19:6803/402333948] boot 2024-03-18T15:09:07.686 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:07 smithi019 ceph-mon[31522]: osdmap e8: 1 total, 1 up, 1 in 2024-03-18T15:09:07.686 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:07 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T15:09:07.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:07 smithi104 ceph-mon[36070]: purged_snaps scrub starts 2024-03-18T15:09:07.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:07 smithi104 ceph-mon[36070]: purged_snaps scrub ok 2024-03-18T15:09:07.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:07 smithi104 ceph-mon[36070]: pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:09:07.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:07 smithi104 ceph-mon[36070]: osd.0 [v2:172.21.15.19:6802/402333948,v1:172.21.15.19:6803/402333948] boot 2024-03-18T15:09:07.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:07 smithi104 ceph-mon[36070]: osdmap e8: 1 total, 1 up, 1 in 2024-03-18T15:09:07.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:07 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T15:09:08.131 INFO:teuthology.orchestra.run.smithi033.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-18T15:09:08.131 INFO:teuthology.orchestra.run.smithi033.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-18T15:09:08.131 INFO:teuthology.orchestra.run.smithi033.stderr: stderr: 10+0 records in 2024-03-18T15:09:08.131 INFO:teuthology.orchestra.run.smithi033.stderr:10+0 records out 2024-03-18T15:09:08.131 INFO:teuthology.orchestra.run.smithi033.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0110811 s, 946 MB/s 2024-03-18T15:09:08.131 INFO:teuthology.orchestra.run.smithi033.stderr:--> Zapping successful for: 2024-03-18T15:09:08.635 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:08 smithi033 ceph-mon[36648]: osdmap e9: 1 total, 1 up, 1 in 2024-03-18T15:09:08.660 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi033:vg_nvme/lv_4 2024-03-18T15:09:08.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:08 smithi104 ceph-mon[36070]: osdmap e9: 1 total, 1 up, 1 in 2024-03-18T15:09:08.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:08 smithi019 ceph-mon[31522]: osdmap e9: 1 total, 1 up, 1 in 2024-03-18T15:09:08.904 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T15:09:09.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:09 smithi104 ceph-mon[36070]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T15:09:09.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:09 smithi019 ceph-mon[31522]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T15:09:09.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:09 smithi033 ceph-mon[36648]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T15:09:10.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:09:10.272+0000 7f0f9fce9700 1 -- 172.21.15.33:0/3407935920 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0f98102e40 msgr2=0x7f0f981096f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:09:10.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:09:10.272+0000 7f0f9fce9700 1 --2- 172.21.15.33:0/3407935920 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0f98102e40 0x7f0f981096f0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f0f9000b0d0 tx=0x7f0f9000b3e0 comp rx=0 tx=0).stop 2024-03-18T15:09:10.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:09:10.272+0000 7f0f9fce9700 1 -- 172.21.15.33:0/3407935920 shutdown_connections 2024-03-18T15:09:10.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:09:10.272+0000 7f0f9fce9700 1 --2- 172.21.15.33:0/3407935920 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f981012a0 0x7f0f981016c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:09:10.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:09:10.272+0000 7f0f9fce9700 1 --2- 172.21.15.33:0/3407935920 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0f98102e40 0x7f0f981096f0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:09:10.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:09:10.272+0000 7f0f9fce9700 1 --2- 172.21.15.33:0/3407935920 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0f981024a0 0x7f0f98102900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:09:10.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:09:10.272+0000 7f0f9fce9700 1 -- 172.21.15.33:0/3407935920 >> 172.21.15.33:0/3407935920 conn(0x7f0f980fc860 msgr2=0x7f0f980fec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:09:10.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:09:10.272+0000 7f0f9fce9700 1 -- 172.21.15.33:0/3407935920 shutdown_connections 2024-03-18T15:09:10.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:09:10.272+0000 7f0f9fce9700 1 -- 172.21.15.33:0/3407935920 wait complete. 2024-03-18T15:09:10.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:09:10.272+0000 7f0f9fce9700 1 Processor -- start 2024-03-18T15:09:10.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:09:10.273+0000 7f0f9fce9700 1 -- start start 2024-03-18T15:09:10.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:09:10.273+0000 7f0f9fce9700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0f981012a0 0x7f0f98198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:09:10.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:09:10.273+0000 7f0f9fce9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f981024a0 0x7f0f98198a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:09:10.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:09:10.273+0000 7f0f9fce9700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0f98102e40 0x7f0f9819faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:09:10.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:09:10.273+0000 7f0f9fce9700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f0f9810ce80 con 0x7f0f98102e40 2024-03-18T15:09:10.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:09:10.273+0000 7f0f9fce9700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f0f9810cd00 con 0x7f0f981012a0 2024-03-18T15:09:10.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:09:10.273+0000 7f0f9fce9700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f0f9810d000 con 0x7f0f981024a0 2024-03-18T15:09:10.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:09:10.273+0000 7f0f9da85700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0f981012a0 0x7f0f98198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:09:10.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:09:10.273+0000 7f0f9da85700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0f981012a0 0x7f0f98198530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:35398/0 (socket says 172.21.15.33:35398) 2024-03-18T15:09:10.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:09:10.273+0000 7f0f9da85700 1 -- 172.21.15.33:0/1207722752 learned_addr learned my addr 172.21.15.33:0/1207722752 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-18T15:09:10.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:09:10.273+0000 7f0f9d284700 1 --2- 172.21.15.33:0/1207722752 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f981024a0 0x7f0f98198a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:09:10.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:09:10.273+0000 7f0f9da85700 1 -- 172.21.15.33:0/1207722752 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f981024a0 msgr2=0x7f0f98198a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:09:10.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:09:10.273+0000 7f0f9da85700 1 --2- 172.21.15.33:0/1207722752 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f981024a0 0x7f0f98198a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:09:10.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:09:10.273+0000 7f0f9da85700 1 -- 172.21.15.33:0/1207722752 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0f98102e40 msgr2=0x7f0f9819faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:09:10.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:09:10.273+0000 7f0f9da85700 1 --2- 172.21.15.33:0/1207722752 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0f98102e40 0x7f0f9819faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:09:10.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:09:10.273+0000 7f0f9da85700 1 -- 172.21.15.33:0/1207722752 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f981a0190 con 0x7f0f981012a0 2024-03-18T15:09:10.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:09:10.273+0000 7f0f9da85700 1 --2- 172.21.15.33:0/1207722752 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0f981012a0 0x7f0f98198530 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f0f9400c970 tx=0x7f0f9400cd30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:09:10.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:09:10.274+0000 7f0f8effd700 1 -- 172.21.15.33:0/1207722752 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f94004220 con 0x7f0f981012a0 2024-03-18T15:09:10.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:09:10.274+0000 7f0f9fce9700 1 -- 172.21.15.33:0/1207722752 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f981a0480 con 0x7f0f981012a0 2024-03-18T15:09:10.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:09:10.274+0000 7f0f8effd700 1 -- 172.21.15.33:0/1207722752 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0f94007d80 con 0x7f0f981012a0 2024-03-18T15:09:10.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:09:10.274+0000 7f0f8effd700 1 -- 172.21.15.33:0/1207722752 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f9400f450 con 0x7f0f981012a0 2024-03-18T15:09:10.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:09:10.274+0000 7f0f9fce9700 1 -- 172.21.15.33:0/1207722752 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f981a0a40 con 0x7f0f981012a0 2024-03-18T15:09:10.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:09:10.275+0000 7f0f9fce9700 1 -- 172.21.15.33:0/1207722752 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f98068f90 con 0x7f0f981012a0 2024-03-18T15:09:10.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:09:10.276+0000 7f0f8effd700 1 -- 172.21.15.33:0/1207722752 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 15) ==== 137234+0+0 (secure 0 0 0) 0x7f0f940078a0 con 0x7f0f981012a0 2024-03-18T15:09:10.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:09:10.277+0000 7f0f8effd700 1 --2- 172.21.15.33:0/1207722752 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f0f840a2800 0x7f0f840a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:09:10.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:09:10.277+0000 7f0f8effd700 1 -- 172.21.15.33:0/1207722752 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1767+0+0 (secure 0 0 0) 0x7f0f940ccf50 con 0x7f0f981012a0 2024-03-18T15:09:10.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:09:10.277+0000 7f0f9d284700 1 --2- 172.21.15.33:0/1207722752 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f0f840a2800 0x7f0f840a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:09:10.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:09:10.278+0000 7f0f9d284700 1 --2- 172.21.15.33:0/1207722752 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f0f840a2800 0x7f0f840a4cc0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f0f88006fd0 tx=0x7f0f88008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:09:10.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:09:10.280+0000 7f0f8effd700 1 -- 172.21.15.33:0/1207722752 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0f94014030 con 0x7f0f981012a0 2024-03-18T15:09:10.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:09:10.496+0000 7f0f9fce9700 1 -- 172.21.15.33:0/1207722752 --> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f0f980fe940 con 0x7f0f840a2800 2024-03-18T15:09:11.453 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:11 smithi033 ceph-mon[36648]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T15:09:11.453 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:11 smithi033 ceph-mon[36648]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:09:11.454 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:11 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T15:09:11.454 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:11 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T15:09:11.454 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:11 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:09:11.454 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:11 smithi033 ceph-mon[36648]: Detected new or changed devices on smithi019 2024-03-18T15:09:11.454 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:11 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:09:11.454 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:11 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:09:11.454 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:11 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:09:11.454 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:11 smithi033 ceph-mon[36648]: Adjusting osd_memory_target on smithi019 to 17095M 2024-03-18T15:09:11.454 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:11 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:09:11.454 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:11 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:09:11.454 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:11 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:09:11.454 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:11 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:09:11.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:11 smithi104 ceph-mon[36070]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T15:09:11.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:11 smithi104 ceph-mon[36070]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:09:11.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:11 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T15:09:11.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:11 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T15:09:11.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:11 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:09:11.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:11 smithi104 ceph-mon[36070]: Detected new or changed devices on smithi019 2024-03-18T15:09:11.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:11 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:09:11.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:11 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:09:11.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:11 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:09:11.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:11 smithi104 ceph-mon[36070]: Adjusting osd_memory_target on smithi019 to 17095M 2024-03-18T15:09:11.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:11 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:09:11.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:11 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:09:11.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:11 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:09:11.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:11 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:09:11.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:11 smithi019 ceph-mon[31522]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T15:09:11.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:11 smithi019 ceph-mon[31522]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:09:11.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:11 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T15:09:11.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:11 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T15:09:11.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:11 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:09:11.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:11 smithi019 ceph-mon[31522]: Detected new or changed devices on smithi019 2024-03-18T15:09:11.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:11 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:09:11.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:11 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:09:11.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:11 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:09:11.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:11 smithi019 ceph-mon[31522]: Adjusting osd_memory_target on smithi019 to 17095M 2024-03-18T15:09:11.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:11 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:09:11.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:11 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:09:11.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:11 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:09:11.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:11 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:09:13.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:13 smithi104 ceph-mon[36070]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T15:09:13.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:13 smithi104 ceph-mon[36070]: from='client.? 172.21.15.33:0/261419849' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3a1134e8-ede7-4dbd-9be1-eefd2cbb43bb"}]: dispatch 2024-03-18T15:09:13.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:13 smithi104 ceph-mon[36070]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3a1134e8-ede7-4dbd-9be1-eefd2cbb43bb"}]: dispatch 2024-03-18T15:09:13.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:13 smithi104 ceph-mon[36070]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3a1134e8-ede7-4dbd-9be1-eefd2cbb43bb"}]': finished 2024-03-18T15:09:13.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:13 smithi104 ceph-mon[36070]: osdmap e10: 2 total, 1 up, 2 in 2024-03-18T15:09:13.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:13 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T15:09:13.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:13 smithi019 ceph-mon[31522]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T15:09:13.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:13 smithi019 ceph-mon[31522]: from='client.? 172.21.15.33:0/261419849' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3a1134e8-ede7-4dbd-9be1-eefd2cbb43bb"}]: dispatch 2024-03-18T15:09:13.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:13 smithi019 ceph-mon[31522]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3a1134e8-ede7-4dbd-9be1-eefd2cbb43bb"}]: dispatch 2024-03-18T15:09:13.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:13 smithi019 ceph-mon[31522]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3a1134e8-ede7-4dbd-9be1-eefd2cbb43bb"}]': finished 2024-03-18T15:09:13.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:13 smithi019 ceph-mon[31522]: osdmap e10: 2 total, 1 up, 2 in 2024-03-18T15:09:13.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:13 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T15:09:13.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:13 smithi033 ceph-mon[36648]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T15:09:13.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:13 smithi033 ceph-mon[36648]: from='client.? 172.21.15.33:0/261419849' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3a1134e8-ede7-4dbd-9be1-eefd2cbb43bb"}]: dispatch 2024-03-18T15:09:13.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:13 smithi033 ceph-mon[36648]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3a1134e8-ede7-4dbd-9be1-eefd2cbb43bb"}]: dispatch 2024-03-18T15:09:13.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:13 smithi033 ceph-mon[36648]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3a1134e8-ede7-4dbd-9be1-eefd2cbb43bb"}]': finished 2024-03-18T15:09:13.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:13 smithi033 ceph-mon[36648]: osdmap e10: 2 total, 1 up, 2 in 2024-03-18T15:09:13.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:13 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T15:09:14.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:14 smithi019 ceph-mon[31522]: from='client.? 172.21.15.33:0/2872040467' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T15:09:14.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:14 smithi104 ceph-mon[36070]: from='client.? 172.21.15.33:0/2872040467' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T15:09:14.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:14 smithi033 ceph-mon[36648]: from='client.? 172.21.15.33:0/2872040467' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T15:09:15.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:15 smithi104 ceph-mon[36070]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T15:09:15.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:15 smithi019 ceph-mon[31522]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T15:09:15.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:15 smithi033 ceph-mon[36648]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T15:09:17.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:17 smithi104 ceph-mon[36070]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T15:09:17.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:17 smithi019 ceph-mon[31522]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T15:09:17.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:17 smithi033 ceph-mon[36648]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T15:09:19.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:19 smithi104 ceph-mon[36070]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T15:09:19.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:19 smithi019 ceph-mon[31522]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T15:09:19.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:19 smithi033 ceph-mon[36648]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T15:09:20.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:20 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-18T15:09:20.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:20 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:09:20.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:20 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-18T15:09:20.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:20 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:09:20.729 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:20 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-18T15:09:20.729 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:20 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:09:21.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:21 smithi019 ceph-mon[31522]: Deploying daemon osd.1 on smithi033 2024-03-18T15:09:21.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:21 smithi019 ceph-mon[31522]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T15:09:21.732 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:21 smithi033 ceph-mon[36648]: Deploying daemon osd.1 on smithi033 2024-03-18T15:09:21.732 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:21 smithi033 ceph-mon[36648]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T15:09:21.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:21 smithi104 ceph-mon[36070]: Deploying daemon osd.1 on smithi033 2024-03-18T15:09:21.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:21 smithi104 ceph-mon[36070]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T15:09:23.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:23 smithi019 ceph-mon[31522]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T15:09:23.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:23 smithi033 ceph-mon[36648]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T15:09:23.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:23 smithi104 ceph-mon[36070]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T15:09:25.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:25 smithi019 ceph-mon[31522]: pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T15:09:25.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:25 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:09:25.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:25 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:09:25.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:25 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:09:25.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:25 smithi033 ceph-mon[36648]: pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T15:09:25.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:25 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:09:25.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:25 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:09:25.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:25 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:09:25.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:25 smithi104 ceph-mon[36070]: pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T15:09:25.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:25 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:09:25.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:25 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:09:25.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:25 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:09:27.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:27 smithi019 ceph-mon[31522]: pgmap v66: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T15:09:27.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:27 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:09:27.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:27 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:09:27.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:27 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:09:27.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:27 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:09:27.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:27 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:09:27.746 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:27 smithi104 ceph-mon[36070]: pgmap v66: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T15:09:27.746 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:27 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:09:27.747 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:27 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:09:27.747 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:27 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:09:27.747 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:27 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:09:27.747 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:27 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:09:27.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:27 smithi033 ceph-mon[36648]: pgmap v66: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T15:09:27.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:27 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:09:27.748 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:27 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:09:27.749 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:27 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:09:27.749 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:27 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:09:27.749 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:27 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:09:28.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:09:28.844+0000 7f0f8effd700 1 -- 172.21.15.33:0/1207722752 <== mgr.14152 v2:172.21.15.19:6800/2017714236 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f0f980fe940 con 0x7f0f840a2800 2024-03-18T15:09:28.845 INFO:teuthology.orchestra.run.smithi033.stdout:Created osd(s) 1 on host 'smithi033' 2024-03-18T15:09:28.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:09:28.846+0000 7f0f9fce9700 1 -- 172.21.15.33:0/1207722752 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f0f840a2800 msgr2=0x7f0f840a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:09:28.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:09:28.846+0000 7f0f9fce9700 1 --2- 172.21.15.33:0/1207722752 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f0f840a2800 0x7f0f840a4cc0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f0f88006fd0 tx=0x7f0f88008040 comp rx=0 tx=0).stop 2024-03-18T15:09:28.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:09:28.846+0000 7f0f9fce9700 1 -- 172.21.15.33:0/1207722752 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0f981012a0 msgr2=0x7f0f98198530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:09:28.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:09:28.846+0000 7f0f9fce9700 1 --2- 172.21.15.33:0/1207722752 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0f981012a0 0x7f0f98198530 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f0f9400c970 tx=0x7f0f9400cd30 comp rx=0 tx=0).stop 2024-03-18T15:09:28.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:09:28.846+0000 7f0f9fce9700 1 -- 172.21.15.33:0/1207722752 shutdown_connections 2024-03-18T15:09:28.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:09:28.846+0000 7f0f9fce9700 1 --2- 172.21.15.33:0/1207722752 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f0f840a2800 0x7f0f840a4cc0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:09:28.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:09:28.846+0000 7f0f9fce9700 1 --2- 172.21.15.33:0/1207722752 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0f98102e40 0x7f0f9819faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:09:28.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:09:28.846+0000 7f0f9fce9700 1 --2- 172.21.15.33:0/1207722752 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0f981012a0 0x7f0f98198530 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:09:28.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:09:28.846+0000 7f0f9fce9700 1 --2- 172.21.15.33:0/1207722752 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f981024a0 0x7f0f98198a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:09:28.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:09:28.846+0000 7f0f9fce9700 1 -- 172.21.15.33:0/1207722752 >> 172.21.15.33:0/1207722752 conn(0x7f0f980fc860 msgr2=0x7f0f980fe230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:09:28.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:09:28.847+0000 7f0f9fce9700 1 -- 172.21.15.33:0/1207722752 shutdown_connections 2024-03-18T15:09:28.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:09:28.847+0000 7f0f9fce9700 1 -- 172.21.15.33:0/1207722752 wait complete. 2024-03-18T15:09:29.423 DEBUG:teuthology.orchestra.run.smithi033:osd.1> sudo journalctl -f -n 0 -u ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@osd.1.service 2024-03-18T15:09:29.425 INFO:tasks.cephadm:Deploying osd.2 on smithi104 with /dev/vg_nvme/lv_4... 2024-03-18T15:09:29.426 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-18T15:09:29.660 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T15:09:30.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:29 smithi033 ceph-mon[36648]: pgmap v67: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T15:09:30.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:29 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:09:30.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:29 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:09:30.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:29 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:09:30.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:29 smithi033 ceph-mon[36648]: from='osd.1 [v2:172.21.15.33:6800/1951880113,v1:172.21.15.33:6801/1951880113]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-18T15:09:30.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:29 smithi033 ceph-mon[36648]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-18T15:09:30.093 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:29 smithi104 ceph-mon[36070]: pgmap v67: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T15:09:30.094 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:29 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:09:30.094 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:29 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:09:30.094 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:29 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:09:30.094 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:29 smithi104 ceph-mon[36070]: from='osd.1 [v2:172.21.15.33:6800/1951880113,v1:172.21.15.33:6801/1951880113]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-18T15:09:30.094 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:29 smithi104 ceph-mon[36070]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-18T15:09:30.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:29 smithi019 ceph-mon[31522]: pgmap v67: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T15:09:30.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:29 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:09:30.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:29 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:09:30.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:29 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:09:30.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:29 smithi019 ceph-mon[31522]: from='osd.1 [v2:172.21.15.33:6800/1951880113,v1:172.21.15.33:6801/1951880113]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-18T15:09:30.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:29 smithi019 ceph-mon[31522]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-18T15:09:30.995 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:30 smithi104 ceph-mon[36070]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-18T15:09:30.995 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:30 smithi104 ceph-mon[36070]: from='osd.1 [v2:172.21.15.33:6800/1951880113,v1:172.21.15.33:6801/1951880113]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-03-18T15:09:30.995 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:30 smithi104 ceph-mon[36070]: osdmap e11: 2 total, 1 up, 2 in 2024-03-18T15:09:30.995 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:30 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T15:09:30.996 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:30 smithi104 ceph-mon[36070]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-03-18T15:09:31.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:30 smithi033 ceph-mon[36648]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-18T15:09:31.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:30 smithi033 ceph-mon[36648]: from='osd.1 [v2:172.21.15.33:6800/1951880113,v1:172.21.15.33:6801/1951880113]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-03-18T15:09:31.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:30 smithi033 ceph-mon[36648]: osdmap e11: 2 total, 1 up, 2 in 2024-03-18T15:09:31.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:30 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T15:09:31.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:30 smithi033 ceph-mon[36648]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-03-18T15:09:31.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:30 smithi019 ceph-mon[31522]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-18T15:09:31.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:30 smithi019 ceph-mon[31522]: from='osd.1 [v2:172.21.15.33:6800/1951880113,v1:172.21.15.33:6801/1951880113]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-03-18T15:09:31.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:30 smithi019 ceph-mon[31522]: osdmap e11: 2 total, 1 up, 2 in 2024-03-18T15:09:31.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:30 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T15:09:31.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:30 smithi019 ceph-mon[31522]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-03-18T15:09:31.502 INFO:journalctl@ceph.osd.1.smithi033.stdout:Mar 18 15:09:31 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-1[41252]: 2024-03-18T15:09:31.251+0000 7f425ff6b700 -1 osd.1 0 waiting for initial osdmap 2024-03-18T15:09:31.503 INFO:journalctl@ceph.osd.1.smithi033.stdout:Mar 18 15:09:31 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-1[41252]: 2024-03-18T15:09:31.254+0000 7f4257d56700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T15:09:31.546 INFO:teuthology.orchestra.run.smithi104.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-18T15:09:31.547 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-18T15:09:31.547 INFO:teuthology.orchestra.run.smithi104.stderr: stderr: 10+0 records in 2024-03-18T15:09:31.547 INFO:teuthology.orchestra.run.smithi104.stderr:10+0 records out 2024-03-18T15:09:31.547 INFO:teuthology.orchestra.run.smithi104.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113911 s, 921 MB/s 2024-03-18T15:09:31.547 INFO:teuthology.orchestra.run.smithi104.stderr:--> Zapping successful for: 2024-03-18T15:09:32.095 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:31 smithi033 ceph-mon[36648]: pgmap v69: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T15:09:32.095 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:31 smithi033 ceph-mon[36648]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]': finished 2024-03-18T15:09:32.095 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:31 smithi033 ceph-mon[36648]: osdmap e12: 2 total, 1 up, 2 in 2024-03-18T15:09:32.095 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:31 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T15:09:32.095 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:31 smithi033 ceph-mon[36648]: from='osd.1 ' entity='osd.1' 2024-03-18T15:09:32.096 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:31 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T15:09:32.161 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:31 smithi104 ceph-mon[36070]: pgmap v69: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T15:09:32.161 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:31 smithi104 ceph-mon[36070]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]': finished 2024-03-18T15:09:32.161 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:31 smithi104 ceph-mon[36070]: osdmap e12: 2 total, 1 up, 2 in 2024-03-18T15:09:32.161 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:31 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T15:09:32.162 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:31 smithi104 ceph-mon[36070]: from='osd.1 ' entity='osd.1' 2024-03-18T15:09:32.162 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:31 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T15:09:32.188 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi104:vg_nvme/lv_4 2024-03-18T15:09:32.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:31 smithi019 ceph-mon[31522]: pgmap v69: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T15:09:32.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:31 smithi019 ceph-mon[31522]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]': finished 2024-03-18T15:09:32.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:31 smithi019 ceph-mon[31522]: osdmap e12: 2 total, 1 up, 2 in 2024-03-18T15:09:32.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:31 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T15:09:32.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:31 smithi019 ceph-mon[31522]: from='osd.1 ' entity='osd.1' 2024-03-18T15:09:32.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:31 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T15:09:32.422 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T15:09:33.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:33 smithi104 ceph-mon[36070]: purged_snaps scrub starts 2024-03-18T15:09:33.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:33 smithi104 ceph-mon[36070]: purged_snaps scrub ok 2024-03-18T15:09:33.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:33 smithi104 ceph-mon[36070]: Detected new or changed devices on smithi033 2024-03-18T15:09:33.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:33 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:09:33.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:33 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:09:33.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:33 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:09:33.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:33 smithi104 ceph-mon[36070]: Adjusting osd_memory_target on smithi033 to 17084M 2024-03-18T15:09:33.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:33 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:09:33.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:33 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:09:33.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:33 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:09:33.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:33 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:09:33.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:33 smithi104 ceph-mon[36070]: pgmap v71: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T15:09:33.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:33 smithi104 ceph-mon[36070]: osd.1 [v2:172.21.15.33:6800/1951880113,v1:172.21.15.33:6801/1951880113] boot 2024-03-18T15:09:33.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:33 smithi104 ceph-mon[36070]: osdmap e13: 2 total, 2 up, 2 in 2024-03-18T15:09:33.446 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:33 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T15:09:33.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:33 smithi019 ceph-mon[31522]: purged_snaps scrub starts 2024-03-18T15:09:33.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:33 smithi019 ceph-mon[31522]: purged_snaps scrub ok 2024-03-18T15:09:33.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:33 smithi019 ceph-mon[31522]: Detected new or changed devices on smithi033 2024-03-18T15:09:33.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:33 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:09:33.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:33 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:09:33.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:33 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:09:33.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:33 smithi019 ceph-mon[31522]: Adjusting osd_memory_target on smithi033 to 17084M 2024-03-18T15:09:33.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:33 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:09:33.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:33 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:09:33.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:33 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:09:33.467 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:33 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:09:33.467 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:33 smithi019 ceph-mon[31522]: pgmap v71: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T15:09:33.467 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:33 smithi019 ceph-mon[31522]: osd.1 [v2:172.21.15.33:6800/1951880113,v1:172.21.15.33:6801/1951880113] boot 2024-03-18T15:09:33.467 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:33 smithi019 ceph-mon[31522]: osdmap e13: 2 total, 2 up, 2 in 2024-03-18T15:09:33.467 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:33 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T15:09:33.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:33 smithi033 ceph-mon[36648]: purged_snaps scrub starts 2024-03-18T15:09:33.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:33 smithi033 ceph-mon[36648]: purged_snaps scrub ok 2024-03-18T15:09:33.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:33 smithi033 ceph-mon[36648]: Detected new or changed devices on smithi033 2024-03-18T15:09:33.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:33 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:09:33.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:33 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:09:33.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:33 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:09:33.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:33 smithi033 ceph-mon[36648]: Adjusting osd_memory_target on smithi033 to 17084M 2024-03-18T15:09:33.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:33 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:09:33.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:33 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:09:33.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:33 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:09:33.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:33 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:09:33.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:33 smithi033 ceph-mon[36648]: pgmap v71: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T15:09:33.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:33 smithi033 ceph-mon[36648]: osd.1 [v2:172.21.15.33:6800/1951880113,v1:172.21.15.33:6801/1951880113] boot 2024-03-18T15:09:33.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:33 smithi033 ceph-mon[36648]: osdmap e13: 2 total, 2 up, 2 in 2024-03-18T15:09:33.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:33 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T15:09:33.807 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:09:33.804+0000 7fd7199ed700 1 -- 172.21.15.104:0/4050242807 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd714102da0 msgr2=0x7fd714109650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:09:33.807 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:09:33.804+0000 7fd7199ed700 1 --2- 172.21.15.104:0/4050242807 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd714102da0 0x7fd714109650 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fd70c00b0a0 tx=0x7fd70c00b3b0 comp rx=0 tx=0).stop 2024-03-18T15:09:33.807 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:09:33.804+0000 7fd7199ed700 1 -- 172.21.15.104:0/4050242807 shutdown_connections 2024-03-18T15:09:33.807 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:09:33.804+0000 7fd7199ed700 1 --2- 172.21.15.104:0/4050242807 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd714102da0 0x7fd714109650 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:09:33.807 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:09:33.804+0000 7fd7199ed700 1 --2- 172.21.15.104:0/4050242807 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd714101200 0x7fd714101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:09:33.808 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:09:33.804+0000 7fd7199ed700 1 --2- 172.21.15.104:0/4050242807 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd714102400 0x7fd714102860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:09:33.808 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:09:33.804+0000 7fd7199ed700 1 -- 172.21.15.104:0/4050242807 >> 172.21.15.104:0/4050242807 conn(0x7fd7140fc7c0 msgr2=0x7fd7140febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:09:33.808 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:09:33.804+0000 7fd7199ed700 1 -- 172.21.15.104:0/4050242807 shutdown_connections 2024-03-18T15:09:33.808 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:09:33.804+0000 7fd7199ed700 1 -- 172.21.15.104:0/4050242807 wait complete. 2024-03-18T15:09:33.808 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:09:33.805+0000 7fd7199ed700 1 Processor -- start 2024-03-18T15:09:33.808 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:09:33.805+0000 7fd7199ed700 1 -- start start 2024-03-18T15:09:33.808 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:09:33.805+0000 7fd7199ed700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd714101200 0x7fd7141974c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:09:33.808 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:09:33.805+0000 7fd7199ed700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd714102400 0x7fd714197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:09:33.808 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:09:33.805+0000 7fd7199ed700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd714102da0 0x7fd71419ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:09:33.808 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:09:33.805+0000 7fd7199ed700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fd71410cee0 con 0x7fd714102400 2024-03-18T15:09:33.808 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:09:33.805+0000 7fd7199ed700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd71410cd60 con 0x7fd714102da0 2024-03-18T15:09:33.808 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:09:33.805+0000 7fd7199ed700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fd71410d060 con 0x7fd714101200 2024-03-18T15:09:33.808 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:09:33.805+0000 7fd712ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd714101200 0x7fd7141974c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:09:33.809 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:09:33.805+0000 7fd712ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd714101200 0x7fd7141974c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33474/0 (socket says 172.21.15.104:33474) 2024-03-18T15:09:33.809 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:09:33.805+0000 7fd712ffd700 1 -- 172.21.15.104:0/228724624 learned_addr learned my addr 172.21.15.104:0/228724624 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-18T15:09:33.809 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:09:33.806+0000 7fd712ffd700 1 -- 172.21.15.104:0/228724624 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd714102da0 msgr2=0x7fd71419ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:09:33.809 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:09:33.806+0000 7fd712ffd700 1 --2- 172.21.15.104:0/228724624 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd714102da0 0x7fd71419ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:09:33.809 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:09:33.806+0000 7fd712ffd700 1 -- 172.21.15.104:0/228724624 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd714102400 msgr2=0x7fd714197a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:09:33.809 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:09:33.806+0000 7fd712ffd700 1 --2- 172.21.15.104:0/228724624 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd714102400 0x7fd714197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:09:33.809 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:09:33.806+0000 7fd712ffd700 1 -- 172.21.15.104:0/228724624 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd71419f120 con 0x7fd714101200 2024-03-18T15:09:33.809 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:09:33.806+0000 7fd712ffd700 1 --2- 172.21.15.104:0/228724624 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd714101200 0x7fd7141974c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fd70800b530 tx=0x7fd70800b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:09:33.809 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:09:33.806+0000 7fd7189eb700 1 -- 172.21.15.104:0/228724624 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd708015020 con 0x7fd714101200 2024-03-18T15:09:33.809 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:09:33.806+0000 7fd7199ed700 1 -- 172.21.15.104:0/228724624 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd71419f410 con 0x7fd714101200 2024-03-18T15:09:33.809 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:09:33.806+0000 7fd7189eb700 1 -- 172.21.15.104:0/228724624 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd708011d10 con 0x7fd714101200 2024-03-18T15:09:33.809 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:09:33.806+0000 7fd7189eb700 1 -- 172.21.15.104:0/228724624 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd70800f2e0 con 0x7fd714101200 2024-03-18T15:09:33.809 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:09:33.806+0000 7fd7199ed700 1 -- 172.21.15.104:0/228724624 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd71419f9d0 con 0x7fd714101200 2024-03-18T15:09:33.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:09:33.807+0000 7fd7199ed700 1 -- 172.21.15.104:0/228724624 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd714068f90 con 0x7fd714101200 2024-03-18T15:09:33.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:09:33.808+0000 7fd7189eb700 1 -- 172.21.15.104:0/228724624 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 15) ==== 137234+0+0 (secure 0 0 0) 0x7fd708018020 con 0x7fd714101200 2024-03-18T15:09:33.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:09:33.808+0000 7fd7189eb700 1 --2- 172.21.15.104:0/228724624 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fd6fc0a2800 0x7fd6fc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:09:33.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:09:33.809+0000 7fd7189eb700 1 -- 172.21.15.104:0/228724624 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(13..13 src has 1..13) ==== 2309+0+0 (secure 0 0 0) 0x7fd7080ccf30 con 0x7fd714101200 2024-03-18T15:09:33.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:09:33.809+0000 7fd7127fc700 1 --2- 172.21.15.104:0/228724624 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fd6fc0a2800 0x7fd6fc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:09:33.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:09:33.810+0000 7fd7127fc700 1 --2- 172.21.15.104:0/228724624 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fd6fc0a2800 0x7fd6fc0a4cc0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd7040097b0 tx=0x7fd704006d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:09:33.813 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:09:33.812+0000 7fd7189eb700 1 -- 172.21.15.104:0/228724624 <== mon.1 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd70801f020 con 0x7fd714101200 2024-03-18T15:09:34.016 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:09:34.015+0000 7fd7199ed700 1 -- 172.21.15.104:0/228724624 --> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fd714062710 con 0x7fd6fc0a2800 2024-03-18T15:09:34.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:34 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T15:09:34.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:34 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T15:09:34.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:34 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:09:34.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:34 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T15:09:34.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:34 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T15:09:34.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:34 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:09:34.644 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:34 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T15:09:34.644 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:34 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T15:09:34.644 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:34 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:09:35.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:35 smithi019 ceph-mon[31522]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:09:35.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:35 smithi019 ceph-mon[31522]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T15:09:35.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:35 smithi019 ceph-mon[31522]: osdmap e14: 2 total, 2 up, 2 in 2024-03-18T15:09:35.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:35 smithi033 ceph-mon[36648]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:09:35.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:35 smithi033 ceph-mon[36648]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T15:09:35.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:35 smithi033 ceph-mon[36648]: osdmap e14: 2 total, 2 up, 2 in 2024-03-18T15:09:35.575 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:35 smithi104 ceph-mon[36070]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:09:35.576 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:35 smithi104 ceph-mon[36070]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T15:09:35.576 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:35 smithi104 ceph-mon[36070]: osdmap e14: 2 total, 2 up, 2 in 2024-03-18T15:09:37.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:37 smithi104 ceph-mon[36070]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T15:09:37.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:37 smithi104 ceph-mon[36070]: from='client.? 172.21.15.104:0/1540085242' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f383a164-a3a0-49fd-a9e8-9c2178d8cfb7"}]: dispatch 2024-03-18T15:09:37.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:37 smithi104 ceph-mon[36070]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f383a164-a3a0-49fd-a9e8-9c2178d8cfb7"}]: dispatch 2024-03-18T15:09:37.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:37 smithi104 ceph-mon[36070]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f383a164-a3a0-49fd-a9e8-9c2178d8cfb7"}]': finished 2024-03-18T15:09:37.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:37 smithi104 ceph-mon[36070]: osdmap e15: 3 total, 2 up, 3 in 2024-03-18T15:09:37.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:37 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T15:09:37.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:37 smithi019 ceph-mon[31522]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T15:09:37.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:37 smithi019 ceph-mon[31522]: from='client.? 172.21.15.104:0/1540085242' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f383a164-a3a0-49fd-a9e8-9c2178d8cfb7"}]: dispatch 2024-03-18T15:09:37.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:37 smithi019 ceph-mon[31522]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f383a164-a3a0-49fd-a9e8-9c2178d8cfb7"}]: dispatch 2024-03-18T15:09:37.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:37 smithi019 ceph-mon[31522]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f383a164-a3a0-49fd-a9e8-9c2178d8cfb7"}]': finished 2024-03-18T15:09:37.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:37 smithi019 ceph-mon[31522]: osdmap e15: 3 total, 2 up, 3 in 2024-03-18T15:09:37.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:37 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T15:09:37.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:37 smithi033 ceph-mon[36648]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T15:09:37.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:37 smithi033 ceph-mon[36648]: from='client.? 172.21.15.104:0/1540085242' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f383a164-a3a0-49fd-a9e8-9c2178d8cfb7"}]: dispatch 2024-03-18T15:09:37.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:37 smithi033 ceph-mon[36648]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f383a164-a3a0-49fd-a9e8-9c2178d8cfb7"}]: dispatch 2024-03-18T15:09:37.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:37 smithi033 ceph-mon[36648]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f383a164-a3a0-49fd-a9e8-9c2178d8cfb7"}]': finished 2024-03-18T15:09:37.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:37 smithi033 ceph-mon[36648]: osdmap e15: 3 total, 2 up, 3 in 2024-03-18T15:09:37.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:37 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T15:09:38.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:38 smithi033 ceph-mon[36648]: from='client.? 172.21.15.104:0/1963594060' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T15:09:38.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:38 smithi104 ceph-mon[36070]: from='client.? 172.21.15.104:0/1963594060' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T15:09:38.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:38 smithi019 ceph-mon[31522]: from='client.? 172.21.15.104:0/1963594060' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T15:09:39.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:39 smithi104 ceph-mon[36070]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T15:09:39.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:39 smithi033 ceph-mon[36648]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T15:09:39.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:39 smithi019 ceph-mon[31522]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T15:09:41.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:41 smithi033 ceph-mon[36648]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T15:09:41.523 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:41 smithi104 ceph-mon[36070]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T15:09:41.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:41 smithi019 ceph-mon[31522]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T15:09:43.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:43 smithi104 ceph-mon[36070]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T15:09:43.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:43 smithi033 ceph-mon[36648]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T15:09:43.518 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:43 smithi019 ceph-mon[31522]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T15:09:44.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:44 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-18T15:09:44.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:44 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:09:44.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:44 smithi033 ceph-mon[36648]: Deploying daemon osd.2 on smithi104 2024-03-18T15:09:44.564 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:44 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-18T15:09:44.565 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:44 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:09:44.565 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:44 smithi104 ceph-mon[36070]: Deploying daemon osd.2 on smithi104 2024-03-18T15:09:44.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:44 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-18T15:09:44.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:44 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:09:44.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:44 smithi019 ceph-mon[31522]: Deploying daemon osd.2 on smithi104 2024-03-18T15:09:45.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:45 smithi019 ceph-mon[31522]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T15:09:45.490 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:45 smithi104 ceph-mon[36070]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T15:09:45.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:45 smithi033 ceph-mon[36648]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T15:09:47.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:47 smithi033 ceph-mon[36648]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T15:09:47.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:47 smithi104 ceph-mon[36070]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T15:09:47.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:47 smithi019 ceph-mon[31522]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T15:09:50.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:49 smithi033 ceph-mon[36648]: pgmap v82: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T15:09:50.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:49 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:09:50.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:49 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:09:50.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:49 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:09:50.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:49 smithi104 ceph-mon[36070]: pgmap v82: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T15:09:50.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:49 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:09:50.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:49 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:09:50.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:49 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:09:50.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:49 smithi019 ceph-mon[31522]: pgmap v82: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T15:09:50.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:49 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:09:50.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:49 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:09:50.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:49 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:09:52.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:51 smithi104 ceph-mon[36070]: pgmap v83: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T15:09:52.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:51 smithi033 ceph-mon[36648]: pgmap v83: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T15:09:52.346 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:09:52.345+0000 7fd7189eb700 1 -- 172.21.15.104:0/228724624 <== mgr.14152 v2:172.21.15.19:6800/2017714236 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fd714062710 con 0x7fd6fc0a2800 2024-03-18T15:09:52.347 INFO:teuthology.orchestra.run.smithi104.stdout:Created osd(s) 2 on host 'smithi104' 2024-03-18T15:09:52.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:09:52.348+0000 7fd7199ed700 1 -- 172.21.15.104:0/228724624 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fd6fc0a2800 msgr2=0x7fd6fc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:09:52.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:09:52.348+0000 7fd7199ed700 1 --2- 172.21.15.104:0/228724624 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fd6fc0a2800 0x7fd6fc0a4cc0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd7040097b0 tx=0x7fd704006d20 comp rx=0 tx=0).stop 2024-03-18T15:09:52.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:09:52.348+0000 7fd7199ed700 1 -- 172.21.15.104:0/228724624 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd714101200 msgr2=0x7fd7141974c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:09:52.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:09:52.348+0000 7fd7199ed700 1 --2- 172.21.15.104:0/228724624 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd714101200 0x7fd7141974c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fd70800b530 tx=0x7fd70800b8f0 comp rx=0 tx=0).stop 2024-03-18T15:09:52.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:09:52.348+0000 7fd7199ed700 1 -- 172.21.15.104:0/228724624 shutdown_connections 2024-03-18T15:09:52.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:09:52.348+0000 7fd7199ed700 1 --2- 172.21.15.104:0/228724624 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fd6fc0a2800 0x7fd6fc0a4cc0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:09:52.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:09:52.348+0000 7fd7199ed700 1 --2- 172.21.15.104:0/228724624 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd714101200 0x7fd7141974c0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:09:52.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:09:52.348+0000 7fd7199ed700 1 --2- 172.21.15.104:0/228724624 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd714102da0 0x7fd71419ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:09:52.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:09:52.348+0000 7fd7199ed700 1 --2- 172.21.15.104:0/228724624 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd714102400 0x7fd714197a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:09:52.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:09:52.348+0000 7fd7199ed700 1 -- 172.21.15.104:0/228724624 >> 172.21.15.104:0/228724624 conn(0x7fd7140fc7c0 msgr2=0x7fd714105700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:09:52.350 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:09:52.348+0000 7fd7199ed700 1 -- 172.21.15.104:0/228724624 shutdown_connections 2024-03-18T15:09:52.350 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:09:52.348+0000 7fd7199ed700 1 -- 172.21.15.104:0/228724624 wait complete. 2024-03-18T15:09:52.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:51 smithi019 ceph-mon[31522]: pgmap v83: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T15:09:53.005 DEBUG:teuthology.orchestra.run.smithi104:osd.2> sudo journalctl -f -n 0 -u ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@osd.2.service 2024-03-18T15:09:53.008 INFO:tasks.cephadm:Deploying osd.3 on smithi019 with /dev/vg_nvme/lv_3... 2024-03-18T15:09:53.009 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-18T15:09:53.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:53 smithi019 ceph-mon[31522]: pgmap v84: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T15:09:53.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:53 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:09:53.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:53 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:09:53.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:53 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:09:53.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:53 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:09:53.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:53 smithi019 ceph-mon[31522]: from='osd.2 [v2:172.21.15.104:6800/1004727949,v1:172.21.15.104:6801/1004727949]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-18T15:09:53.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:53 smithi019 ceph-mon[31522]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-18T15:09:53.474 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:09:53.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:53 smithi104 ceph-mon[36070]: pgmap v84: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T15:09:53.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:53 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:09:53.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:53 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:09:53.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:53 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:09:53.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:53 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:09:53.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:53 smithi104 ceph-mon[36070]: from='osd.2 [v2:172.21.15.104:6800/1004727949,v1:172.21.15.104:6801/1004727949]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-18T15:09:53.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:53 smithi104 ceph-mon[36070]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-18T15:09:53.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:53 smithi033 ceph-mon[36648]: pgmap v84: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T15:09:53.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:53 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:09:53.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:53 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:09:53.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:53 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:09:53.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:53 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:09:53.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:53 smithi033 ceph-mon[36648]: from='osd.2 [v2:172.21.15.104:6800/1004727949,v1:172.21.15.104:6801/1004727949]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-18T15:09:53.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:53 smithi033 ceph-mon[36648]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-18T15:09:54.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:54 smithi019 ceph-mon[31522]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-18T15:09:54.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:54 smithi019 ceph-mon[31522]: osdmap e16: 3 total, 2 up, 3 in 2024-03-18T15:09:54.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:54 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T15:09:54.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:54 smithi019 ceph-mon[31522]: from='osd.2 [v2:172.21.15.104:6800/1004727949,v1:172.21.15.104:6801/1004727949]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-03-18T15:09:54.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:54 smithi019 ceph-mon[31522]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-03-18T15:09:54.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:54 smithi033 ceph-mon[36648]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-18T15:09:54.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:54 smithi033 ceph-mon[36648]: osdmap e16: 3 total, 2 up, 3 in 2024-03-18T15:09:54.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:54 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T15:09:54.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:54 smithi033 ceph-mon[36648]: from='osd.2 [v2:172.21.15.104:6800/1004727949,v1:172.21.15.104:6801/1004727949]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-03-18T15:09:54.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:54 smithi033 ceph-mon[36648]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-03-18T15:09:54.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:54 smithi104 ceph-mon[36070]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-18T15:09:54.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:54 smithi104 ceph-mon[36070]: osdmap e16: 3 total, 2 up, 3 in 2024-03-18T15:09:54.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:54 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T15:09:54.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:54 smithi104 ceph-mon[36070]: from='osd.2 [v2:172.21.15.104:6800/1004727949,v1:172.21.15.104:6801/1004727949]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-03-18T15:09:54.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:54 smithi104 ceph-mon[36070]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-03-18T15:09:54.945 INFO:journalctl@ceph.osd.2.smithi104.stdout:Mar 18 15:09:54 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-2[41206]: 2024-03-18T15:09:54.454+0000 7f0f87d59700 -1 osd.2 0 waiting for initial osdmap 2024-03-18T15:09:54.945 INFO:journalctl@ceph.osd.2.smithi104.stdout:Mar 18 15:09:54 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-2[41206]: 2024-03-18T15:09:54.459+0000 7f0f83b4c700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T15:09:55.673 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:55 smithi104 ceph-mon[36070]: pgmap v86: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T15:09:55.674 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:55 smithi104 ceph-mon[36070]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2024-03-18T15:09:55.674 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:55 smithi104 ceph-mon[36070]: osdmap e17: 3 total, 2 up, 3 in 2024-03-18T15:09:55.674 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:55 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T15:09:55.674 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:55 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T15:09:55.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:55 smithi033 ceph-mon[36648]: pgmap v86: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T15:09:55.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:55 smithi033 ceph-mon[36648]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2024-03-18T15:09:55.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:55 smithi033 ceph-mon[36648]: osdmap e17: 3 total, 2 up, 3 in 2024-03-18T15:09:55.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:55 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T15:09:55.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:55 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T15:09:55.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:55 smithi019 ceph-mon[31522]: pgmap v86: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T15:09:55.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:55 smithi019 ceph-mon[31522]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2024-03-18T15:09:55.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:55 smithi019 ceph-mon[31522]: osdmap e17: 3 total, 2 up, 3 in 2024-03-18T15:09:55.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:55 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T15:09:55.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:55 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T15:09:56.489 INFO:teuthology.orchestra.run.smithi019.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-18T15:09:56.489 INFO:teuthology.orchestra.run.smithi019.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-18T15:09:56.489 INFO:teuthology.orchestra.run.smithi019.stderr: stderr: 10+0 records in 2024-03-18T15:09:56.489 INFO:teuthology.orchestra.run.smithi019.stderr:10+0 records out 2024-03-18T15:09:56.489 INFO:teuthology.orchestra.run.smithi019.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0366065 s, 286 MB/s 2024-03-18T15:09:56.490 INFO:teuthology.orchestra.run.smithi019.stderr:--> Zapping successful for: 2024-03-18T15:09:57.189 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:56 smithi019 ceph-mon[31522]: purged_snaps scrub starts 2024-03-18T15:09:57.189 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:56 smithi019 ceph-mon[31522]: purged_snaps scrub ok 2024-03-18T15:09:57.189 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:56 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T15:09:57.190 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:56 smithi019 ceph-mon[31522]: osd.2 [v2:172.21.15.104:6800/1004727949,v1:172.21.15.104:6801/1004727949] boot 2024-03-18T15:09:57.190 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:56 smithi019 ceph-mon[31522]: osdmap e18: 3 total, 3 up, 3 in 2024-03-18T15:09:57.190 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:56 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T15:09:57.190 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:56 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:09:57.190 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:56 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:09:57.190 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:56 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:09:57.190 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:56 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:09:57.190 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:56 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:09:57.190 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:56 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:09:57.190 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:56 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:09:57.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:56 smithi104 ceph-mon[36070]: purged_snaps scrub starts 2024-03-18T15:09:57.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:56 smithi104 ceph-mon[36070]: purged_snaps scrub ok 2024-03-18T15:09:57.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:56 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T15:09:57.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:56 smithi104 ceph-mon[36070]: osd.2 [v2:172.21.15.104:6800/1004727949,v1:172.21.15.104:6801/1004727949] boot 2024-03-18T15:09:57.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:56 smithi104 ceph-mon[36070]: osdmap e18: 3 total, 3 up, 3 in 2024-03-18T15:09:57.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:56 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T15:09:57.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:56 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:09:57.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:56 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:09:57.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:56 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:09:57.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:56 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:09:57.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:56 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:09:57.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:56 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:09:57.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:56 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:09:57.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:56 smithi033 ceph-mon[36648]: purged_snaps scrub starts 2024-03-18T15:09:57.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:56 smithi033 ceph-mon[36648]: purged_snaps scrub ok 2024-03-18T15:09:57.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:56 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T15:09:57.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:56 smithi033 ceph-mon[36648]: osd.2 [v2:172.21.15.104:6800/1004727949,v1:172.21.15.104:6801/1004727949] boot 2024-03-18T15:09:57.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:56 smithi033 ceph-mon[36648]: osdmap e18: 3 total, 3 up, 3 in 2024-03-18T15:09:57.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:56 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T15:09:57.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:56 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:09:57.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:56 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:09:57.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:56 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:09:57.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:56 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:09:57.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:56 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:09:57.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:56 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:09:57.258 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:56 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:09:57.264 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi019:vg_nvme/lv_3 2024-03-18T15:09:57.715 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:09:58.184 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:57 smithi019 ceph-mon[31522]: Detected new or changed devices on smithi104 2024-03-18T15:09:58.185 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:57 smithi019 ceph-mon[31522]: Adjusting osd_memory_target on smithi104 to 17080M 2024-03-18T15:09:58.185 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:57 smithi019 ceph-mon[31522]: pgmap v89: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T15:09:58.185 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:57 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-18T15:09:58.185 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:57 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:09:58.185 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:57 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:09:58.185 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:57 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:09:58.185 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:57 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-18T15:09:58.185 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:57 smithi019 ceph-mon[31522]: osdmap e19: 3 total, 3 up, 3 in 2024-03-18T15:09:58.185 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:57 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-18T15:09:58.185 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:57 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:09:58.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:57 smithi104 ceph-mon[36070]: Detected new or changed devices on smithi104 2024-03-18T15:09:58.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:57 smithi104 ceph-mon[36070]: Adjusting osd_memory_target on smithi104 to 17080M 2024-03-18T15:09:58.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:57 smithi104 ceph-mon[36070]: pgmap v89: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T15:09:58.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:57 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-18T15:09:58.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:57 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:09:58.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:57 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:09:58.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:57 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:09:58.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:57 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-18T15:09:58.196 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:57 smithi104 ceph-mon[36070]: osdmap e19: 3 total, 3 up, 3 in 2024-03-18T15:09:58.196 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:57 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-18T15:09:58.196 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:57 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:09:58.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:57 smithi033 ceph-mon[36648]: Detected new or changed devices on smithi104 2024-03-18T15:09:58.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:57 smithi033 ceph-mon[36648]: Adjusting osd_memory_target on smithi104 to 17080M 2024-03-18T15:09:58.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:57 smithi033 ceph-mon[36648]: pgmap v89: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T15:09:58.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:57 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-18T15:09:58.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:57 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:09:58.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:57 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:09:58.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:57 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:09:58.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:57 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-18T15:09:58.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:57 smithi033 ceph-mon[36648]: osdmap e19: 3 total, 3 up, 3 in 2024-03-18T15:09:58.258 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:57 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-18T15:09:58.258 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:57 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:09:58.882 INFO:journalctl@ceph.osd.0.smithi019.stdout:Mar 18 15:09:58 smithi019 sudo[50477]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-18T15:09:58.882 INFO:journalctl@ceph.osd.0.smithi019.stdout:Mar 18 15:09:58 smithi019 sudo[50477]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T15:09:58.882 INFO:journalctl@ceph.osd.0.smithi019.stdout:Mar 18 15:09:58 smithi019 sudo[50477]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T15:09:58.882 INFO:journalctl@ceph.osd.0.smithi019.stdout:Mar 18 15:09:58 smithi019 sudo[50477]: pam_unix(sudo:session): session closed for user root 2024-03-18T15:09:59.193 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:58 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-18T15:09:59.194 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:58 smithi019 ceph-mon[31522]: osdmap e20: 3 total, 3 up, 3 in 2024-03-18T15:09:59.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:58 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-18T15:09:59.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:58 smithi104 ceph-mon[36070]: osdmap e20: 3 total, 3 up, 3 in 2024-03-18T15:09:59.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:58 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-18T15:09:59.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:58 smithi033 ceph-mon[36648]: osdmap e20: 3 total, 3 up, 3 in 2024-03-18T15:09:59.448 INFO:journalctl@ceph.osd.0.smithi019.stdout:Mar 18 15:09:59 smithi019 sudo[50497]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-18T15:09:59.448 INFO:journalctl@ceph.osd.0.smithi019.stdout:Mar 18 15:09:59 smithi019 sudo[50497]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T15:09:59.448 INFO:journalctl@ceph.osd.0.smithi019.stdout:Mar 18 15:09:59 smithi019 sudo[50497]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T15:09:59.448 INFO:journalctl@ceph.osd.0.smithi019.stdout:Mar 18 15:09:59 smithi019 sudo[50497]: pam_unix(sudo:session): session closed for user root 2024-03-18T15:09:59.769 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:09:59.767+0000 7f82dcc00700 1 -- 172.21.15.19:0/3488461356 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f82d8076280 msgr2=0x7f82d80746e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:09:59.769 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:09:59.767+0000 7f82dcc00700 1 --2- 172.21.15.19:0/3488461356 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f82d8076280 0x7f82d80746e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f82c8009990 tx=0x7f82c8009ca0 comp rx=0 tx=0).stop 2024-03-18T15:09:59.769 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:09:59.767+0000 7f82dcc00700 1 -- 172.21.15.19:0/3488461356 shutdown_connections 2024-03-18T15:09:59.769 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:09:59.767+0000 7f82dcc00700 1 --2- 172.21.15.19:0/3488461356 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f82d810a200 0x7f82d810c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:09:59.770 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:09:59.767+0000 7f82dcc00700 1 --2- 172.21.15.19:0/3488461356 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f82d8076280 0x7f82d80746e0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:09:59.770 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:09:59.767+0000 7f82dcc00700 1 --2- 172.21.15.19:0/3488461356 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f82d8074cb0 0x7f82d8075130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:09:59.770 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:09:59.767+0000 7f82dcc00700 1 -- 172.21.15.19:0/3488461356 >> 172.21.15.19:0/3488461356 conn(0x7f82d80fd8e0 msgr2=0x7f82d80ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:09:59.770 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:09:59.768+0000 7f82dcc00700 1 -- 172.21.15.19:0/3488461356 shutdown_connections 2024-03-18T15:09:59.770 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:09:59.768+0000 7f82dcc00700 1 -- 172.21.15.19:0/3488461356 wait complete. 2024-03-18T15:09:59.771 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:09:59.769+0000 7f82dcc00700 1 Processor -- start 2024-03-18T15:09:59.771 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:09:59.769+0000 7f82dcc00700 1 -- start start 2024-03-18T15:09:59.771 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:09:59.770+0000 7f82dcc00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f82d8074cb0 0x7f82d819a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:09:59.771 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:09:59.770+0000 7f82dcc00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f82d8076280 0x7f82d819ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:09:59.771 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:09:59.770+0000 7f82dcc00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f82d810a200 0x7f82d81a1d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:09:59.771 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:09:59.770+0000 7f82dcc00700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f82d8110110 con 0x7f82d8074cb0 2024-03-18T15:09:59.771 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:09:59.770+0000 7f82dcc00700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f82d810ff90 con 0x7f82d810a200 2024-03-18T15:09:59.772 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:09:59.770+0000 7f82dcc00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f82d8110290 con 0x7f82d8076280 2024-03-18T15:09:59.772 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:09:59.770+0000 7f82d5e00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f82d8074cb0 0x7f82d819a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:09:59.772 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:09:59.770+0000 7f82d5e00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f82d8074cb0 0x7f82d819a740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:52826/0 (socket says 172.21.15.19:52826) 2024-03-18T15:09:59.772 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:09:59.770+0000 7f82d5e00700 1 -- 172.21.15.19:0/2763442095 learned_addr learned my addr 172.21.15.19:0/2763442095 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:09:59.772 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:09:59.770+0000 7f82d5400700 1 --2- 172.21.15.19:0/2763442095 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f82d8076280 0x7f82d819ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:09:59.772 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:09:59.770+0000 7f82d6800700 1 --2- 172.21.15.19:0/2763442095 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f82d810a200 0x7f82d81a1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:09:59.773 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:09:59.771+0000 7f82d5e00700 1 -- 172.21.15.19:0/2763442095 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f82d8076280 msgr2=0x7f82d819ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:09:59.773 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:09:59.771+0000 7f82d5e00700 1 --2- 172.21.15.19:0/2763442095 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f82d8076280 0x7f82d819ac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:09:59.773 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:09:59.771+0000 7f82d5e00700 1 -- 172.21.15.19:0/2763442095 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f82d810a200 msgr2=0x7f82d81a1d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:09:59.773 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:09:59.771+0000 7f82d5e00700 1 --2- 172.21.15.19:0/2763442095 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f82d810a200 0x7f82d81a1d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:09:59.773 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:09:59.771+0000 7f82d5e00700 1 -- 172.21.15.19:0/2763442095 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82d81a23a0 con 0x7f82d8074cb0 2024-03-18T15:09:59.773 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:09:59.771+0000 7f82d5e00700 1 --2- 172.21.15.19:0/2763442095 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f82d8074cb0 0x7f82d819a740 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f82c8005910 tx=0x7f82c800bbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:09:59.773 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:09:59.771+0000 7f82cea00700 1 -- 172.21.15.19:0/2763442095 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82c8026070 con 0x7f82d8074cb0 2024-03-18T15:09:59.773 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:09:59.771+0000 7f82cea00700 1 -- 172.21.15.19:0/2763442095 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f82c8003d70 con 0x7f82d8074cb0 2024-03-18T15:09:59.773 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:09:59.771+0000 7f82dcc00700 1 -- 172.21.15.19:0/2763442095 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82d81a2630 con 0x7f82d8074cb0 2024-03-18T15:09:59.773 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:09:59.771+0000 7f82cea00700 1 -- 172.21.15.19:0/2763442095 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82c8021410 con 0x7f82d8074cb0 2024-03-18T15:09:59.773 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:09:59.771+0000 7f82dcc00700 1 -- 172.21.15.19:0/2763442095 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82d81a2a80 con 0x7f82d8074cb0 2024-03-18T15:09:59.774 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:09:59.773+0000 7f82dcc00700 1 -- 172.21.15.19:0/2763442095 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82d8068f90 con 0x7f82d8074cb0 2024-03-18T15:09:59.776 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:09:59.775+0000 7f82cea00700 1 -- 172.21.15.19:0/2763442095 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f82c8029070 con 0x7f82d8074cb0 2024-03-18T15:09:59.777 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:09:59.776+0000 7f82cea00700 1 --2- 172.21.15.19:0/2763442095 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f82bc0a2890 0x7f82bc0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:09:59.778 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:09:59.776+0000 7f82cea00700 1 -- 172.21.15.19:0/2763442095 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 3237+0+0 (secure 0 0 0) 0x7f82c80ce600 con 0x7f82d8074cb0 2024-03-18T15:09:59.778 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:09:59.777+0000 7f82d5400700 1 --2- 172.21.15.19:0/2763442095 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f82bc0a2890 0x7f82bc0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:09:59.778 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:09:59.777+0000 7f82d5400700 1 --2- 172.21.15.19:0/2763442095 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f82bc0a2890 0x7f82bc0a4d50 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f82c4009c60 tx=0x7f82c4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:09:59.782 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:09:59.781+0000 7f82cea00700 1 -- 172.21.15.19:0/2763442095 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f82c805d740 con 0x7f82d8074cb0 2024-03-18T15:10:00.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:59 smithi033 ceph-mon[36648]: pgmap v92: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-18T15:10:00.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:59 smithi033 ceph-mon[36648]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-18T15:10:00.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:59 smithi033 ceph-mon[36648]: osdmap e21: 3 total, 3 up, 3 in 2024-03-18T15:10:00.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:09:59 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:10:00.007 INFO:journalctl@ceph.osd.1.smithi033.stdout:Mar 18 15:09:59 smithi033 sudo[45001]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-18T15:10:00.007 INFO:journalctl@ceph.osd.1.smithi033.stdout:Mar 18 15:09:59 smithi033 sudo[45001]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T15:10:00.007 INFO:journalctl@ceph.osd.1.smithi033.stdout:Mar 18 15:09:59 smithi033 sudo[45001]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T15:10:00.007 INFO:journalctl@ceph.osd.1.smithi033.stdout:Mar 18 15:09:59 smithi033 sudo[45001]: pam_unix(sudo:session): session closed for user root 2024-03-18T15:10:00.019 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:10:00.018+0000 7f82dcc00700 1 -- 172.21.15.19:0/2763442095 --> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi019:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f82d8062710 con 0x7f82bc0a2890 2024-03-18T15:10:00.020 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:59 smithi019 ceph-mon[31522]: pgmap v92: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-18T15:10:00.020 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:59 smithi019 ceph-mon[31522]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-18T15:10:00.020 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:59 smithi019 ceph-mon[31522]: osdmap e21: 3 total, 3 up, 3 in 2024-03-18T15:10:00.020 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:09:59 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:10:00.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:59 smithi104 ceph-mon[36070]: pgmap v92: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-18T15:10:00.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:59 smithi104 ceph-mon[36070]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-18T15:10:00.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:59 smithi104 ceph-mon[36070]: osdmap e21: 3 total, 3 up, 3 in 2024-03-18T15:10:00.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:09:59 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:10:00.507 INFO:journalctl@ceph.osd.1.smithi033.stdout:Mar 18 15:10:00 smithi033 sudo[45004]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-18T15:10:00.507 INFO:journalctl@ceph.osd.1.smithi033.stdout:Mar 18 15:10:00 smithi033 sudo[45004]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T15:10:00.507 INFO:journalctl@ceph.osd.1.smithi033.stdout:Mar 18 15:10:00 smithi033 sudo[45004]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T15:10:00.507 INFO:journalctl@ceph.osd.1.smithi033.stdout:Mar 18 15:10:00 smithi033 sudo[45004]: pam_unix(sudo:session): session closed for user root 2024-03-18T15:10:00.663 INFO:journalctl@ceph.osd.2.smithi104.stdout:Mar 18 15:10:00 smithi104 sudo[45268]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-18T15:10:00.663 INFO:journalctl@ceph.osd.2.smithi104.stdout:Mar 18 15:10:00 smithi104 sudo[45268]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T15:10:00.663 INFO:journalctl@ceph.osd.2.smithi104.stdout:Mar 18 15:10:00 smithi104 sudo[45268]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T15:10:00.663 INFO:journalctl@ceph.osd.2.smithi104.stdout:Mar 18 15:10:00 smithi104 sudo[45268]: pam_unix(sudo:session): session closed for user root 2024-03-18T15:10:00.944 INFO:journalctl@ceph.osd.2.smithi104.stdout:Mar 18 15:10:00 smithi104 sudo[45271]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-18T15:10:00.944 INFO:journalctl@ceph.osd.2.smithi104.stdout:Mar 18 15:10:00 smithi104 sudo[45271]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T15:10:00.944 INFO:journalctl@ceph.osd.2.smithi104.stdout:Mar 18 15:10:00 smithi104 sudo[45271]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T15:10:00.945 INFO:journalctl@ceph.osd.2.smithi104.stdout:Mar 18 15:10:00 smithi104 sudo[45271]: pam_unix(sudo:session): session closed for user root 2024-03-18T15:10:01.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:01 smithi019 sudo[50653]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-18T15:10:01.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:01 smithi019 sudo[50653]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T15:10:01.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:01 smithi019 sudo[50653]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T15:10:01.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:01 smithi019 sudo[50653]: pam_unix(sudo:session): session closed for user root 2024-03-18T15:10:02.111 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:01 smithi104 ceph-mon[36070]: overall HEALTH_OK 2024-03-18T15:10:02.111 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:01 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T15:10:02.111 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:01 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T15:10:02.111 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:01 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:10:02.111 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:01 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:10:02.111 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:01 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:10:02.111 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:01 smithi104 ceph-mon[36070]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-18T15:10:02.111 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:02 smithi104 sudo[45274]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-18T15:10:02.112 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:02 smithi104 sudo[45274]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T15:10:02.112 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:02 smithi104 sudo[45274]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T15:10:02.112 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:02 smithi104 sudo[45274]: pam_unix(sudo:session): session closed for user root 2024-03-18T15:10:02.179 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:01 smithi019 sudo[50672]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-18T15:10:02.179 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:01 smithi019 sudo[50672]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T15:10:02.180 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:01 smithi019 sudo[50672]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T15:10:02.180 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:01 smithi019 sudo[50672]: pam_unix(sudo:session): session closed for user root 2024-03-18T15:10:02.180 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:01 smithi019 ceph-mon[31522]: overall HEALTH_OK 2024-03-18T15:10:02.180 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:01 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T15:10:02.180 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:01 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T15:10:02.180 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:01 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:10:02.180 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:01 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:10:02.180 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:01 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:10:02.180 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:01 smithi019 ceph-mon[31522]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-18T15:10:02.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:01 smithi033 ceph-mon[36648]: overall HEALTH_OK 2024-03-18T15:10:02.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:01 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T15:10:02.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:01 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T15:10:02.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:01 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:10:02.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:01 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:10:02.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:01 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:10:02.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:01 smithi033 ceph-mon[36648]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-18T15:10:02.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:02 smithi104 sudo[45277]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-18T15:10:02.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:02 smithi104 sudo[45277]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T15:10:02.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:02 smithi104 sudo[45277]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T15:10:02.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:02 smithi104 sudo[45277]: pam_unix(sudo:session): session closed for user root 2024-03-18T15:10:03.226 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:02 smithi033 sudo[45007]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-18T15:10:03.226 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:02 smithi033 sudo[45007]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T15:10:03.226 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:02 smithi033 sudo[45007]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T15:10:03.226 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:02 smithi033 sudo[45007]: pam_unix(sudo:session): session closed for user root 2024-03-18T15:10:03.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:03 smithi033 sudo[45011]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-18T15:10:03.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:03 smithi033 sudo[45011]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T15:10:03.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:03 smithi033 sudo[45011]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T15:10:03.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:03 smithi033 sudo[45011]: pam_unix(sudo:session): session closed for user root 2024-03-18T15:10:03.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:03 smithi033 ceph-mon[36648]: from='client.14250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi019:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:10:03.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:03 smithi033 ceph-mon[36648]: pgmap v94: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-18T15:10:03.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:03 smithi033 ceph-mon[36648]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-18T15:10:03.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:03 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T15:10:03.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:03 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T15:10:03.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:03 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T15:10:03.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:03 smithi033 ceph-mon[36648]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-18T15:10:03.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:03 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T15:10:03.508 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:03 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T15:10:03.508 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:03 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T15:10:03.508 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:03 smithi033 ceph-mon[36648]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-18T15:10:03.508 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:03 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T15:10:03.508 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:03 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T15:10:03.508 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:03 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T15:10:03.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:03 smithi104 ceph-mon[36070]: from='client.14250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi019:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:10:03.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:03 smithi104 ceph-mon[36070]: pgmap v94: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-18T15:10:03.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:03 smithi104 ceph-mon[36070]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-18T15:10:03.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:03 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T15:10:03.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:03 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T15:10:03.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:03 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T15:10:03.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:03 smithi104 ceph-mon[36070]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-18T15:10:03.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:03 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T15:10:03.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:03 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T15:10:03.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:03 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T15:10:03.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:03 smithi104 ceph-mon[36070]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-18T15:10:03.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:03 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T15:10:03.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:03 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T15:10:03.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:03 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T15:10:03.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:03 smithi019 ceph-mon[31522]: from='client.14250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi019:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:10:03.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:03 smithi019 ceph-mon[31522]: pgmap v94: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-18T15:10:03.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:03 smithi019 ceph-mon[31522]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-18T15:10:03.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:03 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T15:10:03.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:03 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T15:10:03.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:03 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T15:10:03.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:03 smithi019 ceph-mon[31522]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-18T15:10:03.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:03 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T15:10:03.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:03 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T15:10:03.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:03 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T15:10:03.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:03 smithi019 ceph-mon[31522]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-18T15:10:03.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:03 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T15:10:03.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:03 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T15:10:03.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:03 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T15:10:04.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:04 smithi033 ceph-mon[36648]: pgmap v95: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-18T15:10:04.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:04 smithi033 ceph-mon[36648]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-18T15:10:04.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:04 smithi033 ceph-mon[36648]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-18T15:10:04.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:04 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/1923922330' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c409e116-28bf-45dd-a791-539af22abe1e"}]: dispatch 2024-03-18T15:10:04.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:04 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/1923922330' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c409e116-28bf-45dd-a791-539af22abe1e"}]': finished 2024-03-18T15:10:04.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:04 smithi033 ceph-mon[36648]: osdmap e22: 4 total, 3 up, 4 in 2024-03-18T15:10:04.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:04 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T15:10:04.597 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:04 smithi019 ceph-mon[31522]: pgmap v95: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-18T15:10:04.597 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:04 smithi019 ceph-mon[31522]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-18T15:10:04.597 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:04 smithi019 ceph-mon[31522]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-18T15:10:04.597 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:04 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/1923922330' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c409e116-28bf-45dd-a791-539af22abe1e"}]: dispatch 2024-03-18T15:10:04.597 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:04 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/1923922330' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c409e116-28bf-45dd-a791-539af22abe1e"}]': finished 2024-03-18T15:10:04.597 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:04 smithi019 ceph-mon[31522]: osdmap e22: 4 total, 3 up, 4 in 2024-03-18T15:10:04.597 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:04 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T15:10:04.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:04 smithi104 ceph-mon[36070]: pgmap v95: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-18T15:10:04.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:04 smithi104 ceph-mon[36070]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-18T15:10:04.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:04 smithi104 ceph-mon[36070]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-18T15:10:04.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:04 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/1923922330' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c409e116-28bf-45dd-a791-539af22abe1e"}]: dispatch 2024-03-18T15:10:04.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:04 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/1923922330' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c409e116-28bf-45dd-a791-539af22abe1e"}]': finished 2024-03-18T15:10:04.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:04 smithi104 ceph-mon[36070]: osdmap e22: 4 total, 3 up, 4 in 2024-03-18T15:10:04.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:04 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T15:10:05.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:05 smithi104 ceph-mon[36070]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-18T15:10:05.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:05 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/2229366780' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T15:10:05.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:05 smithi019 ceph-mon[31522]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-18T15:10:05.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:05 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/2229366780' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T15:10:05.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:05 smithi033 ceph-mon[36648]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-18T15:10:05.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:05 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/2229366780' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T15:10:07.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:07 smithi104 ceph-mon[36070]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-18T15:10:07.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:07 smithi019 ceph-mon[31522]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-18T15:10:07.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:07 smithi033 ceph-mon[36648]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-18T15:10:09.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:09 smithi019 ceph-mon[31522]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 2.4 KiB/s rd, 62 KiB/s wr, 5 op/s 2024-03-18T15:10:09.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:09 smithi104 ceph-mon[36070]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 2.4 KiB/s rd, 62 KiB/s wr, 5 op/s 2024-03-18T15:10:09.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:09 smithi033 ceph-mon[36648]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 2.4 KiB/s rd, 62 KiB/s wr, 5 op/s 2024-03-18T15:10:11.430 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:11 smithi019 ceph-mon[31522]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 1.6 KiB/s rd, 42 KiB/s wr, 4 op/s 2024-03-18T15:10:11.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:11 smithi104 ceph-mon[36070]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 1.6 KiB/s rd, 42 KiB/s wr, 4 op/s 2024-03-18T15:10:11.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:11 smithi033 ceph-mon[36648]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 1.6 KiB/s rd, 42 KiB/s wr, 4 op/s 2024-03-18T15:10:13.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:13 smithi104 ceph-mon[36070]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 1.2 KiB/s rd, 32 KiB/s wr, 3 op/s 2024-03-18T15:10:13.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:13 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-18T15:10:13.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:13 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:10:13.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:13 smithi019 ceph-mon[31522]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 1.2 KiB/s rd, 32 KiB/s wr, 3 op/s 2024-03-18T15:10:13.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:13 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-18T15:10:13.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:13 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:10:13.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:13 smithi033 ceph-mon[36648]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 1.2 KiB/s rd, 32 KiB/s wr, 3 op/s 2024-03-18T15:10:13.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:13 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-18T15:10:13.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:13 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:10:14.377 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:14 smithi019 ceph-mon[31522]: Deploying daemon osd.3 on smithi019 2024-03-18T15:10:14.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:14 smithi104 ceph-mon[36070]: Deploying daemon osd.3 on smithi019 2024-03-18T15:10:14.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:14 smithi033 ceph-mon[36648]: Deploying daemon osd.3 on smithi019 2024-03-18T15:10:15.680 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:15 smithi019 ceph-mon[31522]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 1004 B/s rd, 26 KiB/s wr, 2 op/s 2024-03-18T15:10:15.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:15 smithi104 ceph-mon[36070]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 1004 B/s rd, 26 KiB/s wr, 2 op/s 2024-03-18T15:10:15.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:15 smithi033 ceph-mon[36648]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 1004 B/s rd, 26 KiB/s wr, 2 op/s 2024-03-18T15:10:17.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:17 smithi019 ceph-mon[31522]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2024-03-18T15:10:17.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:17 smithi104 ceph-mon[36070]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2024-03-18T15:10:17.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:17 smithi033 ceph-mon[36648]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2024-03-18T15:10:19.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:19 smithi104 ceph-mon[36070]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2024-03-18T15:10:19.695 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:19 smithi019 ceph-mon[31522]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2024-03-18T15:10:19.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:19 smithi033 ceph-mon[36648]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2024-03-18T15:10:21.605 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:21 smithi019 ceph-mon[31522]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-18T15:10:21.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:21 smithi104 ceph-mon[36070]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-18T15:10:21.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:21 smithi033 ceph-mon[36648]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-18T15:10:22.551 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:22 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:10:22.551 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:22 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:10:22.551 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:22 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:10:22.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:22 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:10:22.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:22 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:10:22.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:22 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:10:22.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:22 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:10:22.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:22 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:10:22.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:22 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:10:23.380 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:23 smithi019 ceph-mon[31522]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-18T15:10:23.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:23 smithi104 ceph-mon[36070]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-18T15:10:23.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:23 smithi033 ceph-mon[36648]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-18T15:10:25.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:25 smithi019 ceph-mon[31522]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-18T15:10:25.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:25 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:10:25.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:25 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:10:25.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:25 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:10:25.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:25 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:10:25.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:25 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:10:25.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:25 smithi019 ceph-mon[31522]: from='osd.3 [v2:172.21.15.19:6810/3571525854,v1:172.21.15.19:6811/3571525854]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-18T15:10:26.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:25 smithi104 ceph-mon[36070]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-18T15:10:26.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:25 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:10:26.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:25 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:10:26.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:25 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:10:26.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:25 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:10:26.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:25 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:10:26.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:25 smithi104 ceph-mon[36070]: from='osd.3 [v2:172.21.15.19:6810/3571525854,v1:172.21.15.19:6811/3571525854]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-18T15:10:26.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:25 smithi033 ceph-mon[36648]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-18T15:10:26.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:25 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:10:26.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:25 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:10:26.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:25 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:10:26.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:25 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:10:26.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:25 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:10:26.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:25 smithi033 ceph-mon[36648]: from='osd.3 [v2:172.21.15.19:6810/3571525854,v1:172.21.15.19:6811/3571525854]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-18T15:10:26.571 INFO:teuthology.orchestra.run.smithi019.stdout:Created osd(s) 3 on host 'smithi019' 2024-03-18T15:10:26.572 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:10:26.569+0000 7f82cea00700 1 -- 172.21.15.19:0/2763442095 <== mgr.14152 v2:172.21.15.19:6800/2017714236 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f82d8062710 con 0x7f82bc0a2890 2024-03-18T15:10:26.575 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:10:26.573+0000 7f82dcc00700 1 -- 172.21.15.19:0/2763442095 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f82bc0a2890 msgr2=0x7f82bc0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:10:26.575 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:10:26.573+0000 7f82dcc00700 1 --2- 172.21.15.19:0/2763442095 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f82bc0a2890 0x7f82bc0a4d50 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f82c4009c60 tx=0x7f82c4009260 comp rx=0 tx=0).stop 2024-03-18T15:10:26.575 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:10:26.573+0000 7f82dcc00700 1 -- 172.21.15.19:0/2763442095 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f82d8074cb0 msgr2=0x7f82d819a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:10:26.575 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:10:26.573+0000 7f82dcc00700 1 --2- 172.21.15.19:0/2763442095 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f82d8074cb0 0x7f82d819a740 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f82c8005910 tx=0x7f82c800bbb0 comp rx=0 tx=0).stop 2024-03-18T15:10:26.575 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:10:26.574+0000 7f82dcc00700 1 -- 172.21.15.19:0/2763442095 shutdown_connections 2024-03-18T15:10:26.576 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:10:26.574+0000 7f82dcc00700 1 --2- 172.21.15.19:0/2763442095 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f82bc0a2890 0x7f82bc0a4d50 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:10:26.576 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:10:26.574+0000 7f82dcc00700 1 --2- 172.21.15.19:0/2763442095 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f82d810a200 0x7f82d81a1d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:10:26.576 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:10:26.574+0000 7f82dcc00700 1 --2- 172.21.15.19:0/2763442095 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f82d8074cb0 0x7f82d819a740 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:10:26.576 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:10:26.574+0000 7f82dcc00700 1 --2- 172.21.15.19:0/2763442095 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f82d8076280 0x7f82d819ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:10:26.576 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:10:26.574+0000 7f82dcc00700 1 -- 172.21.15.19:0/2763442095 >> 172.21.15.19:0/2763442095 conn(0x7f82d80fd8e0 msgr2=0x7f82d8106150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:10:26.576 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:10:26.574+0000 7f82dcc00700 1 -- 172.21.15.19:0/2763442095 shutdown_connections 2024-03-18T15:10:26.576 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:10:26.574+0000 7f82dcc00700 1 -- 172.21.15.19:0/2763442095 wait complete. 2024-03-18T15:10:27.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:26 smithi104 ceph-mon[36070]: from='osd.3 [v2:172.21.15.19:6810/3571525854,v1:172.21.15.19:6811/3571525854]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-18T15:10:27.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:26 smithi104 ceph-mon[36070]: osdmap e23: 4 total, 3 up, 4 in 2024-03-18T15:10:27.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:26 smithi104 ceph-mon[36070]: from='osd.3 [v2:172.21.15.19:6810/3571525854,v1:172.21.15.19:6811/3571525854]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]: dispatch 2024-03-18T15:10:27.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:26 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T15:10:27.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:26 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:10:27.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:26 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:10:27.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:26 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:10:27.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:26 smithi019 ceph-mon[31522]: from='osd.3 [v2:172.21.15.19:6810/3571525854,v1:172.21.15.19:6811/3571525854]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-18T15:10:27.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:26 smithi019 ceph-mon[31522]: osdmap e23: 4 total, 3 up, 4 in 2024-03-18T15:10:27.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:26 smithi019 ceph-mon[31522]: from='osd.3 [v2:172.21.15.19:6810/3571525854,v1:172.21.15.19:6811/3571525854]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]: dispatch 2024-03-18T15:10:27.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:26 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T15:10:27.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:26 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:10:27.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:26 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:10:27.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:26 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:10:27.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:26 smithi033 ceph-mon[36648]: from='osd.3 [v2:172.21.15.19:6810/3571525854,v1:172.21.15.19:6811/3571525854]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-18T15:10:27.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:26 smithi033 ceph-mon[36648]: osdmap e23: 4 total, 3 up, 4 in 2024-03-18T15:10:27.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:26 smithi033 ceph-mon[36648]: from='osd.3 [v2:172.21.15.19:6810/3571525854,v1:172.21.15.19:6811/3571525854]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]: dispatch 2024-03-18T15:10:27.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:26 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T15:10:27.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:26 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:10:27.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:26 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:10:27.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:26 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:10:27.364 DEBUG:teuthology.orchestra.run.smithi019:osd.3> sudo journalctl -f -n 0 -u ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@osd.3.service 2024-03-18T15:10:27.372 INFO:tasks.cephadm:Deploying osd.4 on smithi033 with /dev/vg_nvme/lv_3... 2024-03-18T15:10:27.372 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-18T15:10:27.630 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T15:10:28.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:27 smithi033 ceph-mon[36648]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-18T15:10:28.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:27 smithi033 ceph-mon[36648]: from='osd.3 [v2:172.21.15.19:6810/3571525854,v1:172.21.15.19:6811/3571525854]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]': finished 2024-03-18T15:10:28.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:27 smithi033 ceph-mon[36648]: osdmap e24: 4 total, 3 up, 4 in 2024-03-18T15:10:28.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:27 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T15:10:28.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:27 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T15:10:28.073 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:27 smithi019 ceph-mon[31522]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-18T15:10:28.073 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:27 smithi019 ceph-mon[31522]: from='osd.3 [v2:172.21.15.19:6810/3571525854,v1:172.21.15.19:6811/3571525854]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]': finished 2024-03-18T15:10:28.074 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:27 smithi019 ceph-mon[31522]: osdmap e24: 4 total, 3 up, 4 in 2024-03-18T15:10:28.074 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:27 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T15:10:28.074 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:27 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T15:10:28.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:27 smithi104 ceph-mon[36070]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-18T15:10:28.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:27 smithi104 ceph-mon[36070]: from='osd.3 [v2:172.21.15.19:6810/3571525854,v1:172.21.15.19:6811/3571525854]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]': finished 2024-03-18T15:10:28.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:27 smithi104 ceph-mon[36070]: osdmap e24: 4 total, 3 up, 4 in 2024-03-18T15:10:28.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:27 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T15:10:28.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:27 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T15:10:28.953 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:28 smithi033 ceph-mon[36648]: purged_snaps scrub starts 2024-03-18T15:10:28.953 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:28 smithi033 ceph-mon[36648]: purged_snaps scrub ok 2024-03-18T15:10:28.953 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:28 smithi033 ceph-mon[36648]: osd.3 [v2:172.21.15.19:6810/3571525854,v1:172.21.15.19:6811/3571525854] boot 2024-03-18T15:10:28.953 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:28 smithi033 ceph-mon[36648]: osdmap e25: 4 total, 4 up, 4 in 2024-03-18T15:10:28.953 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:28 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T15:10:28.953 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:28 smithi033 ceph-mon[36648]: osdmap e26: 4 total, 4 up, 4 in 2024-03-18T15:10:29.040 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:28 smithi019 ceph-mon[31522]: purged_snaps scrub starts 2024-03-18T15:10:29.040 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:28 smithi019 ceph-mon[31522]: purged_snaps scrub ok 2024-03-18T15:10:29.040 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:28 smithi019 ceph-mon[31522]: osd.3 [v2:172.21.15.19:6810/3571525854,v1:172.21.15.19:6811/3571525854] boot 2024-03-18T15:10:29.040 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:28 smithi019 ceph-mon[31522]: osdmap e25: 4 total, 4 up, 4 in 2024-03-18T15:10:29.040 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:28 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T15:10:29.041 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:28 smithi019 ceph-mon[31522]: osdmap e26: 4 total, 4 up, 4 in 2024-03-18T15:10:29.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:28 smithi104 ceph-mon[36070]: purged_snaps scrub starts 2024-03-18T15:10:29.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:28 smithi104 ceph-mon[36070]: purged_snaps scrub ok 2024-03-18T15:10:29.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:28 smithi104 ceph-mon[36070]: osd.3 [v2:172.21.15.19:6810/3571525854,v1:172.21.15.19:6811/3571525854] boot 2024-03-18T15:10:29.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:28 smithi104 ceph-mon[36070]: osdmap e25: 4 total, 4 up, 4 in 2024-03-18T15:10:29.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:28 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T15:10:29.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:28 smithi104 ceph-mon[36070]: osdmap e26: 4 total, 4 up, 4 in 2024-03-18T15:10:29.539 INFO:teuthology.orchestra.run.smithi033.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-18T15:10:29.540 INFO:teuthology.orchestra.run.smithi033.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-18T15:10:29.540 INFO:teuthology.orchestra.run.smithi033.stderr: stderr: 10+0 records in 2024-03-18T15:10:29.540 INFO:teuthology.orchestra.run.smithi033.stderr:10+0 records out 2024-03-18T15:10:29.540 INFO:teuthology.orchestra.run.smithi033.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.011014 s, 952 MB/s 2024-03-18T15:10:29.540 INFO:teuthology.orchestra.run.smithi033.stderr:--> Zapping successful for: 2024-03-18T15:10:30.116 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:29 smithi033 ceph-mon[36648]: pgmap v112: 1 pgs: 1 peering; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-18T15:10:30.116 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:29 smithi033 ceph-mon[36648]: osdmap e27: 4 total, 4 up, 4 in 2024-03-18T15:10:30.117 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi033:vg_nvme/lv_3 2024-03-18T15:10:30.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:29 smithi104 ceph-mon[36070]: pgmap v112: 1 pgs: 1 peering; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-18T15:10:30.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:29 smithi104 ceph-mon[36070]: osdmap e27: 4 total, 4 up, 4 in 2024-03-18T15:10:30.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:29 smithi019 ceph-mon[31522]: pgmap v112: 1 pgs: 1 peering; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-18T15:10:30.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:29 smithi019 ceph-mon[31522]: osdmap e27: 4 total, 4 up, 4 in 2024-03-18T15:10:30.370 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T15:10:32.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:10:32.013+0000 7f0f6ce23700 1 -- 172.21.15.33:0/733141569 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0f681034d0 msgr2=0x7f0f68103930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:10:32.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:10:32.013+0000 7f0f6ce23700 1 --2- 172.21.15.33:0/733141569 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0f681034d0 0x7f0f68103930 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0f5c0099c0 tx=0x7f0f5c009cd0 comp rx=0 tx=0).stop 2024-03-18T15:10:32.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:10:32.013+0000 7f0f6ce23700 1 -- 172.21.15.33:0/733141569 shutdown_connections 2024-03-18T15:10:32.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:10:32.013+0000 7f0f6ce23700 1 --2- 172.21.15.33:0/733141569 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f681022d0 0x7f0f681026f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:10:32.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:10:32.013+0000 7f0f6ce23700 1 --2- 172.21.15.33:0/733141569 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0f68103e70 0x7f0f6810a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:10:32.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:10:32.013+0000 7f0f6ce23700 1 --2- 172.21.15.33:0/733141569 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0f681034d0 0x7f0f68103930 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:10:32.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:10:32.014+0000 7f0f6ce23700 1 -- 172.21.15.33:0/733141569 >> 172.21.15.33:0/733141569 conn(0x7f0f680fd8b0 msgr2=0x7f0f680ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:10:32.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:10:32.014+0000 7f0f6ce23700 1 -- 172.21.15.33:0/733141569 shutdown_connections 2024-03-18T15:10:32.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:10:32.014+0000 7f0f6ce23700 1 -- 172.21.15.33:0/733141569 wait complete. 2024-03-18T15:10:32.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:10:32.014+0000 7f0f6ce23700 1 Processor -- start 2024-03-18T15:10:32.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:10:32.014+0000 7f0f6ce23700 1 -- start start 2024-03-18T15:10:32.016 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:31 smithi033 ceph-mon[36648]: pgmap v115: 1 pgs: 1 peering; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-18T15:10:32.016 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:31 smithi033 ceph-mon[36648]: osdmap e28: 4 total, 4 up, 4 in 2024-03-18T15:10:32.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:10:32.015+0000 7f0f6ce23700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f681022d0 0x7f0f68198640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:10:32.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:10:32.015+0000 7f0f6ce23700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0f681034d0 0x7f0f68198b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:10:32.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:10:32.015+0000 7f0f6ce23700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0f68103e70 0x7f0f6819fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:10:32.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:10:32.015+0000 7f0f6ce23700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f0f6810dfb0 con 0x7f0f681034d0 2024-03-18T15:10:32.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:10:32.015+0000 7f0f6ce23700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f0f6810de30 con 0x7f0f68103e70 2024-03-18T15:10:32.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:10:32.015+0000 7f0f6ce23700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f0f6810e130 con 0x7f0f681022d0 2024-03-18T15:10:32.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:10:32.015+0000 7f0f6659c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f681022d0 0x7f0f68198640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:10:32.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:10:32.015+0000 7f0f66d9d700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0f68103e70 0x7f0f6819fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:10:32.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:10:32.015+0000 7f0f6659c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f681022d0 0x7f0f68198640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.33:40320/0 (socket says 172.21.15.33:40320) 2024-03-18T15:10:32.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:10:32.015+0000 7f0f6659c700 1 -- 172.21.15.33:0/1890521498 learned_addr learned my addr 172.21.15.33:0/1890521498 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-18T15:10:32.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:10:32.015+0000 7f0f66d9d700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0f68103e70 0x7f0f6819fc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:41898/0 (socket says 172.21.15.33:41898) 2024-03-18T15:10:32.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:10:32.015+0000 7f0f65d9b700 1 --2- 172.21.15.33:0/1890521498 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0f681034d0 0x7f0f68198b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:10:32.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:10:32.015+0000 7f0f66d9d700 1 -- 172.21.15.33:0/1890521498 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f681022d0 msgr2=0x7f0f68198640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:10:32.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:10:32.015+0000 7f0f66d9d700 1 --2- 172.21.15.33:0/1890521498 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f681022d0 0x7f0f68198640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:10:32.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:10:32.015+0000 7f0f66d9d700 1 -- 172.21.15.33:0/1890521498 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0f681034d0 msgr2=0x7f0f68198b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:10:32.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:10:32.015+0000 7f0f66d9d700 1 --2- 172.21.15.33:0/1890521498 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0f681034d0 0x7f0f68198b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:10:32.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:10:32.015+0000 7f0f66d9d700 1 -- 172.21.15.33:0/1890521498 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f681a02a0 con 0x7f0f68103e70 2024-03-18T15:10:32.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:10:32.016+0000 7f0f66d9d700 1 --2- 172.21.15.33:0/1890521498 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0f68103e70 0x7f0f6819fc00 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f0f6000be20 tx=0x7f0f60009ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:10:32.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:10:32.016+0000 7f0f577fe700 1 -- 172.21.15.33:0/1890521498 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f60019070 con 0x7f0f68103e70 2024-03-18T15:10:32.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:10:32.016+0000 7f0f6ce23700 1 -- 172.21.15.33:0/1890521498 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f681a0590 con 0x7f0f68103e70 2024-03-18T15:10:32.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:10:32.016+0000 7f0f577fe700 1 -- 172.21.15.33:0/1890521498 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0f6000cb90 con 0x7f0f68103e70 2024-03-18T15:10:32.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:10:32.016+0000 7f0f577fe700 1 -- 172.21.15.33:0/1890521498 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f600125a0 con 0x7f0f68103e70 2024-03-18T15:10:32.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:10:32.016+0000 7f0f6ce23700 1 -- 172.21.15.33:0/1890521498 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f681a0b00 con 0x7f0f68103e70 2024-03-18T15:10:32.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:10:32.017+0000 7f0f6ce23700 1 -- 172.21.15.33:0/1890521498 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f68068f90 con 0x7f0f68103e70 2024-03-18T15:10:32.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:10:32.019+0000 7f0f577fe700 1 -- 172.21.15.33:0/1890521498 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f0f6001c070 con 0x7f0f68103e70 2024-03-18T15:10:32.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:10:32.019+0000 7f0f577fe700 1 --2- 172.21.15.33:0/1890521498 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f0f500a2890 0x7f0f500a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:10:32.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:10:32.020+0000 7f0f577fe700 1 -- 172.21.15.33:0/1890521498 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 3669+0+0 (secure 0 0 0) 0x7f0f600cd320 con 0x7f0f68103e70 2024-03-18T15:10:32.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:10:32.020+0000 7f0f6659c700 1 --2- 172.21.15.33:0/1890521498 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f0f500a2890 0x7f0f500a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:10:32.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:10:32.021+0000 7f0f6659c700 1 --2- 172.21.15.33:0/1890521498 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f0f500a2890 0x7f0f500a4d50 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0f58009aa0 tx=0x7f0f58009240 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:10:32.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:10:32.023+0000 7f0f577fe700 1 -- 172.21.15.33:0/1890521498 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0f6009b540 con 0x7f0f68103e70 2024-03-18T15:10:32.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:31 smithi104 ceph-mon[36070]: pgmap v115: 1 pgs: 1 peering; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-18T15:10:32.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:31 smithi104 ceph-mon[36070]: osdmap e28: 4 total, 4 up, 4 in 2024-03-18T15:10:32.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:31 smithi019 ceph-mon[31522]: pgmap v115: 1 pgs: 1 peering; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-18T15:10:32.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:31 smithi019 ceph-mon[31522]: osdmap e28: 4 total, 4 up, 4 in 2024-03-18T15:10:32.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:10:32.236+0000 7f0f6ce23700 1 -- 172.21.15.33:0/1890521498 --> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f0f68062710 con 0x7f0f500a2890 2024-03-18T15:10:33.135 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:32 smithi033 ceph-mon[36648]: osdmap e29: 4 total, 4 up, 4 in 2024-03-18T15:10:33.135 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:32 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T15:10:33.135 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:32 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T15:10:33.135 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:32 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:10:33.190 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:32 smithi019 ceph-mon[31522]: osdmap e29: 4 total, 4 up, 4 in 2024-03-18T15:10:33.190 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:32 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T15:10:33.190 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:32 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T15:10:33.190 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:32 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:10:33.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:32 smithi104 ceph-mon[36070]: osdmap e29: 4 total, 4 up, 4 in 2024-03-18T15:10:33.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:32 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T15:10:33.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:32 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T15:10:33.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:32 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:10:34.079 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:33 smithi033 ceph-mon[36648]: pgmap v118: 1 pgs: 1 peering; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-18T15:10:34.079 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:33 smithi033 ceph-mon[36648]: from='client.24215 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:10:34.079 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:33 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:10:34.079 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:33 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:10:34.079 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:33 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:10:34.079 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:33 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:10:34.079 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:33 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:10:34.079 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:33 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:10:34.079 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:33 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:10:34.080 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:33 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:10:34.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:33 smithi019 ceph-mon[31522]: pgmap v118: 1 pgs: 1 peering; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-18T15:10:34.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:33 smithi019 ceph-mon[31522]: from='client.24215 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:10:34.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:33 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:10:34.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:33 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:10:34.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:33 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:10:34.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:33 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:10:34.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:33 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:10:34.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:33 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:10:34.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:33 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:10:34.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:33 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:10:34.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:33 smithi104 ceph-mon[36070]: pgmap v118: 1 pgs: 1 peering; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-18T15:10:34.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:33 smithi104 ceph-mon[36070]: from='client.24215 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:10:34.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:33 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:10:34.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:33 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:10:34.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:33 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:10:34.446 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:33 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:10:34.446 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:33 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:10:34.446 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:33 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:10:34.446 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:33 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:10:34.446 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:33 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:10:35.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:34 smithi033 ceph-mon[36648]: Detected new or changed devices on smithi019 2024-03-18T15:10:35.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:34 smithi033 ceph-mon[36648]: Adjusting osd_memory_target on smithi019 to 8547M 2024-03-18T15:10:35.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:34 smithi104 ceph-mon[36070]: Detected new or changed devices on smithi019 2024-03-18T15:10:35.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:34 smithi104 ceph-mon[36070]: Adjusting osd_memory_target on smithi019 to 8547M 2024-03-18T15:10:35.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:34 smithi019 ceph-mon[31522]: Detected new or changed devices on smithi019 2024-03-18T15:10:35.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:34 smithi019 ceph-mon[31522]: Adjusting osd_memory_target on smithi019 to 8547M 2024-03-18T15:10:36.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:35 smithi033 ceph-mon[36648]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 109 KiB/s, 0 objects/s recovering 2024-03-18T15:10:36.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:35 smithi033 ceph-mon[36648]: from='client.? 172.21.15.33:0/4162130462' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "696eb719-0b3e-43ae-b5f1-ccd73a3b2843"}]: dispatch 2024-03-18T15:10:36.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:35 smithi033 ceph-mon[36648]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "696eb719-0b3e-43ae-b5f1-ccd73a3b2843"}]: dispatch 2024-03-18T15:10:36.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:35 smithi033 ceph-mon[36648]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "696eb719-0b3e-43ae-b5f1-ccd73a3b2843"}]': finished 2024-03-18T15:10:36.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:35 smithi033 ceph-mon[36648]: osdmap e30: 5 total, 4 up, 5 in 2024-03-18T15:10:36.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:35 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T15:10:36.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:35 smithi033 ceph-mon[36648]: from='client.? 172.21.15.33:0/2332373544' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T15:10:36.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:35 smithi104 ceph-mon[36070]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 109 KiB/s, 0 objects/s recovering 2024-03-18T15:10:36.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:35 smithi104 ceph-mon[36070]: from='client.? 172.21.15.33:0/4162130462' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "696eb719-0b3e-43ae-b5f1-ccd73a3b2843"}]: dispatch 2024-03-18T15:10:36.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:35 smithi104 ceph-mon[36070]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "696eb719-0b3e-43ae-b5f1-ccd73a3b2843"}]: dispatch 2024-03-18T15:10:36.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:35 smithi104 ceph-mon[36070]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "696eb719-0b3e-43ae-b5f1-ccd73a3b2843"}]': finished 2024-03-18T15:10:36.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:35 smithi104 ceph-mon[36070]: osdmap e30: 5 total, 4 up, 5 in 2024-03-18T15:10:36.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:35 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T15:10:36.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:35 smithi104 ceph-mon[36070]: from='client.? 172.21.15.33:0/2332373544' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T15:10:36.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:35 smithi019 ceph-mon[31522]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 109 KiB/s, 0 objects/s recovering 2024-03-18T15:10:36.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:35 smithi019 ceph-mon[31522]: from='client.? 172.21.15.33:0/4162130462' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "696eb719-0b3e-43ae-b5f1-ccd73a3b2843"}]: dispatch 2024-03-18T15:10:36.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:35 smithi019 ceph-mon[31522]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "696eb719-0b3e-43ae-b5f1-ccd73a3b2843"}]: dispatch 2024-03-18T15:10:36.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:35 smithi019 ceph-mon[31522]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "696eb719-0b3e-43ae-b5f1-ccd73a3b2843"}]': finished 2024-03-18T15:10:36.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:35 smithi019 ceph-mon[31522]: osdmap e30: 5 total, 4 up, 5 in 2024-03-18T15:10:36.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:35 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T15:10:36.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:35 smithi019 ceph-mon[31522]: from='client.? 172.21.15.33:0/2332373544' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T15:10:38.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:37 smithi033 ceph-mon[36648]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-18T15:10:38.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:37 smithi104 ceph-mon[36070]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-18T15:10:38.456 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:37 smithi019 ceph-mon[31522]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-18T15:10:40.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:39 smithi033 ceph-mon[36648]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-03-18T15:10:40.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:39 smithi104 ceph-mon[36070]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-03-18T15:10:40.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:39 smithi019 ceph-mon[31522]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-03-18T15:10:42.178 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:41 smithi033 ceph-mon[36648]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-03-18T15:10:42.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:41 smithi104 ceph-mon[36070]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-03-18T15:10:42.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:41 smithi019 ceph-mon[31522]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-03-18T15:10:43.255 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:42 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-18T15:10:43.255 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:42 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:10:43.367 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:42 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-18T15:10:43.367 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:42 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:10:43.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:42 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-18T15:10:43.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:42 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:10:43.998 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:43 smithi019 ceph-mon[31522]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-18T15:10:43.998 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:43 smithi019 ceph-mon[31522]: Deploying daemon osd.4 on smithi033 2024-03-18T15:10:44.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:43 smithi033 ceph-mon[36648]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-18T15:10:44.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:43 smithi033 ceph-mon[36648]: Deploying daemon osd.4 on smithi033 2024-03-18T15:10:44.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:43 smithi104 ceph-mon[36070]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-18T15:10:44.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:43 smithi104 ceph-mon[36070]: Deploying daemon osd.4 on smithi033 2024-03-18T15:10:46.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:46 smithi033 ceph-mon[36648]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-18T15:10:46.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:46 smithi104 ceph-mon[36070]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-18T15:10:46.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:46 smithi019 ceph-mon[31522]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-18T15:10:48.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:48 smithi033 ceph-mon[36648]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-18T15:10:48.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:48 smithi104 ceph-mon[36070]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-18T15:10:48.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:48 smithi019 ceph-mon[31522]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-18T15:10:49.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:49 smithi033 ceph-mon[36648]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-18T15:10:49.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:49 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:10:49.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:49 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:10:49.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:49 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:10:49.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:49 smithi104 ceph-mon[36070]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-18T15:10:49.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:49 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:10:49.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:49 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:10:49.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:49 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:10:49.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:49 smithi019 ceph-mon[31522]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-18T15:10:49.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:49 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:10:49.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:49 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:10:49.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:49 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:10:51.556 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:51 smithi033 ceph-mon[36648]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-18T15:10:51.617 INFO:teuthology.orchestra.run.smithi033.stdout:Created osd(s) 4 on host 'smithi033' 2024-03-18T15:10:51.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:10:51.616+0000 7f0f577fe700 1 -- 172.21.15.33:0/1890521498 <== mgr.14152 v2:172.21.15.19:6800/2017714236 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f0f68062710 con 0x7f0f500a2890 2024-03-18T15:10:51.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:10:51.618+0000 7f0f6ce23700 1 -- 172.21.15.33:0/1890521498 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f0f500a2890 msgr2=0x7f0f500a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:10:51.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:10:51.618+0000 7f0f6ce23700 1 --2- 172.21.15.33:0/1890521498 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f0f500a2890 0x7f0f500a4d50 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0f58009aa0 tx=0x7f0f58009240 comp rx=0 tx=0).stop 2024-03-18T15:10:51.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:10:51.618+0000 7f0f6ce23700 1 -- 172.21.15.33:0/1890521498 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0f68103e70 msgr2=0x7f0f6819fc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:10:51.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:10:51.618+0000 7f0f6ce23700 1 --2- 172.21.15.33:0/1890521498 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0f68103e70 0x7f0f6819fc00 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f0f6000be20 tx=0x7f0f60009ed0 comp rx=0 tx=0).stop 2024-03-18T15:10:51.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:10:51.618+0000 7f0f6ce23700 1 -- 172.21.15.33:0/1890521498 shutdown_connections 2024-03-18T15:10:51.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:10:51.618+0000 7f0f6ce23700 1 --2- 172.21.15.33:0/1890521498 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f0f500a2890 0x7f0f500a4d50 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:10:51.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:10:51.618+0000 7f0f6ce23700 1 --2- 172.21.15.33:0/1890521498 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f681022d0 0x7f0f68198640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:10:51.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:10:51.618+0000 7f0f6ce23700 1 --2- 172.21.15.33:0/1890521498 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0f68103e70 0x7f0f6819fc00 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:10:51.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:10:51.618+0000 7f0f6ce23700 1 --2- 172.21.15.33:0/1890521498 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0f681034d0 0x7f0f68198b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:10:51.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:10:51.618+0000 7f0f6ce23700 1 -- 172.21.15.33:0/1890521498 >> 172.21.15.33:0/1890521498 conn(0x7f0f680fd8b0 msgr2=0x7f0f681068a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:10:51.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:10:51.618+0000 7f0f6ce23700 1 -- 172.21.15.33:0/1890521498 shutdown_connections 2024-03-18T15:10:51.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:10:51.619+0000 7f0f6ce23700 1 -- 172.21.15.33:0/1890521498 wait complete. 2024-03-18T15:10:51.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:51 smithi104 ceph-mon[36070]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-18T15:10:51.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:51 smithi019 ceph-mon[31522]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-18T15:10:52.271 DEBUG:teuthology.orchestra.run.smithi033:osd.4> sudo journalctl -f -n 0 -u ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@osd.4.service 2024-03-18T15:10:52.274 INFO:tasks.cephadm:Deploying osd.5 on smithi104 with /dev/vg_nvme/lv_3... 2024-03-18T15:10:52.274 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-18T15:10:52.518 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T15:10:52.837 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:52 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:10:52.838 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:52 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:10:52.838 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:52 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:10:52.838 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:52 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:10:52.838 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:52 smithi033 ceph-mon[36648]: from='osd.4 [v2:172.21.15.33:6808/624901749,v1:172.21.15.33:6809/624901749]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-18T15:10:52.838 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:52 smithi033 ceph-mon[36648]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-18T15:10:52.928 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:52 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:10:52.928 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:52 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:10:52.928 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:52 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:10:52.928 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:52 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:10:52.928 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:52 smithi104 ceph-mon[36070]: from='osd.4 [v2:172.21.15.33:6808/624901749,v1:172.21.15.33:6809/624901749]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-18T15:10:52.928 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:52 smithi104 ceph-mon[36070]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-18T15:10:52.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:52 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:10:52.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:52 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:10:52.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:52 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:10:52.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:52 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:10:52.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:52 smithi019 ceph-mon[31522]: from='osd.4 [v2:172.21.15.33:6808/624901749,v1:172.21.15.33:6809/624901749]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-18T15:10:52.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:52 smithi019 ceph-mon[31522]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-18T15:10:54.001 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:53 smithi104 ceph-mon[36070]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-18T15:10:54.002 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:53 smithi104 ceph-mon[36070]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-18T15:10:54.002 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:53 smithi104 ceph-mon[36070]: from='osd.4 [v2:172.21.15.33:6808/624901749,v1:172.21.15.33:6809/624901749]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-03-18T15:10:54.002 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:53 smithi104 ceph-mon[36070]: osdmap e31: 5 total, 4 up, 5 in 2024-03-18T15:10:54.002 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:53 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T15:10:54.002 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:53 smithi104 ceph-mon[36070]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-03-18T15:10:54.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:53 smithi033 ceph-mon[36648]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-18T15:10:54.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:53 smithi033 ceph-mon[36648]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-18T15:10:54.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:53 smithi033 ceph-mon[36648]: from='osd.4 [v2:172.21.15.33:6808/624901749,v1:172.21.15.33:6809/624901749]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-03-18T15:10:54.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:53 smithi033 ceph-mon[36648]: osdmap e31: 5 total, 4 up, 5 in 2024-03-18T15:10:54.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:53 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T15:10:54.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:53 smithi033 ceph-mon[36648]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-03-18T15:10:54.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:53 smithi019 ceph-mon[31522]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-18T15:10:54.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:53 smithi019 ceph-mon[31522]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-18T15:10:54.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:53 smithi019 ceph-mon[31522]: from='osd.4 [v2:172.21.15.33:6808/624901749,v1:172.21.15.33:6809/624901749]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-03-18T15:10:54.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:53 smithi019 ceph-mon[31522]: osdmap e31: 5 total, 4 up, 5 in 2024-03-18T15:10:54.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:53 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T15:10:54.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:53 smithi019 ceph-mon[31522]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-03-18T15:10:54.549 INFO:teuthology.orchestra.run.smithi104.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-18T15:10:54.549 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-18T15:10:54.549 INFO:teuthology.orchestra.run.smithi104.stderr: stderr: 10+0 records in 2024-03-18T15:10:54.549 INFO:teuthology.orchestra.run.smithi104.stderr:10+0 records out 2024-03-18T15:10:54.549 INFO:teuthology.orchestra.run.smithi104.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0116576 s, 899 MB/s 2024-03-18T15:10:54.549 INFO:teuthology.orchestra.run.smithi104.stderr:--> Zapping successful for: 2024-03-18T15:10:54.676 INFO:journalctl@ceph.osd.4.smithi033.stdout:Mar 18 15:10:54 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-4[47810]: 2024-03-18T15:10:54.372+0000 7fe6d632c700 -1 osd.4 0 waiting for initial osdmap 2024-03-18T15:10:54.676 INFO:journalctl@ceph.osd.4.smithi033.stdout:Mar 18 15:10:54 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-4[47810]: 2024-03-18T15:10:54.378+0000 7fe6ce117700 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T15:10:55.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:54 smithi033 ceph-mon[36648]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]': finished 2024-03-18T15:10:55.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:54 smithi033 ceph-mon[36648]: osdmap e32: 5 total, 4 up, 5 in 2024-03-18T15:10:55.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:54 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T15:10:55.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:54 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T15:10:55.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:54 smithi033 ceph-mon[36648]: from='osd.4 ' entity='osd.4' 2024-03-18T15:10:55.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:54 smithi104 ceph-mon[36070]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]': finished 2024-03-18T15:10:55.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:54 smithi104 ceph-mon[36070]: osdmap e32: 5 total, 4 up, 5 in 2024-03-18T15:10:55.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:54 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T15:10:55.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:54 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T15:10:55.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:54 smithi104 ceph-mon[36070]: from='osd.4 ' entity='osd.4' 2024-03-18T15:10:55.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:54 smithi019 ceph-mon[31522]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]': finished 2024-03-18T15:10:55.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:54 smithi019 ceph-mon[31522]: osdmap e32: 5 total, 4 up, 5 in 2024-03-18T15:10:55.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:54 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T15:10:55.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:54 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T15:10:55.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:54 smithi019 ceph-mon[31522]: from='osd.4 ' entity='osd.4' 2024-03-18T15:10:55.240 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi104:vg_nvme/lv_3 2024-03-18T15:10:55.485 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T15:10:56.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:55 smithi104 ceph-mon[36070]: purged_snaps scrub starts 2024-03-18T15:10:56.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:55 smithi104 ceph-mon[36070]: purged_snaps scrub ok 2024-03-18T15:10:56.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:55 smithi104 ceph-mon[36070]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-18T15:10:56.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:55 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T15:10:56.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:55 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:10:56.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:55 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:10:56.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:55 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:10:56.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:55 smithi104 ceph-mon[36070]: osd.4 [v2:172.21.15.33:6808/624901749,v1:172.21.15.33:6809/624901749] boot 2024-03-18T15:10:56.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:55 smithi104 ceph-mon[36070]: osdmap e33: 5 total, 5 up, 5 in 2024-03-18T15:10:56.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:55 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:10:56.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:55 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T15:10:56.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:55 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:10:56.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:55 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:10:56.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:55 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:10:56.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:55 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:10:56.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:55 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:10:56.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:55 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:10:56.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:55 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:10:56.196 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:55 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:10:56.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:55 smithi019 ceph-mon[31522]: purged_snaps scrub starts 2024-03-18T15:10:56.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:55 smithi019 ceph-mon[31522]: purged_snaps scrub ok 2024-03-18T15:10:56.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:55 smithi019 ceph-mon[31522]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-18T15:10:56.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:55 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T15:10:56.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:55 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:10:56.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:55 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:10:56.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:55 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:10:56.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:55 smithi019 ceph-mon[31522]: osd.4 [v2:172.21.15.33:6808/624901749,v1:172.21.15.33:6809/624901749] boot 2024-03-18T15:10:56.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:55 smithi019 ceph-mon[31522]: osdmap e33: 5 total, 5 up, 5 in 2024-03-18T15:10:56.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:55 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:10:56.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:55 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T15:10:56.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:55 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:10:56.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:55 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:10:56.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:55 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:10:56.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:55 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:10:56.217 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:55 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:10:56.217 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:55 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:10:56.217 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:55 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:10:56.217 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:55 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:10:56.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:55 smithi033 ceph-mon[36648]: purged_snaps scrub starts 2024-03-18T15:10:56.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:55 smithi033 ceph-mon[36648]: purged_snaps scrub ok 2024-03-18T15:10:56.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:55 smithi033 ceph-mon[36648]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-18T15:10:56.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:55 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T15:10:56.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:55 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:10:56.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:55 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:10:56.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:55 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:10:56.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:55 smithi033 ceph-mon[36648]: osd.4 [v2:172.21.15.33:6808/624901749,v1:172.21.15.33:6809/624901749] boot 2024-03-18T15:10:56.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:55 smithi033 ceph-mon[36648]: osdmap e33: 5 total, 5 up, 5 in 2024-03-18T15:10:56.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:55 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:10:56.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:55 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T15:10:56.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:55 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:10:56.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:55 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:10:56.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:55 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:10:56.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:55 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:10:56.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:55 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:10:56.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:55 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:10:56.258 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:55 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:10:56.258 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:55 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:10:57.117 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:10:57.116+0000 7fd75729c700 1 -- 172.21.15.104:0/121505762 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd750102400 msgr2=0x7fd750102860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:10:57.117 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:10:57.116+0000 7fd75729c700 1 --2- 172.21.15.104:0/121505762 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd750102400 0x7fd750102860 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fd740009990 tx=0x7fd740009ca0 comp rx=0 tx=0).stop 2024-03-18T15:10:57.117 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:10:57.116+0000 7fd75729c700 1 -- 172.21.15.104:0/121505762 shutdown_connections 2024-03-18T15:10:57.117 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:10:57.116+0000 7fd75729c700 1 --2- 172.21.15.104:0/121505762 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd750102da0 0x7fd750109650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:10:57.117 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:10:57.116+0000 7fd75729c700 1 --2- 172.21.15.104:0/121505762 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd750101200 0x7fd750101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:10:57.117 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:10:57.116+0000 7fd75729c700 1 --2- 172.21.15.104:0/121505762 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd750102400 0x7fd750102860 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:10:57.117 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:10:57.116+0000 7fd75729c700 1 -- 172.21.15.104:0/121505762 >> 172.21.15.104:0/121505762 conn(0x7fd7500fc7a0 msgr2=0x7fd7500febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:10:57.118 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:10:57.116+0000 7fd75729c700 1 -- 172.21.15.104:0/121505762 shutdown_connections 2024-03-18T15:10:57.118 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:10:57.116+0000 7fd75729c700 1 -- 172.21.15.104:0/121505762 wait complete. 2024-03-18T15:10:57.119 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:10:57.117+0000 7fd75729c700 1 Processor -- start 2024-03-18T15:10:57.120 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:10:57.117+0000 7fd75729c700 1 -- start start 2024-03-18T15:10:57.120 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:10:57.117+0000 7fd75729c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd750101200 0x7fd75006b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:10:57.120 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:10:57.117+0000 7fd75729c700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd750102400 0x7fd75019d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:10:57.120 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:10:57.117+0000 7fd75729c700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd750102da0 0x7fd75019f930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:10:57.120 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:10:57.117+0000 7fd75729c700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fd75010ce60 con 0x7fd750102400 2024-03-18T15:10:57.120 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:10:57.117+0000 7fd75729c700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd75010cce0 con 0x7fd750102da0 2024-03-18T15:10:57.120 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:10:57.117+0000 7fd75729c700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fd75010cfe0 con 0x7fd750101200 2024-03-18T15:10:57.120 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:10:57.117+0000 7fd755038700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd750101200 0x7fd75006b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:10:57.120 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:10:57.117+0000 7fd755038700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd750101200 0x7fd75006b4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:60240/0 (socket says 172.21.15.104:60240) 2024-03-18T15:10:57.120 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:10:57.117+0000 7fd755038700 1 -- 172.21.15.104:0/1383226705 learned_addr learned my addr 172.21.15.104:0/1383226705 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-18T15:10:57.120 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:10:57.117+0000 7fd755839700 1 --2- 172.21.15.104:0/1383226705 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd750102da0 0x7fd75019f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:10:57.120 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:10:57.117+0000 7fd755038700 1 -- 172.21.15.104:0/1383226705 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd750102da0 msgr2=0x7fd75019f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:10:57.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:10:57.117+0000 7fd755038700 1 --2- 172.21.15.104:0/1383226705 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd750102da0 0x7fd75019f930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:10:57.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:10:57.117+0000 7fd755038700 1 -- 172.21.15.104:0/1383226705 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd750102400 msgr2=0x7fd75019d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:10:57.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:10:57.117+0000 7fd755038700 1 --2- 172.21.15.104:0/1383226705 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd750102400 0x7fd75019d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:10:57.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:10:57.117+0000 7fd755038700 1 -- 172.21.15.104:0/1383226705 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7501a0200 con 0x7fd750101200 2024-03-18T15:10:57.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:10:57.117+0000 7fd755839700 1 --2- 172.21.15.104:0/1383226705 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd750102da0 0x7fd75019f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T15:10:57.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:10:57.117+0000 7fd755038700 1 --2- 172.21.15.104:0/1383226705 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd750101200 0x7fd75006b4d0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fd74c00b500 tx=0x7fd74c00b8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:10:57.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:10:57.118+0000 7fd7467fc700 1 -- 172.21.15.104:0/1383226705 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd74c015020 con 0x7fd750101200 2024-03-18T15:10:57.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:10:57.118+0000 7fd75729c700 1 -- 172.21.15.104:0/1383226705 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7501a04f0 con 0x7fd750101200 2024-03-18T15:10:57.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:10:57.118+0000 7fd7467fc700 1 -- 172.21.15.104:0/1383226705 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd74c011d10 con 0x7fd750101200 2024-03-18T15:10:57.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:10:57.118+0000 7fd7467fc700 1 -- 172.21.15.104:0/1383226705 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd74c00f2e0 con 0x7fd750101200 2024-03-18T15:10:57.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:10:57.118+0000 7fd75729c700 1 -- 172.21.15.104:0/1383226705 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7501a0b00 con 0x7fd750101200 2024-03-18T15:10:57.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:10:57.119+0000 7fd75729c700 1 -- 172.21.15.104:0/1383226705 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd750068f90 con 0x7fd750101200 2024-03-18T15:10:57.122 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:10:57.120+0000 7fd7467fc700 1 -- 172.21.15.104:0/1383226705 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fd74c018020 con 0x7fd750101200 2024-03-18T15:10:57.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:10:57.120+0000 7fd7467fc700 1 --2- 172.21.15.104:0/1383226705 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fd73c0a2890 0x7fd73c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:10:57.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:10:57.121+0000 7fd7467fc700 1 -- 172.21.15.104:0/1383226705 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 4101+0+0 (secure 0 0 0) 0x7fd74c0cd200 con 0x7fd750101200 2024-03-18T15:10:57.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:10:57.121+0000 7fd754837700 1 --2- 172.21.15.104:0/1383226705 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fd73c0a2890 0x7fd73c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:10:57.123 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:10:57.122+0000 7fd754837700 1 --2- 172.21.15.104:0/1383226705 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fd73c0a2890 0x7fd73c0a4d50 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fd7400039c0 tx=0x7fd740005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:10:57.126 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:10:57.125+0000 7fd7467fc700 1 -- 172.21.15.104:0/1383226705 <== mon.1 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd74c01f020 con 0x7fd750101200 2024-03-18T15:10:57.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:56 smithi104 ceph-mon[36070]: Detected new or changed devices on smithi033 2024-03-18T15:10:57.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:56 smithi104 ceph-mon[36070]: Adjusting osd_memory_target on smithi033 to 8542M 2024-03-18T15:10:57.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:56 smithi019 ceph-mon[31522]: Detected new or changed devices on smithi033 2024-03-18T15:10:57.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:56 smithi019 ceph-mon[31522]: Adjusting osd_memory_target on smithi033 to 8542M 2024-03-18T15:10:57.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:56 smithi033 ceph-mon[36648]: Detected new or changed devices on smithi033 2024-03-18T15:10:57.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:56 smithi033 ceph-mon[36648]: Adjusting osd_memory_target on smithi033 to 8542M 2024-03-18T15:10:57.337 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:10:57.336+0000 7fd75729c700 1 -- 172.21.15.104:0/1383226705 --> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fd750062710 con 0x7fd73c0a2890 2024-03-18T15:10:58.193 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:57 smithi104 ceph-mon[36070]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 141 MiB used, 447 GiB / 447 GiB avail 2024-03-18T15:10:58.193 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:57 smithi104 ceph-mon[36070]: osdmap e34: 5 total, 5 up, 5 in 2024-03-18T15:10:58.193 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:57 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T15:10:58.193 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:57 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T15:10:58.193 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:57 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:10:58.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:57 smithi019 ceph-mon[31522]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 141 MiB used, 447 GiB / 447 GiB avail 2024-03-18T15:10:58.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:57 smithi019 ceph-mon[31522]: osdmap e34: 5 total, 5 up, 5 in 2024-03-18T15:10:58.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:57 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T15:10:58.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:57 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T15:10:58.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:57 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:10:58.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:57 smithi033 ceph-mon[36648]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 141 MiB used, 447 GiB / 447 GiB avail 2024-03-18T15:10:58.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:57 smithi033 ceph-mon[36648]: osdmap e34: 5 total, 5 up, 5 in 2024-03-18T15:10:58.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:57 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T15:10:58.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:57 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T15:10:58.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:57 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:10:59.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:58 smithi104 ceph-mon[36070]: from='client.24241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:10:59.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:58 smithi019 ceph-mon[31522]: from='client.24241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:10:59.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:58 smithi033 ceph-mon[36648]: from='client.24241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:11:00.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:10:59 smithi104 ceph-mon[36070]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-18T15:11:00.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:10:59 smithi019 ceph-mon[31522]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-18T15:11:00.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:10:59 smithi033 ceph-mon[36648]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-18T15:11:01.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:00 smithi104 ceph-mon[36070]: from='client.? 172.21.15.104:0/409916890' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "21789779-8bb9-42c7-b9ee-437985f11e3d"}]: dispatch 2024-03-18T15:11:01.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:00 smithi104 ceph-mon[36070]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "21789779-8bb9-42c7-b9ee-437985f11e3d"}]: dispatch 2024-03-18T15:11:01.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:00 smithi104 ceph-mon[36070]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "21789779-8bb9-42c7-b9ee-437985f11e3d"}]': finished 2024-03-18T15:11:01.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:00 smithi104 ceph-mon[36070]: osdmap e35: 6 total, 5 up, 6 in 2024-03-18T15:11:01.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:00 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T15:11:01.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:00 smithi104 ceph-mon[36070]: from='client.? 172.21.15.104:0/3523770109' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T15:11:01.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:00 smithi019 ceph-mon[31522]: from='client.? 172.21.15.104:0/409916890' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "21789779-8bb9-42c7-b9ee-437985f11e3d"}]: dispatch 2024-03-18T15:11:01.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:00 smithi019 ceph-mon[31522]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "21789779-8bb9-42c7-b9ee-437985f11e3d"}]: dispatch 2024-03-18T15:11:01.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:00 smithi019 ceph-mon[31522]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "21789779-8bb9-42c7-b9ee-437985f11e3d"}]': finished 2024-03-18T15:11:01.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:00 smithi019 ceph-mon[31522]: osdmap e35: 6 total, 5 up, 6 in 2024-03-18T15:11:01.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:00 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T15:11:01.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:00 smithi019 ceph-mon[31522]: from='client.? 172.21.15.104:0/3523770109' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T15:11:01.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:00 smithi033 ceph-mon[36648]: from='client.? 172.21.15.104:0/409916890' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "21789779-8bb9-42c7-b9ee-437985f11e3d"}]: dispatch 2024-03-18T15:11:01.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:00 smithi033 ceph-mon[36648]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "21789779-8bb9-42c7-b9ee-437985f11e3d"}]: dispatch 2024-03-18T15:11:01.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:00 smithi033 ceph-mon[36648]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "21789779-8bb9-42c7-b9ee-437985f11e3d"}]': finished 2024-03-18T15:11:01.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:00 smithi033 ceph-mon[36648]: osdmap e35: 6 total, 5 up, 6 in 2024-03-18T15:11:01.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:00 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T15:11:01.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:00 smithi033 ceph-mon[36648]: from='client.? 172.21.15.104:0/3523770109' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T15:11:02.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:01 smithi104 ceph-mon[36070]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-18T15:11:02.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:01 smithi019 ceph-mon[31522]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-18T15:11:02.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:01 smithi033 ceph-mon[36648]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-18T15:11:04.115 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:03 smithi104 ceph-mon[36070]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-18T15:11:04.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:03 smithi019 ceph-mon[31522]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-18T15:11:04.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:03 smithi033 ceph-mon[36648]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-18T15:11:06.212 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:05 smithi104 ceph-mon[36070]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-18T15:11:06.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:05 smithi019 ceph-mon[31522]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-18T15:11:06.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:05 smithi033 ceph-mon[36648]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-18T15:11:08.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:07 smithi019 ceph-mon[31522]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-18T15:11:08.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:07 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-18T15:11:08.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:07 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:11:08.230 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:07 smithi104 ceph-mon[36070]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-18T15:11:08.230 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:07 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-18T15:11:08.230 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:07 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:11:08.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:07 smithi033 ceph-mon[36648]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-18T15:11:08.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:07 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-18T15:11:08.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:07 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:11:09.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:08 smithi019 ceph-mon[31522]: Deploying daemon osd.5 on smithi104 2024-03-18T15:11:09.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:08 smithi033 ceph-mon[36648]: Deploying daemon osd.5 on smithi104 2024-03-18T15:11:09.258 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:08 smithi104 ceph-mon[36070]: Deploying daemon osd.5 on smithi104 2024-03-18T15:11:10.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:09 smithi019 ceph-mon[31522]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-18T15:11:10.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:09 smithi033 ceph-mon[36648]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-18T15:11:10.339 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:09 smithi104 ceph-mon[36070]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-18T15:11:12.054 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:11 smithi104 ceph-mon[36070]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-18T15:11:12.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:11 smithi033 ceph-mon[36648]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-18T15:11:12.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:11 smithi019 ceph-mon[31522]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-18T15:11:14.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:13 smithi104 ceph-mon[36070]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-18T15:11:14.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:13 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:11:14.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:13 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:11:14.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:13 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:11:14.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:13 smithi019 ceph-mon[31522]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-18T15:11:14.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:13 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:11:14.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:13 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:11:14.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:13 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:11:14.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:13 smithi033 ceph-mon[36648]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-18T15:11:14.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:13 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:11:14.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:13 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:11:14.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:13 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:11:16.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:15 smithi033 ceph-mon[36648]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-18T15:11:16.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:15 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:11:16.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:15 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:11:16.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:15 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:11:16.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:15 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:11:16.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:15 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:11:16.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:15 smithi104 ceph-mon[36070]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-18T15:11:16.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:15 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:11:16.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:15 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:11:16.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:15 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:11:16.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:15 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:11:16.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:15 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:11:16.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:15 smithi019 ceph-mon[31522]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-18T15:11:16.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:15 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:11:16.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:15 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:11:16.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:15 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:11:16.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:15 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:11:16.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:15 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:11:16.924 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:11:16.923+0000 7fd7467fc700 1 -- 172.21.15.104:0/1383226705 <== mgr.14152 v2:172.21.15.19:6800/2017714236 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fd750062710 con 0x7fd73c0a2890 2024-03-18T15:11:16.925 INFO:teuthology.orchestra.run.smithi104.stdout:Created osd(s) 5 on host 'smithi104' 2024-03-18T15:11:16.926 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:11:16.925+0000 7fd75729c700 1 -- 172.21.15.104:0/1383226705 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fd73c0a2890 msgr2=0x7fd73c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:11:16.926 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:11:16.925+0000 7fd75729c700 1 --2- 172.21.15.104:0/1383226705 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fd73c0a2890 0x7fd73c0a4d50 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fd7400039c0 tx=0x7fd740005fb0 comp rx=0 tx=0).stop 2024-03-18T15:11:16.926 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:11:16.925+0000 7fd75729c700 1 -- 172.21.15.104:0/1383226705 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd750101200 msgr2=0x7fd75006b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:11:16.926 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:11:16.925+0000 7fd75729c700 1 --2- 172.21.15.104:0/1383226705 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd750101200 0x7fd75006b4d0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fd74c00b500 tx=0x7fd74c00b8c0 comp rx=0 tx=0).stop 2024-03-18T15:11:16.926 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:11:16.925+0000 7fd75729c700 1 -- 172.21.15.104:0/1383226705 shutdown_connections 2024-03-18T15:11:16.927 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:11:16.925+0000 7fd75729c700 1 --2- 172.21.15.104:0/1383226705 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fd73c0a2890 0x7fd73c0a4d50 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:11:16.927 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:11:16.925+0000 7fd75729c700 1 --2- 172.21.15.104:0/1383226705 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd750101200 0x7fd75006b4d0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:11:16.927 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:11:16.925+0000 7fd75729c700 1 --2- 172.21.15.104:0/1383226705 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd750102da0 0x7fd75019f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:11:16.927 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:11:16.925+0000 7fd75729c700 1 --2- 172.21.15.104:0/1383226705 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd750102400 0x7fd75019d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:11:16.927 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:11:16.925+0000 7fd75729c700 1 -- 172.21.15.104:0/1383226705 >> 172.21.15.104:0/1383226705 conn(0x7fd7500fc7a0 msgr2=0x7fd7501056a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:11:16.927 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:11:16.925+0000 7fd75729c700 1 -- 172.21.15.104:0/1383226705 shutdown_connections 2024-03-18T15:11:16.927 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:11:16.925+0000 7fd75729c700 1 -- 172.21.15.104:0/1383226705 wait complete. 2024-03-18T15:11:17.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:16 smithi104 ceph-mon[36070]: from='osd.5 [v2:172.21.15.104:6808/1258208297,v1:172.21.15.104:6809/1258208297]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-18T15:11:17.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:16 smithi104 ceph-mon[36070]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-18T15:11:17.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:16 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:11:17.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:16 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:11:17.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:16 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:11:17.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:16 smithi033 ceph-mon[36648]: from='osd.5 [v2:172.21.15.104:6808/1258208297,v1:172.21.15.104:6809/1258208297]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-18T15:11:17.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:16 smithi033 ceph-mon[36648]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-18T15:11:17.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:16 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:11:17.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:16 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:11:17.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:16 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:11:17.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:16 smithi019 ceph-mon[31522]: from='osd.5 [v2:172.21.15.104:6808/1258208297,v1:172.21.15.104:6809/1258208297]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-18T15:11:17.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:16 smithi019 ceph-mon[31522]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-18T15:11:17.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:16 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:11:17.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:16 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:11:17.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:16 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:11:17.621 DEBUG:teuthology.orchestra.run.smithi104:osd.5> sudo journalctl -f -n 0 -u ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@osd.5.service 2024-03-18T15:11:17.623 INFO:tasks.cephadm:Deploying osd.6 on smithi019 with /dev/vg_nvme/lv_2... 2024-03-18T15:11:17.623 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-18T15:11:18.093 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:11:18.181 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:17 smithi104 ceph-mon[36070]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-18T15:11:18.181 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:17 smithi104 ceph-mon[36070]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-18T15:11:18.181 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:17 smithi104 ceph-mon[36070]: from='osd.5 [v2:172.21.15.104:6808/1258208297,v1:172.21.15.104:6809/1258208297]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-03-18T15:11:18.181 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:17 smithi104 ceph-mon[36070]: osdmap e36: 6 total, 5 up, 6 in 2024-03-18T15:11:18.181 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:17 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T15:11:18.181 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:17 smithi104 ceph-mon[36070]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-03-18T15:11:18.182 INFO:journalctl@ceph.osd.5.smithi104.stdout:Mar 18 15:11:17 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-5[48148]: 2024-03-18T15:11:17.986+0000 7f6ec9766700 -1 osd.5 0 waiting for initial osdmap 2024-03-18T15:11:18.182 INFO:journalctl@ceph.osd.5.smithi104.stdout:Mar 18 15:11:17 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-5[48148]: 2024-03-18T15:11:17.994+0000 7f6ec5559700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T15:11:18.238 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:17 smithi019 ceph-mon[31522]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-18T15:11:18.238 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:17 smithi019 ceph-mon[31522]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-18T15:11:18.238 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:17 smithi019 ceph-mon[31522]: from='osd.5 [v2:172.21.15.104:6808/1258208297,v1:172.21.15.104:6809/1258208297]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-03-18T15:11:18.238 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:17 smithi019 ceph-mon[31522]: osdmap e36: 6 total, 5 up, 6 in 2024-03-18T15:11:18.238 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:17 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T15:11:18.238 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:17 smithi019 ceph-mon[31522]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-03-18T15:11:18.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:17 smithi033 ceph-mon[36648]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-18T15:11:18.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:17 smithi033 ceph-mon[36648]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-18T15:11:18.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:17 smithi033 ceph-mon[36648]: from='osd.5 [v2:172.21.15.104:6808/1258208297,v1:172.21.15.104:6809/1258208297]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-03-18T15:11:18.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:17 smithi033 ceph-mon[36648]: osdmap e36: 6 total, 5 up, 6 in 2024-03-18T15:11:18.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:17 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T15:11:18.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:17 smithi033 ceph-mon[36648]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-03-18T15:11:19.174 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:18 smithi104 ceph-mon[36070]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2024-03-18T15:11:19.174 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:18 smithi104 ceph-mon[36070]: osdmap e37: 6 total, 5 up, 6 in 2024-03-18T15:11:19.174 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:18 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T15:11:19.174 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:18 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T15:11:19.247 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:18 smithi019 ceph-mon[31522]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2024-03-18T15:11:19.247 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:18 smithi019 ceph-mon[31522]: osdmap e37: 6 total, 5 up, 6 in 2024-03-18T15:11:19.247 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:19 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T15:11:19.247 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:19 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T15:11:19.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:18 smithi033 ceph-mon[36648]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2024-03-18T15:11:19.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:18 smithi033 ceph-mon[36648]: osdmap e37: 6 total, 5 up, 6 in 2024-03-18T15:11:19.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:18 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T15:11:19.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:18 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T15:11:20.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:20 smithi033 ceph-mon[36648]: purged_snaps scrub starts 2024-03-18T15:11:20.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:20 smithi033 ceph-mon[36648]: purged_snaps scrub ok 2024-03-18T15:11:20.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:20 smithi033 ceph-mon[36648]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-18T15:11:20.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:20 smithi033 ceph-mon[36648]: osd.5 [v2:172.21.15.104:6808/1258208297,v1:172.21.15.104:6809/1258208297] boot 2024-03-18T15:11:20.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:20 smithi033 ceph-mon[36648]: osdmap e38: 6 total, 6 up, 6 in 2024-03-18T15:11:20.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:20 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T15:11:20.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:20 smithi033 ceph-mon[36648]: osdmap e39: 6 total, 6 up, 6 in 2024-03-18T15:11:20.341 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:20 smithi104 ceph-mon[36070]: purged_snaps scrub starts 2024-03-18T15:11:20.341 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:20 smithi104 ceph-mon[36070]: purged_snaps scrub ok 2024-03-18T15:11:20.341 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:20 smithi104 ceph-mon[36070]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-18T15:11:20.341 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:20 smithi104 ceph-mon[36070]: osd.5 [v2:172.21.15.104:6808/1258208297,v1:172.21.15.104:6809/1258208297] boot 2024-03-18T15:11:20.341 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:20 smithi104 ceph-mon[36070]: osdmap e38: 6 total, 6 up, 6 in 2024-03-18T15:11:20.341 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:20 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T15:11:20.341 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:20 smithi104 ceph-mon[36070]: osdmap e39: 6 total, 6 up, 6 in 2024-03-18T15:11:20.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:20 smithi019 ceph-mon[31522]: purged_snaps scrub starts 2024-03-18T15:11:20.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:20 smithi019 ceph-mon[31522]: purged_snaps scrub ok 2024-03-18T15:11:20.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:20 smithi019 ceph-mon[31522]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-18T15:11:20.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:20 smithi019 ceph-mon[31522]: osd.5 [v2:172.21.15.104:6808/1258208297,v1:172.21.15.104:6809/1258208297] boot 2024-03-18T15:11:20.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:20 smithi019 ceph-mon[31522]: osdmap e38: 6 total, 6 up, 6 in 2024-03-18T15:11:20.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:20 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T15:11:20.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:20 smithi019 ceph-mon[31522]: osdmap e39: 6 total, 6 up, 6 in 2024-03-18T15:11:20.959 INFO:teuthology.orchestra.run.smithi019.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-18T15:11:20.960 INFO:teuthology.orchestra.run.smithi019.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-18T15:11:20.960 INFO:teuthology.orchestra.run.smithi019.stderr: stderr: 10+0 records in 2024-03-18T15:11:20.960 INFO:teuthology.orchestra.run.smithi019.stderr:10+0 records out 2024-03-18T15:11:20.960 INFO:teuthology.orchestra.run.smithi019.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.036024 s, 291 MB/s 2024-03-18T15:11:20.960 INFO:teuthology.orchestra.run.smithi019.stderr:--> Zapping successful for: 2024-03-18T15:11:21.812 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:21 smithi019 ceph-mon[31522]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 168 MiB used, 536 GiB / 536 GiB avail 2024-03-18T15:11:21.813 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:21 smithi019 ceph-mon[31522]: Detected new or changed devices on smithi104 2024-03-18T15:11:21.813 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:21 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:11:21.813 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:21 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:11:21.813 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:21 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:11:21.813 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:21 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:11:21.813 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:21 smithi019 ceph-mon[31522]: Adjusting osd_memory_target on smithi104 to 8540M 2024-03-18T15:11:21.813 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:21 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:11:21.813 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:21 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:11:21.813 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:21 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:11:21.813 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:21 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:11:21.813 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:21 smithi019 ceph-mon[31522]: osdmap e40: 6 total, 6 up, 6 in 2024-03-18T15:11:21.817 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi019:vg_nvme/lv_2 2024-03-18T15:11:22.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:21 smithi033 ceph-mon[36648]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 168 MiB used, 536 GiB / 536 GiB avail 2024-03-18T15:11:22.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:21 smithi033 ceph-mon[36648]: Detected new or changed devices on smithi104 2024-03-18T15:11:22.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:21 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:11:22.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:21 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:11:22.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:21 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:11:22.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:21 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:11:22.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:21 smithi033 ceph-mon[36648]: Adjusting osd_memory_target on smithi104 to 8540M 2024-03-18T15:11:22.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:21 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:11:22.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:21 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:11:22.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:21 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:11:22.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:21 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:11:22.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:21 smithi033 ceph-mon[36648]: osdmap e40: 6 total, 6 up, 6 in 2024-03-18T15:11:22.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:21 smithi104 ceph-mon[36070]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 168 MiB used, 536 GiB / 536 GiB avail 2024-03-18T15:11:22.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:21 smithi104 ceph-mon[36070]: Detected new or changed devices on smithi104 2024-03-18T15:11:22.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:21 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:11:22.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:21 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:11:22.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:21 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:11:22.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:21 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:11:22.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:21 smithi104 ceph-mon[36070]: Adjusting osd_memory_target on smithi104 to 8540M 2024-03-18T15:11:22.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:21 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:11:22.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:21 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:11:22.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:21 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:11:22.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:21 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:11:22.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:21 smithi104 ceph-mon[36070]: osdmap e40: 6 total, 6 up, 6 in 2024-03-18T15:11:22.231 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:11:22.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:22 smithi019 ceph-mon[31522]: osdmap e41: 6 total, 6 up, 6 in 2024-03-18T15:11:23.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:22 smithi104 ceph-mon[36070]: osdmap e41: 6 total, 6 up, 6 in 2024-03-18T15:11:23.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:22 smithi033 ceph-mon[36648]: osdmap e41: 6 total, 6 up, 6 in 2024-03-18T15:11:24.187 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:23 smithi019 ceph-mon[31522]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 168 MiB used, 536 GiB / 536 GiB avail 2024-03-18T15:11:24.187 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:23 smithi019 ceph-mon[31522]: osdmap e42: 6 total, 6 up, 6 in 2024-03-18T15:11:24.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:23 smithi104 ceph-mon[36070]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 168 MiB used, 536 GiB / 536 GiB avail 2024-03-18T15:11:24.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:23 smithi104 ceph-mon[36070]: osdmap e42: 6 total, 6 up, 6 in 2024-03-18T15:11:24.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:23 smithi033 ceph-mon[36648]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 168 MiB used, 536 GiB / 536 GiB avail 2024-03-18T15:11:24.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:23 smithi033 ceph-mon[36648]: osdmap e42: 6 total, 6 up, 6 in 2024-03-18T15:11:24.363 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:11:24.362+0000 7f3763200700 1 -- 172.21.15.19:0/1195065974 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3764102000 msgr2=0x7f37641088b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:11:24.363 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:11:24.362+0000 7f3763200700 1 --2- 172.21.15.19:0/1195065974 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3764102000 0x7f37641088b0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f375c009f90 tx=0x7f375c00b330 comp rx=0 tx=0).stop 2024-03-18T15:11:24.364 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:11:24.362+0000 7f3763200700 1 -- 172.21.15.19:0/1195065974 shutdown_connections 2024-03-18T15:11:24.364 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:11:24.362+0000 7f3763200700 1 --2- 172.21.15.19:0/1195065974 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3764102000 0x7f37641088b0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:11:24.364 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:11:24.362+0000 7f3763200700 1 --2- 172.21.15.19:0/1195065974 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3764100460 0x7f3764100880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:11:24.364 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:11:24.362+0000 7f3763200700 1 --2- 172.21.15.19:0/1195065974 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3764101660 0x7f3764101ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:11:24.364 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:11:24.362+0000 7f3763200700 1 -- 172.21.15.19:0/1195065974 >> 172.21.15.19:0/1195065974 conn(0x7f37640fba40 msgr2=0x7f37640fde60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:11:24.364 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:11:24.362+0000 7f3763200700 1 -- 172.21.15.19:0/1195065974 shutdown_connections 2024-03-18T15:11:24.364 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:11:24.362+0000 7f3763200700 1 -- 172.21.15.19:0/1195065974 wait complete. 2024-03-18T15:11:24.365 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:11:24.364+0000 7f3763200700 1 Processor -- start 2024-03-18T15:11:24.365 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:11:24.364+0000 7f3763200700 1 -- start start 2024-03-18T15:11:24.366 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:11:24.364+0000 7f3763200700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3764100460 0x7f376410a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:11:24.366 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:11:24.364+0000 7f3763200700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3764101660 0x7f376419d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:11:24.366 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:11:24.364+0000 7f3763200700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3764102000 0x7f376419f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:11:24.366 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:11:24.364+0000 7f3763200700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f376410cd30 con 0x7f3764100460 2024-03-18T15:11:24.366 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:11:24.364+0000 7f3763200700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f376410cbb0 con 0x7f3764102000 2024-03-18T15:11:24.366 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:11:24.364+0000 7f3763200700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f376410ceb0 con 0x7f3764101660 2024-03-18T15:11:24.366 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:11:24.365+0000 7f3761e00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3764100460 0x7f376410a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:11:24.366 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:11:24.365+0000 7f3761400700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3764101660 0x7f376419d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:11:24.366 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:11:24.365+0000 7f3762800700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3764102000 0x7f376419f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:11:24.366 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:11:24.365+0000 7f3761e00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3764100460 0x7f376410a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:39256/0 (socket says 172.21.15.19:39256) 2024-03-18T15:11:24.367 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:11:24.365+0000 7f3761e00700 1 -- 172.21.15.19:0/1921915275 learned_addr learned my addr 172.21.15.19:0/1921915275 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:11:24.367 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:11:24.365+0000 7f3762800700 1 -- 172.21.15.19:0/1921915275 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3764101660 msgr2=0x7f376419d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:11:24.367 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:11:24.365+0000 7f3762800700 1 --2- 172.21.15.19:0/1921915275 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3764101660 0x7f376419d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:11:24.367 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:11:24.365+0000 7f3762800700 1 -- 172.21.15.19:0/1921915275 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3764100460 msgr2=0x7f376410a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:11:24.368 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:11:24.365+0000 7f3762800700 1 --2- 172.21.15.19:0/1921915275 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3764100460 0x7f376410a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:11:24.368 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:11:24.365+0000 7f3762800700 1 -- 172.21.15.19:0/1921915275 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f376419ffe0 con 0x7f3764102000 2024-03-18T15:11:24.368 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:11:24.365+0000 7f3762800700 1 --2- 172.21.15.19:0/1921915275 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3764102000 0x7f376419f710 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f375c009f90 tx=0x7f375c007c50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:11:24.368 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:11:24.366+0000 7f375aa00700 1 -- 172.21.15.19:0/1921915275 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f375c021020 con 0x7f3764102000 2024-03-18T15:11:24.368 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:11:24.366+0000 7f375aa00700 1 -- 172.21.15.19:0/1921915275 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f375c01d950 con 0x7f3764102000 2024-03-18T15:11:24.368 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:11:24.366+0000 7f3763200700 1 -- 172.21.15.19:0/1921915275 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37641a0270 con 0x7f3764102000 2024-03-18T15:11:24.368 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:11:24.366+0000 7f375aa00700 1 -- 172.21.15.19:0/1921915275 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f375c0052b0 con 0x7f3764102000 2024-03-18T15:11:24.368 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:11:24.366+0000 7f3763200700 1 -- 172.21.15.19:0/1921915275 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37641a0770 con 0x7f3764102000 2024-03-18T15:11:24.368 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:11:24.367+0000 7f3763200700 1 -- 172.21.15.19:0/1921915275 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3764068f90 con 0x7f3764102000 2024-03-18T15:11:24.370 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:11:24.369+0000 7f375aa00700 1 -- 172.21.15.19:0/1921915275 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f375c024020 con 0x7f3764102000 2024-03-18T15:11:24.371 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:11:24.370+0000 7f375aa00700 1 --2- 172.21.15.19:0/1921915275 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f37480a2910 0x7f37480a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:11:24.372 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:11:24.370+0000 7f375aa00700 1 -- 172.21.15.19:0/1921915275 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 4533+0+0 (secure 0 0 0) 0x7f375c00f020 con 0x7f3764102000 2024-03-18T15:11:24.372 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:11:24.371+0000 7f3761e00700 1 --2- 172.21.15.19:0/1921915275 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f37480a2910 0x7f37480a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:11:24.372 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:11:24.371+0000 7f3761e00700 1 --2- 172.21.15.19:0/1921915275 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f37480a2910 0x7f37480a4dd0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f3754009750 tx=0x7f3754009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:11:24.374 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:11:24.373+0000 7f375aa00700 1 -- 172.21.15.19:0/1921915275 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f375c09d290 con 0x7f3764102000 2024-03-18T15:11:24.614 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:11:24.613+0000 7f3763200700 1 -- 172.21.15.19:0/1921915275 --> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi019:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f3764062710 con 0x7f37480a2910 2024-03-18T15:11:25.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:24 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T15:11:25.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:24 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T15:11:25.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:24 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:11:25.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:24 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T15:11:25.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:24 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T15:11:25.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:24 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:11:25.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:24 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T15:11:25.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:24 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T15:11:25.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:24 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:11:26.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:25 smithi104 ceph-mon[36070]: pgmap v157: 1 pgs: 1 peering; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 136 KiB/s, 0 objects/s recovering 2024-03-18T15:11:26.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:25 smithi104 ceph-mon[36070]: from='client.24251 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi019:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:11:26.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:25 smithi019 ceph-mon[31522]: pgmap v157: 1 pgs: 1 peering; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 136 KiB/s, 0 objects/s recovering 2024-03-18T15:11:26.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:25 smithi019 ceph-mon[31522]: from='client.24251 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi019:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:11:26.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:25 smithi033 ceph-mon[36648]: pgmap v157: 1 pgs: 1 peering; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 136 KiB/s, 0 objects/s recovering 2024-03-18T15:11:26.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:25 smithi033 ceph-mon[36648]: from='client.24251 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi019:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:11:28.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:27 smithi104 ceph-mon[36070]: pgmap v158: 1 pgs: 1 peering; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-18T15:11:28.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:27 smithi019 ceph-mon[31522]: pgmap v158: 1 pgs: 1 peering; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-18T15:11:28.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:27 smithi033 ceph-mon[36648]: pgmap v158: 1 pgs: 1 peering; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-18T15:11:29.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:28 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/3362748385' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0e92a2ef-43cd-487c-9203-e219158aef8a"}]: dispatch 2024-03-18T15:11:29.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:28 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/3362748385' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0e92a2ef-43cd-487c-9203-e219158aef8a"}]': finished 2024-03-18T15:11:29.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:28 smithi104 ceph-mon[36070]: osdmap e43: 7 total, 6 up, 7 in 2024-03-18T15:11:29.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:28 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T15:11:29.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:28 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/3362748385' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0e92a2ef-43cd-487c-9203-e219158aef8a"}]: dispatch 2024-03-18T15:11:29.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:28 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/3362748385' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0e92a2ef-43cd-487c-9203-e219158aef8a"}]': finished 2024-03-18T15:11:29.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:28 smithi019 ceph-mon[31522]: osdmap e43: 7 total, 6 up, 7 in 2024-03-18T15:11:29.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:28 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T15:11:29.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:28 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/3362748385' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0e92a2ef-43cd-487c-9203-e219158aef8a"}]: dispatch 2024-03-18T15:11:29.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:28 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/3362748385' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0e92a2ef-43cd-487c-9203-e219158aef8a"}]': finished 2024-03-18T15:11:29.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:28 smithi033 ceph-mon[36648]: osdmap e43: 7 total, 6 up, 7 in 2024-03-18T15:11:29.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:28 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T15:11:30.178 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:29 smithi104 ceph-mon[36070]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-03-18T15:11:30.178 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:29 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/2780728908' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T15:11:30.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:29 smithi019 ceph-mon[31522]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-03-18T15:11:30.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:29 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/2780728908' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T15:11:30.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:29 smithi033 ceph-mon[36648]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-03-18T15:11:30.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:29 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/2780728908' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T15:11:32.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:31 smithi104 ceph-mon[36070]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-18T15:11:32.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:31 smithi019 ceph-mon[31522]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-18T15:11:32.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:31 smithi033 ceph-mon[36648]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-18T15:11:34.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:33 smithi104 ceph-mon[36070]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-18T15:11:34.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:33 smithi019 ceph-mon[31522]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-18T15:11:34.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:33 smithi033 ceph-mon[36648]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-18T15:11:36.105 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:35 smithi019 ceph-mon[31522]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-18T15:11:36.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:35 smithi104 ceph-mon[36070]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-18T15:11:36.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:35 smithi033 ceph-mon[36648]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-18T15:11:37.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:37 smithi019 ceph-mon[31522]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-18T15:11:37.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:37 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-18T15:11:37.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:37 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:11:38.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:37 smithi104 ceph-mon[36070]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-18T15:11:38.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:37 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-18T15:11:38.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:37 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:11:38.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:37 smithi033 ceph-mon[36648]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-18T15:11:38.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:37 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-18T15:11:38.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:37 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:11:39.167 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:38 smithi019 ceph-mon[31522]: Deploying daemon osd.6 on smithi019 2024-03-18T15:11:39.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:38 smithi033 ceph-mon[36648]: Deploying daemon osd.6 on smithi019 2024-03-18T15:11:39.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:38 smithi104 ceph-mon[36070]: Deploying daemon osd.6 on smithi019 2024-03-18T15:11:40.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:39 smithi019 ceph-mon[31522]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-18T15:11:40.244 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:39 smithi104 ceph-mon[36070]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-18T15:11:40.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:39 smithi033 ceph-mon[36648]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-18T15:11:42.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:41 smithi019 ceph-mon[31522]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-18T15:11:42.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:41 smithi033 ceph-mon[36648]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-18T15:11:42.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:41 smithi104 ceph-mon[36070]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-18T15:11:44.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:43 smithi019 ceph-mon[31522]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-18T15:11:44.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:43 smithi033 ceph-mon[36648]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-18T15:11:44.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:43 smithi104 ceph-mon[36070]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-18T15:11:46.158 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:45 smithi019 ceph-mon[31522]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-18T15:11:46.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:45 smithi033 ceph-mon[36648]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-18T15:11:46.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:45 smithi104 ceph-mon[36070]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-18T15:11:47.653 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:47 smithi019 ceph-mon[31522]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-18T15:11:47.653 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:47 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:11:47.653 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:47 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:11:47.653 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:47 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:11:47.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:47 smithi104 ceph-mon[36070]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-18T15:11:47.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:47 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:11:47.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:47 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:11:47.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:47 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:11:47.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:47 smithi033 ceph-mon[36648]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-18T15:11:47.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:47 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:11:47.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:47 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:11:47.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:47 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:11:49.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:49 smithi104 ceph-mon[36070]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-18T15:11:49.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:49 smithi019 ceph-mon[31522]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-18T15:11:49.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:49 smithi033 ceph-mon[36648]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-18T15:11:51.468 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:11:51.466+0000 7f375aa00700 1 -- 172.21.15.19:0/1921915275 <== mgr.14152 v2:172.21.15.19:6800/2017714236 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f3764062710 con 0x7f37480a2910 2024-03-18T15:11:51.468 INFO:teuthology.orchestra.run.smithi019.stdout:Created osd(s) 6 on host 'smithi019' 2024-03-18T15:11:51.472 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:11:51.470+0000 7f3763200700 1 -- 172.21.15.19:0/1921915275 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f37480a2910 msgr2=0x7f37480a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:11:51.472 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:11:51.470+0000 7f3763200700 1 --2- 172.21.15.19:0/1921915275 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f37480a2910 0x7f37480a4dd0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f3754009750 tx=0x7f3754009310 comp rx=0 tx=0).stop 2024-03-18T15:11:51.472 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:11:51.470+0000 7f3763200700 1 -- 172.21.15.19:0/1921915275 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3764102000 msgr2=0x7f376419f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:11:51.472 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:11:51.470+0000 7f3763200700 1 --2- 172.21.15.19:0/1921915275 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3764102000 0x7f376419f710 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f375c009f90 tx=0x7f375c007c50 comp rx=0 tx=0).stop 2024-03-18T15:11:51.472 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:11:51.471+0000 7f3763200700 1 -- 172.21.15.19:0/1921915275 shutdown_connections 2024-03-18T15:11:51.472 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:11:51.471+0000 7f3763200700 1 --2- 172.21.15.19:0/1921915275 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f37480a2910 0x7f37480a4dd0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:11:51.472 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:11:51.471+0000 7f3763200700 1 --2- 172.21.15.19:0/1921915275 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3764102000 0x7f376419f710 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:11:51.472 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:11:51.471+0000 7f3763200700 1 --2- 172.21.15.19:0/1921915275 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3764100460 0x7f376410a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:11:51.473 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:11:51.471+0000 7f3763200700 1 --2- 172.21.15.19:0/1921915275 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3764101660 0x7f376419d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:11:51.473 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:11:51.471+0000 7f3763200700 1 -- 172.21.15.19:0/1921915275 >> 172.21.15.19:0/1921915275 conn(0x7f37640fba40 msgr2=0x7f3764104890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:11:51.473 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:11:51.471+0000 7f3763200700 1 -- 172.21.15.19:0/1921915275 shutdown_connections 2024-03-18T15:11:51.473 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:11:51.471+0000 7f3763200700 1 -- 172.21.15.19:0/1921915275 wait complete. 2024-03-18T15:11:51.492 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:51 smithi019 ceph-mon[31522]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-18T15:11:51.492 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:51 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:11:51.492 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:51 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:11:51.492 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:51 smithi019 ceph-mon[31522]: from='osd.6 [v2:172.21.15.19:6818/2782482301,v1:172.21.15.19:6819/2782482301]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-18T15:11:51.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:51 smithi033 ceph-mon[36648]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-18T15:11:51.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:51 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:11:51.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:51 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:11:51.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:51 smithi033 ceph-mon[36648]: from='osd.6 [v2:172.21.15.19:6818/2782482301,v1:172.21.15.19:6819/2782482301]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-18T15:11:51.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:51 smithi104 ceph-mon[36070]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-18T15:11:51.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:51 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:11:51.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:51 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:11:51.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:51 smithi104 ceph-mon[36070]: from='osd.6 [v2:172.21.15.19:6818/2782482301,v1:172.21.15.19:6819/2782482301]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-18T15:11:52.313 DEBUG:teuthology.orchestra.run.smithi019:osd.6> sudo journalctl -f -n 0 -u ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@osd.6.service 2024-03-18T15:11:52.317 INFO:tasks.cephadm:Deploying osd.7 on smithi033 with /dev/vg_nvme/lv_2... 2024-03-18T15:11:52.317 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-18T15:11:52.568 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:52 smithi019 ceph-mon[31522]: from='osd.6 [v2:172.21.15.19:6818/2782482301,v1:172.21.15.19:6819/2782482301]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-18T15:11:52.568 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:52 smithi019 ceph-mon[31522]: osdmap e44: 7 total, 6 up, 7 in 2024-03-18T15:11:52.568 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:52 smithi019 ceph-mon[31522]: from='osd.6 [v2:172.21.15.19:6818/2782482301,v1:172.21.15.19:6819/2782482301]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]: dispatch 2024-03-18T15:11:52.568 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:52 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T15:11:52.568 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:52 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:11:52.568 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:52 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:11:52.579 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T15:11:52.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:52 smithi104 ceph-mon[36070]: from='osd.6 [v2:172.21.15.19:6818/2782482301,v1:172.21.15.19:6819/2782482301]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-18T15:11:52.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:52 smithi104 ceph-mon[36070]: osdmap e44: 7 total, 6 up, 7 in 2024-03-18T15:11:52.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:52 smithi104 ceph-mon[36070]: from='osd.6 [v2:172.21.15.19:6818/2782482301,v1:172.21.15.19:6819/2782482301]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]: dispatch 2024-03-18T15:11:52.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:52 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T15:11:52.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:52 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:11:52.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:52 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:11:52.695 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:52 smithi033 ceph-mon[36648]: from='osd.6 [v2:172.21.15.19:6818/2782482301,v1:172.21.15.19:6819/2782482301]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-18T15:11:52.695 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:52 smithi033 ceph-mon[36648]: osdmap e44: 7 total, 6 up, 7 in 2024-03-18T15:11:52.695 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:52 smithi033 ceph-mon[36648]: from='osd.6 [v2:172.21.15.19:6818/2782482301,v1:172.21.15.19:6819/2782482301]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]: dispatch 2024-03-18T15:11:52.695 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:52 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T15:11:52.695 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:52 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:11:52.695 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:52 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:11:53.216 INFO:journalctl@ceph.osd.6.smithi019.stdout:Mar 18 15:11:52 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-6[59600]: 2024-03-18T15:11:52.870+0000 7fde55800700 -1 osd.6 0 waiting for initial osdmap 2024-03-18T15:11:53.216 INFO:journalctl@ceph.osd.6.smithi019.stdout:Mar 18 15:11:52 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-6[59600]: 2024-03-18T15:11:52.896+0000 7fde4b600700 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T15:11:53.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:53 smithi033 ceph-mon[36648]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-18T15:11:53.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:53 smithi033 ceph-mon[36648]: from='osd.6 [v2:172.21.15.19:6818/2782482301,v1:172.21.15.19:6819/2782482301]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]': finished 2024-03-18T15:11:53.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:53 smithi033 ceph-mon[36648]: osdmap e45: 7 total, 6 up, 7 in 2024-03-18T15:11:53.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:53 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T15:11:53.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:53 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T15:11:53.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:53 smithi033 ceph-mon[36648]: from='osd.6 [v2:172.21.15.19:6818/2782482301,v1:172.21.15.19:6819/2782482301]' entity='osd.6' 2024-03-18T15:11:53.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:53 smithi104 ceph-mon[36070]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-18T15:11:53.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:53 smithi104 ceph-mon[36070]: from='osd.6 [v2:172.21.15.19:6818/2782482301,v1:172.21.15.19:6819/2782482301]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]': finished 2024-03-18T15:11:53.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:53 smithi104 ceph-mon[36070]: osdmap e45: 7 total, 6 up, 7 in 2024-03-18T15:11:53.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:53 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T15:11:53.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:53 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T15:11:53.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:53 smithi104 ceph-mon[36070]: from='osd.6 [v2:172.21.15.19:6818/2782482301,v1:172.21.15.19:6819/2782482301]' entity='osd.6' 2024-03-18T15:11:53.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:53 smithi019 ceph-mon[31522]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-18T15:11:53.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:53 smithi019 ceph-mon[31522]: from='osd.6 [v2:172.21.15.19:6818/2782482301,v1:172.21.15.19:6819/2782482301]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]': finished 2024-03-18T15:11:53.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:53 smithi019 ceph-mon[31522]: osdmap e45: 7 total, 6 up, 7 in 2024-03-18T15:11:53.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:53 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T15:11:53.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:53 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T15:11:53.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:53 smithi019 ceph-mon[31522]: from='osd.6 [v2:172.21.15.19:6818/2782482301,v1:172.21.15.19:6819/2782482301]' entity='osd.6' 2024-03-18T15:11:54.349 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:54 smithi019 ceph-mon[31522]: purged_snaps scrub starts 2024-03-18T15:11:54.349 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:54 smithi019 ceph-mon[31522]: purged_snaps scrub ok 2024-03-18T15:11:54.350 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:54 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T15:11:54.350 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:54 smithi019 ceph-mon[31522]: osd.6 [v2:172.21.15.19:6818/2782482301,v1:172.21.15.19:6819/2782482301] boot 2024-03-18T15:11:54.350 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:54 smithi019 ceph-mon[31522]: osdmap e46: 7 total, 7 up, 7 in 2024-03-18T15:11:54.350 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:54 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T15:11:54.537 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:54 smithi033 ceph-mon[36648]: purged_snaps scrub starts 2024-03-18T15:11:54.537 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:54 smithi033 ceph-mon[36648]: purged_snaps scrub ok 2024-03-18T15:11:54.537 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:54 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T15:11:54.537 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:54 smithi033 ceph-mon[36648]: osd.6 [v2:172.21.15.19:6818/2782482301,v1:172.21.15.19:6819/2782482301] boot 2024-03-18T15:11:54.537 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:54 smithi033 ceph-mon[36648]: osdmap e46: 7 total, 7 up, 7 in 2024-03-18T15:11:54.537 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:54 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T15:11:54.585 INFO:teuthology.orchestra.run.smithi033.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-18T15:11:54.585 INFO:teuthology.orchestra.run.smithi033.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-18T15:11:54.586 INFO:teuthology.orchestra.run.smithi033.stderr: stderr: 10+0 records in 2024-03-18T15:11:54.586 INFO:teuthology.orchestra.run.smithi033.stderr:10+0 records out 2024-03-18T15:11:54.586 INFO:teuthology.orchestra.run.smithi033.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0111095 s, 944 MB/s 2024-03-18T15:11:54.586 INFO:teuthology.orchestra.run.smithi033.stderr:--> Zapping successful for: 2024-03-18T15:11:54.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:54 smithi104 ceph-mon[36070]: purged_snaps scrub starts 2024-03-18T15:11:54.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:54 smithi104 ceph-mon[36070]: purged_snaps scrub ok 2024-03-18T15:11:54.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:54 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T15:11:54.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:54 smithi104 ceph-mon[36070]: osd.6 [v2:172.21.15.19:6818/2782482301,v1:172.21.15.19:6819/2782482301] boot 2024-03-18T15:11:54.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:54 smithi104 ceph-mon[36070]: osdmap e46: 7 total, 7 up, 7 in 2024-03-18T15:11:54.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:54 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T15:11:55.435 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi033:vg_nvme/lv_2 2024-03-18T15:11:55.682 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T15:11:56.094 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:55 smithi033 ceph-mon[36648]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-18T15:11:56.094 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:55 smithi033 ceph-mon[36648]: osdmap e47: 7 total, 7 up, 7 in 2024-03-18T15:11:56.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:55 smithi104 ceph-mon[36070]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-18T15:11:56.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:55 smithi104 ceph-mon[36070]: osdmap e47: 7 total, 7 up, 7 in 2024-03-18T15:11:56.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:55 smithi019 ceph-mon[31522]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-18T15:11:56.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:55 smithi019 ceph-mon[31522]: osdmap e47: 7 total, 7 up, 7 in 2024-03-18T15:11:57.331 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:11:57.329+0000 7fb6971ae700 1 -- 172.21.15.33:0/869869548 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb690105760 msgr2=0x7fb690107b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:11:57.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:11:57.329+0000 7fb6971ae700 1 --2- 172.21.15.33:0/869869548 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb690105760 0x7fb690107b50 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb67c009990 tx=0x7fb67c009ca0 comp rx=0 tx=0).stop 2024-03-18T15:11:57.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:11:57.329+0000 7fb6971ae700 1 -- 172.21.15.33:0/869869548 shutdown_connections 2024-03-18T15:11:57.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:11:57.329+0000 7fb6971ae700 1 --2- 172.21.15.33:0/869869548 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb690108090 0x7fb69010a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:11:57.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:11:57.329+0000 7fb6971ae700 1 --2- 172.21.15.33:0/869869548 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb690102e30 0x7fb690105220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:11:57.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:11:57.329+0000 7fb6971ae700 1 --2- 172.21.15.33:0/869869548 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb690105760 0x7fb690107b50 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:11:57.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:11:57.329+0000 7fb6971ae700 1 -- 172.21.15.33:0/869869548 >> 172.21.15.33:0/869869548 conn(0x7fb6900fc860 msgr2=0x7fb6900fec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:11:57.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:11:57.329+0000 7fb6971ae700 1 -- 172.21.15.33:0/869869548 shutdown_connections 2024-03-18T15:11:57.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:11:57.329+0000 7fb6971ae700 1 -- 172.21.15.33:0/869869548 wait complete. 2024-03-18T15:11:57.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:11:57.330+0000 7fb6971ae700 1 Processor -- start 2024-03-18T15:11:57.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:11:57.330+0000 7fb6971ae700 1 -- start start 2024-03-18T15:11:57.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:11:57.330+0000 7fb6971ae700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb690102e30 0x7fb690198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:11:57.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:11:57.330+0000 7fb6971ae700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb690105760 0x7fb690198ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:11:57.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:11:57.330+0000 7fb6971ae700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb690108090 0x7fb69019fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:11:57.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:11:57.330+0000 7fb6971ae700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fb69010df80 con 0x7fb690105760 2024-03-18T15:11:57.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:11:57.330+0000 7fb6971ae700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb69010de00 con 0x7fb690102e30 2024-03-18T15:11:57.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:11:57.330+0000 7fb6971ae700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fb69010e100 con 0x7fb690108090 2024-03-18T15:11:57.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:11:57.330+0000 7fb694f4a700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb690102e30 0x7fb690198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:11:57.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:11:57.330+0000 7fb694f4a700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb690102e30 0x7fb690198590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56030/0 (socket says 172.21.15.33:56030) 2024-03-18T15:11:57.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:11:57.330+0000 7fb694f4a700 1 -- 172.21.15.33:0/1134160644 learned_addr learned my addr 172.21.15.33:0/1134160644 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-18T15:11:57.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:11:57.331+0000 7fb694f4a700 1 -- 172.21.15.33:0/1134160644 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb690108090 msgr2=0x7fb69019fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:11:57.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:11:57.331+0000 7fb694f4a700 1 --2- 172.21.15.33:0/1134160644 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb690108090 0x7fb69019fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:11:57.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:11:57.331+0000 7fb694f4a700 1 -- 172.21.15.33:0/1134160644 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb690105760 msgr2=0x7fb690198ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:11:57.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:11:57.331+0000 7fb694f4a700 1 --2- 172.21.15.33:0/1134160644 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb690105760 0x7fb690198ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:11:57.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:11:57.331+0000 7fb694f4a700 1 -- 172.21.15.33:0/1134160644 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6901a01f0 con 0x7fb690102e30 2024-03-18T15:11:57.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:11:57.331+0000 7fb694f4a700 1 --2- 172.21.15.33:0/1134160644 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb690102e30 0x7fb690198590 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb68400b530 tx=0x7fb68400b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:11:57.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:11:57.331+0000 7fb68dffb700 1 -- 172.21.15.33:0/1134160644 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb684015020 con 0x7fb690102e30 2024-03-18T15:11:57.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:11:57.331+0000 7fb6971ae700 1 -- 172.21.15.33:0/1134160644 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6901a04e0 con 0x7fb690102e30 2024-03-18T15:11:57.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:11:57.331+0000 7fb68dffb700 1 -- 172.21.15.33:0/1134160644 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb684011d10 con 0x7fb690102e30 2024-03-18T15:11:57.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:11:57.331+0000 7fb68dffb700 1 -- 172.21.15.33:0/1134160644 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb68400f2e0 con 0x7fb690102e30 2024-03-18T15:11:57.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:11:57.331+0000 7fb6971ae700 1 -- 172.21.15.33:0/1134160644 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6901a0af0 con 0x7fb690102e30 2024-03-18T15:11:57.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:11:57.332+0000 7fb6971ae700 1 -- 172.21.15.33:0/1134160644 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb690068f90 con 0x7fb690102e30 2024-03-18T15:11:57.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:11:57.333+0000 7fb68dffb700 1 -- 172.21.15.33:0/1134160644 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fb684018020 con 0x7fb690102e30 2024-03-18T15:11:57.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:11:57.334+0000 7fb68dffb700 1 --2- 172.21.15.33:0/1134160644 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fb6800a2910 0x7fb6800a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:11:57.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:11:57.334+0000 7fb68dffb700 1 -- 172.21.15.33:0/1134160644 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 4965+0+0 (secure 0 0 0) 0x7fb6840cd570 con 0x7fb690102e30 2024-03-18T15:11:57.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:11:57.334+0000 7fb68ffff700 1 --2- 172.21.15.33:0/1134160644 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fb6800a2910 0x7fb6800a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:11:57.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:11:57.335+0000 7fb68ffff700 1 --2- 172.21.15.33:0/1134160644 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fb6800a2910 0x7fb6800a4dd0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fb67c005f50 tx=0x7fb67c005e80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:11:57.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:11:57.338+0000 7fb68dffb700 1 -- 172.21.15.33:0/1134160644 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb68401f020 con 0x7fb690102e30 2024-03-18T15:11:57.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:11:57.553+0000 7fb6971ae700 1 -- 172.21.15.33:0/1134160644 --> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fb690062710 con 0x7fb6800a2910 2024-03-18T15:11:58.185 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:57 smithi033 ceph-mon[36648]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-18T15:11:58.186 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:57 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:11:58.186 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:57 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:11:58.186 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:57 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:11:58.186 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:57 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:11:58.186 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:57 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:11:58.186 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:57 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:11:58.186 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:57 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:11:58.186 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:57 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:11:58.186 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:57 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:11:58.186 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:57 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T15:11:58.187 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:57 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T15:11:58.187 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:57 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:11:58.187 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:57 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:11:58.187 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:57 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:11:58.187 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:57 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:11:58.187 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:57 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:11:58.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:57 smithi104 ceph-mon[36070]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-18T15:11:58.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:57 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:11:58.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:57 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:11:58.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:57 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:11:58.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:57 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:11:58.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:57 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:11:58.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:57 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:11:58.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:57 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:11:58.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:57 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:11:58.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:57 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:11:58.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:57 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T15:11:58.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:57 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T15:11:58.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:57 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:11:58.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:57 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:11:58.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:57 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:11:58.196 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:57 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:11:58.196 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:57 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:11:58.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:57 smithi019 ceph-mon[31522]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-18T15:11:58.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:57 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:11:58.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:57 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:11:58.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:57 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:11:58.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:57 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:11:58.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:57 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:11:58.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:57 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:11:58.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:57 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:11:58.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:57 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:11:58.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:57 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:11:58.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:57 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T15:11:58.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:57 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T15:11:58.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:57 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:11:58.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:57 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:11:58.217 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:57 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:11:58.217 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:57 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:11:58.217 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:57 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:11:59.158 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:58 smithi033 ceph-mon[36648]: Detected new or changed devices on smithi019 2024-03-18T15:11:59.158 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:58 smithi033 ceph-mon[36648]: Adjusting osd_memory_target on smithi019 to 5698M 2024-03-18T15:11:59.158 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:58 smithi033 ceph-mon[36648]: from='client.24278 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:11:59.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:58 smithi104 ceph-mon[36070]: Detected new or changed devices on smithi019 2024-03-18T15:11:59.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:58 smithi104 ceph-mon[36070]: Adjusting osd_memory_target on smithi019 to 5698M 2024-03-18T15:11:59.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:58 smithi104 ceph-mon[36070]: from='client.24278 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:11:59.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:58 smithi019 ceph-mon[31522]: Detected new or changed devices on smithi019 2024-03-18T15:11:59.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:58 smithi019 ceph-mon[31522]: Adjusting osd_memory_target on smithi019 to 5698M 2024-03-18T15:11:59.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:58 smithi019 ceph-mon[31522]: from='client.24278 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:12:00.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:11:59 smithi104 ceph-mon[36070]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-18T15:12:00.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:11:59 smithi019 ceph-mon[31522]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-18T15:12:00.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:11:59 smithi033 ceph-mon[36648]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-18T15:12:01.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:00 smithi104 ceph-mon[36070]: from='client.? 172.21.15.33:0/862200232' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a16e39a1-3685-491d-b376-3e80a2084072"}]: dispatch 2024-03-18T15:12:01.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:00 smithi104 ceph-mon[36070]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a16e39a1-3685-491d-b376-3e80a2084072"}]: dispatch 2024-03-18T15:12:01.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:00 smithi104 ceph-mon[36070]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a16e39a1-3685-491d-b376-3e80a2084072"}]': finished 2024-03-18T15:12:01.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:00 smithi104 ceph-mon[36070]: osdmap e48: 8 total, 7 up, 8 in 2024-03-18T15:12:01.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:00 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T15:12:01.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:00 smithi019 ceph-mon[31522]: from='client.? 172.21.15.33:0/862200232' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a16e39a1-3685-491d-b376-3e80a2084072"}]: dispatch 2024-03-18T15:12:01.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:00 smithi019 ceph-mon[31522]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a16e39a1-3685-491d-b376-3e80a2084072"}]: dispatch 2024-03-18T15:12:01.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:00 smithi019 ceph-mon[31522]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a16e39a1-3685-491d-b376-3e80a2084072"}]': finished 2024-03-18T15:12:01.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:00 smithi019 ceph-mon[31522]: osdmap e48: 8 total, 7 up, 8 in 2024-03-18T15:12:01.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:00 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T15:12:01.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:00 smithi033 ceph-mon[36648]: from='client.? 172.21.15.33:0/862200232' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a16e39a1-3685-491d-b376-3e80a2084072"}]: dispatch 2024-03-18T15:12:01.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:00 smithi033 ceph-mon[36648]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a16e39a1-3685-491d-b376-3e80a2084072"}]: dispatch 2024-03-18T15:12:01.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:00 smithi033 ceph-mon[36648]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a16e39a1-3685-491d-b376-3e80a2084072"}]': finished 2024-03-18T15:12:01.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:00 smithi033 ceph-mon[36648]: osdmap e48: 8 total, 7 up, 8 in 2024-03-18T15:12:01.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:00 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T15:12:02.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:01 smithi104 ceph-mon[36070]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-18T15:12:02.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:01 smithi104 ceph-mon[36070]: from='client.? 172.21.15.33:0/3264010460' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T15:12:02.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:01 smithi019 ceph-mon[31522]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-18T15:12:02.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:01 smithi019 ceph-mon[31522]: from='client.? 172.21.15.33:0/3264010460' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T15:12:02.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:01 smithi033 ceph-mon[36648]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-18T15:12:02.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:01 smithi033 ceph-mon[36648]: from='client.? 172.21.15.33:0/3264010460' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T15:12:04.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:03 smithi104 ceph-mon[36070]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-18T15:12:04.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:03 smithi019 ceph-mon[31522]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-18T15:12:04.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:03 smithi033 ceph-mon[36648]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-18T15:12:06.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:05 smithi104 ceph-mon[36070]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-18T15:12:06.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:05 smithi019 ceph-mon[31522]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-18T15:12:06.240 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:05 smithi033 ceph-mon[36648]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-18T15:12:08.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:07 smithi104 ceph-mon[36070]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-18T15:12:08.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:07 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-18T15:12:08.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:07 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:12:08.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:07 smithi019 ceph-mon[31522]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-18T15:12:08.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:07 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-18T15:12:08.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:07 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:12:08.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:07 smithi033 ceph-mon[36648]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-18T15:12:08.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:07 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-18T15:12:08.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:07 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:12:09.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:08 smithi104 ceph-mon[36070]: Deploying daemon osd.7 on smithi033 2024-03-18T15:12:09.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:08 smithi019 ceph-mon[31522]: Deploying daemon osd.7 on smithi033 2024-03-18T15:12:09.247 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:08 smithi033 ceph-mon[36648]: Deploying daemon osd.7 on smithi033 2024-03-18T15:12:10.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:09 smithi104 ceph-mon[36070]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-18T15:12:10.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:09 smithi019 ceph-mon[31522]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-18T15:12:10.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:09 smithi033 ceph-mon[36648]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-18T15:12:12.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:11 smithi019 ceph-mon[31522]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-18T15:12:12.220 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:11 smithi033 ceph-mon[36648]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-18T15:12:12.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:11 smithi104 ceph-mon[36070]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-18T15:12:14.125 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:13 smithi033 ceph-mon[36648]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-18T15:12:14.126 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:13 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:12:14.126 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:13 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:12:14.126 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:13 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:12:14.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:13 smithi019 ceph-mon[31522]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-18T15:12:14.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:13 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:12:14.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:13 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:12:14.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:13 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:12:14.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:13 smithi104 ceph-mon[36070]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-18T15:12:14.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:13 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:12:14.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:13 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:12:14.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:13 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:12:16.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:15 smithi019 ceph-mon[31522]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-18T15:12:16.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:15 smithi033 ceph-mon[36648]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-18T15:12:16.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:15 smithi104 ceph-mon[36070]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-18T15:12:17.188 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:12:17.188+0000 7fb68dffb700 1 -- 172.21.15.33:0/1134160644 <== mgr.14152 v2:172.21.15.19:6800/2017714236 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fb690062710 con 0x7fb6800a2910 2024-03-18T15:12:17.189 INFO:teuthology.orchestra.run.smithi033.stdout:Created osd(s) 7 on host 'smithi033' 2024-03-18T15:12:17.191 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:12:17.190+0000 7fb6971ae700 1 -- 172.21.15.33:0/1134160644 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fb6800a2910 msgr2=0x7fb6800a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:12:17.191 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:12:17.190+0000 7fb6971ae700 1 --2- 172.21.15.33:0/1134160644 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fb6800a2910 0x7fb6800a4dd0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fb67c005f50 tx=0x7fb67c005e80 comp rx=0 tx=0).stop 2024-03-18T15:12:17.191 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:12:17.190+0000 7fb6971ae700 1 -- 172.21.15.33:0/1134160644 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb690102e30 msgr2=0x7fb690198590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:12:17.191 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:12:17.190+0000 7fb6971ae700 1 --2- 172.21.15.33:0/1134160644 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb690102e30 0x7fb690198590 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb68400b530 tx=0x7fb68400b8f0 comp rx=0 tx=0).stop 2024-03-18T15:12:17.191 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:12:17.190+0000 7fb6971ae700 1 -- 172.21.15.33:0/1134160644 shutdown_connections 2024-03-18T15:12:17.191 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:12:17.190+0000 7fb6971ae700 1 --2- 172.21.15.33:0/1134160644 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fb6800a2910 0x7fb6800a4dd0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:12:17.191 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:12:17.190+0000 7fb6971ae700 1 --2- 172.21.15.33:0/1134160644 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb690108090 0x7fb69019fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:12:17.191 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:12:17.190+0000 7fb6971ae700 1 --2- 172.21.15.33:0/1134160644 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb690102e30 0x7fb690198590 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:12:17.191 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:12:17.190+0000 7fb6971ae700 1 --2- 172.21.15.33:0/1134160644 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb690105760 0x7fb690198ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:12:17.191 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:12:17.190+0000 7fb6971ae700 1 -- 172.21.15.33:0/1134160644 >> 172.21.15.33:0/1134160644 conn(0x7fb6900fc860 msgr2=0x7fb690103870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:12:17.191 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:12:17.190+0000 7fb6971ae700 1 -- 172.21.15.33:0/1134160644 shutdown_connections 2024-03-18T15:12:17.191 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:12:17.190+0000 7fb6971ae700 1 -- 172.21.15.33:0/1134160644 wait complete. 2024-03-18T15:12:17.504 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:17 smithi033 ceph-mon[36648]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-18T15:12:17.504 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:17 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:12:17.504 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:17 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:12:17.504 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:17 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:12:17.504 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:17 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:12:17.504 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:17 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:12:17.504 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:17 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:12:17.504 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:17 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:12:17.505 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:17 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:12:17.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:17 smithi104 ceph-mon[36070]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-18T15:12:17.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:17 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:12:17.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:17 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:12:17.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:17 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:12:17.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:17 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:12:17.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:17 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:12:17.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:17 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:12:17.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:17 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:12:17.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:17 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:12:17.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:17 smithi019 ceph-mon[31522]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-18T15:12:17.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:17 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:12:17.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:17 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:12:17.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:17 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:12:17.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:17 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:12:17.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:17 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:12:17.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:17 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:12:17.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:17 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:12:17.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:17 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:12:17.822 DEBUG:teuthology.orchestra.run.smithi033:osd.7> sudo journalctl -f -n 0 -u ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@osd.7.service 2024-03-18T15:12:17.825 INFO:tasks.cephadm:Deploying osd.8 on smithi104 with /dev/vg_nvme/lv_2... 2024-03-18T15:12:17.826 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-18T15:12:18.073 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T15:12:18.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:18 smithi033 ceph-mon[36648]: from='osd.7 [v2:172.21.15.33:6816/4238160988,v1:172.21.15.33:6817/4238160988]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-18T15:12:18.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:18 smithi033 ceph-mon[36648]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-18T15:12:18.520 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:18 smithi104 ceph-mon[36070]: from='osd.7 [v2:172.21.15.33:6816/4238160988,v1:172.21.15.33:6817/4238160988]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-18T15:12:18.520 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:18 smithi104 ceph-mon[36070]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-18T15:12:18.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:18 smithi019 ceph-mon[31522]: from='osd.7 [v2:172.21.15.33:6816/4238160988,v1:172.21.15.33:6817/4238160988]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-18T15:12:18.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:18 smithi019 ceph-mon[31522]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-18T15:12:19.639 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:19 smithi104 ceph-mon[36070]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-18T15:12:19.639 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:19 smithi104 ceph-mon[36070]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-18T15:12:19.639 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:19 smithi104 ceph-mon[36070]: from='osd.7 [v2:172.21.15.33:6816/4238160988,v1:172.21.15.33:6817/4238160988]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-03-18T15:12:19.639 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:19 smithi104 ceph-mon[36070]: osdmap e49: 8 total, 7 up, 8 in 2024-03-18T15:12:19.639 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:19 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T15:12:19.639 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:19 smithi104 ceph-mon[36070]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-03-18T15:12:19.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:19 smithi019 ceph-mon[31522]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-18T15:12:19.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:19 smithi019 ceph-mon[31522]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-18T15:12:19.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:19 smithi019 ceph-mon[31522]: from='osd.7 [v2:172.21.15.33:6816/4238160988,v1:172.21.15.33:6817/4238160988]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-03-18T15:12:19.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:19 smithi019 ceph-mon[31522]: osdmap e49: 8 total, 7 up, 8 in 2024-03-18T15:12:19.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:19 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T15:12:19.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:19 smithi019 ceph-mon[31522]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-03-18T15:12:19.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:19 smithi033 ceph-mon[36648]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-18T15:12:19.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:19 smithi033 ceph-mon[36648]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-18T15:12:19.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:19 smithi033 ceph-mon[36648]: from='osd.7 [v2:172.21.15.33:6816/4238160988,v1:172.21.15.33:6817/4238160988]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-03-18T15:12:19.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:19 smithi033 ceph-mon[36648]: osdmap e49: 8 total, 7 up, 8 in 2024-03-18T15:12:19.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:19 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T15:12:19.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:19 smithi033 ceph-mon[36648]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-03-18T15:12:19.757 INFO:journalctl@ceph.osd.7.smithi033.stdout:Mar 18 15:12:19 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-7[54818]: 2024-03-18T15:12:19.303+0000 7f480fb91700 -1 osd.7 0 waiting for initial osdmap 2024-03-18T15:12:19.757 INFO:journalctl@ceph.osd.7.smithi033.stdout:Mar 18 15:12:19 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-7[54818]: 2024-03-18T15:12:19.315+0000 7f480b984700 -1 osd.7 50 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T15:12:20.230 INFO:teuthology.orchestra.run.smithi104.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-18T15:12:20.230 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-18T15:12:20.230 INFO:teuthology.orchestra.run.smithi104.stderr: stderr: 10+0 records in 2024-03-18T15:12:20.230 INFO:teuthology.orchestra.run.smithi104.stderr:10+0 records out 2024-03-18T15:12:20.230 INFO:teuthology.orchestra.run.smithi104.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0116656 s, 899 MB/s 2024-03-18T15:12:20.231 INFO:teuthology.orchestra.run.smithi104.stderr:--> Zapping successful for: 2024-03-18T15:12:20.530 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:20 smithi104 ceph-mon[36070]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]': finished 2024-03-18T15:12:20.530 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:20 smithi104 ceph-mon[36070]: osdmap e50: 8 total, 7 up, 8 in 2024-03-18T15:12:20.530 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:20 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T15:12:20.595 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:20 smithi033 ceph-mon[36648]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]': finished 2024-03-18T15:12:20.595 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:20 smithi033 ceph-mon[36648]: osdmap e50: 8 total, 7 up, 8 in 2024-03-18T15:12:20.595 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:20 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T15:12:20.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:20 smithi019 ceph-mon[31522]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]': finished 2024-03-18T15:12:20.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:20 smithi019 ceph-mon[31522]: osdmap e50: 8 total, 7 up, 8 in 2024-03-18T15:12:20.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:20 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T15:12:20.910 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi104:vg_nvme/lv_2 2024-03-18T15:12:21.158 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T15:12:21.552 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:21 smithi104 ceph-mon[36070]: purged_snaps scrub starts 2024-03-18T15:12:21.552 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:21 smithi104 ceph-mon[36070]: purged_snaps scrub ok 2024-03-18T15:12:21.552 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:21 smithi104 ceph-mon[36070]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-18T15:12:21.552 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:21 smithi104 ceph-mon[36070]: osd.7 [v2:172.21.15.33:6816/4238160988,v1:172.21.15.33:6817/4238160988] boot 2024-03-18T15:12:21.552 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:21 smithi104 ceph-mon[36070]: osdmap e51: 8 total, 8 up, 8 in 2024-03-18T15:12:21.552 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:21 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T15:12:21.552 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:21 smithi104 ceph-mon[36070]: Detected new or changed devices on smithi033 2024-03-18T15:12:21.552 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:21 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:12:21.553 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:21 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:12:21.553 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:21 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:12:21.553 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:21 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:12:21.553 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:21 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:12:21.553 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:21 smithi104 ceph-mon[36070]: Adjusting osd_memory_target on smithi033 to 5694M 2024-03-18T15:12:21.553 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:21 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:12:21.553 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:21 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:12:21.553 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:21 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:12:21.553 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:21 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:12:21.553 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:21 smithi104 ceph-mon[36070]: osdmap e52: 8 total, 8 up, 8 in 2024-03-18T15:12:21.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:21 smithi019 ceph-mon[31522]: purged_snaps scrub starts 2024-03-18T15:12:21.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:21 smithi019 ceph-mon[31522]: purged_snaps scrub ok 2024-03-18T15:12:21.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:21 smithi019 ceph-mon[31522]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-18T15:12:21.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:21 smithi019 ceph-mon[31522]: osd.7 [v2:172.21.15.33:6816/4238160988,v1:172.21.15.33:6817/4238160988] boot 2024-03-18T15:12:21.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:21 smithi019 ceph-mon[31522]: osdmap e51: 8 total, 8 up, 8 in 2024-03-18T15:12:21.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:21 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T15:12:21.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:21 smithi019 ceph-mon[31522]: Detected new or changed devices on smithi033 2024-03-18T15:12:21.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:21 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:12:21.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:21 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:12:21.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:21 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:12:21.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:21 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:12:21.717 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:21 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:12:21.717 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:21 smithi019 ceph-mon[31522]: Adjusting osd_memory_target on smithi033 to 5694M 2024-03-18T15:12:21.717 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:21 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:12:21.717 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:21 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:12:21.717 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:21 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:12:21.717 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:21 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:12:21.717 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:21 smithi019 ceph-mon[31522]: osdmap e52: 8 total, 8 up, 8 in 2024-03-18T15:12:21.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:21 smithi033 ceph-mon[36648]: purged_snaps scrub starts 2024-03-18T15:12:21.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:21 smithi033 ceph-mon[36648]: purged_snaps scrub ok 2024-03-18T15:12:21.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:21 smithi033 ceph-mon[36648]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-18T15:12:21.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:21 smithi033 ceph-mon[36648]: osd.7 [v2:172.21.15.33:6816/4238160988,v1:172.21.15.33:6817/4238160988] boot 2024-03-18T15:12:21.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:21 smithi033 ceph-mon[36648]: osdmap e51: 8 total, 8 up, 8 in 2024-03-18T15:12:21.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:21 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T15:12:21.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:21 smithi033 ceph-mon[36648]: Detected new or changed devices on smithi033 2024-03-18T15:12:21.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:21 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:12:21.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:21 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:12:21.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:21 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:12:21.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:21 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:12:21.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:21 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:12:21.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:21 smithi033 ceph-mon[36648]: Adjusting osd_memory_target on smithi033 to 5694M 2024-03-18T15:12:21.758 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:21 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:12:21.758 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:21 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:12:21.758 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:21 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:12:21.758 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:21 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:12:21.758 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:21 smithi033 ceph-mon[36648]: osdmap e52: 8 total, 8 up, 8 in 2024-03-18T15:12:22.842 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:12:22.835+0000 7f44d84b4700 1 -- 172.21.15.104:0/3719172874 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f44d01011f0 msgr2=0x7f44d0101610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:12:22.842 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:12:22.835+0000 7f44d84b4700 1 --2- 172.21.15.104:0/3719172874 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f44d01011f0 0x7f44d0101610 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f44cc00b0a0 tx=0x7f44cc00b3b0 comp rx=0 tx=0).stop 2024-03-18T15:12:22.842 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:12:22.835+0000 7f44d84b4700 1 -- 172.21.15.104:0/3719172874 shutdown_connections 2024-03-18T15:12:22.842 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:12:22.835+0000 7f44d84b4700 1 --2- 172.21.15.104:0/3719172874 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f44d01011f0 0x7f44d0101610 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:12:22.842 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:12:22.835+0000 7f44d84b4700 1 --2- 172.21.15.104:0/3719172874 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f44d0102d90 0x7f44d0109640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:12:22.842 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:12:22.835+0000 7f44d84b4700 1 --2- 172.21.15.104:0/3719172874 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f44d01023f0 0x7f44d0102850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:12:22.842 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:12:22.835+0000 7f44d84b4700 1 -- 172.21.15.104:0/3719172874 >> 172.21.15.104:0/3719172874 conn(0x7f44d00fc790 msgr2=0x7f44d00febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:12:22.842 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:12:22.835+0000 7f44d84b4700 1 -- 172.21.15.104:0/3719172874 shutdown_connections 2024-03-18T15:12:22.842 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:12:22.835+0000 7f44d84b4700 1 -- 172.21.15.104:0/3719172874 wait complete. 2024-03-18T15:12:22.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:12:22.836+0000 7f44d84b4700 1 Processor -- start 2024-03-18T15:12:22.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:12:22.836+0000 7f44d84b4700 1 -- start start 2024-03-18T15:12:22.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:12:22.836+0000 7f44d84b4700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f44d01011f0 0x7f44d006b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:12:22.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:12:22.836+0000 7f44d84b4700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f44d01023f0 0x7f44d019d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:12:22.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:12:22.836+0000 7f44d84b4700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f44d0102d90 0x7f44d019f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:12:22.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:12:22.836+0000 7f44d84b4700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f44d010ce30 con 0x7f44d01011f0 2024-03-18T15:12:22.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:12:22.836+0000 7f44d84b4700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f44d010ccb0 con 0x7f44d0102d90 2024-03-18T15:12:22.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:12:22.836+0000 7f44d84b4700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f44d010cfb0 con 0x7f44d01023f0 2024-03-18T15:12:22.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:12:22.836+0000 7f44d5a4f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f44d01023f0 0x7f44d019d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:12:22.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:12:22.836+0000 7f44d5a4f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f44d01023f0 0x7f44d019d230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42662/0 (socket says 172.21.15.104:42662) 2024-03-18T15:12:22.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:12:22.836+0000 7f44d5a4f700 1 -- 172.21.15.104:0/958076577 learned_addr learned my addr 172.21.15.104:0/958076577 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-18T15:12:22.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:12:22.836+0000 7f44d6a51700 1 --2- 172.21.15.104:0/958076577 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f44d0102d90 0x7f44d019f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:12:22.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:12:22.836+0000 7f44d5a4f700 1 -- 172.21.15.104:0/958076577 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f44d0102d90 msgr2=0x7f44d019f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:12:22.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:12:22.836+0000 7f44d5a4f700 1 --2- 172.21.15.104:0/958076577 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f44d0102d90 0x7f44d019f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:12:22.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:12:22.836+0000 7f44d5a4f700 1 -- 172.21.15.104:0/958076577 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f44d01011f0 msgr2=0x7f44d006b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:12:22.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:12:22.836+0000 7f44d5a4f700 1 --2- 172.21.15.104:0/958076577 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f44d01011f0 0x7f44d006b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:12:22.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:12:22.836+0000 7f44d5a4f700 1 -- 172.21.15.104:0/958076577 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44d01a0150 con 0x7f44d01023f0 2024-03-18T15:12:22.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:12:22.837+0000 7f44d5a4f700 1 --2- 172.21.15.104:0/958076577 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f44d01023f0 0x7f44d019d230 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f44c000b530 tx=0x7f44c000b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:12:22.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:12:22.837+0000 7f44c77fe700 1 -- 172.21.15.104:0/958076577 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44c0015020 con 0x7f44d01023f0 2024-03-18T15:12:22.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:12:22.837+0000 7f44c77fe700 1 -- 172.21.15.104:0/958076577 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f44c0011d10 con 0x7f44d01023f0 2024-03-18T15:12:22.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:12:22.837+0000 7f44d84b4700 1 -- 172.21.15.104:0/958076577 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44d01a0440 con 0x7f44d01023f0 2024-03-18T15:12:22.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:12:22.837+0000 7f44c77fe700 1 -- 172.21.15.104:0/958076577 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44c0010e50 con 0x7f44d01023f0 2024-03-18T15:12:22.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:12:22.837+0000 7f44d84b4700 1 -- 172.21.15.104:0/958076577 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44d01a0a00 con 0x7f44d01023f0 2024-03-18T15:12:22.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:12:22.838+0000 7f44d84b4700 1 -- 172.21.15.104:0/958076577 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44d0068f90 con 0x7f44d01023f0 2024-03-18T15:12:22.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:12:22.840+0000 7f44c77fe700 1 -- 172.21.15.104:0/958076577 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f44c0018020 con 0x7f44d01023f0 2024-03-18T15:12:22.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:12:22.840+0000 7f44c77fe700 1 --2- 172.21.15.104:0/958076577 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f44bc0a2a30 0x7f44bc0a4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:12:22.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:12:22.840+0000 7f44c77fe700 1 -- 172.21.15.104:0/958076577 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 5426+0+0 (secure 0 0 0) 0x7f44c00cdce0 con 0x7f44d01023f0 2024-03-18T15:12:22.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:12:22.841+0000 7f44d6250700 1 --2- 172.21.15.104:0/958076577 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f44bc0a2a30 0x7f44bc0a4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:12:22.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:12:22.842+0000 7f44d6250700 1 --2- 172.21.15.104:0/958076577 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f44bc0a2a30 0x7f44bc0a4ef0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f44cc009be0 tx=0x7f44cc008fe0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:12:22.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:12:22.844+0000 7f44c77fe700 1 -- 172.21.15.104:0/958076577 <== mon.1 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f44c001f020 con 0x7f44d01023f0 2024-03-18T15:12:23.052 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:12:23.051+0000 7f44d84b4700 1 -- 172.21.15.104:0/958076577 --> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f44d0062710 con 0x7f44bc0a2a30 2024-03-18T15:12:23.684 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:23 smithi104 ceph-mon[36070]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-03-18T15:12:23.684 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:23 smithi104 ceph-mon[36070]: osdmap e53: 8 total, 8 up, 8 in 2024-03-18T15:12:23.684 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:23 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T15:12:23.685 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:23 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T15:12:23.685 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:23 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:12:23.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:23 smithi019 ceph-mon[31522]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-03-18T15:12:23.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:23 smithi019 ceph-mon[31522]: osdmap e53: 8 total, 8 up, 8 in 2024-03-18T15:12:23.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:23 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T15:12:23.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:23 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T15:12:23.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:23 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:12:23.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:23 smithi033 ceph-mon[36648]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-03-18T15:12:23.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:23 smithi033 ceph-mon[36648]: osdmap e53: 8 total, 8 up, 8 in 2024-03-18T15:12:23.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:23 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T15:12:23.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:23 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T15:12:23.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:23 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:12:24.682 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:24 smithi104 ceph-mon[36070]: from='client.24310 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:12:24.682 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:24 smithi104 ceph-mon[36070]: osdmap e54: 8 total, 8 up, 8 in 2024-03-18T15:12:24.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:24 smithi019 ceph-mon[31522]: from='client.24310 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:12:24.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:24 smithi019 ceph-mon[31522]: osdmap e54: 8 total, 8 up, 8 in 2024-03-18T15:12:24.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:24 smithi033 ceph-mon[36648]: from='client.24310 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:12:24.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:24 smithi033 ceph-mon[36648]: osdmap e54: 8 total, 8 up, 8 in 2024-03-18T15:12:25.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:25 smithi019 ceph-mon[31522]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-03-18T15:12:25.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:25 smithi019 ceph-mon[31522]: osdmap e55: 8 total, 8 up, 8 in 2024-03-18T15:12:25.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:25 smithi104 ceph-mon[36070]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-03-18T15:12:25.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:25 smithi104 ceph-mon[36070]: osdmap e55: 8 total, 8 up, 8 in 2024-03-18T15:12:25.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:25 smithi033 ceph-mon[36648]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-03-18T15:12:25.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:25 smithi033 ceph-mon[36648]: osdmap e55: 8 total, 8 up, 8 in 2024-03-18T15:12:26.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:26 smithi104 ceph-mon[36070]: from='client.? 172.21.15.104:0/2590627913' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d5c3fa92-cd5d-4349-9c7d-d5666bacde16"}]: dispatch 2024-03-18T15:12:26.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:26 smithi104 ceph-mon[36070]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d5c3fa92-cd5d-4349-9c7d-d5666bacde16"}]: dispatch 2024-03-18T15:12:26.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:26 smithi104 ceph-mon[36070]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d5c3fa92-cd5d-4349-9c7d-d5666bacde16"}]': finished 2024-03-18T15:12:26.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:26 smithi104 ceph-mon[36070]: osdmap e56: 9 total, 8 up, 9 in 2024-03-18T15:12:26.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:26 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T15:12:26.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:26 smithi019 ceph-mon[31522]: from='client.? 172.21.15.104:0/2590627913' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d5c3fa92-cd5d-4349-9c7d-d5666bacde16"}]: dispatch 2024-03-18T15:12:26.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:26 smithi019 ceph-mon[31522]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d5c3fa92-cd5d-4349-9c7d-d5666bacde16"}]: dispatch 2024-03-18T15:12:26.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:26 smithi019 ceph-mon[31522]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d5c3fa92-cd5d-4349-9c7d-d5666bacde16"}]': finished 2024-03-18T15:12:26.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:26 smithi019 ceph-mon[31522]: osdmap e56: 9 total, 8 up, 9 in 2024-03-18T15:12:26.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:26 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T15:12:26.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:26 smithi033 ceph-mon[36648]: from='client.? 172.21.15.104:0/2590627913' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d5c3fa92-cd5d-4349-9c7d-d5666bacde16"}]: dispatch 2024-03-18T15:12:26.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:26 smithi033 ceph-mon[36648]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d5c3fa92-cd5d-4349-9c7d-d5666bacde16"}]: dispatch 2024-03-18T15:12:26.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:26 smithi033 ceph-mon[36648]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d5c3fa92-cd5d-4349-9c7d-d5666bacde16"}]': finished 2024-03-18T15:12:26.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:26 smithi033 ceph-mon[36648]: osdmap e56: 9 total, 8 up, 9 in 2024-03-18T15:12:26.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:26 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T15:12:27.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:27 smithi104 ceph-mon[36070]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-18T15:12:27.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:27 smithi104 ceph-mon[36070]: from='client.? 172.21.15.104:0/38297830' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T15:12:27.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:27 smithi019 ceph-mon[31522]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-18T15:12:27.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:27 smithi019 ceph-mon[31522]: from='client.? 172.21.15.104:0/38297830' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T15:12:27.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:27 smithi033 ceph-mon[36648]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-18T15:12:27.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:27 smithi033 ceph-mon[36648]: from='client.? 172.21.15.104:0/38297830' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T15:12:29.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:29 smithi104 ceph-mon[36070]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-18T15:12:29.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:29 smithi019 ceph-mon[31522]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-18T15:12:29.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:29 smithi033 ceph-mon[36648]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-18T15:12:31.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:31 smithi104 ceph-mon[36070]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-18T15:12:31.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:31 smithi019 ceph-mon[31522]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-18T15:12:31.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:31 smithi033 ceph-mon[36648]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-18T15:12:33.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:33 smithi104 ceph-mon[36070]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-18T15:12:33.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:33 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-18T15:12:33.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:33 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:12:33.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:33 smithi019 ceph-mon[31522]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-18T15:12:33.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:33 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-18T15:12:33.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:33 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:12:33.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:33 smithi033 ceph-mon[36648]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-18T15:12:33.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:33 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-18T15:12:33.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:33 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:12:34.621 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:34 smithi104 ceph-mon[36070]: Deploying daemon osd.8 on smithi104 2024-03-18T15:12:34.658 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:34 smithi033 ceph-mon[36648]: Deploying daemon osd.8 on smithi104 2024-03-18T15:12:34.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:34 smithi019 ceph-mon[31522]: Deploying daemon osd.8 on smithi104 2024-03-18T15:12:35.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:35 smithi104 ceph-mon[36070]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-18T15:12:35.671 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:35 smithi033 ceph-mon[36648]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-18T15:12:35.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:35 smithi019 ceph-mon[31522]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-18T15:12:37.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:37 smithi104 ceph-mon[36070]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-18T15:12:37.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:37 smithi019 ceph-mon[31522]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-18T15:12:37.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:37 smithi033 ceph-mon[36648]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-18T15:12:39.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:39 smithi104 ceph-mon[36070]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-18T15:12:39.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:39 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:12:39.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:39 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:12:39.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:39 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:12:39.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:39 smithi019 ceph-mon[31522]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-18T15:12:39.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:39 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:12:39.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:39 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:12:39.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:39 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:12:39.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:39 smithi033 ceph-mon[36648]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-18T15:12:39.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:39 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:12:39.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:39 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:12:39.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:39 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:12:41.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:41 smithi104 ceph-mon[36070]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-18T15:12:41.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:41 smithi019 ceph-mon[31522]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-18T15:12:41.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:41 smithi033 ceph-mon[36648]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-18T15:12:42.788 INFO:teuthology.orchestra.run.smithi104.stdout:Created osd(s) 8 on host 'smithi104' 2024-03-18T15:12:42.788 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:12:42.787+0000 7f44c77fe700 1 -- 172.21.15.104:0/958076577 <== mgr.14152 v2:172.21.15.19:6800/2017714236 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f44d0062710 con 0x7f44bc0a2a30 2024-03-18T15:12:42.790 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:12:42.789+0000 7f44d84b4700 1 -- 172.21.15.104:0/958076577 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f44bc0a2a30 msgr2=0x7f44bc0a4ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:12:42.790 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:12:42.789+0000 7f44d84b4700 1 --2- 172.21.15.104:0/958076577 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f44bc0a2a30 0x7f44bc0a4ef0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f44cc009be0 tx=0x7f44cc008fe0 comp rx=0 tx=0).stop 2024-03-18T15:12:42.790 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:12:42.789+0000 7f44d84b4700 1 -- 172.21.15.104:0/958076577 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f44d01023f0 msgr2=0x7f44d019d230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:12:42.791 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:12:42.789+0000 7f44d84b4700 1 --2- 172.21.15.104:0/958076577 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f44d01023f0 0x7f44d019d230 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f44c000b530 tx=0x7f44c000b8f0 comp rx=0 tx=0).stop 2024-03-18T15:12:42.791 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:12:42.789+0000 7f44d84b4700 1 -- 172.21.15.104:0/958076577 shutdown_connections 2024-03-18T15:12:42.791 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:12:42.789+0000 7f44d84b4700 1 --2- 172.21.15.104:0/958076577 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f44bc0a2a30 0x7f44bc0a4ef0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:12:42.791 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:12:42.789+0000 7f44d84b4700 1 --2- 172.21.15.104:0/958076577 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f44d0102d90 0x7f44d019f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:12:42.791 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:12:42.789+0000 7f44d84b4700 1 --2- 172.21.15.104:0/958076577 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f44d01011f0 0x7f44d006b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:12:42.791 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:12:42.789+0000 7f44d84b4700 1 --2- 172.21.15.104:0/958076577 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f44d01023f0 0x7f44d019d230 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:12:42.791 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:12:42.789+0000 7f44d84b4700 1 -- 172.21.15.104:0/958076577 >> 172.21.15.104:0/958076577 conn(0x7f44d00fc790 msgr2=0x7f44d01056f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:12:42.791 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:12:42.790+0000 7f44d84b4700 1 -- 172.21.15.104:0/958076577 shutdown_connections 2024-03-18T15:12:42.791 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:12:42.790+0000 7f44d84b4700 1 -- 172.21.15.104:0/958076577 wait complete. 2024-03-18T15:12:43.445 DEBUG:teuthology.orchestra.run.smithi104:osd.8> sudo journalctl -f -n 0 -u ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@osd.8.service 2024-03-18T15:12:43.448 INFO:tasks.cephadm:Deploying osd.9 on smithi019 with /dev/vg_nvme/lv_1... 2024-03-18T15:12:43.448 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-18T15:12:43.921 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:12:43.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:43 smithi019 ceph-mon[31522]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-18T15:12:43.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:43 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:12:43.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:43 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:12:43.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:43 smithi019 ceph-mon[31522]: from='osd.8 [v2:172.21.15.104:6816/1014779813,v1:172.21.15.104:6817/1014779813]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-18T15:12:43.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:43 smithi019 ceph-mon[31522]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-18T15:12:43.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:43 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:12:43.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:43 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:12:44.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:43 smithi033 ceph-mon[36648]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-18T15:12:44.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:43 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:12:44.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:43 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:12:44.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:43 smithi033 ceph-mon[36648]: from='osd.8 [v2:172.21.15.104:6816/1014779813,v1:172.21.15.104:6817/1014779813]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-18T15:12:44.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:43 smithi033 ceph-mon[36648]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-18T15:12:44.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:43 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:12:44.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:43 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:12:44.152 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:43 smithi104 ceph-mon[36070]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-18T15:12:44.152 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:43 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:12:44.153 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:43 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:12:44.153 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:43 smithi104 ceph-mon[36070]: from='osd.8 [v2:172.21.15.104:6816/1014779813,v1:172.21.15.104:6817/1014779813]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-18T15:12:44.153 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:43 smithi104 ceph-mon[36070]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-18T15:12:44.153 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:43 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:12:44.153 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:43 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:12:45.051 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:44 smithi019 ceph-mon[31522]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-18T15:12:45.051 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:44 smithi019 ceph-mon[31522]: from='osd.8 [v2:172.21.15.104:6816/1014779813,v1:172.21.15.104:6817/1014779813]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-03-18T15:12:45.052 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:44 smithi019 ceph-mon[31522]: osdmap e57: 9 total, 8 up, 9 in 2024-03-18T15:12:45.052 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:44 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T15:12:45.052 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:44 smithi019 ceph-mon[31522]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-03-18T15:12:45.151 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:44 smithi104 ceph-mon[36070]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-18T15:12:45.152 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:44 smithi104 ceph-mon[36070]: from='osd.8 [v2:172.21.15.104:6816/1014779813,v1:172.21.15.104:6817/1014779813]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-03-18T15:12:45.152 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:44 smithi104 ceph-mon[36070]: osdmap e57: 9 total, 8 up, 9 in 2024-03-18T15:12:45.152 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:44 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T15:12:45.152 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:44 smithi104 ceph-mon[36070]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-03-18T15:12:45.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:44 smithi033 ceph-mon[36648]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-18T15:12:45.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:44 smithi033 ceph-mon[36648]: from='osd.8 [v2:172.21.15.104:6816/1014779813,v1:172.21.15.104:6817/1014779813]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-03-18T15:12:45.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:44 smithi033 ceph-mon[36648]: osdmap e57: 9 total, 8 up, 9 in 2024-03-18T15:12:45.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:44 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T15:12:45.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:44 smithi033 ceph-mon[36648]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-03-18T15:12:45.444 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:12:45 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:12:45.150+0000 7fb007e00700 -1 osd.8 0 waiting for initial osdmap 2024-03-18T15:12:45.445 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:12:45 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:12:45.160+0000 7fafffbeb700 -1 osd.8 58 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T15:12:45.860 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:45 smithi019 ceph-mon[31522]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-18T15:12:45.860 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:45 smithi019 ceph-mon[31522]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2024-03-18T15:12:45.860 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:45 smithi019 ceph-mon[31522]: osdmap e58: 9 total, 8 up, 9 in 2024-03-18T15:12:45.860 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:45 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T15:12:45.861 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:45 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T15:12:45.861 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:45 smithi019 ceph-mon[31522]: from='osd.8 ' entity='osd.8' 2024-03-18T15:12:45.861 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:45 smithi019 ceph-mon[31522]: osd.8 [v2:172.21.15.104:6816/1014779813,v1:172.21.15.104:6817/1014779813] boot 2024-03-18T15:12:45.861 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:45 smithi019 ceph-mon[31522]: osdmap e59: 9 total, 9 up, 9 in 2024-03-18T15:12:45.861 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:45 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T15:12:46.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:45 smithi104 ceph-mon[36070]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-18T15:12:46.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:45 smithi104 ceph-mon[36070]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2024-03-18T15:12:46.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:45 smithi104 ceph-mon[36070]: osdmap e58: 9 total, 8 up, 9 in 2024-03-18T15:12:46.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:45 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T15:12:46.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:45 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T15:12:46.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:45 smithi104 ceph-mon[36070]: from='osd.8 ' entity='osd.8' 2024-03-18T15:12:46.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:45 smithi104 ceph-mon[36070]: osd.8 [v2:172.21.15.104:6816/1014779813,v1:172.21.15.104:6817/1014779813] boot 2024-03-18T15:12:46.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:45 smithi104 ceph-mon[36070]: osdmap e59: 9 total, 9 up, 9 in 2024-03-18T15:12:46.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:45 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T15:12:46.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:45 smithi033 ceph-mon[36648]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-18T15:12:46.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:45 smithi033 ceph-mon[36648]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2024-03-18T15:12:46.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:45 smithi033 ceph-mon[36648]: osdmap e58: 9 total, 8 up, 9 in 2024-03-18T15:12:46.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:45 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T15:12:46.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:45 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T15:12:46.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:45 smithi033 ceph-mon[36648]: from='osd.8 ' entity='osd.8' 2024-03-18T15:12:46.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:45 smithi033 ceph-mon[36648]: osd.8 [v2:172.21.15.104:6816/1014779813,v1:172.21.15.104:6817/1014779813] boot 2024-03-18T15:12:46.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:45 smithi033 ceph-mon[36648]: osdmap e59: 9 total, 9 up, 9 in 2024-03-18T15:12:46.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:45 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T15:12:46.899 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:46 smithi104 ceph-mon[36070]: purged_snaps scrub starts 2024-03-18T15:12:46.900 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:46 smithi104 ceph-mon[36070]: purged_snaps scrub ok 2024-03-18T15:12:46.900 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:46 smithi104 ceph-mon[36070]: osdmap e60: 9 total, 9 up, 9 in 2024-03-18T15:12:47.104 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:46 smithi019 ceph-mon[31522]: purged_snaps scrub starts 2024-03-18T15:12:47.105 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:46 smithi019 ceph-mon[31522]: purged_snaps scrub ok 2024-03-18T15:12:47.105 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:46 smithi019 ceph-mon[31522]: osdmap e60: 9 total, 9 up, 9 in 2024-03-18T15:12:47.105 INFO:teuthology.orchestra.run.smithi019.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-18T15:12:47.105 INFO:teuthology.orchestra.run.smithi019.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-18T15:12:47.105 INFO:teuthology.orchestra.run.smithi019.stderr: stderr: 10+0 records in 2024-03-18T15:12:47.105 INFO:teuthology.orchestra.run.smithi019.stderr:10+0 records out 2024-03-18T15:12:47.105 INFO:teuthology.orchestra.run.smithi019.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0359485 s, 292 MB/s 2024-03-18T15:12:47.105 INFO:teuthology.orchestra.run.smithi019.stderr:--> Zapping successful for: 2024-03-18T15:12:47.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:46 smithi033 ceph-mon[36648]: purged_snaps scrub starts 2024-03-18T15:12:47.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:46 smithi033 ceph-mon[36648]: purged_snaps scrub ok 2024-03-18T15:12:47.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:46 smithi033 ceph-mon[36648]: osdmap e60: 9 total, 9 up, 9 in 2024-03-18T15:12:47.996 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi019:vg_nvme/lv_1 2024-03-18T15:12:48.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:47 smithi019 ceph-mon[31522]: pgmap v215: 1 pgs: 1 peering; 577 KiB data, 291 MiB used, 804 GiB / 805 GiB avail 2024-03-18T15:12:48.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:47 smithi019 ceph-mon[31522]: Detected new or changed devices on smithi104 2024-03-18T15:12:48.217 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:47 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:12:48.217 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:47 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:12:48.217 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:47 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:12:48.217 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:47 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:12:48.217 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:47 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:12:48.217 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:47 smithi019 ceph-mon[31522]: Adjusting osd_memory_target on smithi104 to 5693M 2024-03-18T15:12:48.217 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:47 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:12:48.217 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:47 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:12:48.217 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:47 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:12:48.217 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:47 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:12:48.217 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:47 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:12:48.217 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:47 smithi019 ceph-mon[31522]: osdmap e61: 9 total, 9 up, 9 in 2024-03-18T15:12:48.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:47 smithi033 ceph-mon[36648]: pgmap v215: 1 pgs: 1 peering; 577 KiB data, 291 MiB used, 804 GiB / 805 GiB avail 2024-03-18T15:12:48.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:47 smithi033 ceph-mon[36648]: Detected new or changed devices on smithi104 2024-03-18T15:12:48.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:47 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:12:48.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:47 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:12:48.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:47 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:12:48.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:47 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:12:48.258 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:47 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:12:48.258 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:47 smithi033 ceph-mon[36648]: Adjusting osd_memory_target on smithi104 to 5693M 2024-03-18T15:12:48.258 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:47 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:12:48.258 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:47 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:12:48.258 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:47 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:12:48.258 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:47 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:12:48.258 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:47 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:12:48.258 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:47 smithi033 ceph-mon[36648]: osdmap e61: 9 total, 9 up, 9 in 2024-03-18T15:12:48.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:47 smithi104 ceph-mon[36070]: pgmap v215: 1 pgs: 1 peering; 577 KiB data, 291 MiB used, 804 GiB / 805 GiB avail 2024-03-18T15:12:48.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:47 smithi104 ceph-mon[36070]: Detected new or changed devices on smithi104 2024-03-18T15:12:48.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:47 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:12:48.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:47 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:12:48.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:47 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:12:48.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:47 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:12:48.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:47 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:12:48.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:47 smithi104 ceph-mon[36070]: Adjusting osd_memory_target on smithi104 to 5693M 2024-03-18T15:12:48.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:47 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:12:48.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:47 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:12:48.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:47 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:12:48.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:47 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:12:48.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:47 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:12:48.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:47 smithi104 ceph-mon[36070]: osdmap e61: 9 total, 9 up, 9 in 2024-03-18T15:12:48.454 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:12:49.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:48 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:12:49.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:48 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:12:49.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:48 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:12:49.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:48 smithi019 ceph-mon[31522]: osdmap e62: 9 total, 9 up, 9 in 2024-03-18T15:12:49.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:48 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:12:49.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:48 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:12:49.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:48 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:12:49.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:48 smithi033 ceph-mon[36648]: osdmap e62: 9 total, 9 up, 9 in 2024-03-18T15:12:49.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:48 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:12:49.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:48 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:12:49.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:48 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:12:49.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:48 smithi104 ceph-mon[36070]: osdmap e62: 9 total, 9 up, 9 in 2024-03-18T15:12:50.203 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:49 smithi019 ceph-mon[31522]: pgmap v218: 1 pgs: 1 peering; 577 KiB data, 291 MiB used, 804 GiB / 805 GiB avail 2024-03-18T15:12:50.203 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:49 smithi019 ceph-mon[31522]: osdmap e63: 9 total, 9 up, 9 in 2024-03-18T15:12:50.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:49 smithi033 ceph-mon[36648]: pgmap v218: 1 pgs: 1 peering; 577 KiB data, 291 MiB used, 804 GiB / 805 GiB avail 2024-03-18T15:12:50.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:49 smithi033 ceph-mon[36648]: osdmap e63: 9 total, 9 up, 9 in 2024-03-18T15:12:50.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:49 smithi104 ceph-mon[36070]: pgmap v218: 1 pgs: 1 peering; 577 KiB data, 291 MiB used, 804 GiB / 805 GiB avail 2024-03-18T15:12:50.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:49 smithi104 ceph-mon[36070]: osdmap e63: 9 total, 9 up, 9 in 2024-03-18T15:12:50.834 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:12:50.833+0000 7f0460000700 1 -- 172.21.15.19:0/3237293212 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0458105750 msgr2=0x7f0458105bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:12:50.834 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:12:50.833+0000 7f0460000700 1 --2- 172.21.15.19:0/3237293212 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0458105750 0x7f0458105bb0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f04440098f0 tx=0x7f0444009c00 comp rx=0 tx=0).stop 2024-03-18T15:12:50.834 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:12:50.834+0000 7f0460000700 1 -- 172.21.15.19:0/3237293212 shutdown_connections 2024-03-18T15:12:50.834 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:12:50.834+0000 7f0460000700 1 --2- 172.21.15.19:0/3237293212 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f04581060f0 0x7f045810c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:12:50.835 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:12:50.834+0000 7f0460000700 1 --2- 172.21.15.19:0/3237293212 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0458104550 0x7f0458104970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:12:50.835 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:12:50.834+0000 7f0460000700 1 --2- 172.21.15.19:0/3237293212 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0458105750 0x7f0458105bb0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:12:50.835 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:12:50.834+0000 7f0460000700 1 -- 172.21.15.19:0/3237293212 >> 172.21.15.19:0/3237293212 conn(0x7f04580ffaf0 msgr2=0x7f0458101f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:12:50.835 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:12:50.834+0000 7f0460000700 1 -- 172.21.15.19:0/3237293212 shutdown_connections 2024-03-18T15:12:50.835 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:12:50.834+0000 7f0460000700 1 -- 172.21.15.19:0/3237293212 wait complete. 2024-03-18T15:12:50.836 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:12:50.835+0000 7f0460000700 1 Processor -- start 2024-03-18T15:12:50.836 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:12:50.835+0000 7f0460000700 1 -- start start 2024-03-18T15:12:50.836 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:12:50.835+0000 7f0460000700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0458104550 0x7f045819a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:12:50.836 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:12:50.835+0000 7f0460000700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0458105750 0x7f045819acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:12:50.836 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:12:50.835+0000 7f0460000700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f04581060f0 0x7f04581a1d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:12:50.836 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:12:50.836+0000 7f0460000700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f0458110160 con 0x7f04581060f0 2024-03-18T15:12:50.836 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:12:50.836+0000 7f0460000700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f045810ffe0 con 0x7f0458104550 2024-03-18T15:12:50.836 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:12:50.836+0000 7f0460000700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f04581102e0 con 0x7f0458105750 2024-03-18T15:12:50.836 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:12:50.836+0000 7f045e000700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f04581060f0 0x7f04581a1d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:12:50.836 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:12:50.836+0000 7f045e000700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f04581060f0 0x7f04581a1d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:37778/0 (socket says 172.21.15.19:37778) 2024-03-18T15:12:50.836 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:12:50.836+0000 7f045d600700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0458104550 0x7f045819a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:12:50.837 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:12:50.836+0000 7f045e000700 1 -- 172.21.15.19:0/2537027451 learned_addr learned my addr 172.21.15.19:0/2537027451 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:12:50.837 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:12:50.836+0000 7f045cc00700 1 --2- 172.21.15.19:0/2537027451 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0458105750 0x7f045819acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:12:50.838 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:12:50.836+0000 7f045d600700 1 -- 172.21.15.19:0/2537027451 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0458105750 msgr2=0x7f045819acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:12:50.838 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:12:50.836+0000 7f045d600700 1 --2- 172.21.15.19:0/2537027451 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0458105750 0x7f045819acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:12:50.838 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:12:50.836+0000 7f045d600700 1 -- 172.21.15.19:0/2537027451 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f04581060f0 msgr2=0x7f04581a1d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:12:50.838 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:12:50.836+0000 7f045d600700 1 --2- 172.21.15.19:0/2537027451 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f04581060f0 0x7f04581a1d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:12:50.838 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:12:50.836+0000 7f045d600700 1 -- 172.21.15.19:0/2537027451 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04581a23d0 con 0x7f0458104550 2024-03-18T15:12:50.838 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:12:50.836+0000 7f045cc00700 1 --2- 172.21.15.19:0/2537027451 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0458105750 0x7f045819acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:12:50.838 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:12:50.836+0000 7f045e000700 1 --2- 172.21.15.19:0/2537027451 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f04581060f0 0x7f04581a1d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:12:50.838 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:12:50.837+0000 7f045d600700 1 --2- 172.21.15.19:0/2537027451 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0458104550 0x7f045819a770 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f044c00e9c0 tx=0x7f044c00ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:12:50.838 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:12:50.837+0000 7f0456000700 1 -- 172.21.15.19:0/2537027451 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f044c004220 con 0x7f0458104550 2024-03-18T15:12:50.838 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:12:50.837+0000 7f0456000700 1 -- 172.21.15.19:0/2537027451 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f044c0043c0 con 0x7f0458104550 2024-03-18T15:12:50.838 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:12:50.837+0000 7f0456000700 1 -- 172.21.15.19:0/2537027451 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f044c003d80 con 0x7f0458104550 2024-03-18T15:12:50.839 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:12:50.837+0000 7f0460000700 1 -- 172.21.15.19:0/2537027451 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04581a26c0 con 0x7f0458104550 2024-03-18T15:12:50.839 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:12:50.837+0000 7f0460000700 1 -- 172.21.15.19:0/2537027451 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04581a2bb0 con 0x7f0458104550 2024-03-18T15:12:50.839 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:12:50.838+0000 7f0460000700 1 -- 172.21.15.19:0/2537027451 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0458068f90 con 0x7f0458104550 2024-03-18T15:12:50.840 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:12:50.840+0000 7f0456000700 1 -- 172.21.15.19:0/2537027451 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f044c00c7c0 con 0x7f0458104550 2024-03-18T15:12:50.842 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:12:50.842+0000 7f0456000700 1 --2- 172.21.15.19:0/2537027451 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f04400a2890 0x7f04400a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:12:50.842 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:12:50.842+0000 7f0456000700 1 -- 172.21.15.19:0/2537027451 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 5829+0+0 (secure 0 0 0) 0x7f044c0ce320 con 0x7f0458104550 2024-03-18T15:12:50.842 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:12:50.842+0000 7f045cc00700 1 --2- 172.21.15.19:0/2537027451 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f04400a2890 0x7f04400a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:12:50.843 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:12:50.843+0000 7f045cc00700 1 --2- 172.21.15.19:0/2537027451 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f04400a2890 0x7f04400a4d50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f04440098c0 tx=0x7f044401a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:12:50.846 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:12:50.846+0000 7f0456000700 1 -- 172.21.15.19:0/2537027451 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f044c09bcd0 con 0x7f0458104550 2024-03-18T15:12:51.084 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:12:51.083+0000 7f0460000700 1 -- 172.21.15.19:0/2537027451 --> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi019:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f0458062710 con 0x7f04400a2890 2024-03-18T15:12:52.241 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:51 smithi019 ceph-mon[31522]: pgmap v221: 1 pgs: 1 peering; 577 KiB data, 291 MiB used, 804 GiB / 805 GiB avail 2024-03-18T15:12:52.242 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:51 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T15:12:52.242 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:51 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T15:12:52.242 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:51 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:12:52.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:51 smithi033 ceph-mon[36648]: pgmap v221: 1 pgs: 1 peering; 577 KiB data, 291 MiB used, 804 GiB / 805 GiB avail 2024-03-18T15:12:52.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:51 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T15:12:52.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:51 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T15:12:52.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:51 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:12:52.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:51 smithi104 ceph-mon[36070]: pgmap v221: 1 pgs: 1 peering; 577 KiB data, 291 MiB used, 804 GiB / 805 GiB avail 2024-03-18T15:12:52.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:51 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T15:12:52.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:51 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T15:12:52.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:51 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:12:53.217 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:52 smithi019 ceph-mon[31522]: from='client.24320 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi019:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:12:53.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:52 smithi033 ceph-mon[36648]: from='client.24320 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi019:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:12:53.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:52 smithi104 ceph-mon[36070]: from='client.24320 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi019:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:12:54.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:54 smithi033 ceph-mon[36648]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-03-18T15:12:54.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:54 smithi104 ceph-mon[36070]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-03-18T15:12:54.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:54 smithi019 ceph-mon[31522]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-03-18T15:12:56.182 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:56 smithi019 ceph-mon[31522]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-03-18T15:12:56.182 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:56 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/1423969070' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eed2e9b2-26d3-4c46-88ec-34b5596fe991"}]: dispatch 2024-03-18T15:12:56.182 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:56 smithi019 ceph-mon[31522]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eed2e9b2-26d3-4c46-88ec-34b5596fe991"}]: dispatch 2024-03-18T15:12:56.182 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:56 smithi019 ceph-mon[31522]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "eed2e9b2-26d3-4c46-88ec-34b5596fe991"}]': finished 2024-03-18T15:12:56.182 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:56 smithi019 ceph-mon[31522]: osdmap e64: 10 total, 9 up, 10 in 2024-03-18T15:12:56.182 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:56 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T15:12:56.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:56 smithi033 ceph-mon[36648]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-03-18T15:12:56.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:56 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/1423969070' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eed2e9b2-26d3-4c46-88ec-34b5596fe991"}]: dispatch 2024-03-18T15:12:56.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:56 smithi033 ceph-mon[36648]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eed2e9b2-26d3-4c46-88ec-34b5596fe991"}]: dispatch 2024-03-18T15:12:56.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:56 smithi033 ceph-mon[36648]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "eed2e9b2-26d3-4c46-88ec-34b5596fe991"}]': finished 2024-03-18T15:12:56.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:56 smithi033 ceph-mon[36648]: osdmap e64: 10 total, 9 up, 10 in 2024-03-18T15:12:56.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:56 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T15:12:56.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:56 smithi104 ceph-mon[36070]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-03-18T15:12:56.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:56 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/1423969070' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eed2e9b2-26d3-4c46-88ec-34b5596fe991"}]: dispatch 2024-03-18T15:12:56.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:56 smithi104 ceph-mon[36070]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eed2e9b2-26d3-4c46-88ec-34b5596fe991"}]: dispatch 2024-03-18T15:12:56.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:56 smithi104 ceph-mon[36070]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "eed2e9b2-26d3-4c46-88ec-34b5596fe991"}]': finished 2024-03-18T15:12:56.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:56 smithi104 ceph-mon[36070]: osdmap e64: 10 total, 9 up, 10 in 2024-03-18T15:12:56.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:56 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T15:12:57.345 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:57 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/1863919738' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T15:12:57.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:57 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/1863919738' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T15:12:57.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:57 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/1863919738' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T15:12:58.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:12:58 smithi104 ceph-mon[36070]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-03-18T15:12:58.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:12:58 smithi019 ceph-mon[31522]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-03-18T15:12:58.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:12:58 smithi033 ceph-mon[36648]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-03-18T15:13:00.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:00 smithi019 ceph-mon[31522]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-03-18T15:13:00.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:00 smithi104 ceph-mon[36070]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-03-18T15:13:00.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:00 smithi033 ceph-mon[36648]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-03-18T15:13:02.389 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:02 smithi019 ceph-mon[31522]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-18T15:13:02.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:02 smithi104 ceph-mon[36070]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-18T15:13:02.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:02 smithi033 ceph-mon[36648]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-18T15:13:04.211 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:04 smithi019 ceph-mon[31522]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-03-18T15:13:04.309 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:04 smithi033 ceph-mon[36648]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-03-18T15:13:04.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:04 smithi104 ceph-mon[36070]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-03-18T15:13:05.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:05 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-18T15:13:05.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:05 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:13:05.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:05 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-18T15:13:05.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:05 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:13:05.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:05 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-18T15:13:05.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:05 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:13:06.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:06 smithi104 ceph-mon[36070]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-03-18T15:13:06.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:06 smithi104 ceph-mon[36070]: Deploying daemon osd.9 on smithi019 2024-03-18T15:13:06.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:06 smithi019 ceph-mon[31522]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-03-18T15:13:06.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:06 smithi019 ceph-mon[31522]: Deploying daemon osd.9 on smithi019 2024-03-18T15:13:06.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:06 smithi033 ceph-mon[36648]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-03-18T15:13:06.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:06 smithi033 ceph-mon[36648]: Deploying daemon osd.9 on smithi019 2024-03-18T15:13:08.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:08 smithi104 ceph-mon[36070]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-03-18T15:13:08.458 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:08 smithi019 ceph-mon[31522]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-03-18T15:13:08.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:08 smithi033 ceph-mon[36648]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-03-18T15:13:10.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:10 smithi019 ceph-mon[31522]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-03-18T15:13:10.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:10 smithi104 ceph-mon[36070]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-03-18T15:13:10.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:10 smithi033 ceph-mon[36648]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-03-18T15:13:12.340 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:12 smithi019 ceph-mon[31522]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-03-18T15:13:12.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:12 smithi104 ceph-mon[36070]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-03-18T15:13:12.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:12 smithi033 ceph-mon[36648]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-03-18T15:13:13.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:13 smithi104 ceph-mon[36070]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-03-18T15:13:13.464 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:13 smithi019 ceph-mon[31522]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-03-18T15:13:13.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:13 smithi033 ceph-mon[36648]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-03-18T15:13:14.530 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:14 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:13:14.530 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:14 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:13:14.530 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:14 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:13:14.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:14 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:13:14.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:14 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:13:14.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:14 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:13:14.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:14 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:13:14.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:14 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:13:14.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:14 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:13:15.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:15 smithi104 ceph-mon[36070]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-03-18T15:13:15.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:15 smithi019 ceph-mon[31522]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-03-18T15:13:15.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:15 smithi033 ceph-mon[36648]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-03-18T15:13:17.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:17 smithi019 ceph-mon[31522]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-03-18T15:13:17.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:17 smithi019 ceph-mon[31522]: from='osd.9 [v2:172.21.15.19:6826/2641151009,v1:172.21.15.19:6827/2641151009]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-18T15:13:17.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:17 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:13:17.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:17 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:13:17.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:17 smithi104 ceph-mon[36070]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-03-18T15:13:17.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:17 smithi104 ceph-mon[36070]: from='osd.9 [v2:172.21.15.19:6826/2641151009,v1:172.21.15.19:6827/2641151009]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-18T15:13:17.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:17 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:13:17.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:17 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:13:17.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:17 smithi033 ceph-mon[36648]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-03-18T15:13:17.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:17 smithi033 ceph-mon[36648]: from='osd.9 [v2:172.21.15.19:6826/2641151009,v1:172.21.15.19:6827/2641151009]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-18T15:13:17.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:17 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:13:17.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:17 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:13:18.301 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:18 smithi019 ceph-mon[31522]: from='osd.9 [v2:172.21.15.19:6826/2641151009,v1:172.21.15.19:6827/2641151009]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-18T15:13:18.524 INFO:teuthology.orchestra.run.smithi019.stdout:Created osd(s) 9 on host 'smithi019' 2024-03-18T15:13:18.525 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:13:18.522+0000 7f0456000700 1 -- 172.21.15.19:0/2537027451 <== mgr.14152 v2:172.21.15.19:6800/2017714236 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f0458062710 con 0x7f04400a2890 2024-03-18T15:13:18.527 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:13:18.526+0000 7f0460000700 1 -- 172.21.15.19:0/2537027451 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f04400a2890 msgr2=0x7f04400a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:13:18.527 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:13:18.526+0000 7f0460000700 1 --2- 172.21.15.19:0/2537027451 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f04400a2890 0x7f04400a4d50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f04440098c0 tx=0x7f044401a040 comp rx=0 tx=0).stop 2024-03-18T15:13:18.527 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:13:18.526+0000 7f0460000700 1 -- 172.21.15.19:0/2537027451 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0458104550 msgr2=0x7f045819a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:13:18.527 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:13:18.526+0000 7f0460000700 1 --2- 172.21.15.19:0/2537027451 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0458104550 0x7f045819a770 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f044c00e9c0 tx=0x7f044c00ecd0 comp rx=0 tx=0).stop 2024-03-18T15:13:18.528 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:13:18.527+0000 7f0460000700 1 -- 172.21.15.19:0/2537027451 shutdown_connections 2024-03-18T15:13:18.528 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:13:18.527+0000 7f0460000700 1 --2- 172.21.15.19:0/2537027451 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f04400a2890 0x7f04400a4d50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:13:18.528 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:13:18.527+0000 7f0460000700 1 --2- 172.21.15.19:0/2537027451 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f04581060f0 0x7f04581a1d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:13:18.528 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:13:18.527+0000 7f0460000700 1 --2- 172.21.15.19:0/2537027451 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0458104550 0x7f045819a770 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:13:18.528 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:13:18.527+0000 7f0460000700 1 --2- 172.21.15.19:0/2537027451 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0458105750 0x7f045819acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:13:18.528 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:13:18.527+0000 7f0460000700 1 -- 172.21.15.19:0/2537027451 >> 172.21.15.19:0/2537027451 conn(0x7f04580ffaf0 msgr2=0x7f0458108a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:13:18.528 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:13:18.527+0000 7f0460000700 1 -- 172.21.15.19:0/2537027451 shutdown_connections 2024-03-18T15:13:18.528 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:13:18.527+0000 7f0460000700 1 -- 172.21.15.19:0/2537027451 wait complete. 2024-03-18T15:13:18.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:18 smithi104 ceph-mon[36070]: from='osd.9 [v2:172.21.15.19:6826/2641151009,v1:172.21.15.19:6827/2641151009]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-18T15:13:18.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:18 smithi104 ceph-mon[36070]: osdmap e65: 10 total, 9 up, 10 in 2024-03-18T15:13:18.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:18 smithi104 ceph-mon[36070]: from='osd.9 [v2:172.21.15.19:6826/2641151009,v1:172.21.15.19:6827/2641151009]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]: dispatch 2024-03-18T15:13:18.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:18 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T15:13:18.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:18 smithi104 ceph-mon[36070]: from='osd.9 [v2:172.21.15.19:6826/2641151009,v1:172.21.15.19:6827/2641151009]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]': finished 2024-03-18T15:13:18.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:18 smithi019 ceph-mon[31522]: osdmap e65: 10 total, 9 up, 10 in 2024-03-18T15:13:18.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:18 smithi019 ceph-mon[31522]: from='osd.9 [v2:172.21.15.19:6826/2641151009,v1:172.21.15.19:6827/2641151009]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]: dispatch 2024-03-18T15:13:18.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:18 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T15:13:18.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:18 smithi019 ceph-mon[31522]: from='osd.9 [v2:172.21.15.19:6826/2641151009,v1:172.21.15.19:6827/2641151009]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]': finished 2024-03-18T15:13:18.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:18 smithi033 ceph-mon[36648]: from='osd.9 [v2:172.21.15.19:6826/2641151009,v1:172.21.15.19:6827/2641151009]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-18T15:13:18.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:18 smithi033 ceph-mon[36648]: osdmap e65: 10 total, 9 up, 10 in 2024-03-18T15:13:18.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:18 smithi033 ceph-mon[36648]: from='osd.9 [v2:172.21.15.19:6826/2641151009,v1:172.21.15.19:6827/2641151009]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]: dispatch 2024-03-18T15:13:18.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:18 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T15:13:18.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:18 smithi033 ceph-mon[36648]: from='osd.9 [v2:172.21.15.19:6826/2641151009,v1:172.21.15.19:6827/2641151009]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]': finished 2024-03-18T15:13:19.355 DEBUG:teuthology.orchestra.run.smithi019:osd.9> sudo journalctl -f -n 0 -u ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@osd.9.service 2024-03-18T15:13:19.359 INFO:tasks.cephadm:Deploying osd.10 on smithi033 with /dev/vg_nvme/lv_1... 2024-03-18T15:13:19.359 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-18T15:13:19.618 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T15:13:19.642 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:19 smithi019 ceph-mon[31522]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-03-18T15:13:19.642 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:19 smithi019 ceph-mon[31522]: osdmap e66: 10 total, 9 up, 10 in 2024-03-18T15:13:19.642 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:19 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T15:13:19.642 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:19 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T15:13:19.642 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:19 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:13:19.642 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:19 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:13:19.642 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:19 smithi019 ceph-mon[31522]: from='osd.9 [v2:172.21.15.19:6826/2641151009,v1:172.21.15.19:6827/2641151009]' entity='osd.9' 2024-03-18T15:13:19.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:19 smithi104 ceph-mon[36070]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-03-18T15:13:19.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:19 smithi104 ceph-mon[36070]: osdmap e66: 10 total, 9 up, 10 in 2024-03-18T15:13:19.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:19 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T15:13:19.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:19 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T15:13:19.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:19 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:13:19.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:19 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:13:19.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:19 smithi104 ceph-mon[36070]: from='osd.9 [v2:172.21.15.19:6826/2641151009,v1:172.21.15.19:6827/2641151009]' entity='osd.9' 2024-03-18T15:13:19.740 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:19 smithi033 ceph-mon[36648]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-03-18T15:13:19.740 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:19 smithi033 ceph-mon[36648]: osdmap e66: 10 total, 9 up, 10 in 2024-03-18T15:13:19.740 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:19 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T15:13:19.741 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:19 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T15:13:19.741 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:19 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:13:19.741 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:19 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:13:19.741 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:19 smithi033 ceph-mon[36648]: from='osd.9 [v2:172.21.15.19:6826/2641151009,v1:172.21.15.19:6827/2641151009]' entity='osd.9' 2024-03-18T15:13:20.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:20 smithi033 ceph-mon[36648]: purged_snaps scrub starts 2024-03-18T15:13:20.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:20 smithi033 ceph-mon[36648]: purged_snaps scrub ok 2024-03-18T15:13:20.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:20 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T15:13:20.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:20 smithi033 ceph-mon[36648]: osd.9 [v2:172.21.15.19:6826/2641151009,v1:172.21.15.19:6827/2641151009] boot 2024-03-18T15:13:20.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:20 smithi033 ceph-mon[36648]: osdmap e67: 10 total, 10 up, 10 in 2024-03-18T15:13:20.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:20 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T15:13:20.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:20 smithi104 ceph-mon[36070]: purged_snaps scrub starts 2024-03-18T15:13:20.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:20 smithi104 ceph-mon[36070]: purged_snaps scrub ok 2024-03-18T15:13:20.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:20 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T15:13:20.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:20 smithi104 ceph-mon[36070]: osd.9 [v2:172.21.15.19:6826/2641151009,v1:172.21.15.19:6827/2641151009] boot 2024-03-18T15:13:20.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:20 smithi104 ceph-mon[36070]: osdmap e67: 10 total, 10 up, 10 in 2024-03-18T15:13:20.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:20 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T15:13:20.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:20 smithi019 ceph-mon[31522]: purged_snaps scrub starts 2024-03-18T15:13:20.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:20 smithi019 ceph-mon[31522]: purged_snaps scrub ok 2024-03-18T15:13:20.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:20 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T15:13:20.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:20 smithi019 ceph-mon[31522]: osd.9 [v2:172.21.15.19:6826/2641151009,v1:172.21.15.19:6827/2641151009] boot 2024-03-18T15:13:20.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:20 smithi019 ceph-mon[31522]: osdmap e67: 10 total, 10 up, 10 in 2024-03-18T15:13:20.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:20 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T15:13:21.810 INFO:teuthology.orchestra.run.smithi033.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-18T15:13:21.810 INFO:teuthology.orchestra.run.smithi033.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-18T15:13:21.810 INFO:teuthology.orchestra.run.smithi033.stderr: stderr: 10+0 records in 2024-03-18T15:13:21.810 INFO:teuthology.orchestra.run.smithi033.stderr:10+0 records out 2024-03-18T15:13:21.810 INFO:teuthology.orchestra.run.smithi033.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0109007 s, 962 MB/s 2024-03-18T15:13:21.811 INFO:teuthology.orchestra.run.smithi033.stderr:--> Zapping successful for: 2024-03-18T15:13:22.198 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:22 smithi033 ceph-mon[36648]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-03-18T15:13:22.198 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:22 smithi033 ceph-mon[36648]: osdmap e68: 10 total, 10 up, 10 in 2024-03-18T15:13:22.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:22 smithi019 ceph-mon[31522]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-03-18T15:13:22.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:22 smithi019 ceph-mon[31522]: osdmap e68: 10 total, 10 up, 10 in 2024-03-18T15:13:22.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:22 smithi104 ceph-mon[36070]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-03-18T15:13:22.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:22 smithi104 ceph-mon[36070]: osdmap e68: 10 total, 10 up, 10 in 2024-03-18T15:13:22.521 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi033:vg_nvme/lv_1 2024-03-18T15:13:22.777 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T15:13:24.104 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:24 smithi033 ceph-mon[36648]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-18T15:13:24.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:13:24.297+0000 7f2227859700 1 -- 172.21.15.33:0/1220836838 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2220103500 msgr2=0x7f2220103960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:13:24.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:13:24.297+0000 7f2227859700 1 --2- 172.21.15.33:0/1220836838 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2220103500 0x7f2220103960 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f22100099c0 tx=0x7f2210009cd0 comp rx=0 tx=0).stop 2024-03-18T15:13:24.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:13:24.297+0000 7f2227859700 1 -- 172.21.15.33:0/1220836838 shutdown_connections 2024-03-18T15:13:24.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:13:24.297+0000 7f2227859700 1 --2- 172.21.15.33:0/1220836838 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2220103ea0 0x7f222010a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:13:24.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:13:24.297+0000 7f2227859700 1 --2- 172.21.15.33:0/1220836838 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2220102300 0x7f2220102720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:13:24.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:13:24.297+0000 7f2227859700 1 --2- 172.21.15.33:0/1220836838 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2220103500 0x7f2220103960 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:13:24.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:13:24.297+0000 7f2227859700 1 -- 172.21.15.33:0/1220836838 >> 172.21.15.33:0/1220836838 conn(0x7f22200fd8e0 msgr2=0x7f22200ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:13:24.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:13:24.297+0000 7f2227859700 1 -- 172.21.15.33:0/1220836838 shutdown_connections 2024-03-18T15:13:24.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:13:24.297+0000 7f2227859700 1 -- 172.21.15.33:0/1220836838 wait complete. 2024-03-18T15:13:24.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:13:24.298+0000 7f2227859700 1 Processor -- start 2024-03-18T15:13:24.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:13:24.298+0000 7f2227859700 1 -- start start 2024-03-18T15:13:24.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:13:24.298+0000 7f2227859700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2220102300 0x7f22201985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:13:24.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:13:24.298+0000 7f2227859700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2220103500 0x7f2220198af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:13:24.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:13:24.298+0000 7f2227859700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2220103ea0 0x7f222019fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:13:24.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:13:24.298+0000 7f2227859700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f222010df60 con 0x7f2220102300 2024-03-18T15:13:24.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:13:24.298+0000 7f2227859700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f222010dde0 con 0x7f2220103500 2024-03-18T15:13:24.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:13:24.298+0000 7f2227859700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f222010e0e0 con 0x7f2220103ea0 2024-03-18T15:13:24.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:13:24.298+0000 7f2224df4700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2220103500 0x7f2220198af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:13:24.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:13:24.298+0000 7f2224df4700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2220103500 0x7f2220198af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:40386/0 (socket says 172.21.15.33:40386) 2024-03-18T15:13:24.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:13:24.298+0000 7f2224df4700 1 -- 172.21.15.33:0/3749680564 learned_addr learned my addr 172.21.15.33:0/3749680564 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-18T15:13:24.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:13:24.299+0000 7f2225df6700 1 --2- 172.21.15.33:0/3749680564 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2220103ea0 0x7f222019fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:13:24.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:13:24.299+0000 7f2224df4700 1 -- 172.21.15.33:0/3749680564 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2220103ea0 msgr2=0x7f222019fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:13:24.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:13:24.299+0000 7f2224df4700 1 --2- 172.21.15.33:0/3749680564 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2220103ea0 0x7f222019fb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:13:24.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:13:24.299+0000 7f2224df4700 1 -- 172.21.15.33:0/3749680564 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2220102300 msgr2=0x7f22201985b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:13:24.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:13:24.299+0000 7f2224df4700 1 --2- 172.21.15.33:0/3749680564 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2220102300 0x7f22201985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:13:24.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:13:24.299+0000 7f2224df4700 1 -- 172.21.15.33:0/3749680564 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22201a0210 con 0x7f2220103500 2024-03-18T15:13:24.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:13:24.299+0000 7f2225df6700 1 --2- 172.21.15.33:0/3749680564 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2220103ea0 0x7f222019fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T15:13:24.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:13:24.299+0000 7f2224df4700 1 --2- 172.21.15.33:0/3749680564 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2220103500 0x7f2220198af0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f2210003890 tx=0x7f2210003730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:13:24.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:13:24.299+0000 7f22167fc700 1 -- 172.21.15.33:0/3749680564 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2210027070 con 0x7f2220103500 2024-03-18T15:13:24.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:13:24.299+0000 7f22167fc700 1 -- 172.21.15.33:0/3749680564 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2210022470 con 0x7f2220103500 2024-03-18T15:13:24.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:13:24.299+0000 7f2227859700 1 -- 172.21.15.33:0/3749680564 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22201a04a0 con 0x7f2220103500 2024-03-18T15:13:24.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:13:24.299+0000 7f22167fc700 1 -- 172.21.15.33:0/3749680564 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f221000f6a0 con 0x7f2220103500 2024-03-18T15:13:24.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:13:24.299+0000 7f2227859700 1 -- 172.21.15.33:0/3749680564 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22201a0970 con 0x7f2220103500 2024-03-18T15:13:24.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:13:24.300+0000 7f2227859700 1 -- 172.21.15.33:0/3749680564 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2220068f90 con 0x7f2220103500 2024-03-18T15:13:24.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:13:24.301+0000 7f22167fc700 1 -- 172.21.15.33:0/3749680564 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f221002a070 con 0x7f2220103500 2024-03-18T15:13:24.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:13:24.302+0000 7f22167fc700 1 --2- 172.21.15.33:0/3749680564 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f220c0a2890 0x7f220c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:13:24.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:13:24.302+0000 7f22167fc700 1 -- 172.21.15.33:0/3749680564 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 6261+0+0 (secure 0 0 0) 0x7f22100cf2b0 con 0x7f2220103500 2024-03-18T15:13:24.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:13:24.302+0000 7f22255f5700 1 --2- 172.21.15.33:0/3749680564 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f220c0a2890 0x7f220c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:13:24.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:13:24.303+0000 7f22255f5700 1 --2- 172.21.15.33:0/3749680564 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f220c0a2890 0x7f220c0a4d50 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f2220103360 tx=0x7f221c006d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:13:24.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:13:24.306+0000 7f22167fc700 1 -- 172.21.15.33:0/3749680564 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2210013070 con 0x7f2220103500 2024-03-18T15:13:24.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:24 smithi104 ceph-mon[36070]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-18T15:13:24.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:24 smithi019 ceph-mon[31522]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-18T15:13:24.521 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:13:24.521+0000 7f2227859700 1 -- 172.21.15.33:0/3749680564 --> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f2220062710 con 0x7f220c0a2890 2024-03-18T15:13:25.412 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:25 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T15:13:25.412 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:25 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T15:13:25.412 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:25 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:13:25.412 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:25 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:13:25.412 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:25 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:13:25.412 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:25 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:13:25.413 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:25 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:13:25.413 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:25 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:13:25.413 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:25 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:13:25.413 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:25 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:13:25.413 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:25 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:13:25.413 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:25 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:13:25.413 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:25 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:13:25.413 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:25 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:13:25.413 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:25 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:13:25.413 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:25 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:13:25.413 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:25 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:13:25.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:25 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T15:13:25.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:25 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T15:13:25.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:25 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:13:25.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:25 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:13:25.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:25 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:13:25.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:25 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:13:25.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:25 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:13:25.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:25 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:13:25.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:25 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:13:25.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:25 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:13:25.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:25 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:13:25.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:25 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:13:25.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:25 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:13:25.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:25 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:13:25.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:25 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:13:25.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:25 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:13:25.446 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:25 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:13:25.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:25 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T15:13:25.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:25 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T15:13:25.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:25 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:13:25.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:25 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:13:25.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:25 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:13:25.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:25 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:13:25.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:25 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:13:25.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:25 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:13:25.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:25 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:13:25.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:25 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:13:25.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:25 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:13:25.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:25 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:13:25.467 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:25 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:13:25.467 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:25 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:13:25.467 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:25 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:13:25.467 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:25 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:13:25.467 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:25 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:13:26.125 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:26 smithi033 ceph-mon[36648]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-18T15:13:26.125 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:26 smithi033 ceph-mon[36648]: from='client.24347 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:13:26.125 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:26 smithi033 ceph-mon[36648]: Detected new or changed devices on smithi019 2024-03-18T15:13:26.125 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:26 smithi033 ceph-mon[36648]: Adjusting osd_memory_target on smithi019 to 4273M 2024-03-18T15:13:26.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:26 smithi104 ceph-mon[36070]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-18T15:13:26.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:26 smithi104 ceph-mon[36070]: from='client.24347 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:13:26.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:26 smithi104 ceph-mon[36070]: Detected new or changed devices on smithi019 2024-03-18T15:13:26.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:26 smithi104 ceph-mon[36070]: Adjusting osd_memory_target on smithi019 to 4273M 2024-03-18T15:13:26.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:26 smithi019 ceph-mon[31522]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-18T15:13:26.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:26 smithi019 ceph-mon[31522]: from='client.24347 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:13:26.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:26 smithi019 ceph-mon[31522]: Detected new or changed devices on smithi019 2024-03-18T15:13:26.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:26 smithi019 ceph-mon[31522]: Adjusting osd_memory_target on smithi019 to 4273M 2024-03-18T15:13:27.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:27 smithi019 ceph-mon[31522]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-18T15:13:27.345 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:27 smithi104 ceph-mon[36070]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-18T15:13:27.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:27 smithi033 ceph-mon[36648]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-18T15:13:28.359 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:28 smithi104 ceph-mon[36070]: from='client.? 172.21.15.33:0/1763999903' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f3c35ea5-fbe7-4dc4-93c5-b92d1a714db3"}]: dispatch 2024-03-18T15:13:28.359 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:28 smithi104 ceph-mon[36070]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f3c35ea5-fbe7-4dc4-93c5-b92d1a714db3"}]: dispatch 2024-03-18T15:13:28.359 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:28 smithi104 ceph-mon[36070]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f3c35ea5-fbe7-4dc4-93c5-b92d1a714db3"}]': finished 2024-03-18T15:13:28.359 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:28 smithi104 ceph-mon[36070]: osdmap e69: 11 total, 10 up, 11 in 2024-03-18T15:13:28.360 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:28 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T15:13:28.360 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:28 smithi104 ceph-mon[36070]: from='client.? 172.21.15.33:0/3902982937' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T15:13:28.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:28 smithi019 ceph-mon[31522]: from='client.? 172.21.15.33:0/1763999903' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f3c35ea5-fbe7-4dc4-93c5-b92d1a714db3"}]: dispatch 2024-03-18T15:13:28.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:28 smithi019 ceph-mon[31522]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f3c35ea5-fbe7-4dc4-93c5-b92d1a714db3"}]: dispatch 2024-03-18T15:13:28.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:28 smithi019 ceph-mon[31522]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f3c35ea5-fbe7-4dc4-93c5-b92d1a714db3"}]': finished 2024-03-18T15:13:28.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:28 smithi019 ceph-mon[31522]: osdmap e69: 11 total, 10 up, 11 in 2024-03-18T15:13:28.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:28 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T15:13:28.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:28 smithi019 ceph-mon[31522]: from='client.? 172.21.15.33:0/3902982937' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T15:13:28.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:28 smithi033 ceph-mon[36648]: from='client.? 172.21.15.33:0/1763999903' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f3c35ea5-fbe7-4dc4-93c5-b92d1a714db3"}]: dispatch 2024-03-18T15:13:28.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:28 smithi033 ceph-mon[36648]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f3c35ea5-fbe7-4dc4-93c5-b92d1a714db3"}]: dispatch 2024-03-18T15:13:28.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:28 smithi033 ceph-mon[36648]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f3c35ea5-fbe7-4dc4-93c5-b92d1a714db3"}]': finished 2024-03-18T15:13:28.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:28 smithi033 ceph-mon[36648]: osdmap e69: 11 total, 10 up, 11 in 2024-03-18T15:13:28.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:28 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T15:13:28.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:28 smithi033 ceph-mon[36648]: from='client.? 172.21.15.33:0/3902982937' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T15:13:29.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:29 smithi104 ceph-mon[36070]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-18T15:13:29.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:29 smithi019 ceph-mon[31522]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-18T15:13:29.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:29 smithi033 ceph-mon[36648]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-18T15:13:31.618 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:31 smithi033 ceph-mon[36648]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-18T15:13:31.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:31 smithi104 ceph-mon[36070]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-18T15:13:31.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:31 smithi019 ceph-mon[31522]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-18T15:13:33.629 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:33 smithi033 ceph-mon[36648]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-18T15:13:33.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:33 smithi104 ceph-mon[36070]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-18T15:13:33.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:33 smithi019 ceph-mon[31522]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-18T15:13:35.529 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:35 smithi033 ceph-mon[36648]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-18T15:13:35.529 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:35 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-18T15:13:35.529 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:35 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:13:35.529 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:35 smithi033 ceph-mon[36648]: Deploying daemon osd.10 on smithi033 2024-03-18T15:13:35.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:35 smithi104 ceph-mon[36070]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-18T15:13:35.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:35 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-18T15:13:35.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:35 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:13:35.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:35 smithi104 ceph-mon[36070]: Deploying daemon osd.10 on smithi033 2024-03-18T15:13:35.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:35 smithi019 ceph-mon[31522]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-18T15:13:35.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:35 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-18T15:13:35.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:35 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:13:35.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:35 smithi019 ceph-mon[31522]: Deploying daemon osd.10 on smithi033 2024-03-18T15:13:37.472 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:37 smithi033 ceph-mon[36648]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-18T15:13:37.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:37 smithi104 ceph-mon[36070]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-18T15:13:37.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:37 smithi019 ceph-mon[31522]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-18T15:13:39.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:39 smithi033 ceph-mon[36648]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-18T15:13:39.604 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:39 smithi104 ceph-mon[36070]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-18T15:13:39.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:39 smithi019 ceph-mon[31522]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-18T15:13:41.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:41 smithi104 ceph-mon[36070]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-18T15:13:41.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:41 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:13:41.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:41 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:13:41.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:41 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:13:41.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:41 smithi019 ceph-mon[31522]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-18T15:13:41.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:41 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:13:41.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:41 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:13:41.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:41 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:13:42.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:41 smithi033 ceph-mon[36648]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-18T15:13:42.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:41 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:13:42.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:41 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:13:42.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:41 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:13:43.664 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:43 smithi033 ceph-mon[36648]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-18T15:13:43.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:43 smithi104 ceph-mon[36070]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-18T15:13:43.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:43 smithi019 ceph-mon[31522]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-18T15:13:44.062 INFO:teuthology.orchestra.run.smithi033.stdout:Created osd(s) 10 on host 'smithi033' 2024-03-18T15:13:44.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:13:44.059+0000 7f22167fc700 1 -- 172.21.15.33:0/3749680564 <== mgr.14152 v2:172.21.15.19:6800/2017714236 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f2220062710 con 0x7f220c0a2890 2024-03-18T15:13:44.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:13:44.061+0000 7f2227859700 1 -- 172.21.15.33:0/3749680564 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f220c0a2890 msgr2=0x7f220c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:13:44.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:13:44.061+0000 7f2227859700 1 --2- 172.21.15.33:0/3749680564 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f220c0a2890 0x7f220c0a4d50 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f2220103360 tx=0x7f221c006d20 comp rx=0 tx=0).stop 2024-03-18T15:13:44.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:13:44.061+0000 7f2227859700 1 -- 172.21.15.33:0/3749680564 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2220103500 msgr2=0x7f2220198af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:13:44.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:13:44.061+0000 7f2227859700 1 --2- 172.21.15.33:0/3749680564 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2220103500 0x7f2220198af0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f2210003890 tx=0x7f2210003730 comp rx=0 tx=0).stop 2024-03-18T15:13:44.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:13:44.061+0000 7f2227859700 1 -- 172.21.15.33:0/3749680564 shutdown_connections 2024-03-18T15:13:44.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:13:44.061+0000 7f2227859700 1 --2- 172.21.15.33:0/3749680564 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f220c0a2890 0x7f220c0a4d50 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:13:44.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:13:44.061+0000 7f2227859700 1 --2- 172.21.15.33:0/3749680564 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2220103ea0 0x7f222019fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:13:44.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:13:44.061+0000 7f2227859700 1 --2- 172.21.15.33:0/3749680564 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2220102300 0x7f22201985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:13:44.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:13:44.061+0000 7f2227859700 1 --2- 172.21.15.33:0/3749680564 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2220103500 0x7f2220198af0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:13:44.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:13:44.061+0000 7f2227859700 1 -- 172.21.15.33:0/3749680564 >> 172.21.15.33:0/3749680564 conn(0x7f22200fd8e0 msgr2=0x7f2220106880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:13:44.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:13:44.061+0000 7f2227859700 1 -- 172.21.15.33:0/3749680564 shutdown_connections 2024-03-18T15:13:44.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-18T15:13:44.061+0000 7f2227859700 1 -- 172.21.15.33:0/3749680564 wait complete. 2024-03-18T15:13:44.744 DEBUG:teuthology.orchestra.run.smithi033:osd.10> sudo journalctl -f -n 0 -u ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@osd.10.service 2024-03-18T15:13:44.746 INFO:tasks.cephadm:Deploying osd.11 on smithi104 with /dev/vg_nvme/lv_1... 2024-03-18T15:13:44.747 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-18T15:13:44.994 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T15:13:45.116 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:44 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:13:45.116 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:44 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:13:45.116 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:44 smithi104 ceph-mon[36070]: from='osd.10 [v2:172.21.15.33:6824/1346058869,v1:172.21.15.33:6825/1346058869]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-18T15:13:45.117 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:44 smithi104 ceph-mon[36070]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-18T15:13:45.117 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:44 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:13:45.117 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:44 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:13:45.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:44 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:13:45.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:44 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:13:45.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:44 smithi033 ceph-mon[36648]: from='osd.10 [v2:172.21.15.33:6824/1346058869,v1:172.21.15.33:6825/1346058869]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-18T15:13:45.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:44 smithi033 ceph-mon[36648]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-18T15:13:45.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:44 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:13:45.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:44 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:13:45.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:44 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:13:45.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:44 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:13:45.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:44 smithi019 ceph-mon[31522]: from='osd.10 [v2:172.21.15.33:6824/1346058869,v1:172.21.15.33:6825/1346058869]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-18T15:13:45.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:44 smithi019 ceph-mon[31522]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-18T15:13:45.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:44 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:13:45.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:44 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:13:46.178 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:46 smithi019 ceph-mon[31522]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-18T15:13:46.179 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:46 smithi019 ceph-mon[31522]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-18T15:13:46.179 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:46 smithi019 ceph-mon[31522]: osdmap e70: 11 total, 10 up, 11 in 2024-03-18T15:13:46.179 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:46 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T15:13:46.179 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:46 smithi019 ceph-mon[31522]: from='osd.10 [v2:172.21.15.33:6824/1346058869,v1:172.21.15.33:6825/1346058869]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-03-18T15:13:46.179 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:46 smithi019 ceph-mon[31522]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-03-18T15:13:46.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:46 smithi033 ceph-mon[36648]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-18T15:13:46.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:46 smithi033 ceph-mon[36648]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-18T15:13:46.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:46 smithi033 ceph-mon[36648]: osdmap e70: 11 total, 10 up, 11 in 2024-03-18T15:13:46.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:46 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T15:13:46.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:46 smithi033 ceph-mon[36648]: from='osd.10 [v2:172.21.15.33:6824/1346058869,v1:172.21.15.33:6825/1346058869]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-03-18T15:13:46.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:46 smithi033 ceph-mon[36648]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-03-18T15:13:46.257 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:13:46 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:13:46.028+0000 7fc4a6bc2700 -1 osd.10 0 waiting for initial osdmap 2024-03-18T15:13:46.257 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:13:46 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:13:46.040+0000 7fc4a29b5700 -1 osd.10 71 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T15:13:46.308 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:46 smithi104 ceph-mon[36070]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-18T15:13:46.308 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:46 smithi104 ceph-mon[36070]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-18T15:13:46.309 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:46 smithi104 ceph-mon[36070]: osdmap e70: 11 total, 10 up, 11 in 2024-03-18T15:13:46.309 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:46 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T15:13:46.309 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:46 smithi104 ceph-mon[36070]: from='osd.10 [v2:172.21.15.33:6824/1346058869,v1:172.21.15.33:6825/1346058869]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-03-18T15:13:46.309 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:46 smithi104 ceph-mon[36070]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-03-18T15:13:46.955 INFO:teuthology.orchestra.run.smithi104.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-18T15:13:46.955 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-18T15:13:46.955 INFO:teuthology.orchestra.run.smithi104.stderr: stderr: 10+0 records in 2024-03-18T15:13:46.955 INFO:teuthology.orchestra.run.smithi104.stderr:10+0 records out 2024-03-18T15:13:46.955 INFO:teuthology.orchestra.run.smithi104.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.01212 s, 865 MB/s 2024-03-18T15:13:46.955 INFO:teuthology.orchestra.run.smithi104.stderr:--> Zapping successful for: 2024-03-18T15:13:47.106 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:47 smithi033 ceph-mon[36648]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]': finished 2024-03-18T15:13:47.106 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:47 smithi033 ceph-mon[36648]: osdmap e71: 11 total, 10 up, 11 in 2024-03-18T15:13:47.106 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:47 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T15:13:47.106 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:47 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T15:13:47.106 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:47 smithi033 ceph-mon[36648]: osd.10 [v2:172.21.15.33:6824/1346058869,v1:172.21.15.33:6825/1346058869] boot 2024-03-18T15:13:47.106 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:47 smithi033 ceph-mon[36648]: osdmap e72: 11 total, 11 up, 11 in 2024-03-18T15:13:47.107 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:47 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T15:13:47.339 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:47 smithi104 ceph-mon[36070]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]': finished 2024-03-18T15:13:47.339 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:47 smithi104 ceph-mon[36070]: osdmap e71: 11 total, 10 up, 11 in 2024-03-18T15:13:47.339 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:47 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T15:13:47.339 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:47 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T15:13:47.339 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:47 smithi104 ceph-mon[36070]: osd.10 [v2:172.21.15.33:6824/1346058869,v1:172.21.15.33:6825/1346058869] boot 2024-03-18T15:13:47.339 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:47 smithi104 ceph-mon[36070]: osdmap e72: 11 total, 11 up, 11 in 2024-03-18T15:13:47.339 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:47 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T15:13:47.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:47 smithi019 ceph-mon[31522]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]': finished 2024-03-18T15:13:47.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:47 smithi019 ceph-mon[31522]: osdmap e71: 11 total, 10 up, 11 in 2024-03-18T15:13:47.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:47 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T15:13:47.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:47 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T15:13:47.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:47 smithi019 ceph-mon[31522]: osd.10 [v2:172.21.15.33:6824/1346058869,v1:172.21.15.33:6825/1346058869] boot 2024-03-18T15:13:47.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:47 smithi019 ceph-mon[31522]: osdmap e72: 11 total, 11 up, 11 in 2024-03-18T15:13:47.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:47 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T15:13:47.611 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi104:vg_nvme/lv_1 2024-03-18T15:13:47.848 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T15:13:48.311 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:48 smithi104 ceph-mon[36070]: purged_snaps scrub starts 2024-03-18T15:13:48.311 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:48 smithi104 ceph-mon[36070]: purged_snaps scrub ok 2024-03-18T15:13:48.311 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:48 smithi104 ceph-mon[36070]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-18T15:13:48.311 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:48 smithi104 ceph-mon[36070]: osdmap e73: 11 total, 11 up, 11 in 2024-03-18T15:13:48.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:48 smithi019 ceph-mon[31522]: purged_snaps scrub starts 2024-03-18T15:13:48.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:48 smithi019 ceph-mon[31522]: purged_snaps scrub ok 2024-03-18T15:13:48.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:48 smithi019 ceph-mon[31522]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-18T15:13:48.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:48 smithi019 ceph-mon[31522]: osdmap e73: 11 total, 11 up, 11 in 2024-03-18T15:13:48.486 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:48 smithi033 ceph-mon[36648]: purged_snaps scrub starts 2024-03-18T15:13:48.486 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:48 smithi033 ceph-mon[36648]: purged_snaps scrub ok 2024-03-18T15:13:48.486 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:48 smithi033 ceph-mon[36648]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-18T15:13:48.486 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:48 smithi033 ceph-mon[36648]: osdmap e73: 11 total, 11 up, 11 in 2024-03-18T15:13:49.555 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:13:49.554+0000 7f791d228700 1 -- 172.21.15.104:0/2590390733 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7918102da0 msgr2=0x7f7918109650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:13:49.555 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:13:49.554+0000 7f791d228700 1 --2- 172.21.15.104:0/2590390733 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7918102da0 0x7f7918109650 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f791000b0d0 tx=0x7f791000b3e0 comp rx=0 tx=0).stop 2024-03-18T15:13:49.555 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:13:49.554+0000 7f791d228700 1 -- 172.21.15.104:0/2590390733 shutdown_connections 2024-03-18T15:13:49.555 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:13:49.554+0000 7f791d228700 1 --2- 172.21.15.104:0/2590390733 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7918102da0 0x7f7918109650 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:13:49.555 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:13:49.554+0000 7f791d228700 1 --2- 172.21.15.104:0/2590390733 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7918101200 0x7f7918101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:13:49.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:13:49.554+0000 7f791d228700 1 --2- 172.21.15.104:0/2590390733 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7918102400 0x7f7918102860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:13:49.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:13:49.554+0000 7f791d228700 1 -- 172.21.15.104:0/2590390733 >> 172.21.15.104:0/2590390733 conn(0x7f79180fc7a0 msgr2=0x7f79180febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:13:49.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:13:49.554+0000 7f791d228700 1 -- 172.21.15.104:0/2590390733 shutdown_connections 2024-03-18T15:13:49.556 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:13:49.554+0000 7f791d228700 1 -- 172.21.15.104:0/2590390733 wait complete. 2024-03-18T15:13:49.558 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:13:49.555+0000 7f791d228700 1 Processor -- start 2024-03-18T15:13:49.558 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:13:49.555+0000 7f791d228700 1 -- start start 2024-03-18T15:13:49.558 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:13:49.555+0000 7f791d228700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7918101200 0x7f79181985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:13:49.558 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:13:49.555+0000 7f791d228700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7918102400 0x7f7918198ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:13:49.558 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:13:49.555+0000 7f791d228700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7918102da0 0x7f791819fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:13:49.558 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:13:49.555+0000 7f791d228700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f791810ce70 con 0x7f7918102da0 2024-03-18T15:13:49.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:13:49.555+0000 7f791d228700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f791810ccf0 con 0x7f7918102400 2024-03-18T15:13:49.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:13:49.555+0000 7f791d228700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f791810cff0 con 0x7f7918101200 2024-03-18T15:13:49.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:13:49.555+0000 7f7916d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7918101200 0x7f79181985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:13:49.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:13:49.555+0000 7f7916d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7918101200 0x7f79181985a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:52010/0 (socket says 172.21.15.104:52010) 2024-03-18T15:13:49.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:13:49.555+0000 7f7916d9d700 1 -- 172.21.15.104:0/2330907656 learned_addr learned my addr 172.21.15.104:0/2330907656 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-18T15:13:49.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:13:49.555+0000 7f791659c700 1 --2- 172.21.15.104:0/2330907656 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7918102400 0x7f7918198ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:13:49.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:13:49.555+0000 7f7916d9d700 1 -- 172.21.15.104:0/2330907656 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7918102400 msgr2=0x7f7918198ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:13:49.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:13:49.555+0000 7f7916d9d700 1 --2- 172.21.15.104:0/2330907656 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7918102400 0x7f7918198ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:13:49.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:13:49.555+0000 7f7916d9d700 1 -- 172.21.15.104:0/2330907656 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7918102da0 msgr2=0x7f791819fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:13:49.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:13:49.555+0000 7f7916d9d700 1 --2- 172.21.15.104:0/2330907656 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7918102da0 0x7f791819fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:13:49.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:13:49.555+0000 7f7916d9d700 1 -- 172.21.15.104:0/2330907656 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79181a0200 con 0x7f7918101200 2024-03-18T15:13:49.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:13:49.556+0000 7f791659c700 1 --2- 172.21.15.104:0/2330907656 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7918102400 0x7f7918198ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T15:13:49.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:13:49.556+0000 7f7916d9d700 1 --2- 172.21.15.104:0/2330907656 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7918101200 0x7f79181985a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f790800b530 tx=0x7f790800b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:13:49.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:13:49.556+0000 7f78fffff700 1 -- 172.21.15.104:0/2330907656 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7908015020 con 0x7f7918101200 2024-03-18T15:13:49.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:13:49.556+0000 7f791d228700 1 -- 172.21.15.104:0/2330907656 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79181a04f0 con 0x7f7918101200 2024-03-18T15:13:49.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:13:49.556+0000 7f78fffff700 1 -- 172.21.15.104:0/2330907656 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7908011d10 con 0x7f7918101200 2024-03-18T15:13:49.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:13:49.556+0000 7f78fffff700 1 -- 172.21.15.104:0/2330907656 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f790800f2e0 con 0x7f7918101200 2024-03-18T15:13:49.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:13:49.556+0000 7f791d228700 1 -- 172.21.15.104:0/2330907656 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79181a0a80 con 0x7f7918101200 2024-03-18T15:13:49.560 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:49 smithi104 ceph-mon[36070]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-03-18T15:13:49.560 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:49 smithi104 ceph-mon[36070]: Detected new or changed devices on smithi033 2024-03-18T15:13:49.561 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:49 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:13:49.561 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:49 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:13:49.561 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:49 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:13:49.561 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:49 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:13:49.561 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:49 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:13:49.561 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:49 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:13:49.561 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:49 smithi104 ceph-mon[36070]: Adjusting osd_memory_target on smithi033 to 4271M 2024-03-18T15:13:49.561 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:49 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:13:49.561 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:49 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:13:49.561 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:49 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:13:49.561 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:49 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:13:49.561 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:49 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:13:49.561 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:49 smithi104 ceph-mon[36070]: osdmap e74: 11 total, 11 up, 11 in 2024-03-18T15:13:49.562 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:13:49.557+0000 7f791d228700 1 -- 172.21.15.104:0/2330907656 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7918068f90 con 0x7f7918101200 2024-03-18T15:13:49.562 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:13:49.558+0000 7f78fffff700 1 -- 172.21.15.104:0/2330907656 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f7908018020 con 0x7f7918101200 2024-03-18T15:13:49.562 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:13:49.559+0000 7f78fffff700 1 --2- 172.21.15.104:0/2330907656 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f79000a2890 0x7f79000a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:13:49.562 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:13:49.559+0000 7f78fffff700 1 -- 172.21.15.104:0/2330907656 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 6723+0+0 (secure 0 0 0) 0x7f79080cdc50 con 0x7f7918101200 2024-03-18T15:13:49.563 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:13:49.559+0000 7f791659c700 1 --2- 172.21.15.104:0/2330907656 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f79000a2890 0x7f79000a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:13:49.563 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:13:49.560+0000 7f791659c700 1 --2- 172.21.15.104:0/2330907656 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f79000a2890 0x7f79000a4d50 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f790c009be0 tx=0x7f790c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:13:49.564 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:13:49.564+0000 7f78fffff700 1 -- 172.21.15.104:0/2330907656 <== mon.1 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f79080d3020 con 0x7f7918101200 2024-03-18T15:13:49.777 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:13:49.777+0000 7f791d228700 1 -- 172.21.15.104:0/2330907656 --> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f791806a690 con 0x7f79000a2890 2024-03-18T15:13:49.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:49 smithi019 ceph-mon[31522]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-03-18T15:13:49.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:49 smithi019 ceph-mon[31522]: Detected new or changed devices on smithi033 2024-03-18T15:13:49.967 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:49 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:13:49.967 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:49 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:13:49.967 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:49 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:13:49.967 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:49 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:13:49.967 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:49 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:13:49.967 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:49 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:13:49.967 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:49 smithi019 ceph-mon[31522]: Adjusting osd_memory_target on smithi033 to 4271M 2024-03-18T15:13:49.967 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:49 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:13:49.967 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:49 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:13:49.967 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:49 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:13:49.967 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:49 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:13:49.968 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:49 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:13:49.968 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:49 smithi019 ceph-mon[31522]: osdmap e74: 11 total, 11 up, 11 in 2024-03-18T15:13:50.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:49 smithi033 ceph-mon[36648]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-03-18T15:13:50.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:49 smithi033 ceph-mon[36648]: Detected new or changed devices on smithi033 2024-03-18T15:13:50.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:49 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:13:50.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:49 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:13:50.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:49 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:13:50.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:49 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:13:50.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:49 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:13:50.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:49 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:13:50.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:49 smithi033 ceph-mon[36648]: Adjusting osd_memory_target on smithi033 to 4271M 2024-03-18T15:13:50.008 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:49 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:13:50.008 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:49 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:13:50.008 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:49 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:13:50.008 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:49 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:13:50.008 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:49 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:13:50.008 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:49 smithi033 ceph-mon[36648]: osdmap e74: 11 total, 11 up, 11 in 2024-03-18T15:13:50.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:50 smithi104 ceph-mon[36070]: from='client.24367 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:13:50.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:50 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T15:13:50.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:50 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:13:50.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:50 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T15:13:50.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:50 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:13:50.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:50 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:13:50.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:50 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:13:50.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:50 smithi104 ceph-mon[36070]: osdmap e75: 11 total, 11 up, 11 in 2024-03-18T15:13:50.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:50 smithi019 ceph-mon[31522]: from='client.24367 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:13:50.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:50 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T15:13:50.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:50 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:13:50.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:50 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T15:13:50.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:50 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:13:50.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:50 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:13:50.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:50 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:13:50.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:50 smithi019 ceph-mon[31522]: osdmap e75: 11 total, 11 up, 11 in 2024-03-18T15:13:51.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:50 smithi033 ceph-mon[36648]: from='client.24367 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:13:51.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:50 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T15:13:51.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:50 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:13:51.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:50 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T15:13:51.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:50 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:13:51.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:50 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:13:51.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:50 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:13:51.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:50 smithi033 ceph-mon[36648]: osdmap e75: 11 total, 11 up, 11 in 2024-03-18T15:13:52.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:51 smithi104 ceph-mon[36070]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-03-18T15:13:52.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:51 smithi104 ceph-mon[36070]: osdmap e76: 11 total, 11 up, 11 in 2024-03-18T15:13:52.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:51 smithi019 ceph-mon[31522]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-03-18T15:13:52.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:51 smithi019 ceph-mon[31522]: osdmap e76: 11 total, 11 up, 11 in 2024-03-18T15:13:52.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:51 smithi033 ceph-mon[36648]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-03-18T15:13:52.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:51 smithi033 ceph-mon[36648]: osdmap e76: 11 total, 11 up, 11 in 2024-03-18T15:13:53.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:52 smithi104 ceph-mon[36070]: from='client.? 172.21.15.104:0/2949381745' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c8b51571-f807-4b62-90b6-7529e09e59da"}]: dispatch 2024-03-18T15:13:53.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:52 smithi104 ceph-mon[36070]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c8b51571-f807-4b62-90b6-7529e09e59da"}]: dispatch 2024-03-18T15:13:53.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:52 smithi104 ceph-mon[36070]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c8b51571-f807-4b62-90b6-7529e09e59da"}]': finished 2024-03-18T15:13:53.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:52 smithi104 ceph-mon[36070]: osdmap e77: 12 total, 11 up, 12 in 2024-03-18T15:13:53.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:52 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T15:13:53.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:52 smithi019 ceph-mon[31522]: from='client.? 172.21.15.104:0/2949381745' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c8b51571-f807-4b62-90b6-7529e09e59da"}]: dispatch 2024-03-18T15:13:53.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:52 smithi019 ceph-mon[31522]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c8b51571-f807-4b62-90b6-7529e09e59da"}]: dispatch 2024-03-18T15:13:53.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:52 smithi019 ceph-mon[31522]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c8b51571-f807-4b62-90b6-7529e09e59da"}]': finished 2024-03-18T15:13:53.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:52 smithi019 ceph-mon[31522]: osdmap e77: 12 total, 11 up, 12 in 2024-03-18T15:13:53.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:52 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T15:13:53.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:52 smithi033 ceph-mon[36648]: from='client.? 172.21.15.104:0/2949381745' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c8b51571-f807-4b62-90b6-7529e09e59da"}]: dispatch 2024-03-18T15:13:53.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:52 smithi033 ceph-mon[36648]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c8b51571-f807-4b62-90b6-7529e09e59da"}]: dispatch 2024-03-18T15:13:53.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:52 smithi033 ceph-mon[36648]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c8b51571-f807-4b62-90b6-7529e09e59da"}]': finished 2024-03-18T15:13:53.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:52 smithi033 ceph-mon[36648]: osdmap e77: 12 total, 11 up, 12 in 2024-03-18T15:13:53.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:52 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T15:13:54.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:53 smithi019 ceph-mon[31522]: pgmap v265: 1 pgs: 1 peering; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 133 KiB/s, 0 objects/s recovering 2024-03-18T15:13:54.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:53 smithi019 ceph-mon[31522]: from='client.? 172.21.15.104:0/734496323' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T15:13:54.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:53 smithi033 ceph-mon[36648]: pgmap v265: 1 pgs: 1 peering; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 133 KiB/s, 0 objects/s recovering 2024-03-18T15:13:54.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:53 smithi033 ceph-mon[36648]: from='client.? 172.21.15.104:0/734496323' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T15:13:54.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:53 smithi104 ceph-mon[36070]: pgmap v265: 1 pgs: 1 peering; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 133 KiB/s, 0 objects/s recovering 2024-03-18T15:13:54.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:53 smithi104 ceph-mon[36070]: from='client.? 172.21.15.104:0/734496323' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T15:13:56.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:55 smithi019 ceph-mon[31522]: pgmap v267: 1 pgs: 1 peering; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 108 KiB/s, 0 objects/s recovering 2024-03-18T15:13:56.225 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:55 smithi104 ceph-mon[36070]: pgmap v267: 1 pgs: 1 peering; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 108 KiB/s, 0 objects/s recovering 2024-03-18T15:13:56.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:55 smithi033 ceph-mon[36648]: pgmap v267: 1 pgs: 1 peering; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 108 KiB/s, 0 objects/s recovering 2024-03-18T15:13:58.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:57 smithi104 ceph-mon[36070]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-03-18T15:13:58.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:57 smithi033 ceph-mon[36648]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-03-18T15:13:58.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:57 smithi019 ceph-mon[31522]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-03-18T15:14:00.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:59 smithi104 ceph-mon[36070]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-18T15:14:00.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:59 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-18T15:14:00.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:13:59 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:14:00.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:59 smithi033 ceph-mon[36648]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-18T15:14:00.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:59 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-18T15:14:00.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:13:59 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:14:00.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:59 smithi019 ceph-mon[31522]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-18T15:14:00.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:59 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-18T15:14:00.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:13:59 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:14:01.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:00 smithi033 ceph-mon[36648]: Deploying daemon osd.11 on smithi104 2024-03-18T15:14:01.323 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:00 smithi104 ceph-mon[36070]: Deploying daemon osd.11 on smithi104 2024-03-18T15:14:01.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:00 smithi019 ceph-mon[31522]: Deploying daemon osd.11 on smithi104 2024-03-18T15:14:02.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:01 smithi033 ceph-mon[36648]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-18T15:14:02.372 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:01 smithi104 ceph-mon[36070]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-18T15:14:02.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:01 smithi019 ceph-mon[31522]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-18T15:14:04.101 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:03 smithi104 ceph-mon[36070]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-18T15:14:04.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:03 smithi033 ceph-mon[36648]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-18T15:14:04.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:03 smithi019 ceph-mon[31522]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-18T15:14:06.255 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:05 smithi104 ceph-mon[36070]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-18T15:14:06.255 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:05 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:06.255 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:05 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:06.255 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:05 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:14:06.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:05 smithi033 ceph-mon[36648]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-18T15:14:06.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:05 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:06.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:05 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:06.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:05 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:14:06.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:05 smithi019 ceph-mon[31522]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-18T15:14:06.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:05 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:06.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:05 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:06.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:05 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:14:08.136 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:07 smithi104 ceph-mon[36070]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-18T15:14:08.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:07 smithi033 ceph-mon[36648]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-18T15:14:08.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:07 smithi019 ceph-mon[31522]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-18T15:14:08.861 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:14:08.860+0000 7f78fffff700 1 -- 172.21.15.104:0/2330907656 <== mgr.14152 v2:172.21.15.19:6800/2017714236 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f791806a690 con 0x7f79000a2890 2024-03-18T15:14:08.862 INFO:teuthology.orchestra.run.smithi104.stdout:Created osd(s) 11 on host 'smithi104' 2024-03-18T15:14:08.864 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:14:08.863+0000 7f791d228700 1 -- 172.21.15.104:0/2330907656 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f79000a2890 msgr2=0x7f79000a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:08.864 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:14:08.863+0000 7f791d228700 1 --2- 172.21.15.104:0/2330907656 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f79000a2890 0x7f79000a4d50 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f790c009be0 tx=0x7f790c009260 comp rx=0 tx=0).stop 2024-03-18T15:14:08.864 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:14:08.863+0000 7f791d228700 1 -- 172.21.15.104:0/2330907656 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7918101200 msgr2=0x7f79181985a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:08.864 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:14:08.863+0000 7f791d228700 1 --2- 172.21.15.104:0/2330907656 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7918101200 0x7f79181985a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f790800b530 tx=0x7f790800b8f0 comp rx=0 tx=0).stop 2024-03-18T15:14:08.864 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:14:08.863+0000 7f791d228700 1 -- 172.21.15.104:0/2330907656 shutdown_connections 2024-03-18T15:14:08.864 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:14:08.863+0000 7f791d228700 1 --2- 172.21.15.104:0/2330907656 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f79000a2890 0x7f79000a4d50 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:08.864 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:14:08.863+0000 7f791d228700 1 --2- 172.21.15.104:0/2330907656 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7918101200 0x7f79181985a0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:08.865 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:14:08.863+0000 7f791d228700 1 --2- 172.21.15.104:0/2330907656 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7918102da0 0x7f791819fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:08.865 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:14:08.863+0000 7f791d228700 1 --2- 172.21.15.104:0/2330907656 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7918102400 0x7f7918198ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:08.865 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:14:08.863+0000 7f791d228700 1 -- 172.21.15.104:0/2330907656 >> 172.21.15.104:0/2330907656 conn(0x7f79180fc7a0 msgr2=0x7f7918069f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:14:08.865 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:14:08.863+0000 7f791d228700 1 -- 172.21.15.104:0/2330907656 shutdown_connections 2024-03-18T15:14:08.865 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:14:08.863+0000 7f791d228700 1 -- 172.21.15.104:0/2330907656 wait complete. 2024-03-18T15:14:09.532 DEBUG:teuthology.orchestra.run.smithi104:osd.11> sudo journalctl -f -n 0 -u ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@osd.11.service 2024-03-18T15:14:09.534 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-03-18T15:14:09.534 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 -- ceph osd stat -f json 2024-03-18T15:14:09.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:09 smithi019 ceph-mon[31522]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-18T15:14:09.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:09 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:09.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:09 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:09.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:09 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:09.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:09 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:09.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:09 smithi019 ceph-mon[31522]: from='osd.11 [v2:172.21.15.104:6824/3759498281,v1:172.21.15.104:6825/3759498281]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-18T15:14:09.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:09 smithi019 ceph-mon[31522]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-18T15:14:10.010 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:14:10.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:09 smithi104 ceph-mon[36070]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-18T15:14:10.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:09 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:10.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:09 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:10.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:09 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:10.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:09 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:10.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:09 smithi104 ceph-mon[36070]: from='osd.11 [v2:172.21.15.104:6824/3759498281,v1:172.21.15.104:6825/3759498281]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-18T15:14:10.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:09 smithi104 ceph-mon[36070]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-18T15:14:10.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:09 smithi033 ceph-mon[36648]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-18T15:14:10.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:09 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:10.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:09 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:10.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:09 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:10.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:09 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:10.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:09 smithi033 ceph-mon[36648]: from='osd.11 [v2:172.21.15.104:6824/3759498281,v1:172.21.15.104:6825/3759498281]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-18T15:14:10.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:09 smithi033 ceph-mon[36648]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-18T15:14:11.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:11 smithi104 ceph-mon[36070]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-18T15:14:11.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:11 smithi104 ceph-mon[36070]: from='osd.11 [v2:172.21.15.104:6824/3759498281,v1:172.21.15.104:6825/3759498281]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-03-18T15:14:11.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:11 smithi104 ceph-mon[36070]: osdmap e78: 12 total, 11 up, 12 in 2024-03-18T15:14:11.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:11 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T15:14:11.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:11 smithi104 ceph-mon[36070]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-03-18T15:14:11.445 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:14:11 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:14:11.009+0000 7f5e696c6700 -1 osd.11 0 waiting for initial osdmap 2024-03-18T15:14:11.445 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:14:11 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:14:11.025+0000 7f5e654b9700 -1 osd.11 79 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T15:14:11.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:11 smithi019 ceph-mon[31522]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-18T15:14:11.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:11 smithi019 ceph-mon[31522]: from='osd.11 [v2:172.21.15.104:6824/3759498281,v1:172.21.15.104:6825/3759498281]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-03-18T15:14:11.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:11 smithi019 ceph-mon[31522]: osdmap e78: 12 total, 11 up, 12 in 2024-03-18T15:14:11.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:11 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T15:14:11.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:11 smithi019 ceph-mon[31522]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-03-18T15:14:11.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:11 smithi033 ceph-mon[36648]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-18T15:14:11.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:11 smithi033 ceph-mon[36648]: from='osd.11 [v2:172.21.15.104:6824/3759498281,v1:172.21.15.104:6825/3759498281]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-03-18T15:14:11.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:11 smithi033 ceph-mon[36648]: osdmap e78: 12 total, 11 up, 12 in 2024-03-18T15:14:11.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:11 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T15:14:11.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:11 smithi033 ceph-mon[36648]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-03-18T15:14:12.143 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:12 smithi019 ceph-mon[31522]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-18T15:14:12.143 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:12 smithi019 ceph-mon[31522]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2024-03-18T15:14:12.143 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:12 smithi019 ceph-mon[31522]: osdmap e79: 12 total, 11 up, 12 in 2024-03-18T15:14:12.143 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:12 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T15:14:12.143 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:12 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T15:14:12.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:12 smithi104 ceph-mon[36070]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-18T15:14:12.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:12 smithi104 ceph-mon[36070]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2024-03-18T15:14:12.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:12 smithi104 ceph-mon[36070]: osdmap e79: 12 total, 11 up, 12 in 2024-03-18T15:14:12.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:12 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T15:14:12.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:12 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T15:14:12.439 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:12.438+0000 7f22d6200700 1 -- 172.21.15.19:0/427609384 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f22d01056c0 msgr2=0x7f22d0107ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:12.439 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:12.438+0000 7f22d6200700 1 --2- 172.21.15.19:0/427609384 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f22d01056c0 0x7f22d0107ab0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f22bc009990 tx=0x7f22bc009ca0 comp rx=0 tx=0).stop 2024-03-18T15:14:12.439 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:12.438+0000 7f22d6200700 1 -- 172.21.15.19:0/427609384 shutdown_connections 2024-03-18T15:14:12.439 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:12.438+0000 7f22d6200700 1 --2- 172.21.15.19:0/427609384 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22d0107ff0 0x7f22d010a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:12.439 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:12.438+0000 7f22d6200700 1 --2- 172.21.15.19:0/427609384 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f22d0102d90 0x7f22d0105180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:12.439 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:12.438+0000 7f22d6200700 1 --2- 172.21.15.19:0/427609384 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f22d01056c0 0x7f22d0107ab0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:12.439 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:12.438+0000 7f22d6200700 1 -- 172.21.15.19:0/427609384 >> 172.21.15.19:0/427609384 conn(0x7f22d00fc7a0 msgr2=0x7f22d00febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:14:12.439 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:12.438+0000 7f22d6200700 1 -- 172.21.15.19:0/427609384 shutdown_connections 2024-03-18T15:14:12.440 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:12.438+0000 7f22d6200700 1 -- 172.21.15.19:0/427609384 wait complete. 2024-03-18T15:14:12.441 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:12.439+0000 7f22d6200700 1 Processor -- start 2024-03-18T15:14:12.441 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:12.440+0000 7f22d6200700 1 -- start start 2024-03-18T15:14:12.441 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:12.440+0000 7f22d6200700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22d0102d90 0x7f22d019a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:14:12.441 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:12.440+0000 7f22d6200700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f22d01056c0 0x7f22d019ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:14:12.441 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:12.440+0000 7f22d6200700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f22d0107ff0 0x7f22d01a1d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:14:12.441 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:12.440+0000 7f22d6200700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f22d010df60 con 0x7f22d0107ff0 2024-03-18T15:14:12.441 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:12.440+0000 7f22d6200700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f22d010dde0 con 0x7f22d01056c0 2024-03-18T15:14:12.441 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:12.440+0000 7f22d6200700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f22d010e0e0 con 0x7f22d0102d90 2024-03-18T15:14:12.441 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:12.440+0000 7f22cea00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f22d01056c0 0x7f22d019ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:14:12.441 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:12.440+0000 7f22cea00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f22d01056c0 0x7f22d019ace0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:55100/0 (socket says 172.21.15.19:55100) 2024-03-18T15:14:12.442 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:12.441+0000 7f22cea00700 1 -- 172.21.15.19:0/3993737716 learned_addr learned my addr 172.21.15.19:0/3993737716 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:14:12.442 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:12.441+0000 7f22cfe00700 1 --2- 172.21.15.19:0/3993737716 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f22d0107ff0 0x7f22d01a1d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:14:12.442 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:12.441+0000 7f22cea00700 1 -- 172.21.15.19:0/3993737716 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22d0102d90 msgr2=0x7f22d019a7a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-18T15:14:12.443 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:12.441+0000 7f22cea00700 1 --2- 172.21.15.19:0/3993737716 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22d0102d90 0x7f22d019a7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:12.443 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:12.441+0000 7f22cea00700 1 -- 172.21.15.19:0/3993737716 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f22d0107ff0 msgr2=0x7f22d01a1d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:12.443 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:12.441+0000 7f22cea00700 1 --2- 172.21.15.19:0/3993737716 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f22d0107ff0 0x7f22d01a1d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:12.443 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:12.441+0000 7f22cea00700 1 -- 172.21.15.19:0/3993737716 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22d01a2400 con 0x7f22d01056c0 2024-03-18T15:14:12.443 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:12.441+0000 7f22cea00700 1 --2- 172.21.15.19:0/3993737716 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f22d01056c0 0x7f22d019ace0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f22bc00b2e0 tx=0x7f22bc00bbb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:14:12.443 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:12.442+0000 7f22c7e00700 1 -- 172.21.15.19:0/3993737716 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22bc026070 con 0x7f22d01056c0 2024-03-18T15:14:12.443 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:12.442+0000 7f22c7e00700 1 -- 172.21.15.19:0/3993737716 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f22bc00f460 con 0x7f22d01056c0 2024-03-18T15:14:12.443 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:12.442+0000 7f22d6200700 1 -- 172.21.15.19:0/3993737716 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22d01a2690 con 0x7f22d01056c0 2024-03-18T15:14:12.443 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:12.442+0000 7f22c7e00700 1 -- 172.21.15.19:0/3993737716 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22bc021670 con 0x7f22d01056c0 2024-03-18T15:14:12.443 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:12.442+0000 7f22d6200700 1 -- 172.21.15.19:0/3993737716 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22d01a2b40 con 0x7f22d01056c0 2024-03-18T15:14:12.443 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:12.443+0000 7f22d6200700 1 -- 172.21.15.19:0/3993737716 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22d0068f90 con 0x7f22d01056c0 2024-03-18T15:14:12.445 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:12.445+0000 7f22c7e00700 1 -- 172.21.15.19:0/3993737716 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f22bc029070 con 0x7f22d01056c0 2024-03-18T15:14:12.447 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:12.447+0000 7f22c7e00700 1 --2- 172.21.15.19:0/3993737716 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f22b40a2910 0x7f22b40a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:14:12.447 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:12.447+0000 7f22c7e00700 1 -- 172.21.15.19:0/3993737716 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 7127+0+0 (secure 0 0 0) 0x7f22bc0cf3b0 con 0x7f22d01056c0 2024-03-18T15:14:12.447 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:12.447+0000 7f22cf400700 1 --2- 172.21.15.19:0/3993737716 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f22b40a2910 0x7f22b40a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:14:12.447 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:12.448+0000 7f22cf400700 1 --2- 172.21.15.19:0/3993737716 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f22b40a2910 0x7f22b40a4dd0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f22c0009b20 tx=0x7f22c0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:14:12.451 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:12.451+0000 7f22c7e00700 1 -- 172.21.15.19:0/3993737716 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f22bc05d740 con 0x7f22d01056c0 2024-03-18T15:14:12.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:12 smithi033 ceph-mon[36648]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-18T15:14:12.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:12 smithi033 ceph-mon[36648]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2024-03-18T15:14:12.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:12 smithi033 ceph-mon[36648]: osdmap e79: 12 total, 11 up, 12 in 2024-03-18T15:14:12.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:12 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T15:14:12.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:12 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T15:14:12.694 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:12.693+0000 7f22d6200700 1 -- 172.21.15.19:0/3993737716 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f22d0062ee0 con 0x7f22d01056c0 2024-03-18T15:14:12.694 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:12.694+0000 7f22c7e00700 1 -- 172.21.15.19:0/3993737716 <== mon.2 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v80) ==== 74+0+133 (secure 0 0 0) 0x7f22bc013070 con 0x7f22d01056c0 2024-03-18T15:14:12.694 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-18T15:14:12.697 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:12.697+0000 7f22d6200700 1 -- 172.21.15.19:0/3993737716 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f22b40a2910 msgr2=0x7f22b40a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:12.698 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:12.697+0000 7f22d6200700 1 --2- 172.21.15.19:0/3993737716 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f22b40a2910 0x7f22b40a4dd0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f22c0009b20 tx=0x7f22c0009260 comp rx=0 tx=0).stop 2024-03-18T15:14:12.698 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:12.697+0000 7f22d6200700 1 -- 172.21.15.19:0/3993737716 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f22d01056c0 msgr2=0x7f22d019ace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:12.698 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:12.697+0000 7f22d6200700 1 --2- 172.21.15.19:0/3993737716 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f22d01056c0 0x7f22d019ace0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f22bc00b2e0 tx=0x7f22bc00bbb0 comp rx=0 tx=0).stop 2024-03-18T15:14:12.698 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:12.697+0000 7f22d6200700 1 -- 172.21.15.19:0/3993737716 shutdown_connections 2024-03-18T15:14:12.698 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:12.697+0000 7f22d6200700 1 --2- 172.21.15.19:0/3993737716 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f22b40a2910 0x7f22b40a4dd0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:12.698 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:12.697+0000 7f22d6200700 1 --2- 172.21.15.19:0/3993737716 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22d0102d90 0x7f22d019a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:12.698 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:12.697+0000 7f22d6200700 1 --2- 172.21.15.19:0/3993737716 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f22d0107ff0 0x7f22d01a1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:12.698 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:12.697+0000 7f22d6200700 1 --2- 172.21.15.19:0/3993737716 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f22d01056c0 0x7f22d019ace0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:12.698 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:12.697+0000 7f22d6200700 1 -- 172.21.15.19:0/3993737716 >> 172.21.15.19:0/3993737716 conn(0x7f22d00fc7a0 msgr2=0x7f22d0102510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:14:12.698 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:12.698+0000 7f22d6200700 1 -- 172.21.15.19:0/3993737716 shutdown_connections 2024-03-18T15:14:12.698 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:12.698+0000 7f22d6200700 1 -- 172.21.15.19:0/3993737716 wait complete. 2024-03-18T15:14:13.300 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:13 smithi019 ceph-mon[31522]: purged_snaps scrub starts 2024-03-18T15:14:13.300 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:13 smithi019 ceph-mon[31522]: purged_snaps scrub ok 2024-03-18T15:14:13.300 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:13 smithi019 ceph-mon[31522]: osd.11 [v2:172.21.15.104:6824/3759498281,v1:172.21.15.104:6825/3759498281] boot 2024-03-18T15:14:13.300 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:13 smithi019 ceph-mon[31522]: osdmap e80: 12 total, 12 up, 12 in 2024-03-18T15:14:13.300 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:13 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T15:14:13.300 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:13 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:13.301 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:13 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:13.301 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:13 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:14:13.301 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:13 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:14:13.301 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:13 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:14:13.301 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:13 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:14:13.301 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:13 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:13.301 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:13 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:14:13.301 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:13 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:14:13.301 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:13 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:13.301 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:13 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/3993737716' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-18T15:14:13.301 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:13 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:14:13.301 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:13 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:14:13.302 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:13 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:14:13.302 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:13 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:13.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:13 smithi104 ceph-mon[36070]: purged_snaps scrub starts 2024-03-18T15:14:13.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:13 smithi104 ceph-mon[36070]: purged_snaps scrub ok 2024-03-18T15:14:13.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:13 smithi104 ceph-mon[36070]: osd.11 [v2:172.21.15.104:6824/3759498281,v1:172.21.15.104:6825/3759498281] boot 2024-03-18T15:14:13.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:13 smithi104 ceph-mon[36070]: osdmap e80: 12 total, 12 up, 12 in 2024-03-18T15:14:13.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:13 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T15:14:13.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:13 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:13.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:13 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:13.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:13 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:14:13.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:13 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:14:13.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:13 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:14:13.446 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:13 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:14:13.446 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:13 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:13.446 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:13 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:14:13.446 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:13 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:14:13.446 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:13 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:13.446 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:13 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/3993737716' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-18T15:14:13.446 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:13 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:14:13.446 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:13 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:14:13.446 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:13 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:14:13.446 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:13 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:13.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:13 smithi033 ceph-mon[36648]: purged_snaps scrub starts 2024-03-18T15:14:13.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:13 smithi033 ceph-mon[36648]: purged_snaps scrub ok 2024-03-18T15:14:13.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:13 smithi033 ceph-mon[36648]: osd.11 [v2:172.21.15.104:6824/3759498281,v1:172.21.15.104:6825/3759498281] boot 2024-03-18T15:14:13.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:13 smithi033 ceph-mon[36648]: osdmap e80: 12 total, 12 up, 12 in 2024-03-18T15:14:13.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:13 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T15:14:13.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:13 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:13.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:13 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:13.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:13 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:14:13.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:13 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:14:13.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:13 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:14:13.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:13 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:14:13.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:13 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:13.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:13 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:14:13.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:13 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:14:13.508 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:13 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:13.508 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:13 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/3993737716' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-18T15:14:13.508 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:13 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:14:13.508 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:13 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:14:13.508 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:13 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:14:13.508 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:13 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:13.709 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":80,"num_osds":12,"num_up_osds":12,"osd_up_since":1710774852,"num_in_osds":12,"osd_in_since":1710774832,"num_remapped_pgs":0} 2024-03-18T15:14:13.710 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 -- ceph osd dump --format=json 2024-03-18T15:14:14.158 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:14:14.305 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:14 smithi019 ceph-mon[31522]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-18T15:14:14.305 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:14 smithi019 ceph-mon[31522]: Detected new or changed devices on smithi104 2024-03-18T15:14:14.305 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:14 smithi019 ceph-mon[31522]: Adjusting osd_memory_target on smithi104 to 4270M 2024-03-18T15:14:14.305 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:14 smithi019 ceph-mon[31522]: osdmap e81: 12 total, 12 up, 12 in 2024-03-18T15:14:14.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:14 smithi104 ceph-mon[36070]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-18T15:14:14.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:14 smithi104 ceph-mon[36070]: Detected new or changed devices on smithi104 2024-03-18T15:14:14.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:14 smithi104 ceph-mon[36070]: Adjusting osd_memory_target on smithi104 to 4270M 2024-03-18T15:14:14.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:14 smithi104 ceph-mon[36070]: osdmap e81: 12 total, 12 up, 12 in 2024-03-18T15:14:14.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:14 smithi033 ceph-mon[36648]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-18T15:14:14.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:14 smithi033 ceph-mon[36648]: Detected new or changed devices on smithi104 2024-03-18T15:14:14.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:14 smithi033 ceph-mon[36648]: Adjusting osd_memory_target on smithi104 to 4270M 2024-03-18T15:14:14.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:14 smithi033 ceph-mon[36648]: osdmap e81: 12 total, 12 up, 12 in 2024-03-18T15:14:16.336 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:16 smithi019 ceph-mon[31522]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:14:16.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:16 smithi104 ceph-mon[36070]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:14:16.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:16 smithi033 ceph-mon[36648]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:14:16.639 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:16.638+0000 7efe92e00700 1 -- 172.21.15.19:0/4194962247 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efe8c1011d0 msgr2=0x7efe8c101630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:16.639 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:16.638+0000 7efe92e00700 1 --2- 172.21.15.19:0/4194962247 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efe8c1011d0 0x7efe8c101630 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7efe780098f0 tx=0x7efe78009c00 comp rx=0 tx=0).stop 2024-03-18T15:14:16.639 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:16.639+0000 7efe92e00700 1 -- 172.21.15.19:0/4194962247 shutdown_connections 2024-03-18T15:14:16.639 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:16.639+0000 7efe92e00700 1 --2- 172.21.15.19:0/4194962247 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efe8c101b70 0x7efe8c10e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:16.639 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:16.639+0000 7efe92e00700 1 --2- 172.21.15.19:0/4194962247 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efe8c107210 0x7efe8c1075f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:16.639 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:16.639+0000 7efe92e00700 1 --2- 172.21.15.19:0/4194962247 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efe8c1011d0 0x7efe8c101630 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:16.639 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:16.639+0000 7efe92e00700 1 -- 172.21.15.19:0/4194962247 >> 172.21.15.19:0/4194962247 conn(0x7efe8c078100 msgr2=0x7efe8c078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:14:16.640 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:16.639+0000 7efe92e00700 1 -- 172.21.15.19:0/4194962247 shutdown_connections 2024-03-18T15:14:16.640 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:16.639+0000 7efe92e00700 1 -- 172.21.15.19:0/4194962247 wait complete. 2024-03-18T15:14:16.640 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:16.640+0000 7efe92e00700 1 Processor -- start 2024-03-18T15:14:16.640 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:16.640+0000 7efe92e00700 1 -- start start 2024-03-18T15:14:16.641 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:16.640+0000 7efe92e00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efe8c1011d0 0x7efe8c1080f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:14:16.641 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:16.641+0000 7efe92e00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efe8c101b70 0x7efe8c108630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:14:16.641 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:16.641+0000 7efe92e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efe8c107210 0x7efe8c108b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:14:16.641 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:16.641+0000 7efe92e00700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7efe8c1114c0 con 0x7efe8c101b70 2024-03-18T15:14:16.641 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:16.641+0000 7efe92e00700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7efe8c111340 con 0x7efe8c107210 2024-03-18T15:14:16.641 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:16.641+0000 7efe92e00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7efe8c111640 con 0x7efe8c1011d0 2024-03-18T15:14:16.641 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:16.641+0000 7efe8be00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efe8c1011d0 0x7efe8c1080f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:14:16.641 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:16.641+0000 7efe8b400700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efe8c101b70 0x7efe8c108630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:14:16.641 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:16.641+0000 7efe8be00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efe8c1011d0 0x7efe8c1080f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.19:36800/0 (socket says 172.21.15.19:36800) 2024-03-18T15:14:16.642 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:16.641+0000 7efe8be00700 1 -- 172.21.15.19:0/2687504737 learned_addr learned my addr 172.21.15.19:0/2687504737 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:14:16.642 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:16.641+0000 7efe90e00700 1 --2- 172.21.15.19:0/2687504737 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efe8c107210 0x7efe8c108b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:14:16.642 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:16.641+0000 7efe8be00700 1 -- 172.21.15.19:0/2687504737 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efe8c107210 msgr2=0x7efe8c108b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:16.642 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:16.641+0000 7efe8be00700 1 --2- 172.21.15.19:0/2687504737 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efe8c107210 0x7efe8c108b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:16.643 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:16.641+0000 7efe8be00700 1 -- 172.21.15.19:0/2687504737 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efe8c101b70 msgr2=0x7efe8c108630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:16.643 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:16.641+0000 7efe8be00700 1 --2- 172.21.15.19:0/2687504737 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efe8c101b70 0x7efe8c108630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:16.643 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:16.641+0000 7efe8be00700 1 -- 172.21.15.19:0/2687504737 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe8c1a7050 con 0x7efe8c1011d0 2024-03-18T15:14:16.643 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:16.642+0000 7efe8be00700 1 --2- 172.21.15.19:0/2687504737 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efe8c1011d0 0x7efe8c1080f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7efe7c00e9c0 tx=0x7efe7c00ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:14:16.643 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:16.642+0000 7efe88c00700 1 -- 172.21.15.19:0/2687504737 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe7c004a30 con 0x7efe8c1011d0 2024-03-18T15:14:16.643 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:16.642+0000 7efe92e00700 1 -- 172.21.15.19:0/2687504737 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe8c1a7340 con 0x7efe8c1011d0 2024-03-18T15:14:16.643 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:16.642+0000 7efe88c00700 1 -- 172.21.15.19:0/2687504737 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efe7c00cca0 con 0x7efe8c1011d0 2024-03-18T15:14:16.643 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:16.642+0000 7efe88c00700 1 -- 172.21.15.19:0/2687504737 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe7c005680 con 0x7efe8c1011d0 2024-03-18T15:14:16.643 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:16.642+0000 7efe92e00700 1 -- 172.21.15.19:0/2687504737 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe8c1a78a0 con 0x7efe8c1011d0 2024-03-18T15:14:16.644 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:16.643+0000 7efe92e00700 1 -- 172.21.15.19:0/2687504737 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe8c068f90 con 0x7efe8c1011d0 2024-03-18T15:14:16.645 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:16.645+0000 7efe88c00700 1 -- 172.21.15.19:0/2687504737 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7efe7c00c7c0 con 0x7efe8c1011d0 2024-03-18T15:14:16.647 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:16.647+0000 7efe88c00700 1 --2- 172.21.15.19:0/2687504737 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7efe700a2910 0x7efe700a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:14:16.647 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:16.647+0000 7efe88c00700 1 -- 172.21.15.19:0/2687504737 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7efe7c0ce610 con 0x7efe8c1011d0 2024-03-18T15:14:16.647 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:16.647+0000 7efe8b400700 1 --2- 172.21.15.19:0/2687504737 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7efe700a2910 0x7efe700a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:14:16.648 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:16.648+0000 7efe8b400700 1 --2- 172.21.15.19:0/2687504737 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7efe700a2910 0x7efe700a4dd0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7efe78009810 tx=0x7efe78019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:14:16.651 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:16.651+0000 7efe88c00700 1 -- 172.21.15.19:0/2687504737 <== mon.1 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7efe7c09bab0 con 0x7efe8c1011d0 2024-03-18T15:14:16.877 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:16.876+0000 7efe92e00700 1 -- 172.21.15.19:0/2687504737 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7efe8c1a7b30 con 0x7efe8c1011d0 2024-03-18T15:14:16.878 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:16.878+0000 7efe88c00700 1 -- 172.21.15.19:0/2687504737 <== mon.1 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18421 (secure 0 0 0) 0x7efe7c09b3b0 con 0x7efe8c1011d0 2024-03-18T15:14:16.879 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-18T15:14:16.879 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":81,"fsid":"d4dc2ba6-e538-11ee-95c9-87774f69a715","created":"2024-03-18T15:05:57.538551+0000","modified":"2024-03-18T15:14:13.025582+0000","last_up_change":"2024-03-18T15:14:12.009522+0000","last_in_change":"2024-03-18T15:13:52.555387+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T15:09:56.843349+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"37b5a5cd-27ae-4e77-9b05-774b817b2ee9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6802","nonce":402333948},{"type":"v1","addr":"172.21.15.19:6803","nonce":402333948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6804","nonce":402333948},{"type":"v1","addr":"172.21.15.19:6805","nonce":402333948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6808","nonce":402333948},{"type":"v1","addr":"172.21.15.19:6809","nonce":402333948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6806","nonce":402333948},{"type":"v1","addr":"172.21.15.19:6807","nonce":402333948}]},"public_addr":"172.21.15.19:6803/402333948","cluster_addr":"172.21.15.19:6805/402333948","heartbeat_back_addr":"172.21.15.19:6809/402333948","heartbeat_front_addr":"172.21.15.19:6807/402333948","state":["exists","up"]},{"osd":1,"uuid":"3a1134e8-ede7-4dbd-9be1-eefd2cbb43bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1951880113},{"type":"v1","addr":"172.21.15.33:6801","nonce":1951880113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1951880113},{"type":"v1","addr":"172.21.15.33:6803","nonce":1951880113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1951880113},{"type":"v1","addr":"172.21.15.33:6807","nonce":1951880113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1951880113},{"type":"v1","addr":"172.21.15.33:6805","nonce":1951880113}]},"public_addr":"172.21.15.33:6801/1951880113","cluster_addr":"172.21.15.33:6803/1951880113","heartbeat_back_addr":"172.21.15.33:6807/1951880113","heartbeat_front_addr":"172.21.15.33:6805/1951880113","state":["exists","up"]},{"osd":2,"uuid":"f383a164-a3a0-49fd-a9e8-9c2178d8cfb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":1004727949},{"type":"v1","addr":"172.21.15.104:6801","nonce":1004727949}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":1004727949},{"type":"v1","addr":"172.21.15.104:6803","nonce":1004727949}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":1004727949},{"type":"v1","addr":"172.21.15.104:6807","nonce":1004727949}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":1004727949},{"type":"v1","addr":"172.21.15.104:6805","nonce":1004727949}]},"public_addr":"172.21.15.104:6801/1004727949","cluster_addr":"172.21.15.104:6803/1004727949","heartbeat_back_addr":"172.21.15.104:6807/1004727949","heartbeat_front_addr":"172.21.15.104:6805/1004727949","state":["exists","up"]},{"osd":3,"uuid":"c409e116-28bf-45dd-a791-539af22abe1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6810","nonce":3571525854},{"type":"v1","addr":"172.21.15.19:6811","nonce":3571525854}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6812","nonce":3571525854},{"type":"v1","addr":"172.21.15.19:6813","nonce":3571525854}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6816","nonce":3571525854},{"type":"v1","addr":"172.21.15.19:6817","nonce":3571525854}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6814","nonce":3571525854},{"type":"v1","addr":"172.21.15.19:6815","nonce":3571525854}]},"public_addr":"172.21.15.19:6811/3571525854","cluster_addr":"172.21.15.19:6813/3571525854","heartbeat_back_addr":"172.21.15.19:6817/3571525854","heartbeat_front_addr":"172.21.15.19:6815/3571525854","state":["exists","up"]},{"osd":4,"uuid":"696eb719-0b3e-43ae-b5f1-ccd73a3b2843","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":624901749},{"type":"v1","addr":"172.21.15.33:6809","nonce":624901749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":624901749},{"type":"v1","addr":"172.21.15.33:6811","nonce":624901749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":624901749},{"type":"v1","addr":"172.21.15.33:6815","nonce":624901749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":624901749},{"type":"v1","addr":"172.21.15.33:6813","nonce":624901749}]},"public_addr":"172.21.15.33:6809/624901749","cluster_addr":"172.21.15.33:6811/624901749","heartbeat_back_addr":"172.21.15.33:6815/624901749","heartbeat_front_addr":"172.21.15.33:6813/624901749","state":["exists","up"]},{"osd":5,"uuid":"21789779-8bb9-42c7-b9ee-437985f11e3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":1258208297},{"type":"v1","addr":"172.21.15.104:6809","nonce":1258208297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":1258208297},{"type":"v1","addr":"172.21.15.104:6811","nonce":1258208297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":1258208297},{"type":"v1","addr":"172.21.15.104:6815","nonce":1258208297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":1258208297},{"type":"v1","addr":"172.21.15.104:6813","nonce":1258208297}]},"public_addr":"172.21.15.104:6809/1258208297","cluster_addr":"172.21.15.104:6811/1258208297","heartbeat_back_addr":"172.21.15.104:6815/1258208297","heartbeat_front_addr":"172.21.15.104:6813/1258208297","state":["exists","up"]},{"osd":6,"uuid":"0e92a2ef-43cd-487c-9203-e219158aef8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6818","nonce":2782482301},{"type":"v1","addr":"172.21.15.19:6819","nonce":2782482301}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6820","nonce":2782482301},{"type":"v1","addr":"172.21.15.19:6821","nonce":2782482301}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6824","nonce":2782482301},{"type":"v1","addr":"172.21.15.19:6825","nonce":2782482301}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6822","nonce":2782482301},{"type":"v1","addr":"172.21.15.19:6823","nonce":2782482301}]},"public_addr":"172.21.15.19:6819/2782482301","cluster_addr":"172.21.15.19:6821/2782482301","heartbeat_back_addr":"172.21.15.19:6825/2782482301","heartbeat_front_addr":"172.21.15.19:6823/2782482301","state":["exists","up"]},{"osd":7,"uuid":"a16e39a1-3685-491d-b376-3e80a2084072","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":4238160988},{"type":"v1","addr":"172.21.15.33:6817","nonce":4238160988}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":4238160988},{"type":"v1","addr":"172.21.15.33:6819","nonce":4238160988}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":4238160988},{"type":"v1","addr":"172.21.15.33:6823","nonce":4238160988}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":4238160988},{"type":"v1","addr":"172.21.15.33:6821","nonce":4238160988}]},"public_addr":"172.21.15.33:6817/4238160988","cluster_addr":"172.21.15.33:6819/4238160988","heartbeat_back_addr":"172.21.15.33:6823/4238160988","heartbeat_front_addr":"172.21.15.33:6821/4238160988","state":["exists","up"]},{"osd":8,"uuid":"d5c3fa92-cd5d-4349-9c7d-d5666bacde16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":1014779813},{"type":"v1","addr":"172.21.15.104:6817","nonce":1014779813}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":1014779813},{"type":"v1","addr":"172.21.15.104:6819","nonce":1014779813}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":1014779813},{"type":"v1","addr":"172.21.15.104:6823","nonce":1014779813}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":1014779813},{"type":"v1","addr":"172.21.15.104:6821","nonce":1014779813}]},"public_addr":"172.21.15.104:6817/1014779813","cluster_addr":"172.21.15.104:6819/1014779813","heartbeat_back_addr":"172.21.15.104:6823/1014779813","heartbeat_front_addr":"172.21.15.104:6821/1014779813","state":["exists","up"]},{"osd":9,"uuid":"eed2e9b2-26d3-4c46-88ec-34b5596fe991","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6826","nonce":2641151009},{"type":"v1","addr":"172.21.15.19:6827","nonce":2641151009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6828","nonce":2641151009},{"type":"v1","addr":"172.21.15.19:6829","nonce":2641151009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6832","nonce":2641151009},{"type":"v1","addr":"172.21.15.19:6833","nonce":2641151009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6830","nonce":2641151009},{"type":"v1","addr":"172.21.15.19:6831","nonce":2641151009}]},"public_addr":"172.21.15.19:6827/2641151009","cluster_addr":"172.21.15.19:6829/2641151009","heartbeat_back_addr":"172.21.15.19:6833/2641151009","heartbeat_front_addr":"172.21.15.19:6831/2641151009","state":["exists","up"]},{"osd":10,"uuid":"f3c35ea5-fbe7-4dc4-93c5-b92d1a714db3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":1346058869},{"type":"v1","addr":"172.21.15.33:6825","nonce":1346058869}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":1346058869},{"type":"v1","addr":"172.21.15.33:6827","nonce":1346058869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":1346058869},{"type":"v1","addr":"172.21.15.33:6831","nonce":1346058869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":1346058869},{"type":"v1","addr":"172.21.15.33:6829","nonce":1346058869}]},"public_addr":"172.21.15.33:6825/1346058869","cluster_addr":"172.21.15.33:6827/1346058869","heartbeat_back_addr":"172.21.15.33:6831/1346058869","heartbeat_front_addr":"172.21.15.33:6829/1346058869","state":["exists","up"]},{"osd":11,"uuid":"c8b51571-f807-4b62-90b6-7529e09e59da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":3759498281},{"type":"v1","addr":"172.21.15.104:6825","nonce":3759498281}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":3759498281},{"type":"v1","addr":"172.21.15.104:6827","nonce":3759498281}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":3759498281},{"type":"v1","addr":"172.21.15.104:6831","nonce":3759498281}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":3759498281},{"type":"v1","addr":"172.21.15.104:6829","nonce":3759498281}]},"public_addr":"172.21.15.104:6825/3759498281","cluster_addr":"172.21.15.104:6827/3759498281","heartbeat_back_addr":"172.21.15.104:6831/3759498281","heartbeat_front_addr":"172.21.15.104:6829/3759498281","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:09:05.267795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:09:30.234068+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:09:53.870232+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:10:26.471296+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:10:53.180341+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:11:17.801361+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:11:51.364816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:12:18.306408+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:12:43.978744+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:13:18.213964+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:13:45.437544+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:14:10.148580+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.19:0/3860783274":"2024-03-19T15:07:14.150983+0000","172.21.15.19:0/2737223213":"2024-03-19T15:07:14.150983+0000","172.21.15.19:0/176699025":"2024-03-19T15:06:38.688405+0000","172.21.15.19:0/2316969005":"2024-03-19T15:07:14.150983+0000","172.21.15.19:6801/2776008962":"2024-03-19T15:07:14.150983+0000","172.21.15.19:0/2793980440":"2024-03-19T15:06:38.688405+0000","172.21.15.19:6800/3264337803":"2024-03-19T15:06:38.688405+0000","172.21.15.19:6801/3264337803":"2024-03-19T15:06:38.688405+0000","172.21.15.19:0/3166705295":"2024-03-19T15:06:38.688405+0000","172.21.15.19:6800/2776008962":"2024-03-19T15:07:14.150983+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T15:14:16.881 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:16.881+0000 7efe92e00700 1 -- 172.21.15.19:0/2687504737 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7efe700a2910 msgr2=0x7efe700a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:16.881 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:16.881+0000 7efe92e00700 1 --2- 172.21.15.19:0/2687504737 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7efe700a2910 0x7efe700a4dd0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7efe78009810 tx=0x7efe78019040 comp rx=0 tx=0).stop 2024-03-18T15:14:16.882 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:16.881+0000 7efe92e00700 1 -- 172.21.15.19:0/2687504737 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efe8c1011d0 msgr2=0x7efe8c1080f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:16.882 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:16.881+0000 7efe92e00700 1 --2- 172.21.15.19:0/2687504737 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efe8c1011d0 0x7efe8c1080f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7efe7c00e9c0 tx=0x7efe7c00ecd0 comp rx=0 tx=0).stop 2024-03-18T15:14:16.882 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:16.881+0000 7efe92e00700 1 -- 172.21.15.19:0/2687504737 shutdown_connections 2024-03-18T15:14:16.882 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:16.881+0000 7efe92e00700 1 --2- 172.21.15.19:0/2687504737 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7efe700a2910 0x7efe700a4dd0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:16.882 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:16.881+0000 7efe92e00700 1 --2- 172.21.15.19:0/2687504737 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efe8c1011d0 0x7efe8c1080f0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:16.882 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:16.881+0000 7efe92e00700 1 --2- 172.21.15.19:0/2687504737 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efe8c107210 0x7efe8c108b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:16.882 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:16.881+0000 7efe92e00700 1 --2- 172.21.15.19:0/2687504737 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efe8c101b70 0x7efe8c108630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:16.882 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:16.881+0000 7efe92e00700 1 -- 172.21.15.19:0/2687504737 >> 172.21.15.19:0/2687504737 conn(0x7efe8c078100 msgr2=0x7efe8c0fec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:14:16.882 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:16.881+0000 7efe92e00700 1 -- 172.21.15.19:0/2687504737 shutdown_connections 2024-03-18T15:14:16.882 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:16.881+0000 7efe92e00700 1 -- 172.21.15.19:0/2687504737 wait complete. 2024-03-18T15:14:17.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:17 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/2687504737' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:14:17.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:17 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/2687504737' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:14:17.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:17 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/2687504737' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:14:17.762 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T15:09:56.843349+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T15:14:17.763 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-18T15:14:18.245 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:14:18.395 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:18 smithi019 ceph-mon[31522]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:14:18.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:18 smithi104 ceph-mon[36070]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:14:18.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:18 smithi033 ceph-mon[36648]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:14:20.210 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:20 smithi019 ceph-mon[31522]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:14:20.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:20 smithi104 ceph-mon[36070]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:14:20.502 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:20.500+0000 7faad1200700 1 -- 172.21.15.19:0/3180884380 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7faacc100990 msgr2=0x7faacc100e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:20.502 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:20.500+0000 7faad1200700 1 --2- 172.21.15.19:0/3180884380 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7faacc100990 0x7faacc100e10 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7faab4009990 tx=0x7faab4009ca0 comp rx=0 tx=0).stop 2024-03-18T15:14:20.502 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:20.500+0000 7faad1200700 1 -- 172.21.15.19:0/3180884380 shutdown_connections 2024-03-18T15:14:20.502 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:20.500+0000 7faad1200700 1 --2- 172.21.15.19:0/3180884380 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faacc101350 0x7faacc10d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:20.502 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:20.500+0000 7faad1200700 1 --2- 172.21.15.19:0/3180884380 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faacc106990 0x7faacc106d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:20.503 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:20.500+0000 7faad1200700 1 --2- 172.21.15.19:0/3180884380 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7faacc100990 0x7faacc100e10 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:20.503 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:20.500+0000 7faad1200700 1 -- 172.21.15.19:0/3180884380 >> 172.21.15.19:0/3180884380 conn(0x7faacc0777d0 msgr2=0x7faacc0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:14:20.503 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:20.500+0000 7faad1200700 1 -- 172.21.15.19:0/3180884380 shutdown_connections 2024-03-18T15:14:20.503 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:20.500+0000 7faad1200700 1 -- 172.21.15.19:0/3180884380 wait complete. 2024-03-18T15:14:20.503 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:20.501+0000 7faad1200700 1 Processor -- start 2024-03-18T15:14:20.503 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:20.501+0000 7faad1200700 1 -- start start 2024-03-18T15:14:20.505 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:20.502+0000 7faad1200700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faacc100990 0x7faacc1a0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:14:20.505 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:20.502+0000 7faad1200700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7faacc101350 0x7faacc1a0e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:14:20.505 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:20.502+0000 7faad1200700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faacc106990 0x7faacc19ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:14:20.505 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:20.502+0000 7faad1200700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7faacc111520 con 0x7faacc101350 2024-03-18T15:14:20.506 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:20.502+0000 7faad1200700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7faacc1113a0 con 0x7faacc100990 2024-03-18T15:14:20.506 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:20.502+0000 7faad1200700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7faacc1116a0 con 0x7faacc106990 2024-03-18T15:14:20.506 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:20.502+0000 7faac9e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faacc100990 0x7faacc1a0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:14:20.506 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:20.502+0000 7faac1400700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7faacc101350 0x7faacc1a0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:14:20.507 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:20.502+0000 7faac9e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faacc100990 0x7faacc1a0940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:55134/0 (socket says 172.21.15.19:55134) 2024-03-18T15:14:20.507 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:20.502+0000 7faac9e00700 1 -- 172.21.15.19:0/1081021718 learned_addr learned my addr 172.21.15.19:0/1081021718 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:14:20.507 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:20.502+0000 7faaca800700 1 --2- 172.21.15.19:0/1081021718 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faacc106990 0x7faacc19ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:14:20.508 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:20.503+0000 7faac9e00700 1 -- 172.21.15.19:0/1081021718 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faacc106990 msgr2=0x7faacc19ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:20.508 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:20.503+0000 7faac9e00700 1 --2- 172.21.15.19:0/1081021718 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faacc106990 0x7faacc19ab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:20.508 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:20.503+0000 7faac9e00700 1 -- 172.21.15.19:0/1081021718 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7faacc101350 msgr2=0x7faacc1a0e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:20.508 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:20.503+0000 7faac9e00700 1 --2- 172.21.15.19:0/1081021718 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7faacc101350 0x7faacc1a0e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:20.508 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:20.503+0000 7faac9e00700 1 -- 172.21.15.19:0/1081021718 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faacc19b260 con 0x7faacc100990 2024-03-18T15:14:20.508 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:20.503+0000 7faac9e00700 1 --2- 172.21.15.19:0/1081021718 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faacc100990 0x7faacc1a0940 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7faabc00e9a0 tx=0x7faabc00ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:14:20.508 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:20.503+0000 7faac3400700 1 -- 172.21.15.19:0/1081021718 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faabc00ca10 con 0x7faacc100990 2024-03-18T15:14:20.508 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:20.503+0000 7faac3400700 1 -- 172.21.15.19:0/1081021718 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faabc00cbb0 con 0x7faacc100990 2024-03-18T15:14:20.508 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:20.503+0000 7faad1200700 1 -- 172.21.15.19:0/1081021718 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faacc19b550 con 0x7faacc100990 2024-03-18T15:14:20.508 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:20.503+0000 7faac3400700 1 -- 172.21.15.19:0/1081021718 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faabc018680 con 0x7faacc100990 2024-03-18T15:14:20.508 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:20.504+0000 7faad1200700 1 -- 172.21.15.19:0/1081021718 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faacc1a7810 con 0x7faacc100990 2024-03-18T15:14:20.508 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:20.505+0000 7faad1200700 1 -- 172.21.15.19:0/1081021718 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faacc1020d0 con 0x7faacc100990 2024-03-18T15:14:20.510 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:20 smithi033 ceph-mon[36648]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:14:20.510 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:20.506+0000 7faac3400700 1 -- 172.21.15.19:0/1081021718 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7faabc010430 con 0x7faacc100990 2024-03-18T15:14:20.511 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:20.508+0000 7faac3400700 1 --2- 172.21.15.19:0/1081021718 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7faab00a2890 0x7faab00a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:14:20.511 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:20.508+0000 7faac3400700 1 -- 172.21.15.19:0/1081021718 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7faabc0ce780 con 0x7faacc100990 2024-03-18T15:14:20.511 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:20.508+0000 7faac1400700 1 --2- 172.21.15.19:0/1081021718 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7faab00a2890 0x7faab00a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:14:20.511 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:20.509+0000 7faac1400700 1 --2- 172.21.15.19:0/1081021718 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7faab00a2890 0x7faab00a4d50 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7faab40098e0 tx=0x7faab4005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:14:20.512 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:20.512+0000 7faac3400700 1 -- 172.21.15.19:0/1081021718 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7faabc09bcc0 con 0x7faacc100990 2024-03-18T15:14:20.742 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:20.741+0000 7faad1200700 1 -- 172.21.15.19:0/1081021718 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7faacc068f90 con 0x7faacc100990 2024-03-18T15:14:20.742 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:20.742+0000 7faac3400700 1 -- 172.21.15.19:0/1081021718 <== mon.2 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7faacc068f90 con 0x7faacc100990 2024-03-18T15:14:20.743 INFO:teuthology.orchestra.run.smithi019.stdout:pg_num: 1 2024-03-18T15:14:20.746 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:20.746+0000 7faad1200700 1 -- 172.21.15.19:0/1081021718 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7faab00a2890 msgr2=0x7faab00a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:20.747 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:20.746+0000 7faad1200700 1 --2- 172.21.15.19:0/1081021718 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7faab00a2890 0x7faab00a4d50 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7faab40098e0 tx=0x7faab4005960 comp rx=0 tx=0).stop 2024-03-18T15:14:20.747 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:20.746+0000 7faad1200700 1 -- 172.21.15.19:0/1081021718 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faacc100990 msgr2=0x7faacc1a0940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:20.747 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:20.746+0000 7faad1200700 1 --2- 172.21.15.19:0/1081021718 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faacc100990 0x7faacc1a0940 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7faabc00e9a0 tx=0x7faabc00ecb0 comp rx=0 tx=0).stop 2024-03-18T15:14:20.747 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:20.746+0000 7faad1200700 1 -- 172.21.15.19:0/1081021718 shutdown_connections 2024-03-18T15:14:20.747 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:20.746+0000 7faad1200700 1 --2- 172.21.15.19:0/1081021718 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7faab00a2890 0x7faab00a4d50 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:20.747 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:20.746+0000 7faad1200700 1 --2- 172.21.15.19:0/1081021718 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faacc106990 0x7faacc19ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:20.747 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:20.746+0000 7faad1200700 1 --2- 172.21.15.19:0/1081021718 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faacc100990 0x7faacc1a0940 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:20.747 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:20.746+0000 7faad1200700 1 --2- 172.21.15.19:0/1081021718 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7faacc101350 0x7faacc1a0e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:20.747 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:20.746+0000 7faad1200700 1 -- 172.21.15.19:0/1081021718 >> 172.21.15.19:0/1081021718 conn(0x7faacc0777d0 msgr2=0x7faacc103d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:14:20.747 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:20.746+0000 7faad1200700 1 -- 172.21.15.19:0/1081021718 shutdown_connections 2024-03-18T15:14:20.747 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:20.746+0000 7faad1200700 1 -- 172.21.15.19:0/1081021718 wait complete. 2024-03-18T15:14:21.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:21 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/1081021718' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T15:14:21.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:21 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/1081021718' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T15:14:21.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:21 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/1081021718' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T15:14:21.564 INFO:tasks.cephadm:Adding mds.a on smithi019 2024-03-18T15:14:21.564 INFO:tasks.cephadm:Adding mds.d on smithi019 2024-03-18T15:14:21.564 INFO:tasks.cephadm:Adding mds.g on smithi019 2024-03-18T15:14:21.564 INFO:tasks.cephadm:Adding mds.j on smithi019 2024-03-18T15:14:21.564 INFO:tasks.cephadm:Adding mds.b on smithi033 2024-03-18T15:14:21.564 INFO:tasks.cephadm:Adding mds.e on smithi033 2024-03-18T15:14:21.564 INFO:tasks.cephadm:Adding mds.h on smithi033 2024-03-18T15:14:21.564 INFO:tasks.cephadm:Adding mds.k on smithi033 2024-03-18T15:14:21.564 INFO:tasks.cephadm:Adding mds.c on smithi104 2024-03-18T15:14:21.564 INFO:tasks.cephadm:Adding mds.f on smithi104 2024-03-18T15:14:21.564 INFO:tasks.cephadm:Adding mds.i on smithi104 2024-03-18T15:14:21.564 INFO:tasks.cephadm:Adding mds.l on smithi104 2024-03-18T15:14:21.565 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 -- ceph orch apply mds all '12;smithi019=a;smithi019=d;smithi019=g;smithi019=j;smithi033=b;smithi033=e;smithi033=h;smithi033=k;smithi104=c;smithi104=f;smithi104=i;smithi104=l' 2024-03-18T15:14:21.817 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T15:14:22.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:22 smithi104 ceph-mon[36070]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:14:22.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:22 smithi019 ceph-mon[31522]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:14:22.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:22 smithi033 ceph-mon[36648]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:14:23.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:23 smithi104 ceph-mon[36070]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:14:23.465 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:14:23.463+0000 7f3de43c0700 1 -- 172.21.15.104:0/4095435972 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3ddc102da0 msgr2=0x7f3ddc109650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:23.465 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:14:23.463+0000 7f3de43c0700 1 --2- 172.21.15.104:0/4095435972 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3ddc102da0 0x7f3ddc109650 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3dd400b0a0 tx=0x7f3dd400b3b0 comp rx=0 tx=0).stop 2024-03-18T15:14:23.465 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:14:23.463+0000 7f3de43c0700 1 -- 172.21.15.104:0/4095435972 shutdown_connections 2024-03-18T15:14:23.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:14:23.463+0000 7f3de43c0700 1 --2- 172.21.15.104:0/4095435972 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3ddc102da0 0x7f3ddc109650 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:23.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:14:23.463+0000 7f3de43c0700 1 --2- 172.21.15.104:0/4095435972 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3ddc101200 0x7f3ddc101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:23.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:14:23.463+0000 7f3de43c0700 1 --2- 172.21.15.104:0/4095435972 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3ddc102400 0x7f3ddc102860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:23.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:14:23.463+0000 7f3de43c0700 1 -- 172.21.15.104:0/4095435972 >> 172.21.15.104:0/4095435972 conn(0x7f3ddc0fc7c0 msgr2=0x7f3ddc0febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:14:23.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:14:23.463+0000 7f3de43c0700 1 -- 172.21.15.104:0/4095435972 shutdown_connections 2024-03-18T15:14:23.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:14:23.463+0000 7f3de43c0700 1 -- 172.21.15.104:0/4095435972 wait complete. 2024-03-18T15:14:23.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:14:23.463+0000 7f3de43c0700 1 Processor -- start 2024-03-18T15:14:23.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:14:23.463+0000 7f3de43c0700 1 -- start start 2024-03-18T15:14:23.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:14:23.463+0000 7f3de43c0700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3ddc101200 0x7f3ddc06b550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:14:23.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:14:23.463+0000 7f3de43c0700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3ddc102400 0x7f3ddc19f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:14:23.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:14:23.463+0000 7f3de43c0700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3ddc102da0 0x7f3ddc1a1c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:14:23.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:14:23.463+0000 7f3de43c0700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f3ddc10cee0 con 0x7f3ddc102da0 2024-03-18T15:14:23.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:14:23.463+0000 7f3de43c0700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f3ddc10cd60 con 0x7f3ddc101200 2024-03-18T15:14:23.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:14:23.463+0000 7f3de43c0700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f3ddc10d060 con 0x7f3ddc102400 2024-03-18T15:14:23.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:14:23.464+0000 7f3de195b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3ddc102400 0x7f3ddc19f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:14:23.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:14:23.464+0000 7f3de195b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3ddc102400 0x7f3ddc19f720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:33718/0 (socket says 172.21.15.104:33718) 2024-03-18T15:14:23.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:14:23.464+0000 7f3de195b700 1 -- 172.21.15.104:0/3904359884 learned_addr learned my addr 172.21.15.104:0/3904359884 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-03-18T15:14:23.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:14:23.464+0000 7f3de215c700 1 --2- 172.21.15.104:0/3904359884 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3ddc101200 0x7f3ddc06b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:14:23.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:14:23.464+0000 7f3de195b700 1 -- 172.21.15.104:0/3904359884 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3ddc101200 msgr2=0x7f3ddc06b550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:23.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:14:23.464+0000 7f3de195b700 1 --2- 172.21.15.104:0/3904359884 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3ddc101200 0x7f3ddc06b550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:23.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:14:23.464+0000 7f3de195b700 1 -- 172.21.15.104:0/3904359884 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3ddc102da0 msgr2=0x7f3ddc1a1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:23.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:14:23.464+0000 7f3de195b700 1 --2- 172.21.15.104:0/3904359884 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3ddc102da0 0x7f3ddc1a1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:23.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:14:23.464+0000 7f3de195b700 1 -- 172.21.15.104:0/3904359884 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ddc1a2450 con 0x7f3ddc102400 2024-03-18T15:14:23.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:14:23.464+0000 7f3de195b700 1 --2- 172.21.15.104:0/3904359884 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3ddc102400 0x7f3ddc19f720 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3dcc00b530 tx=0x7f3dcc00b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:14:23.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:14:23.464+0000 7f3dd37fe700 1 -- 172.21.15.104:0/3904359884 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dcc015020 con 0x7f3ddc102400 2024-03-18T15:14:23.468 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:14:23.464+0000 7f3dd37fe700 1 -- 172.21.15.104:0/3904359884 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3dcc011d10 con 0x7f3ddc102400 2024-03-18T15:14:23.468 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:14:23.464+0000 7f3de43c0700 1 -- 172.21.15.104:0/3904359884 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ddc1a2740 con 0x7f3ddc102400 2024-03-18T15:14:23.468 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:14:23.464+0000 7f3dd37fe700 1 -- 172.21.15.104:0/3904359884 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dcc00f2e0 con 0x7f3ddc102400 2024-03-18T15:14:23.468 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:14:23.464+0000 7f3de43c0700 1 -- 172.21.15.104:0/3904359884 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ddc1a2d00 con 0x7f3ddc102400 2024-03-18T15:14:23.468 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:23 smithi019 ceph-mon[31522]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:14:23.469 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:14:23.465+0000 7f3de43c0700 1 -- 172.21.15.104:0/3904359884 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ddc068f90 con 0x7f3ddc102400 2024-03-18T15:14:23.469 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:14:23.466+0000 7f3dd37fe700 1 -- 172.21.15.104:0/3904359884 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f3dcc018020 con 0x7f3ddc102400 2024-03-18T15:14:23.469 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:14:23.467+0000 7f3dd37fe700 1 --2- 172.21.15.104:0/3904359884 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f3dc80a2890 0x7f3dc80a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:14:23.469 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:14:23.467+0000 7f3dd37fe700 1 -- 172.21.15.104:0/3904359884 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f3dcc0ce260 con 0x7f3ddc102400 2024-03-18T15:14:23.469 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:14:23.467+0000 7f3de215c700 1 --2- 172.21.15.104:0/3904359884 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f3dc80a2890 0x7f3dc80a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:14:23.469 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:14:23.468+0000 7f3de215c700 1 --2- 172.21.15.104:0/3904359884 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f3dc80a2890 0x7f3dc80a4d50 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f3ddc102260 tx=0x7f3dd8009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:14:23.471 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:14:23.470+0000 7f3dd37fe700 1 -- 172.21.15.104:0/3904359884 <== mon.1 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3dcc01f020 con 0x7f3ddc102400 2024-03-18T15:14:23.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:23 smithi033 ceph-mon[36648]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:14:23.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:14:23.672+0000 7f3de43c0700 1 -- 172.21.15.104:0/3904359884 --> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi019=a;smithi019=d;smithi019=g;smithi019=j;smithi033=b;smithi033=e;smithi033=h;smithi033=k;smithi104=c;smithi104=f;smithi104=i;smithi104=l", "target": ["mon-mgr", ""]}) -- 0x7f3ddc062810 con 0x7f3dc80a2890 2024-03-18T15:14:23.691 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:14:23.690+0000 7f3dd37fe700 1 -- 172.21.15.104:0/3904359884 <== mgr.14152 v2:172.21.15.19:6800/2017714236 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7f3ddc062810 con 0x7f3dc80a2890 2024-03-18T15:14:23.691 INFO:teuthology.orchestra.run.smithi104.stdout:Scheduled mds.all update... 2024-03-18T15:14:23.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:14:23.692+0000 7f3de43c0700 1 -- 172.21.15.104:0/3904359884 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f3dc80a2890 msgr2=0x7f3dc80a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:23.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:14:23.692+0000 7f3de43c0700 1 --2- 172.21.15.104:0/3904359884 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f3dc80a2890 0x7f3dc80a4d50 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f3ddc102260 tx=0x7f3dd8009190 comp rx=0 tx=0).stop 2024-03-18T15:14:23.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:14:23.692+0000 7f3de43c0700 1 -- 172.21.15.104:0/3904359884 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3ddc102400 msgr2=0x7f3ddc19f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:23.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:14:23.692+0000 7f3de43c0700 1 --2- 172.21.15.104:0/3904359884 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3ddc102400 0x7f3ddc19f720 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3dcc00b530 tx=0x7f3dcc00b840 comp rx=0 tx=0).stop 2024-03-18T15:14:23.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:14:23.692+0000 7f3de43c0700 1 -- 172.21.15.104:0/3904359884 shutdown_connections 2024-03-18T15:14:23.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:14:23.692+0000 7f3de43c0700 1 --2- 172.21.15.104:0/3904359884 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f3dc80a2890 0x7f3dc80a4d50 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:23.693 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:14:23.692+0000 7f3de43c0700 1 --2- 172.21.15.104:0/3904359884 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3ddc102da0 0x7f3ddc1a1c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:23.694 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:14:23.692+0000 7f3de43c0700 1 --2- 172.21.15.104:0/3904359884 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3ddc101200 0x7f3ddc06b550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:23.694 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:14:23.692+0000 7f3de43c0700 1 --2- 172.21.15.104:0/3904359884 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3ddc102400 0x7f3ddc19f720 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:23.694 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:14:23.692+0000 7f3de43c0700 1 -- 172.21.15.104:0/3904359884 >> 172.21.15.104:0/3904359884 conn(0x7f3ddc0fc7c0 msgr2=0x7f3ddc105700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:14:23.694 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:14:23.692+0000 7f3de43c0700 1 -- 172.21.15.104:0/3904359884 shutdown_connections 2024-03-18T15:14:23.694 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-18T15:14:23.692+0000 7f3de43c0700 1 -- 172.21.15.104:0/3904359884 wait complete. 2024-03-18T15:14:24.292 DEBUG:teuthology.orchestra.run.smithi019:mds.a> sudo journalctl -f -n 0 -u ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@mds.a.service 2024-03-18T15:14:24.300 DEBUG:teuthology.orchestra.run.smithi019:mds.d> sudo journalctl -f -n 0 -u ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@mds.d.service 2024-03-18T15:14:24.345 DEBUG:teuthology.orchestra.run.smithi019:mds.g> sudo journalctl -f -n 0 -u ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@mds.g.service 2024-03-18T15:14:24.390 DEBUG:teuthology.orchestra.run.smithi019:mds.j> sudo journalctl -f -n 0 -u ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@mds.j.service 2024-03-18T15:14:24.436 DEBUG:teuthology.orchestra.run.smithi033:mds.b> sudo journalctl -f -n 0 -u ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@mds.b.service 2024-03-18T15:14:24.439 DEBUG:teuthology.orchestra.run.smithi033:mds.e> sudo journalctl -f -n 0 -u ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@mds.e.service 2024-03-18T15:14:24.482 DEBUG:teuthology.orchestra.run.smithi033:mds.h> sudo journalctl -f -n 0 -u ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@mds.h.service 2024-03-18T15:14:24.525 DEBUG:teuthology.orchestra.run.smithi033:mds.k> sudo journalctl -f -n 0 -u ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@mds.k.service 2024-03-18T15:14:24.568 DEBUG:teuthology.orchestra.run.smithi104:mds.c> sudo journalctl -f -n 0 -u ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@mds.c.service 2024-03-18T15:14:24.570 DEBUG:teuthology.orchestra.run.smithi104:mds.f> sudo journalctl -f -n 0 -u ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@mds.f.service 2024-03-18T15:14:24.613 DEBUG:teuthology.orchestra.run.smithi104:mds.i> sudo journalctl -f -n 0 -u ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@mds.i.service 2024-03-18T15:14:24.656 DEBUG:teuthology.orchestra.run.smithi104:mds.l> sudo journalctl -f -n 0 -u ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@mds.l.service 2024-03-18T15:14:24.699 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-03-18T15:14:24.700 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T15:14:24.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:24 smithi033 ceph-mon[36648]: from='client.24409 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi019=a;smithi019=d;smithi019=g;smithi019=j;smithi033=b;smithi033=e;smithi033=h;smithi033=k;smithi104=c;smithi104=f;smithi104=i;smithi104=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:14:24.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:24 smithi033 ceph-mon[36648]: Saving service mds.all spec with placement smithi019=a;smithi019=d;smithi019=g;smithi019=j;smithi033=b;smithi033=e;smithi033=h;smithi033=k;smithi104=c;smithi104=f;smithi104=i;smithi104=l;count:12 2024-03-18T15:14:24.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:24 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:24.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:24 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:14:24.928 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:24.928+0000 7fc05b200640 1 -- 172.21.15.19:0/3261208012 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc05c075820 msgr2=0x7fc05c073c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:24.929 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:24.928+0000 7fc05b200640 1 --2- 172.21.15.19:0/3261208012 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc05c075820 0x7fc05c073c70 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fc04c009930 tx=0x7fc04c02bc30 comp rx=0 tx=0).stop 2024-03-18T15:14:24.929 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:24.928+0000 7fc05b200640 1 -- 172.21.15.19:0/3261208012 shutdown_connections 2024-03-18T15:14:24.929 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:24.928+0000 7fc05b200640 1 --2- 172.21.15.19:0/3261208012 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc05c10dbb0 0x7fc05c10ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:24.929 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:24.928+0000 7fc05b200640 1 --2- 172.21.15.19:0/3261208012 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc05c074240 0x7fc05c0746c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:24.929 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:24.928+0000 7fc05b200640 1 --2- 172.21.15.19:0/3261208012 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc05c075820 0x7fc05c073c70 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:24.929 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:24.928+0000 7fc05b200640 1 -- 172.21.15.19:0/3261208012 >> 172.21.15.19:0/3261208012 conn(0x7fc05c0fcb70 msgr2=0x7fc05c0fef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:14:24.929 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:24.928+0000 7fc05b200640 1 -- 172.21.15.19:0/3261208012 shutdown_connections 2024-03-18T15:14:24.929 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:24.928+0000 7fc05b200640 1 -- 172.21.15.19:0/3261208012 wait complete. 2024-03-18T15:14:24.932 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:24.929+0000 7fc05b200640 1 Processor -- start 2024-03-18T15:14:24.932 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:24.929+0000 7fc05b200640 1 -- start start 2024-03-18T15:14:24.932 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:24.929+0000 7fc05b200640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc05c074240 0x7fc05c19df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:14:24.932 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:24.930+0000 7fc05b200640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc05c075820 0x7fc05c19e480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:14:24.932 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:24.930+0000 7fc05b200640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc05c10dbb0 0x7fc05c1a54f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:14:24.932 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:24.930+0000 7fc05b200640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fc05c113850 con 0x7fc05c075820 2024-03-18T15:14:24.932 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:24.930+0000 7fc05b200640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc05c1136d0 con 0x7fc05c074240 2024-03-18T15:14:24.932 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:24.930+0000 7fc05b200640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fc05c1139d0 con 0x7fc05c10dbb0 2024-03-18T15:14:24.933 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:24.930+0000 7fc051400640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc05c075820 0x7fc05c19e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:14:24.933 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:24.930+0000 7fc051400640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc05c075820 0x7fc05c19e480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:42922/0 (socket says 172.21.15.19:42922) 2024-03-18T15:14:24.933 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:24.930+0000 7fc051400640 1 -- 172.21.15.19:0/779636491 learned_addr learned my addr 172.21.15.19:0/779636491 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:14:24.933 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:24.930+0000 7fc05a800640 1 --2- 172.21.15.19:0/779636491 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc05c10dbb0 0x7fc05c1a54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:14:24.933 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:24.930+0000 7fc059e00640 1 --2- 172.21.15.19:0/779636491 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc05c074240 0x7fc05c19df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:14:24.933 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:24.930+0000 7fc051400640 1 -- 172.21.15.19:0/779636491 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc05c10dbb0 msgr2=0x7fc05c1a54f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:24.933 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:24.930+0000 7fc051400640 1 --2- 172.21.15.19:0/779636491 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc05c10dbb0 0x7fc05c1a54f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:24.933 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:24.930+0000 7fc051400640 1 -- 172.21.15.19:0/779636491 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc05c074240 msgr2=0x7fc05c19df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:24.933 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:24.930+0000 7fc051400640 1 --2- 172.21.15.19:0/779636491 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc05c074240 0x7fc05c19df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:24.933 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:24.930+0000 7fc051400640 1 -- 172.21.15.19:0/779636491 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc05c1a5b60 con 0x7fc05c075820 2024-03-18T15:14:24.933 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:24.930+0000 7fc05a800640 1 --2- 172.21.15.19:0/779636491 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc05c10dbb0 0x7fc05c1a54f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:14:24.933 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:24.930+0000 7fc059e00640 1 --2- 172.21.15.19:0/779636491 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc05c074240 0x7fc05c19df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:14:24.934 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:24.931+0000 7fc051400640 1 --2- 172.21.15.19:0/779636491 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc05c075820 0x7fc05c19e480 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fc04400ebc0 tx=0x7fc04400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:14:24.934 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:24.931+0000 7fc053400640 1 -- 172.21.15.19:0/779636491 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc044010630 con 0x7fc05c075820 2024-03-18T15:14:24.934 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:24.931+0000 7fc053400640 1 -- 172.21.15.19:0/779636491 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc044010cb0 con 0x7fc05c075820 2024-03-18T15:14:24.934 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:24.931+0000 7fc05b200640 1 -- 172.21.15.19:0/779636491 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc05c1a5e50 con 0x7fc05c075820 2024-03-18T15:14:24.934 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:24.931+0000 7fc053400640 1 -- 172.21.15.19:0/779636491 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc044018bd0 con 0x7fc05c075820 2024-03-18T15:14:24.934 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:24.931+0000 7fc05b200640 1 -- 172.21.15.19:0/779636491 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc05c1a6360 con 0x7fc05c075820 2024-03-18T15:14:24.934 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:24.932+0000 7fc05b200640 1 -- 172.21.15.19:0/779636491 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc05c103d80 con 0x7fc05c075820 2024-03-18T15:14:24.935 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:24.934+0000 7fc053400640 1 -- 172.21.15.19:0/779636491 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fc0440107d0 con 0x7fc05c075820 2024-03-18T15:14:24.936 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:24.936+0000 7fc053400640 1 --2- 172.21.15.19:0/779636491 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fc0400a2880 0x7fc0400a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:14:24.936 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:24.936+0000 7fc053400640 1 -- 172.21.15.19:0/779636491 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fc0440ceda0 con 0x7fc05c075820 2024-03-18T15:14:24.936 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:24.936+0000 7fc059e00640 1 --2- 172.21.15.19:0/779636491 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fc0400a2880 0x7fc0400a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:14:24.937 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:24.937+0000 7fc059e00640 1 --2- 172.21.15.19:0/779636491 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fc0400a2880 0x7fc0400a4d40 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fc04c000c00 tx=0x7fc04c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:14:24.939 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:24.939+0000 7fc053400640 1 -- 172.21.15.19:0/779636491 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc04409bbf0 con 0x7fc05c075820 2024-03-18T15:14:24.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:24 smithi104 ceph-mon[36070]: from='client.24409 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi019=a;smithi019=d;smithi019=g;smithi019=j;smithi033=b;smithi033=e;smithi033=h;smithi033=k;smithi104=c;smithi104=f;smithi104=i;smithi104=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:14:24.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:24 smithi104 ceph-mon[36070]: Saving service mds.all spec with placement smithi019=a;smithi019=d;smithi019=g;smithi019=j;smithi033=b;smithi033=e;smithi033=h;smithi033=k;smithi104=c;smithi104=f;smithi104=i;smithi104=l;count:12 2024-03-18T15:14:24.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:24 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:24.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:24 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:14:24.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:24 smithi019 ceph-mon[31522]: from='client.24409 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi019=a;smithi019=d;smithi019=g;smithi019=j;smithi033=b;smithi033=e;smithi033=h;smithi033=k;smithi104=c;smithi104=f;smithi104=i;smithi104=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:14:24.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:24 smithi019 ceph-mon[31522]: Saving service mds.all spec with placement smithi019=a;smithi019=d;smithi019=g;smithi019=j;smithi033=b;smithi033=e;smithi033=h;smithi033=k;smithi104=c;smithi104=f;smithi104=i;smithi104=l;count:12 2024-03-18T15:14:24.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:24 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:24.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:24 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:14:25.146 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.146+0000 7fc05b200640 1 -- 172.21.15.19:0/779636491 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc05c062480 con 0x7fc05c075820 2024-03-18T15:14:25.148 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.148+0000 7fc053400640 1 -- 172.21.15.19:0/779636491 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18421 (secure 0 0 0) 0x7fc04409fc90 con 0x7fc05c075820 2024-03-18T15:14:25.149 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-18T15:14:25.149 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":81,"fsid":"d4dc2ba6-e538-11ee-95c9-87774f69a715","created":"2024-03-18T15:05:57.538551+0000","modified":"2024-03-18T15:14:13.025582+0000","last_up_change":"2024-03-18T15:14:12.009522+0000","last_in_change":"2024-03-18T15:13:52.555387+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T15:09:56.843349+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"37b5a5cd-27ae-4e77-9b05-774b817b2ee9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6802","nonce":402333948},{"type":"v1","addr":"172.21.15.19:6803","nonce":402333948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6804","nonce":402333948},{"type":"v1","addr":"172.21.15.19:6805","nonce":402333948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6808","nonce":402333948},{"type":"v1","addr":"172.21.15.19:6809","nonce":402333948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6806","nonce":402333948},{"type":"v1","addr":"172.21.15.19:6807","nonce":402333948}]},"public_addr":"172.21.15.19:6803/402333948","cluster_addr":"172.21.15.19:6805/402333948","heartbeat_back_addr":"172.21.15.19:6809/402333948","heartbeat_front_addr":"172.21.15.19:6807/402333948","state":["exists","up"]},{"osd":1,"uuid":"3a1134e8-ede7-4dbd-9be1-eefd2cbb43bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1951880113},{"type":"v1","addr":"172.21.15.33:6801","nonce":1951880113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1951880113},{"type":"v1","addr":"172.21.15.33:6803","nonce":1951880113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1951880113},{"type":"v1","addr":"172.21.15.33:6807","nonce":1951880113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1951880113},{"type":"v1","addr":"172.21.15.33:6805","nonce":1951880113}]},"public_addr":"172.21.15.33:6801/1951880113","cluster_addr":"172.21.15.33:6803/1951880113","heartbeat_back_addr":"172.21.15.33:6807/1951880113","heartbeat_front_addr":"172.21.15.33:6805/1951880113","state":["exists","up"]},{"osd":2,"uuid":"f383a164-a3a0-49fd-a9e8-9c2178d8cfb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":1004727949},{"type":"v1","addr":"172.21.15.104:6801","nonce":1004727949}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":1004727949},{"type":"v1","addr":"172.21.15.104:6803","nonce":1004727949}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":1004727949},{"type":"v1","addr":"172.21.15.104:6807","nonce":1004727949}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":1004727949},{"type":"v1","addr":"172.21.15.104:6805","nonce":1004727949}]},"public_addr":"172.21.15.104:6801/1004727949","cluster_addr":"172.21.15.104:6803/1004727949","heartbeat_back_addr":"172.21.15.104:6807/1004727949","heartbeat_front_addr":"172.21.15.104:6805/1004727949","state":["exists","up"]},{"osd":3,"uuid":"c409e116-28bf-45dd-a791-539af22abe1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6810","nonce":3571525854},{"type":"v1","addr":"172.21.15.19:6811","nonce":3571525854}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6812","nonce":3571525854},{"type":"v1","addr":"172.21.15.19:6813","nonce":3571525854}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6816","nonce":3571525854},{"type":"v1","addr":"172.21.15.19:6817","nonce":3571525854}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6814","nonce":3571525854},{"type":"v1","addr":"172.21.15.19:6815","nonce":3571525854}]},"public_addr":"172.21.15.19:6811/3571525854","cluster_addr":"172.21.15.19:6813/3571525854","heartbeat_back_addr":"172.21.15.19:6817/3571525854","heartbeat_front_addr":"172.21.15.19:6815/3571525854","state":["exists","up"]},{"osd":4,"uuid":"696eb719-0b3e-43ae-b5f1-ccd73a3b2843","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":624901749},{"type":"v1","addr":"172.21.15.33:6809","nonce":624901749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":624901749},{"type":"v1","addr":"172.21.15.33:6811","nonce":624901749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":624901749},{"type":"v1","addr":"172.21.15.33:6815","nonce":624901749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":624901749},{"type":"v1","addr":"172.21.15.33:6813","nonce":624901749}]},"public_addr":"172.21.15.33:6809/624901749","cluster_addr":"172.21.15.33:6811/624901749","heartbeat_back_addr":"172.21.15.33:6815/624901749","heartbeat_front_addr":"172.21.15.33:6813/624901749","state":["exists","up"]},{"osd":5,"uuid":"21789779-8bb9-42c7-b9ee-437985f11e3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":1258208297},{"type":"v1","addr":"172.21.15.104:6809","nonce":1258208297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":1258208297},{"type":"v1","addr":"172.21.15.104:6811","nonce":1258208297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":1258208297},{"type":"v1","addr":"172.21.15.104:6815","nonce":1258208297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":1258208297},{"type":"v1","addr":"172.21.15.104:6813","nonce":1258208297}]},"public_addr":"172.21.15.104:6809/1258208297","cluster_addr":"172.21.15.104:6811/1258208297","heartbeat_back_addr":"172.21.15.104:6815/1258208297","heartbeat_front_addr":"172.21.15.104:6813/1258208297","state":["exists","up"]},{"osd":6,"uuid":"0e92a2ef-43cd-487c-9203-e219158aef8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6818","nonce":2782482301},{"type":"v1","addr":"172.21.15.19:6819","nonce":2782482301}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6820","nonce":2782482301},{"type":"v1","addr":"172.21.15.19:6821","nonce":2782482301}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6824","nonce":2782482301},{"type":"v1","addr":"172.21.15.19:6825","nonce":2782482301}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6822","nonce":2782482301},{"type":"v1","addr":"172.21.15.19:6823","nonce":2782482301}]},"public_addr":"172.21.15.19:6819/2782482301","cluster_addr":"172.21.15.19:6821/2782482301","heartbeat_back_addr":"172.21.15.19:6825/2782482301","heartbeat_front_addr":"172.21.15.19:6823/2782482301","state":["exists","up"]},{"osd":7,"uuid":"a16e39a1-3685-491d-b376-3e80a2084072","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":4238160988},{"type":"v1","addr":"172.21.15.33:6817","nonce":4238160988}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":4238160988},{"type":"v1","addr":"172.21.15.33:6819","nonce":4238160988}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":4238160988},{"type":"v1","addr":"172.21.15.33:6823","nonce":4238160988}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":4238160988},{"type":"v1","addr":"172.21.15.33:6821","nonce":4238160988}]},"public_addr":"172.21.15.33:6817/4238160988","cluster_addr":"172.21.15.33:6819/4238160988","heartbeat_back_addr":"172.21.15.33:6823/4238160988","heartbeat_front_addr":"172.21.15.33:6821/4238160988","state":["exists","up"]},{"osd":8,"uuid":"d5c3fa92-cd5d-4349-9c7d-d5666bacde16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":1014779813},{"type":"v1","addr":"172.21.15.104:6817","nonce":1014779813}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":1014779813},{"type":"v1","addr":"172.21.15.104:6819","nonce":1014779813}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":1014779813},{"type":"v1","addr":"172.21.15.104:6823","nonce":1014779813}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":1014779813},{"type":"v1","addr":"172.21.15.104:6821","nonce":1014779813}]},"public_addr":"172.21.15.104:6817/1014779813","cluster_addr":"172.21.15.104:6819/1014779813","heartbeat_back_addr":"172.21.15.104:6823/1014779813","heartbeat_front_addr":"172.21.15.104:6821/1014779813","state":["exists","up"]},{"osd":9,"uuid":"eed2e9b2-26d3-4c46-88ec-34b5596fe991","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6826","nonce":2641151009},{"type":"v1","addr":"172.21.15.19:6827","nonce":2641151009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6828","nonce":2641151009},{"type":"v1","addr":"172.21.15.19:6829","nonce":2641151009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6832","nonce":2641151009},{"type":"v1","addr":"172.21.15.19:6833","nonce":2641151009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6830","nonce":2641151009},{"type":"v1","addr":"172.21.15.19:6831","nonce":2641151009}]},"public_addr":"172.21.15.19:6827/2641151009","cluster_addr":"172.21.15.19:6829/2641151009","heartbeat_back_addr":"172.21.15.19:6833/2641151009","heartbeat_front_addr":"172.21.15.19:6831/2641151009","state":["exists","up"]},{"osd":10,"uuid":"f3c35ea5-fbe7-4dc4-93c5-b92d1a714db3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":1346058869},{"type":"v1","addr":"172.21.15.33:6825","nonce":1346058869}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":1346058869},{"type":"v1","addr":"172.21.15.33:6827","nonce":1346058869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":1346058869},{"type":"v1","addr":"172.21.15.33:6831","nonce":1346058869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":1346058869},{"type":"v1","addr":"172.21.15.33:6829","nonce":1346058869}]},"public_addr":"172.21.15.33:6825/1346058869","cluster_addr":"172.21.15.33:6827/1346058869","heartbeat_back_addr":"172.21.15.33:6831/1346058869","heartbeat_front_addr":"172.21.15.33:6829/1346058869","state":["exists","up"]},{"osd":11,"uuid":"c8b51571-f807-4b62-90b6-7529e09e59da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":3759498281},{"type":"v1","addr":"172.21.15.104:6825","nonce":3759498281}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":3759498281},{"type":"v1","addr":"172.21.15.104:6827","nonce":3759498281}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":3759498281},{"type":"v1","addr":"172.21.15.104:6831","nonce":3759498281}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":3759498281},{"type":"v1","addr":"172.21.15.104:6829","nonce":3759498281}]},"public_addr":"172.21.15.104:6825/3759498281","cluster_addr":"172.21.15.104:6827/3759498281","heartbeat_back_addr":"172.21.15.104:6831/3759498281","heartbeat_front_addr":"172.21.15.104:6829/3759498281","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:09:05.267795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:09:30.234068+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:09:53.870232+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:10:26.471296+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:10:53.180341+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:11:17.801361+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:11:51.364816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:12:18.306408+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:12:43.978744+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:13:18.213964+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:13:45.437544+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:14:10.148580+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.19:0/3860783274":"2024-03-19T15:07:14.150983+0000","172.21.15.19:0/2737223213":"2024-03-19T15:07:14.150983+0000","172.21.15.19:0/176699025":"2024-03-19T15:06:38.688405+0000","172.21.15.19:0/2316969005":"2024-03-19T15:07:14.150983+0000","172.21.15.19:6801/2776008962":"2024-03-19T15:07:14.150983+0000","172.21.15.19:0/2793980440":"2024-03-19T15:06:38.688405+0000","172.21.15.19:6800/3264337803":"2024-03-19T15:06:38.688405+0000","172.21.15.19:6801/3264337803":"2024-03-19T15:06:38.688405+0000","172.21.15.19:0/3166705295":"2024-03-19T15:06:38.688405+0000","172.21.15.19:6800/2776008962":"2024-03-19T15:07:14.150983+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T15:14:25.152 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.151+0000 7fc05b200640 1 -- 172.21.15.19:0/779636491 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fc0400a2880 msgr2=0x7fc0400a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:25.152 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.151+0000 7fc05b200640 1 --2- 172.21.15.19:0/779636491 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fc0400a2880 0x7fc0400a4d40 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fc04c000c00 tx=0x7fc04c005fd0 comp rx=0 tx=0).stop 2024-03-18T15:14:25.152 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.151+0000 7fc05b200640 1 -- 172.21.15.19:0/779636491 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc05c075820 msgr2=0x7fc05c19e480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:25.152 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.151+0000 7fc05b200640 1 --2- 172.21.15.19:0/779636491 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc05c075820 0x7fc05c19e480 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fc04400ebc0 tx=0x7fc04400c6a0 comp rx=0 tx=0).stop 2024-03-18T15:14:25.152 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.151+0000 7fc05b200640 1 -- 172.21.15.19:0/779636491 shutdown_connections 2024-03-18T15:14:25.152 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.151+0000 7fc05b200640 1 --2- 172.21.15.19:0/779636491 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc05c10dbb0 0x7fc05c1a54f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:25.152 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.151+0000 7fc05b200640 1 --2- 172.21.15.19:0/779636491 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc05c075820 0x7fc05c19e480 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:25.152 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.151+0000 7fc05b200640 1 --2- 172.21.15.19:0/779636491 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fc0400a2880 0x7fc0400a4d40 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:25.152 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.151+0000 7fc05b200640 1 --2- 172.21.15.19:0/779636491 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc05c074240 0x7fc05c19df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:25.152 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.151+0000 7fc05b200640 1 -- 172.21.15.19:0/779636491 >> 172.21.15.19:0/779636491 conn(0x7fc05c0fcb70 msgr2=0x7fc05c100f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:14:25.152 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.151+0000 7fc05b200640 1 -- 172.21.15.19:0/779636491 shutdown_connections 2024-03-18T15:14:25.153 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.152+0000 7fc05b200640 1 -- 172.21.15.19:0/779636491 wait complete. 2024-03-18T15:14:25.171 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T15:09:56.843349+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T15:14:25.171 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T15:14:25.399 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.399+0000 7ff439000640 1 -- 172.21.15.19:0/61686255 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff4340761a0 msgr2=0x7ff4340765c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:25.399 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.399+0000 7ff439000640 1 --2- 172.21.15.19:0/61686255 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff4340761a0 0x7ff4340765c0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff42c009ca0 tx=0x7ff42c02c0f0 comp rx=0 tx=0).stop 2024-03-18T15:14:25.399 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.399+0000 7ff439000640 1 -- 172.21.15.19:0/61686255 shutdown_connections 2024-03-18T15:14:25.399 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.399+0000 7ff439000640 1 --2- 172.21.15.19:0/61686255 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff434077d80 0x7ff434078620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:25.399 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.399+0000 7ff439000640 1 --2- 172.21.15.19:0/61686255 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff4340773e0 0x7ff434077840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:25.400 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.399+0000 7ff439000640 1 --2- 172.21.15.19:0/61686255 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff4340761a0 0x7ff4340765c0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:25.400 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.399+0000 7ff439000640 1 -- 172.21.15.19:0/61686255 >> 172.21.15.19:0/61686255 conn(0x7ff4340fee50 msgr2=0x7ff434101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:14:25.400 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.399+0000 7ff439000640 1 -- 172.21.15.19:0/61686255 shutdown_connections 2024-03-18T15:14:25.400 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.399+0000 7ff439000640 1 -- 172.21.15.19:0/61686255 wait complete. 2024-03-18T15:14:25.402 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.400+0000 7ff439000640 1 Processor -- start 2024-03-18T15:14:25.402 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.400+0000 7ff439000640 1 -- start start 2024-03-18T15:14:25.402 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.400+0000 7ff439000640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff4340761a0 0x7ff43419dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:14:25.402 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.400+0000 7ff439000640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff4340773e0 0x7ff43419e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:14:25.402 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.401+0000 7ff439000640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff434077d80 0x7ff4341a5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:14:25.402 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.401+0000 7ff439000640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7ff434113970 con 0x7ff434077d80 2024-03-18T15:14:25.402 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.401+0000 7ff439000640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff4341137f0 con 0x7ff4340761a0 2024-03-18T15:14:25.402 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.401+0000 7ff439000640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7ff434113af0 con 0x7ff4340773e0 2024-03-18T15:14:25.403 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.401+0000 7ff432a00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff4340773e0 0x7ff43419e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:14:25.403 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.401+0000 7ff433e00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff434077d80 0x7ff4341a5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:14:25.403 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.401+0000 7ff432a00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff4340773e0 0x7ff43419e4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.19:46592/0 (socket says 172.21.15.19:46592) 2024-03-18T15:14:25.403 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.401+0000 7ff432a00640 1 -- 172.21.15.19:0/108765861 learned_addr learned my addr 172.21.15.19:0/108765861 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:14:25.403 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.401+0000 7ff433400640 1 --2- 172.21.15.19:0/108765861 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff4340761a0 0x7ff43419dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:14:25.403 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.401+0000 7ff432a00640 1 -- 172.21.15.19:0/108765861 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff4340761a0 msgr2=0x7ff43419dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:25.403 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.401+0000 7ff432a00640 1 --2- 172.21.15.19:0/108765861 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff4340761a0 0x7ff43419dfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:25.403 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.401+0000 7ff432a00640 1 -- 172.21.15.19:0/108765861 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff434077d80 msgr2=0x7ff4341a5550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:25.403 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.401+0000 7ff432a00640 1 --2- 172.21.15.19:0/108765861 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff434077d80 0x7ff4341a5550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:25.403 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.401+0000 7ff432a00640 1 -- 172.21.15.19:0/108765861 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4341a5bc0 con 0x7ff4340773e0 2024-03-18T15:14:25.403 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.401+0000 7ff433400640 1 --2- 172.21.15.19:0/108765861 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff4340761a0 0x7ff43419dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:14:25.403 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.402+0000 7ff432a00640 1 --2- 172.21.15.19:0/108765861 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff4340773e0 0x7ff43419e4e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff41c00e9f0 tx=0x7ff41c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:14:25.404 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.402+0000 7ff42be00640 1 -- 172.21.15.19:0/108765861 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff41c00cd80 con 0x7ff4340773e0 2024-03-18T15:14:25.404 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.402+0000 7ff42be00640 1 -- 172.21.15.19:0/108765861 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff41c010910 con 0x7ff4340773e0 2024-03-18T15:14:25.404 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.402+0000 7ff439000640 1 -- 172.21.15.19:0/108765861 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4341a5eb0 con 0x7ff4340773e0 2024-03-18T15:14:25.404 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.402+0000 7ff42be00640 1 -- 172.21.15.19:0/108765861 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff41c018990 con 0x7ff4340773e0 2024-03-18T15:14:25.404 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.402+0000 7ff439000640 1 -- 172.21.15.19:0/108765861 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4341a6370 con 0x7ff4340773e0 2024-03-18T15:14:25.404 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.403+0000 7ff439000640 1 -- 172.21.15.19:0/108765861 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff434110250 con 0x7ff4340773e0 2024-03-18T15:14:25.405 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.405+0000 7ff42be00640 1 -- 172.21.15.19:0/108765861 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7ff41c010ac0 con 0x7ff4340773e0 2024-03-18T15:14:25.406 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.406+0000 7ff42be00640 1 --2- 172.21.15.19:0/108765861 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7ff4180a2800 0x7ff4180a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:14:25.406 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.407+0000 7ff42be00640 1 -- 172.21.15.19:0/108765861 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7ff41c0ceb00 con 0x7ff4340773e0 2024-03-18T15:14:25.407 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.407+0000 7ff433400640 1 --2- 172.21.15.19:0/108765861 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7ff4180a2800 0x7ff4180a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:14:25.407 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.407+0000 7ff433400640 1 --2- 172.21.15.19:0/108765861 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7ff4180a2800 0x7ff4180a4cc0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ff42c02c700 tx=0x7ff42c005ef0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:14:25.413 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.413+0000 7ff42be00640 1 -- 172.21.15.19:0/108765861 <== mon.1 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff41c09b950 con 0x7ff4340773e0 2024-03-18T15:14:25.616 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.616+0000 7ff439000640 1 -- 172.21.15.19:0/108765861 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff43402d400 con 0x7ff4340773e0 2024-03-18T15:14:25.617 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.617+0000 7ff42be00640 1 -- 172.21.15.19:0/108765861 <== mon.1 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7ff41c09f9f0 con 0x7ff4340773e0 2024-03-18T15:14:25.617 INFO:teuthology.orchestra.run.smithi019.stdout:pg_num: 1 2024-03-18T15:14:25.620 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.619+0000 7ff439000640 1 -- 172.21.15.19:0/108765861 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7ff4180a2800 msgr2=0x7ff4180a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:25.620 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.619+0000 7ff439000640 1 --2- 172.21.15.19:0/108765861 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7ff4180a2800 0x7ff4180a4cc0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ff42c02c700 tx=0x7ff42c005ef0 comp rx=0 tx=0).stop 2024-03-18T15:14:25.620 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.619+0000 7ff439000640 1 -- 172.21.15.19:0/108765861 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff4340773e0 msgr2=0x7ff43419e4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:25.620 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.619+0000 7ff439000640 1 --2- 172.21.15.19:0/108765861 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff4340773e0 0x7ff43419e4e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff41c00e9f0 tx=0x7ff41c00eeb0 comp rx=0 tx=0).stop 2024-03-18T15:14:25.620 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.619+0000 7ff439000640 1 -- 172.21.15.19:0/108765861 shutdown_connections 2024-03-18T15:14:25.620 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.619+0000 7ff439000640 1 --2- 172.21.15.19:0/108765861 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff434077d80 0x7ff4341a5550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:25.620 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.619+0000 7ff439000640 1 --2- 172.21.15.19:0/108765861 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff4340773e0 0x7ff43419e4e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:25.620 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.619+0000 7ff439000640 1 --2- 172.21.15.19:0/108765861 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7ff4180a2800 0x7ff4180a4cc0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:25.620 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.619+0000 7ff439000640 1 --2- 172.21.15.19:0/108765861 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff4340761a0 0x7ff43419dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:25.620 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.619+0000 7ff439000640 1 -- 172.21.15.19:0/108765861 >> 172.21.15.19:0/108765861 conn(0x7ff4340fee50 msgr2=0x7ff4341021b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:14:25.621 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.619+0000 7ff439000640 1 -- 172.21.15.19:0/108765861 shutdown_connections 2024-03-18T15:14:25.621 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.620+0000 7ff439000640 1 -- 172.21.15.19:0/108765861 wait complete. 2024-03-18T15:14:25.641 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T15:14:25.776 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:25 smithi104 ceph-mon[36070]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:14:25.776 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:25 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:14:25.776 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:25 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:14:25.776 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:25 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:25.776 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:25 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T15:14:25.776 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:25 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T15:14:25.776 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:25 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:14:25.776 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:25 smithi104 ceph-mon[36070]: Deploying daemon mds.l on smithi104 2024-03-18T15:14:25.776 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:25 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/779636491' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:14:25.777 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:25 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/108765861' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T15:14:25.870 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.869+0000 7f6451800640 1 -- 172.21.15.19:0/472278774 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f644c1055b0 msgr2=0x7f644c10be40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:25.870 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.869+0000 7f6451800640 1 --2- 172.21.15.19:0/472278774 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f644c1055b0 0x7f644c10be40 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f6444009ca0 tx=0x7f644402c100 comp rx=0 tx=0).stop 2024-03-18T15:14:25.870 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.870+0000 7f6451800640 1 -- 172.21.15.19:0/472278774 shutdown_connections 2024-03-18T15:14:25.870 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.870+0000 7f6451800640 1 --2- 172.21.15.19:0/472278774 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f644c1055b0 0x7f644c10be40 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:25.870 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.870+0000 7f6451800640 1 --2- 172.21.15.19:0/472278774 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f644c104bf0 0x7f644c105070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:25.870 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.870+0000 7f6451800640 1 --2- 172.21.15.19:0/472278774 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f644c103a00 0x7f644c103e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:25.870 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.870+0000 7f6451800640 1 -- 172.21.15.19:0/472278774 >> 172.21.15.19:0/472278774 conn(0x7f644c0fef90 msgr2=0x7f644c1013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:14:25.870 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.870+0000 7f6451800640 1 -- 172.21.15.19:0/472278774 shutdown_connections 2024-03-18T15:14:25.871 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.870+0000 7f6451800640 1 -- 172.21.15.19:0/472278774 wait complete. 2024-03-18T15:14:25.873 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.871+0000 7f6451800640 1 Processor -- start 2024-03-18T15:14:25.873 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.871+0000 7f6451800640 1 -- start start 2024-03-18T15:14:25.873 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.871+0000 7f6451800640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f644c103a00 0x7f644c199c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:14:25.873 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.871+0000 7f6451800640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f644c104bf0 0x7f644c19a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:14:25.873 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.871+0000 7f6451800640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f644c1055b0 0x7f644c1a1210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:14:25.873 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.871+0000 7f6451800640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f644c10f660 con 0x7f644c103a00 2024-03-18T15:14:25.873 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.871+0000 7f6451800640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f644c10f4e0 con 0x7f644c1055b0 2024-03-18T15:14:25.873 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.871+0000 7f6451800640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f644c10f7e0 con 0x7f644c104bf0 2024-03-18T15:14:25.873 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.871+0000 7f644aa00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f644c103a00 0x7f644c199c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:14:25.873 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.871+0000 7f644aa00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f644c103a00 0x7f644c199c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:42972/0 (socket says 172.21.15.19:42972) 2024-03-18T15:14:25.873 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.871+0000 7f644aa00640 1 -- 172.21.15.19:0/4138908198 learned_addr learned my addr 172.21.15.19:0/4138908198 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:14:25.873 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.871+0000 7f644b400640 1 --2- 172.21.15.19:0/4138908198 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f644c1055b0 0x7f644c1a1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:14:25.874 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.871+0000 7f6443e00640 1 --2- 172.21.15.19:0/4138908198 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f644c104bf0 0x7f644c19a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:14:25.874 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.872+0000 7f644aa00640 1 -- 172.21.15.19:0/4138908198 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f644c104bf0 msgr2=0x7f644c19a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:25.874 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.872+0000 7f644aa00640 1 --2- 172.21.15.19:0/4138908198 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f644c104bf0 0x7f644c19a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:25.874 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.872+0000 7f644aa00640 1 -- 172.21.15.19:0/4138908198 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f644c1055b0 msgr2=0x7f644c1a1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:25.874 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.872+0000 7f644aa00640 1 --2- 172.21.15.19:0/4138908198 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f644c1055b0 0x7f644c1a1210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:25.874 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.872+0000 7f644aa00640 1 -- 172.21.15.19:0/4138908198 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f644c1a18b0 con 0x7f644c103a00 2024-03-18T15:14:25.874 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.872+0000 7f6443e00640 1 --2- 172.21.15.19:0/4138908198 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f644c104bf0 0x7f644c19a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:14:25.874 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.872+0000 7f644aa00640 1 --2- 172.21.15.19:0/4138908198 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f644c103a00 0x7f644c199c70 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f643c00e9c0 tx=0x7f643c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:14:25.874 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.872+0000 7f6443400640 1 -- 172.21.15.19:0/4138908198 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f643c00cd80 con 0x7f644c103a00 2024-03-18T15:14:25.874 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.872+0000 7f6443400640 1 -- 172.21.15.19:0/4138908198 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f643c0035f0 con 0x7f644c103a00 2024-03-18T15:14:25.874 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.872+0000 7f6451800640 1 -- 172.21.15.19:0/4138908198 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f644c1a1ba0 con 0x7f644c103a00 2024-03-18T15:14:25.874 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.872+0000 7f6443400640 1 -- 172.21.15.19:0/4138908198 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f643c010690 con 0x7f644c103a00 2024-03-18T15:14:25.874 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.873+0000 7f6451800640 1 -- 172.21.15.19:0/4138908198 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f644c1a20f0 con 0x7f644c103a00 2024-03-18T15:14:25.875 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.874+0000 7f6451800640 1 -- 172.21.15.19:0/4138908198 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f644c10acc0 con 0x7f644c103a00 2024-03-18T15:14:25.875 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.876+0000 7f6443400640 1 -- 172.21.15.19:0/4138908198 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f643c0037a0 con 0x7f644c103a00 2024-03-18T15:14:25.877 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.877+0000 7f6443400640 1 --2- 172.21.15.19:0/4138908198 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f64300a2800 0x7f64300a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:14:25.877 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.877+0000 7f6443400640 1 -- 172.21.15.19:0/4138908198 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f643c0cea10 con 0x7f644c103a00 2024-03-18T15:14:25.877 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.877+0000 7f6443e00640 1 --2- 172.21.15.19:0/4138908198 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f64300a2800 0x7f64300a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:14:25.878 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.878+0000 7f6443e00640 1 --2- 172.21.15.19:0/4138908198 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f64300a2800 0x7f64300a4cc0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f64340095f0 tx=0x7f6434005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:14:25.881 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:25.881+0000 7f6443400640 1 -- 172.21.15.19:0/4138908198 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f643c09beb0 con 0x7f644c103a00 2024-03-18T15:14:25.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:25 smithi019 ceph-mon[31522]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:14:25.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:25 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:14:25.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:25 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:14:25.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:25 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:25.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:25 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T15:14:25.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:25 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T15:14:25.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:25 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:14:25.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:25 smithi019 ceph-mon[31522]: Deploying daemon mds.l on smithi104 2024-03-18T15:14:25.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:25 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/779636491' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:14:25.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:25 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/108765861' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T15:14:26.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:25 smithi033 ceph-mon[36648]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:14:26.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:25 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:14:26.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:25 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:14:26.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:25 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:26.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:25 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T15:14:26.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:25 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T15:14:26.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:25 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:14:26.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:25 smithi033 ceph-mon[36648]: Deploying daemon mds.l on smithi104 2024-03-18T15:14:26.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:25 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/779636491' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:14:26.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:25 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/108765861' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T15:14:26.126 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:26.126+0000 7f6451800640 1 -- 172.21.15.19:0/4138908198 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f644c02d1b0 con 0x7f644c103a00 2024-03-18T15:14:26.127 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:26.127+0000 7f6443400640 1 -- 172.21.15.19:0/4138908198 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+569 (secure 0 0 0) 0x7f643c09b7b0 con 0x7f644c103a00 2024-03-18T15:14:26.127 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-18T15:14:26.128 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-03-18T15:14:26.128 INFO:teuthology.orchestra.run.smithi019.stderr:dumped fsmap epoch 1 2024-03-18T15:14:26.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:26.131+0000 7f6451800640 1 -- 172.21.15.19:0/4138908198 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f64300a2800 msgr2=0x7f64300a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:26.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:26.131+0000 7f6451800640 1 --2- 172.21.15.19:0/4138908198 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f64300a2800 0x7f64300a4cc0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f64340095f0 tx=0x7f6434005f90 comp rx=0 tx=0).stop 2024-03-18T15:14:26.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:26.131+0000 7f6451800640 1 -- 172.21.15.19:0/4138908198 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f644c103a00 msgr2=0x7f644c199c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:26.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:26.131+0000 7f6451800640 1 --2- 172.21.15.19:0/4138908198 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f644c103a00 0x7f644c199c70 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f643c00e9c0 tx=0x7f643c00ee80 comp rx=0 tx=0).stop 2024-03-18T15:14:26.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:26.131+0000 7f6451800640 1 -- 172.21.15.19:0/4138908198 shutdown_connections 2024-03-18T15:14:26.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:26.131+0000 7f6451800640 1 --2- 172.21.15.19:0/4138908198 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f64300a2800 0x7f64300a4cc0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:26.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:26.131+0000 7f6451800640 1 --2- 172.21.15.19:0/4138908198 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f644c1055b0 0x7f644c1a1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:26.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:26.131+0000 7f6451800640 1 --2- 172.21.15.19:0/4138908198 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f644c104bf0 0x7f644c19a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:26.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:26.131+0000 7f6451800640 1 --2- 172.21.15.19:0/4138908198 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f644c103a00 0x7f644c199c70 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:26.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:26.131+0000 7f6451800640 1 -- 172.21.15.19:0/4138908198 >> 172.21.15.19:0/4138908198 conn(0x7f644c0fef90 msgr2=0x7f644c107e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:14:26.132 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:26.131+0000 7f6451800640 1 -- 172.21.15.19:0/4138908198 shutdown_connections 2024-03-18T15:14:26.132 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:26.131+0000 7f6451800640 1 -- 172.21.15.19:0/4138908198 wait complete. 2024-03-18T15:14:26.784 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:26 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/4138908198' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:14:26.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:26 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/4138908198' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:14:27.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:26 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/4138908198' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:14:27.177 INFO:journalctl@ceph.mds.l.smithi104.stdout:Mar 18 15:14:26 smithi104 systemd[1]: Starting Ceph mds.l for d4dc2ba6-e538-11ee-95c9-87774f69a715... 2024-03-18T15:14:27.429 INFO:journalctl@ceph.mds.l.smithi104.stdout:Mar 18 15:14:27 smithi104 podman[66284]: 2024-03-18 15:14:27.079423938 +0000 UTC m=+0.017893165 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T15:14:27.429 INFO:journalctl@ceph.mds.l.smithi104.stdout:Mar 18 15:14:27 smithi104 podman[66284]: 2024-03-18 15:14:27.19889186 +0000 UTC m=+0.137361106 container create 4c7a166e007e4dc338410efaed5df891fa433459fad2a8246ae28fc3637e052e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-l, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T15:14:27.429 INFO:journalctl@ceph.mds.l.smithi104.stdout:Mar 18 15:14:27 smithi104 podman[66284]: 2024-03-18 15:14:27.426426514 +0000 UTC m=+0.364895745 container init 4c7a166e007e4dc338410efaed5df891fa433459fad2a8246ae28fc3637e052e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-l, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD) 2024-03-18T15:14:27.429 INFO:journalctl@ceph.mds.l.smithi104.stdout:Mar 18 15:14:27 smithi104 podman[66284]: 2024-03-18 15:14:27.428958454 +0000 UTC m=+0.367427675 container start 4c7a166e007e4dc338410efaed5df891fa433459fad2a8246ae28fc3637e052e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-l, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-18T15:14:27.694 INFO:journalctl@ceph.mds.l.smithi104.stdout:Mar 18 15:14:27 smithi104 ceph-mds[66304]: -- 172.21.15.104:0/2123973725 <== mon.1 v2:172.21.15.104:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f81b887a40 con 0x55f81b85b000 2024-03-18T15:14:27.694 INFO:journalctl@ceph.mds.l.smithi104.stdout:Mar 18 15:14:27 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-l[66300]: starting mds.l at 2024-03-18T15:14:27.694 INFO:journalctl@ceph.mds.l.smithi104.stdout:Mar 18 15:14:27 smithi104 bash[66284]: 4c7a166e007e4dc338410efaed5df891fa433459fad2a8246ae28fc3637e052e 2024-03-18T15:14:27.694 INFO:journalctl@ceph.mds.l.smithi104.stdout:Mar 18 15:14:27 smithi104 systemd[1]: Started Ceph mds.l for d4dc2ba6-e538-11ee-95c9-87774f69a715. 2024-03-18T15:14:27.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:27 smithi019 ceph-mon[31522]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:14:27.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:27 smithi019 ceph-mon[31522]: mds.? [v2:172.21.15.104:6832/482725809,v1:172.21.15.104:6833/482725809] up:boot 2024-03-18T15:14:27.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:27 smithi019 ceph-mon[31522]: fsmap 1 up:standby 2024-03-18T15:14:27.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:27 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-18T15:14:27.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:27 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:27.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:27 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:27.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:27 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:27.967 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:27 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T15:14:27.967 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:27 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T15:14:27.967 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:27 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:14:28.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:27 smithi033 ceph-mon[36648]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:14:28.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:27 smithi033 ceph-mon[36648]: mds.? [v2:172.21.15.104:6832/482725809,v1:172.21.15.104:6833/482725809] up:boot 2024-03-18T15:14:28.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:27 smithi033 ceph-mon[36648]: fsmap 1 up:standby 2024-03-18T15:14:28.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:27 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-18T15:14:28.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:27 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:28.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:27 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:28.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:27 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:28.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:27 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T15:14:28.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:27 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T15:14:28.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:27 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:14:28.152 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T15:14:28.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:27 smithi104 ceph-mon[36070]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:14:28.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:27 smithi104 ceph-mon[36070]: mds.? [v2:172.21.15.104:6832/482725809,v1:172.21.15.104:6833/482725809] up:boot 2024-03-18T15:14:28.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:27 smithi104 ceph-mon[36070]: fsmap 1 up:standby 2024-03-18T15:14:28.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:27 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-18T15:14:28.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:27 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:28.196 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:27 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:28.196 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:27 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:28.196 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:27 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T15:14:28.196 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:27 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T15:14:28.196 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:27 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:14:28.385 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:28.384+0000 7f1777000640 1 -- 172.21.15.19:0/3326372832 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f17701027d0 msgr2=0x7f1770102c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:28.386 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:28.384+0000 7f1777000640 1 --2- 172.21.15.19:0/3326372832 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f17701027d0 0x7f1770102c50 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f175c009900 tx=0x7f175c02bc50 comp rx=0 tx=0).stop 2024-03-18T15:14:28.386 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:28.385+0000 7f1777000640 1 -- 172.21.15.19:0/3326372832 shutdown_connections 2024-03-18T15:14:28.386 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:28.385+0000 7f1777000640 1 --2- 172.21.15.19:0/3326372832 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1770103190 0x7f1770109a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:28.387 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:28.385+0000 7f1777000640 1 --2- 172.21.15.19:0/3326372832 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f17701027d0 0x7f1770102c50 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:28.387 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:28.385+0000 7f1777000640 1 --2- 172.21.15.19:0/3326372832 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f17701015e0 0x7f1770101a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:28.387 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:28.385+0000 7f1777000640 1 -- 172.21.15.19:0/3326372832 >> 172.21.15.19:0/3326372832 conn(0x7f17700fcb70 msgr2=0x7f17700fef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:14:28.387 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:28.385+0000 7f1777000640 1 -- 172.21.15.19:0/3326372832 shutdown_connections 2024-03-18T15:14:28.387 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:28.385+0000 7f1777000640 1 -- 172.21.15.19:0/3326372832 wait complete. 2024-03-18T15:14:28.390 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:28.386+0000 7f1777000640 1 Processor -- start 2024-03-18T15:14:28.390 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:28.386+0000 7f1777000640 1 -- start start 2024-03-18T15:14:28.391 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:28.386+0000 7f1777000640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f17701015e0 0x7f1770075590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:14:28.391 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:28.386+0000 7f1777000640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f17701027d0 0x7f1770073bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:14:28.391 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:28.386+0000 7f1777000640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1770103190 0x7f1770074110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:14:28.391 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:28.386+0000 7f1777000640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f177010d2c0 con 0x7f1770103190 2024-03-18T15:14:28.391 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:28.386+0000 7f1777000640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f177010d140 con 0x7f17701015e0 2024-03-18T15:14:28.391 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:28.386+0000 7f1777000640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f177010d440 con 0x7f17701027d0 2024-03-18T15:14:28.391 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:28.387+0000 7f1776600640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1770103190 0x7f1770074110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:14:28.391 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:28.387+0000 7f1776600640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1770103190 0x7f1770074110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:42986/0 (socket says 172.21.15.19:42986) 2024-03-18T15:14:28.391 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:28.387+0000 7f1776600640 1 -- 172.21.15.19:0/900488332 learned_addr learned my addr 172.21.15.19:0/900488332 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:14:28.391 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:28.387+0000 7f1775c00640 1 --2- 172.21.15.19:0/900488332 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f17701015e0 0x7f1770075590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:14:28.392 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:28.387+0000 7f1775200640 1 --2- 172.21.15.19:0/900488332 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f17701027d0 0x7f1770073bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:14:28.392 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:28.387+0000 7f1775c00640 1 -- 172.21.15.19:0/900488332 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f17701027d0 msgr2=0x7f1770073bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:28.392 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:28.387+0000 7f1775c00640 1 --2- 172.21.15.19:0/900488332 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f17701027d0 0x7f1770073bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:28.392 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:28.387+0000 7f1775c00640 1 -- 172.21.15.19:0/900488332 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1770103190 msgr2=0x7f1770074110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:28.392 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:28.387+0000 7f1775c00640 1 --2- 172.21.15.19:0/900488332 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1770103190 0x7f1770074110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:28.392 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:28.387+0000 7f1775c00640 1 -- 172.21.15.19:0/900488332 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17700749a0 con 0x7f17701015e0 2024-03-18T15:14:28.392 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:28.387+0000 7f1775c00640 1 --2- 172.21.15.19:0/900488332 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f17701015e0 0x7f1770075590 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f176400e9c0 tx=0x7f176400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:14:28.392 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:28.387+0000 7f176e000640 1 -- 172.21.15.19:0/900488332 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f176400ccc0 con 0x7f17701015e0 2024-03-18T15:14:28.392 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:28.387+0000 7f176e000640 1 -- 172.21.15.19:0/900488332 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f17640035f0 con 0x7f17701015e0 2024-03-18T15:14:28.393 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:28.388+0000 7f1777000640 1 -- 172.21.15.19:0/900488332 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17701a1a60 con 0x7f17701015e0 2024-03-18T15:14:28.393 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:28.388+0000 7f176e000640 1 -- 172.21.15.19:0/900488332 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f176400ce60 con 0x7f17701015e0 2024-03-18T15:14:28.393 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:28.388+0000 7f1777000640 1 -- 172.21.15.19:0/900488332 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17701a1f30 con 0x7f17701015e0 2024-03-18T15:14:28.393 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:28.389+0000 7f1777000640 1 -- 172.21.15.19:0/900488332 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1770068940 con 0x7f17701015e0 2024-03-18T15:14:28.393 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:28.390+0000 7f176e000640 1 -- 172.21.15.19:0/900488332 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f17640037a0 con 0x7f17701015e0 2024-03-18T15:14:28.394 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:28.392+0000 7f176e000640 1 --2- 172.21.15.19:0/900488332 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f17580a2800 0x7f17580a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:14:28.394 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:28.392+0000 7f176e000640 1 -- 172.21.15.19:0/900488332 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f17640cebd0 con 0x7f17701015e0 2024-03-18T15:14:28.394 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:28.392+0000 7f1775200640 1 --2- 172.21.15.19:0/900488332 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f17580a2800 0x7f17580a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:14:28.394 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:28.393+0000 7f1775200640 1 --2- 172.21.15.19:0/900488332 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f17580a2800 0x7f17580a4cc0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f175c02c260 tx=0x7f175c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:14:28.396 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:28.396+0000 7f176e000640 1 -- 172.21.15.19:0/900488332 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f176409c070 con 0x7f17701015e0 2024-03-18T15:14:28.641 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:28.641+0000 7f1777000640 1 -- 172.21.15.19:0/900488332 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1770108900 con 0x7f17701015e0 2024-03-18T15:14:28.641 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:28.642+0000 7f176e000640 1 -- 172.21.15.19:0/900488332 <== mon.2 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1393 (secure 0 0 0) 0x7f176409b970 con 0x7f17701015e0 2024-03-18T15:14:28.642 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-18T15:14:28.642 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6833/482725809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":482725809},{"type":"v1","addr":"172.21.15.104:6833","nonce":482725809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-03-18T15:14:28.642 INFO:teuthology.orchestra.run.smithi019.stderr:dumped fsmap epoch 2 2024-03-18T15:14:28.645 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:28.645+0000 7f1777000640 1 -- 172.21.15.19:0/900488332 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f17580a2800 msgr2=0x7f17580a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:28.646 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:28.645+0000 7f1777000640 1 --2- 172.21.15.19:0/900488332 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f17580a2800 0x7f17580a4cc0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f175c02c260 tx=0x7f175c000bc0 comp rx=0 tx=0).stop 2024-03-18T15:14:28.646 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:28.645+0000 7f1777000640 1 -- 172.21.15.19:0/900488332 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f17701015e0 msgr2=0x7f1770075590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:28.646 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:28.645+0000 7f1777000640 1 --2- 172.21.15.19:0/900488332 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f17701015e0 0x7f1770075590 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f176400e9c0 tx=0x7f176400ee80 comp rx=0 tx=0).stop 2024-03-18T15:14:28.646 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:28.645+0000 7f1777000640 1 -- 172.21.15.19:0/900488332 shutdown_connections 2024-03-18T15:14:28.646 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:28.645+0000 7f1777000640 1 --2- 172.21.15.19:0/900488332 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1770103190 0x7f1770074110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:28.646 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:28.645+0000 7f1777000640 1 --2- 172.21.15.19:0/900488332 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f17701027d0 0x7f1770073bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:28.646 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:28.645+0000 7f1777000640 1 --2- 172.21.15.19:0/900488332 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f17580a2800 0x7f17580a4cc0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:28.646 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:28.645+0000 7f1777000640 1 --2- 172.21.15.19:0/900488332 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f17701015e0 0x7f1770075590 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:28.646 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:28.645+0000 7f1777000640 1 -- 172.21.15.19:0/900488332 >> 172.21.15.19:0/900488332 conn(0x7f17700fcb70 msgr2=0x7f1770105ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:14:28.646 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:28.645+0000 7f1777000640 1 -- 172.21.15.19:0/900488332 shutdown_connections 2024-03-18T15:14:28.646 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:28.645+0000 7f1777000640 1 -- 172.21.15.19:0/900488332 wait complete. 2024-03-18T15:14:28.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:28 smithi104 ceph-mon[36070]: Deploying daemon mds.k on smithi033 2024-03-18T15:14:28.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:28 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/900488332' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:14:28.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:28 smithi019 ceph-mon[31522]: Deploying daemon mds.k on smithi033 2024-03-18T15:14:28.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:28 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/900488332' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:14:29.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:28 smithi033 ceph-mon[36648]: Deploying daemon mds.k on smithi033 2024-03-18T15:14:29.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:28 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/900488332' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:14:30.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:29 smithi033 ceph-mon[36648]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:14:30.007 INFO:journalctl@ceph.mds.k.smithi033.stdout:Mar 18 15:14:29 smithi033 systemd[1]: Starting Ceph mds.k for d4dc2ba6-e538-11ee-95c9-87774f69a715... 2024-03-18T15:14:30.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:29 smithi104 ceph-mon[36070]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:14:30.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:29 smithi019 ceph-mon[31522]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:14:30.415 INFO:journalctl@ceph.mds.k.smithi033.stdout:Mar 18 15:14:30 smithi033 podman[66644]: 2024-03-18 15:14:30.045943982 +0000 UTC m=+0.018134583 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T15:14:30.415 INFO:journalctl@ceph.mds.k.smithi033.stdout:Mar 18 15:14:30 smithi033 podman[66644]: 2024-03-18 15:14:30.152871934 +0000 UTC m=+0.125062522 container create 9e9cbccb620682120f0bbfec97aa1333b8797c9193aa243427cc147fa9cfb385 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-k, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD) 2024-03-18T15:14:30.415 INFO:journalctl@ceph.mds.k.smithi033.stdout:Mar 18 15:14:30 smithi033 podman[66644]: 2024-03-18 15:14:30.388725914 +0000 UTC m=+0.360916501 container init 9e9cbccb620682120f0bbfec97aa1333b8797c9193aa243427cc147fa9cfb385 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-k, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-18T15:14:30.416 INFO:journalctl@ceph.mds.k.smithi033.stdout:Mar 18 15:14:30 smithi033 podman[66644]: 2024-03-18 15:14:30.391469821 +0000 UTC m=+0.363660400 container start 9e9cbccb620682120f0bbfec97aa1333b8797c9193aa243427cc147fa9cfb385 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-k, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-03-18T15:14:30.416 INFO:journalctl@ceph.mds.k.smithi033.stdout:Mar 18 15:14:30 smithi033 ceph-mds[66663]: -- 172.21.15.33:0/257293575 <== mon.2 v2:172.21.15.33:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b66c127a40 con 0x55b66c0fac00 2024-03-18T15:14:30.668 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T15:14:30.756 INFO:journalctl@ceph.mds.k.smithi033.stdout:Mar 18 15:14:30 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-k[66659]: starting mds.k at 2024-03-18T15:14:30.756 INFO:journalctl@ceph.mds.k.smithi033.stdout:Mar 18 15:14:30 smithi033 bash[66644]: 9e9cbccb620682120f0bbfec97aa1333b8797c9193aa243427cc147fa9cfb385 2024-03-18T15:14:30.757 INFO:journalctl@ceph.mds.k.smithi033.stdout:Mar 18 15:14:30 smithi033 systemd[1]: Started Ceph mds.k for d4dc2ba6-e538-11ee-95c9-87774f69a715. 2024-03-18T15:14:30.870 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:30.869+0000 7f947f600640 1 -- 172.21.15.19:0/2778301469 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f94781029d0 msgr2=0x7f9478102e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:30.870 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:30.869+0000 7f947f600640 1 --2- 172.21.15.19:0/2778301469 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f94781029d0 0x7f9478102e50 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f9464009860 tx=0x7f946402bb70 comp rx=0 tx=0).stop 2024-03-18T15:14:30.870 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:30.869+0000 7f947f600640 1 -- 172.21.15.19:0/2778301469 shutdown_connections 2024-03-18T15:14:30.870 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:30.869+0000 7f947f600640 1 --2- 172.21.15.19:0/2778301469 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9478103390 0x7f9478109c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:30.870 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:30.869+0000 7f947f600640 1 --2- 172.21.15.19:0/2778301469 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f94781029d0 0x7f9478102e50 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:30.870 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:30.869+0000 7f947f600640 1 --2- 172.21.15.19:0/2778301469 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f94781017e0 0x7f9478101c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:30.870 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:30.869+0000 7f947f600640 1 -- 172.21.15.19:0/2778301469 >> 172.21.15.19:0/2778301469 conn(0x7f94780fcd50 msgr2=0x7f94780ff190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:14:30.871 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:30.870+0000 7f947f600640 1 -- 172.21.15.19:0/2778301469 shutdown_connections 2024-03-18T15:14:30.871 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:30.870+0000 7f947f600640 1 -- 172.21.15.19:0/2778301469 wait complete. 2024-03-18T15:14:30.873 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:30.871+0000 7f947f600640 1 Processor -- start 2024-03-18T15:14:30.873 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:30.871+0000 7f947f600640 1 -- start start 2024-03-18T15:14:30.873 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:30.871+0000 7f947f600640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f94781017e0 0x7f9478199d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:14:30.873 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:30.871+0000 7f947f600640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f94781029d0 0x7f947819a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:14:30.873 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:30.871+0000 7f947f600640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9478103390 0x7f94781a1300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:14:30.873 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:30.871+0000 7f947f600640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f947810d440 con 0x7f94781029d0 2024-03-18T15:14:30.873 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:30.871+0000 7f947f600640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f947810d2c0 con 0x7f9478103390 2024-03-18T15:14:30.873 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:30.871+0000 7f947f600640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f947810d5c0 con 0x7f94781017e0 2024-03-18T15:14:30.873 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:30.871+0000 7f947cc00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f94781017e0 0x7f9478199d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:14:30.874 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:30.871+0000 7f9477e00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f94781029d0 0x7f947819a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:14:30.874 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:30.871+0000 7f947d600640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9478103390 0x7f94781a1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:14:30.874 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:30.871+0000 7f9477e00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f94781029d0 0x7f947819a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:43004/0 (socket says 172.21.15.19:43004) 2024-03-18T15:14:30.874 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:30.871+0000 7f9477e00640 1 -- 172.21.15.19:0/2849754999 learned_addr learned my addr 172.21.15.19:0/2849754999 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:14:30.874 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:30.871+0000 7f947cc00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f94781017e0 0x7f9478199d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.19:46638/0 (socket says 172.21.15.19:46638) 2024-03-18T15:14:30.874 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:30.872+0000 7f947d600640 1 -- 172.21.15.19:0/2849754999 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f94781017e0 msgr2=0x7f9478199d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:30.874 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:30.872+0000 7f947d600640 1 --2- 172.21.15.19:0/2849754999 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f94781017e0 0x7f9478199d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:30.874 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:30.872+0000 7f947d600640 1 -- 172.21.15.19:0/2849754999 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f94781029d0 msgr2=0x7f947819a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:30.874 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:30.872+0000 7f947d600640 1 --2- 172.21.15.19:0/2849754999 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f94781029d0 0x7f947819a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:30.874 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:30.872+0000 7f947d600640 1 -- 172.21.15.19:0/2849754999 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94781a1970 con 0x7f9478103390 2024-03-18T15:14:30.874 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:30.872+0000 7f947cc00640 1 --2- 172.21.15.19:0/2849754999 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f94781017e0 0x7f9478199d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:14:30.874 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:30.872+0000 7f947d600640 1 --2- 172.21.15.19:0/2849754999 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9478103390 0x7f94781a1300 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9470009fd0 tx=0x7f947000ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:14:30.875 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:30.872+0000 7f9475600640 1 -- 172.21.15.19:0/2849754999 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f947000cd60 con 0x7f9478103390 2024-03-18T15:14:30.875 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:30.872+0000 7f947f600640 1 -- 172.21.15.19:0/2849754999 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94781a1c60 con 0x7f9478103390 2024-03-18T15:14:30.875 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:30.873+0000 7f9475600640 1 -- 172.21.15.19:0/2849754999 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9470005020 con 0x7f9478103390 2024-03-18T15:14:30.875 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:30.873+0000 7f9475600640 1 -- 172.21.15.19:0/2849754999 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9470010690 con 0x7f9478103390 2024-03-18T15:14:30.875 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:30.873+0000 7f947f600640 1 -- 172.21.15.19:0/2849754999 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94781a21b0 con 0x7f9478103390 2024-03-18T15:14:30.875 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:30.874+0000 7f947f600640 1 -- 172.21.15.19:0/2849754999 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94781a23e0 con 0x7f9478103390 2024-03-18T15:14:30.875 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:30.876+0000 7f9475600640 1 -- 172.21.15.19:0/2849754999 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f9470005540 con 0x7f9478103390 2024-03-18T15:14:30.877 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:30.877+0000 7f9475600640 1 --2- 172.21.15.19:0/2849754999 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f94600a2800 0x7f94600a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:14:30.877 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:30.877+0000 7f9475600640 1 -- 172.21.15.19:0/2849754999 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f94700cecb0 con 0x7f9478103390 2024-03-18T15:14:30.877 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:30.877+0000 7f947cc00640 1 --2- 172.21.15.19:0/2849754999 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f94600a2800 0x7f94600a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:14:30.878 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:30.878+0000 7f947cc00640 1 --2- 172.21.15.19:0/2849754999 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f94600a2800 0x7f94600a4cc0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f9478102830 tx=0x7f946c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:14:30.881 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:30.881+0000 7f9475600640 1 -- 172.21.15.19:0/2849754999 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f947009c150 con 0x7f9478103390 2024-03-18T15:14:31.127 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:31.127+0000 7f947f600640 1 -- 172.21.15.19:0/2849754999 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f94781a25b0 con 0x7f9478103390 2024-03-18T15:14:31.127 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:31.128+0000 7f9475600640 1 -- 172.21.15.19:0/2849754999 <== mon.2 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2218 (secure 0 0 0) 0x7f947009ba50 con 0x7f9478103390 2024-03-18T15:14:31.128 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-18T15:14:31.128 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6833/482725809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":482725809},{"type":"v1","addr":"172.21.15.104:6833","nonce":482725809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6833/2272279159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":2272279159},{"type":"v1","addr":"172.21.15.33:6833","nonce":2272279159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-03-18T15:14:31.128 INFO:teuthology.orchestra.run.smithi019.stderr:dumped fsmap epoch 3 2024-03-18T15:14:31.132 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:31.132+0000 7f947f600640 1 -- 172.21.15.19:0/2849754999 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f94600a2800 msgr2=0x7f94600a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:31.132 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:31.132+0000 7f947f600640 1 --2- 172.21.15.19:0/2849754999 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f94600a2800 0x7f94600a4cc0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f9478102830 tx=0x7f946c009260 comp rx=0 tx=0).stop 2024-03-18T15:14:31.132 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:31.132+0000 7f947f600640 1 -- 172.21.15.19:0/2849754999 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9478103390 msgr2=0x7f94781a1300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:31.132 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:31.132+0000 7f947f600640 1 --2- 172.21.15.19:0/2849754999 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9478103390 0x7f94781a1300 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9470009fd0 tx=0x7f947000ee50 comp rx=0 tx=0).stop 2024-03-18T15:14:31.132 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:31.132+0000 7f947f600640 1 -- 172.21.15.19:0/2849754999 shutdown_connections 2024-03-18T15:14:31.132 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:31.132+0000 7f947f600640 1 --2- 172.21.15.19:0/2849754999 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f94600a2800 0x7f94600a4cc0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:31.132 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:31.132+0000 7f947f600640 1 --2- 172.21.15.19:0/2849754999 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9478103390 0x7f94781a1300 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:31.132 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:31.132+0000 7f947f600640 1 --2- 172.21.15.19:0/2849754999 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f94781029d0 0x7f947819a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:31.132 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:31.132+0000 7f947f600640 1 --2- 172.21.15.19:0/2849754999 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f94781017e0 0x7f9478199d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:31.133 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:31.132+0000 7f947f600640 1 -- 172.21.15.19:0/2849754999 >> 172.21.15.19:0/2849754999 conn(0x7f94780fcd50 msgr2=0x7f9478073a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:14:31.133 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:31.132+0000 7f947f600640 1 -- 172.21.15.19:0/2849754999 shutdown_connections 2024-03-18T15:14:31.133 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:31.132+0000 7f947f600640 1 -- 172.21.15.19:0/2849754999 wait complete. 2024-03-18T15:14:31.798 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:31 smithi019 ceph-mon[31522]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:14:31.798 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:31 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:31.798 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:31 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:31.798 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:31 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:31.798 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:31 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T15:14:31.798 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:31 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T15:14:31.799 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:31 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:14:31.799 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:31 smithi019 ceph-mon[31522]: Deploying daemon mds.j on smithi019 2024-03-18T15:14:31.799 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:31 smithi019 ceph-mon[31522]: mds.? [v2:172.21.15.33:6832/2272279159,v1:172.21.15.33:6833/2272279159] up:boot 2024-03-18T15:14:31.799 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:31 smithi019 ceph-mon[31522]: fsmap 2 up:standby 2024-03-18T15:14:31.799 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:31 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-18T15:14:31.799 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:31 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/2849754999' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:14:31.813 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:31 smithi033 ceph-mon[36648]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:14:31.813 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:31 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:31.813 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:31 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:31.813 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:31 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:31.813 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:31 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T15:14:31.813 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:31 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T15:14:31.814 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:31 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:14:31.814 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:31 smithi033 ceph-mon[36648]: Deploying daemon mds.j on smithi019 2024-03-18T15:14:31.814 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:31 smithi033 ceph-mon[36648]: mds.? [v2:172.21.15.33:6832/2272279159,v1:172.21.15.33:6833/2272279159] up:boot 2024-03-18T15:14:31.814 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:31 smithi033 ceph-mon[36648]: fsmap 2 up:standby 2024-03-18T15:14:31.814 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:31 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-18T15:14:31.814 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:31 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/2849754999' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:14:31.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:31 smithi104 ceph-mon[36070]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:14:31.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:31 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:31.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:31 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:31.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:31 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:31.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:31 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T15:14:31.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:31 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T15:14:31.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:31 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:14:31.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:31 smithi104 ceph-mon[36070]: Deploying daemon mds.j on smithi019 2024-03-18T15:14:31.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:31 smithi104 ceph-mon[36070]: mds.? [v2:172.21.15.33:6832/2272279159,v1:172.21.15.33:6833/2272279159] up:boot 2024-03-18T15:14:31.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:31 smithi104 ceph-mon[36070]: fsmap 2 up:standby 2024-03-18T15:14:31.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:31 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-18T15:14:31.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:31 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/2849754999' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:14:33.160 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T15:14:33.500 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:33.499+0000 7f987a800640 1 -- 172.21.15.19:0/452516752 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f98740773e0 msgr2=0x7f9874077840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:33.500 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:33.499+0000 7f987a800640 1 --2- 172.21.15.19:0/452516752 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f98740773e0 0x7f9874077840 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f9860009900 tx=0x7f986002bc40 comp rx=0 tx=0).stop 2024-03-18T15:14:33.500 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:33.499+0000 7f987a800640 1 -- 172.21.15.19:0/452516752 shutdown_connections 2024-03-18T15:14:33.500 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:33.499+0000 7f987a800640 1 --2- 172.21.15.19:0/452516752 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9874077d80 0x7f9874078620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:33.500 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:33.499+0000 7f987a800640 1 --2- 172.21.15.19:0/452516752 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f98740773e0 0x7f9874077840 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:33.500 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:33.499+0000 7f987a800640 1 --2- 172.21.15.19:0/452516752 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f98740761a0 0x7f98740765c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:33.500 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:33.499+0000 7f987a800640 1 -- 172.21.15.19:0/452516752 >> 172.21.15.19:0/452516752 conn(0x7f98740fef90 msgr2=0x7f98741013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:14:33.501 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:33.500+0000 7f987a800640 1 -- 172.21.15.19:0/452516752 shutdown_connections 2024-03-18T15:14:33.501 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:33.501+0000 7f987a800640 1 -- 172.21.15.19:0/452516752 wait complete. 2024-03-18T15:14:33.502 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:33.502+0000 7f987a800640 1 Processor -- start 2024-03-18T15:14:33.502 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:33.502+0000 7f987a800640 1 -- start start 2024-03-18T15:14:33.503 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:33.503+0000 7f987a800640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f98740761a0 0x7f987419e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:14:33.503 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:33.503+0000 7f987a800640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f98740773e0 0x7f987419e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:14:33.503 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:33.503+0000 7f987a800640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9874077d80 0x7f98741a5660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:14:33.503 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:33.503+0000 7f987a800640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f9874113ad0 con 0x7f98740761a0 2024-03-18T15:14:33.503 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:33.503+0000 7f987a800640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9874113950 con 0x7f98740773e0 2024-03-18T15:14:33.503 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:33.503+0000 7f987a800640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f9874113c50 con 0x7f9874077d80 2024-03-18T15:14:33.504 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:33.503+0000 7f9872a00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f98740773e0 0x7f987419e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:14:33.504 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:33.503+0000 7f9872a00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f98740773e0 0x7f987419e5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:41684/0 (socket says 172.21.15.19:41684) 2024-03-18T15:14:33.505 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:33.503+0000 7f9872a00640 1 -- 172.21.15.19:0/1320993300 learned_addr learned my addr 172.21.15.19:0/1320993300 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:14:33.505 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:33.504+0000 7f9873e00640 1 --2- 172.21.15.19:0/1320993300 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9874077d80 0x7f98741a5660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:14:33.505 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:33.504+0000 7f9873400640 1 --2- 172.21.15.19:0/1320993300 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f98740761a0 0x7f987419e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:14:33.505 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:33.504+0000 7f9872a00640 1 -- 172.21.15.19:0/1320993300 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9874077d80 msgr2=0x7f98741a5660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:33.505 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:33.504+0000 7f9872a00640 1 --2- 172.21.15.19:0/1320993300 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9874077d80 0x7f98741a5660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:33.505 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:33.504+0000 7f9872a00640 1 -- 172.21.15.19:0/1320993300 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f98740761a0 msgr2=0x7f987419e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:33.505 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:33.504+0000 7f9872a00640 1 --2- 172.21.15.19:0/1320993300 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f98740761a0 0x7f987419e0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:33.505 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:33.504+0000 7f9872a00640 1 -- 172.21.15.19:0/1320993300 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98741a5cd0 con 0x7f98740773e0 2024-03-18T15:14:33.505 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:33.505+0000 7f9873400640 1 --2- 172.21.15.19:0/1320993300 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f98740761a0 0x7f987419e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:14:33.506 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:33.505+0000 7f9872a00640 1 --2- 172.21.15.19:0/1320993300 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f98740773e0 0x7f987419e5f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f986002c8f0 tx=0x7f986002c660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:14:33.506 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:33.505+0000 7f986be00640 1 -- 172.21.15.19:0/1320993300 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9860041070 con 0x7f98740773e0 2024-03-18T15:14:33.506 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:33.505+0000 7f986be00640 1 -- 172.21.15.19:0/1320993300 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9860004450 con 0x7f98740773e0 2024-03-18T15:14:33.506 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:33.505+0000 7f986be00640 1 -- 172.21.15.19:0/1320993300 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9860034440 con 0x7f98740773e0 2024-03-18T15:14:33.506 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:33.506+0000 7f987a800640 1 -- 172.21.15.19:0/1320993300 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98741a5f60 con 0x7f98740773e0 2024-03-18T15:14:33.507 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:33.506+0000 7f987a800640 1 -- 172.21.15.19:0/1320993300 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98741a6370 con 0x7f98740773e0 2024-03-18T15:14:33.507 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:33.508+0000 7f987a800640 1 -- 172.21.15.19:0/1320993300 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98741104c0 con 0x7f98740773e0 2024-03-18T15:14:33.510 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:33.510+0000 7f986be00640 1 -- 172.21.15.19:0/1320993300 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f9860044070 con 0x7f98740773e0 2024-03-18T15:14:33.512 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:33.512+0000 7f986be00640 1 --2- 172.21.15.19:0/1320993300 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f98580a2880 0x7f98580a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:14:33.515 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:33.512+0000 7f986be00640 1 -- 172.21.15.19:0/1320993300 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f98600f06b0 con 0x7f98740773e0 2024-03-18T15:14:33.515 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:33.513+0000 7f9873400640 1 --2- 172.21.15.19:0/1320993300 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f98580a2880 0x7f98580a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:14:33.516 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:33.513+0000 7f9873400640 1 --2- 172.21.15.19:0/1320993300 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f98580a2880 0x7f98580a4d40 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f9864009fc0 tx=0x7f9864005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:14:33.516 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:33.515+0000 7f986be00640 1 -- 172.21.15.19:0/1320993300 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f98600bdb50 con 0x7f98740773e0 2024-03-18T15:14:33.773 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:33.773+0000 7f987a800640 1 -- 172.21.15.19:0/1320993300 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f987402d2d0 con 0x7f98740773e0 2024-03-18T15:14:33.774 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:33.774+0000 7f986be00640 1 -- 172.21.15.19:0/1320993300 <== mon.2 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2218 (secure 0 0 0) 0x7f98600bd450 con 0x7f98740773e0 2024-03-18T15:14:33.774 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-18T15:14:33.774 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6833/482725809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":482725809},{"type":"v1","addr":"172.21.15.104:6833","nonce":482725809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6833/2272279159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":2272279159},{"type":"v1","addr":"172.21.15.33:6833","nonce":2272279159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-03-18T15:14:33.774 INFO:teuthology.orchestra.run.smithi019.stderr:dumped fsmap epoch 3 2024-03-18T15:14:33.777 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:33.776+0000 7f987a800640 1 -- 172.21.15.19:0/1320993300 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f98580a2880 msgr2=0x7f98580a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:33.777 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:33.776+0000 7f987a800640 1 --2- 172.21.15.19:0/1320993300 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f98580a2880 0x7f98580a4d40 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f9864009fc0 tx=0x7f9864005f90 comp rx=0 tx=0).stop 2024-03-18T15:14:33.777 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:33.776+0000 7f987a800640 1 -- 172.21.15.19:0/1320993300 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f98740773e0 msgr2=0x7f987419e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:33.777 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:33.776+0000 7f987a800640 1 --2- 172.21.15.19:0/1320993300 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f98740773e0 0x7f987419e5f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f986002c8f0 tx=0x7f986002c660 comp rx=0 tx=0).stop 2024-03-18T15:14:33.777 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:33.777+0000 7f987a800640 1 -- 172.21.15.19:0/1320993300 shutdown_connections 2024-03-18T15:14:33.777 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:33.777+0000 7f987a800640 1 --2- 172.21.15.19:0/1320993300 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9874077d80 0x7f98741a5660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:33.777 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:33.777+0000 7f987a800640 1 --2- 172.21.15.19:0/1320993300 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f98580a2880 0x7f98580a4d40 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:33.777 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:33.777+0000 7f987a800640 1 --2- 172.21.15.19:0/1320993300 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f98740773e0 0x7f987419e5f0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:33.777 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:33.777+0000 7f987a800640 1 --2- 172.21.15.19:0/1320993300 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f98740761a0 0x7f987419e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:33.777 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:33.777+0000 7f987a800640 1 -- 172.21.15.19:0/1320993300 >> 172.21.15.19:0/1320993300 conn(0x7f98740fef90 msgr2=0x7f9874100600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:14:33.777 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:33.777+0000 7f987a800640 1 -- 172.21.15.19:0/1320993300 shutdown_connections 2024-03-18T15:14:33.777 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:33.777+0000 7f987a800640 1 -- 172.21.15.19:0/1320993300 wait complete. 2024-03-18T15:14:33.800 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:33 smithi019 ceph-mon[31522]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:14:33.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:33 smithi104 ceph-mon[36070]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:14:34.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:33 smithi033 ceph-mon[36648]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:14:34.435 INFO:journalctl@ceph.mds.j.smithi019.stdout:Mar 18 15:14:34 smithi019 podman[71919]: 2024-03-18 15:14:34.146379864 +0000 UTC m=+0.160117367 container create a7249e0bbf372003304b247baf96d946ea71f04b7ff0de5863a98bd6bbbcc057 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-j, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-03-18T15:14:34.435 INFO:journalctl@ceph.mds.j.smithi019.stdout:Mar 18 15:14:34 smithi019 podman[71919]: 2024-03-18 15:14:34.433818643 +0000 UTC m=+0.447556172 container init a7249e0bbf372003304b247baf96d946ea71f04b7ff0de5863a98bd6bbbcc057 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-j, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T15:14:34.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:34 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/1320993300' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:14:34.716 INFO:journalctl@ceph.mds.j.smithi019.stdout:Mar 18 15:14:34 smithi019 podman[71919]: 2024-03-18 15:14:34.444638611 +0000 UTC m=+0.458376071 container start a7249e0bbf372003304b247baf96d946ea71f04b7ff0de5863a98bd6bbbcc057 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-j, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_CLEAN=True) 2024-03-18T15:14:34.716 INFO:journalctl@ceph.mds.j.smithi019.stdout:Mar 18 15:14:34 smithi019 ceph-mds[71937]: -- 172.21.15.19:0/4067493548 <== mon.1 v2:172.21.15.104:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5559199d3a40 con 0x5559199a7000 2024-03-18T15:14:34.716 INFO:journalctl@ceph.mds.j.smithi019.stdout:Mar 18 15:14:34 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-j[71933]: starting mds.j at 2024-03-18T15:14:34.716 INFO:journalctl@ceph.mds.j.smithi019.stdout:Mar 18 15:14:34 smithi019 bash[71919]: a7249e0bbf372003304b247baf96d946ea71f04b7ff0de5863a98bd6bbbcc057 2024-03-18T15:14:34.716 INFO:journalctl@ceph.mds.j.smithi019.stdout:Mar 18 15:14:34 smithi019 systemd[1]: Started Ceph mds.j for d4dc2ba6-e538-11ee-95c9-87774f69a715. 2024-03-18T15:14:34.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:34 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/1320993300' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:14:35.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:34 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/1320993300' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:14:35.805 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T15:14:35.816 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:35 smithi019 ceph-mon[31522]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:14:35.816 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:35 smithi019 ceph-mon[31522]: mds.? [v2:172.21.15.19:6834/1135329545,v1:172.21.15.19:6835/1135329545] up:boot 2024-03-18T15:14:35.816 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:35 smithi019 ceph-mon[31522]: fsmap 3 up:standby 2024-03-18T15:14:35.816 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:35 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-18T15:14:35.816 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:35 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:35.816 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:35 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:35.817 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:35 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:35.817 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:35 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T15:14:35.817 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:35 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T15:14:35.817 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:35 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:14:35.817 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:35 smithi019 ceph-mon[31522]: Deploying daemon mds.d on smithi019 2024-03-18T15:14:35.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:35 smithi104 ceph-mon[36070]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:14:35.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:35 smithi104 ceph-mon[36070]: mds.? [v2:172.21.15.19:6834/1135329545,v1:172.21.15.19:6835/1135329545] up:boot 2024-03-18T15:14:35.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:35 smithi104 ceph-mon[36070]: fsmap 3 up:standby 2024-03-18T15:14:35.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:35 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-18T15:14:35.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:35 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:35.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:35 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:35.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:35 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:35.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:35 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T15:14:35.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:35 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T15:14:35.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:35 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:14:35.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:35 smithi104 ceph-mon[36070]: Deploying daemon mds.d on smithi019 2024-03-18T15:14:36.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:35 smithi033 ceph-mon[36648]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:14:36.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:35 smithi033 ceph-mon[36648]: mds.? [v2:172.21.15.19:6834/1135329545,v1:172.21.15.19:6835/1135329545] up:boot 2024-03-18T15:14:36.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:35 smithi033 ceph-mon[36648]: fsmap 3 up:standby 2024-03-18T15:14:36.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:35 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-18T15:14:36.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:35 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:36.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:35 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:36.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:35 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:36.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:35 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T15:14:36.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:35 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T15:14:36.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:35 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:14:36.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:35 smithi033 ceph-mon[36648]: Deploying daemon mds.d on smithi019 2024-03-18T15:14:36.059 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:36.059+0000 7f798d200640 1 -- 172.21.15.19:0/3394566573 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7988104bf0 msgr2=0x7f7988105070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:36.060 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:36.059+0000 7f798d200640 1 --2- 172.21.15.19:0/3394566573 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7988104bf0 0x7f7988105070 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f7974009860 tx=0x7f797402bb70 comp rx=0 tx=0).stop 2024-03-18T15:14:36.060 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:36.059+0000 7f798d200640 1 -- 172.21.15.19:0/3394566573 shutdown_connections 2024-03-18T15:14:36.060 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:36.059+0000 7f798d200640 1 --2- 172.21.15.19:0/3394566573 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f79881055b0 0x7f798810be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:36.060 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:36.059+0000 7f798d200640 1 --2- 172.21.15.19:0/3394566573 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7988104bf0 0x7f7988105070 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:36.060 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:36.059+0000 7f798d200640 1 --2- 172.21.15.19:0/3394566573 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7988103a00 0x7f7988103e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:36.060 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:36.059+0000 7f798d200640 1 -- 172.21.15.19:0/3394566573 >> 172.21.15.19:0/3394566573 conn(0x7f79880fef90 msgr2=0x7f79881013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:14:36.060 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:36.059+0000 7f798d200640 1 -- 172.21.15.19:0/3394566573 shutdown_connections 2024-03-18T15:14:36.060 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:36.059+0000 7f798d200640 1 -- 172.21.15.19:0/3394566573 wait complete. 2024-03-18T15:14:36.062 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:36.060+0000 7f798d200640 1 Processor -- start 2024-03-18T15:14:36.062 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:36.061+0000 7f798d200640 1 -- start start 2024-03-18T15:14:36.062 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:36.061+0000 7f798d200640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7988103a00 0x7f7988199d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:14:36.062 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:36.061+0000 7f798d200640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7988104bf0 0x7f798819a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:14:36.062 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:36.061+0000 7f798d200640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f79881055b0 0x7f79881a1290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:14:36.062 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:36.061+0000 7f798d200640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f798810f660 con 0x7f7988104bf0 2024-03-18T15:14:36.062 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:36.061+0000 7f798d200640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f798810f4e0 con 0x7f79881055b0 2024-03-18T15:14:36.062 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:36.061+0000 7f798d200640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f798810f7e0 con 0x7f7988103a00 2024-03-18T15:14:36.063 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:36.061+0000 7f7985400640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7988104bf0 0x7f798819a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:14:36.063 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:36.061+0000 7f7985400640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7988104bf0 0x7f798819a260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:48246/0 (socket says 172.21.15.19:48246) 2024-03-18T15:14:36.063 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:36.061+0000 7f7985400640 1 -- 172.21.15.19:0/3945655070 learned_addr learned my addr 172.21.15.19:0/3945655070 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:14:36.063 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:36.061+0000 7f7985e00640 1 --2- 172.21.15.19:0/3945655070 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7988103a00 0x7f7988199d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:14:36.063 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:36.061+0000 7f7986800640 1 --2- 172.21.15.19:0/3945655070 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f79881055b0 0x7f79881a1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:14:36.063 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:36.062+0000 7f7985400640 1 -- 172.21.15.19:0/3945655070 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7988103a00 msgr2=0x7f7988199d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:36.063 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:36.062+0000 7f7985400640 1 --2- 172.21.15.19:0/3945655070 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7988103a00 0x7f7988199d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:36.063 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:36.062+0000 7f7985400640 1 -- 172.21.15.19:0/3945655070 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f79881055b0 msgr2=0x7f79881a1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:36.063 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:36.062+0000 7f7985400640 1 --2- 172.21.15.19:0/3945655070 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f79881055b0 0x7f79881a1290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:36.063 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:36.062+0000 7f7985400640 1 -- 172.21.15.19:0/3945655070 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79881a1930 con 0x7f7988104bf0 2024-03-18T15:14:36.063 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:36.062+0000 7f7985400640 1 --2- 172.21.15.19:0/3945655070 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7988104bf0 0x7f798819a260 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f79740349e0 tx=0x7f7974032450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:14:36.064 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:36.062+0000 7f797ea00640 1 -- 172.21.15.19:0/3945655070 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7974041030 con 0x7f7988104bf0 2024-03-18T15:14:36.064 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:36.062+0000 7f797ea00640 1 -- 172.21.15.19:0/3945655070 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f797402c510 con 0x7f7988104bf0 2024-03-18T15:14:36.064 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:36.062+0000 7f798d200640 1 -- 172.21.15.19:0/3945655070 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79881a1bc0 con 0x7f7988104bf0 2024-03-18T15:14:36.064 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:36.063+0000 7f797ea00640 1 -- 172.21.15.19:0/3945655070 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7974039040 con 0x7f7988104bf0 2024-03-18T15:14:36.064 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:36.063+0000 7f798d200640 1 -- 172.21.15.19:0/3945655070 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79881a2100 con 0x7f7988104bf0 2024-03-18T15:14:36.064 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:36.064+0000 7f798d200640 1 -- 172.21.15.19:0/3945655070 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f798810ad20 con 0x7f7988104bf0 2024-03-18T15:14:36.066 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:36.066+0000 7f797ea00640 1 -- 172.21.15.19:0/3945655070 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f7974044070 con 0x7f7988104bf0 2024-03-18T15:14:36.068 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:36.068+0000 7f797ea00640 1 --2- 172.21.15.19:0/3945655070 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f796c0a2880 0x7f796c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:14:36.068 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:36.069+0000 7f797ea00640 1 -- 172.21.15.19:0/3945655070 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f79740efb10 con 0x7f7988104bf0 2024-03-18T15:14:36.069 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:36.069+0000 7f7985e00640 1 --2- 172.21.15.19:0/3945655070 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f796c0a2880 0x7f796c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:14:36.069 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:36.069+0000 7f7985e00640 1 --2- 172.21.15.19:0/3945655070 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f796c0a2880 0x7f796c0a4d40 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f798819b1f0 tx=0x7f7978005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:14:36.073 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:36.073+0000 7f797ea00640 1 -- 172.21.15.19:0/3945655070 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f79740bd050 con 0x7f7988104bf0 2024-03-18T15:14:36.327 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:36.327+0000 7f798d200640 1 -- 172.21.15.19:0/3945655070 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f798802d1b0 con 0x7f7988104bf0 2024-03-18T15:14:36.329 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:36.328+0000 7f797ea00640 1 -- 172.21.15.19:0/3945655070 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3043 (secure 0 0 0) 0x7f79740bc950 con 0x7f7988104bf0 2024-03-18T15:14:36.329 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-18T15:14:36.329 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6835/1135329545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6834","nonce":1135329545},{"type":"v1","addr":"172.21.15.19:6835","nonce":1135329545}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6833/482725809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":482725809},{"type":"v1","addr":"172.21.15.104:6833","nonce":482725809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6833/2272279159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":2272279159},{"type":"v1","addr":"172.21.15.33:6833","nonce":2272279159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-03-18T15:14:36.329 INFO:teuthology.orchestra.run.smithi019.stderr:dumped fsmap epoch 4 2024-03-18T15:14:36.332 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:36.331+0000 7f798d200640 1 -- 172.21.15.19:0/3945655070 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f796c0a2880 msgr2=0x7f796c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:36.332 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:36.331+0000 7f798d200640 1 --2- 172.21.15.19:0/3945655070 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f796c0a2880 0x7f796c0a4d40 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f798819b1f0 tx=0x7f7978005f90 comp rx=0 tx=0).stop 2024-03-18T15:14:36.332 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:36.331+0000 7f798d200640 1 -- 172.21.15.19:0/3945655070 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7988104bf0 msgr2=0x7f798819a260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:36.332 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:36.331+0000 7f798d200640 1 --2- 172.21.15.19:0/3945655070 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7988104bf0 0x7f798819a260 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f79740349e0 tx=0x7f7974032450 comp rx=0 tx=0).stop 2024-03-18T15:14:36.332 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:36.331+0000 7f798d200640 1 -- 172.21.15.19:0/3945655070 shutdown_connections 2024-03-18T15:14:36.332 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:36.331+0000 7f798d200640 1 --2- 172.21.15.19:0/3945655070 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f796c0a2880 0x7f796c0a4d40 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:36.332 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:36.331+0000 7f798d200640 1 --2- 172.21.15.19:0/3945655070 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f79881055b0 0x7f79881a1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:36.332 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:36.331+0000 7f798d200640 1 --2- 172.21.15.19:0/3945655070 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7988104bf0 0x7f798819a260 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:36.332 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:36.331+0000 7f798d200640 1 --2- 172.21.15.19:0/3945655070 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7988103a00 0x7f7988199d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:36.332 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:36.331+0000 7f798d200640 1 -- 172.21.15.19:0/3945655070 >> 172.21.15.19:0/3945655070 conn(0x7f79880fef90 msgr2=0x7f7988107ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:14:36.332 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:36.332+0000 7f798d200640 1 -- 172.21.15.19:0/3945655070 shutdown_connections 2024-03-18T15:14:36.332 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:36.332+0000 7f798d200640 1 -- 172.21.15.19:0/3945655070 wait complete. 2024-03-18T15:14:36.822 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:36 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/3945655070' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:14:36.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:36 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/3945655070' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:14:37.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:36 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/3945655070' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:14:37.711 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:37 smithi019 ceph-mon[31522]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:14:37.711 INFO:journalctl@ceph.mds.d.smithi019.stdout:Mar 18 15:14:37 smithi019 systemd[1]: Starting Ceph mds.d for d4dc2ba6-e538-11ee-95c9-87774f69a715... 2024-03-18T15:14:37.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:37 smithi104 ceph-mon[36070]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:14:38.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:37 smithi033 ceph-mon[36648]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:14:38.351 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T15:14:38.412 INFO:journalctl@ceph.mds.d.smithi019.stdout:Mar 18 15:14:38 smithi019 podman[72262]: 2024-03-18 15:14:37.994885692 +0000 UTC m=+0.040640918 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T15:14:38.413 INFO:journalctl@ceph.mds.d.smithi019.stdout:Mar 18 15:14:38 smithi019 podman[72262]: 2024-03-18 15:14:38.1172845 +0000 UTC m=+0.163039736 container create 4dbf9e96e798199a089092d680debe016213431657387c3868326059b73d6b80 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-d, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-03-18T15:14:38.413 INFO:journalctl@ceph.mds.d.smithi019.stdout:Mar 18 15:14:38 smithi019 podman[72262]: 2024-03-18 15:14:38.412399527 +0000 UTC m=+0.458154737 container init 4dbf9e96e798199a089092d680debe016213431657387c3868326059b73d6b80 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-d, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-18T15:14:38.585 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:38.583+0000 7f38c5600640 1 -- 172.21.15.19:0/2228243595 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f38c0104c50 msgr2=0x7f38c0107040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:38.585 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:38.583+0000 7f38c5600640 1 --2- 172.21.15.19:0/2228243595 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f38c0104c50 0x7f38c0107040 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f38ac009900 tx=0x7f38ac02bc30 comp rx=0 tx=0).stop 2024-03-18T15:14:38.585 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:38.584+0000 7f38c5600640 1 -- 172.21.15.19:0/2228243595 shutdown_connections 2024-03-18T15:14:38.585 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:38.584+0000 7f38c5600640 1 --2- 172.21.15.19:0/2228243595 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f38c0107580 0x7f38c0109970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:38.585 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:38.584+0000 7f38c5600640 1 --2- 172.21.15.19:0/2228243595 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f38c0104c50 0x7f38c0107040 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:38.585 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:38.584+0000 7f38c5600640 1 --2- 172.21.15.19:0/2228243595 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f38c0102320 0x7f38c0104710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:38.585 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:38.584+0000 7f38c5600640 1 -- 172.21.15.19:0/2228243595 >> 172.21.15.19:0/2228243595 conn(0x7f38c00fbc70 msgr2=0x7f38c00fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:14:38.586 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:38.584+0000 7f38c5600640 1 -- 172.21.15.19:0/2228243595 shutdown_connections 2024-03-18T15:14:38.586 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:38.584+0000 7f38c5600640 1 -- 172.21.15.19:0/2228243595 wait complete. 2024-03-18T15:14:38.587 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:38.585+0000 7f38c5600640 1 Processor -- start 2024-03-18T15:14:38.587 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:38.585+0000 7f38c5600640 1 -- start start 2024-03-18T15:14:38.587 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:38.585+0000 7f38c5600640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f38c0102320 0x7f38c0197b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:14:38.587 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:38.585+0000 7f38c5600640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f38c0104c50 0x7f38c0198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:14:38.587 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:38.585+0000 7f38c5600640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f38c0107580 0x7f38c019f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:14:38.587 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:38.585+0000 7f38c5600640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f38c010d4a0 con 0x7f38c0107580 2024-03-18T15:14:38.587 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:38.585+0000 7f38c5600640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f38c010d320 con 0x7f38c0104c50 2024-03-18T15:14:38.587 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:38.585+0000 7f38c5600640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f38c010d620 con 0x7f38c0102320 2024-03-18T15:14:38.587 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:38.586+0000 7f38bea00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f38c0102320 0x7f38c0197b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:14:38.587 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:38.586+0000 7f38be000640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f38c0104c50 0x7f38c0198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:14:38.588 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:38.586+0000 7f38bea00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f38c0102320 0x7f38c0197b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.19:55642/0 (socket says 172.21.15.19:55642) 2024-03-18T15:14:38.588 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:38.586+0000 7f38bea00640 1 -- 172.21.15.19:0/2137524041 learned_addr learned my addr 172.21.15.19:0/2137524041 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:14:38.588 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:38.586+0000 7f38bf400640 1 --2- 172.21.15.19:0/2137524041 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f38c0107580 0x7f38c019f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:14:38.589 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:38.586+0000 7f38bea00640 1 -- 172.21.15.19:0/2137524041 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f38c0104c50 msgr2=0x7f38c0198080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:38.589 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:38.586+0000 7f38bea00640 1 --2- 172.21.15.19:0/2137524041 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f38c0104c50 0x7f38c0198080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:38.589 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:38.586+0000 7f38bea00640 1 -- 172.21.15.19:0/2137524041 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f38c0107580 msgr2=0x7f38c019f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:38.589 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:38.586+0000 7f38bea00640 1 --2- 172.21.15.19:0/2137524041 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f38c0107580 0x7f38c019f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:38.589 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:38.586+0000 7f38bea00640 1 -- 172.21.15.19:0/2137524041 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38c019f760 con 0x7f38c0102320 2024-03-18T15:14:38.589 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:38.587+0000 7f38bea00640 1 --2- 172.21.15.19:0/2137524041 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f38c0102320 0x7f38c0197b40 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f38b000ea60 tx=0x7f38b000ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:14:38.589 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:38.587+0000 7f38b7400640 1 -- 172.21.15.19:0/2137524041 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38b000cde0 con 0x7f38c0102320 2024-03-18T15:14:38.589 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:38.587+0000 7f38b7400640 1 -- 172.21.15.19:0/2137524041 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f38b00035f0 con 0x7f38c0102320 2024-03-18T15:14:38.589 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:38.587+0000 7f38b7400640 1 -- 172.21.15.19:0/2137524041 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38b0010670 con 0x7f38c0102320 2024-03-18T15:14:38.589 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:38.587+0000 7f38c5600640 1 -- 172.21.15.19:0/2137524041 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38c019fa50 con 0x7f38c0102320 2024-03-18T15:14:38.590 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:38.587+0000 7f38c5600640 1 -- 172.21.15.19:0/2137524041 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38c019ff70 con 0x7f38c0102320 2024-03-18T15:14:38.590 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:38.588+0000 7f38c5600640 1 -- 172.21.15.19:0/2137524041 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38c0068a90 con 0x7f38c0102320 2024-03-18T15:14:38.592 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:38.591+0000 7f38b7400640 1 -- 172.21.15.19:0/2137524041 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f38b00037a0 con 0x7f38c0102320 2024-03-18T15:14:38.593 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:38.592+0000 7f38b7400640 1 --2- 172.21.15.19:0/2137524041 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f38a40a2800 0x7f38a40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:14:38.593 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:38.592+0000 7f38b7400640 1 -- 172.21.15.19:0/2137524041 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f38b00ce950 con 0x7f38c0102320 2024-03-18T15:14:38.594 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:38.593+0000 7f38be000640 1 --2- 172.21.15.19:0/2137524041 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f38a40a2800 0x7f38a40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:14:38.594 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:38.594+0000 7f38be000640 1 --2- 172.21.15.19:0/2137524041 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f38a40a2800 0x7f38a40a4cc0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f38ac02c170 tx=0x7f38ac02df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:14:38.598 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:38.597+0000 7f38b7400640 1 -- 172.21.15.19:0/2137524041 <== mon.1 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f38b009bdf0 con 0x7f38c0102320 2024-03-18T15:14:38.716 INFO:journalctl@ceph.mds.d.smithi019.stdout:Mar 18 15:14:38 smithi019 podman[72262]: 2024-03-18 15:14:38.422443106 +0000 UTC m=+0.468198369 container start 4dbf9e96e798199a089092d680debe016213431657387c3868326059b73d6b80 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-d, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True) 2024-03-18T15:14:38.716 INFO:journalctl@ceph.mds.d.smithi019.stdout:Mar 18 15:14:38 smithi019 ceph-mds[72297]: -- 172.21.15.19:0/1777905979 <== mon.1 v2:172.21.15.104:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x556abae1da40 con 0x556abadf0c00 2024-03-18T15:14:38.716 INFO:journalctl@ceph.mds.d.smithi019.stdout:Mar 18 15:14:38 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-d[72277]: starting mds.d at 2024-03-18T15:14:38.717 INFO:journalctl@ceph.mds.d.smithi019.stdout:Mar 18 15:14:38 smithi019 bash[72262]: 4dbf9e96e798199a089092d680debe016213431657387c3868326059b73d6b80 2024-03-18T15:14:38.717 INFO:journalctl@ceph.mds.d.smithi019.stdout:Mar 18 15:14:38 smithi019 systemd[1]: Started Ceph mds.d for d4dc2ba6-e538-11ee-95c9-87774f69a715. 2024-03-18T15:14:38.861 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:38.860+0000 7f38c5600640 1 -- 172.21.15.19:0/2137524041 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f38c00fd760 con 0x7f38c0102320 2024-03-18T15:14:38.862 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:38.861+0000 7f38b7400640 1 -- 172.21.15.19:0/2137524041 <== mon.1 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3868 (secure 0 0 0) 0x7f38b009b6f0 con 0x7f38c0102320 2024-03-18T15:14:38.862 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-18T15:14:38.862 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6835/1135329545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6834","nonce":1135329545},{"type":"v1","addr":"172.21.15.19:6835","nonce":1135329545}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6833/482725809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":482725809},{"type":"v1","addr":"172.21.15.104:6833","nonce":482725809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6833/2272279159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":2272279159},{"type":"v1","addr":"172.21.15.33:6833","nonce":2272279159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24469,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6837/1778019859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6836","nonce":1778019859},{"type":"v1","addr":"172.21.15.19:6837","nonce":1778019859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[]} 2024-03-18T15:14:38.862 INFO:teuthology.orchestra.run.smithi019.stderr:dumped fsmap epoch 5 2024-03-18T15:14:38.865 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:38.863+0000 7f38c5600640 1 -- 172.21.15.19:0/2137524041 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f38a40a2800 msgr2=0x7f38a40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:38.865 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:38.863+0000 7f38c5600640 1 --2- 172.21.15.19:0/2137524041 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f38a40a2800 0x7f38a40a4cc0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f38ac02c170 tx=0x7f38ac02df90 comp rx=0 tx=0).stop 2024-03-18T15:14:38.865 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:38.863+0000 7f38c5600640 1 -- 172.21.15.19:0/2137524041 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f38c0102320 msgr2=0x7f38c0197b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:38.865 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:38.863+0000 7f38c5600640 1 --2- 172.21.15.19:0/2137524041 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f38c0102320 0x7f38c0197b40 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f38b000ea60 tx=0x7f38b000ef20 comp rx=0 tx=0).stop 2024-03-18T15:14:38.865 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:38.864+0000 7f38c5600640 1 -- 172.21.15.19:0/2137524041 shutdown_connections 2024-03-18T15:14:38.865 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:38.864+0000 7f38c5600640 1 --2- 172.21.15.19:0/2137524041 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f38c0107580 0x7f38c019f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:38.865 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:38.864+0000 7f38c5600640 1 --2- 172.21.15.19:0/2137524041 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f38a40a2800 0x7f38a40a4cc0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:38.865 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:38.864+0000 7f38c5600640 1 --2- 172.21.15.19:0/2137524041 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f38c0104c50 0x7f38c0198080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:38.865 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:38.864+0000 7f38c5600640 1 --2- 172.21.15.19:0/2137524041 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f38c0102320 0x7f38c0197b40 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:38.866 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:38.864+0000 7f38c5600640 1 -- 172.21.15.19:0/2137524041 >> 172.21.15.19:0/2137524041 conn(0x7f38c00fbc70 msgr2=0x7f38c0102c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:14:38.866 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:38.864+0000 7f38c5600640 1 -- 172.21.15.19:0/2137524041 shutdown_connections 2024-03-18T15:14:38.866 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:38.864+0000 7f38c5600640 1 -- 172.21.15.19:0/2137524041 wait complete. 2024-03-18T15:14:39.675 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:39 smithi033 ceph-mon[36648]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:14:39.675 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:39 smithi033 ceph-mon[36648]: mds.? [v2:172.21.15.19:6836/1778019859,v1:172.21.15.19:6837/1778019859] up:boot 2024-03-18T15:14:39.675 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:39 smithi033 ceph-mon[36648]: fsmap 4 up:standby 2024-03-18T15:14:39.675 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:39 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-18T15:14:39.675 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:39 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:39.676 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:39 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:39.676 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:39 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:39.676 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:39 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T15:14:39.676 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:39 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T15:14:39.676 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:39 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:14:39.676 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:39 smithi033 ceph-mon[36648]: Deploying daemon mds.e on smithi033 2024-03-18T15:14:39.676 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:39 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/2137524041' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:14:39.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:39 smithi104 ceph-mon[36070]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:14:39.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:39 smithi104 ceph-mon[36070]: mds.? [v2:172.21.15.19:6836/1778019859,v1:172.21.15.19:6837/1778019859] up:boot 2024-03-18T15:14:39.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:39 smithi104 ceph-mon[36070]: fsmap 4 up:standby 2024-03-18T15:14:39.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:39 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-18T15:14:39.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:39 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:39.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:39 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:39.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:39 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:39.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:39 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T15:14:39.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:39 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T15:14:39.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:39 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:14:39.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:39 smithi104 ceph-mon[36070]: Deploying daemon mds.e on smithi033 2024-03-18T15:14:39.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:39 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/2137524041' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:14:39.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:39 smithi019 ceph-mon[31522]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:14:39.969 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:39 smithi019 ceph-mon[31522]: mds.? [v2:172.21.15.19:6836/1778019859,v1:172.21.15.19:6837/1778019859] up:boot 2024-03-18T15:14:39.969 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:39 smithi019 ceph-mon[31522]: fsmap 4 up:standby 2024-03-18T15:14:39.969 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:39 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-18T15:14:39.970 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:39 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:39.970 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:39 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:39.970 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:39 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:39.970 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:39 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T15:14:39.970 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:39 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T15:14:39.970 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:39 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:14:39.970 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:39 smithi019 ceph-mon[31522]: Deploying daemon mds.e on smithi033 2024-03-18T15:14:39.970 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:39 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/2137524041' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:14:40.884 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T15:14:41.120 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:41.118+0000 7fa827600640 1 -- 172.21.15.19:0/877453842 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa8201055b0 msgr2=0x7fa82010be40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:41.120 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:41.118+0000 7fa827600640 1 --2- 172.21.15.19:0/877453842 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa8201055b0 0x7fa82010be40 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fa818009ca0 tx=0x7fa81802c100 comp rx=0 tx=0).stop 2024-03-18T15:14:41.120 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:41.118+0000 7fa827600640 1 -- 172.21.15.19:0/877453842 shutdown_connections 2024-03-18T15:14:41.120 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:41.118+0000 7fa827600640 1 --2- 172.21.15.19:0/877453842 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa8201055b0 0x7fa82010be40 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:41.120 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:41.118+0000 7fa827600640 1 --2- 172.21.15.19:0/877453842 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa820104bf0 0x7fa820105070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:41.120 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:41.118+0000 7fa827600640 1 --2- 172.21.15.19:0/877453842 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa820103a00 0x7fa820103e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:41.120 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:41.118+0000 7fa827600640 1 -- 172.21.15.19:0/877453842 >> 172.21.15.19:0/877453842 conn(0x7fa8200fef90 msgr2=0x7fa8201013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:14:41.120 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:41.118+0000 7fa827600640 1 -- 172.21.15.19:0/877453842 shutdown_connections 2024-03-18T15:14:41.120 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:41.118+0000 7fa827600640 1 -- 172.21.15.19:0/877453842 wait complete. 2024-03-18T15:14:41.123 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:41.119+0000 7fa827600640 1 Processor -- start 2024-03-18T15:14:41.123 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:41.119+0000 7fa827600640 1 -- start start 2024-03-18T15:14:41.123 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:41.119+0000 7fa827600640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa820103a00 0x7fa820199c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:14:41.123 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:41.120+0000 7fa827600640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa820104bf0 0x7fa82019a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:14:41.123 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:41.120+0000 7fa827600640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa8201055b0 0x7fa8201a1210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:14:41.124 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:41.120+0000 7fa827600640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fa82010f660 con 0x7fa8201055b0 2024-03-18T15:14:41.124 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:41.120+0000 7fa827600640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa82010f4e0 con 0x7fa820104bf0 2024-03-18T15:14:41.124 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:41.120+0000 7fa827600640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fa82010f7e0 con 0x7fa820103a00 2024-03-18T15:14:41.124 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:41.120+0000 7fa81fe00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa820104bf0 0x7fa82019a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:14:41.124 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:41.120+0000 7fa825600640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa8201055b0 0x7fa8201a1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:14:41.124 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:41.120+0000 7fa81fe00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa820104bf0 0x7fa82019a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:41772/0 (socket says 172.21.15.19:41772) 2024-03-18T15:14:41.124 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:41.120+0000 7fa81fe00640 1 -- 172.21.15.19:0/992240203 learned_addr learned my addr 172.21.15.19:0/992240203 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:14:41.124 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:41.120+0000 7fa824c00640 1 --2- 172.21.15.19:0/992240203 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa820103a00 0x7fa820199c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:14:41.124 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:41.120+0000 7fa81fe00640 1 -- 172.21.15.19:0/992240203 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa820103a00 msgr2=0x7fa820199c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:41.124 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:41.120+0000 7fa81fe00640 1 --2- 172.21.15.19:0/992240203 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa820103a00 0x7fa820199c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:41.125 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:41.120+0000 7fa81fe00640 1 -- 172.21.15.19:0/992240203 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa8201055b0 msgr2=0x7fa8201a1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:41.125 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:41.120+0000 7fa81fe00640 1 --2- 172.21.15.19:0/992240203 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa8201055b0 0x7fa8201a1210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:41.125 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:41.120+0000 7fa81fe00640 1 -- 172.21.15.19:0/992240203 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8201a18b0 con 0x7fa820104bf0 2024-03-18T15:14:41.125 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:41.121+0000 7fa81fe00640 1 --2- 172.21.15.19:0/992240203 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa820104bf0 0x7fa82019a1b0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa80c00e9f0 tx=0x7fa80c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:14:41.125 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:41.121+0000 7fa81d600640 1 -- 172.21.15.19:0/992240203 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa80c00cd80 con 0x7fa820104bf0 2024-03-18T15:14:41.125 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:41.121+0000 7fa827600640 1 -- 172.21.15.19:0/992240203 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8201a1ba0 con 0x7fa820104bf0 2024-03-18T15:14:41.125 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:41.121+0000 7fa81d600640 1 -- 172.21.15.19:0/992240203 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa80c010910 con 0x7fa820104bf0 2024-03-18T15:14:41.125 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:41.121+0000 7fa827600640 1 -- 172.21.15.19:0/992240203 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8201a2140 con 0x7fa820104bf0 2024-03-18T15:14:41.125 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:41.121+0000 7fa81d600640 1 -- 172.21.15.19:0/992240203 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa80c018980 con 0x7fa820104bf0 2024-03-18T15:14:41.125 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:41.122+0000 7fa827600640 1 -- 172.21.15.19:0/992240203 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa82010acc0 con 0x7fa820104bf0 2024-03-18T15:14:41.126 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:41.124+0000 7fa81d600640 1 -- 172.21.15.19:0/992240203 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fa80c010ac0 con 0x7fa820104bf0 2024-03-18T15:14:41.126 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:41.125+0000 7fa81d600640 1 --2- 172.21.15.19:0/992240203 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fa8080a2880 0x7fa8080a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:14:41.127 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:41.126+0000 7fa81d600640 1 -- 172.21.15.19:0/992240203 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fa80c0ce980 con 0x7fa820104bf0 2024-03-18T15:14:41.127 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:41.126+0000 7fa824c00640 1 --2- 172.21.15.19:0/992240203 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fa8080a2880 0x7fa8080a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:14:41.127 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:41.126+0000 7fa824c00640 1 --2- 172.21.15.19:0/992240203 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fa8080a2880 0x7fa8080a4d40 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fa82019ab40 tx=0x7fa814005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:14:41.130 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:41.129+0000 7fa81d600640 1 -- 172.21.15.19:0/992240203 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa80c09b7d0 con 0x7fa820104bf0 2024-03-18T15:14:41.185 INFO:journalctl@ceph.mds.e.smithi033.stdout:Mar 18 15:14:40 smithi033 systemd[1]: Starting Ceph mds.e for d4dc2ba6-e538-11ee-95c9-87774f69a715... 2024-03-18T15:14:41.379 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:41.378+0000 7fa827600640 1 -- 172.21.15.19:0/992240203 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa8200623e0 con 0x7fa820104bf0 2024-03-18T15:14:41.380 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:41.379+0000 7fa81d600640 1 -- 172.21.15.19:0/992240203 <== mon.2 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3868 (secure 0 0 0) 0x7fa80c09f870 con 0x7fa820104bf0 2024-03-18T15:14:41.380 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-18T15:14:41.380 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6835/1135329545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6834","nonce":1135329545},{"type":"v1","addr":"172.21.15.19:6835","nonce":1135329545}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6833/482725809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":482725809},{"type":"v1","addr":"172.21.15.104:6833","nonce":482725809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6833/2272279159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":2272279159},{"type":"v1","addr":"172.21.15.33:6833","nonce":2272279159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24469,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6837/1778019859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6836","nonce":1778019859},{"type":"v1","addr":"172.21.15.19:6837","nonce":1778019859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[]} 2024-03-18T15:14:41.381 INFO:teuthology.orchestra.run.smithi019.stderr:dumped fsmap epoch 5 2024-03-18T15:14:41.383 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:41.382+0000 7fa827600640 1 -- 172.21.15.19:0/992240203 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fa8080a2880 msgr2=0x7fa8080a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:41.384 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:41.382+0000 7fa827600640 1 --2- 172.21.15.19:0/992240203 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fa8080a2880 0x7fa8080a4d40 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fa82019ab40 tx=0x7fa814005f90 comp rx=0 tx=0).stop 2024-03-18T15:14:41.384 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:41.382+0000 7fa827600640 1 -- 172.21.15.19:0/992240203 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa820104bf0 msgr2=0x7fa82019a1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:41.384 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:41.382+0000 7fa827600640 1 --2- 172.21.15.19:0/992240203 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa820104bf0 0x7fa82019a1b0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa80c00e9f0 tx=0x7fa80c00eeb0 comp rx=0 tx=0).stop 2024-03-18T15:14:41.384 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:41.382+0000 7fa827600640 1 -- 172.21.15.19:0/992240203 shutdown_connections 2024-03-18T15:14:41.384 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:41.382+0000 7fa827600640 1 --2- 172.21.15.19:0/992240203 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa8201055b0 0x7fa8201a1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:41.384 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:41.382+0000 7fa827600640 1 --2- 172.21.15.19:0/992240203 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fa8080a2880 0x7fa8080a4d40 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:41.384 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:41.382+0000 7fa827600640 1 --2- 172.21.15.19:0/992240203 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa820104bf0 0x7fa82019a1b0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:41.384 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:41.382+0000 7fa827600640 1 --2- 172.21.15.19:0/992240203 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa820103a00 0x7fa820199c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:41.384 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:41.382+0000 7fa827600640 1 -- 172.21.15.19:0/992240203 >> 172.21.15.19:0/992240203 conn(0x7fa8200fef90 msgr2=0x7fa820107e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:14:41.384 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:41.382+0000 7fa827600640 1 -- 172.21.15.19:0/992240203 shutdown_connections 2024-03-18T15:14:41.384 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:41.383+0000 7fa827600640 1 -- 172.21.15.19:0/992240203 wait complete. 2024-03-18T15:14:41.445 INFO:journalctl@ceph.mds.e.smithi033.stdout:Mar 18 15:14:41 smithi033 podman[66969]: 2024-03-18 15:14:41.088626532 +0000 UTC m=+0.018422582 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T15:14:41.445 INFO:journalctl@ceph.mds.e.smithi033.stdout:Mar 18 15:14:41 smithi033 podman[66969]: 2024-03-18 15:14:41.199427507 +0000 UTC m=+0.129223544 container create 59af355be2aee87c099525cd3c86f53b1e7feb5cd39a2a02c180c0d12dd4c596 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-e, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5) 2024-03-18T15:14:41.445 INFO:journalctl@ceph.mds.e.smithi033.stdout:Mar 18 15:14:41 smithi033 podman[66969]: 2024-03-18 15:14:41.418750402 +0000 UTC m=+0.348546434 container init 59af355be2aee87c099525cd3c86f53b1e7feb5cd39a2a02c180c0d12dd4c596 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-e, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0) 2024-03-18T15:14:41.445 INFO:journalctl@ceph.mds.e.smithi033.stdout:Mar 18 15:14:41 smithi033 podman[66969]: 2024-03-18 15:14:41.421756051 +0000 UTC m=+0.351552091 container start 59af355be2aee87c099525cd3c86f53b1e7feb5cd39a2a02c180c0d12dd4c596 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-e, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5) 2024-03-18T15:14:41.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:41 smithi019 ceph-mon[31522]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:14:41.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:41 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/992240203' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:14:41.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:41 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:41.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:41 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:41.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:41 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:41.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:41 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T15:14:41.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:41 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T15:14:41.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:41 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:14:41.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:41 smithi033 ceph-mon[36648]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:14:41.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:41 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/992240203' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:14:41.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:41 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:41.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:41 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:41.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:41 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:41.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:41 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T15:14:41.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:41 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T15:14:41.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:41 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:14:41.757 INFO:journalctl@ceph.mds.e.smithi033.stdout:Mar 18 15:14:41 smithi033 ceph-mds[66988]: -- 172.21.15.33:0/1456086040 <== mon.2 v2:172.21.15.33:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x558fa908ba40 con 0x558fa905e800 2024-03-18T15:14:41.757 INFO:journalctl@ceph.mds.e.smithi033.stdout:Mar 18 15:14:41 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-e[66984]: starting mds.e at 2024-03-18T15:14:41.757 INFO:journalctl@ceph.mds.e.smithi033.stdout:Mar 18 15:14:41 smithi033 bash[66969]: 59af355be2aee87c099525cd3c86f53b1e7feb5cd39a2a02c180c0d12dd4c596 2024-03-18T15:14:41.758 INFO:journalctl@ceph.mds.e.smithi033.stdout:Mar 18 15:14:41 smithi033 systemd[1]: Started Ceph mds.e for d4dc2ba6-e538-11ee-95c9-87774f69a715. 2024-03-18T15:14:41.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:41 smithi104 ceph-mon[36070]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:14:41.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:41 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/992240203' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:14:41.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:41 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:41.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:41 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:41.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:41 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:41.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:41 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T15:14:41.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:41 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T15:14:41.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:41 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:14:42.915 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:42 smithi104 ceph-mon[36070]: Deploying daemon mds.f on smithi104 2024-03-18T15:14:42.915 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:42 smithi104 ceph-mon[36070]: mds.? [v2:172.21.15.33:6834/4051613151,v1:172.21.15.33:6835/4051613151] up:boot 2024-03-18T15:14:42.916 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:42 smithi104 ceph-mon[36070]: fsmap 5 up:standby 2024-03-18T15:14:42.916 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:42 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-18T15:14:42.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:42 smithi019 ceph-mon[31522]: Deploying daemon mds.f on smithi104 2024-03-18T15:14:42.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:42 smithi019 ceph-mon[31522]: mds.? [v2:172.21.15.33:6834/4051613151,v1:172.21.15.33:6835/4051613151] up:boot 2024-03-18T15:14:42.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:42 smithi019 ceph-mon[31522]: fsmap 5 up:standby 2024-03-18T15:14:42.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:42 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-18T15:14:43.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:42 smithi033 ceph-mon[36648]: Deploying daemon mds.f on smithi104 2024-03-18T15:14:43.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:42 smithi033 ceph-mon[36648]: mds.? [v2:172.21.15.33:6834/4051613151,v1:172.21.15.33:6835/4051613151] up:boot 2024-03-18T15:14:43.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:42 smithi033 ceph-mon[36648]: fsmap 5 up:standby 2024-03-18T15:14:43.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:42 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-18T15:14:43.407 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T15:14:43.640 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:43.638+0000 7f3637c00640 1 -- 172.21.15.19:0/3635358066 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3630101380 msgr2=0x7f3630101800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:43.640 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:43.638+0000 7f3637c00640 1 --2- 172.21.15.19:0/3635358066 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3630101380 0x7f3630101800 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f3628009ca0 tx=0x7f362802c100 comp rx=0 tx=0).stop 2024-03-18T15:14:43.640 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:43.638+0000 7f3637c00640 1 -- 172.21.15.19:0/3635358066 shutdown_connections 2024-03-18T15:14:43.640 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:43.638+0000 7f3637c00640 1 --2- 172.21.15.19:0/3635358066 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3630101380 0x7f3630101800 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:43.640 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:43.638+0000 7f3637c00640 1 --2- 172.21.15.19:0/3635358066 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f36301009e0 0x7f3630100e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:43.640 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:43.638+0000 7f3637c00640 1 --2- 172.21.15.19:0/3635358066 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3630100080 0x7f36301004a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:43.640 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:43.638+0000 7f3637c00640 1 -- 172.21.15.19:0/3635358066 >> 172.21.15.19:0/3635358066 conn(0x7f36300fbc40 msgr2=0x7f36300fe080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:14:43.641 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:43.639+0000 7f3637c00640 1 -- 172.21.15.19:0/3635358066 shutdown_connections 2024-03-18T15:14:43.641 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:43.639+0000 7f3637c00640 1 -- 172.21.15.19:0/3635358066 wait complete. 2024-03-18T15:14:43.643 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:43.640+0000 7f3637c00640 1 Processor -- start 2024-03-18T15:14:43.643 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:43.640+0000 7f3637c00640 1 -- start start 2024-03-18T15:14:43.643 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:43.640+0000 7f3637c00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3630100080 0x7f3630199c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:14:43.643 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:43.640+0000 7f3637c00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f36301009e0 0x7f363019a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:14:43.643 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:43.640+0000 7f3637c00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3630101380 0x7f36301a1220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:14:43.643 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:43.640+0000 7f3637c00640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f363010f640 con 0x7f3630101380 2024-03-18T15:14:43.643 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:43.640+0000 7f3637c00640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f363010f4c0 con 0x7f3630100080 2024-03-18T15:14:43.643 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:43.640+0000 7f3637c00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f363010f7c0 con 0x7f36301009e0 2024-03-18T15:14:43.644 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:43.640+0000 7f3635200640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3630100080 0x7f3630199c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:14:43.644 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:43.640+0000 7f3635200640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3630100080 0x7f3630199c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:36600/0 (socket says 172.21.15.19:36600) 2024-03-18T15:14:43.644 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:43.640+0000 7f3635200640 1 -- 172.21.15.19:0/3049776908 learned_addr learned my addr 172.21.15.19:0/3049776908 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:14:43.644 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:43.640+0000 7f362fe00640 1 --2- 172.21.15.19:0/3049776908 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f36301009e0 0x7f363019a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:14:43.644 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:43.640+0000 7f3635c00640 1 --2- 172.21.15.19:0/3049776908 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3630101380 0x7f36301a1220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:14:43.644 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:43.641+0000 7f3635200640 1 -- 172.21.15.19:0/3049776908 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f36301009e0 msgr2=0x7f363019a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:43.644 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:43.641+0000 7f3635200640 1 --2- 172.21.15.19:0/3049776908 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f36301009e0 0x7f363019a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:43.644 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:43.641+0000 7f3635200640 1 -- 172.21.15.19:0/3049776908 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3630101380 msgr2=0x7f36301a1220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:43.644 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:43.641+0000 7f3635200640 1 --2- 172.21.15.19:0/3049776908 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3630101380 0x7f36301a1220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:43.644 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:43.641+0000 7f3635200640 1 -- 172.21.15.19:0/3049776908 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36301a18c0 con 0x7f3630100080 2024-03-18T15:14:43.644 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:43.641+0000 7f362fe00640 1 --2- 172.21.15.19:0/3049776908 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f36301009e0 0x7f363019a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:14:43.644 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:43.641+0000 7f3635200640 1 --2- 172.21.15.19:0/3049776908 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3630100080 0x7f3630199c80 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f362000ea60 tx=0x7f362000ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:14:43.644 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:43.641+0000 7f362d600640 1 -- 172.21.15.19:0/3049776908 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f362000cde0 con 0x7f3630100080 2024-03-18T15:14:43.644 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:43.641+0000 7f362d600640 1 -- 172.21.15.19:0/3049776908 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f36200035f0 con 0x7f3630100080 2024-03-18T15:14:43.644 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:43.641+0000 7f362d600640 1 -- 172.21.15.19:0/3049776908 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3620010670 con 0x7f3630100080 2024-03-18T15:14:43.645 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:43.641+0000 7f3637c00640 1 -- 172.21.15.19:0/3049776908 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36301a1bb0 con 0x7f3630100080 2024-03-18T15:14:43.645 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:43.642+0000 7f3637c00640 1 -- 172.21.15.19:0/3049776908 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36301a20b0 con 0x7f3630100080 2024-03-18T15:14:43.645 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:43.643+0000 7f3637c00640 1 -- 172.21.15.19:0/3049776908 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f363010af30 con 0x7f3630100080 2024-03-18T15:14:43.645 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:43.645+0000 7f362d600640 1 -- 172.21.15.19:0/3049776908 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f36200037a0 con 0x7f3630100080 2024-03-18T15:14:43.647 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:43.646+0000 7f362d600640 1 --2- 172.21.15.19:0/3049776908 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f36140a2800 0x7f36140a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:14:43.647 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:43.646+0000 7f362d600640 1 -- 172.21.15.19:0/3049776908 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f36200ce940 con 0x7f3630100080 2024-03-18T15:14:43.647 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:43.646+0000 7f362fe00640 1 --2- 172.21.15.19:0/3049776908 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f36140a2800 0x7f36140a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:14:43.648 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:43.647+0000 7f362fe00640 1 --2- 172.21.15.19:0/3049776908 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f36140a2800 0x7f36140a4cc0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f363019b180 tx=0x7f361c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:14:43.651 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:43.650+0000 7f362d600640 1 -- 172.21.15.19:0/3049776908 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f362009bde0 con 0x7f3630100080 2024-03-18T15:14:43.896 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:43.894+0000 7f3637c00640 1 -- 172.21.15.19:0/3049776908 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f363002d1b0 con 0x7f3630100080 2024-03-18T15:14:43.896 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:43.895+0000 7f362d600640 1 -- 172.21.15.19:0/3049776908 <== mon.2 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4693 (secure 0 0 0) 0x7f362009b6e0 con 0x7f3630100080 2024-03-18T15:14:43.897 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-18T15:14:43.897 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6835/1135329545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6834","nonce":1135329545},{"type":"v1","addr":"172.21.15.19:6835","nonce":1135329545}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6833/482725809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":482725809},{"type":"v1","addr":"172.21.15.104:6833","nonce":482725809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6833/2272279159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":2272279159},{"type":"v1","addr":"172.21.15.33:6833","nonce":2272279159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24469,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6837/1778019859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6836","nonce":1778019859},{"type":"v1","addr":"172.21.15.19:6837","nonce":1778019859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24482,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/4051613151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":4051613151},{"type":"v1","addr":"172.21.15.33:6835","nonce":4051613151}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-03-18T15:14:43.897 INFO:teuthology.orchestra.run.smithi019.stderr:dumped fsmap epoch 6 2024-03-18T15:14:43.900 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:43.898+0000 7f3637c00640 1 -- 172.21.15.19:0/3049776908 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f36140a2800 msgr2=0x7f36140a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:43.900 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:43.898+0000 7f3637c00640 1 --2- 172.21.15.19:0/3049776908 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f36140a2800 0x7f36140a4cc0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f363019b180 tx=0x7f361c005f90 comp rx=0 tx=0).stop 2024-03-18T15:14:43.900 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:43.898+0000 7f3637c00640 1 -- 172.21.15.19:0/3049776908 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3630100080 msgr2=0x7f3630199c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:43.900 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:43.898+0000 7f3637c00640 1 --2- 172.21.15.19:0/3049776908 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3630100080 0x7f3630199c80 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f362000ea60 tx=0x7f362000ef20 comp rx=0 tx=0).stop 2024-03-18T15:14:43.900 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:43.898+0000 7f3637c00640 1 -- 172.21.15.19:0/3049776908 shutdown_connections 2024-03-18T15:14:43.900 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:43.898+0000 7f3637c00640 1 --2- 172.21.15.19:0/3049776908 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3630101380 0x7f36301a1220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:43.900 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:43.898+0000 7f3637c00640 1 --2- 172.21.15.19:0/3049776908 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f36301009e0 0x7f363019a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:43.900 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:43.898+0000 7f3637c00640 1 --2- 172.21.15.19:0/3049776908 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f36140a2800 0x7f36140a4cc0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:43.900 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:43.898+0000 7f3637c00640 1 --2- 172.21.15.19:0/3049776908 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3630100080 0x7f3630199c80 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:43.900 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:43.899+0000 7f3637c00640 1 -- 172.21.15.19:0/3049776908 >> 172.21.15.19:0/3049776908 conn(0x7f36300fbc40 msgr2=0x7f36300fd2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:14:43.901 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:43.899+0000 7f3637c00640 1 -- 172.21.15.19:0/3049776908 shutdown_connections 2024-03-18T15:14:43.901 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:43.899+0000 7f3637c00640 1 -- 172.21.15.19:0/3049776908 wait complete. 2024-03-18T15:14:43.917 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:43 smithi019 ceph-mon[31522]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:14:43.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:43 smithi104 ceph-mon[36070]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:14:43.945 INFO:journalctl@ceph.mds.f.smithi104.stdout:Mar 18 15:14:43 smithi104 systemd[1]: Starting Ceph mds.f for d4dc2ba6-e538-11ee-95c9-87774f69a715... 2024-03-18T15:14:44.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:43 smithi033 ceph-mon[36648]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:14:44.322 INFO:journalctl@ceph.mds.f.smithi104.stdout:Mar 18 15:14:44 smithi104 podman[66592]: 2024-03-18 15:14:43.911108796 +0000 UTC m=+0.017853366 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T15:14:44.322 INFO:journalctl@ceph.mds.f.smithi104.stdout:Mar 18 15:14:44 smithi104 podman[66592]: 2024-03-18 15:14:44.044014138 +0000 UTC m=+0.150758684 container create 19039ff9850c38be0733eff47fb0ece02c1a98e3c2d5be54be6d43fbffaa0c2c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-f, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T15:14:44.322 INFO:journalctl@ceph.mds.f.smithi104.stdout:Mar 18 15:14:44 smithi104 podman[66592]: 2024-03-18 15:14:44.321812844 +0000 UTC m=+0.428557401 container init 19039ff9850c38be0733eff47fb0ece02c1a98e3c2d5be54be6d43fbffaa0c2c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-f, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, ceph=True, org.label-schema.license=GPLv2) 2024-03-18T15:14:44.625 INFO:journalctl@ceph.mds.f.smithi104.stdout:Mar 18 15:14:44 smithi104 podman[66592]: 2024-03-18 15:14:44.32449989 +0000 UTC m=+0.431244455 container start 19039ff9850c38be0733eff47fb0ece02c1a98e3c2d5be54be6d43fbffaa0c2c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-f, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T15:14:44.626 INFO:journalctl@ceph.mds.f.smithi104.stdout:Mar 18 15:14:44 smithi104 ceph-mds[66611]: -- 172.21.15.104:0/1559369078 <== mon.1 v2:172.21.15.104:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55867184fa40 con 0x558671822c00 2024-03-18T15:14:44.626 INFO:journalctl@ceph.mds.f.smithi104.stdout:Mar 18 15:14:44 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-f[66607]: starting mds.f at 2024-03-18T15:14:44.626 INFO:journalctl@ceph.mds.f.smithi104.stdout:Mar 18 15:14:44 smithi104 bash[66592]: 19039ff9850c38be0733eff47fb0ece02c1a98e3c2d5be54be6d43fbffaa0c2c 2024-03-18T15:14:44.626 INFO:journalctl@ceph.mds.f.smithi104.stdout:Mar 18 15:14:44 smithi104 systemd[1]: Started Ceph mds.f for d4dc2ba6-e538-11ee-95c9-87774f69a715. 2024-03-18T15:14:44.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:44 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/3049776908' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:14:44.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:44 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:44.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:44 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:44.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:44 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:44.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:44 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T15:14:44.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:44 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T15:14:44.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:44 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:14:44.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:44 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/3049776908' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:14:44.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:44 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:44.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:44 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:44.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:44 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:44.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:44 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T15:14:44.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:44 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T15:14:44.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:44 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:14:45.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:44 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/3049776908' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:14:45.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:44 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:45.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:44 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:45.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:44 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:45.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:44 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T15:14:45.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:44 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T15:14:45.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:44 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:14:45.896 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:45 smithi019 ceph-mon[31522]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:14:45.897 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:45 smithi019 ceph-mon[31522]: Deploying daemon mds.c on smithi104 2024-03-18T15:14:45.897 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:45 smithi019 ceph-mon[31522]: mds.? [v2:172.21.15.104:6834/3788595957,v1:172.21.15.104:6835/3788595957] up:boot 2024-03-18T15:14:45.897 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:45 smithi019 ceph-mon[31522]: fsmap 6 up:standby 2024-03-18T15:14:45.897 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:45 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-18T15:14:45.922 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T15:14:45.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:45 smithi104 ceph-mon[36070]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:14:45.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:45 smithi104 ceph-mon[36070]: Deploying daemon mds.c on smithi104 2024-03-18T15:14:45.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:45 smithi104 ceph-mon[36070]: mds.? [v2:172.21.15.104:6834/3788595957,v1:172.21.15.104:6835/3788595957] up:boot 2024-03-18T15:14:45.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:45 smithi104 ceph-mon[36070]: fsmap 6 up:standby 2024-03-18T15:14:45.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:45 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-18T15:14:46.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:45 smithi033 ceph-mon[36648]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:14:46.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:45 smithi033 ceph-mon[36648]: Deploying daemon mds.c on smithi104 2024-03-18T15:14:46.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:45 smithi033 ceph-mon[36648]: mds.? [v2:172.21.15.104:6834/3788595957,v1:172.21.15.104:6835/3788595957] up:boot 2024-03-18T15:14:46.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:45 smithi033 ceph-mon[36648]: fsmap 6 up:standby 2024-03-18T15:14:46.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:45 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-18T15:14:46.143 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:46.141+0000 7f3e2a200640 1 -- 172.21.15.19:0/15826785 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3e241007c0 msgr2=0x7f3e24100c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:46.143 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:46.141+0000 7f3e2a200640 1 --2- 172.21.15.19:0/15826785 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3e241007c0 0x7f3e24100c20 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f3e10009860 tx=0x7f3e1002bb70 comp rx=0 tx=0).stop 2024-03-18T15:14:46.143 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:46.141+0000 7f3e2a200640 1 -- 172.21.15.19:0/15826785 shutdown_connections 2024-03-18T15:14:46.143 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:46.141+0000 7f3e2a200640 1 --2- 172.21.15.19:0/15826785 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3e24101160 0x7f3e241015e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:46.143 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:46.141+0000 7f3e2a200640 1 --2- 172.21.15.19:0/15826785 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3e241007c0 0x7f3e24100c20 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:46.143 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:46.141+0000 7f3e2a200640 1 --2- 172.21.15.19:0/15826785 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3e240ffe60 0x7f3e24100280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:46.143 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:46.141+0000 7f3e2a200640 1 -- 172.21.15.19:0/15826785 >> 172.21.15.19:0/15826785 conn(0x7f3e240fba60 msgr2=0x7f3e240fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:14:46.143 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:46.141+0000 7f3e2a200640 1 -- 172.21.15.19:0/15826785 shutdown_connections 2024-03-18T15:14:46.143 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:46.141+0000 7f3e2a200640 1 -- 172.21.15.19:0/15826785 wait complete. 2024-03-18T15:14:46.145 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:46.143+0000 7f3e2a200640 1 Processor -- start 2024-03-18T15:14:46.145 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:46.143+0000 7f3e2a200640 1 -- start start 2024-03-18T15:14:46.145 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:46.143+0000 7f3e2a200640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3e240ffe60 0x7f3e24199aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:14:46.145 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:46.143+0000 7f3e2a200640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3e241007c0 0x7f3e24199fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:14:46.145 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:46.143+0000 7f3e2a200640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3e24101160 0x7f3e241a1050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:14:46.145 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:46.143+0000 7f3e2a200640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f3e2410f430 con 0x7f3e240ffe60 2024-03-18T15:14:46.145 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:46.143+0000 7f3e2a200640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f3e2410f2b0 con 0x7f3e241007c0 2024-03-18T15:14:46.145 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:46.143+0000 7f3e2a200640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f3e2410f5b0 con 0x7f3e24101160 2024-03-18T15:14:46.146 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:46.144+0000 7f3e23e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3e241007c0 0x7f3e24199fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:14:46.146 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:46.144+0000 7f3e28e00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3e240ffe60 0x7f3e24199aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:14:46.146 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:46.144+0000 7f3e23e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3e241007c0 0x7f3e24199fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:36618/0 (socket says 172.21.15.19:36618) 2024-03-18T15:14:46.146 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:46.144+0000 7f3e23e00640 1 -- 172.21.15.19:0/335796287 learned_addr learned my addr 172.21.15.19:0/335796287 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:14:46.146 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:46.144+0000 7f3e29800640 1 --2- 172.21.15.19:0/335796287 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3e24101160 0x7f3e241a1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:14:46.146 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:46.144+0000 7f3e23e00640 1 -- 172.21.15.19:0/335796287 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3e24101160 msgr2=0x7f3e241a1050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:46.146 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:46.144+0000 7f3e23e00640 1 --2- 172.21.15.19:0/335796287 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3e24101160 0x7f3e241a1050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:46.146 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:46.144+0000 7f3e23e00640 1 -- 172.21.15.19:0/335796287 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3e240ffe60 msgr2=0x7f3e24199aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:46.146 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:46.144+0000 7f3e23e00640 1 --2- 172.21.15.19:0/335796287 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3e240ffe60 0x7f3e24199aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:46.146 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:46.144+0000 7f3e23e00640 1 -- 172.21.15.19:0/335796287 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e241a16c0 con 0x7f3e241007c0 2024-03-18T15:14:46.147 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:46.144+0000 7f3e23e00640 1 --2- 172.21.15.19:0/335796287 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3e241007c0 0x7f3e24199fe0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f3e100349e0 tx=0x7f3e10032450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:14:46.147 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:46.145+0000 7f3e21600640 1 -- 172.21.15.19:0/335796287 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e10041030 con 0x7f3e241007c0 2024-03-18T15:14:46.147 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:46.145+0000 7f3e2a200640 1 -- 172.21.15.19:0/335796287 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e241a1950 con 0x7f3e241007c0 2024-03-18T15:14:46.147 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:46.145+0000 7f3e21600640 1 -- 172.21.15.19:0/335796287 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e1002c510 con 0x7f3e241007c0 2024-03-18T15:14:46.147 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:46.145+0000 7f3e21600640 1 -- 172.21.15.19:0/335796287 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e10039040 con 0x7f3e241007c0 2024-03-18T15:14:46.147 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:46.145+0000 7f3e2a200640 1 -- 172.21.15.19:0/335796287 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e241a1e40 con 0x7f3e241007c0 2024-03-18T15:14:46.148 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:46.146+0000 7f3e2a200640 1 -- 172.21.15.19:0/335796287 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e2410ad50 con 0x7f3e241007c0 2024-03-18T15:14:46.149 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:46.148+0000 7f3e21600640 1 -- 172.21.15.19:0/335796287 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f3e10044070 con 0x7f3e241007c0 2024-03-18T15:14:46.150 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:46.149+0000 7f3e21600640 1 --2- 172.21.15.19:0/335796287 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f3e0c0a2800 0x7f3e0c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:14:46.150 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:46.149+0000 7f3e21600640 1 -- 172.21.15.19:0/335796287 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f3e100efb30 con 0x7f3e241007c0 2024-03-18T15:14:46.150 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:46.150+0000 7f3e28e00640 1 --2- 172.21.15.19:0/335796287 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f3e0c0a2800 0x7f3e0c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:14:46.151 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:46.150+0000 7f3e28e00640 1 --2- 172.21.15.19:0/335796287 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f3e0c0a2800 0x7f3e0c0a4cc0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f3e18009530 tx=0x7f3e18009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:14:46.153 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:46.152+0000 7f3e21600640 1 -- 172.21.15.19:0/335796287 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3e100bcfd0 con 0x7f3e241007c0 2024-03-18T15:14:46.400 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:46.399+0000 7f3e2a200640 1 -- 172.21.15.19:0/335796287 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3e2402d2d0 con 0x7f3e241007c0 2024-03-18T15:14:46.401 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:46.400+0000 7f3e21600640 1 -- 172.21.15.19:0/335796287 <== mon.2 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5521 (secure 0 0 0) 0x7f3e100bc8d0 con 0x7f3e241007c0 2024-03-18T15:14:46.401 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-18T15:14:46.402 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6835/1135329545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6834","nonce":1135329545},{"type":"v1","addr":"172.21.15.19:6835","nonce":1135329545}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6833/482725809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":482725809},{"type":"v1","addr":"172.21.15.104:6833","nonce":482725809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6833/2272279159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":2272279159},{"type":"v1","addr":"172.21.15.33:6833","nonce":2272279159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24469,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6837/1778019859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6836","nonce":1778019859},{"type":"v1","addr":"172.21.15.19:6837","nonce":1778019859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24482,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/4051613151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":4051613151},{"type":"v1","addr":"172.21.15.33:6835","nonce":4051613151}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24499,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3788595957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3788595957},{"type":"v1","addr":"172.21.15.104:6835","nonce":3788595957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[]} 2024-03-18T15:14:46.402 INFO:teuthology.orchestra.run.smithi019.stderr:dumped fsmap epoch 7 2024-03-18T15:14:46.404 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:46.403+0000 7f3e2a200640 1 -- 172.21.15.19:0/335796287 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f3e0c0a2800 msgr2=0x7f3e0c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:46.405 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:46.403+0000 7f3e2a200640 1 --2- 172.21.15.19:0/335796287 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f3e0c0a2800 0x7f3e0c0a4cc0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f3e18009530 tx=0x7f3e18009260 comp rx=0 tx=0).stop 2024-03-18T15:14:46.405 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:46.403+0000 7f3e2a200640 1 -- 172.21.15.19:0/335796287 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3e241007c0 msgr2=0x7f3e24199fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:46.405 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:46.403+0000 7f3e2a200640 1 --2- 172.21.15.19:0/335796287 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3e241007c0 0x7f3e24199fe0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f3e100349e0 tx=0x7f3e10032450 comp rx=0 tx=0).stop 2024-03-18T15:14:46.405 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:46.403+0000 7f3e2a200640 1 -- 172.21.15.19:0/335796287 shutdown_connections 2024-03-18T15:14:46.405 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:46.403+0000 7f3e2a200640 1 --2- 172.21.15.19:0/335796287 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3e24101160 0x7f3e241a1050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:46.405 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:46.403+0000 7f3e2a200640 1 --2- 172.21.15.19:0/335796287 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f3e0c0a2800 0x7f3e0c0a4cc0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:46.405 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:46.403+0000 7f3e2a200640 1 --2- 172.21.15.19:0/335796287 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3e241007c0 0x7f3e24199fe0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:46.405 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:46.403+0000 7f3e2a200640 1 --2- 172.21.15.19:0/335796287 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3e240ffe60 0x7f3e24199aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:46.405 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:46.403+0000 7f3e2a200640 1 -- 172.21.15.19:0/335796287 >> 172.21.15.19:0/335796287 conn(0x7f3e240fba60 msgr2=0x7f3e240fd190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:14:46.405 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:46.404+0000 7f3e2a200640 1 -- 172.21.15.19:0/335796287 shutdown_connections 2024-03-18T15:14:46.405 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:46.404+0000 7f3e2a200640 1 -- 172.21.15.19:0/335796287 wait complete. 2024-03-18T15:14:46.876 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:46 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/335796287' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:14:46.876 INFO:journalctl@ceph.mds.c.smithi104.stdout:Mar 18 15:14:46 smithi104 systemd[1]: Starting Ceph mds.c for d4dc2ba6-e538-11ee-95c9-87774f69a715... 2024-03-18T15:14:46.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:46 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/335796287' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:14:47.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:46 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/335796287' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:14:47.151 INFO:journalctl@ceph.mds.c.smithi104.stdout:Mar 18 15:14:46 smithi104 podman[66905]: 2024-03-18 15:14:46.778639464 +0000 UTC m=+0.017908661 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T15:14:47.151 INFO:journalctl@ceph.mds.c.smithi104.stdout:Mar 18 15:14:46 smithi104 podman[66905]: 2024-03-18 15:14:46.901041632 +0000 UTC m=+0.140310814 container create 57fbc4a22e683530fe9224218e84f8dc7ff25508eb6de5061e62bbb11587bf2d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-c, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS) 2024-03-18T15:14:47.151 INFO:journalctl@ceph.mds.c.smithi104.stdout:Mar 18 15:14:47 smithi104 podman[66905]: 2024-03-18 15:14:47.151230777 +0000 UTC m=+0.390499966 container init 57fbc4a22e683530fe9224218e84f8dc7ff25508eb6de5061e62bbb11587bf2d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-c, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-18T15:14:47.444 INFO:journalctl@ceph.mds.c.smithi104.stdout:Mar 18 15:14:47 smithi104 podman[66905]: 2024-03-18 15:14:47.15382187 +0000 UTC m=+0.393091059 container start 57fbc4a22e683530fe9224218e84f8dc7ff25508eb6de5061e62bbb11587bf2d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-c, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240307, io.buildah.version=1.33.5) 2024-03-18T15:14:47.444 INFO:journalctl@ceph.mds.c.smithi104.stdout:Mar 18 15:14:47 smithi104 ceph-mds[66924]: -- 172.21.15.104:0/3590893593 <== mon.1 v2:172.21.15.104:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557278399a40 con 0x55727836cc00 2024-03-18T15:14:47.445 INFO:journalctl@ceph.mds.c.smithi104.stdout:Mar 18 15:14:47 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-c[66920]: starting mds.c at 2024-03-18T15:14:47.445 INFO:journalctl@ceph.mds.c.smithi104.stdout:Mar 18 15:14:47 smithi104 bash[66905]: 57fbc4a22e683530fe9224218e84f8dc7ff25508eb6de5061e62bbb11587bf2d 2024-03-18T15:14:47.445 INFO:journalctl@ceph.mds.c.smithi104.stdout:Mar 18 15:14:47 smithi104 systemd[1]: Started Ceph mds.c for d4dc2ba6-e538-11ee-95c9-87774f69a715. 2024-03-18T15:14:47.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:47 smithi104 ceph-mon[36070]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:14:47.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:47 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:47.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:47 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:47.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:47 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:47.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:47 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T15:14:47.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:47 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T15:14:47.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:47 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:14:47.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:47 smithi019 ceph-mon[31522]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:14:47.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:47 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:47.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:47 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:47.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:47 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:47.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:47 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T15:14:47.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:47 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T15:14:47.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:47 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:14:48.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:47 smithi033 ceph-mon[36648]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:14:48.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:47 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:48.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:47 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:48.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:47 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:48.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:47 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T15:14:48.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:47 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T15:14:48.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:47 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:14:48.425 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T15:14:48.666 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:48.665+0000 7f8772c00640 1 -- 172.21.15.19:0/2931839157 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f876c1015f0 msgr2=0x7f876c101a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:48.666 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:48.665+0000 7f8772c00640 1 --2- 172.21.15.19:0/2931839157 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f876c1015f0 0x7f876c101a10 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f8760009900 tx=0x7f876002bc40 comp rx=0 tx=0).stop 2024-03-18T15:14:48.667 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:48.665+0000 7f8772c00640 1 -- 172.21.15.19:0/2931839157 shutdown_connections 2024-03-18T15:14:48.667 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:48.665+0000 7f8772c00640 1 --2- 172.21.15.19:0/2931839157 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f876c1031a0 0x7f876c109a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:48.667 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:48.665+0000 7f8772c00640 1 --2- 172.21.15.19:0/2931839157 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f876c1027e0 0x7f876c102c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:48.667 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:48.665+0000 7f8772c00640 1 --2- 172.21.15.19:0/2931839157 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f876c1015f0 0x7f876c101a10 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:48.667 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:48.665+0000 7f8772c00640 1 -- 172.21.15.19:0/2931839157 >> 172.21.15.19:0/2931839157 conn(0x7f876c0fcb40 msgr2=0x7f876c0fefa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:14:48.667 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:48.665+0000 7f8772c00640 1 -- 172.21.15.19:0/2931839157 shutdown_connections 2024-03-18T15:14:48.667 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:48.665+0000 7f8772c00640 1 -- 172.21.15.19:0/2931839157 wait complete. 2024-03-18T15:14:48.669 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:48.666+0000 7f8772c00640 1 Processor -- start 2024-03-18T15:14:48.669 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:48.666+0000 7f8772c00640 1 -- start start 2024-03-18T15:14:48.669 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:48.666+0000 7f8772c00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f876c1015f0 0x7f876c197850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:14:48.669 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:48.667+0000 7f8772c00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f876c1027e0 0x7f876c197d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:14:48.669 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:48.667+0000 7f8772c00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f876c1031a0 0x7f876c19ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:14:48.669 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:48.667+0000 7f8772c00640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f876c10d240 con 0x7f876c1015f0 2024-03-18T15:14:48.669 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:48.667+0000 7f8772c00640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f876c10d0c0 con 0x7f876c1027e0 2024-03-18T15:14:48.669 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:48.667+0000 7f8772c00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f876c10d3c0 con 0x7f876c1031a0 2024-03-18T15:14:48.669 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:48.667+0000 7f8770e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f876c1027e0 0x7f876c197d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:14:48.670 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:48.667+0000 7f8771800640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f876c1015f0 0x7f876c197850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:14:48.670 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:48.667+0000 7f8770e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f876c1027e0 0x7f876c197d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:36648/0 (socket says 172.21.15.19:36648) 2024-03-18T15:14:48.670 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:48.667+0000 7f8770e00640 1 -- 172.21.15.19:0/2872879930 learned_addr learned my addr 172.21.15.19:0/2872879930 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:14:48.670 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:48.667+0000 7f8772200640 1 --2- 172.21.15.19:0/2872879930 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f876c1031a0 0x7f876c19ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:14:48.670 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:48.667+0000 7f8770e00640 1 -- 172.21.15.19:0/2872879930 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f876c1031a0 msgr2=0x7f876c19ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:48.670 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:48.667+0000 7f8770e00640 1 --2- 172.21.15.19:0/2872879930 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f876c1031a0 0x7f876c19ee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:48.670 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:48.667+0000 7f8770e00640 1 -- 172.21.15.19:0/2872879930 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f876c1015f0 msgr2=0x7f876c197850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:48.670 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:48.667+0000 7f8770e00640 1 --2- 172.21.15.19:0/2872879930 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f876c1015f0 0x7f876c197850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:48.670 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:48.667+0000 7f8770e00640 1 -- 172.21.15.19:0/2872879930 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f876c19f470 con 0x7f876c1027e0 2024-03-18T15:14:48.670 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:48.668+0000 7f8770e00640 1 --2- 172.21.15.19:0/2872879930 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f876c1027e0 0x7f876c197d90 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f875800e9f0 tx=0x7f875800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:14:48.670 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:48.668+0000 7f876a000640 1 -- 172.21.15.19:0/2872879930 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f875800cd80 con 0x7f876c1027e0 2024-03-18T15:14:48.670 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:48.668+0000 7f8772c00640 1 -- 172.21.15.19:0/2872879930 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f876c19f760 con 0x7f876c1027e0 2024-03-18T15:14:48.670 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:48.668+0000 7f876a000640 1 -- 172.21.15.19:0/2872879930 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8758010910 con 0x7f876c1027e0 2024-03-18T15:14:48.671 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:48.668+0000 7f8772c00640 1 -- 172.21.15.19:0/2872879930 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f876c19fd00 con 0x7f876c1027e0 2024-03-18T15:14:48.671 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:48.668+0000 7f876a000640 1 -- 172.21.15.19:0/2872879930 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8758018980 con 0x7f876c1027e0 2024-03-18T15:14:48.671 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:48.669+0000 7f8772c00640 1 -- 172.21.15.19:0/2872879930 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f876c108910 con 0x7f876c1027e0 2024-03-18T15:14:48.672 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:48.671+0000 7f876a000640 1 -- 172.21.15.19:0/2872879930 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f8758010ac0 con 0x7f876c1027e0 2024-03-18T15:14:48.674 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:48.673+0000 7f876a000640 1 --2- 172.21.15.19:0/2872879930 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f87540a2800 0x7f87540a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:14:48.674 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:48.673+0000 7f876a000640 1 -- 172.21.15.19:0/2872879930 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f87580ce9d0 con 0x7f876c1027e0 2024-03-18T15:14:48.674 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:48.673+0000 7f8771800640 1 --2- 172.21.15.19:0/2872879930 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f87540a2800 0x7f87540a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:14:48.674 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:48.674+0000 7f8771800640 1 --2- 172.21.15.19:0/2872879930 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f87540a2800 0x7f87540a4cc0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f8760000c00 tx=0x7f8760005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:14:48.678 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:48.677+0000 7f876a000640 1 -- 172.21.15.19:0/2872879930 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f875809b820 con 0x7f876c1027e0 2024-03-18T15:14:48.912 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:48.911+0000 7f8772c00640 1 -- 172.21.15.19:0/2872879930 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f876c0623e0 con 0x7f876c1027e0 2024-03-18T15:14:48.913 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:48.912+0000 7f876a000640 1 -- 172.21.15.19:0/2872879930 <== mon.2 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6346 (secure 0 0 0) 0x7f875809f8c0 con 0x7f876c1027e0 2024-03-18T15:14:48.913 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-18T15:14:48.914 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6835/1135329545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6834","nonce":1135329545},{"type":"v1","addr":"172.21.15.19:6835","nonce":1135329545}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6833/482725809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":482725809},{"type":"v1","addr":"172.21.15.104:6833","nonce":482725809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6833/2272279159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":2272279159},{"type":"v1","addr":"172.21.15.33:6833","nonce":2272279159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24469,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6837/1778019859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6836","nonce":1778019859},{"type":"v1","addr":"172.21.15.19:6837","nonce":1778019859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24482,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/4051613151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":4051613151},{"type":"v1","addr":"172.21.15.33:6835","nonce":4051613151}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24499,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3788595957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3788595957},{"type":"v1","addr":"172.21.15.104:6835","nonce":3788595957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24511,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6837/260984674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":260984674},{"type":"v1","addr":"172.21.15.104:6837","nonce":260984674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-03-18T15:14:48.914 INFO:teuthology.orchestra.run.smithi019.stderr:dumped fsmap epoch 8 2024-03-18T15:14:48.916 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:48.914+0000 7f8772c00640 1 -- 172.21.15.19:0/2872879930 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f87540a2800 msgr2=0x7f87540a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:48.916 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:48.914+0000 7f8772c00640 1 --2- 172.21.15.19:0/2872879930 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f87540a2800 0x7f87540a4cc0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f8760000c00 tx=0x7f8760005fd0 comp rx=0 tx=0).stop 2024-03-18T15:14:48.916 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:48.914+0000 7f8772c00640 1 -- 172.21.15.19:0/2872879930 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f876c1027e0 msgr2=0x7f876c197d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:48.916 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:48.914+0000 7f8772c00640 1 --2- 172.21.15.19:0/2872879930 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f876c1027e0 0x7f876c197d90 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f875800e9f0 tx=0x7f875800eeb0 comp rx=0 tx=0).stop 2024-03-18T15:14:48.916 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:48.914+0000 7f8772c00640 1 -- 172.21.15.19:0/2872879930 shutdown_connections 2024-03-18T15:14:48.916 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:48.914+0000 7f8772c00640 1 --2- 172.21.15.19:0/2872879930 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f876c1031a0 0x7f876c19ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:48.916 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:48.914+0000 7f8772c00640 1 --2- 172.21.15.19:0/2872879930 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f87540a2800 0x7f87540a4cc0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:48.916 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:48.914+0000 7f8772c00640 1 --2- 172.21.15.19:0/2872879930 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f876c1027e0 0x7f876c197d90 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:48.916 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:48.914+0000 7f8772c00640 1 --2- 172.21.15.19:0/2872879930 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f876c1015f0 0x7f876c197850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:48.916 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:48.915+0000 7f8772c00640 1 -- 172.21.15.19:0/2872879930 >> 172.21.15.19:0/2872879930 conn(0x7f876c0fcb40 msgr2=0x7f876c105ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:14:48.917 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:48.915+0000 7f8772c00640 1 -- 172.21.15.19:0/2872879930 shutdown_connections 2024-03-18T15:14:48.917 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:48.915+0000 7f8772c00640 1 -- 172.21.15.19:0/2872879930 wait complete. 2024-03-18T15:14:48.931 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:48 smithi019 ceph-mon[31522]: Deploying daemon mds.h on smithi033 2024-03-18T15:14:48.931 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:48 smithi019 ceph-mon[31522]: mds.? [v2:172.21.15.104:6836/260984674,v1:172.21.15.104:6837/260984674] up:boot 2024-03-18T15:14:48.931 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:48 smithi019 ceph-mon[31522]: fsmap 7 up:standby 2024-03-18T15:14:48.931 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:48 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-18T15:14:48.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:48 smithi104 ceph-mon[36070]: Deploying daemon mds.h on smithi033 2024-03-18T15:14:48.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:48 smithi104 ceph-mon[36070]: mds.? [v2:172.21.15.104:6836/260984674,v1:172.21.15.104:6837/260984674] up:boot 2024-03-18T15:14:48.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:48 smithi104 ceph-mon[36070]: fsmap 7 up:standby 2024-03-18T15:14:48.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:48 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-18T15:14:49.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:48 smithi033 ceph-mon[36648]: Deploying daemon mds.h on smithi033 2024-03-18T15:14:49.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:48 smithi033 ceph-mon[36648]: mds.? [v2:172.21.15.104:6836/260984674,v1:172.21.15.104:6837/260984674] up:boot 2024-03-18T15:14:49.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:48 smithi033 ceph-mon[36648]: fsmap 7 up:standby 2024-03-18T15:14:49.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:48 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-18T15:14:49.662 INFO:journalctl@ceph.mds.h.smithi033.stdout:Mar 18 15:14:49 smithi033 systemd[1]: Starting Ceph mds.h for d4dc2ba6-e538-11ee-95c9-87774f69a715... 2024-03-18T15:14:49.934 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:49 smithi033 ceph-mon[36648]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:14:49.935 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:49 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/2872879930' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:14:49.935 INFO:journalctl@ceph.mds.h.smithi033.stdout:Mar 18 15:14:49 smithi033 podman[67279]: 2024-03-18 15:14:49.683782356 +0000 UTC m=+0.018354270 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T15:14:49.935 INFO:journalctl@ceph.mds.h.smithi033.stdout:Mar 18 15:14:49 smithi033 podman[67279]: 2024-03-18 15:14:49.812179885 +0000 UTC m=+0.146751791 container create 10fcda2e0d01b9d8509174d765797d74ae0f24036efbdb9df414a583d970e07b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-h, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T15:14:49.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:49 smithi104 ceph-mon[36070]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:14:49.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:49 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/2872879930' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:14:49.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:49 smithi019 ceph-mon[31522]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:14:49.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:49 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/2872879930' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:14:50.507 INFO:journalctl@ceph.mds.h.smithi033.stdout:Mar 18 15:14:50 smithi033 podman[67279]: 2024-03-18 15:14:50.056984017 +0000 UTC m=+0.391555930 container init 10fcda2e0d01b9d8509174d765797d74ae0f24036efbdb9df414a583d970e07b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-h, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_CLEAN=True) 2024-03-18T15:14:50.507 INFO:journalctl@ceph.mds.h.smithi033.stdout:Mar 18 15:14:50 smithi033 podman[67279]: 2024-03-18 15:14:50.059814184 +0000 UTC m=+0.394386097 container start 10fcda2e0d01b9d8509174d765797d74ae0f24036efbdb9df414a583d970e07b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-h, org.label-schema.build-date=20240307, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5) 2024-03-18T15:14:50.507 INFO:journalctl@ceph.mds.h.smithi033.stdout:Mar 18 15:14:50 smithi033 ceph-mds[67298]: -- 172.21.15.33:0/1805547479 <== mon.2 v2:172.21.15.33:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f1bf6f1a40 con 0x55f1bf6c4c00 2024-03-18T15:14:50.507 INFO:journalctl@ceph.mds.h.smithi033.stdout:Mar 18 15:14:50 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-h[67294]: starting mds.h at 2024-03-18T15:14:50.507 INFO:journalctl@ceph.mds.h.smithi033.stdout:Mar 18 15:14:50 smithi033 bash[67279]: 10fcda2e0d01b9d8509174d765797d74ae0f24036efbdb9df414a583d970e07b 2024-03-18T15:14:50.507 INFO:journalctl@ceph.mds.h.smithi033.stdout:Mar 18 15:14:50 smithi033 systemd[1]: Started Ceph mds.h for d4dc2ba6-e538-11ee-95c9-87774f69a715. 2024-03-18T15:14:50.936 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T15:14:51.170 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:51.169+0000 7fd36a800640 1 -- 172.21.15.19:0/151413197 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd364107e00 msgr2=0x7fd364108220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:51.170 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:51.169+0000 7fd36a800640 1 --2- 172.21.15.19:0/151413197 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd364107e00 0x7fd364108220 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd354009be0 tx=0x7fd35402bef0 comp rx=0 tx=0).stop 2024-03-18T15:14:51.170 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:51.169+0000 7fd36a800640 1 -- 172.21.15.19:0/151413197 shutdown_connections 2024-03-18T15:14:51.171 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:51.169+0000 7fd36a800640 1 --2- 172.21.15.19:0/151413197 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd3641099b0 0x7fd364110240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:51.171 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:51.169+0000 7fd36a800640 1 --2- 172.21.15.19:0/151413197 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd364108ff0 0x7fd364109470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:51.171 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:51.169+0000 7fd36a800640 1 --2- 172.21.15.19:0/151413197 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd364107e00 0x7fd364108220 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:51.171 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:51.169+0000 7fd36a800640 1 -- 172.21.15.19:0/151413197 >> 172.21.15.19:0/151413197 conn(0x7fd364076dc0 msgr2=0x7fd3640791e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:14:51.171 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:51.169+0000 7fd36a800640 1 -- 172.21.15.19:0/151413197 shutdown_connections 2024-03-18T15:14:51.171 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:51.169+0000 7fd36a800640 1 -- 172.21.15.19:0/151413197 wait complete. 2024-03-18T15:14:51.173 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:51.170+0000 7fd36a800640 1 Processor -- start 2024-03-18T15:14:51.173 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:51.170+0000 7fd36a800640 1 -- start start 2024-03-18T15:14:51.173 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:51.170+0000 7fd36a800640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd364107e00 0x7fd36419e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:14:51.174 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:51.170+0000 7fd36a800640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd364108ff0 0x7fd36419e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:14:51.174 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:51.170+0000 7fd36a800640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd3641099b0 0x7fd3641a5690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:14:51.174 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:51.170+0000 7fd36a800640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fd364113aa0 con 0x7fd364107e00 2024-03-18T15:14:51.174 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:51.170+0000 7fd36a800640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd364113920 con 0x7fd3641099b0 2024-03-18T15:14:51.174 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:51.170+0000 7fd36a800640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fd364113c20 con 0x7fd364108ff0 2024-03-18T15:14:51.174 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:51.171+0000 7fd362a00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd364108ff0 0x7fd36419e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:14:51.174 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:51.171+0000 7fd363400640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd364107e00 0x7fd36419e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:14:51.174 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:51.171+0000 7fd363e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd3641099b0 0x7fd3641a5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:14:51.174 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:51.171+0000 7fd362a00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd364108ff0 0x7fd36419e620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.19:54106/0 (socket says 172.21.15.19:54106) 2024-03-18T15:14:51.174 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:51.171+0000 7fd363e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd3641099b0 0x7fd3641a5690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:36656/0 (socket says 172.21.15.19:36656) 2024-03-18T15:14:51.174 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:51.171+0000 7fd363400640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd364107e00 0x7fd36419e0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:56770/0 (socket says 172.21.15.19:56770) 2024-03-18T15:14:51.174 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:51.171+0000 7fd362a00640 1 -- 172.21.15.19:0/3752642780 learned_addr learned my addr 172.21.15.19:0/3752642780 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:14:51.174 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:51.171+0000 7fd363e00640 1 -- 172.21.15.19:0/3752642780 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd364108ff0 msgr2=0x7fd36419e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:51.175 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:51.171+0000 7fd363e00640 1 --2- 172.21.15.19:0/3752642780 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd364108ff0 0x7fd36419e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:51.175 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:51.171+0000 7fd363e00640 1 -- 172.21.15.19:0/3752642780 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd364107e00 msgr2=0x7fd36419e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:51.175 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:51.171+0000 7fd363e00640 1 --2- 172.21.15.19:0/3752642780 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd364107e00 0x7fd36419e0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:51.175 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:51.171+0000 7fd363e00640 1 -- 172.21.15.19:0/3752642780 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3641a5d00 con 0x7fd3641099b0 2024-03-18T15:14:51.175 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:51.171+0000 7fd363400640 1 --2- 172.21.15.19:0/3752642780 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd364107e00 0x7fd36419e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:14:51.175 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:51.171+0000 7fd363e00640 1 --2- 172.21.15.19:0/3752642780 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd3641099b0 0x7fd3641a5690 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd35c00e970 tx=0x7fd35c00ee30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:14:51.175 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:51.172+0000 7fd35be00640 1 -- 172.21.15.19:0/3752642780 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd35c00ccc0 con 0x7fd3641099b0 2024-03-18T15:14:51.175 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:51.172+0000 7fd35be00640 1 -- 172.21.15.19:0/3752642780 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd35c005020 con 0x7fd3641099b0 2024-03-18T15:14:51.175 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:51.172+0000 7fd35be00640 1 -- 172.21.15.19:0/3752642780 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd35c004a00 con 0x7fd3641099b0 2024-03-18T15:14:51.175 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:51.172+0000 7fd36a800640 1 -- 172.21.15.19:0/3752642780 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3641a5ff0 con 0x7fd3641099b0 2024-03-18T15:14:51.175 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:51.172+0000 7fd36a800640 1 -- 172.21.15.19:0/3752642780 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3641a6460 con 0x7fd3641099b0 2024-03-18T15:14:51.176 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:51.173+0000 7fd36a800640 1 -- 172.21.15.19:0/3752642780 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd36410f0c0 con 0x7fd3641099b0 2024-03-18T15:14:51.176 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:51.175+0000 7fd35be00640 1 -- 172.21.15.19:0/3752642780 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fd35c0051d0 con 0x7fd3641099b0 2024-03-18T15:14:51.177 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:51.176+0000 7fd35be00640 1 --2- 172.21.15.19:0/3752642780 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fd3480a2880 0x7fd3480a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:14:51.177 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:51.176+0000 7fd35be00640 1 -- 172.21.15.19:0/3752642780 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fd35c0ce530 con 0x7fd3641099b0 2024-03-18T15:14:51.177 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:51.177+0000 7fd363400640 1 --2- 172.21.15.19:0/3752642780 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fd3480a2880 0x7fd3480a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:14:51.178 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:51.177+0000 7fd363400640 1 --2- 172.21.15.19:0/3752642780 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fd3480a2880 0x7fd3480a4d40 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fd354034fd0 tx=0x7fd354000f40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:14:51.181 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:51.181+0000 7fd35be00640 1 -- 172.21.15.19:0/3752642780 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd35c09b9d0 con 0x7fd3641099b0 2024-03-18T15:14:51.428 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:51.427+0000 7fd36a800640 1 -- 172.21.15.19:0/3752642780 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd36402d1b0 con 0x7fd3641099b0 2024-03-18T15:14:51.429 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:51.428+0000 7fd35be00640 1 -- 172.21.15.19:0/3752642780 <== mon.2 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7168 (secure 0 0 0) 0x7fd35c09b2d0 con 0x7fd3641099b0 2024-03-18T15:14:51.430 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-18T15:14:51.430 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6835/1135329545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6834","nonce":1135329545},{"type":"v1","addr":"172.21.15.19:6835","nonce":1135329545}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6833/482725809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":482725809},{"type":"v1","addr":"172.21.15.104:6833","nonce":482725809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6833/2272279159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":2272279159},{"type":"v1","addr":"172.21.15.33:6833","nonce":2272279159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24469,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6837/1778019859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6836","nonce":1778019859},{"type":"v1","addr":"172.21.15.19:6837","nonce":1778019859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24482,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/4051613151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":4051613151},{"type":"v1","addr":"172.21.15.33:6835","nonce":4051613151}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24499,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3788595957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3788595957},{"type":"v1","addr":"172.21.15.104:6835","nonce":3788595957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24506,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/160187378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":160187378},{"type":"v1","addr":"172.21.15.33:6837","nonce":160187378}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24511,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6837/260984674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":260984674},{"type":"v1","addr":"172.21.15.104:6837","nonce":260984674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-03-18T15:14:51.430 INFO:teuthology.orchestra.run.smithi019.stderr:dumped fsmap epoch 9 2024-03-18T15:14:51.433 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:51.431+0000 7fd36a800640 1 -- 172.21.15.19:0/3752642780 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fd3480a2880 msgr2=0x7fd3480a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:51.433 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:51.431+0000 7fd36a800640 1 --2- 172.21.15.19:0/3752642780 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fd3480a2880 0x7fd3480a4d40 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fd354034fd0 tx=0x7fd354000f40 comp rx=0 tx=0).stop 2024-03-18T15:14:51.433 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:51.431+0000 7fd36a800640 1 -- 172.21.15.19:0/3752642780 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd3641099b0 msgr2=0x7fd3641a5690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:51.433 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:51.431+0000 7fd36a800640 1 --2- 172.21.15.19:0/3752642780 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd3641099b0 0x7fd3641a5690 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd35c00e970 tx=0x7fd35c00ee30 comp rx=0 tx=0).stop 2024-03-18T15:14:51.433 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:51.431+0000 7fd36a800640 1 -- 172.21.15.19:0/3752642780 shutdown_connections 2024-03-18T15:14:51.433 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:51.431+0000 7fd36a800640 1 --2- 172.21.15.19:0/3752642780 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fd3480a2880 0x7fd3480a4d40 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:51.433 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:51.431+0000 7fd36a800640 1 --2- 172.21.15.19:0/3752642780 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd3641099b0 0x7fd3641a5690 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:51.433 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:51.431+0000 7fd36a800640 1 --2- 172.21.15.19:0/3752642780 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd364108ff0 0x7fd36419e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:51.433 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:51.431+0000 7fd36a800640 1 --2- 172.21.15.19:0/3752642780 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd364107e00 0x7fd36419e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:51.433 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:51.432+0000 7fd36a800640 1 -- 172.21.15.19:0/3752642780 >> 172.21.15.19:0/3752642780 conn(0x7fd364076dc0 msgr2=0x7fd36410c290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:14:51.434 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:51.432+0000 7fd36a800640 1 -- 172.21.15.19:0/3752642780 shutdown_connections 2024-03-18T15:14:51.434 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:51.432+0000 7fd36a800640 1 -- 172.21.15.19:0/3752642780 wait complete. 2024-03-18T15:14:51.450 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:51 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:51.450 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:51 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:51.450 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:51 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:51.451 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:51 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T15:14:51.451 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:51 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T15:14:51.451 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:51 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:14:51.451 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:51 smithi019 ceph-mon[31522]: Deploying daemon mds.i on smithi104 2024-03-18T15:14:51.451 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:51 smithi019 ceph-mon[31522]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:14:51.451 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:51 smithi019 ceph-mon[31522]: mds.? [v2:172.21.15.33:6836/160187378,v1:172.21.15.33:6837/160187378] up:boot 2024-03-18T15:14:51.451 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:51 smithi019 ceph-mon[31522]: fsmap 8 up:standby 2024-03-18T15:14:51.451 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:51 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-18T15:14:51.451 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:51 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-18T15:14:51.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:51 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:51.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:51 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:51.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:51 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:51.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:51 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T15:14:51.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:51 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T15:14:51.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:51 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:14:51.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:51 smithi033 ceph-mon[36648]: Deploying daemon mds.i on smithi104 2024-03-18T15:14:51.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:51 smithi033 ceph-mon[36648]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:14:51.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:51 smithi033 ceph-mon[36648]: mds.? [v2:172.21.15.33:6836/160187378,v1:172.21.15.33:6837/160187378] up:boot 2024-03-18T15:14:51.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:51 smithi033 ceph-mon[36648]: fsmap 8 up:standby 2024-03-18T15:14:51.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:51 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-18T15:14:51.508 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:51 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-18T15:14:51.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:51 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:51.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:51 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:51.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:51 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:51.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:51 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T15:14:51.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:51 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T15:14:51.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:51 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:14:51.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:51 smithi104 ceph-mon[36070]: Deploying daemon mds.i on smithi104 2024-03-18T15:14:51.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:51 smithi104 ceph-mon[36070]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:14:51.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:51 smithi104 ceph-mon[36070]: mds.? [v2:172.21.15.33:6836/160187378,v1:172.21.15.33:6837/160187378] up:boot 2024-03-18T15:14:51.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:51 smithi104 ceph-mon[36070]: fsmap 8 up:standby 2024-03-18T15:14:51.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:51 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-18T15:14:51.696 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:51 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-18T15:14:52.657 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:52 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/3752642780' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:14:52.658 INFO:journalctl@ceph.mds.i.smithi104.stdout:Mar 18 15:14:52 smithi104 systemd[1]: Starting Ceph mds.i for d4dc2ba6-e538-11ee-95c9-87774f69a715... 2024-03-18T15:14:52.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:52 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/3752642780' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:14:52.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:52 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/3752642780' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:14:52.912 INFO:journalctl@ceph.mds.i.smithi104.stdout:Mar 18 15:14:52 smithi104 podman[67217]: 2024-03-18 15:14:52.559869912 +0000 UTC m=+0.017857720 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T15:14:52.912 INFO:journalctl@ceph.mds.i.smithi104.stdout:Mar 18 15:14:52 smithi104 podman[67217]: 2024-03-18 15:14:52.675980026 +0000 UTC m=+0.133967862 container create 128cfe8487678eda64be790cae928f6249e791c30f525500e1fd51604bb891d9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-i, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T15:14:52.912 INFO:journalctl@ceph.mds.i.smithi104.stdout:Mar 18 15:14:52 smithi104 podman[67217]: 2024-03-18 15:14:52.912091579 +0000 UTC m=+0.370079396 container init 128cfe8487678eda64be790cae928f6249e791c30f525500e1fd51604bb891d9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-i, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307) 2024-03-18T15:14:53.194 INFO:journalctl@ceph.mds.i.smithi104.stdout:Mar 18 15:14:52 smithi104 podman[67217]: 2024-03-18 15:14:52.914932619 +0000 UTC m=+0.372920426 container start 128cfe8487678eda64be790cae928f6249e791c30f525500e1fd51604bb891d9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-i, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307) 2024-03-18T15:14:53.194 INFO:journalctl@ceph.mds.i.smithi104.stdout:Mar 18 15:14:52 smithi104 ceph-mds[67236]: -- 172.21.15.104:0/37251147 <== mon.1 v2:172.21.15.104:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b396c45a40 con 0x55b396c18c00 2024-03-18T15:14:53.195 INFO:journalctl@ceph.mds.i.smithi104.stdout:Mar 18 15:14:52 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-i[67232]: starting mds.i at 2024-03-18T15:14:53.195 INFO:journalctl@ceph.mds.i.smithi104.stdout:Mar 18 15:14:53 smithi104 bash[67217]: 128cfe8487678eda64be790cae928f6249e791c30f525500e1fd51604bb891d9 2024-03-18T15:14:53.195 INFO:journalctl@ceph.mds.i.smithi104.stdout:Mar 18 15:14:53 smithi104 systemd[1]: Started Ceph mds.i for d4dc2ba6-e538-11ee-95c9-87774f69a715. 2024-03-18T15:14:53.455 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T15:14:53.678 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:53.677+0000 7fd30f800640 1 -- 172.21.15.19:0/1583630625 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd3081055b0 msgr2=0x7fd30810be40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:53.679 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:53.677+0000 7fd30f800640 1 --2- 172.21.15.19:0/1583630625 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd3081055b0 0x7fd30810be40 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd300009ca0 tx=0x7fd30002c100 comp rx=0 tx=0).stop 2024-03-18T15:14:53.679 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:53.677+0000 7fd30f800640 1 -- 172.21.15.19:0/1583630625 shutdown_connections 2024-03-18T15:14:53.679 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:53.677+0000 7fd30f800640 1 --2- 172.21.15.19:0/1583630625 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd3081055b0 0x7fd30810be40 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:53.679 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:53.677+0000 7fd30f800640 1 --2- 172.21.15.19:0/1583630625 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd308104bf0 0x7fd308105070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:53.679 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:53.677+0000 7fd30f800640 1 --2- 172.21.15.19:0/1583630625 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd308103a00 0x7fd308103e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:53.679 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:53.677+0000 7fd30f800640 1 -- 172.21.15.19:0/1583630625 >> 172.21.15.19:0/1583630625 conn(0x7fd3080fef90 msgr2=0x7fd3081013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:14:53.679 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:53.677+0000 7fd30f800640 1 -- 172.21.15.19:0/1583630625 shutdown_connections 2024-03-18T15:14:53.679 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:53.677+0000 7fd30f800640 1 -- 172.21.15.19:0/1583630625 wait complete. 2024-03-18T15:14:53.683 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:53.678+0000 7fd30f800640 1 Processor -- start 2024-03-18T15:14:53.683 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:53.678+0000 7fd30f800640 1 -- start start 2024-03-18T15:14:53.683 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:53.678+0000 7fd30f800640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd308103a00 0x7fd308199c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:14:53.683 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:53.679+0000 7fd30f800640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd308104bf0 0x7fd30819a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:14:53.683 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:53.679+0000 7fd30f800640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd3081055b0 0x7fd3081a1230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:14:53.683 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:53.679+0000 7fd30f800640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fd30810f680 con 0x7fd308103a00 2024-03-18T15:14:53.683 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:53.679+0000 7fd30f800640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd30810f500 con 0x7fd308104bf0 2024-03-18T15:14:53.683 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:53.679+0000 7fd30f800640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fd30810f800 con 0x7fd3081055b0 2024-03-18T15:14:53.684 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:53.679+0000 7fd30ce00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd308103a00 0x7fd308199c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:14:53.684 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:53.679+0000 7fd30ce00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd308103a00 0x7fd308199c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:49098/0 (socket says 172.21.15.19:49098) 2024-03-18T15:14:53.684 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:53.679+0000 7fd30ce00640 1 -- 172.21.15.19:0/2854839631 learned_addr learned my addr 172.21.15.19:0/2854839631 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:14:53.684 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:53.679+0000 7fd307e00640 1 --2- 172.21.15.19:0/2854839631 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd308104bf0 0x7fd30819a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:14:53.684 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:53.679+0000 7fd30d800640 1 --2- 172.21.15.19:0/2854839631 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd3081055b0 0x7fd3081a1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:14:53.684 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:53.679+0000 7fd30ce00640 1 -- 172.21.15.19:0/2854839631 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd3081055b0 msgr2=0x7fd3081a1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:53.684 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:53.679+0000 7fd30ce00640 1 --2- 172.21.15.19:0/2854839631 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd3081055b0 0x7fd3081a1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:53.684 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:53.679+0000 7fd30ce00640 1 -- 172.21.15.19:0/2854839631 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd308104bf0 msgr2=0x7fd30819a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:53.684 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:53.679+0000 7fd30ce00640 1 --2- 172.21.15.19:0/2854839631 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd308104bf0 0x7fd30819a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:53.684 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:53.679+0000 7fd30ce00640 1 -- 172.21.15.19:0/2854839631 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3081a18d0 con 0x7fd308103a00 2024-03-18T15:14:53.684 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:53.680+0000 7fd30ce00640 1 --2- 172.21.15.19:0/2854839631 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd308103a00 0x7fd308199c90 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fd2fc00e9c0 tx=0x7fd2fc00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:14:53.684 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:53.680+0000 7fd305600640 1 -- 172.21.15.19:0/2854839631 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2fc00cd80 con 0x7fd308103a00 2024-03-18T15:14:53.684 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:53.680+0000 7fd305600640 1 -- 172.21.15.19:0/2854839631 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd2fc0035f0 con 0x7fd308103a00 2024-03-18T15:14:53.685 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:53.680+0000 7fd30f800640 1 -- 172.21.15.19:0/2854839631 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3081a1bc0 con 0x7fd308103a00 2024-03-18T15:14:53.685 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:53.680+0000 7fd305600640 1 -- 172.21.15.19:0/2854839631 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2fc010690 con 0x7fd308103a00 2024-03-18T15:14:53.685 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:53.680+0000 7fd30f800640 1 -- 172.21.15.19:0/2854839631 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3081a2110 con 0x7fd308103a00 2024-03-18T15:14:53.685 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:53.682+0000 7fd30f800640 1 -- 172.21.15.19:0/2854839631 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd30810ad20 con 0x7fd308103a00 2024-03-18T15:14:53.685 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:53.684+0000 7fd305600640 1 -- 172.21.15.19:0/2854839631 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fd2fc0037a0 con 0x7fd308103a00 2024-03-18T15:14:53.686 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:53.685+0000 7fd305600640 1 --2- 172.21.15.19:0/2854839631 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fd2f00a2800 0x7fd2f00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:14:53.686 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:53.685+0000 7fd305600640 1 -- 172.21.15.19:0/2854839631 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fd2fc0cea10 con 0x7fd308103a00 2024-03-18T15:14:53.686 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:53.685+0000 7fd307e00640 1 --2- 172.21.15.19:0/2854839631 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fd2f00a2800 0x7fd2f00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:14:53.687 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:53.686+0000 7fd307e00640 1 --2- 172.21.15.19:0/2854839631 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fd2f00a2800 0x7fd2f00a4cc0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fd30819b190 tx=0x7fd2f4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:14:53.696 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:53.695+0000 7fd305600640 1 -- 172.21.15.19:0/2854839631 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd2fc09beb0 con 0x7fd308103a00 2024-03-18T15:14:53.959 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:53.958+0000 7fd30f800640 1 -- 172.21.15.19:0/2854839631 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd30802d1b0 con 0x7fd308103a00 2024-03-18T15:14:53.960 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:53.959+0000 7fd305600640 1 -- 172.21.15.19:0/2854839631 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+7995 (secure 0 0 0) 0x7fd2fc09b7b0 con 0x7fd308103a00 2024-03-18T15:14:53.961 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-18T15:14:53.961 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6835/1135329545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6834","nonce":1135329545},{"type":"v1","addr":"172.21.15.19:6835","nonce":1135329545}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6833/482725809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":482725809},{"type":"v1","addr":"172.21.15.104:6833","nonce":482725809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6833/2272279159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":2272279159},{"type":"v1","addr":"172.21.15.33:6833","nonce":2272279159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24469,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6837/1778019859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6836","nonce":1778019859},{"type":"v1","addr":"172.21.15.19:6837","nonce":1778019859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24482,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/4051613151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":4051613151},{"type":"v1","addr":"172.21.15.33:6835","nonce":4051613151}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24499,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3788595957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3788595957},{"type":"v1","addr":"172.21.15.104:6835","nonce":3788595957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24506,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/160187378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":160187378},{"type":"v1","addr":"172.21.15.33:6837","nonce":160187378}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24511,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6837/260984674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":260984674},{"type":"v1","addr":"172.21.15.104:6837","nonce":260984674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24529,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6839/805341436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6838","nonce":805341436},{"type":"v1","addr":"172.21.15.104:6839","nonce":805341436}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-03-18T15:14:53.961 INFO:teuthology.orchestra.run.smithi019.stderr:dumped fsmap epoch 10 2024-03-18T15:14:53.963 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:53.962+0000 7fd30f800640 1 -- 172.21.15.19:0/2854839631 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fd2f00a2800 msgr2=0x7fd2f00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:53.963 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:53.962+0000 7fd30f800640 1 --2- 172.21.15.19:0/2854839631 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fd2f00a2800 0x7fd2f00a4cc0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fd30819b190 tx=0x7fd2f4005f90 comp rx=0 tx=0).stop 2024-03-18T15:14:53.963 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:53.962+0000 7fd30f800640 1 -- 172.21.15.19:0/2854839631 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd308103a00 msgr2=0x7fd308199c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:53.963 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:53.962+0000 7fd30f800640 1 --2- 172.21.15.19:0/2854839631 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd308103a00 0x7fd308199c90 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fd2fc00e9c0 tx=0x7fd2fc00ee80 comp rx=0 tx=0).stop 2024-03-18T15:14:53.963 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:53.962+0000 7fd30f800640 1 -- 172.21.15.19:0/2854839631 shutdown_connections 2024-03-18T15:14:53.963 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:53.962+0000 7fd30f800640 1 --2- 172.21.15.19:0/2854839631 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd3081055b0 0x7fd3081a1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:53.963 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:53.962+0000 7fd30f800640 1 --2- 172.21.15.19:0/2854839631 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fd2f00a2800 0x7fd2f00a4cc0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:53.963 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:53.962+0000 7fd30f800640 1 --2- 172.21.15.19:0/2854839631 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd308104bf0 0x7fd30819a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:53.963 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:53.962+0000 7fd30f800640 1 --2- 172.21.15.19:0/2854839631 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd308103a00 0x7fd308199c90 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:53.964 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:53.962+0000 7fd30f800640 1 -- 172.21.15.19:0/2854839631 >> 172.21.15.19:0/2854839631 conn(0x7fd3080fef90 msgr2=0x7fd308107ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:14:53.964 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:53.962+0000 7fd30f800640 1 -- 172.21.15.19:0/2854839631 shutdown_connections 2024-03-18T15:14:53.964 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:53.962+0000 7fd30f800640 1 -- 172.21.15.19:0/2854839631 wait complete. 2024-03-18T15:14:54.359 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:54 smithi019 ceph-mon[31522]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:14:54.359 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:54 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:54.359 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:54 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:54.359 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:54 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:54.359 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:54 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T15:14:54.359 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:54 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T15:14:54.359 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:54 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:14:54.359 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:54 smithi019 ceph-mon[31522]: mds.? [v2:172.21.15.104:6838/805341436,v1:172.21.15.104:6839/805341436] up:boot 2024-03-18T15:14:54.359 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:54 smithi019 ceph-mon[31522]: fsmap 9 up:standby 2024-03-18T15:14:54.360 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:54 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-18T15:14:54.360 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:54 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/2854839631' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:14:54.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:54 smithi104 ceph-mon[36070]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:14:54.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:54 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:54.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:54 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:54.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:54 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:54.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:54 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T15:14:54.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:54 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T15:14:54.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:54 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:14:54.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:54 smithi104 ceph-mon[36070]: mds.? [v2:172.21.15.104:6838/805341436,v1:172.21.15.104:6839/805341436] up:boot 2024-03-18T15:14:54.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:54 smithi104 ceph-mon[36070]: fsmap 9 up:standby 2024-03-18T15:14:54.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:54 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-18T15:14:54.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:54 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/2854839631' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:14:54.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:54 smithi033 ceph-mon[36648]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:14:54.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:54 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:54.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:54 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:54.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:54 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:54.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:54 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T15:14:54.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:54 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T15:14:54.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:54 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:14:54.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:54 smithi033 ceph-mon[36648]: mds.? [v2:172.21.15.104:6838/805341436,v1:172.21.15.104:6839/805341436] up:boot 2024-03-18T15:14:54.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:54 smithi033 ceph-mon[36648]: fsmap 9 up:standby 2024-03-18T15:14:54.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:54 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-18T15:14:54.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:54 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/2854839631' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:14:55.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:55 smithi104 ceph-mon[36070]: Deploying daemon mds.g on smithi019 2024-03-18T15:14:55.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:55 smithi019 ceph-mon[31522]: Deploying daemon mds.g on smithi019 2024-03-18T15:14:55.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:55 smithi033 ceph-mon[36648]: Deploying daemon mds.g on smithi019 2024-03-18T15:14:55.983 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T15:14:56.209 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:56.207+0000 7efd5d400640 1 -- 172.21.15.19:0/1836701182 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efd581006e0 msgr2=0x7efd58100b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:56.209 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:56.207+0000 7efd5d400640 1 --2- 172.21.15.19:0/1836701182 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efd581006e0 0x7efd58100b00 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7efd48009900 tx=0x7efd4802bc50 comp rx=0 tx=0).stop 2024-03-18T15:14:56.209 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:56.207+0000 7efd5d400640 1 -- 172.21.15.19:0/1836701182 shutdown_connections 2024-03-18T15:14:56.209 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:56.207+0000 7efd5d400640 1 --2- 172.21.15.19:0/1836701182 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efd58102290 0x7efd58108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:56.209 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:56.207+0000 7efd5d400640 1 --2- 172.21.15.19:0/1836701182 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efd581018d0 0x7efd58101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:56.209 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:56.207+0000 7efd5d400640 1 --2- 172.21.15.19:0/1836701182 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efd581006e0 0x7efd58100b00 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:56.210 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:56.207+0000 7efd5d400640 1 -- 172.21.15.19:0/1836701182 >> 172.21.15.19:0/1836701182 conn(0x7efd580fbc30 msgr2=0x7efd580fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:14:56.210 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:56.207+0000 7efd5d400640 1 -- 172.21.15.19:0/1836701182 shutdown_connections 2024-03-18T15:14:56.210 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:56.208+0000 7efd5d400640 1 -- 172.21.15.19:0/1836701182 wait complete. 2024-03-18T15:14:56.212 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:56.209+0000 7efd5d400640 1 Processor -- start 2024-03-18T15:14:56.212 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:56.209+0000 7efd5d400640 1 -- start start 2024-03-18T15:14:56.212 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:56.210+0000 7efd5d400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efd581006e0 0x7efd58104670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:14:56.212 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:56.210+0000 7efd5d400640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efd581018d0 0x7efd58102cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:14:56.213 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:56.210+0000 7efd5d400640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efd58102290 0x7efd581031f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:14:56.213 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:56.210+0000 7efd5d400640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7efd5810c3a0 con 0x7efd58102290 2024-03-18T15:14:56.213 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:56.210+0000 7efd5d400640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7efd5810c220 con 0x7efd581006e0 2024-03-18T15:14:56.213 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:56.210+0000 7efd5d400640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7efd5810c520 con 0x7efd581018d0 2024-03-18T15:14:56.213 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:56.210+0000 7efd57200640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efd58102290 0x7efd581031f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:14:56.213 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:56.210+0000 7efd56800640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efd581006e0 0x7efd58104670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:14:56.213 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:56.210+0000 7efd57200640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efd58102290 0x7efd581031f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:49106/0 (socket says 172.21.15.19:49106) 2024-03-18T15:14:56.213 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:56.210+0000 7efd57200640 1 -- 172.21.15.19:0/2387725028 learned_addr learned my addr 172.21.15.19:0/2387725028 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:14:56.213 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:56.210+0000 7efd4fe00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efd581018d0 0x7efd58102cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:14:56.213 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:56.210+0000 7efd4fe00640 1 -- 172.21.15.19:0/2387725028 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efd581006e0 msgr2=0x7efd58104670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:56.214 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:56.210+0000 7efd4fe00640 1 --2- 172.21.15.19:0/2387725028 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efd581006e0 0x7efd58104670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:56.214 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:56.211+0000 7efd4fe00640 1 -- 172.21.15.19:0/2387725028 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efd58102290 msgr2=0x7efd581031f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:56.214 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:56.211+0000 7efd4fe00640 1 --2- 172.21.15.19:0/2387725028 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efd58102290 0x7efd581031f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:56.214 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:56.211+0000 7efd4fe00640 1 -- 172.21.15.19:0/2387725028 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd58103a80 con 0x7efd581018d0 2024-03-18T15:14:56.214 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:56.211+0000 7efd57200640 1 --2- 172.21.15.19:0/2387725028 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efd58102290 0x7efd581031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:14:56.214 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:56.211+0000 7efd4fe00640 1 --2- 172.21.15.19:0/2387725028 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efd581018d0 0x7efd58102cb0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7efd4000e9f0 tx=0x7efd4000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:14:56.214 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:56.211+0000 7efd4f400640 1 -- 172.21.15.19:0/2387725028 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd4000ccc0 con 0x7efd581018d0 2024-03-18T15:14:56.214 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:56.211+0000 7efd4f400640 1 -- 172.21.15.19:0/2387725028 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efd40005020 con 0x7efd581018d0 2024-03-18T15:14:56.214 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:56.211+0000 7efd5d400640 1 -- 172.21.15.19:0/2387725028 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd581a1c50 con 0x7efd581018d0 2024-03-18T15:14:56.214 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:56.211+0000 7efd4f400640 1 -- 172.21.15.19:0/2387725028 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd400049e0 con 0x7efd581018d0 2024-03-18T15:14:56.214 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:56.211+0000 7efd5d400640 1 -- 172.21.15.19:0/2387725028 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd581a21a0 con 0x7efd581018d0 2024-03-18T15:14:56.215 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:56.213+0000 7efd5d400640 1 -- 172.21.15.19:0/2387725028 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd581a23d0 con 0x7efd581018d0 2024-03-18T15:14:56.215 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:56.214+0000 7efd4f400640 1 -- 172.21.15.19:0/2387725028 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7efd40005560 con 0x7efd581018d0 2024-03-18T15:14:56.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:56.216+0000 7efd4f400640 1 --2- 172.21.15.19:0/2387725028 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7efd3c0a2880 0x7efd3c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:14:56.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:56.216+0000 7efd4f400640 1 -- 172.21.15.19:0/2387725028 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7efd400ce5c0 con 0x7efd581018d0 2024-03-18T15:14:56.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:56.216+0000 7efd56800640 1 --2- 172.21.15.19:0/2387725028 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7efd3c0a2880 0x7efd3c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:14:56.218 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:56.217+0000 7efd56800640 1 --2- 172.21.15.19:0/2387725028 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7efd3c0a2880 0x7efd3c0a4d40 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7efd4802c190 tx=0x7efd48000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:14:56.221 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:56.220+0000 7efd4f400640 1 -- 172.21.15.19:0/2387725028 <== mon.1 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7efd400051d0 con 0x7efd581018d0 2024-03-18T15:14:56.392 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:56 smithi019 ceph-mon[31522]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:14:56.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:56 smithi104 ceph-mon[36070]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:14:56.472 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:56.471+0000 7efd5d400640 1 -- 172.21.15.19:0/2387725028 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efd581a2760 con 0x7efd581018d0 2024-03-18T15:14:56.473 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:56.472+0000 7efd4f400640 1 -- 172.21.15.19:0/2387725028 <== mon.1 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+7995 (secure 0 0 0) 0x7efd4009ba60 con 0x7efd581018d0 2024-03-18T15:14:56.473 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-18T15:14:56.473 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6835/1135329545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6834","nonce":1135329545},{"type":"v1","addr":"172.21.15.19:6835","nonce":1135329545}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6833/482725809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":482725809},{"type":"v1","addr":"172.21.15.104:6833","nonce":482725809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6833/2272279159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":2272279159},{"type":"v1","addr":"172.21.15.33:6833","nonce":2272279159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24469,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6837/1778019859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6836","nonce":1778019859},{"type":"v1","addr":"172.21.15.19:6837","nonce":1778019859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24482,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/4051613151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":4051613151},{"type":"v1","addr":"172.21.15.33:6835","nonce":4051613151}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24499,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3788595957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3788595957},{"type":"v1","addr":"172.21.15.104:6835","nonce":3788595957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24506,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/160187378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":160187378},{"type":"v1","addr":"172.21.15.33:6837","nonce":160187378}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24511,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6837/260984674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":260984674},{"type":"v1","addr":"172.21.15.104:6837","nonce":260984674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24529,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6839/805341436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6838","nonce":805341436},{"type":"v1","addr":"172.21.15.104:6839","nonce":805341436}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-03-18T15:14:56.474 INFO:teuthology.orchestra.run.smithi019.stderr:dumped fsmap epoch 10 2024-03-18T15:14:56.476 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:56.474+0000 7efd5d400640 1 -- 172.21.15.19:0/2387725028 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7efd3c0a2880 msgr2=0x7efd3c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:56.476 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:56.474+0000 7efd5d400640 1 --2- 172.21.15.19:0/2387725028 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7efd3c0a2880 0x7efd3c0a4d40 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7efd4802c190 tx=0x7efd48000bc0 comp rx=0 tx=0).stop 2024-03-18T15:14:56.476 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:56.474+0000 7efd5d400640 1 -- 172.21.15.19:0/2387725028 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efd581018d0 msgr2=0x7efd58102cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:56.476 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:56.474+0000 7efd5d400640 1 --2- 172.21.15.19:0/2387725028 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efd581018d0 0x7efd58102cb0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7efd4000e9f0 tx=0x7efd4000eeb0 comp rx=0 tx=0).stop 2024-03-18T15:14:56.476 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:56.475+0000 7efd5d400640 1 -- 172.21.15.19:0/2387725028 shutdown_connections 2024-03-18T15:14:56.476 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:56.475+0000 7efd5d400640 1 --2- 172.21.15.19:0/2387725028 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efd58102290 0x7efd581031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:56.476 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:56.475+0000 7efd5d400640 1 --2- 172.21.15.19:0/2387725028 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efd581018d0 0x7efd58102cb0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:56.476 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:56.475+0000 7efd5d400640 1 --2- 172.21.15.19:0/2387725028 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7efd3c0a2880 0x7efd3c0a4d40 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:56.477 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:56.475+0000 7efd5d400640 1 --2- 172.21.15.19:0/2387725028 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efd581006e0 0x7efd58104670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:56.477 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:56.475+0000 7efd5d400640 1 -- 172.21.15.19:0/2387725028 >> 172.21.15.19:0/2387725028 conn(0x7efd580fbc30 msgr2=0x7efd580fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:14:56.477 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:56.475+0000 7efd5d400640 1 -- 172.21.15.19:0/2387725028 shutdown_connections 2024-03-18T15:14:56.477 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:56.475+0000 7efd5d400640 1 -- 172.21.15.19:0/2387725028 wait complete. 2024-03-18T15:14:56.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:56 smithi033 ceph-mon[36648]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:14:57.082 INFO:journalctl@ceph.mds.g.smithi019.stdout:Mar 18 15:14:56 smithi019 podman[72907]: 2024-03-18 15:14:56.822308233 +0000 UTC m=+0.194354498 container create 939a29143c15835a69062bbfc9bdf99b705204671b92a06c26da1cc6a8de37c1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-g, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-03-18T15:14:57.342 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:57 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/2387725028' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:14:57.342 INFO:journalctl@ceph.mds.g.smithi019.stdout:Mar 18 15:14:57 smithi019 podman[72907]: 2024-03-18 15:14:57.201148057 +0000 UTC m=+0.573194324 container init 939a29143c15835a69062bbfc9bdf99b705204671b92a06c26da1cc6a8de37c1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-g, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-03-18T15:14:57.342 INFO:journalctl@ceph.mds.g.smithi019.stdout:Mar 18 15:14:57 smithi019 podman[72907]: 2024-03-18 15:14:57.212010405 +0000 UTC m=+0.584056702 container start 939a29143c15835a69062bbfc9bdf99b705204671b92a06c26da1cc6a8de37c1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-g, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-18T15:14:57.342 INFO:journalctl@ceph.mds.g.smithi019.stdout:Mar 18 15:14:57 smithi019 ceph-mds[72925]: -- 172.21.15.19:0/512787663 <== mon.0 v2:172.21.15.19:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d044ccba40 con 0x55d044c9ec00 2024-03-18T15:14:57.343 INFO:journalctl@ceph.mds.g.smithi019.stdout:Mar 18 15:14:57 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-g[72921]: starting mds.g at 2024-03-18T15:14:57.345 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:57 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/2387725028' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:14:57.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:57 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/2387725028' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:14:57.634 INFO:journalctl@ceph.mds.g.smithi019.stdout:Mar 18 15:14:57 smithi019 bash[72907]: 939a29143c15835a69062bbfc9bdf99b705204671b92a06c26da1cc6a8de37c1 2024-03-18T15:14:57.634 INFO:journalctl@ceph.mds.g.smithi019.stdout:Mar 18 15:14:57 smithi019 systemd[1]: Started Ceph mds.g for d4dc2ba6-e538-11ee-95c9-87774f69a715. 2024-03-18T15:14:58.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:58 smithi104 ceph-mon[36070]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:14:58.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:58 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:58.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:58 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:58.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:58 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:58.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:58 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T15:14:58.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:58 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T15:14:58.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:58 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:14:58.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:58 smithi104 ceph-mon[36070]: Deploying daemon mds.a on smithi019 2024-03-18T15:14:58.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:58 smithi019 ceph-mon[31522]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:14:58.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:58 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:58.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:58 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:58.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:58 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:58.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:58 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T15:14:58.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:58 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T15:14:58.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:58 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:14:58.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:58 smithi019 ceph-mon[31522]: Deploying daemon mds.a on smithi019 2024-03-18T15:14:58.496 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T15:14:58.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:58 smithi033 ceph-mon[36648]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:14:58.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:58 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:58.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:58 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:58.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:58 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:14:58.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:58 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T15:14:58.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:58 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T15:14:58.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:58 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:14:58.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:58 smithi033 ceph-mon[36648]: Deploying daemon mds.a on smithi019 2024-03-18T15:14:58.735 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:58.734+0000 7fcab8000640 1 -- 172.21.15.19:0/2952611788 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcab0100a50 msgr2=0x7fcab0100eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:58.736 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:58.734+0000 7fcab8000640 1 --2- 172.21.15.19:0/2952611788 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcab0100a50 0x7fcab0100eb0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fca9c009860 tx=0x7fca9c02bc20 comp rx=0 tx=0).stop 2024-03-18T15:14:58.736 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:58.734+0000 7fcab8000640 1 -- 172.21.15.19:0/2952611788 shutdown_connections 2024-03-18T15:14:58.736 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:58.734+0000 7fcab8000640 1 --2- 172.21.15.19:0/2952611788 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fcab01013f0 0x7fcab0101870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:58.736 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:58.734+0000 7fcab8000640 1 --2- 172.21.15.19:0/2952611788 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcab0100a50 0x7fcab0100eb0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:58.736 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:58.734+0000 7fcab8000640 1 --2- 172.21.15.19:0/2952611788 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcab01000f0 0x7fcab0100510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:58.736 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:58.734+0000 7fcab8000640 1 -- 172.21.15.19:0/2952611788 >> 172.21.15.19:0/2952611788 conn(0x7fcab00fbc70 msgr2=0x7fcab00fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:14:58.736 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:58.734+0000 7fcab8000640 1 -- 172.21.15.19:0/2952611788 shutdown_connections 2024-03-18T15:14:58.736 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:58.734+0000 7fcab8000640 1 -- 172.21.15.19:0/2952611788 wait complete. 2024-03-18T15:14:58.738 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:58.735+0000 7fcab8000640 1 Processor -- start 2024-03-18T15:14:58.738 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:58.735+0000 7fcab8000640 1 -- start start 2024-03-18T15:14:58.738 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:58.736+0000 7fcab8000640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcab01000f0 0x7fcab0199d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:14:58.738 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:58.736+0000 7fcab8000640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fcab0100a50 0x7fcab019a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:14:58.738 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:58.736+0000 7fcab8000640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcab01013f0 0x7fcab01a12a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:14:58.738 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:58.736+0000 7fcab8000640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fcab010f660 con 0x7fcab0100a50 2024-03-18T15:14:58.738 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:58.736+0000 7fcab8000640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fcab010f4e0 con 0x7fcab01013f0 2024-03-18T15:14:58.738 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:58.736+0000 7fcab8000640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fcab010f7e0 con 0x7fcab01000f0 2024-03-18T15:14:58.738 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:58.736+0000 7fcab4c00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fcab0100a50 0x7fcab019a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:14:58.738 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:58.736+0000 7fcab5600640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcab01000f0 0x7fcab0199d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:14:58.738 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:58.736+0000 7fcab5600640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcab01000f0 0x7fcab0199d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.19:53302/0 (socket says 172.21.15.19:53302) 2024-03-18T15:14:58.739 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:58.736+0000 7fcab5600640 1 -- 172.21.15.19:0/1836015228 learned_addr learned my addr 172.21.15.19:0/1836015228 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:14:58.739 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:58.736+0000 7fcab6000640 1 --2- 172.21.15.19:0/1836015228 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcab01013f0 0x7fcab01a12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:14:58.739 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:58.736+0000 7fcab5600640 1 -- 172.21.15.19:0/1836015228 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcab01013f0 msgr2=0x7fcab01a12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:58.739 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:58.736+0000 7fcab5600640 1 --2- 172.21.15.19:0/1836015228 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcab01013f0 0x7fcab01a12a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:58.739 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:58.736+0000 7fcab5600640 1 -- 172.21.15.19:0/1836015228 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fcab0100a50 msgr2=0x7fcab019a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:58.739 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:58.736+0000 7fcab5600640 1 --2- 172.21.15.19:0/1836015228 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fcab0100a50 0x7fcab019a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:58.739 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:58.736+0000 7fcab5600640 1 -- 172.21.15.19:0/1836015228 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcab01a1940 con 0x7fcab01000f0 2024-03-18T15:14:58.739 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:58.736+0000 7fcab6000640 1 --2- 172.21.15.19:0/1836015228 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcab01013f0 0x7fcab01a12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T15:14:58.739 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:58.737+0000 7fcab5600640 1 --2- 172.21.15.19:0/1836015228 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcab01000f0 0x7fcab0199d30 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fcaa400e9f0 tx=0x7fcaa400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:14:58.740 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:58.737+0000 7fcaae000640 1 -- 172.21.15.19:0/1836015228 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaa400cd80 con 0x7fcab01000f0 2024-03-18T15:14:58.740 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:58.737+0000 7fcaae000640 1 -- 172.21.15.19:0/1836015228 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcaa4010910 con 0x7fcab01000f0 2024-03-18T15:14:58.740 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:58.737+0000 7fcab8000640 1 -- 172.21.15.19:0/1836015228 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcab01a1c30 con 0x7fcab01000f0 2024-03-18T15:14:58.740 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:58.737+0000 7fcaae000640 1 -- 172.21.15.19:0/1836015228 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaa40188f0 con 0x7fcab01000f0 2024-03-18T15:14:58.740 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:58.737+0000 7fcab8000640 1 -- 172.21.15.19:0/1836015228 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcab01a2180 con 0x7fcab01000f0 2024-03-18T15:14:58.740 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:58.738+0000 7fcab8000640 1 -- 172.21.15.19:0/1836015228 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcab010afc0 con 0x7fcab01000f0 2024-03-18T15:14:58.741 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:58.740+0000 7fcaae000640 1 -- 172.21.15.19:0/1836015228 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fcaa4010ac0 con 0x7fcab01000f0 2024-03-18T15:14:58.743 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:58.742+0000 7fcaae000640 1 --2- 172.21.15.19:0/1836015228 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fca980a2800 0x7fca980a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:14:58.743 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:58.742+0000 7fcaae000640 1 -- 172.21.15.19:0/1836015228 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fcaa40ceb40 con 0x7fcab01000f0 2024-03-18T15:14:58.743 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:58.742+0000 7fcab4c00640 1 --2- 172.21.15.19:0/1836015228 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fca980a2800 0x7fca980a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:14:58.744 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:58.743+0000 7fcab4c00640 1 --2- 172.21.15.19:0/1836015228 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fca980a2800 0x7fca980a4cc0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fca9c009830 tx=0x7fca9c034b50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:14:58.747 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:58.746+0000 7fcaae000640 1 -- 172.21.15.19:0/1836015228 <== mon.1 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fcaa409bfe0 con 0x7fcab01000f0 2024-03-18T15:14:58.996 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:58.995+0000 7fcab8000640 1 -- 172.21.15.19:0/1836015228 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcab00623e0 con 0x7fcab01000f0 2024-03-18T15:14:58.996 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:58.996+0000 7fcaae000640 1 -- 172.21.15.19:0/1836015228 <== mon.1 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8821 (secure 0 0 0) 0x7fcaa409b8e0 con 0x7fcab01000f0 2024-03-18T15:14:58.997 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-18T15:14:58.997 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6835/1135329545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6834","nonce":1135329545},{"type":"v1","addr":"172.21.15.19:6835","nonce":1135329545}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14457,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6839/3833518557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6838","nonce":3833518557},{"type":"v1","addr":"172.21.15.19:6839","nonce":3833518557}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6833/482725809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":482725809},{"type":"v1","addr":"172.21.15.104:6833","nonce":482725809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6833/2272279159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":2272279159},{"type":"v1","addr":"172.21.15.33:6833","nonce":2272279159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24469,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6837/1778019859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6836","nonce":1778019859},{"type":"v1","addr":"172.21.15.19:6837","nonce":1778019859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24482,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/4051613151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":4051613151},{"type":"v1","addr":"172.21.15.33:6835","nonce":4051613151}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24499,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3788595957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3788595957},{"type":"v1","addr":"172.21.15.104:6835","nonce":3788595957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24506,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/160187378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":160187378},{"type":"v1","addr":"172.21.15.33:6837","nonce":160187378}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24511,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6837/260984674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":260984674},{"type":"v1","addr":"172.21.15.104:6837","nonce":260984674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24529,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6839/805341436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6838","nonce":805341436},{"type":"v1","addr":"172.21.15.104:6839","nonce":805341436}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-03-18T15:14:58.997 INFO:teuthology.orchestra.run.smithi019.stderr:dumped fsmap epoch 11 2024-03-18T15:14:59.002 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:58.998+0000 7fcab8000640 1 -- 172.21.15.19:0/1836015228 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fca980a2800 msgr2=0x7fca980a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:59.002 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:58.998+0000 7fcab8000640 1 --2- 172.21.15.19:0/1836015228 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fca980a2800 0x7fca980a4cc0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fca9c009830 tx=0x7fca9c034b50 comp rx=0 tx=0).stop 2024-03-18T15:14:59.002 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:58.998+0000 7fcab8000640 1 -- 172.21.15.19:0/1836015228 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcab01000f0 msgr2=0x7fcab0199d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:14:59.002 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:58.998+0000 7fcab8000640 1 --2- 172.21.15.19:0/1836015228 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcab01000f0 0x7fcab0199d30 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fcaa400e9f0 tx=0x7fcaa400eeb0 comp rx=0 tx=0).stop 2024-03-18T15:14:59.002 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:58.998+0000 7fcab8000640 1 -- 172.21.15.19:0/1836015228 shutdown_connections 2024-03-18T15:14:59.002 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:58.998+0000 7fcab8000640 1 --2- 172.21.15.19:0/1836015228 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fca980a2800 0x7fca980a4cc0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:59.002 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:58.998+0000 7fcab8000640 1 --2- 172.21.15.19:0/1836015228 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcab01013f0 0x7fcab01a12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:59.002 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:58.998+0000 7fcab8000640 1 --2- 172.21.15.19:0/1836015228 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fcab0100a50 0x7fcab019a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:59.002 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:58.998+0000 7fcab8000640 1 --2- 172.21.15.19:0/1836015228 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcab01000f0 0x7fcab0199d30 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:14:59.002 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:58.998+0000 7fcab8000640 1 -- 172.21.15.19:0/1836015228 >> 172.21.15.19:0/1836015228 conn(0x7fcab00fbc70 msgr2=0x7fcab00fd380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:14:59.002 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:58.999+0000 7fcab8000640 1 -- 172.21.15.19:0/1836015228 shutdown_connections 2024-03-18T15:14:59.002 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:14:58.999+0000 7fcab8000640 1 -- 172.21.15.19:0/1836015228 wait complete. 2024-03-18T15:14:59.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:59 smithi019 ceph-mon[31522]: mds.? [v2:172.21.15.19:6838/3833518557,v1:172.21.15.19:6839/3833518557] up:boot 2024-03-18T15:14:59.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:59 smithi019 ceph-mon[31522]: fsmap 10 up:standby 2024-03-18T15:14:59.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:59 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-18T15:14:59.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:59 smithi019 ceph-mon[31522]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:14:59.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:14:59 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/1836015228' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:14:59.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:59 smithi104 ceph-mon[36070]: mds.? [v2:172.21.15.19:6838/3833518557,v1:172.21.15.19:6839/3833518557] up:boot 2024-03-18T15:14:59.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:59 smithi104 ceph-mon[36070]: fsmap 10 up:standby 2024-03-18T15:14:59.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:59 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-18T15:14:59.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:59 smithi104 ceph-mon[36070]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:14:59.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:14:59 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/1836015228' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:14:59.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:59 smithi033 ceph-mon[36648]: mds.? [v2:172.21.15.19:6838/3833518557,v1:172.21.15.19:6839/3833518557] up:boot 2024-03-18T15:14:59.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:59 smithi033 ceph-mon[36648]: fsmap 10 up:standby 2024-03-18T15:14:59.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:59 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-18T15:14:59.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:59 smithi033 ceph-mon[36648]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:14:59.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:14:59 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/1836015228' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:15:00.966 INFO:journalctl@ceph.mds.a.smithi019.stdout:Mar 18 15:15:00 smithi019 systemd[1]: Starting Ceph mds.a for d4dc2ba6-e538-11ee-95c9-87774f69a715... 2024-03-18T15:15:01.020 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T15:15:01.251 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:01.249+0000 7fb712a00640 1 -- 172.21.15.19:0/2624144338 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb70c1099b0 msgr2=0x7fb70c110240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:01.251 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:01.249+0000 7fb712a00640 1 --2- 172.21.15.19:0/2624144338 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb70c1099b0 0x7fb70c110240 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb704009ca0 tx=0x7fb70402c100 comp rx=0 tx=0).stop 2024-03-18T15:15:01.251 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:01.249+0000 7fb712a00640 1 -- 172.21.15.19:0/2624144338 shutdown_connections 2024-03-18T15:15:01.251 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:01.249+0000 7fb712a00640 1 --2- 172.21.15.19:0/2624144338 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb70c1099b0 0x7fb70c110240 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:01.251 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:01.249+0000 7fb712a00640 1 --2- 172.21.15.19:0/2624144338 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb70c108ff0 0x7fb70c109470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:01.251 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:01.249+0000 7fb712a00640 1 --2- 172.21.15.19:0/2624144338 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb70c107e00 0x7fb70c108220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:01.252 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:01.249+0000 7fb712a00640 1 -- 172.21.15.19:0/2624144338 >> 172.21.15.19:0/2624144338 conn(0x7fb70c076dc0 msgr2=0x7fb70c0791e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:01.252 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:01.250+0000 7fb712a00640 1 -- 172.21.15.19:0/2624144338 shutdown_connections 2024-03-18T15:15:01.252 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:01.250+0000 7fb712a00640 1 -- 172.21.15.19:0/2624144338 wait complete. 2024-03-18T15:15:01.254 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:01.251+0000 7fb712a00640 1 Processor -- start 2024-03-18T15:15:01.254 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:01.251+0000 7fb712a00640 1 -- start start 2024-03-18T15:15:01.254 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:01.251+0000 7fb712a00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb70c107e00 0x7fb70c19e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:01.254 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:01.251+0000 7fb712a00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb70c108ff0 0x7fb70c19e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:01.254 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:01.251+0000 7fb712a00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb70c1099b0 0x7fb70c1a5640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:01.254 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:01.251+0000 7fb712a00640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fb70c113a80 con 0x7fb70c1099b0 2024-03-18T15:15:01.254 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:01.251+0000 7fb712a00640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb70c113900 con 0x7fb70c107e00 2024-03-18T15:15:01.254 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:01.251+0000 7fb712a00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fb70c113c00 con 0x7fb70c108ff0 2024-03-18T15:15:01.254 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:01.251+0000 7fb70be00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb70c107e00 0x7fb70c19e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:01.254 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:01.251+0000 7fb70be00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb70c107e00 0x7fb70c19e090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:46172/0 (socket says 172.21.15.19:46172) 2024-03-18T15:15:01.255 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:01.251+0000 7fb70be00640 1 -- 172.21.15.19:0/1051303919 learned_addr learned my addr 172.21.15.19:0/1051303919 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:15:01.255 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:01.251+0000 7fb710a00640 1 --2- 172.21.15.19:0/1051303919 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb70c1099b0 0x7fb70c1a5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:01.255 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:01.251+0000 7fb703e00640 1 --2- 172.21.15.19:0/1051303919 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb70c108ff0 0x7fb70c19e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:01.255 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:01.252+0000 7fb70be00640 1 -- 172.21.15.19:0/1051303919 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb70c108ff0 msgr2=0x7fb70c19e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:01.255 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:01.252+0000 7fb70be00640 1 --2- 172.21.15.19:0/1051303919 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb70c108ff0 0x7fb70c19e5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:01.255 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:01.252+0000 7fb70be00640 1 -- 172.21.15.19:0/1051303919 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb70c1099b0 msgr2=0x7fb70c1a5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:01.255 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:01.252+0000 7fb70be00640 1 --2- 172.21.15.19:0/1051303919 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb70c1099b0 0x7fb70c1a5640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:01.255 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:01.252+0000 7fb70be00640 1 -- 172.21.15.19:0/1051303919 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb70c1a5cb0 con 0x7fb70c107e00 2024-03-18T15:15:01.255 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:01.252+0000 7fb70be00640 1 --2- 172.21.15.19:0/1051303919 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb70c107e00 0x7fb70c19e090 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb6fc00e9c0 tx=0x7fb6fc00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:01.255 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:01.252+0000 7fb709600640 1 -- 172.21.15.19:0/1051303919 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6fc00cd80 con 0x7fb70c107e00 2024-03-18T15:15:01.255 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:01.252+0000 7fb709600640 1 -- 172.21.15.19:0/1051303919 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb6fc0035f0 con 0x7fb70c107e00 2024-03-18T15:15:01.255 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:01.252+0000 7fb712a00640 1 -- 172.21.15.19:0/1051303919 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb70c1a5fa0 con 0x7fb70c107e00 2024-03-18T15:15:01.255 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:01.252+0000 7fb709600640 1 -- 172.21.15.19:0/1051303919 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6fc010690 con 0x7fb70c107e00 2024-03-18T15:15:01.255 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:01.253+0000 7fb712a00640 1 -- 172.21.15.19:0/1051303919 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb70c1a63b0 con 0x7fb70c107e00 2024-03-18T15:15:01.256 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:01.254+0000 7fb712a00640 1 -- 172.21.15.19:0/1051303919 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb70c10f0c0 con 0x7fb70c107e00 2024-03-18T15:15:01.256 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:01.256+0000 7fb709600640 1 -- 172.21.15.19:0/1051303919 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fb6fc0037a0 con 0x7fb70c107e00 2024-03-18T15:15:01.258 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:01.257+0000 7fb709600640 1 --2- 172.21.15.19:0/1051303919 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fb6f00a2800 0x7fb6f00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:01.258 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:01.257+0000 7fb709600640 1 -- 172.21.15.19:0/1051303919 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fb6fc0ce9c0 con 0x7fb70c107e00 2024-03-18T15:15:01.258 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:01.257+0000 7fb703e00640 1 --2- 172.21.15.19:0/1051303919 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fb6f00a2800 0x7fb6f00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:01.259 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:01.258+0000 7fb703e00640 1 --2- 172.21.15.19:0/1051303919 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fb6f00a2800 0x7fb6f00a4cc0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fb70c19f5a0 tx=0x7fb6f4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:01.263 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:01.262+0000 7fb709600640 1 -- 172.21.15.19:0/1051303919 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb6fc09be60 con 0x7fb70c107e00 2024-03-18T15:15:01.340 INFO:journalctl@ceph.mds.a.smithi019.stdout:Mar 18 15:15:01 smithi019 podman[73249]: 2024-03-18 15:15:00.923910497 +0000 UTC m=+0.038505475 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T15:15:01.340 INFO:journalctl@ceph.mds.a.smithi019.stdout:Mar 18 15:15:01 smithi019 podman[73249]: 2024-03-18 15:15:01.043646986 +0000 UTC m=+0.158242001 container create a8b0d96b5a0e94c7b33d7fc240b564844c8270741a9da1240f02ee77d01cff6c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-a, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0) 2024-03-18T15:15:01.340 INFO:journalctl@ceph.mds.a.smithi019.stdout:Mar 18 15:15:01 smithi019 podman[73249]: 2024-03-18 15:15:01.339151837 +0000 UTC m=+0.453746859 container init a8b0d96b5a0e94c7b33d7fc240b564844c8270741a9da1240f02ee77d01cff6c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD) 2024-03-18T15:15:01.532 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:01.531+0000 7fb712a00640 1 -- 172.21.15.19:0/1051303919 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb70c02d1b0 con 0x7fb70c107e00 2024-03-18T15:15:01.533 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:01.532+0000 7fb709600640 1 -- 172.21.15.19:0/1051303919 <== mon.2 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8821 (secure 0 0 0) 0x7fb6fc09b760 con 0x7fb70c107e00 2024-03-18T15:15:01.533 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-18T15:15:01.533 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6835/1135329545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6834","nonce":1135329545},{"type":"v1","addr":"172.21.15.19:6835","nonce":1135329545}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14457,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6839/3833518557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6838","nonce":3833518557},{"type":"v1","addr":"172.21.15.19:6839","nonce":3833518557}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6833/482725809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":482725809},{"type":"v1","addr":"172.21.15.104:6833","nonce":482725809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6833/2272279159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":2272279159},{"type":"v1","addr":"172.21.15.33:6833","nonce":2272279159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24469,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6837/1778019859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6836","nonce":1778019859},{"type":"v1","addr":"172.21.15.19:6837","nonce":1778019859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24482,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/4051613151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":4051613151},{"type":"v1","addr":"172.21.15.33:6835","nonce":4051613151}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24499,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3788595957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3788595957},{"type":"v1","addr":"172.21.15.104:6835","nonce":3788595957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24506,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/160187378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":160187378},{"type":"v1","addr":"172.21.15.33:6837","nonce":160187378}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24511,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6837/260984674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":260984674},{"type":"v1","addr":"172.21.15.104:6837","nonce":260984674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24529,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6839/805341436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6838","nonce":805341436},{"type":"v1","addr":"172.21.15.104:6839","nonce":805341436}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-03-18T15:15:01.534 INFO:teuthology.orchestra.run.smithi019.stderr:dumped fsmap epoch 11 2024-03-18T15:15:01.536 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:01.535+0000 7fb712a00640 1 -- 172.21.15.19:0/1051303919 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fb6f00a2800 msgr2=0x7fb6f00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:01.536 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:01.535+0000 7fb712a00640 1 --2- 172.21.15.19:0/1051303919 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fb6f00a2800 0x7fb6f00a4cc0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fb70c19f5a0 tx=0x7fb6f4005f90 comp rx=0 tx=0).stop 2024-03-18T15:15:01.536 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:01.535+0000 7fb712a00640 1 -- 172.21.15.19:0/1051303919 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb70c107e00 msgr2=0x7fb70c19e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:01.536 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:01.535+0000 7fb712a00640 1 --2- 172.21.15.19:0/1051303919 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb70c107e00 0x7fb70c19e090 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb6fc00e9c0 tx=0x7fb6fc00ee80 comp rx=0 tx=0).stop 2024-03-18T15:15:01.536 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:01.535+0000 7fb712a00640 1 -- 172.21.15.19:0/1051303919 shutdown_connections 2024-03-18T15:15:01.536 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:01.535+0000 7fb712a00640 1 --2- 172.21.15.19:0/1051303919 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb70c1099b0 0x7fb70c1a5640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:01.536 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:01.535+0000 7fb712a00640 1 --2- 172.21.15.19:0/1051303919 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb70c108ff0 0x7fb70c19e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:01.537 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:01.535+0000 7fb712a00640 1 --2- 172.21.15.19:0/1051303919 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fb6f00a2800 0x7fb6f00a4cc0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:01.537 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:01.535+0000 7fb712a00640 1 --2- 172.21.15.19:0/1051303919 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb70c107e00 0x7fb70c19e090 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:01.537 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:01.535+0000 7fb712a00640 1 -- 172.21.15.19:0/1051303919 >> 172.21.15.19:0/1051303919 conn(0x7fb70c076dc0 msgr2=0x7fb70c10c290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:01.537 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:01.535+0000 7fb712a00640 1 -- 172.21.15.19:0/1051303919 shutdown_connections 2024-03-18T15:15:01.537 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:01.535+0000 7fb712a00640 1 -- 172.21.15.19:0/1051303919 wait complete. 2024-03-18T15:15:01.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:01 smithi104 ceph-mon[36070]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:15:01.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:01 smithi019 ceph-mon[31522]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:15:01.715 INFO:journalctl@ceph.mds.a.smithi019.stdout:Mar 18 15:15:01 smithi019 podman[73249]: 2024-03-18 15:15:01.350730558 +0000 UTC m=+0.465325607 container start a8b0d96b5a0e94c7b33d7fc240b564844c8270741a9da1240f02ee77d01cff6c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True) 2024-03-18T15:15:01.715 INFO:journalctl@ceph.mds.a.smithi019.stdout:Mar 18 15:15:01 smithi019 ceph-mds[73307]: -- 172.21.15.19:0/2011377561 <== mon.1 v2:172.21.15.104:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x555a311f5a40 con 0x555a311c9000 2024-03-18T15:15:01.716 INFO:journalctl@ceph.mds.a.smithi019.stdout:Mar 18 15:15:01 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-a[73284]: starting mds.a at 2024-03-18T15:15:01.716 INFO:journalctl@ceph.mds.a.smithi019.stdout:Mar 18 15:15:01 smithi019 bash[73249]: a8b0d96b5a0e94c7b33d7fc240b564844c8270741a9da1240f02ee77d01cff6c 2024-03-18T15:15:01.716 INFO:journalctl@ceph.mds.a.smithi019.stdout:Mar 18 15:15:01 smithi019 systemd[1]: Started Ceph mds.a for d4dc2ba6-e538-11ee-95c9-87774f69a715. 2024-03-18T15:15:01.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:01 smithi033 ceph-mon[36648]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:15:02.585 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:02 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/1051303919' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:15:02.585 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:02 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:15:02.585 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:02 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:15:02.585 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:02 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:15:02.585 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:02 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T15:15:02.585 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:02 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T15:15:02.585 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:02 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:15:02.585 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:02 smithi033 ceph-mon[36648]: Deploying daemon mds.b on smithi033 2024-03-18T15:15:02.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:02 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/1051303919' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:15:02.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:02 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:15:02.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:02 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:15:02.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:02 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:15:02.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:02 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T15:15:02.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:02 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T15:15:02.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:02 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:15:02.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:02 smithi104 ceph-mon[36070]: Deploying daemon mds.b on smithi033 2024-03-18T15:15:02.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:02 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/1051303919' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:15:02.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:02 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:15:02.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:02 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:15:02.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:02 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:15:02.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:02 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T15:15:02.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:02 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T15:15:02.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:02 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:15:02.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:02 smithi019 ceph-mon[31522]: Deploying daemon mds.b on smithi033 2024-03-18T15:15:03.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:03 smithi033 ceph-mon[36648]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:15:03.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:03 smithi033 ceph-mon[36648]: mds.? [v2:172.21.15.19:6840/3176583319,v1:172.21.15.19:6841/3176583319] up:boot 2024-03-18T15:15:03.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:03 smithi033 ceph-mon[36648]: fsmap 11 up:standby 2024-03-18T15:15:03.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:03 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-18T15:15:03.561 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T15:15:03.653 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:03 smithi019 ceph-mon[31522]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:15:03.653 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:03 smithi019 ceph-mon[31522]: mds.? [v2:172.21.15.19:6840/3176583319,v1:172.21.15.19:6841/3176583319] up:boot 2024-03-18T15:15:03.653 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:03 smithi019 ceph-mon[31522]: fsmap 11 up:standby 2024-03-18T15:15:03.653 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:03 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-18T15:15:03.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:03 smithi104 ceph-mon[36070]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:15:03.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:03 smithi104 ceph-mon[36070]: mds.? [v2:172.21.15.19:6840/3176583319,v1:172.21.15.19:6841/3176583319] up:boot 2024-03-18T15:15:03.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:03 smithi104 ceph-mon[36070]: fsmap 11 up:standby 2024-03-18T15:15:03.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:03 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-18T15:15:03.793 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:03.792+0000 7fc8d0c00640 1 -- 172.21.15.19:0/3515570517 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc8cc1027d0 msgr2=0x7fc8cc102c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:03.794 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:03.792+0000 7fc8d0c00640 1 --2- 172.21.15.19:0/3515570517 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc8cc1027d0 0x7fc8cc102c50 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc8b4009900 tx=0x7fc8b402bc40 comp rx=0 tx=0).stop 2024-03-18T15:15:03.794 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:03.792+0000 7fc8d0c00640 1 -- 172.21.15.19:0/3515570517 shutdown_connections 2024-03-18T15:15:03.794 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:03.792+0000 7fc8d0c00640 1 --2- 172.21.15.19:0/3515570517 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc8cc103190 0x7fc8cc109a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:03.794 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:03.792+0000 7fc8d0c00640 1 --2- 172.21.15.19:0/3515570517 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc8cc1027d0 0x7fc8cc102c50 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:03.794 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:03.792+0000 7fc8d0c00640 1 --2- 172.21.15.19:0/3515570517 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc8cc1015e0 0x7fc8cc101a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:03.794 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:03.792+0000 7fc8d0c00640 1 -- 172.21.15.19:0/3515570517 >> 172.21.15.19:0/3515570517 conn(0x7fc8cc0fcb70 msgr2=0x7fc8cc0fef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:03.794 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:03.792+0000 7fc8d0c00640 1 -- 172.21.15.19:0/3515570517 shutdown_connections 2024-03-18T15:15:03.794 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:03.792+0000 7fc8d0c00640 1 -- 172.21.15.19:0/3515570517 wait complete. 2024-03-18T15:15:03.796 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:03.793+0000 7fc8d0c00640 1 Processor -- start 2024-03-18T15:15:03.796 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:03.793+0000 7fc8d0c00640 1 -- start start 2024-03-18T15:15:03.796 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:03.794+0000 7fc8d0c00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc8cc1015e0 0x7fc8cc075590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:03.796 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:03.794+0000 7fc8d0c00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc8cc1027d0 0x7fc8cc073bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:03.796 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:03.794+0000 7fc8d0c00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc8cc103190 0x7fc8cc074110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:03.796 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:03.794+0000 7fc8d0c00640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fc8cc10d2c0 con 0x7fc8cc1027d0 2024-03-18T15:15:03.797 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:03.794+0000 7fc8d0c00640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc8cc10d140 con 0x7fc8cc103190 2024-03-18T15:15:03.797 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:03.794+0000 7fc8d0c00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fc8cc10d440 con 0x7fc8cc1015e0 2024-03-18T15:15:03.797 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:03.794+0000 7fc8c3e00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc8cc1027d0 0x7fc8cc073bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:03.797 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:03.794+0000 7fc8cbe00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc8cc103190 0x7fc8cc074110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:03.797 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:03.794+0000 7fc8c3e00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc8cc1027d0 0x7fc8cc073bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:59254/0 (socket says 172.21.15.19:59254) 2024-03-18T15:15:03.797 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:03.794+0000 7fc8c3e00640 1 -- 172.21.15.19:0/3589432552 learned_addr learned my addr 172.21.15.19:0/3589432552 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:15:03.797 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:03.794+0000 7fc8cb400640 1 --2- 172.21.15.19:0/3589432552 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc8cc1015e0 0x7fc8cc075590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:03.797 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:03.795+0000 7fc8cbe00640 1 -- 172.21.15.19:0/3589432552 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc8cc1015e0 msgr2=0x7fc8cc075590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:03.797 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:03.795+0000 7fc8cbe00640 1 --2- 172.21.15.19:0/3589432552 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc8cc1015e0 0x7fc8cc075590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:03.797 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:03.795+0000 7fc8cbe00640 1 -- 172.21.15.19:0/3589432552 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc8cc1027d0 msgr2=0x7fc8cc073bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:03.797 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:03.795+0000 7fc8cbe00640 1 --2- 172.21.15.19:0/3589432552 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc8cc1027d0 0x7fc8cc073bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:03.797 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:03.795+0000 7fc8cbe00640 1 -- 172.21.15.19:0/3589432552 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8cc0749a0 con 0x7fc8cc103190 2024-03-18T15:15:03.797 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:03.795+0000 7fc8cbe00640 1 --2- 172.21.15.19:0/3589432552 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc8cc103190 0x7fc8cc074110 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc8c4009fd0 tx=0x7fc8c400ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:03.797 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:03.795+0000 7fc8c8c00640 1 -- 172.21.15.19:0/3589432552 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8c400ce20 con 0x7fc8cc103190 2024-03-18T15:15:03.798 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:03.795+0000 7fc8c8c00640 1 -- 172.21.15.19:0/3589432552 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc8c4005020 con 0x7fc8cc103190 2024-03-18T15:15:03.798 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:03.795+0000 7fc8c8c00640 1 -- 172.21.15.19:0/3589432552 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8c4010720 con 0x7fc8cc103190 2024-03-18T15:15:03.798 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:03.795+0000 7fc8d0c00640 1 -- 172.21.15.19:0/3589432552 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8cc1a19d0 con 0x7fc8cc103190 2024-03-18T15:15:03.798 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:03.795+0000 7fc8d0c00640 1 -- 172.21.15.19:0/3589432552 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8cc1a1f40 con 0x7fc8cc103190 2024-03-18T15:15:03.798 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:03.797+0000 7fc8d0c00640 1 -- 172.21.15.19:0/3589432552 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8cc1088a0 con 0x7fc8cc103190 2024-03-18T15:15:03.799 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:03.798+0000 7fc8c8c00640 1 -- 172.21.15.19:0/3589432552 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fc8c4005540 con 0x7fc8cc103190 2024-03-18T15:15:03.801 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:03.799+0000 7fc8c8c00640 1 --2- 172.21.15.19:0/3589432552 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fc8b00a2880 0x7fc8b00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:03.801 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:03.800+0000 7fc8c8c00640 1 -- 172.21.15.19:0/3589432552 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fc8c40ceba0 con 0x7fc8cc103190 2024-03-18T15:15:03.801 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:03.800+0000 7fc8cb400640 1 --2- 172.21.15.19:0/3589432552 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fc8b00a2880 0x7fc8b00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:03.801 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:03.801+0000 7fc8cb400640 1 --2- 172.21.15.19:0/3589432552 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fc8b00a2880 0x7fc8b00a4d40 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fc8bc005fd0 tx=0x7fc8bc00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:03.805 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:03.804+0000 7fc8c8c00640 1 -- 172.21.15.19:0/3589432552 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc8c409c040 con 0x7fc8cc103190 2024-03-18T15:15:03.983 INFO:journalctl@ceph.mds.b.smithi033.stdout:Mar 18 15:15:03 smithi033 systemd[1]: Starting Ceph mds.b for d4dc2ba6-e538-11ee-95c9-87774f69a715... 2024-03-18T15:15:04.047 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:04.046+0000 7fc8d0c00640 1 -- 172.21.15.19:0/3589432552 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc8cc02d570 con 0x7fc8cc103190 2024-03-18T15:15:04.048 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:04.047+0000 7fc8c8c00640 1 -- 172.21.15.19:0/3589432552 <== mon.2 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9647 (secure 0 0 0) 0x7fc8c409b940 con 0x7fc8cc103190 2024-03-18T15:15:04.049 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-18T15:15:04.049 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6835/1135329545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6834","nonce":1135329545},{"type":"v1","addr":"172.21.15.19:6835","nonce":1135329545}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14457,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6839/3833518557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6838","nonce":3833518557},{"type":"v1","addr":"172.21.15.19:6839","nonce":3833518557}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14475,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6841/3176583319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6840","nonce":3176583319},{"type":"v1","addr":"172.21.15.19:6841","nonce":3176583319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6833/482725809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":482725809},{"type":"v1","addr":"172.21.15.104:6833","nonce":482725809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6833/2272279159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":2272279159},{"type":"v1","addr":"172.21.15.33:6833","nonce":2272279159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24469,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6837/1778019859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6836","nonce":1778019859},{"type":"v1","addr":"172.21.15.19:6837","nonce":1778019859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24482,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/4051613151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":4051613151},{"type":"v1","addr":"172.21.15.33:6835","nonce":4051613151}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24499,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3788595957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3788595957},{"type":"v1","addr":"172.21.15.104:6835","nonce":3788595957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24506,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/160187378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":160187378},{"type":"v1","addr":"172.21.15.33:6837","nonce":160187378}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24511,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6837/260984674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":260984674},{"type":"v1","addr":"172.21.15.104:6837","nonce":260984674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24529,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6839/805341436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6838","nonce":805341436},{"type":"v1","addr":"172.21.15.104:6839","nonce":805341436}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-03-18T15:15:04.049 INFO:teuthology.orchestra.run.smithi019.stderr:dumped fsmap epoch 12 2024-03-18T15:15:04.051 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:04.050+0000 7fc8d0c00640 1 -- 172.21.15.19:0/3589432552 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fc8b00a2880 msgr2=0x7fc8b00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:04.051 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:04.050+0000 7fc8d0c00640 1 --2- 172.21.15.19:0/3589432552 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fc8b00a2880 0x7fc8b00a4d40 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fc8bc005fd0 tx=0x7fc8bc00b410 comp rx=0 tx=0).stop 2024-03-18T15:15:04.052 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:04.050+0000 7fc8d0c00640 1 -- 172.21.15.19:0/3589432552 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc8cc103190 msgr2=0x7fc8cc074110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:04.052 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:04.050+0000 7fc8d0c00640 1 --2- 172.21.15.19:0/3589432552 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc8cc103190 0x7fc8cc074110 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc8c4009fd0 tx=0x7fc8c400ef10 comp rx=0 tx=0).stop 2024-03-18T15:15:04.052 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:04.050+0000 7fc8d0c00640 1 -- 172.21.15.19:0/3589432552 shutdown_connections 2024-03-18T15:15:04.052 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:04.050+0000 7fc8d0c00640 1 --2- 172.21.15.19:0/3589432552 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fc8b00a2880 0x7fc8b00a4d40 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:04.052 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:04.050+0000 7fc8d0c00640 1 --2- 172.21.15.19:0/3589432552 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc8cc103190 0x7fc8cc074110 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:04.052 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:04.050+0000 7fc8d0c00640 1 --2- 172.21.15.19:0/3589432552 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc8cc1027d0 0x7fc8cc073bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:04.052 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:04.050+0000 7fc8d0c00640 1 --2- 172.21.15.19:0/3589432552 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc8cc1015e0 0x7fc8cc075590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:04.052 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:04.050+0000 7fc8d0c00640 1 -- 172.21.15.19:0/3589432552 >> 172.21.15.19:0/3589432552 conn(0x7fc8cc0fcb70 msgr2=0x7fc8cc105a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:04.052 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:04.050+0000 7fc8d0c00640 1 -- 172.21.15.19:0/3589432552 shutdown_connections 2024-03-18T15:15:04.052 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:04.050+0000 7fc8d0c00640 1 -- 172.21.15.19:0/3589432552 wait complete. 2024-03-18T15:15:04.257 INFO:journalctl@ceph.mds.b.smithi033.stdout:Mar 18 15:15:03 smithi033 podman[67584]: 2024-03-18 15:15:03.88583968 +0000 UTC m=+0.018307861 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T15:15:04.257 INFO:journalctl@ceph.mds.b.smithi033.stdout:Mar 18 15:15:03 smithi033 podman[67584]: 2024-03-18 15:15:03.992123097 +0000 UTC m=+0.124591316 container create 79906bb04e4463e396726f62ab3386b98ad98a96ef6db02808826ad54bea588a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-b, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-18T15:15:04.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:04 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/3589432552' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:15:04.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:04 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/3589432552' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:15:04.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:04 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/3589432552' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:15:04.757 INFO:journalctl@ceph.mds.b.smithi033.stdout:Mar 18 15:15:04 smithi033 podman[67584]: 2024-03-18 15:15:04.261386683 +0000 UTC m=+0.393854866 container init 79906bb04e4463e396726f62ab3386b98ad98a96ef6db02808826ad54bea588a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-b, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307) 2024-03-18T15:15:04.757 INFO:journalctl@ceph.mds.b.smithi033.stdout:Mar 18 15:15:04 smithi033 podman[67584]: 2024-03-18 15:15:04.26408999 +0000 UTC m=+0.396558167 container start 79906bb04e4463e396726f62ab3386b98ad98a96ef6db02808826ad54bea588a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-b, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5) 2024-03-18T15:15:04.757 INFO:journalctl@ceph.mds.b.smithi033.stdout:Mar 18 15:15:04 smithi033 ceph-mds[67603]: -- 172.21.15.33:0/2462477301 <== mon.2 v2:172.21.15.33:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e307cd5a40 con 0x55e307ca9000 2024-03-18T15:15:04.757 INFO:journalctl@ceph.mds.b.smithi033.stdout:Mar 18 15:15:04 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-b[67599]: starting mds.b at 2024-03-18T15:15:04.757 INFO:journalctl@ceph.mds.b.smithi033.stdout:Mar 18 15:15:04 smithi033 bash[67584]: 79906bb04e4463e396726f62ab3386b98ad98a96ef6db02808826ad54bea588a 2024-03-18T15:15:04.757 INFO:journalctl@ceph.mds.b.smithi033.stdout:Mar 18 15:15:04 smithi033 systemd[1]: Started Ceph mds.b for d4dc2ba6-e538-11ee-95c9-87774f69a715. 2024-03-18T15:15:05.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:05 smithi104 ceph-mon[36070]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:15:05.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:05 smithi104 ceph-mon[36070]: mds.? [v2:172.21.15.33:6838/1903591535,v1:172.21.15.33:6839/1903591535] up:boot 2024-03-18T15:15:05.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:05 smithi104 ceph-mon[36070]: fsmap 12 up:standby 2024-03-18T15:15:05.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:05 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-18T15:15:05.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:05 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:15:05.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:05 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:15:05.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:05 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:15:05.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:05 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:15:05.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:05 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:15:05.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:05 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:15:05.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:05 smithi019 ceph-mon[31522]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:15:05.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:05 smithi019 ceph-mon[31522]: mds.? [v2:172.21.15.33:6838/1903591535,v1:172.21.15.33:6839/1903591535] up:boot 2024-03-18T15:15:05.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:05 smithi019 ceph-mon[31522]: fsmap 12 up:standby 2024-03-18T15:15:05.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:05 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-18T15:15:05.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:05 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:15:05.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:05 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:15:05.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:05 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:15:05.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:05 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:15:05.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:05 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:15:05.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:05 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:15:05.740 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:05 smithi033 ceph-mon[36648]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:15:05.740 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:05 smithi033 ceph-mon[36648]: mds.? [v2:172.21.15.33:6838/1903591535,v1:172.21.15.33:6839/1903591535] up:boot 2024-03-18T15:15:05.740 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:05 smithi033 ceph-mon[36648]: fsmap 12 up:standby 2024-03-18T15:15:05.740 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:05 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-18T15:15:05.740 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:05 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:15:05.740 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:05 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:15:05.741 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:05 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:15:05.741 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:05 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:15:05.741 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:05 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:15:05.741 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:05 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:15:06.073 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T15:15:06.288 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:06.286+0000 7f9c16600640 1 -- 172.21.15.19:0/3494119015 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c101031a0 msgr2=0x7f9c10109a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:06.288 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:06.286+0000 7f9c16600640 1 --2- 172.21.15.19:0/3494119015 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c101031a0 0x7f9c10109a30 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f9c08009ca0 tx=0x7f9c0802c0f0 comp rx=0 tx=0).stop 2024-03-18T15:15:06.288 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:06.286+0000 7f9c16600640 1 -- 172.21.15.19:0/3494119015 shutdown_connections 2024-03-18T15:15:06.288 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:06.286+0000 7f9c16600640 1 --2- 172.21.15.19:0/3494119015 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c101031a0 0x7f9c10109a30 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:06.288 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:06.286+0000 7f9c16600640 1 --2- 172.21.15.19:0/3494119015 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9c101027e0 0x7f9c10102c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:06.288 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:06.286+0000 7f9c16600640 1 --2- 172.21.15.19:0/3494119015 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9c101015f0 0x7f9c10101a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:06.288 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:06.286+0000 7f9c16600640 1 -- 172.21.15.19:0/3494119015 >> 172.21.15.19:0/3494119015 conn(0x7f9c100fcb40 msgr2=0x7f9c100fefa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:06.288 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:06.286+0000 7f9c16600640 1 -- 172.21.15.19:0/3494119015 shutdown_connections 2024-03-18T15:15:06.288 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:06.286+0000 7f9c16600640 1 -- 172.21.15.19:0/3494119015 wait complete. 2024-03-18T15:15:06.291 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:06.287+0000 7f9c16600640 1 Processor -- start 2024-03-18T15:15:06.291 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:06.288+0000 7f9c16600640 1 -- start start 2024-03-18T15:15:06.291 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:06.288+0000 7f9c16600640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9c101015f0 0x7f9c10197820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:06.291 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:06.288+0000 7f9c16600640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9c101027e0 0x7f9c10197d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:06.291 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:06.288+0000 7f9c16600640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c101031a0 0x7f9c1019edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:06.291 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:06.288+0000 7f9c16600640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f9c1010d270 con 0x7f9c101027e0 2024-03-18T15:15:06.291 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:06.288+0000 7f9c16600640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9c1010d0f0 con 0x7f9c101015f0 2024-03-18T15:15:06.291 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:06.288+0000 7f9c16600640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f9c1010d3f0 con 0x7f9c101031a0 2024-03-18T15:15:06.291 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:06.288+0000 7f9c15200640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9c101015f0 0x7f9c10197820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:06.291 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:06.288+0000 7f9c0fe00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9c101027e0 0x7f9c10197d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:06.291 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:06.288+0000 7f9c15200640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9c101015f0 0x7f9c10197820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:33936/0 (socket says 172.21.15.19:33936) 2024-03-18T15:15:06.291 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:06.288+0000 7f9c15200640 1 -- 172.21.15.19:0/2350304573 learned_addr learned my addr 172.21.15.19:0/2350304573 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:15:06.291 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:06.288+0000 7f9c15c00640 1 --2- 172.21.15.19:0/2350304573 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c101031a0 0x7f9c1019edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:06.292 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:06.289+0000 7f9c15200640 1 -- 172.21.15.19:0/2350304573 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c101031a0 msgr2=0x7f9c1019edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:06.292 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:06.289+0000 7f9c15200640 1 --2- 172.21.15.19:0/2350304573 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c101031a0 0x7f9c1019edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:06.292 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:06.289+0000 7f9c15200640 1 -- 172.21.15.19:0/2350304573 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9c101027e0 msgr2=0x7f9c10197d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:06.292 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:06.289+0000 7f9c15200640 1 --2- 172.21.15.19:0/2350304573 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9c101027e0 0x7f9c10197d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:06.292 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:06.289+0000 7f9c15200640 1 -- 172.21.15.19:0/2350304573 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c1019f440 con 0x7f9c101015f0 2024-03-18T15:15:06.292 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:06.289+0000 7f9c15200640 1 --2- 172.21.15.19:0/2350304573 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9c101015f0 0x7f9c10197820 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f9c0400e9c0 tx=0x7f9c0400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:06.292 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:06.289+0000 7f9c0d600640 1 -- 172.21.15.19:0/2350304573 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c0400cd80 con 0x7f9c101015f0 2024-03-18T15:15:06.292 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:06.289+0000 7f9c0d600640 1 -- 172.21.15.19:0/2350304573 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9c040035f0 con 0x7f9c101015f0 2024-03-18T15:15:06.292 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:06.290+0000 7f9c16600640 1 -- 172.21.15.19:0/2350304573 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c1019f730 con 0x7f9c101015f0 2024-03-18T15:15:06.292 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:06.290+0000 7f9c0d600640 1 -- 172.21.15.19:0/2350304573 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c04010690 con 0x7f9c101015f0 2024-03-18T15:15:06.292 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:06.290+0000 7f9c16600640 1 -- 172.21.15.19:0/2350304573 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c1019fc80 con 0x7f9c101015f0 2024-03-18T15:15:06.293 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:06.291+0000 7f9c16600640 1 -- 172.21.15.19:0/2350304573 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c101088b0 con 0x7f9c101015f0 2024-03-18T15:15:06.293 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:06.292+0000 7f9c0d600640 1 -- 172.21.15.19:0/2350304573 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f9c040037a0 con 0x7f9c101015f0 2024-03-18T15:15:06.295 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:06.294+0000 7f9c0d600640 1 --2- 172.21.15.19:0/2350304573 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f9bf80a2880 0x7f9bf80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:06.295 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:06.294+0000 7f9c0d600640 1 -- 172.21.15.19:0/2350304573 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f9c040ce930 con 0x7f9c101015f0 2024-03-18T15:15:06.295 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:06.294+0000 7f9c0fe00640 1 --2- 172.21.15.19:0/2350304573 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f9bf80a2880 0x7f9bf80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:06.296 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:06.295+0000 7f9c0fe00640 1 --2- 172.21.15.19:0/2350304573 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f9bf80a2880 0x7f9bf80a4d40 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f9c10198d30 tx=0x7f9bfc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:06.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:06.298+0000 7f9c0d600640 1 -- 172.21.15.19:0/2350304573 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9c0409bdd0 con 0x7f9c101015f0 2024-03-18T15:15:06.552 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:06.551+0000 7f9c16600640 1 -- 172.21.15.19:0/2350304573 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9c1002d570 con 0x7f9c101015f0 2024-03-18T15:15:06.555 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:06.554+0000 7f9c0d600640 1 -- 172.21.15.19:0/2350304573 <== mon.2 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10473 (secure 0 0 0) 0x7f9c040860e0 con 0x7f9c101015f0 2024-03-18T15:15:06.555 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-18T15:15:06.555 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6835/1135329545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6834","nonce":1135329545},{"type":"v1","addr":"172.21.15.19:6835","nonce":1135329545}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14457,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6839/3833518557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6838","nonce":3833518557},{"type":"v1","addr":"172.21.15.19:6839","nonce":3833518557}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14475,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6841/3176583319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6840","nonce":3176583319},{"type":"v1","addr":"172.21.15.19:6841","nonce":3176583319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6833/482725809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":482725809},{"type":"v1","addr":"172.21.15.104:6833","nonce":482725809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6833/2272279159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":2272279159},{"type":"v1","addr":"172.21.15.33:6833","nonce":2272279159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24469,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6837/1778019859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6836","nonce":1778019859},{"type":"v1","addr":"172.21.15.19:6837","nonce":1778019859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24482,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/4051613151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":4051613151},{"type":"v1","addr":"172.21.15.33:6835","nonce":4051613151}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24499,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3788595957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3788595957},{"type":"v1","addr":"172.21.15.104:6835","nonce":3788595957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24506,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/160187378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":160187378},{"type":"v1","addr":"172.21.15.33:6837","nonce":160187378}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24511,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6837/260984674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":260984674},{"type":"v1","addr":"172.21.15.104:6837","nonce":260984674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24529,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6839/805341436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6838","nonce":805341436},{"type":"v1","addr":"172.21.15.104:6839","nonce":805341436}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24560,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/1903591535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":1903591535},{"type":"v1","addr":"172.21.15.33:6839","nonce":1903591535}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[]} 2024-03-18T15:15:06.556 INFO:teuthology.orchestra.run.smithi019.stderr:dumped fsmap epoch 13 2024-03-18T15:15:06.559 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:06.557+0000 7f9c16600640 1 -- 172.21.15.19:0/2350304573 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f9bf80a2880 msgr2=0x7f9bf80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:06.559 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:06.557+0000 7f9c16600640 1 --2- 172.21.15.19:0/2350304573 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f9bf80a2880 0x7f9bf80a4d40 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f9c10198d30 tx=0x7f9bfc005f90 comp rx=0 tx=0).stop 2024-03-18T15:15:06.559 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:06.557+0000 7f9c16600640 1 -- 172.21.15.19:0/2350304573 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9c101015f0 msgr2=0x7f9c10197820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:06.559 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:06.557+0000 7f9c16600640 1 --2- 172.21.15.19:0/2350304573 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9c101015f0 0x7f9c10197820 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f9c0400e9c0 tx=0x7f9c0400ee80 comp rx=0 tx=0).stop 2024-03-18T15:15:06.559 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:06.558+0000 7f9c16600640 1 -- 172.21.15.19:0/2350304573 shutdown_connections 2024-03-18T15:15:06.559 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:06.558+0000 7f9c16600640 1 --2- 172.21.15.19:0/2350304573 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c101031a0 0x7f9c1019edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:06.559 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:06.558+0000 7f9c16600640 1 --2- 172.21.15.19:0/2350304573 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9c101027e0 0x7f9c10197d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:06.560 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:06.558+0000 7f9c16600640 1 --2- 172.21.15.19:0/2350304573 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f9bf80a2880 0x7f9bf80a4d40 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:06.560 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:06.558+0000 7f9c16600640 1 --2- 172.21.15.19:0/2350304573 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9c101015f0 0x7f9c10197820 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:06.560 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:06.558+0000 7f9c16600640 1 -- 172.21.15.19:0/2350304573 >> 172.21.15.19:0/2350304573 conn(0x7f9c100fcb40 msgr2=0x7f9c10105a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:06.560 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:06.558+0000 7f9c16600640 1 -- 172.21.15.19:0/2350304573 shutdown_connections 2024-03-18T15:15:06.560 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:06.558+0000 7f9c16600640 1 -- 172.21.15.19:0/2350304573 wait complete. 2024-03-18T15:15:06.580 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T15:15:06.778 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:06.776+0000 7fc3f1800640 1 -- 172.21.15.19:0/2564375711 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc3ec1033a0 msgr2=0x7fc3ec109c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:06.778 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:06.776+0000 7fc3f1800640 1 --2- 172.21.15.19:0/2564375711 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc3ec1033a0 0x7fc3ec109c30 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fc3e4009860 tx=0x7fc3e402bb70 comp rx=0 tx=0).stop 2024-03-18T15:15:06.778 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:06.776+0000 7fc3f1800640 1 -- 172.21.15.19:0/2564375711 shutdown_connections 2024-03-18T15:15:06.778 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:06.776+0000 7fc3f1800640 1 --2- 172.21.15.19:0/2564375711 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc3ec1033a0 0x7fc3ec109c30 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:06.778 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:06.776+0000 7fc3f1800640 1 --2- 172.21.15.19:0/2564375711 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc3ec1029e0 0x7fc3ec102e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:06.778 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:06.776+0000 7fc3f1800640 1 --2- 172.21.15.19:0/2564375711 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc3ec1017f0 0x7fc3ec101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:06.778 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:06.776+0000 7fc3f1800640 1 -- 172.21.15.19:0/2564375711 >> 172.21.15.19:0/2564375711 conn(0x7fc3ec0fcd80 msgr2=0x7fc3ec0ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:06.778 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:06.776+0000 7fc3f1800640 1 -- 172.21.15.19:0/2564375711 shutdown_connections 2024-03-18T15:15:06.778 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:06.777+0000 7fc3f1800640 1 -- 172.21.15.19:0/2564375711 wait complete. 2024-03-18T15:15:06.780 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:06.778+0000 7fc3f1800640 1 Processor -- start 2024-03-18T15:15:06.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:06.778+0000 7fc3f1800640 1 -- start start 2024-03-18T15:15:06.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:06.778+0000 7fc3f1800640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc3ec1017f0 0x7fc3ec075590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:06.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:06.778+0000 7fc3f1800640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc3ec1029e0 0x7fc3ec073bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:06.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:06.778+0000 7fc3f1800640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc3ec1033a0 0x7fc3ec074110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:06.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:06.778+0000 7fc3f1800640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fc3ec10d450 con 0x7fc3ec1033a0 2024-03-18T15:15:06.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:06.778+0000 7fc3f1800640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc3ec10d2d0 con 0x7fc3ec1017f0 2024-03-18T15:15:06.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:06.778+0000 7fc3f1800640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fc3ec10d5d0 con 0x7fc3ec1029e0 2024-03-18T15:15:06.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:06.778+0000 7fc3eaa00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc3ec1017f0 0x7fc3ec075590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:06.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:06.778+0000 7fc3eaa00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc3ec1017f0 0x7fc3ec075590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:33966/0 (socket says 172.21.15.19:33966) 2024-03-18T15:15:06.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:06.778+0000 7fc3eaa00640 1 -- 172.21.15.19:0/366796166 learned_addr learned my addr 172.21.15.19:0/366796166 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:15:06.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:06.778+0000 7fc3ea000640 1 --2- 172.21.15.19:0/366796166 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc3ec1029e0 0x7fc3ec073bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:06.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:06.778+0000 7fc3eb400640 1 --2- 172.21.15.19:0/366796166 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc3ec1033a0 0x7fc3ec074110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:06.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:06.779+0000 7fc3eaa00640 1 -- 172.21.15.19:0/366796166 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc3ec1029e0 msgr2=0x7fc3ec073bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:06.782 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:06.779+0000 7fc3eaa00640 1 --2- 172.21.15.19:0/366796166 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc3ec1029e0 0x7fc3ec073bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:06.782 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:06.779+0000 7fc3eaa00640 1 -- 172.21.15.19:0/366796166 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc3ec1033a0 msgr2=0x7fc3ec074110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:06.782 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:06.779+0000 7fc3eaa00640 1 --2- 172.21.15.19:0/366796166 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc3ec1033a0 0x7fc3ec074110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:06.782 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:06.779+0000 7fc3eaa00640 1 -- 172.21.15.19:0/366796166 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3ec0749a0 con 0x7fc3ec1017f0 2024-03-18T15:15:06.782 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:06.779+0000 7fc3ea000640 1 --2- 172.21.15.19:0/366796166 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc3ec1029e0 0x7fc3ec073bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:15:06.782 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:06.779+0000 7fc3eaa00640 1 --2- 172.21.15.19:0/366796166 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc3ec1017f0 0x7fc3ec075590 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc3d400e9f0 tx=0x7fc3d400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:06.782 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:06.779+0000 7fc3e3400640 1 -- 172.21.15.19:0/366796166 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3d400ccc0 con 0x7fc3ec1017f0 2024-03-18T15:15:06.782 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:06.779+0000 7fc3e3400640 1 -- 172.21.15.19:0/366796166 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc3d40035f0 con 0x7fc3ec1017f0 2024-03-18T15:15:06.782 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:06.779+0000 7fc3e3400640 1 -- 172.21.15.19:0/366796166 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3d400ce60 con 0x7fc3ec1017f0 2024-03-18T15:15:06.782 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:06.779+0000 7fc3f1800640 1 -- 172.21.15.19:0/366796166 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3ec1a1b80 con 0x7fc3ec1017f0 2024-03-18T15:15:06.782 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:06.780+0000 7fc3f1800640 1 -- 172.21.15.19:0/366796166 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3ec1a2080 con 0x7fc3ec1017f0 2024-03-18T15:15:06.783 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:06.781+0000 7fc3f1800640 1 -- 172.21.15.19:0/366796166 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3ec108b10 con 0x7fc3ec1017f0 2024-03-18T15:15:06.783 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:06.783+0000 7fc3e3400640 1 -- 172.21.15.19:0/366796166 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fc3d40037a0 con 0x7fc3ec1017f0 2024-03-18T15:15:06.785 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:06.784+0000 7fc3e3400640 1 --2- 172.21.15.19:0/366796166 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fc3d00a2800 0x7fc3d00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:06.785 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:06.784+0000 7fc3e3400640 1 -- 172.21.15.19:0/366796166 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fc3d40ce8d0 con 0x7fc3ec1017f0 2024-03-18T15:15:06.785 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:06.784+0000 7fc3ea000640 1 --2- 172.21.15.19:0/366796166 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fc3d00a2800 0x7fc3d00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:06.786 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:06.785+0000 7fc3ea000640 1 --2- 172.21.15.19:0/366796166 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fc3d00a2800 0x7fc3d00a4cc0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fc3dc0099a0 tx=0x7fc3dc009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:06.789 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:06.788+0000 7fc3e3400640 1 -- 172.21.15.19:0/366796166 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc3d409bd70 con 0x7fc3ec1017f0 2024-03-18T15:15:07.017 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.016+0000 7fc3f1800640 1 -- 172.21.15.19:0/366796166 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc3ec0623e0 con 0x7fc3ec1017f0 2024-03-18T15:15:07.019 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.018+0000 7fc3e3400640 1 -- 172.21.15.19:0/366796166 <== mon.2 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18421 (secure 0 0 0) 0x7fc3d409b670 con 0x7fc3ec1017f0 2024-03-18T15:15:07.020 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-18T15:15:07.020 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":81,"fsid":"d4dc2ba6-e538-11ee-95c9-87774f69a715","created":"2024-03-18T15:05:57.538551+0000","modified":"2024-03-18T15:14:13.025582+0000","last_up_change":"2024-03-18T15:14:12.009522+0000","last_in_change":"2024-03-18T15:13:52.555387+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T15:09:56.843349+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"37b5a5cd-27ae-4e77-9b05-774b817b2ee9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6802","nonce":402333948},{"type":"v1","addr":"172.21.15.19:6803","nonce":402333948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6804","nonce":402333948},{"type":"v1","addr":"172.21.15.19:6805","nonce":402333948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6808","nonce":402333948},{"type":"v1","addr":"172.21.15.19:6809","nonce":402333948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6806","nonce":402333948},{"type":"v1","addr":"172.21.15.19:6807","nonce":402333948}]},"public_addr":"172.21.15.19:6803/402333948","cluster_addr":"172.21.15.19:6805/402333948","heartbeat_back_addr":"172.21.15.19:6809/402333948","heartbeat_front_addr":"172.21.15.19:6807/402333948","state":["exists","up"]},{"osd":1,"uuid":"3a1134e8-ede7-4dbd-9be1-eefd2cbb43bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1951880113},{"type":"v1","addr":"172.21.15.33:6801","nonce":1951880113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1951880113},{"type":"v1","addr":"172.21.15.33:6803","nonce":1951880113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1951880113},{"type":"v1","addr":"172.21.15.33:6807","nonce":1951880113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1951880113},{"type":"v1","addr":"172.21.15.33:6805","nonce":1951880113}]},"public_addr":"172.21.15.33:6801/1951880113","cluster_addr":"172.21.15.33:6803/1951880113","heartbeat_back_addr":"172.21.15.33:6807/1951880113","heartbeat_front_addr":"172.21.15.33:6805/1951880113","state":["exists","up"]},{"osd":2,"uuid":"f383a164-a3a0-49fd-a9e8-9c2178d8cfb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":1004727949},{"type":"v1","addr":"172.21.15.104:6801","nonce":1004727949}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":1004727949},{"type":"v1","addr":"172.21.15.104:6803","nonce":1004727949}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":1004727949},{"type":"v1","addr":"172.21.15.104:6807","nonce":1004727949}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":1004727949},{"type":"v1","addr":"172.21.15.104:6805","nonce":1004727949}]},"public_addr":"172.21.15.104:6801/1004727949","cluster_addr":"172.21.15.104:6803/1004727949","heartbeat_back_addr":"172.21.15.104:6807/1004727949","heartbeat_front_addr":"172.21.15.104:6805/1004727949","state":["exists","up"]},{"osd":3,"uuid":"c409e116-28bf-45dd-a791-539af22abe1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6810","nonce":3571525854},{"type":"v1","addr":"172.21.15.19:6811","nonce":3571525854}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6812","nonce":3571525854},{"type":"v1","addr":"172.21.15.19:6813","nonce":3571525854}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6816","nonce":3571525854},{"type":"v1","addr":"172.21.15.19:6817","nonce":3571525854}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6814","nonce":3571525854},{"type":"v1","addr":"172.21.15.19:6815","nonce":3571525854}]},"public_addr":"172.21.15.19:6811/3571525854","cluster_addr":"172.21.15.19:6813/3571525854","heartbeat_back_addr":"172.21.15.19:6817/3571525854","heartbeat_front_addr":"172.21.15.19:6815/3571525854","state":["exists","up"]},{"osd":4,"uuid":"696eb719-0b3e-43ae-b5f1-ccd73a3b2843","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":624901749},{"type":"v1","addr":"172.21.15.33:6809","nonce":624901749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":624901749},{"type":"v1","addr":"172.21.15.33:6811","nonce":624901749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":624901749},{"type":"v1","addr":"172.21.15.33:6815","nonce":624901749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":624901749},{"type":"v1","addr":"172.21.15.33:6813","nonce":624901749}]},"public_addr":"172.21.15.33:6809/624901749","cluster_addr":"172.21.15.33:6811/624901749","heartbeat_back_addr":"172.21.15.33:6815/624901749","heartbeat_front_addr":"172.21.15.33:6813/624901749","state":["exists","up"]},{"osd":5,"uuid":"21789779-8bb9-42c7-b9ee-437985f11e3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":1258208297},{"type":"v1","addr":"172.21.15.104:6809","nonce":1258208297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":1258208297},{"type":"v1","addr":"172.21.15.104:6811","nonce":1258208297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":1258208297},{"type":"v1","addr":"172.21.15.104:6815","nonce":1258208297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":1258208297},{"type":"v1","addr":"172.21.15.104:6813","nonce":1258208297}]},"public_addr":"172.21.15.104:6809/1258208297","cluster_addr":"172.21.15.104:6811/1258208297","heartbeat_back_addr":"172.21.15.104:6815/1258208297","heartbeat_front_addr":"172.21.15.104:6813/1258208297","state":["exists","up"]},{"osd":6,"uuid":"0e92a2ef-43cd-487c-9203-e219158aef8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6818","nonce":2782482301},{"type":"v1","addr":"172.21.15.19:6819","nonce":2782482301}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6820","nonce":2782482301},{"type":"v1","addr":"172.21.15.19:6821","nonce":2782482301}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6824","nonce":2782482301},{"type":"v1","addr":"172.21.15.19:6825","nonce":2782482301}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6822","nonce":2782482301},{"type":"v1","addr":"172.21.15.19:6823","nonce":2782482301}]},"public_addr":"172.21.15.19:6819/2782482301","cluster_addr":"172.21.15.19:6821/2782482301","heartbeat_back_addr":"172.21.15.19:6825/2782482301","heartbeat_front_addr":"172.21.15.19:6823/2782482301","state":["exists","up"]},{"osd":7,"uuid":"a16e39a1-3685-491d-b376-3e80a2084072","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":4238160988},{"type":"v1","addr":"172.21.15.33:6817","nonce":4238160988}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":4238160988},{"type":"v1","addr":"172.21.15.33:6819","nonce":4238160988}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":4238160988},{"type":"v1","addr":"172.21.15.33:6823","nonce":4238160988}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":4238160988},{"type":"v1","addr":"172.21.15.33:6821","nonce":4238160988}]},"public_addr":"172.21.15.33:6817/4238160988","cluster_addr":"172.21.15.33:6819/4238160988","heartbeat_back_addr":"172.21.15.33:6823/4238160988","heartbeat_front_addr":"172.21.15.33:6821/4238160988","state":["exists","up"]},{"osd":8,"uuid":"d5c3fa92-cd5d-4349-9c7d-d5666bacde16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":1014779813},{"type":"v1","addr":"172.21.15.104:6817","nonce":1014779813}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":1014779813},{"type":"v1","addr":"172.21.15.104:6819","nonce":1014779813}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":1014779813},{"type":"v1","addr":"172.21.15.104:6823","nonce":1014779813}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":1014779813},{"type":"v1","addr":"172.21.15.104:6821","nonce":1014779813}]},"public_addr":"172.21.15.104:6817/1014779813","cluster_addr":"172.21.15.104:6819/1014779813","heartbeat_back_addr":"172.21.15.104:6823/1014779813","heartbeat_front_addr":"172.21.15.104:6821/1014779813","state":["exists","up"]},{"osd":9,"uuid":"eed2e9b2-26d3-4c46-88ec-34b5596fe991","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6826","nonce":2641151009},{"type":"v1","addr":"172.21.15.19:6827","nonce":2641151009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6828","nonce":2641151009},{"type":"v1","addr":"172.21.15.19:6829","nonce":2641151009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6832","nonce":2641151009},{"type":"v1","addr":"172.21.15.19:6833","nonce":2641151009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6830","nonce":2641151009},{"type":"v1","addr":"172.21.15.19:6831","nonce":2641151009}]},"public_addr":"172.21.15.19:6827/2641151009","cluster_addr":"172.21.15.19:6829/2641151009","heartbeat_back_addr":"172.21.15.19:6833/2641151009","heartbeat_front_addr":"172.21.15.19:6831/2641151009","state":["exists","up"]},{"osd":10,"uuid":"f3c35ea5-fbe7-4dc4-93c5-b92d1a714db3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":1346058869},{"type":"v1","addr":"172.21.15.33:6825","nonce":1346058869}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":1346058869},{"type":"v1","addr":"172.21.15.33:6827","nonce":1346058869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":1346058869},{"type":"v1","addr":"172.21.15.33:6831","nonce":1346058869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":1346058869},{"type":"v1","addr":"172.21.15.33:6829","nonce":1346058869}]},"public_addr":"172.21.15.33:6825/1346058869","cluster_addr":"172.21.15.33:6827/1346058869","heartbeat_back_addr":"172.21.15.33:6831/1346058869","heartbeat_front_addr":"172.21.15.33:6829/1346058869","state":["exists","up"]},{"osd":11,"uuid":"c8b51571-f807-4b62-90b6-7529e09e59da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":3759498281},{"type":"v1","addr":"172.21.15.104:6825","nonce":3759498281}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":3759498281},{"type":"v1","addr":"172.21.15.104:6827","nonce":3759498281}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":3759498281},{"type":"v1","addr":"172.21.15.104:6831","nonce":3759498281}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":3759498281},{"type":"v1","addr":"172.21.15.104:6829","nonce":3759498281}]},"public_addr":"172.21.15.104:6825/3759498281","cluster_addr":"172.21.15.104:6827/3759498281","heartbeat_back_addr":"172.21.15.104:6831/3759498281","heartbeat_front_addr":"172.21.15.104:6829/3759498281","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:09:05.267795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:09:30.234068+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:09:53.870232+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:10:26.471296+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:10:53.180341+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:11:17.801361+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:11:51.364816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:12:18.306408+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:12:43.978744+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:13:18.213964+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:13:45.437544+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:14:10.148580+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.19:0/3860783274":"2024-03-19T15:07:14.150983+0000","172.21.15.19:0/2737223213":"2024-03-19T15:07:14.150983+0000","172.21.15.19:0/176699025":"2024-03-19T15:06:38.688405+0000","172.21.15.19:0/2316969005":"2024-03-19T15:07:14.150983+0000","172.21.15.19:6801/2776008962":"2024-03-19T15:07:14.150983+0000","172.21.15.19:0/2793980440":"2024-03-19T15:06:38.688405+0000","172.21.15.19:6800/3264337803":"2024-03-19T15:06:38.688405+0000","172.21.15.19:6801/3264337803":"2024-03-19T15:06:38.688405+0000","172.21.15.19:0/3166705295":"2024-03-19T15:06:38.688405+0000","172.21.15.19:6800/2776008962":"2024-03-19T15:07:14.150983+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T15:15:07.022 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.021+0000 7fc3f1800640 1 -- 172.21.15.19:0/366796166 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fc3d00a2800 msgr2=0x7fc3d00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:07.022 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.021+0000 7fc3f1800640 1 --2- 172.21.15.19:0/366796166 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fc3d00a2800 0x7fc3d00a4cc0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fc3dc0099a0 tx=0x7fc3dc009500 comp rx=0 tx=0).stop 2024-03-18T15:15:07.022 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.021+0000 7fc3f1800640 1 -- 172.21.15.19:0/366796166 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc3ec1017f0 msgr2=0x7fc3ec075590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:07.022 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.021+0000 7fc3f1800640 1 --2- 172.21.15.19:0/366796166 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc3ec1017f0 0x7fc3ec075590 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc3d400e9f0 tx=0x7fc3d400eeb0 comp rx=0 tx=0).stop 2024-03-18T15:15:07.023 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.021+0000 7fc3f1800640 1 -- 172.21.15.19:0/366796166 shutdown_connections 2024-03-18T15:15:07.023 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.021+0000 7fc3f1800640 1 --2- 172.21.15.19:0/366796166 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc3ec1033a0 0x7fc3ec074110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:07.023 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.021+0000 7fc3f1800640 1 --2- 172.21.15.19:0/366796166 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc3ec1029e0 0x7fc3ec073bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:07.023 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.021+0000 7fc3f1800640 1 --2- 172.21.15.19:0/366796166 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fc3d00a2800 0x7fc3d00a4cc0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:07.023 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.021+0000 7fc3f1800640 1 --2- 172.21.15.19:0/366796166 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc3ec1017f0 0x7fc3ec075590 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:07.023 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.021+0000 7fc3f1800640 1 -- 172.21.15.19:0/366796166 >> 172.21.15.19:0/366796166 conn(0x7fc3ec0fcd80 msgr2=0x7fc3ec105ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:07.023 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.021+0000 7fc3f1800640 1 -- 172.21.15.19:0/366796166 shutdown_connections 2024-03-18T15:15:07.023 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.021+0000 7fc3f1800640 1 -- 172.21.15.19:0/366796166 wait complete. 2024-03-18T15:15:07.041 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T15:09:56.843349+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T15:15:07.041 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T15:15:07.244 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.242+0000 7f4361a00640 1 -- 172.21.15.19:0/1460696016 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f435c0773e0 msgr2=0x7f435c077840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:07.244 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.242+0000 7f4361a00640 1 --2- 172.21.15.19:0/1460696016 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f435c0773e0 0x7f435c077840 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f4348009860 tx=0x7f434802bb70 comp rx=0 tx=0).stop 2024-03-18T15:15:07.244 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.242+0000 7f4361a00640 1 -- 172.21.15.19:0/1460696016 shutdown_connections 2024-03-18T15:15:07.244 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.242+0000 7f4361a00640 1 --2- 172.21.15.19:0/1460696016 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f435c077d80 0x7f435c078620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:07.244 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.242+0000 7f4361a00640 1 --2- 172.21.15.19:0/1460696016 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f435c0773e0 0x7f435c077840 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:07.244 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.242+0000 7f4361a00640 1 --2- 172.21.15.19:0/1460696016 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f435c0761a0 0x7f435c0765c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:07.244 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.242+0000 7f4361a00640 1 -- 172.21.15.19:0/1460696016 >> 172.21.15.19:0/1460696016 conn(0x7f435c0fef90 msgr2=0x7f435c1013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:07.244 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.242+0000 7f4361a00640 1 -- 172.21.15.19:0/1460696016 shutdown_connections 2024-03-18T15:15:07.244 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.242+0000 7f4361a00640 1 -- 172.21.15.19:0/1460696016 wait complete. 2024-03-18T15:15:07.246 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.243+0000 7f4361a00640 1 Processor -- start 2024-03-18T15:15:07.246 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.243+0000 7f4361a00640 1 -- start start 2024-03-18T15:15:07.246 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.244+0000 7f4361a00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f435c0761a0 0x7f435c19e130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:07.247 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.244+0000 7f4361a00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f435c0773e0 0x7f435c19e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:07.247 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.244+0000 7f4361a00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f435c077d80 0x7f435c1a56e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:07.247 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.244+0000 7f4361a00640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f435c113a90 con 0x7f435c0761a0 2024-03-18T15:15:07.247 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.244+0000 7f4361a00640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f435c113910 con 0x7f435c0773e0 2024-03-18T15:15:07.247 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.244+0000 7f4361a00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f435c113c10 con 0x7f435c077d80 2024-03-18T15:15:07.247 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.244+0000 7f435aa00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f435c0761a0 0x7f435c19e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:07.247 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.244+0000 7f435b400640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f435c077d80 0x7f435c1a56e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:07.247 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.244+0000 7f435aa00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f435c0761a0 0x7f435c19e130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:59330/0 (socket says 172.21.15.19:59330) 2024-03-18T15:15:07.247 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.244+0000 7f435aa00640 1 -- 172.21.15.19:0/748133040 learned_addr learned my addr 172.21.15.19:0/748133040 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:15:07.247 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.244+0000 7f435b400640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f435c077d80 0x7f435c1a56e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.19:40232/0 (socket says 172.21.15.19:40232) 2024-03-18T15:15:07.247 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.244+0000 7f435a000640 1 --2- 172.21.15.19:0/748133040 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f435c0773e0 0x7f435c19e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:07.247 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.244+0000 7f435b400640 1 -- 172.21.15.19:0/748133040 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f435c0773e0 msgr2=0x7f435c19e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:07.247 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.244+0000 7f435b400640 1 --2- 172.21.15.19:0/748133040 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f435c0773e0 0x7f435c19e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:07.248 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.244+0000 7f435b400640 1 -- 172.21.15.19:0/748133040 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f435c0761a0 msgr2=0x7f435c19e130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:07.248 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.244+0000 7f435b400640 1 --2- 172.21.15.19:0/748133040 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f435c0761a0 0x7f435c19e130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:07.248 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.244+0000 7f435b400640 1 -- 172.21.15.19:0/748133040 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f435c1a5d50 con 0x7f435c077d80 2024-03-18T15:15:07.248 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.245+0000 7f435b400640 1 --2- 172.21.15.19:0/748133040 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f435c077d80 0x7f435c1a56e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f4354009fd0 tx=0x7f435400ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:07.248 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.245+0000 7f4353400640 1 -- 172.21.15.19:0/748133040 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f435400ce20 con 0x7f435c077d80 2024-03-18T15:15:07.248 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.245+0000 7f4353400640 1 -- 172.21.15.19:0/748133040 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4354005020 con 0x7f435c077d80 2024-03-18T15:15:07.248 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.245+0000 7f4361a00640 1 -- 172.21.15.19:0/748133040 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f435c1a5fe0 con 0x7f435c077d80 2024-03-18T15:15:07.248 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.245+0000 7f4353400640 1 -- 172.21.15.19:0/748133040 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4354010720 con 0x7f435c077d80 2024-03-18T15:15:07.248 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.245+0000 7f4361a00640 1 -- 172.21.15.19:0/748133040 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f435c1a6550 con 0x7f435c077d80 2024-03-18T15:15:07.249 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.246+0000 7f4361a00640 1 -- 172.21.15.19:0/748133040 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f435c110370 con 0x7f435c077d80 2024-03-18T15:15:07.249 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.248+0000 7f4353400640 1 -- 172.21.15.19:0/748133040 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f4354005540 con 0x7f435c077d80 2024-03-18T15:15:07.250 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.249+0000 7f4353400640 1 --2- 172.21.15.19:0/748133040 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f43400a2880 0x7f43400a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:07.250 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.249+0000 7f4353400640 1 -- 172.21.15.19:0/748133040 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f43540cebe0 con 0x7f435c077d80 2024-03-18T15:15:07.250 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.249+0000 7f435aa00640 1 --2- 172.21.15.19:0/748133040 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f43400a2880 0x7f43400a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:07.251 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.250+0000 7f435aa00640 1 --2- 172.21.15.19:0/748133040 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f43400a2880 0x7f43400a4d40 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f434c005c80 tx=0x7f434c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:07.254 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.253+0000 7f4353400640 1 -- 172.21.15.19:0/748133040 <== mon.1 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f435409c080 con 0x7f435c077d80 2024-03-18T15:15:07.499 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.498+0000 7f4361a00640 1 -- 172.21.15.19:0/748133040 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f435c0623e0 con 0x7f435c077d80 2024-03-18T15:15:07.499 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.498+0000 7f4353400640 1 -- 172.21.15.19:0/748133040 <== mon.1 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7f435409b980 con 0x7f435c077d80 2024-03-18T15:15:07.500 INFO:teuthology.orchestra.run.smithi019.stdout:pg_num: 1 2024-03-18T15:15:07.503 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.502+0000 7f4361a00640 1 -- 172.21.15.19:0/748133040 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f43400a2880 msgr2=0x7f43400a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:07.504 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.502+0000 7f4361a00640 1 --2- 172.21.15.19:0/748133040 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f43400a2880 0x7f43400a4d40 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f434c005c80 tx=0x7f434c009260 comp rx=0 tx=0).stop 2024-03-18T15:15:07.504 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.502+0000 7f4361a00640 1 -- 172.21.15.19:0/748133040 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f435c077d80 msgr2=0x7f435c1a56e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:07.504 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.502+0000 7f4361a00640 1 --2- 172.21.15.19:0/748133040 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f435c077d80 0x7f435c1a56e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f4354009fd0 tx=0x7f435400ef10 comp rx=0 tx=0).stop 2024-03-18T15:15:07.504 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.502+0000 7f4361a00640 1 -- 172.21.15.19:0/748133040 shutdown_connections 2024-03-18T15:15:07.504 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.502+0000 7f4361a00640 1 --2- 172.21.15.19:0/748133040 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f435c077d80 0x7f435c1a56e0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:07.504 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.502+0000 7f4361a00640 1 --2- 172.21.15.19:0/748133040 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f43400a2880 0x7f43400a4d40 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:07.504 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.502+0000 7f4361a00640 1 --2- 172.21.15.19:0/748133040 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f435c0773e0 0x7f435c19e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:07.504 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.502+0000 7f4361a00640 1 --2- 172.21.15.19:0/748133040 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f435c0761a0 0x7f435c19e130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:07.504 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.502+0000 7f4361a00640 1 -- 172.21.15.19:0/748133040 >> 172.21.15.19:0/748133040 conn(0x7f435c0fef90 msgr2=0x7f435c100a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:07.504 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.503+0000 7f4361a00640 1 -- 172.21.15.19:0/748133040 shutdown_connections 2024-03-18T15:15:07.504 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.503+0000 7f4361a00640 1 -- 172.21.15.19:0/748133040 wait complete. 2024-03-18T15:15:07.521 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-18T15:15:07.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:07 smithi104 ceph-mon[36070]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:15:07.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:07 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:15:07.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:07 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:15:07.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:07 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/2350304573' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:15:07.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:07 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:15:07.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:07 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:15:07.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:07 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/366796166' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:15:07.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:07 smithi019 ceph-mon[31522]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:15:07.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:07 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:15:07.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:07 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:15:07.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:07 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/2350304573' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:15:07.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:07 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:15:07.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:07 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:15:07.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:07 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/366796166' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:15:07.717 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.716+0000 7ff14ac00640 1 -- 172.21.15.19:0/1347338966 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff1440773e0 msgr2=0x7ff144077840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:07.717 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.716+0000 7ff14ac00640 1 --2- 172.21.15.19:0/1347338966 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff1440773e0 0x7ff144077840 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ff130009900 tx=0x7ff13002bc30 comp rx=0 tx=0).stop 2024-03-18T15:15:07.717 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.716+0000 7ff14ac00640 1 -- 172.21.15.19:0/1347338966 shutdown_connections 2024-03-18T15:15:07.717 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.716+0000 7ff14ac00640 1 --2- 172.21.15.19:0/1347338966 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff144077d80 0x7ff144078620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:07.717 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.716+0000 7ff14ac00640 1 --2- 172.21.15.19:0/1347338966 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff1440773e0 0x7ff144077840 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:07.718 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.716+0000 7ff14ac00640 1 --2- 172.21.15.19:0/1347338966 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff1440761a0 0x7ff1440765c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:07.718 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.716+0000 7ff14ac00640 1 -- 172.21.15.19:0/1347338966 >> 172.21.15.19:0/1347338966 conn(0x7ff1440fedc0 msgr2=0x7ff1441011e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:07.718 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.716+0000 7ff14ac00640 1 -- 172.21.15.19:0/1347338966 shutdown_connections 2024-03-18T15:15:07.718 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.716+0000 7ff14ac00640 1 -- 172.21.15.19:0/1347338966 wait complete. 2024-03-18T15:15:07.719 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.717+0000 7ff14ac00640 1 Processor -- start 2024-03-18T15:15:07.719 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.717+0000 7ff14ac00640 1 -- start start 2024-03-18T15:15:07.719 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.718+0000 7ff14ac00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff1440761a0 0x7ff14419dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:07.719 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.718+0000 7ff14ac00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff1440773e0 0x7ff14419e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:07.720 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.718+0000 7ff14ac00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff144077d80 0x7ff1441a5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:07.720 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.718+0000 7ff14ac00640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7ff1441138c0 con 0x7ff1440761a0 2024-03-18T15:15:07.720 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.718+0000 7ff14ac00640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff144113740 con 0x7ff144077d80 2024-03-18T15:15:07.720 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.718+0000 7ff14ac00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7ff144113a40 con 0x7ff1440773e0 2024-03-18T15:15:07.720 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.718+0000 7ff14a200640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff144077d80 0x7ff1441a5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:07.720 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.718+0000 7ff14a200640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff144077d80 0x7ff1441a5490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:34004/0 (socket says 172.21.15.19:34004) 2024-03-18T15:15:07.720 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.718+0000 7ff14a200640 1 -- 172.21.15.19:0/3826987739 learned_addr learned my addr 172.21.15.19:0/3826987739 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:15:07.720 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.718+0000 7ff149800640 1 --2- 172.21.15.19:0/3826987739 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff1440761a0 0x7ff14419dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:07.720 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.718+0000 7ff148e00640 1 --2- 172.21.15.19:0/3826987739 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff1440773e0 0x7ff14419e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:07.720 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.718+0000 7ff14a200640 1 -- 172.21.15.19:0/3826987739 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff1440773e0 msgr2=0x7ff14419e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:07.720 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.718+0000 7ff14a200640 1 --2- 172.21.15.19:0/3826987739 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff1440773e0 0x7ff14419e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:07.721 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.718+0000 7ff14a200640 1 -- 172.21.15.19:0/3826987739 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff1440761a0 msgr2=0x7ff14419dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:07.721 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.718+0000 7ff14a200640 1 --2- 172.21.15.19:0/3826987739 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff1440761a0 0x7ff14419dee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:07.721 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.718+0000 7ff14a200640 1 -- 172.21.15.19:0/3826987739 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1441a5b00 con 0x7ff144077d80 2024-03-18T15:15:07.722 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.719+0000 7ff14a200640 1 --2- 172.21.15.19:0/3826987739 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff144077d80 0x7ff1441a5490 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ff13c00eb10 tx=0x7ff13c00c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:07.722 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.719+0000 7ff142000640 1 -- 172.21.15.19:0/3826987739 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff13c004b20 con 0x7ff144077d80 2024-03-18T15:15:07.722 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.719+0000 7ff142000640 1 -- 172.21.15.19:0/3826987739 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff13c004cc0 con 0x7ff144077d80 2024-03-18T15:15:07.722 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.719+0000 7ff142000640 1 -- 172.21.15.19:0/3826987739 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff13c010760 con 0x7ff144077d80 2024-03-18T15:15:07.722 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.719+0000 7ff14ac00640 1 -- 172.21.15.19:0/3826987739 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1441a5df0 con 0x7ff144077d80 2024-03-18T15:15:07.722 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.719+0000 7ff14ac00640 1 -- 172.21.15.19:0/3826987739 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1441a6260 con 0x7ff144077d80 2024-03-18T15:15:07.722 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.720+0000 7ff14ac00640 1 -- 172.21.15.19:0/3826987739 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff144110330 con 0x7ff144077d80 2024-03-18T15:15:07.723 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.722+0000 7ff142000640 1 -- 172.21.15.19:0/3826987739 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7ff13c005020 con 0x7ff144077d80 2024-03-18T15:15:07.725 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.724+0000 7ff142000640 1 --2- 172.21.15.19:0/3826987739 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7ff12c0a2880 0x7ff12c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:07.725 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.724+0000 7ff142000640 1 -- 172.21.15.19:0/3826987739 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7ff13c0cec90 con 0x7ff144077d80 2024-03-18T15:15:07.725 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.724+0000 7ff149800640 1 --2- 172.21.15.19:0/3826987739 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7ff12c0a2880 0x7ff12c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:07.725 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.724+0000 7ff149800640 1 --2- 172.21.15.19:0/3826987739 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7ff12c0a2880 0x7ff12c0a4d40 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7ff138009bd0 tx=0x7ff138005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:07.729 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.728+0000 7ff142000640 1 -- 172.21.15.19:0/3826987739 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff13c09c130 con 0x7ff144077d80 2024-03-18T15:15:07.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:07 smithi033 ceph-mon[36648]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:15:07.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:07 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:15:07.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:07 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:15:07.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:07 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/2350304573' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:15:07.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:07 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:15:07.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:07 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:15:07.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:07 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/366796166' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:15:07.936 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.935+0000 7ff14ac00640 1 -- 172.21.15.19:0/3826987739 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7ff1440624e0 con 0x7ff144077d80 2024-03-18T15:15:07.936 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.935+0000 7ff142000640 1 -- 172.21.15.19:0/3826987739 <== mon.2 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v81) ==== 84+0+65 (secure 0 0 0) 0x7ff13c09ba30 con 0x7ff144077d80 2024-03-18T15:15:07.937 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-18T15:15:07.937 INFO:teuthology.orchestra.run.smithi019.stdout:[ 2024-03-18T15:15:07.937 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:15:07.937 INFO:teuthology.orchestra.run.smithi019.stdout: "poolnum": 1, 2024-03-18T15:15:07.937 INFO:teuthology.orchestra.run.smithi019.stdout: "poolname": ".mgr" 2024-03-18T15:15:07.937 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:15:07.937 INFO:teuthology.orchestra.run.smithi019.stdout:] 2024-03-18T15:15:07.939 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.938+0000 7ff14ac00640 1 -- 172.21.15.19:0/3826987739 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7ff12c0a2880 msgr2=0x7ff12c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:07.939 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.938+0000 7ff14ac00640 1 --2- 172.21.15.19:0/3826987739 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7ff12c0a2880 0x7ff12c0a4d40 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7ff138009bd0 tx=0x7ff138005f90 comp rx=0 tx=0).stop 2024-03-18T15:15:07.939 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.938+0000 7ff14ac00640 1 -- 172.21.15.19:0/3826987739 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff144077d80 msgr2=0x7ff1441a5490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:07.939 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.938+0000 7ff14ac00640 1 --2- 172.21.15.19:0/3826987739 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff144077d80 0x7ff1441a5490 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ff13c00eb10 tx=0x7ff13c00c2d0 comp rx=0 tx=0).stop 2024-03-18T15:15:07.939 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.938+0000 7ff14ac00640 1 -- 172.21.15.19:0/3826987739 shutdown_connections 2024-03-18T15:15:07.940 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.938+0000 7ff14ac00640 1 --2- 172.21.15.19:0/3826987739 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7ff12c0a2880 0x7ff12c0a4d40 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:07.940 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.938+0000 7ff14ac00640 1 --2- 172.21.15.19:0/3826987739 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff144077d80 0x7ff1441a5490 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:07.940 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.938+0000 7ff14ac00640 1 --2- 172.21.15.19:0/3826987739 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff1440773e0 0x7ff14419e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:07.940 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.938+0000 7ff14ac00640 1 --2- 172.21.15.19:0/3826987739 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff1440761a0 0x7ff14419dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:07.940 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.938+0000 7ff14ac00640 1 -- 172.21.15.19:0/3826987739 >> 172.21.15.19:0/3826987739 conn(0x7ff1440fedc0 msgr2=0x7ff144100460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:07.940 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.938+0000 7ff14ac00640 1 -- 172.21.15.19:0/3826987739 shutdown_connections 2024-03-18T15:15:07.940 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:07.938+0000 7ff14ac00640 1 -- 172.21.15.19:0/3826987739 wait complete. 2024-03-18T15:15:07.957 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-18T15:15:07.958 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-18T15:15:08.152 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:08.151+0000 7fc92be00640 1 -- 172.21.15.19:0/4232151932 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc92c104b30 msgr2=0x7fc92c106f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:08.152 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:08.151+0000 7fc92be00640 1 --2- 172.21.15.19:0/4232151932 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc92c104b30 0x7fc92c106f20 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fc914009900 tx=0x7fc91402bc30 comp rx=0 tx=0).stop 2024-03-18T15:15:08.153 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:08.151+0000 7fc92be00640 1 -- 172.21.15.19:0/4232151932 shutdown_connections 2024-03-18T15:15:08.153 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:08.151+0000 7fc92be00640 1 --2- 172.21.15.19:0/4232151932 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc92c107460 0x7fc92c109850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:08.153 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:08.151+0000 7fc92be00640 1 --2- 172.21.15.19:0/4232151932 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc92c104b30 0x7fc92c106f20 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:08.153 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:08.151+0000 7fc92be00640 1 --2- 172.21.15.19:0/4232151932 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc92c102200 0x7fc92c1045f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:08.153 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:08.151+0000 7fc92be00640 1 -- 172.21.15.19:0/4232151932 >> 172.21.15.19:0/4232151932 conn(0x7fc92c0fbb50 msgr2=0x7fc92c0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:08.153 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:08.151+0000 7fc92be00640 1 -- 172.21.15.19:0/4232151932 shutdown_connections 2024-03-18T15:15:08.153 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:08.151+0000 7fc92be00640 1 -- 172.21.15.19:0/4232151932 wait complete. 2024-03-18T15:15:08.155 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:08.152+0000 7fc92be00640 1 Processor -- start 2024-03-18T15:15:08.155 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:08.152+0000 7fc92be00640 1 -- start start 2024-03-18T15:15:08.155 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:08.153+0000 7fc92be00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc92c102200 0x7fc92c1979f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:08.155 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:08.153+0000 7fc92be00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc92c104b30 0x7fc92c197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:08.156 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:08.153+0000 7fc92be00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc92c107460 0x7fc92c19efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:08.156 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:08.153+0000 7fc92be00640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fc92c10d370 con 0x7fc92c104b30 2024-03-18T15:15:08.156 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:08.153+0000 7fc92be00640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc92c10d1f0 con 0x7fc92c107460 2024-03-18T15:15:08.156 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:08.153+0000 7fc92be00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fc92c10d4f0 con 0x7fc92c102200 2024-03-18T15:15:08.156 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:08.153+0000 7fc92aa00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc92c102200 0x7fc92c1979f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:08.156 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:08.153+0000 7fc92aa00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc92c102200 0x7fc92c1979f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.19:40276/0 (socket says 172.21.15.19:40276) 2024-03-18T15:15:08.156 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:08.153+0000 7fc92aa00640 1 -- 172.21.15.19:0/1863919381 learned_addr learned my addr 172.21.15.19:0/1863919381 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:15:08.156 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:08.153+0000 7fc92b400640 1 --2- 172.21.15.19:0/1863919381 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc92c107460 0x7fc92c19efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:08.156 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:08.153+0000 7fc923e00640 1 --2- 172.21.15.19:0/1863919381 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc92c104b30 0x7fc92c197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:08.156 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:08.154+0000 7fc92aa00640 1 -- 172.21.15.19:0/1863919381 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc92c107460 msgr2=0x7fc92c19efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:08.156 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:08.154+0000 7fc92aa00640 1 --2- 172.21.15.19:0/1863919381 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc92c107460 0x7fc92c19efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:08.156 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:08.154+0000 7fc92aa00640 1 -- 172.21.15.19:0/1863919381 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc92c104b30 msgr2=0x7fc92c197f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:08.156 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:08.154+0000 7fc92aa00640 1 --2- 172.21.15.19:0/1863919381 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc92c104b30 0x7fc92c197f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:08.156 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:08.154+0000 7fc92aa00640 1 -- 172.21.15.19:0/1863919381 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc92c19f610 con 0x7fc92c102200 2024-03-18T15:15:08.157 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:08.154+0000 7fc923e00640 1 --2- 172.21.15.19:0/1863919381 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc92c104b30 0x7fc92c197f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T15:15:08.157 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:08.154+0000 7fc92aa00640 1 --2- 172.21.15.19:0/1863919381 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc92c102200 0x7fc92c1979f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fc91c00ea60 tx=0x7fc91c00ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:08.157 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:08.154+0000 7fc923400640 1 -- 172.21.15.19:0/1863919381 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc91c00cde0 con 0x7fc92c102200 2024-03-18T15:15:08.157 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:08.154+0000 7fc923400640 1 -- 172.21.15.19:0/1863919381 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc91c0035f0 con 0x7fc92c102200 2024-03-18T15:15:08.157 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:08.154+0000 7fc92be00640 1 -- 172.21.15.19:0/1863919381 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc92c19f900 con 0x7fc92c102200 2024-03-18T15:15:08.157 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:08.154+0000 7fc923400640 1 -- 172.21.15.19:0/1863919381 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc91c010670 con 0x7fc92c102200 2024-03-18T15:15:08.157 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:08.154+0000 7fc92be00640 1 -- 172.21.15.19:0/1863919381 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc92c19fe20 con 0x7fc92c102200 2024-03-18T15:15:08.157 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:08.155+0000 7fc92be00640 1 -- 172.21.15.19:0/1863919381 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc92c0fd640 con 0x7fc92c102200 2024-03-18T15:15:08.158 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:08.157+0000 7fc923400640 1 -- 172.21.15.19:0/1863919381 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fc91c0037a0 con 0x7fc92c102200 2024-03-18T15:15:08.160 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:08.158+0000 7fc923400640 1 --2- 172.21.15.19:0/1863919381 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fc9100a2800 0x7fc9100a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:08.160 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:08.159+0000 7fc923400640 1 -- 172.21.15.19:0/1863919381 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fc91c0ce9f0 con 0x7fc92c102200 2024-03-18T15:15:08.160 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:08.159+0000 7fc923e00640 1 --2- 172.21.15.19:0/1863919381 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fc9100a2800 0x7fc9100a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:08.160 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:08.159+0000 7fc923e00640 1 --2- 172.21.15.19:0/1863919381 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fc9100a2800 0x7fc9100a4cc0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fc91402c170 tx=0x7fc91402df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:08.163 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:08.162+0000 7fc923400640 1 -- 172.21.15.19:0/1863919381 <== mon.1 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc91c09be90 con 0x7fc92c102200 2024-03-18T15:15:08.384 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:08.383+0000 7fc92be00640 1 -- 172.21.15.19:0/1863919381 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fc92c02d1b0 con 0x7fc92c102200 2024-03-18T15:15:08.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:08.996+0000 7fc923400640 1 -- 172.21.15.19:0/1863919381 <== mon.1 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v82) ==== 138+0+0 (secure 0 0 0) 0x7fc92c02d1b0 con 0x7fc92c102200 2024-03-18T15:15:08.997 INFO:teuthology.orchestra.run.smithi019.stderr:pool 'cephfs_metadata' created 2024-03-18T15:15:09.002 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:09.001+0000 7fc92be00640 1 -- 172.21.15.19:0/1863919381 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fc9100a2800 msgr2=0x7fc9100a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:09.003 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:09.001+0000 7fc92be00640 1 --2- 172.21.15.19:0/1863919381 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fc9100a2800 0x7fc9100a4cc0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fc91402c170 tx=0x7fc91402df90 comp rx=0 tx=0).stop 2024-03-18T15:15:09.003 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:09.001+0000 7fc92be00640 1 -- 172.21.15.19:0/1863919381 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc92c102200 msgr2=0x7fc92c1979f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:09.003 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:09.001+0000 7fc92be00640 1 --2- 172.21.15.19:0/1863919381 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc92c102200 0x7fc92c1979f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fc91c00ea60 tx=0x7fc91c00ef20 comp rx=0 tx=0).stop 2024-03-18T15:15:09.004 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:09.002+0000 7fc92be00640 1 -- 172.21.15.19:0/1863919381 shutdown_connections 2024-03-18T15:15:09.004 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:09.002+0000 7fc92be00640 1 --2- 172.21.15.19:0/1863919381 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fc9100a2800 0x7fc9100a4cc0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:09.005 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:09.002+0000 7fc92be00640 1 --2- 172.21.15.19:0/1863919381 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc92c107460 0x7fc92c19efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:09.005 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:09.002+0000 7fc92be00640 1 --2- 172.21.15.19:0/1863919381 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc92c104b30 0x7fc92c197f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:09.005 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:09.002+0000 7fc92be00640 1 --2- 172.21.15.19:0/1863919381 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc92c102200 0x7fc92c1979f0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:09.005 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:09.002+0000 7fc92be00640 1 -- 172.21.15.19:0/1863919381 >> 172.21.15.19:0/1863919381 conn(0x7fc92c0fbb50 msgr2=0x7fc92c102b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:09.005 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:09.004+0000 7fc92be00640 1 -- 172.21.15.19:0/1863919381 shutdown_connections 2024-03-18T15:15:09.008 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:09.004+0000 7fc92be00640 1 -- 172.21.15.19:0/1863919381 wait complete. 2024-03-18T15:15:09.030 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-03-18T15:15:09.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:08 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/748133040' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T15:15:09.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:08 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:15:09.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:08 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:15:09.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:08 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:15:09.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:08 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:15:09.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:08 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:15:09.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:08 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:15:09.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:08 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/3826987739' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-18T15:15:09.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:08 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/748133040' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T15:15:09.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:08 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:15:09.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:08 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:15:09.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:08 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:15:09.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:08 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:15:09.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:08 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:15:09.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:08 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:15:09.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:08 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/3826987739' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-18T15:15:09.232 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:09.231+0000 7fa124600640 1 -- 172.21.15.19:0/2430210739 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa11c10dde0 msgr2=0x7fa11c1101d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:09.232 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:09.231+0000 7fa124600640 1 --2- 172.21.15.19:0/2430210739 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa11c10dde0 0x7fa11c1101d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa114009be0 tx=0x7fa11402bef0 comp rx=0 tx=0).stop 2024-03-18T15:15:09.232 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:09.231+0000 7fa124600640 1 -- 172.21.15.19:0/2430210739 shutdown_connections 2024-03-18T15:15:09.233 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:09.231+0000 7fa124600640 1 --2- 172.21.15.19:0/2430210739 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa11c10dde0 0x7fa11c1101d0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:09.233 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:09.231+0000 7fa124600640 1 --2- 172.21.15.19:0/2430210739 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa11c074240 0x7fa11c0746c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:09.233 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:09.231+0000 7fa124600640 1 --2- 172.21.15.19:0/2430210739 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa11c075820 0x7fa11c073c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:09.233 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:09.231+0000 7fa124600640 1 -- 172.21.15.19:0/2430210739 >> 172.21.15.19:0/2430210739 conn(0x7fa11c0fcd80 msgr2=0x7fa11c0ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:09.233 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:09.231+0000 7fa124600640 1 -- 172.21.15.19:0/2430210739 shutdown_connections 2024-03-18T15:15:09.233 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:09.231+0000 7fa124600640 1 -- 172.21.15.19:0/2430210739 wait complete. 2024-03-18T15:15:09.235 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:09.232+0000 7fa124600640 1 Processor -- start 2024-03-18T15:15:09.235 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:09.232+0000 7fa124600640 1 -- start start 2024-03-18T15:15:09.235 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:09.233+0000 7fa124600640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa11c074240 0x7fa11c19e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:09.236 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:09.233+0000 7fa124600640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa11c075820 0x7fa11c19e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:09.236 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:09.233+0000 7fa124600640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa11c10dde0 0x7fa11c1a56b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:09.236 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:09.233+0000 7fa124600640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fa11c113ab0 con 0x7fa11c074240 2024-03-18T15:15:09.236 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:09.233+0000 7fa124600640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa11c113930 con 0x7fa11c075820 2024-03-18T15:15:09.236 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:09.233+0000 7fa124600640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fa11c113c30 con 0x7fa11c10dde0 2024-03-18T15:15:09.236 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:09.233+0000 7fa121c00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa11c074240 0x7fa11c19e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:09.236 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:09.233+0000 7fa121200640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa11c075820 0x7fa11c19e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:09.236 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:09.233+0000 7fa121c00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa11c074240 0x7fa11c19e100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:59394/0 (socket says 172.21.15.19:59394) 2024-03-18T15:15:09.236 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:09.233+0000 7fa121c00640 1 -- 172.21.15.19:0/284039637 learned_addr learned my addr 172.21.15.19:0/284039637 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:15:09.236 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:09.233+0000 7fa122600640 1 --2- 172.21.15.19:0/284039637 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa11c10dde0 0x7fa11c1a56b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:09.236 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:09.233+0000 7fa121200640 1 -- 172.21.15.19:0/284039637 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa11c10dde0 msgr2=0x7fa11c1a56b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:09.236 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:09.233+0000 7fa121200640 1 --2- 172.21.15.19:0/284039637 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa11c10dde0 0x7fa11c1a56b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:09.236 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:09.233+0000 7fa121200640 1 -- 172.21.15.19:0/284039637 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa11c074240 msgr2=0x7fa11c19e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:09.237 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:09.233+0000 7fa121200640 1 --2- 172.21.15.19:0/284039637 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa11c074240 0x7fa11c19e100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:09.237 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:09.233+0000 7fa121200640 1 -- 172.21.15.19:0/284039637 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa11c1a5d20 con 0x7fa11c075820 2024-03-18T15:15:09.237 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:09.234+0000 7fa121c00640 1 --2- 172.21.15.19:0/284039637 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa11c074240 0x7fa11c19e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:15:09.237 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:09.234+0000 7fa121200640 1 --2- 172.21.15.19:0/284039637 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa11c075820 0x7fa11c19e640 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fa10c00e9f0 tx=0x7fa10c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:09.237 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:09.234+0000 7fa11a000640 1 -- 172.21.15.19:0/284039637 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa10c00ccc0 con 0x7fa11c075820 2024-03-18T15:15:09.237 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:09.234+0000 7fa11a000640 1 -- 172.21.15.19:0/284039637 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa10c010910 con 0x7fa11c075820 2024-03-18T15:15:09.237 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:09.234+0000 7fa124600640 1 -- 172.21.15.19:0/284039637 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa11c1a5fb0 con 0x7fa11c075820 2024-03-18T15:15:09.237 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:09.234+0000 7fa11a000640 1 -- 172.21.15.19:0/284039637 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa10c018960 con 0x7fa11c075820 2024-03-18T15:15:09.237 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:09.234+0000 7fa124600640 1 -- 172.21.15.19:0/284039637 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa11c1a64d0 con 0x7fa11c075820 2024-03-18T15:15:09.237 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:09.235+0000 7fa124600640 1 -- 172.21.15.19:0/284039637 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa11c103fb0 con 0x7fa11c075820 2024-03-18T15:15:09.238 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:09.237+0000 7fa11a000640 1 -- 172.21.15.19:0/284039637 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fa10c010ac0 con 0x7fa11c075820 2024-03-18T15:15:09.239 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:09.238+0000 7fa11a000640 1 --2- 172.21.15.19:0/284039637 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fa1040a2800 0x7fa1040a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:09.240 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:09.239+0000 7fa11a000640 1 -- 172.21.15.19:0/284039637 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7497+0+0 (secure 0 0 0) 0x7fa10c0ceb80 con 0x7fa11c075820 2024-03-18T15:15:09.240 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:09.239+0000 7fa121c00640 1 --2- 172.21.15.19:0/284039637 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fa1040a2800 0x7fa1040a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:09.240 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:09.239+0000 7fa121c00640 1 --2- 172.21.15.19:0/284039637 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fa1040a2800 0x7fa1040a4cc0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fa1100058c0 tx=0x7fa110009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:09.245 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:09.242+0000 7fa11a000640 1 -- 172.21.15.19:0/284039637 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa10c09b860 con 0x7fa11c075820 2024-03-18T15:15:09.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:08 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/748133040' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T15:15:09.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:08 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:15:09.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:08 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:15:09.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:08 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:15:09.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:08 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:15:09.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:08 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:15:09.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:08 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:15:09.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:08 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/3826987739' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-18T15:15:09.456 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:09.455+0000 7fa124600640 1 -- 172.21.15.19:0/284039637 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) -- 0x7fa11c062480 con 0x7fa11c075820 2024-03-18T15:15:10.068 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:10.067+0000 7fa11a000640 1 -- 172.21.15.19:0/284039637 <== mon.2 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v83) ==== 171+0+0 (secure 0 0 0) 0x7fa10c09f900 con 0x7fa11c075820 2024-03-18T15:15:10.069 INFO:teuthology.orchestra.run.smithi019.stderr:pool 'cephfs_data' created 2024-03-18T15:15:10.074 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:10.073+0000 7fa124600640 1 -- 172.21.15.19:0/284039637 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fa1040a2800 msgr2=0x7fa1040a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:10.075 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:10.073+0000 7fa124600640 1 --2- 172.21.15.19:0/284039637 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fa1040a2800 0x7fa1040a4cc0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fa1100058c0 tx=0x7fa110009260 comp rx=0 tx=0).stop 2024-03-18T15:15:10.075 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:10.073+0000 7fa124600640 1 -- 172.21.15.19:0/284039637 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa11c075820 msgr2=0x7fa11c19e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:10.075 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:10.073+0000 7fa124600640 1 --2- 172.21.15.19:0/284039637 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa11c075820 0x7fa11c19e640 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fa10c00e9f0 tx=0x7fa10c00eeb0 comp rx=0 tx=0).stop 2024-03-18T15:15:10.075 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:10.074+0000 7fa124600640 1 -- 172.21.15.19:0/284039637 shutdown_connections 2024-03-18T15:15:10.075 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:10.074+0000 7fa124600640 1 --2- 172.21.15.19:0/284039637 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa11c10dde0 0x7fa11c1a56b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:10.075 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:10.074+0000 7fa124600640 1 --2- 172.21.15.19:0/284039637 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fa1040a2800 0x7fa1040a4cc0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:10.075 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:10.074+0000 7fa124600640 1 --2- 172.21.15.19:0/284039637 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa11c075820 0x7fa11c19e640 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:10.075 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:10.074+0000 7fa124600640 1 --2- 172.21.15.19:0/284039637 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa11c074240 0x7fa11c19e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:10.075 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:10.074+0000 7fa124600640 1 -- 172.21.15.19:0/284039637 >> 172.21.15.19:0/284039637 conn(0x7fa11c0fcd80 msgr2=0x7fa11c101180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:10.076 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:10.074+0000 7fa124600640 1 -- 172.21.15.19:0/284039637 shutdown_connections 2024-03-18T15:15:10.076 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:10.075+0000 7fa124600640 1 -- 172.21.15.19:0/284039637 wait complete. 2024-03-18T15:15:10.113 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-18T15:15:10.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:10.297+0000 7f369f600640 1 -- 172.21.15.19:0/194489893 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3698104c10 msgr2=0x7f3698105090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:10.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:10.297+0000 7f369f600640 1 --2- 172.21.15.19:0/194489893 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3698104c10 0x7f3698105090 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f3684009860 tx=0x7f368402bb70 comp rx=0 tx=0).stop 2024-03-18T15:15:10.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:10.297+0000 7f369f600640 1 -- 172.21.15.19:0/194489893 shutdown_connections 2024-03-18T15:15:10.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:10.297+0000 7f369f600640 1 --2- 172.21.15.19:0/194489893 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f36981055d0 0x7f369810be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:10.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:10.297+0000 7f369f600640 1 --2- 172.21.15.19:0/194489893 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3698104c10 0x7f3698105090 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:10.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:10.297+0000 7f369f600640 1 --2- 172.21.15.19:0/194489893 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3698103a20 0x7f3698103e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:10.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:10.297+0000 7f369f600640 1 -- 172.21.15.19:0/194489893 >> 172.21.15.19:0/194489893 conn(0x7f36980fef90 msgr2=0x7f36981013d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:10.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:10.297+0000 7f369f600640 1 -- 172.21.15.19:0/194489893 shutdown_connections 2024-03-18T15:15:10.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:10.298+0000 7f369f600640 1 -- 172.21.15.19:0/194489893 wait complete. 2024-03-18T15:15:10.301 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:10.298+0000 7f369f600640 1 Processor -- start 2024-03-18T15:15:10.301 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:10.299+0000 7f369f600640 1 -- start start 2024-03-18T15:15:10.302 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:10.299+0000 7f369f600640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3698103a20 0x7f3698199d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:10.302 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:10.299+0000 7f369f600640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3698104c10 0x7f369819a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:10.302 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:10.299+0000 7f369f600640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f36981055d0 0x7f36981a1310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:10.302 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:10.299+0000 7f369f600640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f369810f6a0 con 0x7f3698103a20 2024-03-18T15:15:10.302 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:10.299+0000 7f369f600640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f369810f520 con 0x7f36981055d0 2024-03-18T15:15:10.302 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:10.299+0000 7f369f600640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f369810f820 con 0x7f3698104c10 2024-03-18T15:15:10.302 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:10.299+0000 7f369cc00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3698103a20 0x7f3698199d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:10.302 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:10.299+0000 7f369cc00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3698103a20 0x7f3698199d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:59414/0 (socket says 172.21.15.19:59414) 2024-03-18T15:15:10.302 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:10.299+0000 7f369cc00640 1 -- 172.21.15.19:0/3667664384 learned_addr learned my addr 172.21.15.19:0/3667664384 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:15:10.302 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:10.299+0000 7f3697e00640 1 --2- 172.21.15.19:0/3667664384 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3698104c10 0x7f369819a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:10.302 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:10.299+0000 7f369d600640 1 --2- 172.21.15.19:0/3667664384 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f36981055d0 0x7f36981a1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:10.302 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:10.300+0000 7f3697e00640 1 -- 172.21.15.19:0/3667664384 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f36981055d0 msgr2=0x7f36981a1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:10.303 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:10.300+0000 7f3697e00640 1 --2- 172.21.15.19:0/3667664384 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f36981055d0 0x7f36981a1310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:10.303 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:10.300+0000 7f3697e00640 1 -- 172.21.15.19:0/3667664384 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3698103a20 msgr2=0x7f3698199d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:10.303 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:10.300+0000 7f3697e00640 1 --2- 172.21.15.19:0/3667664384 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3698103a20 0x7f3698199d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:10.303 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:10.300+0000 7f3697e00640 1 -- 172.21.15.19:0/3667664384 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36981a1980 con 0x7f3698104c10 2024-03-18T15:15:10.303 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:10.300+0000 7f3697e00640 1 --2- 172.21.15.19:0/3667664384 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3698104c10 0x7f369819a2a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f368402c0b0 tx=0x7f3684032430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:10.303 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:10.300+0000 7f3695600640 1 -- 172.21.15.19:0/3667664384 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3684041020 con 0x7f3698104c10 2024-03-18T15:15:10.303 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:10.300+0000 7f369f600640 1 -- 172.21.15.19:0/3667664384 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36981a1c10 con 0x7f3698104c10 2024-03-18T15:15:10.303 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:10.300+0000 7f369f600640 1 -- 172.21.15.19:0/3667664384 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36981a2120 con 0x7f3698104c10 2024-03-18T15:15:10.303 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:10.300+0000 7f3695600640 1 -- 172.21.15.19:0/3667664384 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f368402c9f0 con 0x7f3698104c10 2024-03-18T15:15:10.303 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:10.300+0000 7f3695600640 1 -- 172.21.15.19:0/3667664384 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3684039040 con 0x7f3698104c10 2024-03-18T15:15:10.304 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:10.302+0000 7f369f600640 1 -- 172.21.15.19:0/3667664384 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3698068a20 con 0x7f3698104c10 2024-03-18T15:15:10.304 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:10.303+0000 7f3695600640 1 -- 172.21.15.19:0/3667664384 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f3684044020 con 0x7f3698104c10 2024-03-18T15:15:10.305 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:10.304+0000 7f3695600640 1 --2- 172.21.15.19:0/3667664384 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f36800a2880 0x7f36800a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:10.306 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:10.305+0000 7f3695600640 1 -- 172.21.15.19:0/3667664384 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7879+0+0 (secure 0 0 0) 0x7f36840efa30 con 0x7f3698104c10 2024-03-18T15:15:10.306 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:10.305+0000 7f369cc00640 1 --2- 172.21.15.19:0/3667664384 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f36800a2880 0x7f36800a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:10.306 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:10.306+0000 7f369cc00640 1 --2- 172.21.15.19:0/3667664384 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f36800a2880 0x7f36800a4d40 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f368c0058c0 tx=0x7f368c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:10.309 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:10.308+0000 7f3695600640 1 -- 172.21.15.19:0/3667664384 <== mon.1 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f368404b030 con 0x7f3698104c10 2024-03-18T15:15:10.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:10 smithi104 ceph-mon[36070]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:15:10.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:10 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/1863919381' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-18T15:15:10.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:10 smithi104 ceph-mon[36070]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-18T15:15:10.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:10 smithi104 ceph-mon[36070]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-18T15:15:10.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:10 smithi104 ceph-mon[36070]: osdmap e82: 12 total, 12 up, 12 in 2024-03-18T15:15:10.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:10 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:15:10.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:10 smithi019 ceph-mon[31522]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:15:10.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:10 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/1863919381' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-18T15:15:10.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:10 smithi019 ceph-mon[31522]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-18T15:15:10.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:10 smithi019 ceph-mon[31522]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-18T15:15:10.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:10 smithi019 ceph-mon[31522]: osdmap e82: 12 total, 12 up, 12 in 2024-03-18T15:15:10.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:10 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:15:10.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:10 smithi033 ceph-mon[36648]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:15:10.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:10 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/1863919381' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-18T15:15:10.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:10 smithi033 ceph-mon[36648]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-18T15:15:10.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:10 smithi033 ceph-mon[36648]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-18T15:15:10.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:10 smithi033 ceph-mon[36648]: osdmap e82: 12 total, 12 up, 12 in 2024-03-18T15:15:10.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:10 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:15:10.587 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:10.586+0000 7f369f600640 1 -- 172.21.15.19:0/3667664384 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f36980623e0 con 0x7f3698104c10 2024-03-18T15:15:11.117 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.116+0000 7f3695600640 1 -- 172.21.15.19:0/3667664384 <== mon.1 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7f36840bcbe0 con 0x7f3698104c10 2024-03-18T15:15:11.117 INFO:teuthology.orchestra.run.smithi019.stderr:new fs with metadata pool 2 and data pool 3 2024-03-18T15:15:11.121 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.120+0000 7f369f600640 1 -- 172.21.15.19:0/3667664384 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f36800a2880 msgr2=0x7f36800a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:11.121 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.120+0000 7f369f600640 1 --2- 172.21.15.19:0/3667664384 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f36800a2880 0x7f36800a4d40 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f368c0058c0 tx=0x7f368c005f90 comp rx=0 tx=0).stop 2024-03-18T15:15:11.121 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.120+0000 7f369f600640 1 -- 172.21.15.19:0/3667664384 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3698104c10 msgr2=0x7f369819a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:11.121 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.120+0000 7f369f600640 1 --2- 172.21.15.19:0/3667664384 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3698104c10 0x7f369819a2a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f368402c0b0 tx=0x7f3684032430 comp rx=0 tx=0).stop 2024-03-18T15:15:11.121 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.120+0000 7f369f600640 1 -- 172.21.15.19:0/3667664384 shutdown_connections 2024-03-18T15:15:11.121 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.120+0000 7f369f600640 1 --2- 172.21.15.19:0/3667664384 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f36800a2880 0x7f36800a4d40 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:11.121 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.120+0000 7f369f600640 1 --2- 172.21.15.19:0/3667664384 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f36981055d0 0x7f36981a1310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:11.121 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.120+0000 7f369f600640 1 --2- 172.21.15.19:0/3667664384 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3698104c10 0x7f369819a2a0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:11.122 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.120+0000 7f369f600640 1 --2- 172.21.15.19:0/3667664384 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3698103a20 0x7f3698199d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:11.122 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.120+0000 7f369f600640 1 -- 172.21.15.19:0/3667664384 >> 172.21.15.19:0/3667664384 conn(0x7f36980fef90 msgr2=0x7f3698107fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:11.122 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.120+0000 7f369f600640 1 -- 172.21.15.19:0/3667664384 shutdown_connections 2024-03-18T15:15:11.122 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.120+0000 7f369f600640 1 -- 172.21.15.19:0/3667664384 wait complete. 2024-03-18T15:15:11.147 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-03-18T15:15:11.147 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-03-18T15:15:11.358 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.357+0000 7fa661000640 1 -- 172.21.15.19:0/148833352 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa65c1073a0 msgr2=0x7fa65c1097f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:11.359 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.357+0000 7fa661000640 1 --2- 172.21.15.19:0/148833352 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa65c1073a0 0x7fa65c1097f0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fa654009ca0 tx=0x7fa65402c100 comp rx=0 tx=0).stop 2024-03-18T15:15:11.359 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.357+0000 7fa661000640 1 -- 172.21.15.19:0/148833352 shutdown_connections 2024-03-18T15:15:11.359 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.357+0000 7fa661000640 1 --2- 172.21.15.19:0/148833352 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa65c1073a0 0x7fa65c1097f0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:11.359 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.357+0000 7fa661000640 1 --2- 172.21.15.19:0/148833352 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa65c104a70 0x7fa65c106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:11.359 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.357+0000 7fa661000640 1 --2- 172.21.15.19:0/148833352 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa65c102140 0x7fa65c104530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:11.359 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.357+0000 7fa661000640 1 -- 172.21.15.19:0/148833352 >> 172.21.15.19:0/148833352 conn(0x7fa65c0fbb90 msgr2=0x7fa65c0fdfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:11.359 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.357+0000 7fa661000640 1 -- 172.21.15.19:0/148833352 shutdown_connections 2024-03-18T15:15:11.359 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.357+0000 7fa661000640 1 -- 172.21.15.19:0/148833352 wait complete. 2024-03-18T15:15:11.361 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.358+0000 7fa661000640 1 Processor -- start 2024-03-18T15:15:11.361 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.358+0000 7fa661000640 1 -- start start 2024-03-18T15:15:11.361 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.359+0000 7fa661000640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa65c102140 0x7fa65c199b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:11.362 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.359+0000 7fa661000640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa65c104a70 0x7fa65c19a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:11.362 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.359+0000 7fa661000640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa65c1073a0 0x7fa65c1a1140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:11.362 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.359+0000 7fa661000640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fa65c10d360 con 0x7fa65c102140 2024-03-18T15:15:11.362 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.359+0000 7fa661000640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa65c10d1e0 con 0x7fa65c1073a0 2024-03-18T15:15:11.362 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.359+0000 7fa661000640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fa65c10d4e0 con 0x7fa65c104a70 2024-03-18T15:15:11.362 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.359+0000 7fa65b400640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa65c102140 0x7fa65c199b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:11.362 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.359+0000 7fa65b400640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa65c102140 0x7fa65c199b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:59432/0 (socket says 172.21.15.19:59432) 2024-03-18T15:15:11.362 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.359+0000 7fa65b400640 1 -- 172.21.15.19:0/768143876 learned_addr learned my addr 172.21.15.19:0/768143876 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:15:11.362 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.359+0000 7fa65be00640 1 --2- 172.21.15.19:0/768143876 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa65c1073a0 0x7fa65c1a1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:11.362 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.359+0000 7fa653e00640 1 --2- 172.21.15.19:0/768143876 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa65c104a70 0x7fa65c19a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:11.362 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.359+0000 7fa65b400640 1 -- 172.21.15.19:0/768143876 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa65c104a70 msgr2=0x7fa65c19a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:11.362 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.359+0000 7fa65b400640 1 --2- 172.21.15.19:0/768143876 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa65c104a70 0x7fa65c19a0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:11.363 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.359+0000 7fa65b400640 1 -- 172.21.15.19:0/768143876 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa65c1073a0 msgr2=0x7fa65c1a1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:11.363 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.359+0000 7fa65b400640 1 --2- 172.21.15.19:0/768143876 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa65c1073a0 0x7fa65c1a1140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:11.363 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.359+0000 7fa65b400640 1 -- 172.21.15.19:0/768143876 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa65c1a17b0 con 0x7fa65c102140 2024-03-18T15:15:11.363 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.360+0000 7fa65b400640 1 --2- 172.21.15.19:0/768143876 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa65c102140 0x7fa65c199b90 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fa64c00e9f0 tx=0x7fa64c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:11.363 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.360+0000 7fa658c00640 1 -- 172.21.15.19:0/768143876 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa64c00cd80 con 0x7fa65c102140 2024-03-18T15:15:11.363 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.360+0000 7fa658c00640 1 -- 172.21.15.19:0/768143876 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa64c005500 con 0x7fa65c102140 2024-03-18T15:15:11.363 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.360+0000 7fa661000640 1 -- 172.21.15.19:0/768143876 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa65c1a1aa0 con 0x7fa65c102140 2024-03-18T15:15:11.363 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.360+0000 7fa658c00640 1 -- 172.21.15.19:0/768143876 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa64c010690 con 0x7fa65c102140 2024-03-18T15:15:11.363 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.360+0000 7fa661000640 1 -- 172.21.15.19:0/768143876 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa65c1a1fa0 con 0x7fa65c102140 2024-03-18T15:15:11.363 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.361+0000 7fa661000640 1 -- 172.21.15.19:0/768143876 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa65c108e90 con 0x7fa65c102140 2024-03-18T15:15:11.364 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.363+0000 7fa658c00640 1 -- 172.21.15.19:0/768143876 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fa64c0046c0 con 0x7fa65c102140 2024-03-18T15:15:11.367 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.366+0000 7fa658c00640 1 --2- 172.21.15.19:0/768143876 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fa6400a2800 0x7fa6400a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:11.367 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.366+0000 7fa658c00640 1 -- 172.21.15.19:0/768143876 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7979+0+0 (secure 0 0 0) 0x7fa64c0cef80 con 0x7fa65c102140 2024-03-18T15:15:11.367 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.366+0000 7fa653e00640 1 --2- 172.21.15.19:0/768143876 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fa6400a2800 0x7fa6400a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:11.368 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.367+0000 7fa653e00640 1 --2- 172.21.15.19:0/768143876 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fa6400a2800 0x7fa6400a4cc0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fa65c19b0a0 tx=0x7fa64400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:11.370 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.369+0000 7fa658c00640 1 -- 172.21.15.19:0/768143876 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa64c053740 con 0x7fa65c102140 2024-03-18T15:15:11.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:11 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/284039637' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-18T15:15:11.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:11 smithi104 ceph-mon[36070]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-18T15:15:11.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:11 smithi104 ceph-mon[36070]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-03-18T15:15:11.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:11 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:15:11.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:11 smithi104 ceph-mon[36070]: osdmap e83: 12 total, 12 up, 12 in 2024-03-18T15:15:11.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:11 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:15:11.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:11 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:15:11.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:11 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:15:11.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:11 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/3667664384' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-18T15:15:11.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:11 smithi104 ceph-mon[36070]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-18T15:15:11.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:11 smithi104 ceph-mon[36070]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-18T15:15:11.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:11 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:15:11.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:11 smithi104 ceph-mon[36070]: osdmap e84: 12 total, 12 up, 12 in 2024-03-18T15:15:11.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:11 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:15:11.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:11 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:15:11.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:11 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:15:11.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:11 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:15:11.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:11 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/284039637' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-18T15:15:11.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:11 smithi019 ceph-mon[31522]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-18T15:15:11.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:11 smithi019 ceph-mon[31522]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-03-18T15:15:11.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:11 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:15:11.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:11 smithi019 ceph-mon[31522]: osdmap e83: 12 total, 12 up, 12 in 2024-03-18T15:15:11.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:11 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:15:11.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:11 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:15:11.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:11 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:15:11.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:11 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/3667664384' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-18T15:15:11.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:11 smithi019 ceph-mon[31522]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-18T15:15:11.467 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:11 smithi019 ceph-mon[31522]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-18T15:15:11.467 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:11 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:15:11.467 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:11 smithi019 ceph-mon[31522]: osdmap e84: 12 total, 12 up, 12 in 2024-03-18T15:15:11.467 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:11 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:15:11.467 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:11 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:15:11.467 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:11 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:15:11.467 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:11 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:15:11.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:11 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/284039637' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-18T15:15:11.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:11 smithi033 ceph-mon[36648]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-18T15:15:11.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:11 smithi033 ceph-mon[36648]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-03-18T15:15:11.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:11 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:15:11.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:11 smithi033 ceph-mon[36648]: osdmap e83: 12 total, 12 up, 12 in 2024-03-18T15:15:11.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:11 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:15:11.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:11 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:15:11.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:11 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:15:11.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:11 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/3667664384' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-18T15:15:11.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:11 smithi033 ceph-mon[36648]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-18T15:15:11.508 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:11 smithi033 ceph-mon[36648]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-18T15:15:11.508 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:11 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:15:11.508 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:11 smithi033 ceph-mon[36648]: osdmap e84: 12 total, 12 up, 12 in 2024-03-18T15:15:11.508 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:11 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:15:11.508 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:11 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:15:11.508 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:11 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:15:11.508 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:11 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:15:11.573 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.571+0000 7fa661000640 1 -- 172.21.15.19:0/768143876 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) -- 0x7fa65c02d250 con 0x7fa65c102140 2024-03-18T15:15:11.644 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.643+0000 7fa658c00640 1 -- 172.21.15.19:0/768143876 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v85) ==== 157+0+0 (secure 0 0 0) 0x7fa64c09c0d0 con 0x7fa65c102140 2024-03-18T15:15:11.648 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.647+0000 7fa661000640 1 -- 172.21.15.19:0/768143876 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fa6400a2800 msgr2=0x7fa6400a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:11.648 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.647+0000 7fa661000640 1 --2- 172.21.15.19:0/768143876 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fa6400a2800 0x7fa6400a4cc0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fa65c19b0a0 tx=0x7fa64400b390 comp rx=0 tx=0).stop 2024-03-18T15:15:11.648 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.647+0000 7fa661000640 1 -- 172.21.15.19:0/768143876 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa65c102140 msgr2=0x7fa65c199b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:11.649 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.647+0000 7fa661000640 1 --2- 172.21.15.19:0/768143876 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa65c102140 0x7fa65c199b90 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fa64c00e9f0 tx=0x7fa64c00edf0 comp rx=0 tx=0).stop 2024-03-18T15:15:11.649 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.648+0000 7fa661000640 1 -- 172.21.15.19:0/768143876 shutdown_connections 2024-03-18T15:15:11.649 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.648+0000 7fa661000640 1 --2- 172.21.15.19:0/768143876 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fa6400a2800 0x7fa6400a4cc0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:11.649 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.648+0000 7fa661000640 1 --2- 172.21.15.19:0/768143876 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa65c1073a0 0x7fa65c1a1140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:11.650 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.648+0000 7fa661000640 1 --2- 172.21.15.19:0/768143876 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa65c104a70 0x7fa65c19a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:11.650 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.648+0000 7fa661000640 1 --2- 172.21.15.19:0/768143876 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa65c102140 0x7fa65c199b90 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:11.650 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.648+0000 7fa661000640 1 -- 172.21.15.19:0/768143876 >> 172.21.15.19:0/768143876 conn(0x7fa65c0fbb90 msgr2=0x7fa65c101770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:11.650 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.648+0000 7fa661000640 1 -- 172.21.15.19:0/768143876 shutdown_connections 2024-03-18T15:15:11.650 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.648+0000 7fa661000640 1 -- 172.21.15.19:0/768143876 wait complete. 2024-03-18T15:15:11.669 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-03-18T15:15:11.901 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.900+0000 7fc850000640 1 -- 172.21.15.19:0/3008052133 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc848104c10 msgr2=0x7fc848105090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:11.901 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.900+0000 7fc850000640 1 --2- 172.21.15.19:0/3008052133 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc848104c10 0x7fc848105090 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fc834009860 tx=0x7fc83402bb70 comp rx=0 tx=0).stop 2024-03-18T15:15:11.902 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.900+0000 7fc850000640 1 -- 172.21.15.19:0/3008052133 shutdown_connections 2024-03-18T15:15:11.902 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.900+0000 7fc850000640 1 --2- 172.21.15.19:0/3008052133 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc8481055d0 0x7fc84810be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:11.902 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.900+0000 7fc850000640 1 --2- 172.21.15.19:0/3008052133 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc848104c10 0x7fc848105090 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:11.902 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.900+0000 7fc850000640 1 --2- 172.21.15.19:0/3008052133 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc848103a20 0x7fc848103e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:11.902 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.900+0000 7fc850000640 1 -- 172.21.15.19:0/3008052133 >> 172.21.15.19:0/3008052133 conn(0x7fc8480fefd0 msgr2=0x7fc8481013f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:11.902 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.900+0000 7fc850000640 1 -- 172.21.15.19:0/3008052133 shutdown_connections 2024-03-18T15:15:11.902 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.900+0000 7fc850000640 1 -- 172.21.15.19:0/3008052133 wait complete. 2024-03-18T15:15:11.904 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.901+0000 7fc850000640 1 Processor -- start 2024-03-18T15:15:11.904 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.901+0000 7fc850000640 1 -- start start 2024-03-18T15:15:11.904 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.902+0000 7fc850000640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc848103a20 0x7fc848199d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:11.904 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.902+0000 7fc850000640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc848104c10 0x7fc84819a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:11.904 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.902+0000 7fc850000640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc8481055d0 0x7fc8481a12b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:11.905 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.902+0000 7fc850000640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fc84810f6a0 con 0x7fc8481055d0 2024-03-18T15:15:11.905 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.902+0000 7fc850000640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc84810f520 con 0x7fc848103a20 2024-03-18T15:15:11.905 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.902+0000 7fc850000640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fc84810f820 con 0x7fc848104c10 2024-03-18T15:15:11.905 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.902+0000 7fc84e000640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc8481055d0 0x7fc8481a12b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:11.905 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.902+0000 7fc84d600640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc848103a20 0x7fc848199d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:11.905 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.902+0000 7fc84d600640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc848103a20 0x7fc848199d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:34088/0 (socket says 172.21.15.19:34088) 2024-03-18T15:15:11.905 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.902+0000 7fc84d600640 1 -- 172.21.15.19:0/2774315481 learned_addr learned my addr 172.21.15.19:0/2774315481 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:15:11.905 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.902+0000 7fc84cc00640 1 --2- 172.21.15.19:0/2774315481 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc848104c10 0x7fc84819a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:11.905 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.902+0000 7fc84d600640 1 -- 172.21.15.19:0/2774315481 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc848104c10 msgr2=0x7fc84819a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:11.905 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.902+0000 7fc84d600640 1 --2- 172.21.15.19:0/2774315481 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc848104c10 0x7fc84819a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:11.905 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.902+0000 7fc84d600640 1 -- 172.21.15.19:0/2774315481 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc8481055d0 msgr2=0x7fc8481a12b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:11.905 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.902+0000 7fc84d600640 1 --2- 172.21.15.19:0/2774315481 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc8481055d0 0x7fc8481a12b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:11.905 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.903+0000 7fc84d600640 1 -- 172.21.15.19:0/2774315481 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8481a1950 con 0x7fc848103a20 2024-03-18T15:15:11.906 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.903+0000 7fc84d600640 1 --2- 172.21.15.19:0/2774315481 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc848103a20 0x7fc848199d40 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc83c00e9a0 tx=0x7fc83c00ee60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:11.906 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.903+0000 7fc846000640 1 -- 172.21.15.19:0/2774315481 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc83c00cd80 con 0x7fc848103a20 2024-03-18T15:15:11.906 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.903+0000 7fc846000640 1 -- 172.21.15.19:0/2774315481 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc83c010910 con 0x7fc848103a20 2024-03-18T15:15:11.906 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.903+0000 7fc850000640 1 -- 172.21.15.19:0/2774315481 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8481a1c40 con 0x7fc848103a20 2024-03-18T15:15:11.906 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.903+0000 7fc846000640 1 -- 172.21.15.19:0/2774315481 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc83c018a30 con 0x7fc848103a20 2024-03-18T15:15:11.906 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.903+0000 7fc850000640 1 -- 172.21.15.19:0/2774315481 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8481a2190 con 0x7fc848103a20 2024-03-18T15:15:11.906 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.904+0000 7fc850000640 1 -- 172.21.15.19:0/2774315481 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc84810ad40 con 0x7fc848103a20 2024-03-18T15:15:11.907 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.906+0000 7fc846000640 1 -- 172.21.15.19:0/2774315481 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fc83c010ac0 con 0x7fc848103a20 2024-03-18T15:15:11.908 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.907+0000 7fc846000640 1 --2- 172.21.15.19:0/2774315481 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fc8300a2880 0x7fc8300a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:11.909 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.908+0000 7fc846000640 1 -- 172.21.15.19:0/2774315481 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 8277+0+0 (secure 0 0 0) 0x7fc83c0cee60 con 0x7fc848103a20 2024-03-18T15:15:11.909 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.908+0000 7fc84cc00640 1 --2- 172.21.15.19:0/2774315481 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fc8300a2880 0x7fc8300a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:11.909 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.909+0000 7fc84cc00640 1 --2- 172.21.15.19:0/2774315481 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fc8300a2880 0x7fc8300a4d40 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fc834009830 tx=0x7fc834034ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:11.913 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:11.912+0000 7fc846000640 1 -- 172.21.15.19:0/2774315481 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc83c053740 con 0x7fc848103a20 2024-03-18T15:15:12.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:12.130+0000 7fc850000640 1 -- 172.21.15.19:0/2774315481 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fc8480623e0 con 0x7fc848103a20 2024-03-18T15:15:12.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:12 smithi104 ceph-mon[36070]: pgmap v311: 97 pgs: 8 creating+peering, 1 active+clean, 88 unknown; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:15:12.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:12 smithi104 ceph-mon[36070]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-18T15:15:12.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:12 smithi104 ceph-mon[36070]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-18T15:15:12.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:12 smithi104 ceph-mon[36070]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-18T15:15:12.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:12 smithi104 ceph-mon[36070]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-18T15:15:12.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:12 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/768143876' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-18T15:15:12.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:12 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/768143876' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-03-18T15:15:12.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:12 smithi104 ceph-mon[36070]: osdmap e85: 12 total, 12 up, 12 in 2024-03-18T15:15:12.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:12 smithi104 ceph-mon[36070]: mds.? [v2:172.21.15.33:6838/1903591535,v1:172.21.15.33:6839/1903591535] up:active 2024-03-18T15:15:12.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:12 smithi104 ceph-mon[36070]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-18T15:15:12.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:12 smithi019 ceph-mon[31522]: pgmap v311: 97 pgs: 8 creating+peering, 1 active+clean, 88 unknown; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:15:12.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:12 smithi019 ceph-mon[31522]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-18T15:15:12.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:12 smithi019 ceph-mon[31522]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-18T15:15:12.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:12 smithi019 ceph-mon[31522]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-18T15:15:12.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:12 smithi019 ceph-mon[31522]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-18T15:15:12.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:12 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/768143876' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-18T15:15:12.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:12 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/768143876' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-03-18T15:15:12.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:12 smithi019 ceph-mon[31522]: osdmap e85: 12 total, 12 up, 12 in 2024-03-18T15:15:12.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:12 smithi019 ceph-mon[31522]: mds.? [v2:172.21.15.33:6838/1903591535,v1:172.21.15.33:6839/1903591535] up:active 2024-03-18T15:15:12.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:12 smithi019 ceph-mon[31522]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-18T15:15:12.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:12 smithi033 ceph-mon[36648]: pgmap v311: 97 pgs: 8 creating+peering, 1 active+clean, 88 unknown; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:15:12.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:12 smithi033 ceph-mon[36648]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-18T15:15:12.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:12 smithi033 ceph-mon[36648]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-18T15:15:12.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:12 smithi033 ceph-mon[36648]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-18T15:15:12.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:12 smithi033 ceph-mon[36648]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-18T15:15:12.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:12 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/768143876' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-18T15:15:12.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:12 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/768143876' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-03-18T15:15:12.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:12 smithi033 ceph-mon[36648]: osdmap e85: 12 total, 12 up, 12 in 2024-03-18T15:15:12.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:12 smithi033 ceph-mon[36648]: mds.? [v2:172.21.15.33:6838/1903591535,v1:172.21.15.33:6839/1903591535] up:active 2024-03-18T15:15:12.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:12 smithi033 ceph-mon[36648]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-18T15:15:13.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:13 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/2774315481' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-18T15:15:13.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:13 smithi104 ceph-mon[36070]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-18T15:15:13.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:13 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/2774315481' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-18T15:15:13.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:13 smithi019 ceph-mon[31522]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-18T15:15:13.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:13 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/2774315481' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-18T15:15:13.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:13 smithi033 ceph-mon[36648]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-18T15:15:14.080 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:14.078+0000 7fc846000640 1 -- 172.21.15.19:0/2774315481 <== mon.2 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v87) ==== 228+0+0 (secure 0 0 0) 0x7fc83c09be70 con 0x7fc848103a20 2024-03-18T15:15:14.080 INFO:teuthology.orchestra.run.smithi019.stderr:pool 'cephfs_data_ec' created 2024-03-18T15:15:14.085 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:14.084+0000 7fc850000640 1 -- 172.21.15.19:0/2774315481 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fc8300a2880 msgr2=0x7fc8300a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:14.086 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:14.084+0000 7fc850000640 1 --2- 172.21.15.19:0/2774315481 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fc8300a2880 0x7fc8300a4d40 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fc834009830 tx=0x7fc834034ab0 comp rx=0 tx=0).stop 2024-03-18T15:15:14.086 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:14.084+0000 7fc850000640 1 -- 172.21.15.19:0/2774315481 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc848103a20 msgr2=0x7fc848199d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:14.086 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:14.084+0000 7fc850000640 1 --2- 172.21.15.19:0/2774315481 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc848103a20 0x7fc848199d40 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc83c00e9a0 tx=0x7fc83c00ee60 comp rx=0 tx=0).stop 2024-03-18T15:15:14.086 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:14.084+0000 7fc850000640 1 -- 172.21.15.19:0/2774315481 shutdown_connections 2024-03-18T15:15:14.086 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:14.085+0000 7fc850000640 1 --2- 172.21.15.19:0/2774315481 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc8481055d0 0x7fc8481a12b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:14.086 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:14.085+0000 7fc850000640 1 --2- 172.21.15.19:0/2774315481 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc848104c10 0x7fc84819a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:14.086 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:14.085+0000 7fc850000640 1 --2- 172.21.15.19:0/2774315481 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fc8300a2880 0x7fc8300a4d40 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:14.086 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:14.085+0000 7fc850000640 1 --2- 172.21.15.19:0/2774315481 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc848103a20 0x7fc848199d40 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:14.086 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:14.085+0000 7fc850000640 1 -- 172.21.15.19:0/2774315481 >> 172.21.15.19:0/2774315481 conn(0x7fc8480fefd0 msgr2=0x7fc848107f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:14.087 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:14.085+0000 7fc850000640 1 -- 172.21.15.19:0/2774315481 shutdown_connections 2024-03-18T15:15:14.087 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:14.085+0000 7fc850000640 1 -- 172.21.15.19:0/2774315481 wait complete. 2024-03-18T15:15:14.111 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-03-18T15:15:14.301 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:14.300+0000 7fb40f200640 1 -- 172.21.15.19:0/3026832565 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb410104980 msgr2=0x7fb410106d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:14.301 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:14.300+0000 7fb40f200640 1 --2- 172.21.15.19:0/3026832565 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb410104980 0x7fb410106d70 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc009900 tx=0x7fb3fc02bc30 comp rx=0 tx=0).stop 2024-03-18T15:15:14.301 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:14.300+0000 7fb40f200640 1 -- 172.21.15.19:0/3026832565 shutdown_connections 2024-03-18T15:15:14.302 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:14.300+0000 7fb40f200640 1 --2- 172.21.15.19:0/3026832565 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb4101072b0 0x7fb410109700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:14.302 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:14.300+0000 7fb40f200640 1 --2- 172.21.15.19:0/3026832565 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb410104980 0x7fb410106d70 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:14.302 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:14.300+0000 7fb40f200640 1 --2- 172.21.15.19:0/3026832565 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb410102050 0x7fb410104440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:14.302 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:14.300+0000 7fb40f200640 1 -- 172.21.15.19:0/3026832565 >> 172.21.15.19:0/3026832565 conn(0x7fb4100fba80 msgr2=0x7fb4100fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:14.302 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:14.300+0000 7fb40f200640 1 -- 172.21.15.19:0/3026832565 shutdown_connections 2024-03-18T15:15:14.302 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:14.300+0000 7fb40f200640 1 -- 172.21.15.19:0/3026832565 wait complete. 2024-03-18T15:15:14.304 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:14.301+0000 7fb40f200640 1 Processor -- start 2024-03-18T15:15:14.304 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:14.301+0000 7fb40f200640 1 -- start start 2024-03-18T15:15:14.304 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:14.301+0000 7fb40f200640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb410102050 0x7fb4101019c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:14.304 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:14.301+0000 7fb40f200640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb410104980 0x7fb410100000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:14.304 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:14.301+0000 7fb40f200640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb4101072b0 0x7fb410100540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:14.304 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:14.301+0000 7fb40f200640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fb41010d280 con 0x7fb410104980 2024-03-18T15:15:14.304 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:14.301+0000 7fb40f200640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb41010d100 con 0x7fb410102050 2024-03-18T15:15:14.304 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:14.301+0000 7fb40f200640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fb41010d400 con 0x7fb4101072b0 2024-03-18T15:15:14.305 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:14.301+0000 7fb40de00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb410102050 0x7fb4101019c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:14.305 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:14.301+0000 7fb40de00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb410102050 0x7fb4101019c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:36700/0 (socket says 172.21.15.19:36700) 2024-03-18T15:15:14.305 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:14.301+0000 7fb40de00640 1 -- 172.21.15.19:0/4286448678 learned_addr learned my addr 172.21.15.19:0/4286448678 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:15:14.305 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:14.301+0000 7fb40d400640 1 --2- 172.21.15.19:0/4286448678 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb410104980 0x7fb410100000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:14.305 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:14.301+0000 7fb40e800640 1 --2- 172.21.15.19:0/4286448678 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb4101072b0 0x7fb410100540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:14.305 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:14.302+0000 7fb40de00640 1 -- 172.21.15.19:0/4286448678 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb4101072b0 msgr2=0x7fb410100540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:14.305 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:14.302+0000 7fb40de00640 1 --2- 172.21.15.19:0/4286448678 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb4101072b0 0x7fb410100540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:14.305 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:14.302+0000 7fb40de00640 1 -- 172.21.15.19:0/4286448678 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb410104980 msgr2=0x7fb410100000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:14.305 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:14.302+0000 7fb40de00640 1 --2- 172.21.15.19:0/4286448678 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb410104980 0x7fb410100000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:14.305 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:14.302+0000 7fb40de00640 1 -- 172.21.15.19:0/4286448678 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb410100dd0 con 0x7fb410102050 2024-03-18T15:15:14.305 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:14.302+0000 7fb40de00640 1 --2- 172.21.15.19:0/4286448678 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb410102050 0x7fb4101019c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fb40000e9f0 tx=0x7fb40000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:14.305 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:14.302+0000 7fb406a00640 1 -- 172.21.15.19:0/4286448678 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb40000cd80 con 0x7fb410102050 2024-03-18T15:15:14.305 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:14.302+0000 7fb406a00640 1 -- 172.21.15.19:0/4286448678 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb400010910 con 0x7fb410102050 2024-03-18T15:15:14.305 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:14.302+0000 7fb40f200640 1 -- 172.21.15.19:0/4286448678 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4101a19c0 con 0x7fb410102050 2024-03-18T15:15:14.306 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:14.302+0000 7fb406a00640 1 -- 172.21.15.19:0/4286448678 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb400018950 con 0x7fb410102050 2024-03-18T15:15:14.306 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:14.302+0000 7fb40f200640 1 -- 172.21.15.19:0/4286448678 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4101a1ee0 con 0x7fb410102050 2024-03-18T15:15:14.306 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:14.303+0000 7fb40f200640 1 -- 172.21.15.19:0/4286448678 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4100fd570 con 0x7fb410102050 2024-03-18T15:15:14.306 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:14.305+0000 7fb406a00640 1 -- 172.21.15.19:0/4286448678 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fb400010ac0 con 0x7fb410102050 2024-03-18T15:15:14.307 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:14.306+0000 7fb406a00640 1 --2- 172.21.15.19:0/4286448678 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fb3f40a2800 0x7fb3f40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:14.307 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:14.306+0000 7fb406a00640 1 -- 172.21.15.19:0/4286448678 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8770+0+0 (secure 0 0 0) 0x7fb4000cf1e0 con 0x7fb410102050 2024-03-18T15:15:14.307 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:14.307+0000 7fb40d400640 1 --2- 172.21.15.19:0/4286448678 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fb3f40a2800 0x7fb3f40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:14.308 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:14.307+0000 7fb40d400640 1 --2- 172.21.15.19:0/4286448678 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fb3f40a2800 0x7fb3f40a4cc0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc02c170 tx=0x7fb3fc02df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:14.311 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:14.310+0000 7fb406a00640 1 -- 172.21.15.19:0/4286448678 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb40009c000 con 0x7fb410102050 2024-03-18T15:15:14.380 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:14 smithi033 ceph-mon[36648]: pgmap v314: 97 pgs: 36 active+clean, 18 creating+peering, 43 unknown; 578 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s wr, 2 op/s 2024-03-18T15:15:14.380 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:14 smithi033 ceph-mon[36648]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-03-18T15:15:14.380 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:14 smithi033 ceph-mon[36648]: Cluster is now healthy 2024-03-18T15:15:14.380 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:14 smithi033 ceph-mon[36648]: osdmap e86: 12 total, 12 up, 12 in 2024-03-18T15:15:14.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:14 smithi104 ceph-mon[36070]: pgmap v314: 97 pgs: 36 active+clean, 18 creating+peering, 43 unknown; 578 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s wr, 2 op/s 2024-03-18T15:15:14.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:14 smithi104 ceph-mon[36070]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-03-18T15:15:14.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:14 smithi104 ceph-mon[36070]: Cluster is now healthy 2024-03-18T15:15:14.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:14 smithi104 ceph-mon[36070]: osdmap e86: 12 total, 12 up, 12 in 2024-03-18T15:15:14.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:14 smithi019 ceph-mon[31522]: pgmap v314: 97 pgs: 36 active+clean, 18 creating+peering, 43 unknown; 578 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s wr, 2 op/s 2024-03-18T15:15:14.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:14 smithi019 ceph-mon[31522]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-03-18T15:15:14.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:14 smithi019 ceph-mon[31522]: Cluster is now healthy 2024-03-18T15:15:14.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:14 smithi019 ceph-mon[31522]: osdmap e86: 12 total, 12 up, 12 in 2024-03-18T15:15:14.512 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:14.511+0000 7fb40f200640 1 -- 172.21.15.19:0/4286448678 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) -- 0x7fb41002d1b0 con 0x7fb410102050 2024-03-18T15:15:14.756 INFO:journalctl@ceph.mds.b.smithi033.stdout:Mar 18 15:15:14 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-b[67599]: 2024-03-18T15:15:14.378+0000 7f90acfba700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-18T15:15:15.083 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:15.082+0000 7fb406a00640 1 -- 172.21.15.19:0/4286448678 <== mon.2 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v88) ==== 169+0+0 (secure 0 0 0) 0x7fb40009b900 con 0x7fb410102050 2024-03-18T15:15:15.084 INFO:teuthology.orchestra.run.smithi019.stderr:set pool 4 allow_ec_overwrites to true 2024-03-18T15:15:15.089 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:15.088+0000 7fb40f200640 1 -- 172.21.15.19:0/4286448678 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fb3f40a2800 msgr2=0x7fb3f40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:15.089 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:15.088+0000 7fb40f200640 1 --2- 172.21.15.19:0/4286448678 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fb3f40a2800 0x7fb3f40a4cc0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc02c170 tx=0x7fb3fc02df90 comp rx=0 tx=0).stop 2024-03-18T15:15:15.089 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:15.088+0000 7fb40f200640 1 -- 172.21.15.19:0/4286448678 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb410102050 msgr2=0x7fb4101019c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:15.090 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:15.088+0000 7fb40f200640 1 --2- 172.21.15.19:0/4286448678 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb410102050 0x7fb4101019c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fb40000e9f0 tx=0x7fb40000eeb0 comp rx=0 tx=0).stop 2024-03-18T15:15:15.093 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:15.092+0000 7fb40f200640 1 -- 172.21.15.19:0/4286448678 shutdown_connections 2024-03-18T15:15:15.093 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:15.092+0000 7fb40f200640 1 --2- 172.21.15.19:0/4286448678 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb4101072b0 0x7fb410100540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:15.093 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:15.092+0000 7fb40f200640 1 --2- 172.21.15.19:0/4286448678 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb410104980 0x7fb410100000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:15.093 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:15.092+0000 7fb40f200640 1 --2- 172.21.15.19:0/4286448678 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fb3f40a2800 0x7fb3f40a4cc0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:15.093 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:15.092+0000 7fb40f200640 1 --2- 172.21.15.19:0/4286448678 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb410102050 0x7fb4101019c0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:15.093 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:15.092+0000 7fb40f200640 1 -- 172.21.15.19:0/4286448678 >> 172.21.15.19:0/4286448678 conn(0x7fb4100fba80 msgr2=0x7fb4101028f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:15.094 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:15.093+0000 7fb40f200640 1 -- 172.21.15.19:0/4286448678 shutdown_connections 2024-03-18T15:15:15.096 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:15.094+0000 7fb40f200640 1 -- 172.21.15.19:0/4286448678 wait complete. 2024-03-18T15:15:15.133 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-03-18T15:15:15.321 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:15.319+0000 7f3aa3400640 1 -- 172.21.15.19:0/848924792 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3a9c077d80 msgr2=0x7f3a9c078620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:15.321 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:15.319+0000 7f3aa3400640 1 --2- 172.21.15.19:0/848924792 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3a9c077d80 0x7f3a9c078620 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f3a94009ca0 tx=0x7f3a9402c100 comp rx=0 tx=0).stop 2024-03-18T15:15:15.321 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:15.319+0000 7f3aa3400640 1 -- 172.21.15.19:0/848924792 shutdown_connections 2024-03-18T15:15:15.321 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:15.319+0000 7f3aa3400640 1 --2- 172.21.15.19:0/848924792 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3a9c077d80 0x7f3a9c078620 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:15.321 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:15.319+0000 7f3aa3400640 1 --2- 172.21.15.19:0/848924792 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3a9c0773e0 0x7f3a9c077840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:15.321 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:15.319+0000 7f3aa3400640 1 --2- 172.21.15.19:0/848924792 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3a9c0761a0 0x7f3a9c0765c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:15.321 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:15.319+0000 7f3aa3400640 1 -- 172.21.15.19:0/848924792 >> 172.21.15.19:0/848924792 conn(0x7f3a9c0fefa0 msgr2=0x7f3a9c1013c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:15.321 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:15.320+0000 7f3aa3400640 1 -- 172.21.15.19:0/848924792 shutdown_connections 2024-03-18T15:15:15.321 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:15.320+0000 7f3aa3400640 1 -- 172.21.15.19:0/848924792 wait complete. 2024-03-18T15:15:15.323 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:15.320+0000 7f3aa3400640 1 Processor -- start 2024-03-18T15:15:15.324 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:15.321+0000 7f3aa3400640 1 -- start start 2024-03-18T15:15:15.324 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:15.321+0000 7f3aa3400640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3a9c0761a0 0x7f3a9c19e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:15.324 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:15.321+0000 7f3aa3400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3a9c0773e0 0x7f3a9c19e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:15.324 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:15.321+0000 7f3aa3400640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3a9c077d80 0x7f3a9c1a5610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:15.324 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:15.321+0000 7f3aa3400640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f3a9c113a70 con 0x7f3a9c0761a0 2024-03-18T15:15:15.324 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:15.321+0000 7f3aa3400640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f3a9c1138f0 con 0x7f3a9c0773e0 2024-03-18T15:15:15.324 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:15.321+0000 7f3aa3400640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f3a9c113bf0 con 0x7f3a9c077d80 2024-03-18T15:15:15.324 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:15.321+0000 7f3a9be00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3a9c0773e0 0x7f3a9c19e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:15.324 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:15.321+0000 7f3a9be00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3a9c0773e0 0x7f3a9c19e5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:36728/0 (socket says 172.21.15.19:36728) 2024-03-18T15:15:15.324 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:15.321+0000 7f3a9be00640 1 -- 172.21.15.19:0/3102855390 learned_addr learned my addr 172.21.15.19:0/3102855390 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:15:15.324 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:15.321+0000 7f3aa0a00640 1 --2- 172.21.15.19:0/3102855390 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3a9c0761a0 0x7f3a9c19e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:15.324 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:15.321+0000 7f3aa1400640 1 --2- 172.21.15.19:0/3102855390 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3a9c077d80 0x7f3a9c1a5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:15.325 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:15.322+0000 7f3a9be00640 1 -- 172.21.15.19:0/3102855390 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3a9c077d80 msgr2=0x7f3a9c1a5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:15.325 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:15.322+0000 7f3a9be00640 1 --2- 172.21.15.19:0/3102855390 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3a9c077d80 0x7f3a9c1a5610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:15.325 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:15.322+0000 7f3a9be00640 1 -- 172.21.15.19:0/3102855390 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3a9c0761a0 msgr2=0x7f3a9c19e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:15.325 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:15.322+0000 7f3a9be00640 1 --2- 172.21.15.19:0/3102855390 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3a9c0761a0 0x7f3a9c19e060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:15.325 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:15.322+0000 7f3a9be00640 1 -- 172.21.15.19:0/3102855390 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a9c1a5c80 con 0x7f3a9c0773e0 2024-03-18T15:15:15.325 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:15.322+0000 7f3a9be00640 1 --2- 172.21.15.19:0/3102855390 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3a9c0773e0 0x7f3a9c19e5a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f3a8800e9f0 tx=0x7f3a8800edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:15.325 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:15.322+0000 7f3a99600640 1 -- 172.21.15.19:0/3102855390 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a8800cd80 con 0x7f3a9c0773e0 2024-03-18T15:15:15.325 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:15.322+0000 7f3a99600640 1 -- 172.21.15.19:0/3102855390 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a88005500 con 0x7f3a9c0773e0 2024-03-18T15:15:15.325 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:15.322+0000 7f3a99600640 1 -- 172.21.15.19:0/3102855390 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a88010690 con 0x7f3a9c0773e0 2024-03-18T15:15:15.325 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:15.322+0000 7f3aa3400640 1 -- 172.21.15.19:0/3102855390 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a9c1a5f70 con 0x7f3a9c0773e0 2024-03-18T15:15:15.325 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:15.322+0000 7f3aa3400640 1 -- 172.21.15.19:0/3102855390 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a9c1a63e0 con 0x7f3a9c0773e0 2024-03-18T15:15:15.326 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:15.324+0000 7f3aa3400640 1 -- 172.21.15.19:0/3102855390 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a9c110360 con 0x7f3a9c0773e0 2024-03-18T15:15:15.327 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:15.326+0000 7f3a99600640 1 -- 172.21.15.19:0/3102855390 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f3a880046c0 con 0x7f3a9c0773e0 2024-03-18T15:15:15.329 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:15.328+0000 7f3a99600640 1 --2- 172.21.15.19:0/3102855390 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f3a840a2880 0x7f3a840a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:15.329 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:15.328+0000 7f3a99600640 1 -- 172.21.15.19:0/3102855390 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8770+0+0 (secure 0 0 0) 0x7f3a880ceff0 con 0x7f3a9c0773e0 2024-03-18T15:15:15.329 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:15.328+0000 7f3aa0a00640 1 --2- 172.21.15.19:0/3102855390 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f3a840a2880 0x7f3a840a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:15.330 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:15.329+0000 7f3aa0a00640 1 --2- 172.21.15.19:0/3102855390 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f3a840a2880 0x7f3a840a4d40 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f3a90009bd0 tx=0x7f3a90005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:15.333 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:15.332+0000 7f3a99600640 1 -- 172.21.15.19:0/3102855390 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3a8809be70 con 0x7f3a9c0773e0 2024-03-18T15:15:15.367 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:15 smithi019 ceph-mon[31522]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-18T15:15:15.367 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:15 smithi019 ceph-mon[31522]: osdmap e87: 12 total, 12 up, 12 in 2024-03-18T15:15:15.367 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:15 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/4286448678' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-18T15:15:15.367 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:15 smithi019 ceph-mon[31522]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-18T15:15:15.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:15 smithi104 ceph-mon[36070]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-18T15:15:15.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:15 smithi104 ceph-mon[36070]: osdmap e87: 12 total, 12 up, 12 in 2024-03-18T15:15:15.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:15 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/4286448678' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-18T15:15:15.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:15 smithi104 ceph-mon[36070]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-18T15:15:15.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:15 smithi033 ceph-mon[36648]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-18T15:15:15.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:15 smithi033 ceph-mon[36648]: osdmap e87: 12 total, 12 up, 12 in 2024-03-18T15:15:15.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:15 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/4286448678' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-18T15:15:15.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:15 smithi033 ceph-mon[36648]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-18T15:15:15.581 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:15.579+0000 7f3aa3400640 1 -- 172.21.15.19:0/3102855390 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) -- 0x7f3a9c0624e0 con 0x7f3a9c0773e0 2024-03-18T15:15:16.106 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.104+0000 7f3a99600640 1 -- 172.21.15.19:0/3102855390 <== mon.2 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v16) ==== 137+0+0 (secure 0 0 0) 0x7f3a8809b770 con 0x7f3a9c0773e0 2024-03-18T15:15:16.106 INFO:teuthology.orchestra.run.smithi019.stderr:added data pool 4 to fsmap 2024-03-18T15:15:16.109 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.108+0000 7f3aa3400640 1 -- 172.21.15.19:0/3102855390 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f3a840a2880 msgr2=0x7f3a840a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:16.109 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.108+0000 7f3aa3400640 1 --2- 172.21.15.19:0/3102855390 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f3a840a2880 0x7f3a840a4d40 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f3a90009bd0 tx=0x7f3a90005f90 comp rx=0 tx=0).stop 2024-03-18T15:15:16.109 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.108+0000 7f3aa3400640 1 -- 172.21.15.19:0/3102855390 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3a9c0773e0 msgr2=0x7f3a9c19e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:16.110 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.108+0000 7f3aa3400640 1 --2- 172.21.15.19:0/3102855390 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3a9c0773e0 0x7f3a9c19e5a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f3a8800e9f0 tx=0x7f3a8800edf0 comp rx=0 tx=0).stop 2024-03-18T15:15:16.111 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.109+0000 7f3aa3400640 1 -- 172.21.15.19:0/3102855390 shutdown_connections 2024-03-18T15:15:16.111 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.109+0000 7f3aa3400640 1 --2- 172.21.15.19:0/3102855390 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3a9c077d80 0x7f3a9c1a5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:16.111 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.109+0000 7f3aa3400640 1 --2- 172.21.15.19:0/3102855390 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f3a840a2880 0x7f3a840a4d40 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:16.111 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.109+0000 7f3aa3400640 1 --2- 172.21.15.19:0/3102855390 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3a9c0773e0 0x7f3a9c19e5a0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:16.111 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.109+0000 7f3aa3400640 1 --2- 172.21.15.19:0/3102855390 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3a9c0761a0 0x7f3a9c19e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:16.111 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.109+0000 7f3aa3400640 1 -- 172.21.15.19:0/3102855390 >> 172.21.15.19:0/3102855390 conn(0x7f3a9c0fefa0 msgr2=0x7f3a9c100aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:16.112 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.109+0000 7f3aa3400640 1 -- 172.21.15.19:0/3102855390 shutdown_connections 2024-03-18T15:15:16.112 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.109+0000 7f3aa3400640 1 -- 172.21.15.19:0/3102855390 wait complete. 2024-03-18T15:15:16.134 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T15:15:16.338 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.336+0000 7f9525600640 1 -- 172.21.15.19:0/1977110107 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9520102320 msgr2=0x7f9520104710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:16.338 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.336+0000 7f9525600640 1 --2- 172.21.15.19:0/1977110107 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9520102320 0x7f9520104710 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f9510009900 tx=0x7f951002bc30 comp rx=0 tx=0).stop 2024-03-18T15:15:16.338 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.336+0000 7f9525600640 1 -- 172.21.15.19:0/1977110107 shutdown_connections 2024-03-18T15:15:16.338 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.336+0000 7f9525600640 1 --2- 172.21.15.19:0/1977110107 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9520107580 0x7f9520109970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:16.338 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.336+0000 7f9525600640 1 --2- 172.21.15.19:0/1977110107 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9520104c50 0x7f9520107040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:16.338 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.336+0000 7f9525600640 1 --2- 172.21.15.19:0/1977110107 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9520102320 0x7f9520104710 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:16.338 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.336+0000 7f9525600640 1 -- 172.21.15.19:0/1977110107 >> 172.21.15.19:0/1977110107 conn(0x7f95200fbcb0 msgr2=0x7f95200fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:16.338 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.336+0000 7f9525600640 1 -- 172.21.15.19:0/1977110107 shutdown_connections 2024-03-18T15:15:16.339 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.336+0000 7f9525600640 1 -- 172.21.15.19:0/1977110107 wait complete. 2024-03-18T15:15:16.341 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.337+0000 7f9525600640 1 Processor -- start 2024-03-18T15:15:16.341 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.337+0000 7f9525600640 1 -- start start 2024-03-18T15:15:16.341 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.338+0000 7f9525600640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9520102320 0x7f9520101c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:16.341 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.338+0000 7f9525600640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9520104c50 0x7f95201002d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:16.341 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.338+0000 7f9525600640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9520107580 0x7f9520100810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:16.341 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.338+0000 7f9525600640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f952010d4e0 con 0x7f9520104c50 2024-03-18T15:15:16.341 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.338+0000 7f9525600640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f952010d360 con 0x7f9520102320 2024-03-18T15:15:16.341 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.338+0000 7f9525600640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f952010d660 con 0x7f9520107580 2024-03-18T15:15:16.341 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.338+0000 7f951ea00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9520102320 0x7f9520101c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:16.341 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.338+0000 7f951ea00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9520102320 0x7f9520101c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:36740/0 (socket says 172.21.15.19:36740) 2024-03-18T15:15:16.342 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.338+0000 7f951ea00640 1 -- 172.21.15.19:0/269625952 learned_addr learned my addr 172.21.15.19:0/269625952 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:15:16.342 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.338+0000 7f951e000640 1 --2- 172.21.15.19:0/269625952 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9520104c50 0x7f95201002d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:16.342 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.338+0000 7f951f400640 1 --2- 172.21.15.19:0/269625952 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9520107580 0x7f9520100810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:16.342 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.338+0000 7f951ea00640 1 -- 172.21.15.19:0/269625952 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9520107580 msgr2=0x7f9520100810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:16.342 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.338+0000 7f951ea00640 1 --2- 172.21.15.19:0/269625952 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9520107580 0x7f9520100810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:16.342 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.338+0000 7f951ea00640 1 -- 172.21.15.19:0/269625952 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9520104c50 msgr2=0x7f95201002d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:16.342 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.338+0000 7f951ea00640 1 --2- 172.21.15.19:0/269625952 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9520104c50 0x7f95201002d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:16.342 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.338+0000 7f951ea00640 1 -- 172.21.15.19:0/269625952 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95201010d0 con 0x7f9520102320 2024-03-18T15:15:16.342 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.339+0000 7f951ea00640 1 --2- 172.21.15.19:0/269625952 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9520102320 0x7f9520101c40 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f9510000c00 tx=0x7f951002c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:16.342 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.339+0000 7f9517400640 1 -- 172.21.15.19:0/269625952 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9510043020 con 0x7f9520102320 2024-03-18T15:15:16.342 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.339+0000 7f9517400640 1 -- 172.21.15.19:0/269625952 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f95100054e0 con 0x7f9520102320 2024-03-18T15:15:16.342 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.339+0000 7f9517400640 1 -- 172.21.15.19:0/269625952 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9510003c40 con 0x7f9520102320 2024-03-18T15:15:16.342 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.339+0000 7f9525600640 1 -- 172.21.15.19:0/269625952 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95201a1c10 con 0x7f9520102320 2024-03-18T15:15:16.342 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.339+0000 7f9525600640 1 -- 172.21.15.19:0/269625952 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95201a20e0 con 0x7f9520102320 2024-03-18T15:15:16.343 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.341+0000 7f9525600640 1 -- 172.21.15.19:0/269625952 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95200fd7a0 con 0x7f9520102320 2024-03-18T15:15:16.343 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.342+0000 7f9517400640 1 -- 172.21.15.19:0/269625952 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f9510046020 con 0x7f9520102320 2024-03-18T15:15:16.344 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.343+0000 7f9517400640 1 --2- 172.21.15.19:0/269625952 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f95040a2880 0x7f95040a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:16.344 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.343+0000 7f9517400640 1 -- 172.21.15.19:0/269625952 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f95100f0d70 con 0x7f9520102320 2024-03-18T15:15:16.344 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.343+0000 7f951e000640 1 --2- 172.21.15.19:0/269625952 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f95040a2880 0x7f95040a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:16.345 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.344+0000 7f951e000640 1 --2- 172.21.15.19:0/269625952 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f95040a2880 0x7f95040a4d40 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f95201019b0 tx=0x7f950c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:16.348 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.347+0000 7f9517400640 1 -- 172.21.15.19:0/269625952 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f951004d020 con 0x7f9520102320 2024-03-18T15:15:16.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:16 smithi104 ceph-mon[36070]: pgmap v317: 129 pgs: 78 active+clean, 13 creating+peering, 38 unknown; 578 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s wr, 3 op/s 2024-03-18T15:15:16.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:16 smithi104 ceph-mon[36070]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-18T15:15:16.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:16 smithi104 ceph-mon[36070]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-03-18T15:15:16.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:16 smithi104 ceph-mon[36070]: osdmap e88: 12 total, 12 up, 12 in 2024-03-18T15:15:16.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:16 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/3102855390' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-18T15:15:16.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:16 smithi104 ceph-mon[36070]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-18T15:15:16.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:16 smithi019 ceph-mon[31522]: pgmap v317: 129 pgs: 78 active+clean, 13 creating+peering, 38 unknown; 578 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s wr, 3 op/s 2024-03-18T15:15:16.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:16 smithi019 ceph-mon[31522]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-18T15:15:16.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:16 smithi019 ceph-mon[31522]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-03-18T15:15:16.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:16 smithi019 ceph-mon[31522]: osdmap e88: 12 total, 12 up, 12 in 2024-03-18T15:15:16.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:16 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/3102855390' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-18T15:15:16.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:16 smithi019 ceph-mon[31522]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-18T15:15:16.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:16 smithi033 ceph-mon[36648]: pgmap v317: 129 pgs: 78 active+clean, 13 creating+peering, 38 unknown; 578 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s wr, 3 op/s 2024-03-18T15:15:16.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:16 smithi033 ceph-mon[36648]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-18T15:15:16.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:16 smithi033 ceph-mon[36648]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-03-18T15:15:16.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:16 smithi033 ceph-mon[36648]: osdmap e88: 12 total, 12 up, 12 in 2024-03-18T15:15:16.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:16 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/3102855390' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-18T15:15:16.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:16 smithi033 ceph-mon[36648]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-18T15:15:16.591 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.590+0000 7f9525600640 1 -- 172.21.15.19:0/269625952 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f95200623e0 con 0x7f9520102320 2024-03-18T15:15:16.592 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.591+0000 7f9517400640 1 -- 172.21.15.19:0/269625952 <== mon.2 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+11791 (secure 0 0 0) 0x7f95100bdb70 con 0x7f9520102320 2024-03-18T15:15:16.592 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-18T15:15:16.592 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6835/1135329545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6834","nonce":1135329545},{"type":"v1","addr":"172.21.15.19:6835","nonce":1135329545}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14457,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6839/3833518557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6838","nonce":3833518557},{"type":"v1","addr":"172.21.15.19:6839","nonce":3833518557}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14475,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6841/3176583319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6840","nonce":3176583319},{"type":"v1","addr":"172.21.15.19:6841","nonce":3176583319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6833/482725809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":482725809},{"type":"v1","addr":"172.21.15.104:6833","nonce":482725809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6833/2272279159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":2272279159},{"type":"v1","addr":"172.21.15.33:6833","nonce":2272279159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24469,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6837/1778019859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6836","nonce":1778019859},{"type":"v1","addr":"172.21.15.19:6837","nonce":1778019859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24482,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/4051613151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":4051613151},{"type":"v1","addr":"172.21.15.33:6835","nonce":4051613151}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24499,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3788595957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3788595957},{"type":"v1","addr":"172.21.15.104:6835","nonce":3788595957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24506,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/160187378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":160187378},{"type":"v1","addr":"172.21.15.33:6837","nonce":160187378}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24511,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6837/260984674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":260984674},{"type":"v1","addr":"172.21.15.104:6837","nonce":260984674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24529,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6839/805341436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6838","nonce":805341436},{"type":"v1","addr":"172.21.15.104:6839","nonce":805341436}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T15:15:10.589737+0000","modified":"2024-03-18T15:15:15.582611+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24560},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24560":{"gid":24560,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.33:6839/1903591535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":1903591535},{"type":"v1","addr":"172.21.15.33:6839","nonce":1903591535}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24560,"qdb_cluster":[24560]},"id":1}]} 2024-03-18T15:15:16.592 INFO:teuthology.orchestra.run.smithi019.stderr:dumped fsmap epoch 16 2024-03-18T15:15:16.594 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.593+0000 7f9525600640 1 -- 172.21.15.19:0/269625952 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f95040a2880 msgr2=0x7f95040a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:16.594 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.593+0000 7f9525600640 1 --2- 172.21.15.19:0/269625952 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f95040a2880 0x7f95040a4d40 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f95201019b0 tx=0x7f950c005f90 comp rx=0 tx=0).stop 2024-03-18T15:15:16.595 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.593+0000 7f9525600640 1 -- 172.21.15.19:0/269625952 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9520102320 msgr2=0x7f9520101c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:16.595 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.593+0000 7f9525600640 1 --2- 172.21.15.19:0/269625952 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9520102320 0x7f9520101c40 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f9510000c00 tx=0x7f951002c650 comp rx=0 tx=0).stop 2024-03-18T15:15:16.595 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.593+0000 7f9525600640 1 -- 172.21.15.19:0/269625952 shutdown_connections 2024-03-18T15:15:16.595 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.593+0000 7f9525600640 1 --2- 172.21.15.19:0/269625952 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9520107580 0x7f9520100810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:16.595 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.593+0000 7f9525600640 1 --2- 172.21.15.19:0/269625952 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9520104c50 0x7f95201002d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:16.595 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.593+0000 7f9525600640 1 --2- 172.21.15.19:0/269625952 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f95040a2880 0x7f95040a4d40 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:16.595 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.593+0000 7f9525600640 1 --2- 172.21.15.19:0/269625952 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9520102320 0x7f9520101c40 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:16.595 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.593+0000 7f9525600640 1 -- 172.21.15.19:0/269625952 >> 172.21.15.19:0/269625952 conn(0x7f95200fbcb0 msgr2=0x7f9520102d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:16.595 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.593+0000 7f9525600640 1 -- 172.21.15.19:0/269625952 shutdown_connections 2024-03-18T15:15:16.595 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.593+0000 7f9525600640 1 -- 172.21.15.19:0/269625952 wait complete. 2024-03-18T15:15:16.613 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T15:15:16.809 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.807+0000 7fd081c00640 1 -- 172.21.15.19:0/2800442355 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd07c102130 msgr2=0x7fd07c104520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:16.809 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.807+0000 7fd081c00640 1 --2- 172.21.15.19:0/2800442355 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd07c102130 0x7fd07c104520 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fd06c009900 tx=0x7fd06c02bc30 comp rx=0 tx=0).stop 2024-03-18T15:15:16.809 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.807+0000 7fd081c00640 1 -- 172.21.15.19:0/2800442355 shutdown_connections 2024-03-18T15:15:16.809 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.807+0000 7fd081c00640 1 --2- 172.21.15.19:0/2800442355 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd07c107390 0x7fd07c109780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:16.809 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.807+0000 7fd081c00640 1 --2- 172.21.15.19:0/2800442355 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd07c104a60 0x7fd07c106e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:16.809 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.807+0000 7fd081c00640 1 --2- 172.21.15.19:0/2800442355 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd07c102130 0x7fd07c104520 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:16.809 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.807+0000 7fd081c00640 1 -- 172.21.15.19:0/2800442355 >> 172.21.15.19:0/2800442355 conn(0x7fd07c0fba60 msgr2=0x7fd07c0fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:16.809 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.807+0000 7fd081c00640 1 -- 172.21.15.19:0/2800442355 shutdown_connections 2024-03-18T15:15:16.810 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.807+0000 7fd081c00640 1 -- 172.21.15.19:0/2800442355 wait complete. 2024-03-18T15:15:16.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.808+0000 7fd081c00640 1 Processor -- start 2024-03-18T15:15:16.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.808+0000 7fd081c00640 1 -- start start 2024-03-18T15:15:16.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.809+0000 7fd081c00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd07c102130 0x7fd07c199b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:16.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.809+0000 7fd081c00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd07c104a60 0x7fd07c19a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:16.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.809+0000 7fd081c00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd07c107390 0x7fd07c1a1120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:16.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.809+0000 7fd081c00640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fd07c10d2e0 con 0x7fd07c104a60 2024-03-18T15:15:16.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.809+0000 7fd081c00640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd07c10d160 con 0x7fd07c107390 2024-03-18T15:15:16.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.809+0000 7fd081c00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fd07c10d460 con 0x7fd07c102130 2024-03-18T15:15:16.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.809+0000 7fd081200640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd07c107390 0x7fd07c1a1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:16.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.809+0000 7fd07b400640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd07c104a60 0x7fd07c19a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:16.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.809+0000 7fd081200640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd07c107390 0x7fd07c1a1120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:36754/0 (socket says 172.21.15.19:36754) 2024-03-18T15:15:16.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.809+0000 7fd081200640 1 -- 172.21.15.19:0/311255568 learned_addr learned my addr 172.21.15.19:0/311255568 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:15:16.813 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.809+0000 7fd07be00640 1 --2- 172.21.15.19:0/311255568 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd07c102130 0x7fd07c199b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:16.813 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.809+0000 7fd07b400640 1 -- 172.21.15.19:0/311255568 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd07c102130 msgr2=0x7fd07c199b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:16.813 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.809+0000 7fd07b400640 1 --2- 172.21.15.19:0/311255568 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd07c102130 0x7fd07c199b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:16.813 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.809+0000 7fd07b400640 1 -- 172.21.15.19:0/311255568 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd07c107390 msgr2=0x7fd07c1a1120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:16.813 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.809+0000 7fd07b400640 1 --2- 172.21.15.19:0/311255568 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd07c107390 0x7fd07c1a1120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:16.813 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.809+0000 7fd07b400640 1 -- 172.21.15.19:0/311255568 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd07c1a1790 con 0x7fd07c104a60 2024-03-18T15:15:16.813 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.809+0000 7fd081200640 1 --2- 172.21.15.19:0/311255568 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd07c107390 0x7fd07c1a1120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:15:16.813 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.810+0000 7fd07b400640 1 --2- 172.21.15.19:0/311255568 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd07c104a60 0x7fd07c19a0b0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fd06800e9c0 tx=0x7fd06800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:16.813 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.810+0000 7fd078c00640 1 -- 172.21.15.19:0/311255568 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd06800ccc0 con 0x7fd07c104a60 2024-03-18T15:15:16.813 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.810+0000 7fd078c00640 1 -- 172.21.15.19:0/311255568 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd068005020 con 0x7fd07c104a60 2024-03-18T15:15:16.813 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.810+0000 7fd081c00640 1 -- 172.21.15.19:0/311255568 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd07c1a1a80 con 0x7fd07c104a60 2024-03-18T15:15:16.813 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.810+0000 7fd078c00640 1 -- 172.21.15.19:0/311255568 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd068004a00 con 0x7fd07c104a60 2024-03-18T15:15:16.813 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.810+0000 7fd081c00640 1 -- 172.21.15.19:0/311255568 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd07c1a1f80 con 0x7fd07c104a60 2024-03-18T15:15:16.814 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.811+0000 7fd081c00640 1 -- 172.21.15.19:0/311255568 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd07c108e80 con 0x7fd07c104a60 2024-03-18T15:15:16.817 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.817+0000 7fd078c00640 1 -- 172.21.15.19:0/311255568 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fd0680051d0 con 0x7fd07c104a60 2024-03-18T15:15:16.819 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.818+0000 7fd078c00640 1 --2- 172.21.15.19:0/311255568 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fd0600a2840 0x7fd0600a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:16.819 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.818+0000 7fd078c00640 1 -- 172.21.15.19:0/311255568 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fd0680cec90 con 0x7fd07c104a60 2024-03-18T15:15:16.819 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.818+0000 7fd07be00640 1 --2- 172.21.15.19:0/311255568 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fd0600a2840 0x7fd0600a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:16.820 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.819+0000 7fd07be00640 1 --2- 172.21.15.19:0/311255568 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fd0600a2840 0x7fd0600a4d00 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fd06c02c7b0 tx=0x7fd06c02c880 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:16.821 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:16.820+0000 7fd078c00640 1 -- 172.21.15.19:0/311255568 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd06809bb80 con 0x7fd07c104a60 2024-03-18T15:15:17.024 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.023+0000 7fd081c00640 1 -- 172.21.15.19:0/311255568 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd07c02d400 con 0x7fd07c104a60 2024-03-18T15:15:17.028 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.026+0000 7fd078c00640 1 -- 172.21.15.19:0/311255568 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24093 (secure 0 0 0) 0x7fd06809b480 con 0x7fd07c104a60 2024-03-18T15:15:17.028 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-18T15:15:17.028 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":89,"fsid":"d4dc2ba6-e538-11ee-95c9-87774f69a715","created":"2024-03-18T15:05:57.538551+0000","modified":"2024-03-18T15:15:15.582418+0000","last_up_change":"2024-03-18T15:14:12.009522+0000","last_in_change":"2024-03-18T15:13:52.555387+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T15:09:56.843349+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T15:15:08.941047+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T15:15:10.044671+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T15:15:13.621916+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"37b5a5cd-27ae-4e77-9b05-774b817b2ee9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6802","nonce":402333948},{"type":"v1","addr":"172.21.15.19:6803","nonce":402333948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6804","nonce":402333948},{"type":"v1","addr":"172.21.15.19:6805","nonce":402333948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6808","nonce":402333948},{"type":"v1","addr":"172.21.15.19:6809","nonce":402333948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6806","nonce":402333948},{"type":"v1","addr":"172.21.15.19:6807","nonce":402333948}]},"public_addr":"172.21.15.19:6803/402333948","cluster_addr":"172.21.15.19:6805/402333948","heartbeat_back_addr":"172.21.15.19:6809/402333948","heartbeat_front_addr":"172.21.15.19:6807/402333948","state":["exists","up"]},{"osd":1,"uuid":"3a1134e8-ede7-4dbd-9be1-eefd2cbb43bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1951880113},{"type":"v1","addr":"172.21.15.33:6801","nonce":1951880113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1951880113},{"type":"v1","addr":"172.21.15.33:6803","nonce":1951880113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1951880113},{"type":"v1","addr":"172.21.15.33:6807","nonce":1951880113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1951880113},{"type":"v1","addr":"172.21.15.33:6805","nonce":1951880113}]},"public_addr":"172.21.15.33:6801/1951880113","cluster_addr":"172.21.15.33:6803/1951880113","heartbeat_back_addr":"172.21.15.33:6807/1951880113","heartbeat_front_addr":"172.21.15.33:6805/1951880113","state":["exists","up"]},{"osd":2,"uuid":"f383a164-a3a0-49fd-a9e8-9c2178d8cfb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":1004727949},{"type":"v1","addr":"172.21.15.104:6801","nonce":1004727949}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":1004727949},{"type":"v1","addr":"172.21.15.104:6803","nonce":1004727949}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":1004727949},{"type":"v1","addr":"172.21.15.104:6807","nonce":1004727949}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":1004727949},{"type":"v1","addr":"172.21.15.104:6805","nonce":1004727949}]},"public_addr":"172.21.15.104:6801/1004727949","cluster_addr":"172.21.15.104:6803/1004727949","heartbeat_back_addr":"172.21.15.104:6807/1004727949","heartbeat_front_addr":"172.21.15.104:6805/1004727949","state":["exists","up"]},{"osd":3,"uuid":"c409e116-28bf-45dd-a791-539af22abe1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6810","nonce":3571525854},{"type":"v1","addr":"172.21.15.19:6811","nonce":3571525854}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6812","nonce":3571525854},{"type":"v1","addr":"172.21.15.19:6813","nonce":3571525854}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6816","nonce":3571525854},{"type":"v1","addr":"172.21.15.19:6817","nonce":3571525854}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6814","nonce":3571525854},{"type":"v1","addr":"172.21.15.19:6815","nonce":3571525854}]},"public_addr":"172.21.15.19:6811/3571525854","cluster_addr":"172.21.15.19:6813/3571525854","heartbeat_back_addr":"172.21.15.19:6817/3571525854","heartbeat_front_addr":"172.21.15.19:6815/3571525854","state":["exists","up"]},{"osd":4,"uuid":"696eb719-0b3e-43ae-b5f1-ccd73a3b2843","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":624901749},{"type":"v1","addr":"172.21.15.33:6809","nonce":624901749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":624901749},{"type":"v1","addr":"172.21.15.33:6811","nonce":624901749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":624901749},{"type":"v1","addr":"172.21.15.33:6815","nonce":624901749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":624901749},{"type":"v1","addr":"172.21.15.33:6813","nonce":624901749}]},"public_addr":"172.21.15.33:6809/624901749","cluster_addr":"172.21.15.33:6811/624901749","heartbeat_back_addr":"172.21.15.33:6815/624901749","heartbeat_front_addr":"172.21.15.33:6813/624901749","state":["exists","up"]},{"osd":5,"uuid":"21789779-8bb9-42c7-b9ee-437985f11e3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":1258208297},{"type":"v1","addr":"172.21.15.104:6809","nonce":1258208297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":1258208297},{"type":"v1","addr":"172.21.15.104:6811","nonce":1258208297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":1258208297},{"type":"v1","addr":"172.21.15.104:6815","nonce":1258208297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":1258208297},{"type":"v1","addr":"172.21.15.104:6813","nonce":1258208297}]},"public_addr":"172.21.15.104:6809/1258208297","cluster_addr":"172.21.15.104:6811/1258208297","heartbeat_back_addr":"172.21.15.104:6815/1258208297","heartbeat_front_addr":"172.21.15.104:6813/1258208297","state":["exists","up"]},{"osd":6,"uuid":"0e92a2ef-43cd-487c-9203-e219158aef8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6818","nonce":2782482301},{"type":"v1","addr":"172.21.15.19:6819","nonce":2782482301}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6820","nonce":2782482301},{"type":"v1","addr":"172.21.15.19:6821","nonce":2782482301}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6824","nonce":2782482301},{"type":"v1","addr":"172.21.15.19:6825","nonce":2782482301}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6822","nonce":2782482301},{"type":"v1","addr":"172.21.15.19:6823","nonce":2782482301}]},"public_addr":"172.21.15.19:6819/2782482301","cluster_addr":"172.21.15.19:6821/2782482301","heartbeat_back_addr":"172.21.15.19:6825/2782482301","heartbeat_front_addr":"172.21.15.19:6823/2782482301","state":["exists","up"]},{"osd":7,"uuid":"a16e39a1-3685-491d-b376-3e80a2084072","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":4238160988},{"type":"v1","addr":"172.21.15.33:6817","nonce":4238160988}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":4238160988},{"type":"v1","addr":"172.21.15.33:6819","nonce":4238160988}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":4238160988},{"type":"v1","addr":"172.21.15.33:6823","nonce":4238160988}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":4238160988},{"type":"v1","addr":"172.21.15.33:6821","nonce":4238160988}]},"public_addr":"172.21.15.33:6817/4238160988","cluster_addr":"172.21.15.33:6819/4238160988","heartbeat_back_addr":"172.21.15.33:6823/4238160988","heartbeat_front_addr":"172.21.15.33:6821/4238160988","state":["exists","up"]},{"osd":8,"uuid":"d5c3fa92-cd5d-4349-9c7d-d5666bacde16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":1014779813},{"type":"v1","addr":"172.21.15.104:6817","nonce":1014779813}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":1014779813},{"type":"v1","addr":"172.21.15.104:6819","nonce":1014779813}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":1014779813},{"type":"v1","addr":"172.21.15.104:6823","nonce":1014779813}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":1014779813},{"type":"v1","addr":"172.21.15.104:6821","nonce":1014779813}]},"public_addr":"172.21.15.104:6817/1014779813","cluster_addr":"172.21.15.104:6819/1014779813","heartbeat_back_addr":"172.21.15.104:6823/1014779813","heartbeat_front_addr":"172.21.15.104:6821/1014779813","state":["exists","up"]},{"osd":9,"uuid":"eed2e9b2-26d3-4c46-88ec-34b5596fe991","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6826","nonce":2641151009},{"type":"v1","addr":"172.21.15.19:6827","nonce":2641151009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6828","nonce":2641151009},{"type":"v1","addr":"172.21.15.19:6829","nonce":2641151009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6832","nonce":2641151009},{"type":"v1","addr":"172.21.15.19:6833","nonce":2641151009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6830","nonce":2641151009},{"type":"v1","addr":"172.21.15.19:6831","nonce":2641151009}]},"public_addr":"172.21.15.19:6827/2641151009","cluster_addr":"172.21.15.19:6829/2641151009","heartbeat_back_addr":"172.21.15.19:6833/2641151009","heartbeat_front_addr":"172.21.15.19:6831/2641151009","state":["exists","up"]},{"osd":10,"uuid":"f3c35ea5-fbe7-4dc4-93c5-b92d1a714db3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":1346058869},{"type":"v1","addr":"172.21.15.33:6825","nonce":1346058869}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":1346058869},{"type":"v1","addr":"172.21.15.33:6827","nonce":1346058869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":1346058869},{"type":"v1","addr":"172.21.15.33:6831","nonce":1346058869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":1346058869},{"type":"v1","addr":"172.21.15.33:6829","nonce":1346058869}]},"public_addr":"172.21.15.33:6825/1346058869","cluster_addr":"172.21.15.33:6827/1346058869","heartbeat_back_addr":"172.21.15.33:6831/1346058869","heartbeat_front_addr":"172.21.15.33:6829/1346058869","state":["exists","up"]},{"osd":11,"uuid":"c8b51571-f807-4b62-90b6-7529e09e59da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":3759498281},{"type":"v1","addr":"172.21.15.104:6825","nonce":3759498281}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":3759498281},{"type":"v1","addr":"172.21.15.104:6827","nonce":3759498281}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":3759498281},{"type":"v1","addr":"172.21.15.104:6831","nonce":3759498281}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":3759498281},{"type":"v1","addr":"172.21.15.104:6829","nonce":3759498281}]},"public_addr":"172.21.15.104:6825/3759498281","cluster_addr":"172.21.15.104:6827/3759498281","heartbeat_back_addr":"172.21.15.104:6831/3759498281","heartbeat_front_addr":"172.21.15.104:6829/3759498281","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:09:05.267795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:09:30.234068+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:09:53.870232+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:10:26.471296+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:10:53.180341+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:11:17.801361+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:11:51.364816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:12:18.306408+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:12:43.978744+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:13:18.213964+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:13:45.437544+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:14:10.148580+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.19:0/3860783274":"2024-03-19T15:07:14.150983+0000","172.21.15.19:0/2737223213":"2024-03-19T15:07:14.150983+0000","172.21.15.19:0/176699025":"2024-03-19T15:06:38.688405+0000","172.21.15.19:0/2316969005":"2024-03-19T15:07:14.150983+0000","172.21.15.19:6801/2776008962":"2024-03-19T15:07:14.150983+0000","172.21.15.19:0/2793980440":"2024-03-19T15:06:38.688405+0000","172.21.15.19:6800/3264337803":"2024-03-19T15:06:38.688405+0000","172.21.15.19:6801/3264337803":"2024-03-19T15:06:38.688405+0000","172.21.15.19:0/3166705295":"2024-03-19T15:06:38.688405+0000","172.21.15.19:6800/2776008962":"2024-03-19T15:07:14.150983+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T15:15:17.030 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.029+0000 7fd081c00640 1 -- 172.21.15.19:0/311255568 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fd0600a2840 msgr2=0x7fd0600a4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:17.031 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.029+0000 7fd081c00640 1 --2- 172.21.15.19:0/311255568 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fd0600a2840 0x7fd0600a4d00 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fd06c02c7b0 tx=0x7fd06c02c880 comp rx=0 tx=0).stop 2024-03-18T15:15:17.031 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.029+0000 7fd081c00640 1 -- 172.21.15.19:0/311255568 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd07c104a60 msgr2=0x7fd07c19a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:17.031 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.029+0000 7fd081c00640 1 --2- 172.21.15.19:0/311255568 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd07c104a60 0x7fd07c19a0b0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fd06800e9c0 tx=0x7fd06800ee80 comp rx=0 tx=0).stop 2024-03-18T15:15:17.031 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.029+0000 7fd081c00640 1 -- 172.21.15.19:0/311255568 shutdown_connections 2024-03-18T15:15:17.031 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.029+0000 7fd081c00640 1 --2- 172.21.15.19:0/311255568 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fd0600a2840 0x7fd0600a4d00 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:17.031 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.029+0000 7fd081c00640 1 --2- 172.21.15.19:0/311255568 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd07c107390 0x7fd07c1a1120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:17.031 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.029+0000 7fd081c00640 1 --2- 172.21.15.19:0/311255568 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd07c104a60 0x7fd07c19a0b0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:17.031 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.029+0000 7fd081c00640 1 --2- 172.21.15.19:0/311255568 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd07c102130 0x7fd07c199b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:17.031 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.029+0000 7fd081c00640 1 -- 172.21.15.19:0/311255568 >> 172.21.15.19:0/311255568 conn(0x7fd07c0fba60 msgr2=0x7fd07c1018b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:17.031 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.030+0000 7fd081c00640 1 -- 172.21.15.19:0/311255568 shutdown_connections 2024-03-18T15:15:17.031 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.030+0000 7fd081c00640 1 -- 172.21.15.19:0/311255568 wait complete. 2024-03-18T15:15:17.050 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T15:15:17.248 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.246+0000 7fb673200640 1 -- 172.21.15.19:0/3983927093 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb66c102070 msgr2=0x7fb66c104460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:17.249 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.246+0000 7fb673200640 1 --2- 172.21.15.19:0/3983927093 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb66c102070 0x7fb66c104460 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb660009900 tx=0x7fb66002bc50 comp rx=0 tx=0).stop 2024-03-18T15:15:17.249 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.246+0000 7fb673200640 1 -- 172.21.15.19:0/3983927093 shutdown_connections 2024-03-18T15:15:17.249 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.246+0000 7fb673200640 1 --2- 172.21.15.19:0/3983927093 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb66c1072d0 0x7fb66c109720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:17.249 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.246+0000 7fb673200640 1 --2- 172.21.15.19:0/3983927093 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb66c1049a0 0x7fb66c106d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:17.249 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.246+0000 7fb673200640 1 --2- 172.21.15.19:0/3983927093 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb66c102070 0x7fb66c104460 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:17.249 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.246+0000 7fb673200640 1 -- 172.21.15.19:0/3983927093 >> 172.21.15.19:0/3983927093 conn(0x7fb66c0fba60 msgr2=0x7fb66c0fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:17.249 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.246+0000 7fb673200640 1 -- 172.21.15.19:0/3983927093 shutdown_connections 2024-03-18T15:15:17.249 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.246+0000 7fb673200640 1 -- 172.21.15.19:0/3983927093 wait complete. 2024-03-18T15:15:17.249 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.247+0000 7fb673200640 1 Processor -- start 2024-03-18T15:15:17.249 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.248+0000 7fb673200640 1 -- start start 2024-03-18T15:15:17.251 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.248+0000 7fb673200640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb66c102070 0x7fb66c1019e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:17.251 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.248+0000 7fb673200640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb66c1049a0 0x7fb66c100020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:17.252 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.248+0000 7fb673200640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb66c1072d0 0x7fb66c100560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:17.252 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.248+0000 7fb673200640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fb66c10d2d0 con 0x7fb66c1072d0 2024-03-18T15:15:17.252 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.248+0000 7fb673200640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb66c10d150 con 0x7fb66c1049a0 2024-03-18T15:15:17.252 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.248+0000 7fb673200640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fb66c10d450 con 0x7fb66c102070 2024-03-18T15:15:17.252 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.248+0000 7fb671400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb66c1049a0 0x7fb66c100020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:17.252 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.248+0000 7fb671400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb66c1049a0 0x7fb66c100020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:36768/0 (socket says 172.21.15.19:36768) 2024-03-18T15:15:17.252 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.248+0000 7fb671400640 1 -- 172.21.15.19:0/1471811895 learned_addr learned my addr 172.21.15.19:0/1471811895 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:15:17.252 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.248+0000 7fb671e00640 1 --2- 172.21.15.19:0/1471811895 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb66c102070 0x7fb66c1019e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:17.252 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.248+0000 7fb672800640 1 --2- 172.21.15.19:0/1471811895 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb66c1072d0 0x7fb66c100560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:17.252 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.249+0000 7fb671400640 1 -- 172.21.15.19:0/1471811895 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb66c102070 msgr2=0x7fb66c1019e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:17.252 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.249+0000 7fb671400640 1 --2- 172.21.15.19:0/1471811895 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb66c102070 0x7fb66c1019e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:17.252 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.249+0000 7fb671400640 1 -- 172.21.15.19:0/1471811895 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb66c1072d0 msgr2=0x7fb66c100560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:17.252 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.249+0000 7fb671400640 1 --2- 172.21.15.19:0/1471811895 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb66c1072d0 0x7fb66c100560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:17.253 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.249+0000 7fb671400640 1 -- 172.21.15.19:0/1471811895 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb66c100df0 con 0x7fb66c1049a0 2024-03-18T15:15:17.253 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.249+0000 7fb671400640 1 --2- 172.21.15.19:0/1471811895 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb66c1049a0 0x7fb66c100020 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fb65c00e9f0 tx=0x7fb65c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:17.253 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.249+0000 7fb666a00640 1 -- 172.21.15.19:0/1471811895 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb65c00ccc0 con 0x7fb66c1049a0 2024-03-18T15:15:17.253 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.249+0000 7fb673200640 1 -- 172.21.15.19:0/1471811895 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb66c1a19d0 con 0x7fb66c1049a0 2024-03-18T15:15:17.253 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.249+0000 7fb666a00640 1 -- 172.21.15.19:0/1471811895 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb65c005500 con 0x7fb66c1049a0 2024-03-18T15:15:17.253 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.249+0000 7fb666a00640 1 -- 172.21.15.19:0/1471811895 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb65c010690 con 0x7fb66c1049a0 2024-03-18T15:15:17.253 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.249+0000 7fb673200640 1 -- 172.21.15.19:0/1471811895 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb66c1a1ed0 con 0x7fb66c1049a0 2024-03-18T15:15:17.253 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.250+0000 7fb673200640 1 -- 172.21.15.19:0/1471811895 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb66c0fd550 con 0x7fb66c1049a0 2024-03-18T15:15:17.253 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.253+0000 7fb666a00640 1 -- 172.21.15.19:0/1471811895 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fb65c0046c0 con 0x7fb66c1049a0 2024-03-18T15:15:17.255 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.254+0000 7fb666a00640 1 --2- 172.21.15.19:0/1471811895 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fb6540a2800 0x7fb6540a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:17.255 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.254+0000 7fb666a00640 1 -- 172.21.15.19:0/1471811895 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fb65c0cf0c0 con 0x7fb66c1049a0 2024-03-18T15:15:17.255 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.254+0000 7fb671e00640 1 --2- 172.21.15.19:0/1471811895 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fb6540a2800 0x7fb6540a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:17.256 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.255+0000 7fb671e00640 1 --2- 172.21.15.19:0/1471811895 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fb6540a2800 0x7fb6540a4cc0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fb66002c260 tx=0x7fb660000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:17.259 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.258+0000 7fb666a00640 1 -- 172.21.15.19:0/1471811895 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb65c09bec0 con 0x7fb66c1049a0 2024-03-18T15:15:17.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:17 smithi104 ceph-mon[36070]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-03-18T15:15:17.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:17 smithi104 ceph-mon[36070]: osdmap e89: 12 total, 12 up, 12 in 2024-03-18T15:15:17.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:17 smithi104 ceph-mon[36070]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-18T15:15:17.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:17 smithi104 ceph-mon[36070]: pgmap v320: 129 pgs: 110 active+clean, 3 creating+peering, 16 unknown; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s wr, 8 op/s 2024-03-18T15:15:17.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:17 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/269625952' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:15:17.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:17 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/311255568' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:15:17.456 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.455+0000 7fb673200640 1 -- 172.21.15.19:0/1471811895 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb66c0623e0 con 0x7fb66c1049a0 2024-03-18T15:15:17.458 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.457+0000 7fb666a00640 1 -- 172.21.15.19:0/1471811895 <== mon.2 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24093 (secure 0 0 0) 0x7fb65c09b7c0 con 0x7fb66c1049a0 2024-03-18T15:15:17.459 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-18T15:15:17.459 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":89,"fsid":"d4dc2ba6-e538-11ee-95c9-87774f69a715","created":"2024-03-18T15:05:57.538551+0000","modified":"2024-03-18T15:15:15.582418+0000","last_up_change":"2024-03-18T15:14:12.009522+0000","last_in_change":"2024-03-18T15:13:52.555387+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T15:09:56.843349+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T15:15:08.941047+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T15:15:10.044671+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T15:15:13.621916+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"37b5a5cd-27ae-4e77-9b05-774b817b2ee9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6802","nonce":402333948},{"type":"v1","addr":"172.21.15.19:6803","nonce":402333948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6804","nonce":402333948},{"type":"v1","addr":"172.21.15.19:6805","nonce":402333948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6808","nonce":402333948},{"type":"v1","addr":"172.21.15.19:6809","nonce":402333948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6806","nonce":402333948},{"type":"v1","addr":"172.21.15.19:6807","nonce":402333948}]},"public_addr":"172.21.15.19:6803/402333948","cluster_addr":"172.21.15.19:6805/402333948","heartbeat_back_addr":"172.21.15.19:6809/402333948","heartbeat_front_addr":"172.21.15.19:6807/402333948","state":["exists","up"]},{"osd":1,"uuid":"3a1134e8-ede7-4dbd-9be1-eefd2cbb43bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1951880113},{"type":"v1","addr":"172.21.15.33:6801","nonce":1951880113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1951880113},{"type":"v1","addr":"172.21.15.33:6803","nonce":1951880113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1951880113},{"type":"v1","addr":"172.21.15.33:6807","nonce":1951880113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1951880113},{"type":"v1","addr":"172.21.15.33:6805","nonce":1951880113}]},"public_addr":"172.21.15.33:6801/1951880113","cluster_addr":"172.21.15.33:6803/1951880113","heartbeat_back_addr":"172.21.15.33:6807/1951880113","heartbeat_front_addr":"172.21.15.33:6805/1951880113","state":["exists","up"]},{"osd":2,"uuid":"f383a164-a3a0-49fd-a9e8-9c2178d8cfb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":1004727949},{"type":"v1","addr":"172.21.15.104:6801","nonce":1004727949}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":1004727949},{"type":"v1","addr":"172.21.15.104:6803","nonce":1004727949}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":1004727949},{"type":"v1","addr":"172.21.15.104:6807","nonce":1004727949}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":1004727949},{"type":"v1","addr":"172.21.15.104:6805","nonce":1004727949}]},"public_addr":"172.21.15.104:6801/1004727949","cluster_addr":"172.21.15.104:6803/1004727949","heartbeat_back_addr":"172.21.15.104:6807/1004727949","heartbeat_front_addr":"172.21.15.104:6805/1004727949","state":["exists","up"]},{"osd":3,"uuid":"c409e116-28bf-45dd-a791-539af22abe1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6810","nonce":3571525854},{"type":"v1","addr":"172.21.15.19:6811","nonce":3571525854}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6812","nonce":3571525854},{"type":"v1","addr":"172.21.15.19:6813","nonce":3571525854}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6816","nonce":3571525854},{"type":"v1","addr":"172.21.15.19:6817","nonce":3571525854}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6814","nonce":3571525854},{"type":"v1","addr":"172.21.15.19:6815","nonce":3571525854}]},"public_addr":"172.21.15.19:6811/3571525854","cluster_addr":"172.21.15.19:6813/3571525854","heartbeat_back_addr":"172.21.15.19:6817/3571525854","heartbeat_front_addr":"172.21.15.19:6815/3571525854","state":["exists","up"]},{"osd":4,"uuid":"696eb719-0b3e-43ae-b5f1-ccd73a3b2843","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":624901749},{"type":"v1","addr":"172.21.15.33:6809","nonce":624901749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":624901749},{"type":"v1","addr":"172.21.15.33:6811","nonce":624901749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":624901749},{"type":"v1","addr":"172.21.15.33:6815","nonce":624901749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":624901749},{"type":"v1","addr":"172.21.15.33:6813","nonce":624901749}]},"public_addr":"172.21.15.33:6809/624901749","cluster_addr":"172.21.15.33:6811/624901749","heartbeat_back_addr":"172.21.15.33:6815/624901749","heartbeat_front_addr":"172.21.15.33:6813/624901749","state":["exists","up"]},{"osd":5,"uuid":"21789779-8bb9-42c7-b9ee-437985f11e3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":1258208297},{"type":"v1","addr":"172.21.15.104:6809","nonce":1258208297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":1258208297},{"type":"v1","addr":"172.21.15.104:6811","nonce":1258208297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":1258208297},{"type":"v1","addr":"172.21.15.104:6815","nonce":1258208297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":1258208297},{"type":"v1","addr":"172.21.15.104:6813","nonce":1258208297}]},"public_addr":"172.21.15.104:6809/1258208297","cluster_addr":"172.21.15.104:6811/1258208297","heartbeat_back_addr":"172.21.15.104:6815/1258208297","heartbeat_front_addr":"172.21.15.104:6813/1258208297","state":["exists","up"]},{"osd":6,"uuid":"0e92a2ef-43cd-487c-9203-e219158aef8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6818","nonce":2782482301},{"type":"v1","addr":"172.21.15.19:6819","nonce":2782482301}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6820","nonce":2782482301},{"type":"v1","addr":"172.21.15.19:6821","nonce":2782482301}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6824","nonce":2782482301},{"type":"v1","addr":"172.21.15.19:6825","nonce":2782482301}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6822","nonce":2782482301},{"type":"v1","addr":"172.21.15.19:6823","nonce":2782482301}]},"public_addr":"172.21.15.19:6819/2782482301","cluster_addr":"172.21.15.19:6821/2782482301","heartbeat_back_addr":"172.21.15.19:6825/2782482301","heartbeat_front_addr":"172.21.15.19:6823/2782482301","state":["exists","up"]},{"osd":7,"uuid":"a16e39a1-3685-491d-b376-3e80a2084072","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":4238160988},{"type":"v1","addr":"172.21.15.33:6817","nonce":4238160988}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":4238160988},{"type":"v1","addr":"172.21.15.33:6819","nonce":4238160988}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":4238160988},{"type":"v1","addr":"172.21.15.33:6823","nonce":4238160988}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":4238160988},{"type":"v1","addr":"172.21.15.33:6821","nonce":4238160988}]},"public_addr":"172.21.15.33:6817/4238160988","cluster_addr":"172.21.15.33:6819/4238160988","heartbeat_back_addr":"172.21.15.33:6823/4238160988","heartbeat_front_addr":"172.21.15.33:6821/4238160988","state":["exists","up"]},{"osd":8,"uuid":"d5c3fa92-cd5d-4349-9c7d-d5666bacde16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":1014779813},{"type":"v1","addr":"172.21.15.104:6817","nonce":1014779813}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":1014779813},{"type":"v1","addr":"172.21.15.104:6819","nonce":1014779813}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":1014779813},{"type":"v1","addr":"172.21.15.104:6823","nonce":1014779813}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":1014779813},{"type":"v1","addr":"172.21.15.104:6821","nonce":1014779813}]},"public_addr":"172.21.15.104:6817/1014779813","cluster_addr":"172.21.15.104:6819/1014779813","heartbeat_back_addr":"172.21.15.104:6823/1014779813","heartbeat_front_addr":"172.21.15.104:6821/1014779813","state":["exists","up"]},{"osd":9,"uuid":"eed2e9b2-26d3-4c46-88ec-34b5596fe991","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6826","nonce":2641151009},{"type":"v1","addr":"172.21.15.19:6827","nonce":2641151009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6828","nonce":2641151009},{"type":"v1","addr":"172.21.15.19:6829","nonce":2641151009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6832","nonce":2641151009},{"type":"v1","addr":"172.21.15.19:6833","nonce":2641151009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6830","nonce":2641151009},{"type":"v1","addr":"172.21.15.19:6831","nonce":2641151009}]},"public_addr":"172.21.15.19:6827/2641151009","cluster_addr":"172.21.15.19:6829/2641151009","heartbeat_back_addr":"172.21.15.19:6833/2641151009","heartbeat_front_addr":"172.21.15.19:6831/2641151009","state":["exists","up"]},{"osd":10,"uuid":"f3c35ea5-fbe7-4dc4-93c5-b92d1a714db3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":1346058869},{"type":"v1","addr":"172.21.15.33:6825","nonce":1346058869}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":1346058869},{"type":"v1","addr":"172.21.15.33:6827","nonce":1346058869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":1346058869},{"type":"v1","addr":"172.21.15.33:6831","nonce":1346058869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":1346058869},{"type":"v1","addr":"172.21.15.33:6829","nonce":1346058869}]},"public_addr":"172.21.15.33:6825/1346058869","cluster_addr":"172.21.15.33:6827/1346058869","heartbeat_back_addr":"172.21.15.33:6831/1346058869","heartbeat_front_addr":"172.21.15.33:6829/1346058869","state":["exists","up"]},{"osd":11,"uuid":"c8b51571-f807-4b62-90b6-7529e09e59da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":3759498281},{"type":"v1","addr":"172.21.15.104:6825","nonce":3759498281}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":3759498281},{"type":"v1","addr":"172.21.15.104:6827","nonce":3759498281}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":3759498281},{"type":"v1","addr":"172.21.15.104:6831","nonce":3759498281}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":3759498281},{"type":"v1","addr":"172.21.15.104:6829","nonce":3759498281}]},"public_addr":"172.21.15.104:6825/3759498281","cluster_addr":"172.21.15.104:6827/3759498281","heartbeat_back_addr":"172.21.15.104:6831/3759498281","heartbeat_front_addr":"172.21.15.104:6829/3759498281","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:09:05.267795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:09:30.234068+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:09:53.870232+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:10:26.471296+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:10:53.180341+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:11:17.801361+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:11:51.364816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:12:18.306408+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:12:43.978744+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:13:18.213964+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:13:45.437544+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:14:10.148580+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.19:0/3860783274":"2024-03-19T15:07:14.150983+0000","172.21.15.19:0/2737223213":"2024-03-19T15:07:14.150983+0000","172.21.15.19:0/176699025":"2024-03-19T15:06:38.688405+0000","172.21.15.19:0/2316969005":"2024-03-19T15:07:14.150983+0000","172.21.15.19:6801/2776008962":"2024-03-19T15:07:14.150983+0000","172.21.15.19:0/2793980440":"2024-03-19T15:06:38.688405+0000","172.21.15.19:6800/3264337803":"2024-03-19T15:06:38.688405+0000","172.21.15.19:6801/3264337803":"2024-03-19T15:06:38.688405+0000","172.21.15.19:0/3166705295":"2024-03-19T15:06:38.688405+0000","172.21.15.19:6800/2776008962":"2024-03-19T15:07:14.150983+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T15:15:17.461 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.459+0000 7fb673200640 1 -- 172.21.15.19:0/1471811895 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fb6540a2800 msgr2=0x7fb6540a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:17.461 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.459+0000 7fb673200640 1 --2- 172.21.15.19:0/1471811895 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fb6540a2800 0x7fb6540a4cc0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fb66002c260 tx=0x7fb660000bc0 comp rx=0 tx=0).stop 2024-03-18T15:15:17.461 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.459+0000 7fb673200640 1 -- 172.21.15.19:0/1471811895 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb66c1049a0 msgr2=0x7fb66c100020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:17.462 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.459+0000 7fb673200640 1 --2- 172.21.15.19:0/1471811895 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb66c1049a0 0x7fb66c100020 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fb65c00e9f0 tx=0x7fb65c00eeb0 comp rx=0 tx=0).stop 2024-03-18T15:15:17.462 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.460+0000 7fb673200640 1 -- 172.21.15.19:0/1471811895 shutdown_connections 2024-03-18T15:15:17.462 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.460+0000 7fb673200640 1 --2- 172.21.15.19:0/1471811895 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb66c1072d0 0x7fb66c100560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:17.462 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.460+0000 7fb673200640 1 --2- 172.21.15.19:0/1471811895 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fb6540a2800 0x7fb6540a4cc0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:17.462 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.460+0000 7fb673200640 1 --2- 172.21.15.19:0/1471811895 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb66c1049a0 0x7fb66c100020 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:17.462 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.460+0000 7fb673200640 1 --2- 172.21.15.19:0/1471811895 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb66c102070 0x7fb66c1019e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:17.463 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.460+0000 7fb673200640 1 -- 172.21.15.19:0/1471811895 >> 172.21.15.19:0/1471811895 conn(0x7fb66c0fba60 msgr2=0x7fb66c1029b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:17.463 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.460+0000 7fb673200640 1 -- 172.21.15.19:0/1471811895 shutdown_connections 2024-03-18T15:15:17.463 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.460+0000 7fb673200640 1 -- 172.21.15.19:0/1471811895 wait complete. 2024-03-18T15:15:17.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:17 smithi019 ceph-mon[31522]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-03-18T15:15:17.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:17 smithi019 ceph-mon[31522]: osdmap e89: 12 total, 12 up, 12 in 2024-03-18T15:15:17.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:17 smithi019 ceph-mon[31522]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-18T15:15:17.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:17 smithi019 ceph-mon[31522]: pgmap v320: 129 pgs: 110 active+clean, 3 creating+peering, 16 unknown; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s wr, 8 op/s 2024-03-18T15:15:17.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:17 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/269625952' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:15:17.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:17 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/311255568' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:15:17.496 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T15:15:17.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:17 smithi033 ceph-mon[36648]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-03-18T15:15:17.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:17 smithi033 ceph-mon[36648]: osdmap e89: 12 total, 12 up, 12 in 2024-03-18T15:15:17.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:17 smithi033 ceph-mon[36648]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-18T15:15:17.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:17 smithi033 ceph-mon[36648]: pgmap v320: 129 pgs: 110 active+clean, 3 creating+peering, 16 unknown; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s wr, 8 op/s 2024-03-18T15:15:17.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:17 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/269625952' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:15:17.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:17 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/311255568' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:15:17.709 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.708+0000 7ff8baa00640 1 -- 172.21.15.19:0/1165496098 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff8b41094c0 msgr2=0x7ff8b410b950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:17.710 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.708+0000 7ff8baa00640 1 --2- 172.21.15.19:0/1165496098 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff8b41094c0 0x7ff8b410b950 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7ff8ac009ca0 tx=0x7ff8ac02c100 comp rx=0 tx=0).stop 2024-03-18T15:15:17.710 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.708+0000 7ff8baa00640 1 -- 172.21.15.19:0/1165496098 shutdown_connections 2024-03-18T15:15:17.710 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.708+0000 7ff8baa00640 1 --2- 172.21.15.19:0/1165496098 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff8b41094c0 0x7ff8b410b950 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:17.710 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.708+0000 7ff8baa00640 1 --2- 172.21.15.19:0/1165496098 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff8b4074240 0x7ff8b40746c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:17.710 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.708+0000 7ff8baa00640 1 --2- 172.21.15.19:0/1165496098 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff8b4075820 0x7ff8b4073c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:17.710 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.708+0000 7ff8baa00640 1 -- 172.21.15.19:0/1165496098 >> 172.21.15.19:0/1165496098 conn(0x7ff8b40fcb90 msgr2=0x7ff8b40fefb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:17.711 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.708+0000 7ff8baa00640 1 -- 172.21.15.19:0/1165496098 shutdown_connections 2024-03-18T15:15:17.711 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.708+0000 7ff8baa00640 1 -- 172.21.15.19:0/1165496098 wait complete. 2024-03-18T15:15:17.714 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.709+0000 7ff8baa00640 1 Processor -- start 2024-03-18T15:15:17.714 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.709+0000 7ff8baa00640 1 -- start start 2024-03-18T15:15:17.714 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.710+0000 7ff8baa00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff8b4074240 0x7ff8b4199a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:17.714 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.710+0000 7ff8baa00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff8b4075820 0x7ff8b4199f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:17.714 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.710+0000 7ff8baa00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff8b41094c0 0x7ff8b41a1000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:17.714 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.710+0000 7ff8baa00640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7ff8b410f460 con 0x7ff8b4074240 2024-03-18T15:15:17.715 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.710+0000 7ff8baa00640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff8b410f2e0 con 0x7ff8b41094c0 2024-03-18T15:15:17.715 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.710+0000 7ff8baa00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7ff8b410f5e0 con 0x7ff8b4075820 2024-03-18T15:15:17.715 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.710+0000 7ff8b9600640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff8b4074240 0x7ff8b4199a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:17.715 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.710+0000 7ff8b9600640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff8b4074240 0x7ff8b4199a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:36512/0 (socket says 172.21.15.19:36512) 2024-03-18T15:15:17.715 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.710+0000 7ff8b9600640 1 -- 172.21.15.19:0/3471948888 learned_addr learned my addr 172.21.15.19:0/3471948888 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:15:17.715 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.710+0000 7ff8ba000640 1 --2- 172.21.15.19:0/3471948888 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff8b41094c0 0x7ff8b41a1000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:17.715 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.710+0000 7ff8b8c00640 1 --2- 172.21.15.19:0/3471948888 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff8b4075820 0x7ff8b4199f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:17.715 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.710+0000 7ff8ba000640 1 -- 172.21.15.19:0/3471948888 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff8b4075820 msgr2=0x7ff8b4199f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:17.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.710+0000 7ff8ba000640 1 --2- 172.21.15.19:0/3471948888 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff8b4075820 0x7ff8b4199f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:17.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.710+0000 7ff8ba000640 1 -- 172.21.15.19:0/3471948888 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff8b4074240 msgr2=0x7ff8b4199a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:17.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.710+0000 7ff8ba000640 1 --2- 172.21.15.19:0/3471948888 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff8b4074240 0x7ff8b4199a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:17.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.710+0000 7ff8ba000640 1 -- 172.21.15.19:0/3471948888 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8b41a1670 con 0x7ff8b41094c0 2024-03-18T15:15:17.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.710+0000 7ff8b9600640 1 --2- 172.21.15.19:0/3471948888 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff8b4074240 0x7ff8b4199a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:15:17.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.711+0000 7ff8ba000640 1 --2- 172.21.15.19:0/3471948888 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff8b41094c0 0x7ff8b41a1000 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ff8ac037040 tx=0x7ff8ac035d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:17.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.711+0000 7ff8b2000640 1 -- 172.21.15.19:0/3471948888 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8ac039040 con 0x7ff8b41094c0 2024-03-18T15:15:17.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.711+0000 7ff8baa00640 1 -- 172.21.15.19:0/3471948888 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8b41a1900 con 0x7ff8b41094c0 2024-03-18T15:15:17.717 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.711+0000 7ff8b2000640 1 -- 172.21.15.19:0/3471948888 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff8ac044070 con 0x7ff8b41094c0 2024-03-18T15:15:17.717 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.711+0000 7ff8b2000640 1 -- 172.21.15.19:0/3471948888 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8ac004350 con 0x7ff8b41094c0 2024-03-18T15:15:17.717 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.711+0000 7ff8baa00640 1 -- 172.21.15.19:0/3471948888 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8b41a1dc0 con 0x7ff8b41094c0 2024-03-18T15:15:17.717 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.712+0000 7ff8baa00640 1 -- 172.21.15.19:0/3471948888 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8b41081f0 con 0x7ff8b41094c0 2024-03-18T15:15:17.718 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.714+0000 7ff8b2000640 1 -- 172.21.15.19:0/3471948888 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7ff8ac0044f0 con 0x7ff8b41094c0 2024-03-18T15:15:17.718 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.715+0000 7ff8b2000640 1 --2- 172.21.15.19:0/3471948888 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7ff89c0a2800 0x7ff89c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:17.718 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.715+0000 7ff8b2000640 1 -- 172.21.15.19:0/3471948888 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7ff8ac0f1210 con 0x7ff8b41094c0 2024-03-18T15:15:17.718 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.716+0000 7ff8b9600640 1 --2- 172.21.15.19:0/3471948888 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7ff89c0a2800 0x7ff89c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:17.719 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.716+0000 7ff8b9600640 1 --2- 172.21.15.19:0/3471948888 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7ff89c0a2800 0x7ff89c0a4cc0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ff8a8011500 tx=0x7ff8a8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:17.725 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.724+0000 7ff8b2000640 1 -- 172.21.15.19:0/3471948888 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff8ac0be010 con 0x7ff8b41094c0 2024-03-18T15:15:17.959 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.958+0000 7ff8baa00640 1 -- 172.21.15.19:0/3471948888 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff8b4062480 con 0x7ff8b41094c0 2024-03-18T15:15:17.960 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.959+0000 7ff8b2000640 1 -- 172.21.15.19:0/3471948888 <== mon.2 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+11791 (secure 0 0 0) 0x7ff8ac0098b0 con 0x7ff8b41094c0 2024-03-18T15:15:17.960 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-18T15:15:17.960 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6835/1135329545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6834","nonce":1135329545},{"type":"v1","addr":"172.21.15.19:6835","nonce":1135329545}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14457,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6839/3833518557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6838","nonce":3833518557},{"type":"v1","addr":"172.21.15.19:6839","nonce":3833518557}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14475,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6841/3176583319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6840","nonce":3176583319},{"type":"v1","addr":"172.21.15.19:6841","nonce":3176583319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6833/482725809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":482725809},{"type":"v1","addr":"172.21.15.104:6833","nonce":482725809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6833/2272279159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":2272279159},{"type":"v1","addr":"172.21.15.33:6833","nonce":2272279159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24469,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6837/1778019859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6836","nonce":1778019859},{"type":"v1","addr":"172.21.15.19:6837","nonce":1778019859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24482,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/4051613151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":4051613151},{"type":"v1","addr":"172.21.15.33:6835","nonce":4051613151}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24499,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3788595957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3788595957},{"type":"v1","addr":"172.21.15.104:6835","nonce":3788595957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24506,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/160187378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":160187378},{"type":"v1","addr":"172.21.15.33:6837","nonce":160187378}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24511,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6837/260984674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":260984674},{"type":"v1","addr":"172.21.15.104:6837","nonce":260984674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24529,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6839/805341436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6838","nonce":805341436},{"type":"v1","addr":"172.21.15.104:6839","nonce":805341436}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T15:15:10.589737+0000","modified":"2024-03-18T15:15:15.582611+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24560},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24560":{"gid":24560,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.33:6839/1903591535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":1903591535},{"type":"v1","addr":"172.21.15.33:6839","nonce":1903591535}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24560,"qdb_cluster":[24560]},"id":1}]} 2024-03-18T15:15:17.961 INFO:teuthology.orchestra.run.smithi019.stderr:dumped fsmap epoch 16 2024-03-18T15:15:17.962 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.961+0000 7ff8baa00640 1 -- 172.21.15.19:0/3471948888 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7ff89c0a2800 msgr2=0x7ff89c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:17.962 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.961+0000 7ff8baa00640 1 --2- 172.21.15.19:0/3471948888 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7ff89c0a2800 0x7ff89c0a4cc0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ff8a8011500 tx=0x7ff8a8005f90 comp rx=0 tx=0).stop 2024-03-18T15:15:17.963 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.961+0000 7ff8baa00640 1 -- 172.21.15.19:0/3471948888 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff8b41094c0 msgr2=0x7ff8b41a1000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:17.963 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.961+0000 7ff8baa00640 1 --2- 172.21.15.19:0/3471948888 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff8b41094c0 0x7ff8b41a1000 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ff8ac037040 tx=0x7ff8ac035d10 comp rx=0 tx=0).stop 2024-03-18T15:15:17.963 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.961+0000 7ff8baa00640 1 -- 172.21.15.19:0/3471948888 shutdown_connections 2024-03-18T15:15:17.963 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.961+0000 7ff8baa00640 1 --2- 172.21.15.19:0/3471948888 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7ff89c0a2800 0x7ff89c0a4cc0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:17.963 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.961+0000 7ff8baa00640 1 --2- 172.21.15.19:0/3471948888 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff8b41094c0 0x7ff8b41a1000 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:17.963 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.961+0000 7ff8baa00640 1 --2- 172.21.15.19:0/3471948888 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff8b4075820 0x7ff8b4199f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:17.963 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.961+0000 7ff8baa00640 1 --2- 172.21.15.19:0/3471948888 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff8b4074240 0x7ff8b4199a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:17.963 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.961+0000 7ff8baa00640 1 -- 172.21.15.19:0/3471948888 >> 172.21.15.19:0/3471948888 conn(0x7ff8b40fcb90 msgr2=0x7ff8b41053c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:17.963 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.962+0000 7ff8baa00640 1 -- 172.21.15.19:0/3471948888 shutdown_connections 2024-03-18T15:15:17.963 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:17.962+0000 7ff8baa00640 1 -- 172.21.15.19:0/3471948888 wait complete. 2024-03-18T15:15:17.982 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T15:15:18.198 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.196+0000 7fd06be00640 1 -- 172.21.15.19:0/823734274 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd06c103810 msgr2=0x7fd06c103c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:18.198 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.196+0000 7fd06be00640 1 --2- 172.21.15.19:0/823734274 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd06c103810 0x7fd06c103c30 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd05c009900 tx=0x7fd05c02bc30 comp rx=0 tx=0).stop 2024-03-18T15:15:18.198 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.196+0000 7fd06be00640 1 -- 172.21.15.19:0/823734274 shutdown_connections 2024-03-18T15:15:18.198 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.196+0000 7fd06be00640 1 --2- 172.21.15.19:0/823734274 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd06c1053c0 0x7fd06c10bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:18.198 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.196+0000 7fd06be00640 1 --2- 172.21.15.19:0/823734274 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd06c104a00 0x7fd06c104e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:18.198 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.196+0000 7fd06be00640 1 --2- 172.21.15.19:0/823734274 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd06c103810 0x7fd06c103c30 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:18.198 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.196+0000 7fd06be00640 1 -- 172.21.15.19:0/823734274 >> 172.21.15.19:0/823734274 conn(0x7fd06c0fed80 msgr2=0x7fd06c1011c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:18.198 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.196+0000 7fd06be00640 1 -- 172.21.15.19:0/823734274 shutdown_connections 2024-03-18T15:15:18.198 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.196+0000 7fd06be00640 1 -- 172.21.15.19:0/823734274 wait complete. 2024-03-18T15:15:18.200 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.197+0000 7fd06be00640 1 Processor -- start 2024-03-18T15:15:18.201 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.198+0000 7fd06be00640 1 -- start start 2024-03-18T15:15:18.201 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.198+0000 7fd06be00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd06c103810 0x7fd06c199af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:18.201 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.198+0000 7fd06be00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd06c104a00 0x7fd06c19a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:18.201 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.198+0000 7fd06be00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd06c1053c0 0x7fd06c1a10a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:18.201 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.198+0000 7fd06be00640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fd06c10f4b0 con 0x7fd06c104a00 2024-03-18T15:15:18.201 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.198+0000 7fd06be00640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd06c10f330 con 0x7fd06c103810 2024-03-18T15:15:18.201 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.198+0000 7fd06be00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fd06c10f630 con 0x7fd06c1053c0 2024-03-18T15:15:18.201 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.198+0000 7fd06aa00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd06c103810 0x7fd06c199af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:18.201 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.198+0000 7fd06aa00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd06c103810 0x7fd06c199af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:36824/0 (socket says 172.21.15.19:36824) 2024-03-18T15:15:18.201 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.198+0000 7fd06aa00640 1 -- 172.21.15.19:0/3620641539 learned_addr learned my addr 172.21.15.19:0/3620641539 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:15:18.201 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.198+0000 7fd06a000640 1 --2- 172.21.15.19:0/3620641539 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd06c104a00 0x7fd06c19a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:18.201 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.198+0000 7fd06b400640 1 --2- 172.21.15.19:0/3620641539 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd06c1053c0 0x7fd06c1a10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:18.201 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.199+0000 7fd06aa00640 1 -- 172.21.15.19:0/3620641539 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd06c1053c0 msgr2=0x7fd06c1a10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:18.202 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.199+0000 7fd06aa00640 1 --2- 172.21.15.19:0/3620641539 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd06c1053c0 0x7fd06c1a10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:18.202 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.199+0000 7fd06aa00640 1 -- 172.21.15.19:0/3620641539 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd06c104a00 msgr2=0x7fd06c19a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:18.202 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.199+0000 7fd06aa00640 1 --2- 172.21.15.19:0/3620641539 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd06c104a00 0x7fd06c19a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:18.202 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.199+0000 7fd06aa00640 1 -- 172.21.15.19:0/3620641539 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd06c1a1710 con 0x7fd06c103810 2024-03-18T15:15:18.202 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.199+0000 7fd06b400640 1 --2- 172.21.15.19:0/3620641539 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd06c1053c0 0x7fd06c1a10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T15:15:18.202 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.199+0000 7fd06aa00640 1 --2- 172.21.15.19:0/3620641539 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd06c103810 0x7fd06c199af0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fd05c000c00 tx=0x7fd05c02c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:18.202 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.199+0000 7fd070a00640 1 -- 172.21.15.19:0/3620641539 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd05c043020 con 0x7fd06c103810 2024-03-18T15:15:18.202 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.199+0000 7fd070a00640 1 -- 172.21.15.19:0/3620641539 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd05c0054e0 con 0x7fd06c103810 2024-03-18T15:15:18.202 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.199+0000 7fd070a00640 1 -- 172.21.15.19:0/3620641539 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd05c003c40 con 0x7fd06c103810 2024-03-18T15:15:18.202 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.199+0000 7fd06be00640 1 -- 172.21.15.19:0/3620641539 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd06c1a19a0 con 0x7fd06c103810 2024-03-18T15:15:18.202 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.200+0000 7fd06be00640 1 -- 172.21.15.19:0/3620641539 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd06c1a1ef0 con 0x7fd06c103810 2024-03-18T15:15:18.203 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.201+0000 7fd06be00640 1 -- 172.21.15.19:0/3620641539 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd06c10ab30 con 0x7fd06c103810 2024-03-18T15:15:18.203 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.202+0000 7fd070a00640 1 -- 172.21.15.19:0/3620641539 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fd05c046020 con 0x7fd06c103810 2024-03-18T15:15:18.204 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.203+0000 7fd070a00640 1 --2- 172.21.15.19:0/3620641539 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fd0500a2800 0x7fd0500a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:18.204 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.204+0000 7fd070a00640 1 -- 172.21.15.19:0/3620641539 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fd05c0f0d70 con 0x7fd06c103810 2024-03-18T15:15:18.205 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.204+0000 7fd06a000640 1 --2- 172.21.15.19:0/3620641539 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fd0500a2800 0x7fd0500a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:18.205 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.204+0000 7fd06a000640 1 --2- 172.21.15.19:0/3620641539 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fd0500a2800 0x7fd0500a4cc0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fd0540095f0 tx=0x7fd054005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:18.208 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.208+0000 7fd070a00640 1 -- 172.21.15.19:0/3620641539 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd05c04d020 con 0x7fd06c103810 2024-03-18T15:15:18.405 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.403+0000 7fd06be00640 1 -- 172.21.15.19:0/3620641539 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd06c0623e0 con 0x7fd06c103810 2024-03-18T15:15:18.407 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.407+0000 7fd070a00640 1 -- 172.21.15.19:0/3620641539 <== mon.2 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24093 (secure 0 0 0) 0x7fd05c0bdb70 con 0x7fd06c103810 2024-03-18T15:15:18.408 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-18T15:15:18.408 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":89,"fsid":"d4dc2ba6-e538-11ee-95c9-87774f69a715","created":"2024-03-18T15:05:57.538551+0000","modified":"2024-03-18T15:15:15.582418+0000","last_up_change":"2024-03-18T15:14:12.009522+0000","last_in_change":"2024-03-18T15:13:52.555387+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T15:09:56.843349+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T15:15:08.941047+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T15:15:10.044671+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T15:15:13.621916+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"37b5a5cd-27ae-4e77-9b05-774b817b2ee9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6802","nonce":402333948},{"type":"v1","addr":"172.21.15.19:6803","nonce":402333948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6804","nonce":402333948},{"type":"v1","addr":"172.21.15.19:6805","nonce":402333948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6808","nonce":402333948},{"type":"v1","addr":"172.21.15.19:6809","nonce":402333948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6806","nonce":402333948},{"type":"v1","addr":"172.21.15.19:6807","nonce":402333948}]},"public_addr":"172.21.15.19:6803/402333948","cluster_addr":"172.21.15.19:6805/402333948","heartbeat_back_addr":"172.21.15.19:6809/402333948","heartbeat_front_addr":"172.21.15.19:6807/402333948","state":["exists","up"]},{"osd":1,"uuid":"3a1134e8-ede7-4dbd-9be1-eefd2cbb43bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1951880113},{"type":"v1","addr":"172.21.15.33:6801","nonce":1951880113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1951880113},{"type":"v1","addr":"172.21.15.33:6803","nonce":1951880113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1951880113},{"type":"v1","addr":"172.21.15.33:6807","nonce":1951880113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1951880113},{"type":"v1","addr":"172.21.15.33:6805","nonce":1951880113}]},"public_addr":"172.21.15.33:6801/1951880113","cluster_addr":"172.21.15.33:6803/1951880113","heartbeat_back_addr":"172.21.15.33:6807/1951880113","heartbeat_front_addr":"172.21.15.33:6805/1951880113","state":["exists","up"]},{"osd":2,"uuid":"f383a164-a3a0-49fd-a9e8-9c2178d8cfb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":1004727949},{"type":"v1","addr":"172.21.15.104:6801","nonce":1004727949}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":1004727949},{"type":"v1","addr":"172.21.15.104:6803","nonce":1004727949}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":1004727949},{"type":"v1","addr":"172.21.15.104:6807","nonce":1004727949}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":1004727949},{"type":"v1","addr":"172.21.15.104:6805","nonce":1004727949}]},"public_addr":"172.21.15.104:6801/1004727949","cluster_addr":"172.21.15.104:6803/1004727949","heartbeat_back_addr":"172.21.15.104:6807/1004727949","heartbeat_front_addr":"172.21.15.104:6805/1004727949","state":["exists","up"]},{"osd":3,"uuid":"c409e116-28bf-45dd-a791-539af22abe1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6810","nonce":3571525854},{"type":"v1","addr":"172.21.15.19:6811","nonce":3571525854}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6812","nonce":3571525854},{"type":"v1","addr":"172.21.15.19:6813","nonce":3571525854}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6816","nonce":3571525854},{"type":"v1","addr":"172.21.15.19:6817","nonce":3571525854}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6814","nonce":3571525854},{"type":"v1","addr":"172.21.15.19:6815","nonce":3571525854}]},"public_addr":"172.21.15.19:6811/3571525854","cluster_addr":"172.21.15.19:6813/3571525854","heartbeat_back_addr":"172.21.15.19:6817/3571525854","heartbeat_front_addr":"172.21.15.19:6815/3571525854","state":["exists","up"]},{"osd":4,"uuid":"696eb719-0b3e-43ae-b5f1-ccd73a3b2843","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":624901749},{"type":"v1","addr":"172.21.15.33:6809","nonce":624901749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":624901749},{"type":"v1","addr":"172.21.15.33:6811","nonce":624901749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":624901749},{"type":"v1","addr":"172.21.15.33:6815","nonce":624901749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":624901749},{"type":"v1","addr":"172.21.15.33:6813","nonce":624901749}]},"public_addr":"172.21.15.33:6809/624901749","cluster_addr":"172.21.15.33:6811/624901749","heartbeat_back_addr":"172.21.15.33:6815/624901749","heartbeat_front_addr":"172.21.15.33:6813/624901749","state":["exists","up"]},{"osd":5,"uuid":"21789779-8bb9-42c7-b9ee-437985f11e3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":1258208297},{"type":"v1","addr":"172.21.15.104:6809","nonce":1258208297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":1258208297},{"type":"v1","addr":"172.21.15.104:6811","nonce":1258208297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":1258208297},{"type":"v1","addr":"172.21.15.104:6815","nonce":1258208297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":1258208297},{"type":"v1","addr":"172.21.15.104:6813","nonce":1258208297}]},"public_addr":"172.21.15.104:6809/1258208297","cluster_addr":"172.21.15.104:6811/1258208297","heartbeat_back_addr":"172.21.15.104:6815/1258208297","heartbeat_front_addr":"172.21.15.104:6813/1258208297","state":["exists","up"]},{"osd":6,"uuid":"0e92a2ef-43cd-487c-9203-e219158aef8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6818","nonce":2782482301},{"type":"v1","addr":"172.21.15.19:6819","nonce":2782482301}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6820","nonce":2782482301},{"type":"v1","addr":"172.21.15.19:6821","nonce":2782482301}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6824","nonce":2782482301},{"type":"v1","addr":"172.21.15.19:6825","nonce":2782482301}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6822","nonce":2782482301},{"type":"v1","addr":"172.21.15.19:6823","nonce":2782482301}]},"public_addr":"172.21.15.19:6819/2782482301","cluster_addr":"172.21.15.19:6821/2782482301","heartbeat_back_addr":"172.21.15.19:6825/2782482301","heartbeat_front_addr":"172.21.15.19:6823/2782482301","state":["exists","up"]},{"osd":7,"uuid":"a16e39a1-3685-491d-b376-3e80a2084072","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":4238160988},{"type":"v1","addr":"172.21.15.33:6817","nonce":4238160988}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":4238160988},{"type":"v1","addr":"172.21.15.33:6819","nonce":4238160988}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":4238160988},{"type":"v1","addr":"172.21.15.33:6823","nonce":4238160988}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":4238160988},{"type":"v1","addr":"172.21.15.33:6821","nonce":4238160988}]},"public_addr":"172.21.15.33:6817/4238160988","cluster_addr":"172.21.15.33:6819/4238160988","heartbeat_back_addr":"172.21.15.33:6823/4238160988","heartbeat_front_addr":"172.21.15.33:6821/4238160988","state":["exists","up"]},{"osd":8,"uuid":"d5c3fa92-cd5d-4349-9c7d-d5666bacde16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":1014779813},{"type":"v1","addr":"172.21.15.104:6817","nonce":1014779813}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":1014779813},{"type":"v1","addr":"172.21.15.104:6819","nonce":1014779813}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":1014779813},{"type":"v1","addr":"172.21.15.104:6823","nonce":1014779813}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":1014779813},{"type":"v1","addr":"172.21.15.104:6821","nonce":1014779813}]},"public_addr":"172.21.15.104:6817/1014779813","cluster_addr":"172.21.15.104:6819/1014779813","heartbeat_back_addr":"172.21.15.104:6823/1014779813","heartbeat_front_addr":"172.21.15.104:6821/1014779813","state":["exists","up"]},{"osd":9,"uuid":"eed2e9b2-26d3-4c46-88ec-34b5596fe991","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6826","nonce":2641151009},{"type":"v1","addr":"172.21.15.19:6827","nonce":2641151009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6828","nonce":2641151009},{"type":"v1","addr":"172.21.15.19:6829","nonce":2641151009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6832","nonce":2641151009},{"type":"v1","addr":"172.21.15.19:6833","nonce":2641151009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6830","nonce":2641151009},{"type":"v1","addr":"172.21.15.19:6831","nonce":2641151009}]},"public_addr":"172.21.15.19:6827/2641151009","cluster_addr":"172.21.15.19:6829/2641151009","heartbeat_back_addr":"172.21.15.19:6833/2641151009","heartbeat_front_addr":"172.21.15.19:6831/2641151009","state":["exists","up"]},{"osd":10,"uuid":"f3c35ea5-fbe7-4dc4-93c5-b92d1a714db3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":1346058869},{"type":"v1","addr":"172.21.15.33:6825","nonce":1346058869}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":1346058869},{"type":"v1","addr":"172.21.15.33:6827","nonce":1346058869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":1346058869},{"type":"v1","addr":"172.21.15.33:6831","nonce":1346058869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":1346058869},{"type":"v1","addr":"172.21.15.33:6829","nonce":1346058869}]},"public_addr":"172.21.15.33:6825/1346058869","cluster_addr":"172.21.15.33:6827/1346058869","heartbeat_back_addr":"172.21.15.33:6831/1346058869","heartbeat_front_addr":"172.21.15.33:6829/1346058869","state":["exists","up"]},{"osd":11,"uuid":"c8b51571-f807-4b62-90b6-7529e09e59da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":3759498281},{"type":"v1","addr":"172.21.15.104:6825","nonce":3759498281}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":3759498281},{"type":"v1","addr":"172.21.15.104:6827","nonce":3759498281}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":3759498281},{"type":"v1","addr":"172.21.15.104:6831","nonce":3759498281}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":3759498281},{"type":"v1","addr":"172.21.15.104:6829","nonce":3759498281}]},"public_addr":"172.21.15.104:6825/3759498281","cluster_addr":"172.21.15.104:6827/3759498281","heartbeat_back_addr":"172.21.15.104:6831/3759498281","heartbeat_front_addr":"172.21.15.104:6829/3759498281","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:09:05.267795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:09:30.234068+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:09:53.870232+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:10:26.471296+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:10:53.180341+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:11:17.801361+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:11:51.364816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:12:18.306408+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:12:43.978744+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:13:18.213964+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:13:45.437544+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:14:10.148580+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.19:0/3860783274":"2024-03-19T15:07:14.150983+0000","172.21.15.19:0/2737223213":"2024-03-19T15:07:14.150983+0000","172.21.15.19:0/176699025":"2024-03-19T15:06:38.688405+0000","172.21.15.19:0/2316969005":"2024-03-19T15:07:14.150983+0000","172.21.15.19:6801/2776008962":"2024-03-19T15:07:14.150983+0000","172.21.15.19:0/2793980440":"2024-03-19T15:06:38.688405+0000","172.21.15.19:6800/3264337803":"2024-03-19T15:06:38.688405+0000","172.21.15.19:6801/3264337803":"2024-03-19T15:06:38.688405+0000","172.21.15.19:0/3166705295":"2024-03-19T15:06:38.688405+0000","172.21.15.19:6800/2776008962":"2024-03-19T15:07:14.150983+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T15:15:18.410 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.409+0000 7fd06be00640 1 -- 172.21.15.19:0/3620641539 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fd0500a2800 msgr2=0x7fd0500a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:18.411 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.409+0000 7fd06be00640 1 --2- 172.21.15.19:0/3620641539 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fd0500a2800 0x7fd0500a4cc0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fd0540095f0 tx=0x7fd054005f90 comp rx=0 tx=0).stop 2024-03-18T15:15:18.411 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.409+0000 7fd06be00640 1 -- 172.21.15.19:0/3620641539 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd06c103810 msgr2=0x7fd06c199af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:18.411 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.409+0000 7fd06be00640 1 --2- 172.21.15.19:0/3620641539 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd06c103810 0x7fd06c199af0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fd05c000c00 tx=0x7fd05c02c650 comp rx=0 tx=0).stop 2024-03-18T15:15:18.411 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.409+0000 7fd06be00640 1 -- 172.21.15.19:0/3620641539 shutdown_connections 2024-03-18T15:15:18.411 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.409+0000 7fd06be00640 1 --2- 172.21.15.19:0/3620641539 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd06c1053c0 0x7fd06c1a10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:18.411 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.409+0000 7fd06be00640 1 --2- 172.21.15.19:0/3620641539 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd06c104a00 0x7fd06c19a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:18.411 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.409+0000 7fd06be00640 1 --2- 172.21.15.19:0/3620641539 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fd0500a2800 0x7fd0500a4cc0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:18.411 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.409+0000 7fd06be00640 1 --2- 172.21.15.19:0/3620641539 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd06c103810 0x7fd06c199af0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:18.411 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.409+0000 7fd06be00640 1 -- 172.21.15.19:0/3620641539 >> 172.21.15.19:0/3620641539 conn(0x7fd06c0fed80 msgr2=0x7fd06c107d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:18.412 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.409+0000 7fd06be00640 1 -- 172.21.15.19:0/3620641539 shutdown_connections 2024-03-18T15:15:18.412 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.409+0000 7fd06be00640 1 -- 172.21.15.19:0/3620641539 wait complete. 2024-03-18T15:15:18.425 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:18 smithi019 ceph-mon[31522]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-18T15:15:18.425 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:18 smithi019 ceph-mon[31522]: Cluster is now healthy 2024-03-18T15:15:18.425 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:18 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/1471811895' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:15:18.425 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:18 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/3471948888' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:15:18.428 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 16, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T15:15:10.589737+0000', 'modified': '2024-03-18T15:15:15.582611+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24560}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24560': {'gid': 24560, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.33:6839/1903591535', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.33:6838', 'nonce': 1903591535}, {'type': 'v1', 'addr': '172.21.15.33:6839', 'nonce': 1903591535}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 24560, 'qdb_cluster': [24560]} 2024-03-18T15:15:18.428 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-18T15:15:18.428 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-18T15:15:18.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:18 smithi104 ceph-mon[36070]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-18T15:15:18.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:18 smithi104 ceph-mon[36070]: Cluster is now healthy 2024-03-18T15:15:18.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:18 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/1471811895' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:15:18.445 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:18 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/3471948888' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:15:18.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:18 smithi033 ceph-mon[36648]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-18T15:15:18.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:18 smithi033 ceph-mon[36648]: Cluster is now healthy 2024-03-18T15:15:18.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:18 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/1471811895' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:15:18.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:18 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/3471948888' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:15:18.641 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.640+0000 7fa8a7200640 1 -- 172.21.15.19:0/344261065 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa8a80fff40 msgr2=0x7fa8a8100360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:18.641 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.640+0000 7fa8a7200640 1 --2- 172.21.15.19:0/344261065 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa8a80fff40 0x7fa8a8100360 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fa898009900 tx=0x7fa89802bc40 comp rx=0 tx=0).stop 2024-03-18T15:15:18.642 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.640+0000 7fa8a7200640 1 -- 172.21.15.19:0/344261065 shutdown_connections 2024-03-18T15:15:18.642 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.640+0000 7fa8a7200640 1 --2- 172.21.15.19:0/344261065 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa8a8101240 0x7fa8a81016c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:18.642 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.640+0000 7fa8a7200640 1 --2- 172.21.15.19:0/344261065 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa8a81008a0 0x7fa8a8100d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:18.642 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.640+0000 7fa8a7200640 1 --2- 172.21.15.19:0/344261065 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa8a80fff40 0x7fa8a8100360 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:18.642 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.640+0000 7fa8a7200640 1 -- 172.21.15.19:0/344261065 >> 172.21.15.19:0/344261065 conn(0x7fa8a80fbb60 msgr2=0x7fa8a80fdf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:18.642 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.640+0000 7fa8a7200640 1 -- 172.21.15.19:0/344261065 shutdown_connections 2024-03-18T15:15:18.642 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.640+0000 7fa8a7200640 1 -- 172.21.15.19:0/344261065 wait complete. 2024-03-18T15:15:18.644 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.641+0000 7fa8a7200640 1 Processor -- start 2024-03-18T15:15:18.644 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.641+0000 7fa8a7200640 1 -- start start 2024-03-18T15:15:18.644 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.641+0000 7fa8a7200640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa8a80fff40 0x7fa8a8199ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:18.644 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.641+0000 7fa8a7200640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa8a81008a0 0x7fa8a819a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:18.644 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.641+0000 7fa8a7200640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa8a8101240 0x7fa8a81a1150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:18.645 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.642+0000 7fa8a7200640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fa8a810f580 con 0x7fa8a8101240 2024-03-18T15:15:18.645 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.642+0000 7fa8a7200640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa8a810f400 con 0x7fa8a81008a0 2024-03-18T15:15:18.645 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.642+0000 7fa8a7200640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fa8a810f700 con 0x7fa8a80fff40 2024-03-18T15:15:18.645 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.642+0000 7fa8a5400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa8a81008a0 0x7fa8a819a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:18.645 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.642+0000 7fa8a5400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa8a81008a0 0x7fa8a819a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:36850/0 (socket says 172.21.15.19:36850) 2024-03-18T15:15:18.645 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.642+0000 7fa8a5400640 1 -- 172.21.15.19:0/3065145584 learned_addr learned my addr 172.21.15.19:0/3065145584 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:15:18.645 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.642+0000 7fa8a6800640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa8a8101240 0x7fa8a81a1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:18.645 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.642+0000 7fa8a5e00640 1 --2- 172.21.15.19:0/3065145584 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa8a80fff40 0x7fa8a8199ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:18.645 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.642+0000 7fa8a5400640 1 -- 172.21.15.19:0/3065145584 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa8a80fff40 msgr2=0x7fa8a8199ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:18.645 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.642+0000 7fa8a5400640 1 --2- 172.21.15.19:0/3065145584 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa8a80fff40 0x7fa8a8199ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:18.645 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.642+0000 7fa8a5400640 1 -- 172.21.15.19:0/3065145584 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa8a8101240 msgr2=0x7fa8a81a1150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:18.645 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.642+0000 7fa8a5400640 1 --2- 172.21.15.19:0/3065145584 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa8a8101240 0x7fa8a81a1150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:18.645 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.642+0000 7fa8a5400640 1 -- 172.21.15.19:0/3065145584 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8a81a17c0 con 0x7fa8a81008a0 2024-03-18T15:15:18.646 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.642+0000 7fa8a5e00640 1 --2- 172.21.15.19:0/3065145584 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa8a80fff40 0x7fa8a8199ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T15:15:18.646 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.642+0000 7fa8a5400640 1 --2- 172.21.15.19:0/3065145584 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa8a81008a0 0x7fa8a819a0e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fa89400ea60 tx=0x7fa89400ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:18.646 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.643+0000 7fa89ea00640 1 -- 172.21.15.19:0/3065145584 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa89400cde0 con 0x7fa8a81008a0 2024-03-18T15:15:18.646 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.643+0000 7fa89ea00640 1 -- 172.21.15.19:0/3065145584 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa8940035f0 con 0x7fa8a81008a0 2024-03-18T15:15:18.646 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.643+0000 7fa89ea00640 1 -- 172.21.15.19:0/3065145584 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa894010670 con 0x7fa8a81008a0 2024-03-18T15:15:18.646 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.643+0000 7fa8a7200640 1 -- 172.21.15.19:0/3065145584 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8a81a1ab0 con 0x7fa8a81008a0 2024-03-18T15:15:18.646 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.643+0000 7fa8a7200640 1 -- 172.21.15.19:0/3065145584 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8a81a2050 con 0x7fa8a81008a0 2024-03-18T15:15:18.647 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.646+0000 7fa89ea00640 1 -- 172.21.15.19:0/3065145584 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fa8940037a0 con 0x7fa8a81008a0 2024-03-18T15:15:18.647 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.646+0000 7fa8af842740 1 Processor -- start 2024-03-18T15:15:18.655 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.646+0000 7fa8af842740 1 -- start start 2024-03-18T15:15:18.655 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.646+0000 7fa8af842740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cee91f9340 0x55cee91f9760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:18.655 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.646+0000 7fa8af842740 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55cee91f9d30 0x55cee91fe830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:18.655 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.646+0000 7fa8af842740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cee91feea0 0x55cee9201330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:18.655 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.647+0000 7fa8af842740 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x55cee915dc50 con 0x55cee91f9d30 2024-03-18T15:15:18.655 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.647+0000 7fa8af842740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55cee91069d0 con 0x55cee91f9340 2024-03-18T15:15:18.656 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.647+0000 7fa8af842740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55cee9159cf0 con 0x55cee91feea0 2024-03-18T15:15:18.656 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.647+0000 7fa8a6800640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cee91f9340 0x55cee91f9760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:18.656 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.647+0000 7fa8a6800640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cee91f9340 0x55cee91f9760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:36858/0 (socket says 172.21.15.19:36858) 2024-03-18T15:15:18.656 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.647+0000 7fa8a6800640 1 -- 172.21.15.19:0/3531914693 learned_addr learned my addr 172.21.15.19:0/3531914693 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:15:18.656 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.647+0000 7fa8a5e00640 1 --2- 172.21.15.19:0/3531914693 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55cee91f9d30 0x55cee91fe830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:18.656 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.647+0000 7fa8a5400640 1 --2- 172.21.15.19:0/3531914693 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cee91feea0 0x55cee9201330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:18.656 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.647+0000 7fa8a6800640 1 -- 172.21.15.19:0/3531914693 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cee91feea0 msgr2=0x55cee9201330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:15:18.656 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.647+0000 7fa8a6800640 1 --2- 172.21.15.19:0/3531914693 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cee91feea0 0x55cee9201330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:18.656 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.647+0000 7fa8a6800640 1 -- 172.21.15.19:0/3531914693 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55cee91f9d30 msgr2=0x55cee91fe830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:15:18.656 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.647+0000 7fa8a6800640 1 --2- 172.21.15.19:0/3531914693 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55cee91f9d30 0x55cee91fe830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:18.656 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.647+0000 7fa8a6800640 1 -- 172.21.15.19:0/3531914693 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cee9120f10 con 0x55cee91f9340 2024-03-18T15:15:18.656 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.647+0000 7fa89ea00640 1 --2- 172.21.15.19:0/3065145584 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fa88c0a2880 0x7fa88c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:18.656 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.647+0000 7fa8a6800640 1 --2- 172.21.15.19:0/3531914693 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cee91f9340 0x55cee91f9760 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fa8a0009fd0 tx=0x7fa8a00118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f55997b4c48b4720 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:18.657 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.648+0000 7fa89ea00640 1 -- 172.21.15.19:0/3065145584 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fa8940cefe0 con 0x7fa8a81008a0 2024-03-18T15:15:18.657 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.648+0000 7fa893000640 1 -- 172.21.15.19:0/3531914693 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8a0004d20 con 0x55cee91f9340 2024-03-18T15:15:18.657 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.648+0000 7fa893000640 1 -- 172.21.15.19:0/3531914693 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa8a0004050 con 0x55cee91f9340 2024-03-18T15:15:18.657 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.648+0000 7fa8a5e00640 1 --2- 172.21.15.19:0/3065145584 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fa88c0a2880 0x7fa88c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:18.657 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.648+0000 7fa893000640 1 -- 172.21.15.19:0/3531914693 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8a0005260 con 0x55cee91f9340 2024-03-18T15:15:18.657 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.648+0000 7fa8af842740 1 -- 172.21.15.19:0/3531914693 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cee91f9340 msgr2=0x55cee91f9760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:18.657 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.648+0000 7fa8af842740 1 --2- 172.21.15.19:0/3531914693 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cee91f9340 0x55cee91f9760 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fa8a0009fd0 tx=0x7fa8a00118d0 comp rx=0 tx=0).stop 2024-03-18T15:15:18.657 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.649+0000 7fa8a5e00640 1 --2- 172.21.15.19:0/3065145584 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fa88c0a2880 0x7fa88c0a4d40 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fa89802c250 tx=0x7fa89802cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:18.657 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.649+0000 7fa8af842740 1 -- 172.21.15.19:0/3531914693 shutdown_connections 2024-03-18T15:15:18.657 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.649+0000 7fa8af842740 1 --2- 172.21.15.19:0/3531914693 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cee91feea0 0x55cee9201330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:18.657 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.649+0000 7fa8af842740 1 --2- 172.21.15.19:0/3531914693 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55cee91f9d30 0x55cee91fe830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:18.657 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.649+0000 7fa8af842740 1 --2- 172.21.15.19:0/3531914693 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cee91f9340 0x55cee91f9760 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:18.657 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.649+0000 7fa8af842740 1 -- 172.21.15.19:0/3531914693 >> 172.21.15.19:0/3531914693 conn(0x55cee91f48b0 msgr2=0x55cee91f4cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:18.657 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.649+0000 7fa8af842740 1 -- 172.21.15.19:0/3531914693 shutdown_connections 2024-03-18T15:15:18.658 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.649+0000 7fa8af842740 1 -- 172.21.15.19:0/3531914693 wait complete. 2024-03-18T15:15:18.658 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.649+0000 7fa8af842740 1 -- start start 2024-03-18T15:15:18.658 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.649+0000 7fa8af842740 1 Processor -- start 2024-03-18T15:15:18.658 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.650+0000 7fa8af842740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cee91f9340 0x55cee92034c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:18.658 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.650+0000 7fa8af842740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cee91f9d30 0x55cee9203a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:18.658 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.650+0000 7fa8af842740 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55cee91feea0 0x55cee920aa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:18.658 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.650+0000 7fa8af842740 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x55cee9201e50 con 0x55cee91feea0 2024-03-18T15:15:18.658 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.650+0000 7fa8af842740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55cee9119e20 con 0x55cee91f9d30 2024-03-18T15:15:18.658 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.650+0000 7fa8af842740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55cee9201fd0 con 0x55cee91f9340 2024-03-18T15:15:18.658 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.650+0000 7fa8af842740 10 client.0 _ll_register_callbacks cb 0x55cee91197f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T15:15:18.658 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.650+0000 7fa8a6800640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55cee91feea0 0x55cee920aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:18.658 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.650+0000 7fa8a5400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cee91f9d30 0x55cee9203a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:18.659 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.650+0000 7fa8a6800640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55cee91feea0 0x55cee920aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:36590/0 (socket says 172.21.15.19:36590) 2024-03-18T15:15:18.659 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.650+0000 7fa8a6800640 1 -- 172.21.15.19:0/2946031719 learned_addr learned my addr 172.21.15.19:0/2946031719 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:15:18.659 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.650+0000 7fa8a5e00640 1 --2- 172.21.15.19:0/2946031719 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cee91f9340 0x55cee92034c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:18.659 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.650+0000 7fa8a5400640 1 -- 172.21.15.19:0/2946031719 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cee91f9340 msgr2=0x55cee92034c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:15:18.659 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.650+0000 7fa8a5400640 1 --2- 172.21.15.19:0/2946031719 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cee91f9340 0x55cee92034c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:18.659 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.650+0000 7fa8a5400640 1 -- 172.21.15.19:0/2946031719 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55cee91feea0 msgr2=0x55cee920aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:15:18.659 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.650+0000 7fa8a5400640 1 --2- 172.21.15.19:0/2946031719 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55cee91feea0 0x55cee920aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:18.659 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.650+0000 7fa8a5400640 1 -- 172.21.15.19:0/2946031719 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cee8e722d0 con 0x55cee91f9d30 2024-03-18T15:15:18.659 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.650+0000 7fa8a5e00640 1 --2- 172.21.15.19:0/2946031719 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cee91f9340 0x55cee92034c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T15:15:18.659 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.651+0000 7fa8a5400640 1 --2- 172.21.15.19:0/2946031719 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cee91f9d30 0x55cee9203a00 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fa89401f000 tx=0x7fa894097a30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=91d5a71e7c494b73 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:18.659 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.651+0000 7fa887400640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-18T15:15:18.659 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.651+0000 7fa887400640 1 -- 172.21.15.19:0/2946031719 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa89409b080 con 0x55cee91f9d30 2024-03-18T15:15:18.659 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.651+0000 7fa887400640 1 -- 172.21.15.19:0/2946031719 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fa894003b30 con 0x55cee91f9d30 2024-03-18T15:15:18.660 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.651+0000 7fa8af842740 1 -- 172.21.15.19:0/2946031719 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa8a0004050 con 0x55cee91f9d30 2024-03-18T15:15:18.660 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.651+0000 7fa887400640 1 -- 172.21.15.19:0/2946031719 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) ==== 24+0+0 (secure 0 0 0) 0x7fa89409be20 con 0x55cee91f9d30 2024-03-18T15:15:18.660 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.651+0000 7fa8af842740 10 client.24668 fetch_fsmap learned FSMap version 16 2024-03-18T15:15:18.660 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.651+0000 7fa8af842740 1 -- 172.21.15.19:0/2946031719 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=4+}) -- 0x55cee8f3c6b0 con 0x55cee91f9d30 2024-03-18T15:15:18.660 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.651+0000 7fa887400640 1 -- 172.21.15.19:0/2946031719 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa8a0004050 con 0x55cee91f9d30 2024-03-18T15:15:18.660 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.652+0000 7fa887400640 1 -- 172.21.15.19:0/2946031719 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 16) ==== 8342+0+0 (secure 0 0 0) 0x7fa8940d10f0 con 0x55cee91f9d30 2024-03-18T15:15:18.660 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.652+0000 7fa8af842740 10 client.24668 fetch_fsmap finished waiting for FSMap version 16 2024-03-18T15:15:18.660 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.652+0000 7fa8af842740 10 client.24668 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-18T15:15:18.660 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.652+0000 7fa8af842740 20 client.24668 populate_metadata read hostname 'smithi019' 2024-03-18T15:15:18.660 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.652+0000 7fa8af842740 1 --2- 172.21.15.19:0/2946031719 >> [v2:172.21.15.33:6838/1903591535,v1:172.21.15.33:6839/1903591535] conn(0x55cee9210190 0x55cee9211600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:18.660 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.652+0000 7fa8af842740 4 client.24668 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T15:15:18.660 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.652+0000 7fa8af842740 1 -- 172.21.15.19:0/2946031719 --> [v2:172.21.15.33:6838/1903591535,v1:172.21.15.33:6839/1903591535] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cee91bb2b0 con 0x55cee9210190 2024-03-18T15:15:18.660 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.652+0000 7fa8a5e00640 1 --2- 172.21.15.19:0/2946031719 >> [v2:172.21.15.33:6838/1903591535,v1:172.21.15.33:6839/1903591535] conn(0x55cee9210190 0x55cee9211600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:18.660 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.652+0000 7fa8a5e00640 1 --2- 172.21.15.19:0/2946031719 >> [v2:172.21.15.33:6838/1903591535,v1:172.21.15.33:6839/1903591535] conn(0x55cee9210190 0x55cee9211600 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d36aea8f52f2a555 server_cookie=a09d780701f5898 in_seq=0 out_seq=0 2024-03-18T15:15:18.661 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.652+0000 7fa887400640 10 client.24668 ms_handle_connect on v2:172.21.15.33:6838/1903591535 2024-03-18T15:15:18.661 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.653+0000 7fa887400640 1 -- 172.21.15.19:0/2946031719 <== mds.0 v2:172.21.15.33:6838/1903591535 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55cee91bb2b0 con 0x55cee9210190 2024-03-18T15:15:18.661 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.653+0000 7fa887400640 10 client.24668 handle_command_reply: tid=0 2024-03-18T15:15:18.661 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.653+0000 7fa8af842740 1 client.24668 shutdown 2024-03-18T15:15:18.661 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.653+0000 7fa8af842740 1 -- 172.21.15.19:0/2946031719 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cee91f9d30 msgr2=0x55cee9203a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:18.661 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.653+0000 7fa8af842740 1 --2- 172.21.15.19:0/2946031719 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cee91f9d30 0x55cee9203a00 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fa89401f000 tx=0x7fa894097a30 comp rx=0 tx=0).stop 2024-03-18T15:15:18.661 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.654+0000 7fa8af842740 1 -- 172.21.15.19:0/2946031719 shutdown_connections 2024-03-18T15:15:18.661 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.654+0000 7fa8af842740 1 --2- 172.21.15.19:0/2946031719 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55cee91feea0 0x55cee920aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:18.661 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.654+0000 7fa8af842740 1 --2- 172.21.15.19:0/2946031719 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cee91f9d30 0x55cee9203a00 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:18.661 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.654+0000 7fa8af842740 1 --2- 172.21.15.19:0/2946031719 >> [v2:172.21.15.33:6838/1903591535,v1:172.21.15.33:6839/1903591535] conn(0x55cee9210190 0x55cee9211600 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:18.661 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.654+0000 7fa8af842740 1 --2- 172.21.15.19:0/2946031719 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cee91f9340 0x55cee92034c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:18.661 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.654+0000 7fa887400640 0 client.24668 ms_handle_reset on v2:172.21.15.33:6838/1903591535 2024-03-18T15:15:18.661 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.654+0000 7fa8af842740 1 -- 172.21.15.19:0/2946031719 >> 172.21.15.19:0/2946031719 conn(0x55cee91f48b0 msgr2=0x55cee91f3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:18.661 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.654+0000 7fa8af842740 1 -- 172.21.15.19:0/2946031719 shutdown_connections 2024-03-18T15:15:18.662 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.654+0000 7fa8af842740 1 -- 172.21.15.19:0/2946031719 wait complete. 2024-03-18T15:15:18.662 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.654+0000 7fa8af842740 20 client.24668 trim_cache size 0 max 16384 2024-03-18T15:15:18.679 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.677+0000 7fa8af842740 1 Processor -- start 2024-03-18T15:15:18.679 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.677+0000 7fa8af842740 1 -- start start 2024-03-18T15:15:18.679 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.678+0000 7fa8af842740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cee91f9340 0x55cee91f38f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:18.679 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.678+0000 7fa8af842740 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55cee91f9d30 0x55cee920aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:18.679 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.678+0000 7fa8af842740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cee91feea0 0x55cee91fa780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:18.679 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.678+0000 7fa8af842740 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x55cee920f940 con 0x55cee91f9d30 2024-03-18T15:15:18.680 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.678+0000 7fa8af842740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55cee9210010 con 0x55cee91feea0 2024-03-18T15:15:18.680 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.678+0000 7fa8af842740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55cee92106e0 con 0x55cee91f9340 2024-03-18T15:15:18.680 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.678+0000 7fa8a5e00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cee91f9340 0x55cee91f38f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:18.680 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.678+0000 7fa8a6800640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cee91feea0 0x55cee91fa780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:18.680 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.678+0000 7fa8a5400640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55cee91f9d30 0x55cee920aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:18.680 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.678+0000 7fa8a5e00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cee91f9340 0x55cee91f38f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.19:55622/0 (socket says 172.21.15.19:55622) 2024-03-18T15:15:18.680 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.678+0000 7fa8a5e00640 1 -- 172.21.15.19:0/1975951360 learned_addr learned my addr 172.21.15.19:0/1975951360 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:15:18.681 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.679+0000 7fa8a5e00640 1 -- 172.21.15.19:0/1975951360 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cee91feea0 msgr2=0x55cee91fa780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:15:18.681 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.679+0000 7fa8a5e00640 1 --2- 172.21.15.19:0/1975951360 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cee91feea0 0x55cee91fa780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:18.681 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.679+0000 7fa8a5e00640 1 -- 172.21.15.19:0/1975951360 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55cee91f9d30 msgr2=0x55cee920aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:15:18.681 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.679+0000 7fa8a5e00640 1 --2- 172.21.15.19:0/1975951360 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55cee91f9d30 0x55cee920aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:18.681 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.679+0000 7fa8a5e00640 1 -- 172.21.15.19:0/1975951360 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cee91feb50 con 0x55cee91f9340 2024-03-18T15:15:18.681 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.679+0000 7fa8a5400640 1 --2- 172.21.15.19:0/1975951360 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55cee91f9d30 0x55cee920aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T15:15:18.681 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.679+0000 7fa8a5e00640 1 --2- 172.21.15.19:0/1975951360 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cee91f9340 0x55cee91f38f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa898035000 tx=0x7fa898040070 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fb09b3be87ce517 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:18.681 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.679+0000 7fa892600640 1 -- 172.21.15.19:0/1975951360 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa898054a30 con 0x55cee91f9340 2024-03-18T15:15:18.681 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.679+0000 7fa892600640 1 -- 172.21.15.19:0/1975951360 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa898054800 con 0x55cee91f9340 2024-03-18T15:15:18.681 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.679+0000 7fa892600640 1 -- 172.21.15.19:0/1975951360 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa898005550 con 0x55cee91f9340 2024-03-18T15:15:18.683 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.680+0000 7fa8af842740 1 -- 172.21.15.19:0/1975951360 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cee91f9340 msgr2=0x55cee91f38f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:18.683 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.680+0000 7fa8af842740 1 --2- 172.21.15.19:0/1975951360 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cee91f9340 0x55cee91f38f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa898035000 tx=0x7fa898040070 comp rx=0 tx=0).stop 2024-03-18T15:15:18.683 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.680+0000 7fa8af842740 1 -- 172.21.15.19:0/1975951360 shutdown_connections 2024-03-18T15:15:18.683 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.680+0000 7fa8af842740 1 --2- 172.21.15.19:0/1975951360 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cee91feea0 0x55cee91fa780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:18.683 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.680+0000 7fa8af842740 1 --2- 172.21.15.19:0/1975951360 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55cee91f9d30 0x55cee920aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:18.683 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.680+0000 7fa8af842740 1 --2- 172.21.15.19:0/1975951360 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cee91f9340 0x55cee91f38f0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:18.683 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.680+0000 7fa8af842740 1 -- 172.21.15.19:0/1975951360 >> 172.21.15.19:0/1975951360 conn(0x55cee91f48b0 msgr2=0x55cee91f31e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:18.683 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.680+0000 7fa8af842740 1 -- 172.21.15.19:0/1975951360 shutdown_connections 2024-03-18T15:15:18.683 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.680+0000 7fa8af842740 1 -- 172.21.15.19:0/1975951360 wait complete. 2024-03-18T15:15:18.683 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.681+0000 7fa8af842740 1 -- start start 2024-03-18T15:15:18.684 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.681+0000 7fa8af842740 1 Processor -- start 2024-03-18T15:15:18.684 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.682+0000 7fa8af842740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cee91f9340 0x55cee9206ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:18.684 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.682+0000 7fa8af842740 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55cee91f9d30 0x55cee9206ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:18.684 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.682+0000 7fa8af842740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cee91feea0 0x55cee920a010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:18.684 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.682+0000 7fa8af842740 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x55cee9211480 con 0x55cee91f9d30 2024-03-18T15:15:18.684 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.682+0000 7fa8af842740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55cee9210db0 con 0x55cee91feea0 2024-03-18T15:15:18.684 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.682+0000 7fa8af842740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55cee9211b50 con 0x55cee91f9340 2024-03-18T15:15:18.684 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.682+0000 7fa8af842740 10 client.0 _ll_register_callbacks cb 0x7fa894097310 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T15:15:18.684 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.682+0000 7fa8a5400640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55cee91f9d30 0x55cee9206ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:18.684 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.682+0000 7fa8a6800640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cee91feea0 0x55cee920a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:18.684 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.682+0000 7fa8a5400640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55cee91f9d30 0x55cee9206ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:36614/0 (socket says 172.21.15.19:36614) 2024-03-18T15:15:18.684 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.682+0000 7fa8a5400640 1 -- 172.21.15.19:0/3608705092 learned_addr learned my addr 172.21.15.19:0/3608705092 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:15:18.684 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.682+0000 7fa8a5e00640 1 --2- 172.21.15.19:0/3608705092 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cee91f9340 0x55cee9206ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:18.687 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.682+0000 7fa8a5e00640 1 -- 172.21.15.19:0/3608705092 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cee91feea0 msgr2=0x55cee920a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:15:18.687 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.682+0000 7fa8a5e00640 1 --2- 172.21.15.19:0/3608705092 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cee91feea0 0x55cee920a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:18.687 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.682+0000 7fa8a5e00640 1 -- 172.21.15.19:0/3608705092 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55cee91f9d30 msgr2=0x55cee9206ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:15:18.687 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.682+0000 7fa8a5e00640 1 --2- 172.21.15.19:0/3608705092 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55cee91f9d30 0x55cee9206ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:18.688 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.682+0000 7fa8a5e00640 1 -- 172.21.15.19:0/3608705092 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cee920a600 con 0x55cee91f9340 2024-03-18T15:15:18.688 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.683+0000 7fa8a5e00640 1 --2- 172.21.15.19:0/3608705092 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cee91f9340 0x55cee9206ab0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa89802c7c0 tx=0x7fa8980056b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2b054550a1feea32 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:18.688 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.683+0000 7fa887400640 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-03-18T15:15:18.688 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.683+0000 7fa887400640 1 -- 172.21.15.19:0/3608705092 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8980543a0 con 0x55cee91f9340 2024-03-18T15:15:18.688 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.683+0000 7fa887400640 1 -- 172.21.15.19:0/3608705092 <== mon.1 v2:172.21.15.104:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fa898059040 con 0x55cee91f9340 2024-03-18T15:15:18.688 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.683+0000 7fa8af842740 1 -- 172.21.15.19:0/3608705092 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55cee916b190 con 0x55cee91f9340 2024-03-18T15:15:18.688 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.683+0000 7fa887400640 1 -- 172.21.15.19:0/3608705092 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) ==== 24+0+0 (secure 0 0 0) 0x7fa898054540 con 0x55cee91f9340 2024-03-18T15:15:18.688 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.683+0000 7fa8af842740 10 client.24688 fetch_fsmap learned FSMap version 16 2024-03-18T15:15:18.688 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.683+0000 7fa8af842740 1 -- 172.21.15.19:0/3608705092 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=4+}) -- 0x55cee9276800 con 0x55cee91f9340 2024-03-18T15:15:18.688 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.683+0000 7fa887400640 1 -- 172.21.15.19:0/3608705092 <== mon.1 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55cee916b190 con 0x55cee91f9340 2024-03-18T15:15:18.688 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.684+0000 7fa887400640 1 -- 172.21.15.19:0/3608705092 <== mon.1 v2:172.21.15.104:3300/0 5 ==== fsmap(e 16) ==== 8342+0+0 (secure 0 0 0) 0x7fa898003d70 con 0x55cee91f9340 2024-03-18T15:15:18.688 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.684+0000 7fa8af842740 10 client.24688 fetch_fsmap finished waiting for FSMap version 16 2024-03-18T15:15:18.688 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.684+0000 7fa8af842740 10 client.24688 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-18T15:15:18.689 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.684+0000 7fa8af842740 20 client.24688 populate_metadata read hostname 'smithi019' 2024-03-18T15:15:18.689 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.684+0000 7fa8af842740 1 --2- 172.21.15.19:0/3608705092 >> [v2:172.21.15.33:6838/1903591535,v1:172.21.15.33:6839/1903591535] conn(0x55cee927ae50 0x55cee9277960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:18.689 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.684+0000 7fa8af842740 4 client.24688 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-18T15:15:18.689 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.684+0000 7fa8af842740 1 -- 172.21.15.19:0/3608705092 --> [v2:172.21.15.33:6838/1903591535,v1:172.21.15.33:6839/1903591535] -- command(tid 0: {"prefix": "status"}) -- 0x55cee91654e0 con 0x55cee927ae50 2024-03-18T15:15:18.689 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.684+0000 7fa8a5400640 1 --2- 172.21.15.19:0/3608705092 >> [v2:172.21.15.33:6838/1903591535,v1:172.21.15.33:6839/1903591535] conn(0x55cee927ae50 0x55cee9277960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:18.689 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.684+0000 7fa8a5400640 1 --2- 172.21.15.19:0/3608705092 >> [v2:172.21.15.33:6838/1903591535,v1:172.21.15.33:6839/1903591535] conn(0x55cee927ae50 0x55cee9277960 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a1f81262432029a8 server_cookie=93ef2195a1519900 in_seq=0 out_seq=0 2024-03-18T15:15:18.689 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.684+0000 7fa887400640 10 client.24688 ms_handle_connect on v2:172.21.15.33:6838/1903591535 2024-03-18T15:15:18.689 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.685+0000 7fa887400640 1 -- 172.21.15.19:0/3608705092 <== mds.0 v2:172.21.15.33:6838/1903591535 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55cee91654e0 con 0x55cee927ae50 2024-03-18T15:15:18.689 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.685+0000 7fa887400640 10 client.24688 handle_command_reply: tid=0 2024-03-18T15:15:18.689 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.685+0000 7fa8af842740 1 client.24688 shutdown 2024-03-18T15:15:18.689 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.685+0000 7fa8af842740 1 -- 172.21.15.19:0/3608705092 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cee91f9340 msgr2=0x55cee9206ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:18.689 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.685+0000 7fa8af842740 1 --2- 172.21.15.19:0/3608705092 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cee91f9340 0x55cee9206ab0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa89802c7c0 tx=0x7fa8980056b0 comp rx=0 tx=0).stop 2024-03-18T15:15:18.689 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.685+0000 7fa8af842740 1 -- 172.21.15.19:0/3608705092 shutdown_connections 2024-03-18T15:15:18.689 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.685+0000 7fa8af842740 1 --2- 172.21.15.19:0/3608705092 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cee91feea0 0x55cee920a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:18.690 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.685+0000 7fa8af842740 1 --2- 172.21.15.19:0/3608705092 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55cee91f9d30 0x55cee9206ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:18.690 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.685+0000 7fa8af842740 1 --2- 172.21.15.19:0/3608705092 >> [v2:172.21.15.33:6838/1903591535,v1:172.21.15.33:6839/1903591535] conn(0x55cee927ae50 0x55cee9277960 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:18.690 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.685+0000 7fa8af842740 1 --2- 172.21.15.19:0/3608705092 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55cee91f9340 0x55cee9206ab0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:18.690 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.685+0000 7fa8af842740 1 -- 172.21.15.19:0/3608705092 >> 172.21.15.19:0/3608705092 conn(0x55cee91f48b0 msgr2=0x55cee91f3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:18.690 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.685+0000 7fa887400640 0 client.24688 ms_handle_reset on v2:172.21.15.33:6838/1903591535 2024-03-18T15:15:18.690 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.685+0000 7fa8af842740 1 -- 172.21.15.19:0/3608705092 shutdown_connections 2024-03-18T15:15:18.690 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.685+0000 7fa8af842740 1 -- 172.21.15.19:0/3608705092 wait complete. 2024-03-18T15:15:18.690 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.685+0000 7fa8af842740 20 client.24688 trim_cache size 0 max 16384 2024-03-18T15:15:18.690 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-03-18T15:15:18.691 INFO:teuthology.orchestra.run.smithi019.stdout: "cluster_fsid": "d4dc2ba6-e538-11ee-95c9-87774f69a715", 2024-03-18T15:15:18.691 INFO:teuthology.orchestra.run.smithi019.stdout: "whoami": 0, 2024-03-18T15:15:18.691 INFO:teuthology.orchestra.run.smithi019.stdout: "id": 24560, 2024-03-18T15:15:18.691 INFO:teuthology.orchestra.run.smithi019.stdout: "want_state": "up:active", 2024-03-18T15:15:18.691 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "up:active", 2024-03-18T15:15:18.691 INFO:teuthology.orchestra.run.smithi019.stdout: "fs_name": "cephfs", 2024-03-18T15:15:18.691 INFO:teuthology.orchestra.run.smithi019.stdout: "rank_uptime": 7.5689367570000003, 2024-03-18T15:15:18.691 INFO:teuthology.orchestra.run.smithi019.stdout: "mdsmap_epoch": 16, 2024-03-18T15:15:18.691 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap_epoch": 89, 2024-03-18T15:15:18.691 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap_epoch_barrier": 88, 2024-03-18T15:15:18.691 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 14.395677147000001 2024-03-18T15:15:18.691 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-03-18T15:15:18.692 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.686+0000 7fa8a7200640 1 -- 172.21.15.19:0/3065145584 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fa88c0a2880 msgr2=0x7fa88c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:18.692 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.686+0000 7fa8a7200640 1 --2- 172.21.15.19:0/3065145584 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fa88c0a2880 0x7fa88c0a4d40 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fa89802c250 tx=0x7fa89802cf90 comp rx=0 tx=0).stop 2024-03-18T15:15:18.692 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.686+0000 7fa8a7200640 1 -- 172.21.15.19:0/3065145584 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa8a81008a0 msgr2=0x7fa8a819a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:18.692 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.686+0000 7fa8a7200640 1 --2- 172.21.15.19:0/3065145584 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa8a81008a0 0x7fa8a819a0e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fa89400ea60 tx=0x7fa89400ef20 comp rx=0 tx=0).stop 2024-03-18T15:15:18.692 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.686+0000 7fa8a7200640 1 -- 172.21.15.19:0/3065145584 shutdown_connections 2024-03-18T15:15:18.692 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.686+0000 7fa8a7200640 1 --2- 172.21.15.19:0/3065145584 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa8a8101240 0x7fa8a81a1150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:18.692 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.686+0000 7fa8a7200640 1 --2- 172.21.15.19:0/3065145584 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fa88c0a2880 0x7fa88c0a4d40 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:18.693 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.686+0000 7fa8a7200640 1 --2- 172.21.15.19:0/3065145584 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa8a81008a0 0x7fa8a819a0e0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:18.693 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.686+0000 7fa8a7200640 1 --2- 172.21.15.19:0/3065145584 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa8a80fff40 0x7fa8a8199ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:18.693 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.686+0000 7fa8a7200640 1 -- 172.21.15.19:0/3065145584 >> 172.21.15.19:0/3065145584 conn(0x7fa8a80fbb60 msgr2=0x7fa8a80fd220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:18.693 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.686+0000 7fa8a7200640 1 -- 172.21.15.19:0/3065145584 shutdown_connections 2024-03-18T15:15:18.693 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:18.686+0000 7fa8a7200640 1 -- 172.21.15.19:0/3065145584 wait complete. 2024-03-18T15:15:18.704 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-18T15:15:18.704 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-03-18T15:15:18.704 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 -- ceph fs ls 2024-03-18T15:15:19.158 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:15:19.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:19 smithi104 ceph-mon[36070]: pgmap v321: 129 pgs: 122 active+clean, 3 creating+peering, 4 unknown; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 783 B/s wr, 6 op/s 2024-03-18T15:15:19.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:19 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/3620641539' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:15:19.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:19 smithi019 ceph-mon[31522]: pgmap v321: 129 pgs: 122 active+clean, 3 creating+peering, 4 unknown; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 783 B/s wr, 6 op/s 2024-03-18T15:15:19.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:19 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/3620641539' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:15:19.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:19 smithi033 ceph-mon[36648]: pgmap v321: 129 pgs: 122 active+clean, 3 creating+peering, 4 unknown; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 783 B/s wr, 6 op/s 2024-03-18T15:15:19.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:19 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/3620641539' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:15:21.430 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:21 smithi019 ceph-mon[31522]: pgmap v322: 129 pgs: 129 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 329 B/s wr, 4 op/s 2024-03-18T15:15:21.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:21 smithi104 ceph-mon[36070]: pgmap v322: 129 pgs: 129 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 329 B/s wr, 4 op/s 2024-03-18T15:15:21.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:21 smithi033 ceph-mon[36648]: pgmap v322: 129 pgs: 129 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 329 B/s wr, 4 op/s 2024-03-18T15:15:21.808 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:21.807+0000 7fd687000700 1 -- 172.21.15.19:0/1107354265 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd6801044e0 msgr2=0x7fd680104960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:21.808 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:21.807+0000 7fd687000700 1 --2- 172.21.15.19:0/1107354265 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd6801044e0 0x7fd680104960 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fd66c009920 tx=0x7fd66c009c30 comp rx=0 tx=0).stop 2024-03-18T15:15:21.809 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:21.807+0000 7fd687000700 1 -- 172.21.15.19:0/1107354265 shutdown_connections 2024-03-18T15:15:21.809 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:21.807+0000 7fd687000700 1 --2- 172.21.15.19:0/1107354265 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd680104ea0 0x7fd680111360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:21.809 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:21.807+0000 7fd687000700 1 --2- 172.21.15.19:0/1107354265 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd68010a590 0x7fd68010a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:21.809 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:21.807+0000 7fd687000700 1 --2- 172.21.15.19:0/1107354265 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd6801044e0 0x7fd680104960 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:21.809 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:21.807+0000 7fd687000700 1 -- 172.21.15.19:0/1107354265 >> 172.21.15.19:0/1107354265 conn(0x7fd6800fffb0 msgr2=0x7fd6801023d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:21.809 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:21.807+0000 7fd687000700 1 -- 172.21.15.19:0/1107354265 shutdown_connections 2024-03-18T15:15:21.809 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:21.807+0000 7fd687000700 1 -- 172.21.15.19:0/1107354265 wait complete. 2024-03-18T15:15:21.810 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:21.808+0000 7fd687000700 1 Processor -- start 2024-03-18T15:15:21.811 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:21.808+0000 7fd687000700 1 -- start start 2024-03-18T15:15:21.811 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:21.809+0000 7fd687000700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd6801044e0 0x7fd6801a0880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:21.811 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:21.809+0000 7fd687000700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd680104ea0 0x7fd6801a0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:21.811 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:21.809+0000 7fd687000700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd68010a590 0x7fd68019aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:21.811 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:21.809+0000 7fd687000700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fd6801147e0 con 0x7fd68010a590 2024-03-18T15:15:21.811 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:21.809+0000 7fd687000700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd680114660 con 0x7fd6801044e0 2024-03-18T15:15:21.811 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:21.809+0000 7fd687000700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fd680114960 con 0x7fd680104ea0 2024-03-18T15:15:21.811 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:21.809+0000 7fd67fe00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd6801044e0 0x7fd6801a0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:21.811 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:21.809+0000 7fd67fe00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd6801044e0 0x7fd6801a0880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:36906/0 (socket says 172.21.15.19:36906) 2024-03-18T15:15:21.811 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:21.809+0000 7fd67fe00700 1 -- 172.21.15.19:0/3459370027 learned_addr learned my addr 172.21.15.19:0/3459370027 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:15:21.811 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:21.809+0000 7fd67f400700 1 --2- 172.21.15.19:0/3459370027 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd680104ea0 0x7fd6801a0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:21.811 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:21.809+0000 7fd685000700 1 --2- 172.21.15.19:0/3459370027 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd68010a590 0x7fd68019aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:21.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:21.809+0000 7fd67fe00700 1 -- 172.21.15.19:0/3459370027 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd680104ea0 msgr2=0x7fd6801a0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:21.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:21.809+0000 7fd67fe00700 1 --2- 172.21.15.19:0/3459370027 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd680104ea0 0x7fd6801a0dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:21.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:21.809+0000 7fd67fe00700 1 -- 172.21.15.19:0/3459370027 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd68010a590 msgr2=0x7fd68019aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:21.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:21.809+0000 7fd67fe00700 1 --2- 172.21.15.19:0/3459370027 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd68010a590 0x7fd68019aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:21.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:21.809+0000 7fd67fe00700 1 -- 172.21.15.19:0/3459370027 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd68019b1a0 con 0x7fd6801044e0 2024-03-18T15:15:21.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:21.810+0000 7fd67fe00700 1 --2- 172.21.15.19:0/3459370027 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd6801044e0 0x7fd6801a0880 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fd67000e9a0 tx=0x7fd67000ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:21.813 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:21.810+0000 7fd67cc00700 1 -- 172.21.15.19:0/3459370027 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd67000cbd0 con 0x7fd6801044e0 2024-03-18T15:15:21.813 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:21.810+0000 7fd67cc00700 1 -- 172.21.15.19:0/3459370027 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd67000cd70 con 0x7fd6801044e0 2024-03-18T15:15:21.813 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:21.810+0000 7fd67cc00700 1 -- 172.21.15.19:0/3459370027 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd670018750 con 0x7fd6801044e0 2024-03-18T15:15:21.813 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:21.810+0000 7fd687000700 1 -- 172.21.15.19:0/3459370027 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd68019b490 con 0x7fd6801044e0 2024-03-18T15:15:21.813 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:21.810+0000 7fd687000700 1 -- 172.21.15.19:0/3459370027 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6801a7750 con 0x7fd6801044e0 2024-03-18T15:15:21.813 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:21.811+0000 7fd687000700 1 -- 172.21.15.19:0/3459370027 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd680068f90 con 0x7fd6801044e0 2024-03-18T15:15:21.814 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:21.813+0000 7fd67cc00700 1 -- 172.21.15.19:0/3459370027 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fd670010430 con 0x7fd6801044e0 2024-03-18T15:15:21.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:21.814+0000 7fd67cc00700 1 --2- 172.21.15.19:0/3459370027 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fd6640a2910 0x7fd6640a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:21.816 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:21.814+0000 7fd67cc00700 1 -- 172.21.15.19:0/3459370027 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fd6700cef40 con 0x7fd6801044e0 2024-03-18T15:15:21.816 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:21.815+0000 7fd67f400700 1 --2- 172.21.15.19:0/3459370027 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fd6640a2910 0x7fd6640a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:21.816 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:21.815+0000 7fd67f400700 1 --2- 172.21.15.19:0/3459370027 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fd6640a2910 0x7fd6640a4dd0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fd66c00c010 tx=0x7fd66c019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:21.819 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:21.818+0000 7fd67cc00700 1 -- 172.21.15.19:0/3459370027 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd67009bd40 con 0x7fd6801044e0 2024-03-18T15:15:22.106 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:22.105+0000 7fd687000700 1 -- 172.21.15.19:0/3459370027 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fd68002db70 con 0x7fd6801044e0 2024-03-18T15:15:22.107 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:22.106+0000 7fd67cc00700 1 -- 172.21.15.19:0/3459370027 <== mon.2 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v16) ==== 53+0+88 (secure 0 0 0) 0x7fd670097030 con 0x7fd6801044e0 2024-03-18T15:15:22.107 INFO:teuthology.orchestra.run.smithi019.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-18T15:15:22.110 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:22.109+0000 7fd687000700 1 -- 172.21.15.19:0/3459370027 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fd6640a2910 msgr2=0x7fd6640a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:22.110 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:22.109+0000 7fd687000700 1 --2- 172.21.15.19:0/3459370027 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fd6640a2910 0x7fd6640a4dd0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fd66c00c010 tx=0x7fd66c019040 comp rx=0 tx=0).stop 2024-03-18T15:15:22.111 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:22.109+0000 7fd687000700 1 -- 172.21.15.19:0/3459370027 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd6801044e0 msgr2=0x7fd6801a0880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:22.111 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:22.109+0000 7fd687000700 1 --2- 172.21.15.19:0/3459370027 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd6801044e0 0x7fd6801a0880 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fd67000e9a0 tx=0x7fd67000ecb0 comp rx=0 tx=0).stop 2024-03-18T15:15:22.111 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:22.109+0000 7fd687000700 1 -- 172.21.15.19:0/3459370027 shutdown_connections 2024-03-18T15:15:22.111 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:22.109+0000 7fd687000700 1 --2- 172.21.15.19:0/3459370027 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fd6640a2910 0x7fd6640a4dd0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:22.111 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:22.109+0000 7fd687000700 1 --2- 172.21.15.19:0/3459370027 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd68010a590 0x7fd68019aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:22.111 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:22.109+0000 7fd687000700 1 --2- 172.21.15.19:0/3459370027 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd6801044e0 0x7fd6801a0880 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:22.111 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:22.109+0000 7fd687000700 1 --2- 172.21.15.19:0/3459370027 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd680104ea0 0x7fd6801a0dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:22.111 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:22.109+0000 7fd687000700 1 -- 172.21.15.19:0/3459370027 >> 172.21.15.19:0/3459370027 conn(0x7fd6800fffb0 msgr2=0x7fd680101780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:22.111 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:22.109+0000 7fd687000700 1 -- 172.21.15.19:0/3459370027 shutdown_connections 2024-03-18T15:15:22.111 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:22.109+0000 7fd687000700 1 -- 172.21.15.19:0/3459370027 wait complete. 2024-03-18T15:15:22.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:22 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/3459370027' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-18T15:15:22.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:22 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/3459370027' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-18T15:15:22.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:22 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/3459370027' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-18T15:15:23.013 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-18T15:15:23.013 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-18T15:15:23.014 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi019.front.sepia.ceph.com 2024-03-18T15:15:23.014 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-03-18T15:15:23.014 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-03-18T15:15:23.014 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-18T15:15:23.014 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-18T15:15:23.014 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-18T15:15:23.014 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-03-18T15:15:23.014 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T15:15:23.014 DEBUG:teuthology.orchestra.run.smithi019:> ip addr 2024-03-18T15:15:23.077 INFO:teuthology.orchestra.run.smithi019.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-18T15:15:23.077 INFO:teuthology.orchestra.run.smithi019.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-18T15:15:23.077 INFO:teuthology.orchestra.run.smithi019.stdout: inet 127.0.0.1/8 scope host lo 2024-03-18T15:15:23.077 INFO:teuthology.orchestra.run.smithi019.stdout: valid_lft forever preferred_lft forever 2024-03-18T15:15:23.077 INFO:teuthology.orchestra.run.smithi019.stdout: inet6 ::1/128 scope host 2024-03-18T15:15:23.078 INFO:teuthology.orchestra.run.smithi019.stdout: valid_lft forever preferred_lft forever 2024-03-18T15:15:23.078 INFO:teuthology.orchestra.run.smithi019.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T15:15:23.078 INFO:teuthology.orchestra.run.smithi019.stdout: link/ether 0c:c4:7a:6b:fd:12 brd ff:ff:ff:ff:ff:ff 2024-03-18T15:15:23.078 INFO:teuthology.orchestra.run.smithi019.stdout: altname enp7s0f0 2024-03-18T15:15:23.078 INFO:teuthology.orchestra.run.smithi019.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T15:15:23.078 INFO:teuthology.orchestra.run.smithi019.stdout: link/ether 0c:c4:7a:6b:fd:13 brd ff:ff:ff:ff:ff:ff 2024-03-18T15:15:23.078 INFO:teuthology.orchestra.run.smithi019.stdout: altname enp7s0f1 2024-03-18T15:15:23.078 INFO:teuthology.orchestra.run.smithi019.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-18T15:15:23.078 INFO:teuthology.orchestra.run.smithi019.stdout: link/ether 0c:c4:7a:bd:16:1e brd ff:ff:ff:ff:ff:ff 2024-03-18T15:15:23.078 INFO:teuthology.orchestra.run.smithi019.stdout: altname enp1s0f0 2024-03-18T15:15:23.078 INFO:teuthology.orchestra.run.smithi019.stdout: inet 172.21.15.19/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-18T15:15:23.078 INFO:teuthology.orchestra.run.smithi019.stdout: valid_lft 42006sec preferred_lft 42006sec 2024-03-18T15:15:23.079 INFO:teuthology.orchestra.run.smithi019.stdout: inet6 fe80::ec4:7aff:febd:161e/64 scope link noprefixroute 2024-03-18T15:15:23.079 INFO:teuthology.orchestra.run.smithi019.stdout: valid_lft forever preferred_lft forever 2024-03-18T15:15:23.079 INFO:teuthology.orchestra.run.smithi019.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T15:15:23.079 INFO:teuthology.orchestra.run.smithi019.stdout: link/ether 0c:c4:7a:bd:16:1f brd ff:ff:ff:ff:ff:ff 2024-03-18T15:15:23.079 INFO:teuthology.orchestra.run.smithi019.stdout: altname enp1s0f1 2024-03-18T15:15:23.080 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-18T15:15:23.080 DEBUG:teuthology.orchestra.run.smithi019:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-18T15:15:23.080 DEBUG:teuthology.orchestra.run.smithi019:> set -e 2024-03-18T15:15:23.080 DEBUG:teuthology.orchestra.run.smithi019:> sudo ip link add name ceph-brx type bridge 2024-03-18T15:15:23.080 DEBUG:teuthology.orchestra.run.smithi019:> sudo ip addr flush dev ceph-brx 2024-03-18T15:15:23.080 DEBUG:teuthology.orchestra.run.smithi019:> sudo ip link set ceph-brx up 2024-03-18T15:15:23.081 DEBUG:teuthology.orchestra.run.smithi019:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-18T15:15:23.081 DEBUG:teuthology.orchestra.run.smithi019:> ') 2024-03-18T15:15:23.190 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:23 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T15:15:23.503 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:23 smithi019 ceph-mon[31522]: pgmap v323: 129 pgs: 129 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 3 op/s 2024-03-18T15:15:23.608 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:23 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T15:15:23.622 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T15:15:23.622 DEBUG:teuthology.orchestra.run.smithi019:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-18T15:15:23.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:23 smithi104 ceph-mon[36070]: pgmap v323: 129 pgs: 129 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 3 op/s 2024-03-18T15:15:23.701 INFO:teuthology.orchestra.run.smithi019.stdout:1 2024-03-18T15:15:23.705 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T15:15:23.706 DEBUG:teuthology.orchestra.run.smithi019:> route 2024-03-18T15:15:23.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:23 smithi033 ceph-mon[36648]: pgmap v323: 129 pgs: 129 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 3 op/s 2024-03-18T15:15:23.772 INFO:teuthology.orchestra.run.smithi019.stdout:Kernel IP routing table 2024-03-18T15:15:23.772 INFO:teuthology.orchestra.run.smithi019.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-18T15:15:23.773 INFO:teuthology.orchestra.run.smithi019.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-18T15:15:23.773 INFO:teuthology.orchestra.run.smithi019.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-18T15:15:23.773 INFO:teuthology.orchestra.run.smithi019.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-18T15:15:23.774 DEBUG:teuthology.orchestra.run.smithi019:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-18T15:15:23.774 DEBUG:teuthology.orchestra.run.smithi019:> set -e 2024-03-18T15:15:23.774 DEBUG:teuthology.orchestra.run.smithi019:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-18T15:15:23.774 DEBUG:teuthology.orchestra.run.smithi019:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-18T15:15:23.774 DEBUG:teuthology.orchestra.run.smithi019:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-18T15:15:23.774 DEBUG:teuthology.orchestra.run.smithi019:> ') 2024-03-18T15:15:23.887 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:23 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T15:15:24.617 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:24 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T15:15:24.628 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T15:15:24.628 DEBUG:teuthology.orchestra.run.smithi019:> ip netns list 2024-03-18T15:15:24.678 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T15:15:24.679 DEBUG:teuthology.orchestra.run.smithi019:> ip netns list-id 2024-03-18T15:15:24.729 DEBUG:teuthology.orchestra.run.smithi019:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-18T15:15:24.729 DEBUG:teuthology.orchestra.run.smithi019:> set -e 2024-03-18T15:15:24.730 DEBUG:teuthology.orchestra.run.smithi019:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-18T15:15:24.730 DEBUG:teuthology.orchestra.run.smithi019:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-03-18T15:15:24.730 DEBUG:teuthology.orchestra.run.smithi019:> ') 2024-03-18T15:15:24.835 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:24 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T15:15:24.931 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:24 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T15:15:24.941 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-03-18T15:15:24.941 DEBUG:teuthology.orchestra.run.smithi019:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-18T15:15:24.941 DEBUG:teuthology.orchestra.run.smithi019:> set -e 2024-03-18T15:15:24.941 DEBUG:teuthology.orchestra.run.smithi019:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-03-18T15:15:24.941 DEBUG:teuthology.orchestra.run.smithi019:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-18T15:15:24.941 DEBUG:teuthology.orchestra.run.smithi019:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-03-18T15:15:24.941 DEBUG:teuthology.orchestra.run.smithi019:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-03-18T15:15:24.942 DEBUG:teuthology.orchestra.run.smithi019:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-03-18T15:15:24.942 DEBUG:teuthology.orchestra.run.smithi019:> ') 2024-03-18T15:15:25.052 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:25 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T15:15:25.285 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:25 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T15:15:25.296 DEBUG:teuthology.orchestra.run.smithi019:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-18T15:15:25.296 DEBUG:teuthology.orchestra.run.smithi019:> set -e 2024-03-18T15:15:25.296 DEBUG:teuthology.orchestra.run.smithi019:> sudo ip link set brx.0 up 2024-03-18T15:15:25.296 DEBUG:teuthology.orchestra.run.smithi019:> sudo ip link set dev brx.0 master ceph-brx 2024-03-18T15:15:25.296 DEBUG:teuthology.orchestra.run.smithi019:> ') 2024-03-18T15:15:25.398 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:25 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T15:15:25.433 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:25 smithi019 ceph-mon[31522]: pgmap v324: 129 pgs: 129 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 222 B/s wr, 3 op/s 2024-03-18T15:15:25.521 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:25 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T15:15:25.531 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-03-18T15:15:25.531 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-18T15:15:25.531 DEBUG:teuthology.orchestra.run.smithi019:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-03-18T15:15:25.577 INFO:teuthology.orchestra.run.smithi019.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-03-18T15:15:25.577 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-18T15:15:25.577 DEBUG:teuthology.orchestra.run.smithi019:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-03-18T15:15:25.671 DEBUG:teuthology.orchestra.run.smithi019:> sudo modprobe fuse 2024-03-18T15:15:25.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:25 smithi104 ceph-mon[36070]: pgmap v324: 129 pgs: 129 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 222 B/s wr, 3 op/s 2024-03-18T15:15:25.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:25 smithi033 ceph-mon[36648]: pgmap v324: 129 pgs: 129 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 222 B/s wr, 3 op/s 2024-03-18T15:15:25.767 DEBUG:teuthology.orchestra.run.smithi019:> cat /proc/self/mounts | awk '{print $2}' 2024-03-18T15:15:25.821 INFO:teuthology.orchestra.run.smithi019.stdout:/proc 2024-03-18T15:15:25.821 INFO:teuthology.orchestra.run.smithi019.stdout:/sys 2024-03-18T15:15:25.821 INFO:teuthology.orchestra.run.smithi019.stdout:/dev 2024-03-18T15:15:25.821 INFO:teuthology.orchestra.run.smithi019.stdout:/sys/kernel/security 2024-03-18T15:15:25.821 INFO:teuthology.orchestra.run.smithi019.stdout:/dev/shm 2024-03-18T15:15:25.822 INFO:teuthology.orchestra.run.smithi019.stdout:/dev/pts 2024-03-18T15:15:25.822 INFO:teuthology.orchestra.run.smithi019.stdout:/run 2024-03-18T15:15:25.822 INFO:teuthology.orchestra.run.smithi019.stdout:/sys/fs/cgroup 2024-03-18T15:15:25.822 INFO:teuthology.orchestra.run.smithi019.stdout:/sys/fs/pstore 2024-03-18T15:15:25.822 INFO:teuthology.orchestra.run.smithi019.stdout:/sys/fs/bpf 2024-03-18T15:15:25.822 INFO:teuthology.orchestra.run.smithi019.stdout:/ 2024-03-18T15:15:25.822 INFO:teuthology.orchestra.run.smithi019.stdout:/sys/fs/selinux 2024-03-18T15:15:25.822 INFO:teuthology.orchestra.run.smithi019.stdout:/proc/sys/fs/binfmt_misc 2024-03-18T15:15:25.822 INFO:teuthology.orchestra.run.smithi019.stdout:/dev/hugepages 2024-03-18T15:15:25.822 INFO:teuthology.orchestra.run.smithi019.stdout:/dev/mqueue 2024-03-18T15:15:25.822 INFO:teuthology.orchestra.run.smithi019.stdout:/sys/kernel/debug 2024-03-18T15:15:25.822 INFO:teuthology.orchestra.run.smithi019.stdout:/sys/kernel/tracing 2024-03-18T15:15:25.823 INFO:teuthology.orchestra.run.smithi019.stdout:/sys/fs/fuse/connections 2024-03-18T15:15:25.823 INFO:teuthology.orchestra.run.smithi019.stdout:/sys/kernel/config 2024-03-18T15:15:25.823 INFO:teuthology.orchestra.run.smithi019.stdout:/run/credentials/systemd-sysctl.service 2024-03-18T15:15:25.823 INFO:teuthology.orchestra.run.smithi019.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-18T15:15:25.823 INFO:teuthology.orchestra.run.smithi019.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-18T15:15:25.823 INFO:teuthology.orchestra.run.smithi019.stdout:/var/lib/nfs/rpc_pipefs 2024-03-18T15:15:25.823 INFO:teuthology.orchestra.run.smithi019.stdout:/run/user/1000 2024-03-18T15:15:25.823 INFO:teuthology.orchestra.run.smithi019.stdout:/var/lib/ceph 2024-03-18T15:15:25.823 INFO:teuthology.orchestra.run.smithi019.stdout:/var/lib/containers/storage/overlay 2024-03-18T15:15:25.823 INFO:teuthology.orchestra.run.smithi019.stdout:/var/lib/containers/storage/overlay/b66b5134a4ea952a92928f21fb342e7679f069674a569961be2c1b90d235945e/merged 2024-03-18T15:15:25.823 INFO:teuthology.orchestra.run.smithi019.stdout:/var/lib/containers/storage/overlay/e60a447f27e2255bcf4d845a3c2b10cdce0d123e7f9c1600f01d88156fc78fa6/merged 2024-03-18T15:15:25.823 INFO:teuthology.orchestra.run.smithi019.stdout:/run/user/0 2024-03-18T15:15:25.823 INFO:teuthology.orchestra.run.smithi019.stdout:/proc/sys/fs/binfmt_misc 2024-03-18T15:15:25.823 INFO:teuthology.orchestra.run.smithi019.stdout:/var/lib/containers/storage/overlay/fcb443e7538fc4219fea707f22f042e6977d78e6e510f2473c9fc6ed58aa1d34/merged 2024-03-18T15:15:25.824 INFO:teuthology.orchestra.run.smithi019.stdout:/var/lib/containers/storage/overlay/e3a32159e383b655b5643b5c40f57f9ae93b443eb771b78e93af30c1cca13282/merged 2024-03-18T15:15:25.824 INFO:teuthology.orchestra.run.smithi019.stdout:/var/lib/containers/storage/overlay/c216e79ccf5d731cc08d1498cd6fffe6c6d29d80151efd5cf5140ed9388582c0/merged 2024-03-18T15:15:25.824 INFO:teuthology.orchestra.run.smithi019.stdout:/var/lib/containers/storage/overlay/eba1e8dfb65d6f9f3e04558d977da6b5383b5beecbb34d0e5106b89f9e9180e8/merged 2024-03-18T15:15:25.824 INFO:teuthology.orchestra.run.smithi019.stdout:/var/lib/containers/storage/overlay/9f4f55fd97845b81e87f9d9c7089affd7ca50033e50a03c261d853276da91aa8/merged 2024-03-18T15:15:25.824 INFO:teuthology.orchestra.run.smithi019.stdout:/var/lib/containers/storage/overlay/49d7e6a26ceb5c7a3cd0f5e158afc92b78433a8e69424c1bc233934a6f777e5e/merged 2024-03-18T15:15:25.824 INFO:teuthology.orchestra.run.smithi019.stdout:/var/lib/containers/storage/overlay/c6831eb51c280a14b72d1e50e88539fce99785bb58ad511a621d332e0274480c/merged 2024-03-18T15:15:25.824 INFO:teuthology.orchestra.run.smithi019.stdout:/var/lib/containers/storage/overlay/c06b6969a39b74e3b6a133480921beb81a0059fb3cdb30846e7535612f53cf3e/merged 2024-03-18T15:15:25.824 INFO:teuthology.orchestra.run.smithi019.stdout:/run/netns 2024-03-18T15:15:25.824 INFO:teuthology.orchestra.run.smithi019.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-18T15:15:25.824 INFO:teuthology.orchestra.run.smithi019.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-18T15:15:25.825 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T15:15:25.825 DEBUG:teuthology.orchestra.run.smithi019:> ls /sys/fs/fuse/connections 2024-03-18T15:15:25.878 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-18T15:15:25.879 DEBUG:teuthology.orchestra.run.smithi019:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-03-18T15:15:25.884 DEBUG:teuthology.orchestra.run.smithi019:> sudo modprobe fuse 2024-03-18T15:15:25.966 DEBUG:teuthology.orchestra.run.smithi019:> cat /proc/self/mounts | awk '{print $2}' 2024-03-18T15:15:26.020 INFO:teuthology.orchestra.run.smithi019.stdout:/proc 2024-03-18T15:15:26.020 INFO:teuthology.orchestra.run.smithi019.stdout:/sys 2024-03-18T15:15:26.020 INFO:teuthology.orchestra.run.smithi019.stdout:/dev 2024-03-18T15:15:26.020 INFO:teuthology.orchestra.run.smithi019.stdout:/sys/kernel/security 2024-03-18T15:15:26.020 INFO:teuthology.orchestra.run.smithi019.stdout:/dev/shm 2024-03-18T15:15:26.020 INFO:teuthology.orchestra.run.smithi019.stdout:/dev/pts 2024-03-18T15:15:26.020 INFO:teuthology.orchestra.run.smithi019.stdout:/run 2024-03-18T15:15:26.021 INFO:teuthology.orchestra.run.smithi019.stdout:/sys/fs/cgroup 2024-03-18T15:15:26.021 INFO:teuthology.orchestra.run.smithi019.stdout:/sys/fs/pstore 2024-03-18T15:15:26.021 INFO:teuthology.orchestra.run.smithi019.stdout:/sys/fs/bpf 2024-03-18T15:15:26.021 INFO:teuthology.orchestra.run.smithi019.stdout:/ 2024-03-18T15:15:26.021 INFO:teuthology.orchestra.run.smithi019.stdout:/sys/fs/selinux 2024-03-18T15:15:26.021 INFO:teuthology.orchestra.run.smithi019.stdout:/proc/sys/fs/binfmt_misc 2024-03-18T15:15:26.021 INFO:teuthology.orchestra.run.smithi019.stdout:/dev/hugepages 2024-03-18T15:15:26.021 INFO:teuthology.orchestra.run.smithi019.stdout:/dev/mqueue 2024-03-18T15:15:26.021 INFO:teuthology.orchestra.run.smithi019.stdout:/sys/kernel/debug 2024-03-18T15:15:26.021 INFO:teuthology.orchestra.run.smithi019.stdout:/sys/kernel/tracing 2024-03-18T15:15:26.021 INFO:teuthology.orchestra.run.smithi019.stdout:/sys/fs/fuse/connections 2024-03-18T15:15:26.021 INFO:teuthology.orchestra.run.smithi019.stdout:/sys/kernel/config 2024-03-18T15:15:26.021 INFO:teuthology.orchestra.run.smithi019.stdout:/run/credentials/systemd-sysctl.service 2024-03-18T15:15:26.022 INFO:teuthology.orchestra.run.smithi019.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-18T15:15:26.022 INFO:teuthology.orchestra.run.smithi019.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-18T15:15:26.022 INFO:teuthology.orchestra.run.smithi019.stdout:/var/lib/nfs/rpc_pipefs 2024-03-18T15:15:26.022 INFO:teuthology.orchestra.run.smithi019.stdout:/run/user/1000 2024-03-18T15:15:26.022 INFO:teuthology.orchestra.run.smithi019.stdout:/var/lib/ceph 2024-03-18T15:15:26.022 INFO:teuthology.orchestra.run.smithi019.stdout:/var/lib/containers/storage/overlay 2024-03-18T15:15:26.022 INFO:teuthology.orchestra.run.smithi019.stdout:/var/lib/containers/storage/overlay/b66b5134a4ea952a92928f21fb342e7679f069674a569961be2c1b90d235945e/merged 2024-03-18T15:15:26.022 INFO:teuthology.orchestra.run.smithi019.stdout:/var/lib/containers/storage/overlay/e60a447f27e2255bcf4d845a3c2b10cdce0d123e7f9c1600f01d88156fc78fa6/merged 2024-03-18T15:15:26.022 INFO:teuthology.orchestra.run.smithi019.stdout:/run/user/0 2024-03-18T15:15:26.022 INFO:teuthology.orchestra.run.smithi019.stdout:/proc/sys/fs/binfmt_misc 2024-03-18T15:15:26.022 INFO:teuthology.orchestra.run.smithi019.stdout:/var/lib/containers/storage/overlay/fcb443e7538fc4219fea707f22f042e6977d78e6e510f2473c9fc6ed58aa1d34/merged 2024-03-18T15:15:26.022 INFO:teuthology.orchestra.run.smithi019.stdout:/var/lib/containers/storage/overlay/e3a32159e383b655b5643b5c40f57f9ae93b443eb771b78e93af30c1cca13282/merged 2024-03-18T15:15:26.022 INFO:teuthology.orchestra.run.smithi019.stdout:/var/lib/containers/storage/overlay/c216e79ccf5d731cc08d1498cd6fffe6c6d29d80151efd5cf5140ed9388582c0/merged 2024-03-18T15:15:26.023 INFO:teuthology.orchestra.run.smithi019.stdout:/var/lib/containers/storage/overlay/eba1e8dfb65d6f9f3e04558d977da6b5383b5beecbb34d0e5106b89f9e9180e8/merged 2024-03-18T15:15:26.023 INFO:teuthology.orchestra.run.smithi019.stdout:/var/lib/containers/storage/overlay/9f4f55fd97845b81e87f9d9c7089affd7ca50033e50a03c261d853276da91aa8/merged 2024-03-18T15:15:26.023 INFO:teuthology.orchestra.run.smithi019.stdout:/var/lib/containers/storage/overlay/49d7e6a26ceb5c7a3cd0f5e158afc92b78433a8e69424c1bc233934a6f777e5e/merged 2024-03-18T15:15:26.023 INFO:teuthology.orchestra.run.smithi019.stdout:/var/lib/containers/storage/overlay/c6831eb51c280a14b72d1e50e88539fce99785bb58ad511a621d332e0274480c/merged 2024-03-18T15:15:26.023 INFO:teuthology.orchestra.run.smithi019.stdout:/var/lib/containers/storage/overlay/c06b6969a39b74e3b6a133480921beb81a0059fb3cdb30846e7535612f53cf3e/merged 2024-03-18T15:15:26.023 INFO:teuthology.orchestra.run.smithi019.stdout:/run/netns 2024-03-18T15:15:26.023 INFO:teuthology.orchestra.run.smithi019.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-18T15:15:26.023 INFO:teuthology.orchestra.run.smithi019.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-18T15:15:26.024 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T15:15:26.024 DEBUG:teuthology.orchestra.run.smithi019:> ls /sys/fs/fuse/connections 2024-03-18T15:15:26.094 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi019.stderr:2024-03-18T15:15:26.093+0000 7faf84644480 -1 init, newargv = 0x56545d911580 newargc=15 2024-03-18T15:15:26.095 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi019.stderr:ceph-fuse[75442]: starting ceph client 2024-03-18T15:15:26.104 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi019.stderr:ceph-fuse[75442]: starting fuse 2024-03-18T15:15:27.172 DEBUG:teuthology.orchestra.run.smithi019:> sudo modprobe fuse 2024-03-18T15:15:27.281 DEBUG:teuthology.orchestra.run.smithi019:> cat /proc/self/mounts | awk '{print $2}' 2024-03-18T15:15:27.341 INFO:teuthology.orchestra.run.smithi019.stdout:/proc 2024-03-18T15:15:27.341 INFO:teuthology.orchestra.run.smithi019.stdout:/sys 2024-03-18T15:15:27.341 INFO:teuthology.orchestra.run.smithi019.stdout:/dev 2024-03-18T15:15:27.341 INFO:teuthology.orchestra.run.smithi019.stdout:/sys/kernel/security 2024-03-18T15:15:27.341 INFO:teuthology.orchestra.run.smithi019.stdout:/dev/shm 2024-03-18T15:15:27.341 INFO:teuthology.orchestra.run.smithi019.stdout:/dev/pts 2024-03-18T15:15:27.341 INFO:teuthology.orchestra.run.smithi019.stdout:/run 2024-03-18T15:15:27.341 INFO:teuthology.orchestra.run.smithi019.stdout:/sys/fs/cgroup 2024-03-18T15:15:27.341 INFO:teuthology.orchestra.run.smithi019.stdout:/sys/fs/pstore 2024-03-18T15:15:27.341 INFO:teuthology.orchestra.run.smithi019.stdout:/sys/fs/bpf 2024-03-18T15:15:27.341 INFO:teuthology.orchestra.run.smithi019.stdout:/ 2024-03-18T15:15:27.342 INFO:teuthology.orchestra.run.smithi019.stdout:/sys/fs/selinux 2024-03-18T15:15:27.342 INFO:teuthology.orchestra.run.smithi019.stdout:/proc/sys/fs/binfmt_misc 2024-03-18T15:15:27.342 INFO:teuthology.orchestra.run.smithi019.stdout:/dev/hugepages 2024-03-18T15:15:27.342 INFO:teuthology.orchestra.run.smithi019.stdout:/dev/mqueue 2024-03-18T15:15:27.342 INFO:teuthology.orchestra.run.smithi019.stdout:/sys/kernel/debug 2024-03-18T15:15:27.342 INFO:teuthology.orchestra.run.smithi019.stdout:/sys/kernel/tracing 2024-03-18T15:15:27.342 INFO:teuthology.orchestra.run.smithi019.stdout:/sys/fs/fuse/connections 2024-03-18T15:15:27.342 INFO:teuthology.orchestra.run.smithi019.stdout:/sys/kernel/config 2024-03-18T15:15:27.342 INFO:teuthology.orchestra.run.smithi019.stdout:/run/credentials/systemd-sysctl.service 2024-03-18T15:15:27.342 INFO:teuthology.orchestra.run.smithi019.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-18T15:15:27.342 INFO:teuthology.orchestra.run.smithi019.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-18T15:15:27.342 INFO:teuthology.orchestra.run.smithi019.stdout:/var/lib/nfs/rpc_pipefs 2024-03-18T15:15:27.342 INFO:teuthology.orchestra.run.smithi019.stdout:/run/user/1000 2024-03-18T15:15:27.343 INFO:teuthology.orchestra.run.smithi019.stdout:/var/lib/ceph 2024-03-18T15:15:27.343 INFO:teuthology.orchestra.run.smithi019.stdout:/var/lib/containers/storage/overlay 2024-03-18T15:15:27.343 INFO:teuthology.orchestra.run.smithi019.stdout:/var/lib/containers/storage/overlay/b66b5134a4ea952a92928f21fb342e7679f069674a569961be2c1b90d235945e/merged 2024-03-18T15:15:27.343 INFO:teuthology.orchestra.run.smithi019.stdout:/var/lib/containers/storage/overlay/e60a447f27e2255bcf4d845a3c2b10cdce0d123e7f9c1600f01d88156fc78fa6/merged 2024-03-18T15:15:27.343 INFO:teuthology.orchestra.run.smithi019.stdout:/run/user/0 2024-03-18T15:15:27.343 INFO:teuthology.orchestra.run.smithi019.stdout:/proc/sys/fs/binfmt_misc 2024-03-18T15:15:27.343 INFO:teuthology.orchestra.run.smithi019.stdout:/var/lib/containers/storage/overlay/fcb443e7538fc4219fea707f22f042e6977d78e6e510f2473c9fc6ed58aa1d34/merged 2024-03-18T15:15:27.343 INFO:teuthology.orchestra.run.smithi019.stdout:/var/lib/containers/storage/overlay/e3a32159e383b655b5643b5c40f57f9ae93b443eb771b78e93af30c1cca13282/merged 2024-03-18T15:15:27.343 INFO:teuthology.orchestra.run.smithi019.stdout:/var/lib/containers/storage/overlay/c216e79ccf5d731cc08d1498cd6fffe6c6d29d80151efd5cf5140ed9388582c0/merged 2024-03-18T15:15:27.343 INFO:teuthology.orchestra.run.smithi019.stdout:/var/lib/containers/storage/overlay/eba1e8dfb65d6f9f3e04558d977da6b5383b5beecbb34d0e5106b89f9e9180e8/merged 2024-03-18T15:15:27.343 INFO:teuthology.orchestra.run.smithi019.stdout:/var/lib/containers/storage/overlay/9f4f55fd97845b81e87f9d9c7089affd7ca50033e50a03c261d853276da91aa8/merged 2024-03-18T15:15:27.343 INFO:teuthology.orchestra.run.smithi019.stdout:/var/lib/containers/storage/overlay/49d7e6a26ceb5c7a3cd0f5e158afc92b78433a8e69424c1bc233934a6f777e5e/merged 2024-03-18T15:15:27.343 INFO:teuthology.orchestra.run.smithi019.stdout:/var/lib/containers/storage/overlay/c6831eb51c280a14b72d1e50e88539fce99785bb58ad511a621d332e0274480c/merged 2024-03-18T15:15:27.344 INFO:teuthology.orchestra.run.smithi019.stdout:/var/lib/containers/storage/overlay/c06b6969a39b74e3b6a133480921beb81a0059fb3cdb30846e7535612f53cf3e/merged 2024-03-18T15:15:27.344 INFO:teuthology.orchestra.run.smithi019.stdout:/run/netns 2024-03-18T15:15:27.344 INFO:teuthology.orchestra.run.smithi019.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-18T15:15:27.344 INFO:teuthology.orchestra.run.smithi019.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-18T15:15:27.344 INFO:teuthology.orchestra.run.smithi019.stdout:/home/ubuntu/cephtest/mnt.admin 2024-03-18T15:15:27.345 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T15:15:27.345 DEBUG:teuthology.orchestra.run.smithi019:> ls /sys/fs/fuse/connections 2024-03-18T15:15:27.401 INFO:teuthology.orchestra.run.smithi019.stdout:118 2024-03-18T15:15:27.401 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [118] 2024-03-18T15:15:27.401 DEBUG:teuthology.orchestra.run.smithi019:> sudo stdin-killer -- python3 -c ' 2024-03-18T15:15:27.401 DEBUG:teuthology.orchestra.run.smithi019:> import glob 2024-03-18T15:15:27.401 DEBUG:teuthology.orchestra.run.smithi019:> import re 2024-03-18T15:15:27.402 DEBUG:teuthology.orchestra.run.smithi019:> import os 2024-03-18T15:15:27.402 DEBUG:teuthology.orchestra.run.smithi019:> import subprocess 2024-03-18T15:15:27.402 DEBUG:teuthology.orchestra.run.smithi019:> 2024-03-18T15:15:27.402 DEBUG:teuthology.orchestra.run.smithi019:> def _find_admin_socket(client_name): 2024-03-18T15:15:27.402 DEBUG:teuthology.orchestra.run.smithi019:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-03-18T15:15:27.402 DEBUG:teuthology.orchestra.run.smithi019:> files = glob.glob(asok_path) 2024-03-18T15:15:27.402 DEBUG:teuthology.orchestra.run.smithi019:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-03-18T15:15:27.402 DEBUG:teuthology.orchestra.run.smithi019:> 2024-03-18T15:15:27.402 DEBUG:teuthology.orchestra.run.smithi019:> # Given a non-glob path, it better be there 2024-03-18T15:15:27.402 DEBUG:teuthology.orchestra.run.smithi019:> if "*" not in asok_path: 2024-03-18T15:15:27.402 DEBUG:teuthology.orchestra.run.smithi019:> assert(len(files) == 1) 2024-03-18T15:15:27.402 DEBUG:teuthology.orchestra.run.smithi019:> return files[0] 2024-03-18T15:15:27.402 DEBUG:teuthology.orchestra.run.smithi019:> 2024-03-18T15:15:27.402 DEBUG:teuthology.orchestra.run.smithi019:> for f in files: 2024-03-18T15:15:27.403 DEBUG:teuthology.orchestra.run.smithi019:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-18T15:15:27.403 DEBUG:teuthology.orchestra.run.smithi019:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-18T15:15:27.403 DEBUG:teuthology.orchestra.run.smithi019:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-18T15:15:27.403 DEBUG:teuthology.orchestra.run.smithi019:> contents = proc_f.read() 2024-03-18T15:15:27.403 DEBUG:teuthology.orchestra.run.smithi019:> if mountpoint in contents: 2024-03-18T15:15:27.403 DEBUG:teuthology.orchestra.run.smithi019:> return f 2024-03-18T15:15:27.403 DEBUG:teuthology.orchestra.run.smithi019:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-18T15:15:27.403 DEBUG:teuthology.orchestra.run.smithi019:> 2024-03-18T15:15:27.403 DEBUG:teuthology.orchestra.run.smithi019:> print(_find_admin_socket("client.admin")) 2024-03-18T15:15:27.403 DEBUG:teuthology.orchestra.run.smithi019:> ' 2024-03-18T15:15:27.589 INFO:teuthology.orchestra.run.smithi019.stdout:/var/run/ceph/ceph-client.admin.75442.asok 2024-03-18T15:15:27.593 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:27 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T15:15:27.603 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:27 smithi019 ceph-mon[31522]: pgmap v325: 129 pgs: 129 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:15:27.607 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.75442.asok 2024-03-18T15:15:27.607 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T15:15:27.607 DEBUG:teuthology.orchestra.run.smithi019:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.75442.asok status 2024-03-18T15:15:27.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:27 smithi104 ceph-mon[36070]: pgmap v325: 129 pgs: 129 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:15:27.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:27 smithi033 ceph-mon[36648]: pgmap v325: 129 pgs: 129 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:15:27.849 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-03-18T15:15:27.849 INFO:teuthology.orchestra.run.smithi019.stdout: "metadata": { 2024-03-18T15:15:27.849 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph_sha1": "3d78f5ba07bf18880172c07d4d7b4ac00c9d30af", 2024-03-18T15:15:27.849 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph_version": "ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev)", 2024-03-18T15:15:27.849 INFO:teuthology.orchestra.run.smithi019.stdout: "entity_id": "admin", 2024-03-18T15:15:27.849 INFO:teuthology.orchestra.run.smithi019.stdout: "hostname": "smithi019", 2024-03-18T15:15:27.849 INFO:teuthology.orchestra.run.smithi019.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-03-18T15:15:27.849 INFO:teuthology.orchestra.run.smithi019.stdout: "pid": "75442", 2024-03-18T15:15:27.849 INFO:teuthology.orchestra.run.smithi019.stdout: "root": "/" 2024-03-18T15:15:27.850 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:15:27.850 INFO:teuthology.orchestra.run.smithi019.stdout: "dentry_count": 0, 2024-03-18T15:15:27.850 INFO:teuthology.orchestra.run.smithi019.stdout: "dentry_pinned_count": 0, 2024-03-18T15:15:27.850 INFO:teuthology.orchestra.run.smithi019.stdout: "id": 24686, 2024-03-18T15:15:27.850 INFO:teuthology.orchestra.run.smithi019.stdout: "inst": { 2024-03-18T15:15:27.850 INFO:teuthology.orchestra.run.smithi019.stdout: "name": { 2024-03-18T15:15:27.850 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "client", 2024-03-18T15:15:27.850 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 24686 2024-03-18T15:15:27.850 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:15:27.850 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": { 2024-03-18T15:15:27.850 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-03-18T15:15:27.850 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:0", 2024-03-18T15:15:27.850 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 1734837826 2024-03-18T15:15:27.851 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:15:27.851 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:15:27.851 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": { 2024-03-18T15:15:27.851 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-03-18T15:15:27.851 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:0", 2024-03-18T15:15:27.851 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 1734837826 2024-03-18T15:15:27.851 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:15:27.851 INFO:teuthology.orchestra.run.smithi019.stdout: "inst_str": "client.24686 172.21.15.19:0/1734837826", 2024-03-18T15:15:27.851 INFO:teuthology.orchestra.run.smithi019.stdout: "addr_str": "172.21.15.19:0/1734837826", 2024-03-18T15:15:27.851 INFO:teuthology.orchestra.run.smithi019.stdout: "inode_count": 1, 2024-03-18T15:15:27.851 INFO:teuthology.orchestra.run.smithi019.stdout: "mds_epoch": 16, 2024-03-18T15:15:27.851 INFO:teuthology.orchestra.run.smithi019.stdout: "osd_epoch": 89, 2024-03-18T15:15:27.851 INFO:teuthology.orchestra.run.smithi019.stdout: "osd_epoch_barrier": 0, 2024-03-18T15:15:27.851 INFO:teuthology.orchestra.run.smithi019.stdout: "blocklisted": false, 2024-03-18T15:15:27.851 INFO:teuthology.orchestra.run.smithi019.stdout: "fs_name": "cephfs" 2024-03-18T15:15:27.852 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-03-18T15:15:27.864 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T15:15:27.865 DEBUG:teuthology.orchestra.run.smithi019:> stat --file-system '--printf=%T 2024-03-18T15:15:27.865 DEBUG:teuthology.orchestra.run.smithi019:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-03-18T15:15:27.924 INFO:teuthology.orchestra.run.smithi019.stdout:fuseblk 2024-03-18T15:15:27.924 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-03-18T15:15:27.924 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T15:15:27.924 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-03-18T15:15:28.027 DEBUG:teuthology.orchestra.run.smithi019:> (cd /home/ubuntu/cephtest/mnt.admin && exec stdin-killer --timeout=900 -- bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-03-18T15:15:28.147 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:28 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-18T15:15:28.182 INFO:teuthology.orchestra.run.smithi019.stdout:# file: . 2024-03-18T15:15:28.182 INFO:teuthology.orchestra.run.smithi019.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-03-18T15:15:28.182 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-18T15:15:28.183 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T15:15:28.192 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-03-18T15:15:28.192 DEBUG:teuthology.orchestra.run.smithi019:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T15:15:28.251 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T15:15:28.480 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:28.479+0000 7fcbd1600640 1 -- 172.21.15.19:0/2062303211 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcbcc100840 msgr2=0x7fcbcc100ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:28.480 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:28.479+0000 7fcbd1600640 1 --2- 172.21.15.19:0/2062303211 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcbcc100840 0x7fcbcc100ca0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fcbb4009900 tx=0x7fcbb402bcd0 comp rx=0 tx=0).stop 2024-03-18T15:15:28.480 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:28.479+0000 7fcbd1600640 1 -- 172.21.15.19:0/2062303211 shutdown_connections 2024-03-18T15:15:28.480 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:28.479+0000 7fcbd1600640 1 --2- 172.21.15.19:0/2062303211 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fcbcc1011e0 0x7fcbcc101660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:28.480 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:28.479+0000 7fcbd1600640 1 --2- 172.21.15.19:0/2062303211 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcbcc100840 0x7fcbcc100ca0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:28.480 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:28.479+0000 7fcbd1600640 1 --2- 172.21.15.19:0/2062303211 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcbcc0ffee0 0x7fcbcc100300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:28.480 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:28.479+0000 7fcbd1600640 1 -- 172.21.15.19:0/2062303211 >> 172.21.15.19:0/2062303211 conn(0x7fcbcc0fba60 msgr2=0x7fcbcc0fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:28.480 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:28.479+0000 7fcbd1600640 1 -- 172.21.15.19:0/2062303211 shutdown_connections 2024-03-18T15:15:28.481 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:28.480+0000 7fcbd1600640 1 -- 172.21.15.19:0/2062303211 wait complete. 2024-03-18T15:15:28.483 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:28.481+0000 7fcbd1600640 1 Processor -- start 2024-03-18T15:15:28.483 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:28.481+0000 7fcbd1600640 1 -- start start 2024-03-18T15:15:28.483 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:28.481+0000 7fcbd1600640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcbcc0ffee0 0x7fcbcc199ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:28.483 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:28.481+0000 7fcbd1600640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fcbcc100840 0x7fcbcc19a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:28.483 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:28.481+0000 7fcbd1600640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcbcc1011e0 0x7fcbcc1a1080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:28.483 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:28.482+0000 7fcbd1600640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fcbcc10f4c0 con 0x7fcbcc100840 2024-03-18T15:15:28.483 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:28.482+0000 7fcbd1600640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fcbcc10f340 con 0x7fcbcc0ffee0 2024-03-18T15:15:28.483 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:28.482+0000 7fcbd1600640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fcbcc10f640 con 0x7fcbcc1011e0 2024-03-18T15:15:28.483 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:28.482+0000 7fcbcbe00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcbcc0ffee0 0x7fcbcc199ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:28.483 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:28.482+0000 7fcbcbe00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcbcc0ffee0 0x7fcbcc199ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:33884/0 (socket says 172.21.15.19:33884) 2024-03-18T15:15:28.483 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:28.482+0000 7fcbcbe00640 1 -- 172.21.15.19:0/2087299541 learned_addr learned my addr 172.21.15.19:0/2087299541 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:15:28.483 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:28.482+0000 7fcbcb400640 1 --2- 172.21.15.19:0/2087299541 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fcbcc100840 0x7fcbcc19a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:28.484 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:28.482+0000 7fcbd0c00640 1 --2- 172.21.15.19:0/2087299541 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcbcc1011e0 0x7fcbcc1a1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:28.484 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:28.482+0000 7fcbcbe00640 1 -- 172.21.15.19:0/2087299541 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcbcc1011e0 msgr2=0x7fcbcc1a1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:28.484 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:28.482+0000 7fcbcbe00640 1 --2- 172.21.15.19:0/2087299541 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcbcc1011e0 0x7fcbcc1a1080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:28.484 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:28.482+0000 7fcbcbe00640 1 -- 172.21.15.19:0/2087299541 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fcbcc100840 msgr2=0x7fcbcc19a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:28.484 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:28.482+0000 7fcbcbe00640 1 --2- 172.21.15.19:0/2087299541 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fcbcc100840 0x7fcbcc19a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:28.484 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:28.482+0000 7fcbcbe00640 1 -- 172.21.15.19:0/2087299541 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbcc1a16f0 con 0x7fcbcc0ffee0 2024-03-18T15:15:28.484 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:28.482+0000 7fcbd0c00640 1 --2- 172.21.15.19:0/2087299541 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcbcc1011e0 0x7fcbcc1a1080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T15:15:28.484 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:28.482+0000 7fcbcb400640 1 --2- 172.21.15.19:0/2087299541 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fcbcc100840 0x7fcbcc19a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T15:15:28.484 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:28.482+0000 7fcbcbe00640 1 --2- 172.21.15.19:0/2087299541 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcbcc0ffee0 0x7fcbcc199ad0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fcbbc00eb90 tx=0x7fcbbc00c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:28.484 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:28.483+0000 7fcbc8c00640 1 -- 172.21.15.19:0/2087299541 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbbc004d20 con 0x7fcbcc0ffee0 2024-03-18T15:15:28.484 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:28.483+0000 7fcbc8c00640 1 -- 172.21.15.19:0/2087299541 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcbbc00ee90 con 0x7fcbcc0ffee0 2024-03-18T15:15:28.484 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:28.483+0000 7fcbd1600640 1 -- 172.21.15.19:0/2087299541 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbcc1a19e0 con 0x7fcbcc0ffee0 2024-03-18T15:15:28.484 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:28.483+0000 7fcbc8c00640 1 -- 172.21.15.19:0/2087299541 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbbc010830 con 0x7fcbcc0ffee0 2024-03-18T15:15:28.484 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:28.483+0000 7fcbd1600640 1 -- 172.21.15.19:0/2087299541 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbcc1a1f00 con 0x7fcbcc0ffee0 2024-03-18T15:15:28.485 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:28.484+0000 7fcbd1600640 1 -- 172.21.15.19:0/2087299541 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcbcc10ac30 con 0x7fcbcc0ffee0 2024-03-18T15:15:28.485 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:28.485+0000 7fcbc8c00640 1 -- 172.21.15.19:0/2087299541 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fcbbc005020 con 0x7fcbcc0ffee0 2024-03-18T15:15:28.487 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:28.487+0000 7fcbc8c00640 1 --2- 172.21.15.19:0/2087299541 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fcbb00a2880 0x7fcbb00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:28.487 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:28.487+0000 7fcbc8c00640 1 -- 172.21.15.19:0/2087299541 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fcbbc0cf400 con 0x7fcbcc0ffee0 2024-03-18T15:15:28.487 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:28.487+0000 7fcbcb400640 1 --2- 172.21.15.19:0/2087299541 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fcbb00a2880 0x7fcbb00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:28.487 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:28.488+0000 7fcbcb400640 1 --2- 172.21.15.19:0/2087299541 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fcbb00a2880 0x7fcbb00a4d40 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fcbb402c2e0 tx=0x7fcbb4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:28.491 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:28.491+0000 7fcbc8c00640 1 -- 172.21.15.19:0/2087299541 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fcbbc09c200 con 0x7fcbcc0ffee0 2024-03-18T15:15:28.695 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:28.694+0000 7fcbd1600640 1 -- 172.21.15.19:0/2087299541 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fcbcc0623e0 con 0x7fcbcc0ffee0 2024-03-18T15:15:28.696 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:28.696+0000 7fcbc8c00640 1 -- 172.21.15.19:0/2087299541 <== mon.2 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24093 (secure 0 0 0) 0x7fcbbc09bb00 con 0x7fcbcc0ffee0 2024-03-18T15:15:28.697 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-18T15:15:28.697 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":89,"fsid":"d4dc2ba6-e538-11ee-95c9-87774f69a715","created":"2024-03-18T15:05:57.538551+0000","modified":"2024-03-18T15:15:15.582418+0000","last_up_change":"2024-03-18T15:14:12.009522+0000","last_in_change":"2024-03-18T15:13:52.555387+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T15:09:56.843349+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T15:15:08.941047+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T15:15:10.044671+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T15:15:13.621916+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"37b5a5cd-27ae-4e77-9b05-774b817b2ee9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6802","nonce":402333948},{"type":"v1","addr":"172.21.15.19:6803","nonce":402333948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6804","nonce":402333948},{"type":"v1","addr":"172.21.15.19:6805","nonce":402333948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6808","nonce":402333948},{"type":"v1","addr":"172.21.15.19:6809","nonce":402333948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6806","nonce":402333948},{"type":"v1","addr":"172.21.15.19:6807","nonce":402333948}]},"public_addr":"172.21.15.19:6803/402333948","cluster_addr":"172.21.15.19:6805/402333948","heartbeat_back_addr":"172.21.15.19:6809/402333948","heartbeat_front_addr":"172.21.15.19:6807/402333948","state":["exists","up"]},{"osd":1,"uuid":"3a1134e8-ede7-4dbd-9be1-eefd2cbb43bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1951880113},{"type":"v1","addr":"172.21.15.33:6801","nonce":1951880113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1951880113},{"type":"v1","addr":"172.21.15.33:6803","nonce":1951880113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1951880113},{"type":"v1","addr":"172.21.15.33:6807","nonce":1951880113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1951880113},{"type":"v1","addr":"172.21.15.33:6805","nonce":1951880113}]},"public_addr":"172.21.15.33:6801/1951880113","cluster_addr":"172.21.15.33:6803/1951880113","heartbeat_back_addr":"172.21.15.33:6807/1951880113","heartbeat_front_addr":"172.21.15.33:6805/1951880113","state":["exists","up"]},{"osd":2,"uuid":"f383a164-a3a0-49fd-a9e8-9c2178d8cfb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":1004727949},{"type":"v1","addr":"172.21.15.104:6801","nonce":1004727949}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":1004727949},{"type":"v1","addr":"172.21.15.104:6803","nonce":1004727949}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":1004727949},{"type":"v1","addr":"172.21.15.104:6807","nonce":1004727949}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":1004727949},{"type":"v1","addr":"172.21.15.104:6805","nonce":1004727949}]},"public_addr":"172.21.15.104:6801/1004727949","cluster_addr":"172.21.15.104:6803/1004727949","heartbeat_back_addr":"172.21.15.104:6807/1004727949","heartbeat_front_addr":"172.21.15.104:6805/1004727949","state":["exists","up"]},{"osd":3,"uuid":"c409e116-28bf-45dd-a791-539af22abe1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6810","nonce":3571525854},{"type":"v1","addr":"172.21.15.19:6811","nonce":3571525854}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6812","nonce":3571525854},{"type":"v1","addr":"172.21.15.19:6813","nonce":3571525854}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6816","nonce":3571525854},{"type":"v1","addr":"172.21.15.19:6817","nonce":3571525854}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6814","nonce":3571525854},{"type":"v1","addr":"172.21.15.19:6815","nonce":3571525854}]},"public_addr":"172.21.15.19:6811/3571525854","cluster_addr":"172.21.15.19:6813/3571525854","heartbeat_back_addr":"172.21.15.19:6817/3571525854","heartbeat_front_addr":"172.21.15.19:6815/3571525854","state":["exists","up"]},{"osd":4,"uuid":"696eb719-0b3e-43ae-b5f1-ccd73a3b2843","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":624901749},{"type":"v1","addr":"172.21.15.33:6809","nonce":624901749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":624901749},{"type":"v1","addr":"172.21.15.33:6811","nonce":624901749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":624901749},{"type":"v1","addr":"172.21.15.33:6815","nonce":624901749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":624901749},{"type":"v1","addr":"172.21.15.33:6813","nonce":624901749}]},"public_addr":"172.21.15.33:6809/624901749","cluster_addr":"172.21.15.33:6811/624901749","heartbeat_back_addr":"172.21.15.33:6815/624901749","heartbeat_front_addr":"172.21.15.33:6813/624901749","state":["exists","up"]},{"osd":5,"uuid":"21789779-8bb9-42c7-b9ee-437985f11e3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":1258208297},{"type":"v1","addr":"172.21.15.104:6809","nonce":1258208297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":1258208297},{"type":"v1","addr":"172.21.15.104:6811","nonce":1258208297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":1258208297},{"type":"v1","addr":"172.21.15.104:6815","nonce":1258208297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":1258208297},{"type":"v1","addr":"172.21.15.104:6813","nonce":1258208297}]},"public_addr":"172.21.15.104:6809/1258208297","cluster_addr":"172.21.15.104:6811/1258208297","heartbeat_back_addr":"172.21.15.104:6815/1258208297","heartbeat_front_addr":"172.21.15.104:6813/1258208297","state":["exists","up"]},{"osd":6,"uuid":"0e92a2ef-43cd-487c-9203-e219158aef8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6818","nonce":2782482301},{"type":"v1","addr":"172.21.15.19:6819","nonce":2782482301}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6820","nonce":2782482301},{"type":"v1","addr":"172.21.15.19:6821","nonce":2782482301}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6824","nonce":2782482301},{"type":"v1","addr":"172.21.15.19:6825","nonce":2782482301}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6822","nonce":2782482301},{"type":"v1","addr":"172.21.15.19:6823","nonce":2782482301}]},"public_addr":"172.21.15.19:6819/2782482301","cluster_addr":"172.21.15.19:6821/2782482301","heartbeat_back_addr":"172.21.15.19:6825/2782482301","heartbeat_front_addr":"172.21.15.19:6823/2782482301","state":["exists","up"]},{"osd":7,"uuid":"a16e39a1-3685-491d-b376-3e80a2084072","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":4238160988},{"type":"v1","addr":"172.21.15.33:6817","nonce":4238160988}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":4238160988},{"type":"v1","addr":"172.21.15.33:6819","nonce":4238160988}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":4238160988},{"type":"v1","addr":"172.21.15.33:6823","nonce":4238160988}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":4238160988},{"type":"v1","addr":"172.21.15.33:6821","nonce":4238160988}]},"public_addr":"172.21.15.33:6817/4238160988","cluster_addr":"172.21.15.33:6819/4238160988","heartbeat_back_addr":"172.21.15.33:6823/4238160988","heartbeat_front_addr":"172.21.15.33:6821/4238160988","state":["exists","up"]},{"osd":8,"uuid":"d5c3fa92-cd5d-4349-9c7d-d5666bacde16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":1014779813},{"type":"v1","addr":"172.21.15.104:6817","nonce":1014779813}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":1014779813},{"type":"v1","addr":"172.21.15.104:6819","nonce":1014779813}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":1014779813},{"type":"v1","addr":"172.21.15.104:6823","nonce":1014779813}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":1014779813},{"type":"v1","addr":"172.21.15.104:6821","nonce":1014779813}]},"public_addr":"172.21.15.104:6817/1014779813","cluster_addr":"172.21.15.104:6819/1014779813","heartbeat_back_addr":"172.21.15.104:6823/1014779813","heartbeat_front_addr":"172.21.15.104:6821/1014779813","state":["exists","up"]},{"osd":9,"uuid":"eed2e9b2-26d3-4c46-88ec-34b5596fe991","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6826","nonce":2641151009},{"type":"v1","addr":"172.21.15.19:6827","nonce":2641151009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6828","nonce":2641151009},{"type":"v1","addr":"172.21.15.19:6829","nonce":2641151009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6832","nonce":2641151009},{"type":"v1","addr":"172.21.15.19:6833","nonce":2641151009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6830","nonce":2641151009},{"type":"v1","addr":"172.21.15.19:6831","nonce":2641151009}]},"public_addr":"172.21.15.19:6827/2641151009","cluster_addr":"172.21.15.19:6829/2641151009","heartbeat_back_addr":"172.21.15.19:6833/2641151009","heartbeat_front_addr":"172.21.15.19:6831/2641151009","state":["exists","up"]},{"osd":10,"uuid":"f3c35ea5-fbe7-4dc4-93c5-b92d1a714db3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":1346058869},{"type":"v1","addr":"172.21.15.33:6825","nonce":1346058869}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":1346058869},{"type":"v1","addr":"172.21.15.33:6827","nonce":1346058869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":1346058869},{"type":"v1","addr":"172.21.15.33:6831","nonce":1346058869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":1346058869},{"type":"v1","addr":"172.21.15.33:6829","nonce":1346058869}]},"public_addr":"172.21.15.33:6825/1346058869","cluster_addr":"172.21.15.33:6827/1346058869","heartbeat_back_addr":"172.21.15.33:6831/1346058869","heartbeat_front_addr":"172.21.15.33:6829/1346058869","state":["exists","up"]},{"osd":11,"uuid":"c8b51571-f807-4b62-90b6-7529e09e59da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":3759498281},{"type":"v1","addr":"172.21.15.104:6825","nonce":3759498281}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":3759498281},{"type":"v1","addr":"172.21.15.104:6827","nonce":3759498281}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":3759498281},{"type":"v1","addr":"172.21.15.104:6831","nonce":3759498281}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":3759498281},{"type":"v1","addr":"172.21.15.104:6829","nonce":3759498281}]},"public_addr":"172.21.15.104:6825/3759498281","cluster_addr":"172.21.15.104:6827/3759498281","heartbeat_back_addr":"172.21.15.104:6831/3759498281","heartbeat_front_addr":"172.21.15.104:6829/3759498281","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:09:05.267795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:09:30.234068+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:09:53.870232+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:10:26.471296+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:10:53.180341+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:11:17.801361+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:11:51.364816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:12:18.306408+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:12:43.978744+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:13:18.213964+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:13:45.437544+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:14:10.148580+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.19:0/3860783274":"2024-03-19T15:07:14.150983+0000","172.21.15.19:0/2737223213":"2024-03-19T15:07:14.150983+0000","172.21.15.19:0/176699025":"2024-03-19T15:06:38.688405+0000","172.21.15.19:0/2316969005":"2024-03-19T15:07:14.150983+0000","172.21.15.19:6801/2776008962":"2024-03-19T15:07:14.150983+0000","172.21.15.19:0/2793980440":"2024-03-19T15:06:38.688405+0000","172.21.15.19:6800/3264337803":"2024-03-19T15:06:38.688405+0000","172.21.15.19:6801/3264337803":"2024-03-19T15:06:38.688405+0000","172.21.15.19:0/3166705295":"2024-03-19T15:06:38.688405+0000","172.21.15.19:6800/2776008962":"2024-03-19T15:07:14.150983+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T15:15:28.699 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:28.699+0000 7fcbd1600640 1 -- 172.21.15.19:0/2087299541 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fcbb00a2880 msgr2=0x7fcbb00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:28.700 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:28.699+0000 7fcbd1600640 1 --2- 172.21.15.19:0/2087299541 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fcbb00a2880 0x7fcbb00a4d40 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fcbb402c2e0 tx=0x7fcbb4005fd0 comp rx=0 tx=0).stop 2024-03-18T15:15:28.700 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:28.699+0000 7fcbd1600640 1 -- 172.21.15.19:0/2087299541 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcbcc0ffee0 msgr2=0x7fcbcc199ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:28.700 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:28.699+0000 7fcbd1600640 1 --2- 172.21.15.19:0/2087299541 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcbcc0ffee0 0x7fcbcc199ad0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fcbbc00eb90 tx=0x7fcbbc00c6a0 comp rx=0 tx=0).stop 2024-03-18T15:15:28.700 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:28.699+0000 7fcbd1600640 1 -- 172.21.15.19:0/2087299541 shutdown_connections 2024-03-18T15:15:28.700 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:28.699+0000 7fcbd1600640 1 --2- 172.21.15.19:0/2087299541 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcbcc1011e0 0x7fcbcc1a1080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:28.700 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:28.699+0000 7fcbd1600640 1 --2- 172.21.15.19:0/2087299541 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fcbcc100840 0x7fcbcc19a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:28.700 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:28.699+0000 7fcbd1600640 1 --2- 172.21.15.19:0/2087299541 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fcbb00a2880 0x7fcbb00a4d40 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:28.700 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:28.699+0000 7fcbd1600640 1 --2- 172.21.15.19:0/2087299541 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcbcc0ffee0 0x7fcbcc199ad0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:28.700 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:28.699+0000 7fcbd1600640 1 -- 172.21.15.19:0/2087299541 >> 172.21.15.19:0/2087299541 conn(0x7fcbcc0fba60 msgr2=0x7fcbcc0fd560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:28.700 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:28.700+0000 7fcbd1600640 1 -- 172.21.15.19:0/2087299541 shutdown_connections 2024-03-18T15:15:28.700 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:28.700+0000 7fcbd1600640 1 -- 172.21.15.19:0/2087299541 wait complete. 2024-03-18T15:15:28.722 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T15:09:56.843349+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T15:15:08.941047+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T15:15:10.044671+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-18T15:15:13.621916+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-18T15:15:28.723 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T15:15:28.950 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:28.949+0000 7f4d27000640 1 -- 172.21.15.19:0/3802134932 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4d20108ff0 msgr2=0x7f4d20109470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:28.950 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:28.949+0000 7f4d27000640 1 --2- 172.21.15.19:0/3802134932 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4d20108ff0 0x7f4d20109470 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f4d10009ca0 tx=0x7f4d1002c100 comp rx=0 tx=0).stop 2024-03-18T15:15:28.950 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:28.949+0000 7f4d27000640 1 -- 172.21.15.19:0/3802134932 shutdown_connections 2024-03-18T15:15:28.950 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:28.949+0000 7f4d27000640 1 --2- 172.21.15.19:0/3802134932 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4d201099b0 0x7f4d20110240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:28.950 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:28.949+0000 7f4d27000640 1 --2- 172.21.15.19:0/3802134932 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4d20108ff0 0x7f4d20109470 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:28.950 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:28.949+0000 7f4d27000640 1 --2- 172.21.15.19:0/3802134932 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4d20107e00 0x7f4d20108220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:28.950 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:28.949+0000 7f4d27000640 1 -- 172.21.15.19:0/3802134932 >> 172.21.15.19:0/3802134932 conn(0x7f4d20076da0 msgr2=0x7f4d200791e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:28.950 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:28.949+0000 7f4d27000640 1 -- 172.21.15.19:0/3802134932 shutdown_connections 2024-03-18T15:15:28.950 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:28.949+0000 7f4d27000640 1 -- 172.21.15.19:0/3802134932 wait complete. 2024-03-18T15:15:28.953 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:28.950+0000 7f4d27000640 1 Processor -- start 2024-03-18T15:15:28.953 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:28.951+0000 7f4d27000640 1 -- start start 2024-03-18T15:15:28.953 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:28.951+0000 7f4d27000640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4d20107e00 0x7f4d2019e130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:28.953 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:28.951+0000 7f4d27000640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4d20108ff0 0x7f4d2019e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:28.953 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:28.951+0000 7f4d27000640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4d201099b0 0x7f4d201a56e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:28.953 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:28.951+0000 7f4d27000640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f4d20113a60 con 0x7f4d201099b0 2024-03-18T15:15:28.953 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:28.951+0000 7f4d27000640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4d201138e0 con 0x7f4d20107e00 2024-03-18T15:15:28.953 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:28.951+0000 7f4d27000640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f4d20113be0 con 0x7f4d20108ff0 2024-03-18T15:15:28.953 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:28.951+0000 7f4d1fe00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4d20107e00 0x7f4d2019e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:28.953 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:28.951+0000 7f4d1fe00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4d20107e00 0x7f4d2019e130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:33896/0 (socket says 172.21.15.19:33896) 2024-03-18T15:15:28.954 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:28.951+0000 7f4d1fe00640 1 -- 172.21.15.19:0/2744501659 learned_addr learned my addr 172.21.15.19:0/2744501659 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:15:28.954 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:28.951+0000 7f4d25000640 1 --2- 172.21.15.19:0/2744501659 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4d201099b0 0x7f4d201a56e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:28.954 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:28.951+0000 7f4d1f400640 1 --2- 172.21.15.19:0/2744501659 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4d20108ff0 0x7f4d2019e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:28.954 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:28.952+0000 7f4d1fe00640 1 -- 172.21.15.19:0/2744501659 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4d20108ff0 msgr2=0x7f4d2019e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:28.954 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:28.952+0000 7f4d1fe00640 1 --2- 172.21.15.19:0/2744501659 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4d20108ff0 0x7f4d2019e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:28.954 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:28.952+0000 7f4d1fe00640 1 -- 172.21.15.19:0/2744501659 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4d201099b0 msgr2=0x7f4d201a56e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:28.954 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:28.952+0000 7f4d1fe00640 1 --2- 172.21.15.19:0/2744501659 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4d201099b0 0x7f4d201a56e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:28.954 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:28.952+0000 7f4d1fe00640 1 -- 172.21.15.19:0/2744501659 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d201a5d50 con 0x7f4d20107e00 2024-03-18T15:15:28.954 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:28.952+0000 7f4d1f400640 1 --2- 172.21.15.19:0/2744501659 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4d20108ff0 0x7f4d2019e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:15:28.954 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:28.952+0000 7f4d1fe00640 1 --2- 172.21.15.19:0/2744501659 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4d20107e00 0x7f4d2019e130 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f4d0800e9c0 tx=0x7f4d0800edc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:28.954 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:28.952+0000 7f4d1cc00640 1 -- 172.21.15.19:0/2744501659 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d0800ccc0 con 0x7f4d20107e00 2024-03-18T15:15:28.954 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:28.952+0000 7f4d1cc00640 1 -- 172.21.15.19:0/2744501659 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d080035f0 con 0x7f4d20107e00 2024-03-18T15:15:28.954 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:28.952+0000 7f4d27000640 1 -- 172.21.15.19:0/2744501659 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d201a5fe0 con 0x7f4d20107e00 2024-03-18T15:15:28.955 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:28.952+0000 7f4d1cc00640 1 -- 172.21.15.19:0/2744501659 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d0800ce60 con 0x7f4d20107e00 2024-03-18T15:15:28.955 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:28.952+0000 7f4d27000640 1 -- 172.21.15.19:0/2744501659 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d201a64f0 con 0x7f4d20107e00 2024-03-18T15:15:28.955 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:28.953+0000 7f4d27000640 1 -- 172.21.15.19:0/2744501659 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d2010f1a0 con 0x7f4d20107e00 2024-03-18T15:15:28.955 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:28.955+0000 7f4d1cc00640 1 -- 172.21.15.19:0/2744501659 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f4d080037a0 con 0x7f4d20107e00 2024-03-18T15:15:28.956 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:28.956+0000 7f4d1cc00640 1 --2- 172.21.15.19:0/2744501659 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f4d040a2880 0x7f4d040a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:28.957 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:28.957+0000 7f4d1cc00640 1 -- 172.21.15.19:0/2744501659 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f4d080cefe0 con 0x7f4d20107e00 2024-03-18T15:15:28.957 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:28.957+0000 7f4d1f400640 1 --2- 172.21.15.19:0/2744501659 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f4d040a2880 0x7f4d040a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:28.957 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:28.958+0000 7f4d1f400640 1 --2- 172.21.15.19:0/2744501659 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f4d040a2880 0x7f4d040a4d40 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f4d1002c710 tx=0x7f4d10005ef0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:28.960 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:28.961+0000 7f4d1cc00640 1 -- 172.21.15.19:0/2744501659 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4d0809bde0 con 0x7f4d20107e00 2024-03-18T15:15:29.166 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.166+0000 7f4d27000640 1 -- 172.21.15.19:0/2744501659 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f4d20062410 con 0x7f4d20107e00 2024-03-18T15:15:29.166 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.166+0000 7f4d1cc00640 1 -- 172.21.15.19:0/2744501659 <== mon.2 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) ==== 93+0+10 (secure 0 0 0) 0x7f4d080860f0 con 0x7f4d20107e00 2024-03-18T15:15:29.167 INFO:teuthology.orchestra.run.smithi019.stdout:pg_num: 1 2024-03-18T15:15:29.170 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.169+0000 7f4d27000640 1 -- 172.21.15.19:0/2744501659 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f4d040a2880 msgr2=0x7f4d040a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:29.170 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.170+0000 7f4d27000640 1 --2- 172.21.15.19:0/2744501659 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f4d040a2880 0x7f4d040a4d40 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f4d1002c710 tx=0x7f4d10005ef0 comp rx=0 tx=0).stop 2024-03-18T15:15:29.170 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.170+0000 7f4d27000640 1 -- 172.21.15.19:0/2744501659 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4d20107e00 msgr2=0x7f4d2019e130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:29.170 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.170+0000 7f4d27000640 1 --2- 172.21.15.19:0/2744501659 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4d20107e00 0x7f4d2019e130 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f4d0800e9c0 tx=0x7f4d0800edc0 comp rx=0 tx=0).stop 2024-03-18T15:15:29.170 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.170+0000 7f4d27000640 1 -- 172.21.15.19:0/2744501659 shutdown_connections 2024-03-18T15:15:29.170 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.170+0000 7f4d27000640 1 --2- 172.21.15.19:0/2744501659 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4d201099b0 0x7f4d201a56e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:29.170 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.170+0000 7f4d27000640 1 --2- 172.21.15.19:0/2744501659 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4d20108ff0 0x7f4d2019e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:29.171 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.170+0000 7f4d27000640 1 --2- 172.21.15.19:0/2744501659 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f4d040a2880 0x7f4d040a4d40 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:29.171 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.170+0000 7f4d27000640 1 --2- 172.21.15.19:0/2744501659 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4d20107e00 0x7f4d2019e130 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:29.171 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.170+0000 7f4d27000640 1 -- 172.21.15.19:0/2744501659 >> 172.21.15.19:0/2744501659 conn(0x7f4d20076da0 msgr2=0x7f4d2010c370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:29.171 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.170+0000 7f4d27000640 1 -- 172.21.15.19:0/2744501659 shutdown_connections 2024-03-18T15:15:29.171 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.170+0000 7f4d27000640 1 -- 172.21.15.19:0/2744501659 wait complete. 2024-03-18T15:15:29.190 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T15:15:29.419 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.419+0000 7f6b4d600640 1 -- 172.21.15.19:0/3137322158 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6b481098b0 msgr2=0x7f6b48110140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:29.419 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.419+0000 7f6b4d600640 1 --2- 172.21.15.19:0/3137322158 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6b481098b0 0x7f6b48110140 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f6b40009900 tx=0x7f6b4002bc40 comp rx=0 tx=0).stop 2024-03-18T15:15:29.419 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.419+0000 7f6b4d600640 1 -- 172.21.15.19:0/3137322158 shutdown_connections 2024-03-18T15:15:29.419 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.419+0000 7f6b4d600640 1 --2- 172.21.15.19:0/3137322158 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6b481098b0 0x7f6b48110140 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:29.419 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.419+0000 7f6b4d600640 1 --2- 172.21.15.19:0/3137322158 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6b48108ef0 0x7f6b48109370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:29.419 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.419+0000 7f6b4d600640 1 --2- 172.21.15.19:0/3137322158 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6b48107d00 0x7f6b48108120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:29.419 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.419+0000 7f6b4d600640 1 -- 172.21.15.19:0/3137322158 >> 172.21.15.19:0/3137322158 conn(0x7f6b48076cb0 msgr2=0x7f6b480790d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:29.420 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.419+0000 7f6b4d600640 1 -- 172.21.15.19:0/3137322158 shutdown_connections 2024-03-18T15:15:29.420 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.419+0000 7f6b4d600640 1 -- 172.21.15.19:0/3137322158 wait complete. 2024-03-18T15:15:29.421 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.420+0000 7f6b4d600640 1 Processor -- start 2024-03-18T15:15:29.421 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.420+0000 7f6b4d600640 1 -- start start 2024-03-18T15:15:29.421 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.420+0000 7f6b4d600640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6b48107d00 0x7f6b4819df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:29.421 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.420+0000 7f6b4d600640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6b48108ef0 0x7f6b4819e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:29.421 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.420+0000 7f6b4d600640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6b481098b0 0x7f6b481a54e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:29.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.420+0000 7f6b4d600640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f6b481139b0 con 0x7f6b48108ef0 2024-03-18T15:15:29.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.420+0000 7f6b4d600640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6b48113830 con 0x7f6b48107d00 2024-03-18T15:15:29.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.420+0000 7f6b4d600640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f6b48113b30 con 0x7f6b481098b0 2024-03-18T15:15:29.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.421+0000 7f6b47e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6b48107d00 0x7f6b4819df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:29.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.421+0000 7f6b47e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6b48107d00 0x7f6b4819df30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:33914/0 (socket says 172.21.15.19:33914) 2024-03-18T15:15:29.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.421+0000 7f6b47e00640 1 -- 172.21.15.19:0/3251282299 learned_addr learned my addr 172.21.15.19:0/3251282299 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:15:29.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.421+0000 7f6b3f400640 1 --2- 172.21.15.19:0/3251282299 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6b48108ef0 0x7f6b4819e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:29.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.421+0000 7f6b4cc00640 1 --2- 172.21.15.19:0/3251282299 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6b481098b0 0x7f6b481a54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:29.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.421+0000 7f6b47e00640 1 -- 172.21.15.19:0/3251282299 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6b481098b0 msgr2=0x7f6b481a54e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:29.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.421+0000 7f6b47e00640 1 --2- 172.21.15.19:0/3251282299 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6b481098b0 0x7f6b481a54e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:29.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.421+0000 7f6b47e00640 1 -- 172.21.15.19:0/3251282299 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6b48108ef0 msgr2=0x7f6b4819e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:29.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.421+0000 7f6b47e00640 1 --2- 172.21.15.19:0/3251282299 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6b48108ef0 0x7f6b4819e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:29.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.421+0000 7f6b47e00640 1 -- 172.21.15.19:0/3251282299 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b481a5b50 con 0x7f6b48107d00 2024-03-18T15:15:29.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.421+0000 7f6b3f400640 1 --2- 172.21.15.19:0/3251282299 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6b48108ef0 0x7f6b4819e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T15:15:29.423 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.421+0000 7f6b47e00640 1 --2- 172.21.15.19:0/3251282299 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6b48107d00 0x7f6b4819df30 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f6b3800ebc0 tx=0x7f6b3800c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:29.423 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.422+0000 7f6b45600640 1 -- 172.21.15.19:0/3251282299 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b38003b70 con 0x7f6b48107d00 2024-03-18T15:15:29.423 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.422+0000 7f6b45600640 1 -- 172.21.15.19:0/3251282299 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b38003d10 con 0x7f6b48107d00 2024-03-18T15:15:29.423 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.422+0000 7f6b45600640 1 -- 172.21.15.19:0/3251282299 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b38019980 con 0x7f6b48107d00 2024-03-18T15:15:29.424 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.422+0000 7f6b4d600640 1 -- 172.21.15.19:0/3251282299 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b481a5e40 con 0x7f6b48107d00 2024-03-18T15:15:29.424 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.422+0000 7f6b4d600640 1 -- 172.21.15.19:0/3251282299 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b481a6250 con 0x7f6b48107d00 2024-03-18T15:15:29.424 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.423+0000 7f6b4d600640 1 -- 172.21.15.19:0/3251282299 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b48068940 con 0x7f6b48107d00 2024-03-18T15:15:29.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.425+0000 7f6b45600640 1 -- 172.21.15.19:0/3251282299 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f6b38019b20 con 0x7f6b48107d00 2024-03-18T15:15:29.426 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.426+0000 7f6b45600640 1 --2- 172.21.15.19:0/3251282299 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f6b2c0a2880 0x7f6b2c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:29.426 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.426+0000 7f6b45600640 1 -- 172.21.15.19:0/3251282299 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f6b380cf510 con 0x7f6b48107d00 2024-03-18T15:15:29.426 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.426+0000 7f6b3f400640 1 --2- 172.21.15.19:0/3251282299 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f6b2c0a2880 0x7f6b2c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:29.427 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.427+0000 7f6b3f400640 1 --2- 172.21.15.19:0/3251282299 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f6b2c0a2880 0x7f6b2c0a4d40 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f6b4819f440 tx=0x7f6b30005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:29.430 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.430+0000 7f6b45600640 1 -- 172.21.15.19:0/3251282299 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6b3809c310 con 0x7f6b48107d00 2024-03-18T15:15:29.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:29 smithi019 ceph-mon[31522]: pgmap v326: 129 pgs: 129 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-03-18T15:15:29.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:29 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/2087299541' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:15:29.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:29 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/2744501659' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T15:15:29.632 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.632+0000 7f6b4d600640 1 -- 172.21.15.19:0/3251282299 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f6b4810efc0 con 0x7f6b48107d00 2024-03-18T15:15:29.633 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.633+0000 7f6b45600640 1 -- 172.21.15.19:0/3251282299 <== mon.2 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) ==== 104+0+11 (secure 0 0 0) 0x7f6b4810efc0 con 0x7f6b48107d00 2024-03-18T15:15:29.633 INFO:teuthology.orchestra.run.smithi019.stdout:pg_num: 32 2024-03-18T15:15:29.636 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.636+0000 7f6b4d600640 1 -- 172.21.15.19:0/3251282299 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f6b2c0a2880 msgr2=0x7f6b2c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:29.636 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.636+0000 7f6b4d600640 1 --2- 172.21.15.19:0/3251282299 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f6b2c0a2880 0x7f6b2c0a4d40 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f6b4819f440 tx=0x7f6b30005f90 comp rx=0 tx=0).stop 2024-03-18T15:15:29.636 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.636+0000 7f6b4d600640 1 -- 172.21.15.19:0/3251282299 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6b48107d00 msgr2=0x7f6b4819df30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:29.636 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.636+0000 7f6b4d600640 1 --2- 172.21.15.19:0/3251282299 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6b48107d00 0x7f6b4819df30 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f6b3800ebc0 tx=0x7f6b3800c6a0 comp rx=0 tx=0).stop 2024-03-18T15:15:29.636 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.636+0000 7f6b4d600640 1 -- 172.21.15.19:0/3251282299 shutdown_connections 2024-03-18T15:15:29.636 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.636+0000 7f6b4d600640 1 --2- 172.21.15.19:0/3251282299 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6b481098b0 0x7f6b481a54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:29.636 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.636+0000 7f6b4d600640 1 --2- 172.21.15.19:0/3251282299 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6b48108ef0 0x7f6b4819e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:29.636 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.636+0000 7f6b4d600640 1 --2- 172.21.15.19:0/3251282299 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f6b2c0a2880 0x7f6b2c0a4d40 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:29.637 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.636+0000 7f6b4d600640 1 --2- 172.21.15.19:0/3251282299 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6b48107d00 0x7f6b4819df30 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:29.637 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.636+0000 7f6b4d600640 1 -- 172.21.15.19:0/3251282299 >> 172.21.15.19:0/3251282299 conn(0x7f6b48076cb0 msgr2=0x7f6b4810c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:29.637 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.636+0000 7f6b4d600640 1 -- 172.21.15.19:0/3251282299 shutdown_connections 2024-03-18T15:15:29.637 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.636+0000 7f6b4d600640 1 -- 172.21.15.19:0/3251282299 wait complete. 2024-03-18T15:15:29.655 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T15:15:29.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:29 smithi104 ceph-mon[36070]: pgmap v326: 129 pgs: 129 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-03-18T15:15:29.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:29 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/2087299541' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:15:29.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:29 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/2744501659' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T15:15:29.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:29 smithi033 ceph-mon[36648]: pgmap v326: 129 pgs: 129 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-03-18T15:15:29.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:29 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/2087299541' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:15:29.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:29 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/2744501659' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T15:15:29.883 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.882+0000 7f0e77200640 1 -- 172.21.15.19:0/1805156309 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0e70101810 msgr2=0x7f0e70101c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:29.883 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.882+0000 7f0e77200640 1 --2- 172.21.15.19:0/1805156309 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0e70101810 0x7f0e70101c30 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f0e60009ca0 tx=0x7f0e6002c0f0 comp rx=0 tx=0).stop 2024-03-18T15:15:29.883 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.883+0000 7f0e77200640 1 -- 172.21.15.19:0/1805156309 shutdown_connections 2024-03-18T15:15:29.883 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.883+0000 7f0e77200640 1 --2- 172.21.15.19:0/1805156309 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0e701033c0 0x7f0e70109c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:29.883 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.883+0000 7f0e77200640 1 --2- 172.21.15.19:0/1805156309 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0e70102a00 0x7f0e70102e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:29.883 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.883+0000 7f0e77200640 1 --2- 172.21.15.19:0/1805156309 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0e70101810 0x7f0e70101c30 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:29.883 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.883+0000 7f0e77200640 1 -- 172.21.15.19:0/1805156309 >> 172.21.15.19:0/1805156309 conn(0x7f0e700fcd80 msgr2=0x7f0e700ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:29.884 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.883+0000 7f0e77200640 1 -- 172.21.15.19:0/1805156309 shutdown_connections 2024-03-18T15:15:29.884 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.883+0000 7f0e77200640 1 -- 172.21.15.19:0/1805156309 wait complete. 2024-03-18T15:15:29.884 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.884+0000 7f0e77200640 1 Processor -- start 2024-03-18T15:15:29.884 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.884+0000 7f0e77200640 1 -- start start 2024-03-18T15:15:29.886 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.885+0000 7f0e77200640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0e70101810 0x7f0e70199e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:29.886 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.885+0000 7f0e77200640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0e70102a00 0x7f0e7019a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:29.886 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.885+0000 7f0e77200640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0e701033c0 0x7f0e701a1320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:29.886 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.885+0000 7f0e77200640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f0e7010d460 con 0x7f0e701033c0 2024-03-18T15:15:29.886 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.885+0000 7f0e77200640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f0e7010d2e0 con 0x7f0e70102a00 2024-03-18T15:15:29.886 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.885+0000 7f0e77200640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f0e7010d5e0 con 0x7f0e70101810 2024-03-18T15:15:29.886 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.885+0000 7f0e67400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0e70102a00 0x7f0e7019a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:29.886 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.885+0000 7f0e67400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0e70102a00 0x7f0e7019a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:33924/0 (socket says 172.21.15.19:33924) 2024-03-18T15:15:29.886 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.885+0000 7f0e67400640 1 -- 172.21.15.19:0/397509324 learned_addr learned my addr 172.21.15.19:0/397509324 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:15:29.886 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.885+0000 7f0e75200640 1 --2- 172.21.15.19:0/397509324 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0e701033c0 0x7f0e701a1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:29.886 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.885+0000 7f0e6fe00640 1 --2- 172.21.15.19:0/397509324 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0e70101810 0x7f0e70199e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:29.886 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.886+0000 7f0e67400640 1 -- 172.21.15.19:0/397509324 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0e70101810 msgr2=0x7f0e70199e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:29.886 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.886+0000 7f0e67400640 1 --2- 172.21.15.19:0/397509324 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0e70101810 0x7f0e70199e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:29.887 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.886+0000 7f0e67400640 1 -- 172.21.15.19:0/397509324 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0e701033c0 msgr2=0x7f0e701a1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:29.887 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.886+0000 7f0e67400640 1 --2- 172.21.15.19:0/397509324 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0e701033c0 0x7f0e701a1320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:29.887 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.886+0000 7f0e67400640 1 -- 172.21.15.19:0/397509324 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e701a19c0 con 0x7f0e70102a00 2024-03-18T15:15:29.887 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.886+0000 7f0e67400640 1 --2- 172.21.15.19:0/397509324 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0e70102a00 0x7f0e7019a340 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f0e5800d950 tx=0x7f0e5800de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:29.888 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.886+0000 7f0e6d600640 1 -- 172.21.15.19:0/397509324 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e58017030 con 0x7f0e70102a00 2024-03-18T15:15:29.888 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.886+0000 7f0e6d600640 1 -- 172.21.15.19:0/397509324 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0e58004020 con 0x7f0e70102a00 2024-03-18T15:15:29.888 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.886+0000 7f0e6d600640 1 -- 172.21.15.19:0/397509324 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e580047a0 con 0x7f0e70102a00 2024-03-18T15:15:29.888 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.886+0000 7f0e77200640 1 -- 172.21.15.19:0/397509324 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e701a1cb0 con 0x7f0e70102a00 2024-03-18T15:15:29.888 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.886+0000 7f0e77200640 1 -- 172.21.15.19:0/397509324 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e701a21d0 con 0x7f0e70102a00 2024-03-18T15:15:29.888 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.888+0000 7f0e77200640 1 -- 172.21.15.19:0/397509324 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e701a2400 con 0x7f0e70102a00 2024-03-18T15:15:29.889 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.889+0000 7f0e6d600640 1 -- 172.21.15.19:0/397509324 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f0e580041d0 con 0x7f0e70102a00 2024-03-18T15:15:29.890 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.890+0000 7f0e6d600640 1 --2- 172.21.15.19:0/397509324 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f0e540a2880 0x7f0e540a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:29.890 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.890+0000 7f0e6d600640 1 -- 172.21.15.19:0/397509324 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f0e580ce670 con 0x7f0e70102a00 2024-03-18T15:15:29.891 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.891+0000 7f0e6fe00640 1 --2- 172.21.15.19:0/397509324 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f0e540a2880 0x7f0e540a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:29.891 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.892+0000 7f0e6fe00640 1 --2- 172.21.15.19:0/397509324 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f0e540a2880 0x7f0e540a4d40 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f0e6002c630 tx=0x7f0e60009360 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:29.894 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:29.894+0000 7f0e6d600640 1 -- 172.21.15.19:0/397509324 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0e5809b4d0 con 0x7f0e70102a00 2024-03-18T15:15:30.097 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.097+0000 7f0e77200640 1 -- 172.21.15.19:0/397509324 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f0e701a25d0 con 0x7f0e70102a00 2024-03-18T15:15:30.097 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.097+0000 7f0e6d600640 1 -- 172.21.15.19:0/397509324 <== mon.2 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) ==== 100+0+11 (secure 0 0 0) 0x7f0e5809add0 con 0x7f0e70102a00 2024-03-18T15:15:30.098 INFO:teuthology.orchestra.run.smithi019.stdout:pg_num: 64 2024-03-18T15:15:30.101 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.101+0000 7f0e77200640 1 -- 172.21.15.19:0/397509324 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f0e540a2880 msgr2=0x7f0e540a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:30.101 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.101+0000 7f0e77200640 1 --2- 172.21.15.19:0/397509324 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f0e540a2880 0x7f0e540a4d40 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f0e6002c630 tx=0x7f0e60009360 comp rx=0 tx=0).stop 2024-03-18T15:15:30.101 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.101+0000 7f0e77200640 1 -- 172.21.15.19:0/397509324 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0e70102a00 msgr2=0x7f0e7019a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:30.101 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.101+0000 7f0e77200640 1 --2- 172.21.15.19:0/397509324 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0e70102a00 0x7f0e7019a340 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f0e5800d950 tx=0x7f0e5800de10 comp rx=0 tx=0).stop 2024-03-18T15:15:30.101 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.101+0000 7f0e77200640 1 -- 172.21.15.19:0/397509324 shutdown_connections 2024-03-18T15:15:30.101 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.101+0000 7f0e77200640 1 --2- 172.21.15.19:0/397509324 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0e701033c0 0x7f0e701a1320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:30.102 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.101+0000 7f0e77200640 1 --2- 172.21.15.19:0/397509324 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f0e540a2880 0x7f0e540a4d40 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:30.102 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.101+0000 7f0e77200640 1 --2- 172.21.15.19:0/397509324 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0e70102a00 0x7f0e7019a340 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:30.102 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.101+0000 7f0e77200640 1 --2- 172.21.15.19:0/397509324 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0e70101810 0x7f0e70199e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:30.102 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.101+0000 7f0e77200640 1 -- 172.21.15.19:0/397509324 >> 172.21.15.19:0/397509324 conn(0x7f0e700fcd80 msgr2=0x7f0e70108220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:30.102 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.101+0000 7f0e77200640 1 -- 172.21.15.19:0/397509324 shutdown_connections 2024-03-18T15:15:30.102 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.101+0000 7f0e77200640 1 -- 172.21.15.19:0/397509324 wait complete. 2024-03-18T15:15:30.121 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-18T15:15:30.343 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.342+0000 7f6ca6000640 1 -- 172.21.15.19:0/2221085485 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6ca0102140 msgr2=0x7f6ca0104530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:30.343 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.342+0000 7f6ca6000640 1 --2- 172.21.15.19:0/2221085485 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6ca0102140 0x7f6ca0104530 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f6c90009900 tx=0x7f6c9002bc50 comp rx=0 tx=0).stop 2024-03-18T15:15:30.343 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.343+0000 7f6ca6000640 1 -- 172.21.15.19:0/2221085485 shutdown_connections 2024-03-18T15:15:30.343 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.343+0000 7f6ca6000640 1 --2- 172.21.15.19:0/2221085485 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6ca01073a0 0x7f6ca01097f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:30.343 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.343+0000 7f6ca6000640 1 --2- 172.21.15.19:0/2221085485 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6ca0104a70 0x7f6ca0106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:30.343 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.343+0000 7f6ca6000640 1 --2- 172.21.15.19:0/2221085485 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6ca0102140 0x7f6ca0104530 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:30.344 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.343+0000 7f6ca6000640 1 -- 172.21.15.19:0/2221085485 >> 172.21.15.19:0/2221085485 conn(0x7f6ca00fbb50 msgr2=0x7f6ca00fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:30.344 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.343+0000 7f6ca6000640 1 -- 172.21.15.19:0/2221085485 shutdown_connections 2024-03-18T15:15:30.344 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.343+0000 7f6ca6000640 1 -- 172.21.15.19:0/2221085485 wait complete. 2024-03-18T15:15:30.344 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.344+0000 7f6ca6000640 1 Processor -- start 2024-03-18T15:15:30.344 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.344+0000 7f6ca6000640 1 -- start start 2024-03-18T15:15:30.345 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.345+0000 7f6ca6000640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6ca0102140 0x7f6ca0199c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:30.345 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.345+0000 7f6ca6000640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6ca0104a70 0x7f6ca019a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:30.346 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.345+0000 7f6ca6000640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6ca01073a0 0x7f6ca01a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:30.346 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.345+0000 7f6ca6000640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f6ca010d3b0 con 0x7f6ca0104a70 2024-03-18T15:15:30.346 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.345+0000 7f6ca6000640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6ca010d230 con 0x7f6ca01073a0 2024-03-18T15:15:30.346 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.345+0000 7f6ca6000640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f6ca010d530 con 0x7f6ca0102140 2024-03-18T15:15:30.346 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.345+0000 7f6c9fe00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6ca0104a70 0x7f6ca019a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:30.346 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.345+0000 7f6ca5600640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6ca01073a0 0x7f6ca01a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:30.346 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.345+0000 7f6c9fe00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6ca0104a70 0x7f6ca019a160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:51566/0 (socket says 172.21.15.19:51566) 2024-03-18T15:15:30.346 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.345+0000 7f6c9fe00640 1 -- 172.21.15.19:0/10578497 learned_addr learned my addr 172.21.15.19:0/10578497 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:15:30.346 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.345+0000 7f6ca4c00640 1 --2- 172.21.15.19:0/10578497 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6ca0102140 0x7f6ca0199c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:30.346 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.346+0000 7f6c9fe00640 1 -- 172.21.15.19:0/10578497 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6ca0102140 msgr2=0x7f6ca0199c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:30.346 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.346+0000 7f6c9fe00640 1 --2- 172.21.15.19:0/10578497 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6ca0102140 0x7f6ca0199c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:30.346 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.346+0000 7f6c9fe00640 1 -- 172.21.15.19:0/10578497 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6ca01073a0 msgr2=0x7f6ca01a1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:30.346 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.346+0000 7f6c9fe00640 1 --2- 172.21.15.19:0/10578497 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6ca01073a0 0x7f6ca01a1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:30.347 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.346+0000 7f6c9fe00640 1 -- 172.21.15.19:0/10578497 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ca01a1830 con 0x7f6ca0104a70 2024-03-18T15:15:30.347 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.346+0000 7f6ca5600640 1 --2- 172.21.15.19:0/10578497 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6ca01073a0 0x7f6ca01a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:15:30.347 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.346+0000 7f6c9fe00640 1 --2- 172.21.15.19:0/10578497 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6ca0104a70 0x7f6ca019a160 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f6c8c00e9c0 tx=0x7f6c8c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:30.347 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.346+0000 7f6c9d600640 1 -- 172.21.15.19:0/10578497 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c8c00cd80 con 0x7f6ca0104a70 2024-03-18T15:15:30.348 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.346+0000 7f6c9d600640 1 -- 172.21.15.19:0/10578497 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6c8c005500 con 0x7f6ca0104a70 2024-03-18T15:15:30.348 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.346+0000 7f6ca6000640 1 -- 172.21.15.19:0/10578497 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ca01a1b20 con 0x7f6ca0104a70 2024-03-18T15:15:30.348 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.346+0000 7f6c9d600640 1 -- 172.21.15.19:0/10578497 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c8c010690 con 0x7f6ca0104a70 2024-03-18T15:15:30.348 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.346+0000 7f6ca6000640 1 -- 172.21.15.19:0/10578497 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ca01a2090 con 0x7f6ca0104a70 2024-03-18T15:15:30.348 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.348+0000 7f6ca6000640 1 -- 172.21.15.19:0/10578497 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ca0108ef0 con 0x7f6ca0104a70 2024-03-18T15:15:30.350 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.350+0000 7f6c9d600640 1 -- 172.21.15.19:0/10578497 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f6c8c0046c0 con 0x7f6ca0104a70 2024-03-18T15:15:30.352 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.352+0000 7f6c9d600640 1 --2- 172.21.15.19:0/10578497 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f6c840a2880 0x7f6c840a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:30.352 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.352+0000 7f6c9d600640 1 -- 172.21.15.19:0/10578497 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f6c8c0cf070 con 0x7f6ca0104a70 2024-03-18T15:15:30.352 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.352+0000 7f6ca4c00640 1 --2- 172.21.15.19:0/10578497 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f6c840a2880 0x7f6c840a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:30.353 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.353+0000 7f6ca4c00640 1 --2- 172.21.15.19:0/10578497 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f6c840a2880 0x7f6c840a4d40 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f6c9002c260 tx=0x7f6c90000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:30.356 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.356+0000 7f6c9d600640 1 -- 172.21.15.19:0/10578497 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6c8c09be70 con 0x7f6ca0104a70 2024-03-18T15:15:30.567 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.567+0000 7f6ca6000640 1 -- 172.21.15.19:0/10578497 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f6ca00623e0 con 0x7f6ca0104a70 2024-03-18T15:15:30.568 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.568+0000 7f6c9d600640 1 -- 172.21.15.19:0/10578497 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v89) ==== 103+0+11 (secure 0 0 0) 0x7f6c8c09b770 con 0x7f6ca0104a70 2024-03-18T15:15:30.568 INFO:teuthology.orchestra.run.smithi019.stdout:pg_num: 32 2024-03-18T15:15:30.571 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.571+0000 7f6ca6000640 1 -- 172.21.15.19:0/10578497 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f6c840a2880 msgr2=0x7f6c840a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:30.572 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.571+0000 7f6ca6000640 1 --2- 172.21.15.19:0/10578497 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f6c840a2880 0x7f6c840a4d40 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f6c9002c260 tx=0x7f6c90000bc0 comp rx=0 tx=0).stop 2024-03-18T15:15:30.572 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.571+0000 7f6ca6000640 1 -- 172.21.15.19:0/10578497 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6ca0104a70 msgr2=0x7f6ca019a160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:30.572 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.571+0000 7f6ca6000640 1 --2- 172.21.15.19:0/10578497 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6ca0104a70 0x7f6ca019a160 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f6c8c00e9c0 tx=0x7f6c8c00ee80 comp rx=0 tx=0).stop 2024-03-18T15:15:30.572 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.571+0000 7f6ca6000640 1 -- 172.21.15.19:0/10578497 shutdown_connections 2024-03-18T15:15:30.572 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.571+0000 7f6ca6000640 1 --2- 172.21.15.19:0/10578497 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f6c840a2880 0x7f6c840a4d40 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:30.572 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.571+0000 7f6ca6000640 1 --2- 172.21.15.19:0/10578497 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6ca01073a0 0x7f6ca01a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:30.572 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.571+0000 7f6ca6000640 1 --2- 172.21.15.19:0/10578497 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6ca0104a70 0x7f6ca019a160 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:30.572 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.571+0000 7f6ca6000640 1 --2- 172.21.15.19:0/10578497 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6ca0102140 0x7f6ca0199c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:30.572 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.571+0000 7f6ca6000640 1 -- 172.21.15.19:0/10578497 >> 172.21.15.19:0/10578497 conn(0x7f6ca00fbb50 msgr2=0x7f6ca01018b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:30.572 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.571+0000 7f6ca6000640 1 -- 172.21.15.19:0/10578497 shutdown_connections 2024-03-18T15:15:30.572 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.571+0000 7f6ca6000640 1 -- 172.21.15.19:0/10578497 wait complete. 2024-03-18T15:15:30.591 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-18T15:15:30.670 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:30 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/3251282299' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T15:15:30.670 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:30 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/397509324' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T15:15:30.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:30 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/3251282299' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T15:15:30.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:30 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/397509324' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T15:15:30.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:30 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/3251282299' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T15:15:30.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:30 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/397509324' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T15:15:30.811 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.811+0000 7f3003a00640 1 -- 172.21.15.19:0/4268354347 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2ffc1022b0 msgr2=0x7f2ffc108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:30.811 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.811+0000 7f3003a00640 1 --2- 172.21.15.19:0/4268354347 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2ffc1022b0 0x7f2ffc108b40 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f2ff4009900 tx=0x7f2ff402bc40 comp rx=0 tx=0).stop 2024-03-18T15:15:30.811 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.811+0000 7f3003a00640 1 -- 172.21.15.19:0/4268354347 shutdown_connections 2024-03-18T15:15:30.811 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.811+0000 7f3003a00640 1 --2- 172.21.15.19:0/4268354347 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2ffc1022b0 0x7f2ffc108b40 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:30.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.811+0000 7f3003a00640 1 --2- 172.21.15.19:0/4268354347 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2ffc1018f0 0x7f2ffc101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:30.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.811+0000 7f3003a00640 1 --2- 172.21.15.19:0/4268354347 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2ffc100700 0x7f2ffc100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:30.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.811+0000 7f3003a00640 1 -- 172.21.15.19:0/4268354347 >> 172.21.15.19:0/4268354347 conn(0x7f2ffc0fbc90 msgr2=0x7f2ffc0fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:30.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.811+0000 7f3003a00640 1 -- 172.21.15.19:0/4268354347 shutdown_connections 2024-03-18T15:15:30.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.811+0000 7f3003a00640 1 -- 172.21.15.19:0/4268354347 wait complete. 2024-03-18T15:15:30.814 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.812+0000 7f3003a00640 1 Processor -- start 2024-03-18T15:15:30.814 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.812+0000 7f3003a00640 1 -- start start 2024-03-18T15:15:30.814 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.812+0000 7f3003a00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2ffc100700 0x7f2ffc197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:30.814 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.813+0000 7f3003a00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2ffc1018f0 0x7f2ffc197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:30.814 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.813+0000 7f3003a00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2ffc1022b0 0x7f2ffc19f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:30.814 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.813+0000 7f3003a00640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f2ffc10c360 con 0x7f2ffc1018f0 2024-03-18T15:15:30.814 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.813+0000 7f3003a00640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2ffc10c1e0 con 0x7f2ffc100700 2024-03-18T15:15:30.814 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.813+0000 7f3003a00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f2ffc10c4e0 con 0x7f2ffc1022b0 2024-03-18T15:15:30.814 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.813+0000 7f3001000640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2ffc100700 0x7f2ffc197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:30.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.813+0000 7f3001000640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2ffc100700 0x7f2ffc197a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:33968/0 (socket says 172.21.15.19:33968) 2024-03-18T15:15:30.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.813+0000 7f3001000640 1 -- 172.21.15.19:0/731625546 learned_addr learned my addr 172.21.15.19:0/731625546 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:15:30.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.813+0000 7f2ffbe00640 1 --2- 172.21.15.19:0/731625546 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2ffc1018f0 0x7f2ffc197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:30.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.813+0000 7f3001a00640 1 --2- 172.21.15.19:0/731625546 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2ffc1022b0 0x7f2ffc19f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:30.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.813+0000 7f3001000640 1 -- 172.21.15.19:0/731625546 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2ffc1022b0 msgr2=0x7f2ffc19f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:30.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.813+0000 7f3001000640 1 --2- 172.21.15.19:0/731625546 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2ffc1022b0 0x7f2ffc19f040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:30.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.813+0000 7f3001000640 1 -- 172.21.15.19:0/731625546 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2ffc1018f0 msgr2=0x7f2ffc197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:30.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.813+0000 7f3001000640 1 --2- 172.21.15.19:0/731625546 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2ffc1018f0 0x7f2ffc197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:30.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.813+0000 7f3001000640 1 -- 172.21.15.19:0/731625546 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ffc19f6b0 con 0x7f2ffc100700 2024-03-18T15:15:30.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.813+0000 7f3001a00640 1 --2- 172.21.15.19:0/731625546 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2ffc1022b0 0x7f2ffc19f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T15:15:30.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.813+0000 7f2ffbe00640 1 --2- 172.21.15.19:0/731625546 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2ffc1018f0 0x7f2ffc197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T15:15:30.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.814+0000 7f3001000640 1 --2- 172.21.15.19:0/731625546 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2ffc100700 0x7f2ffc197a90 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f2fec009d20 tx=0x7f2fec00c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:30.816 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.814+0000 7f2ff9600640 1 -- 172.21.15.19:0/731625546 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fec010630 con 0x7f2ffc100700 2024-03-18T15:15:30.816 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.814+0000 7f3003a00640 1 -- 172.21.15.19:0/731625546 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ffc19f9a0 con 0x7f2ffc100700 2024-03-18T15:15:30.816 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.814+0000 7f2ff9600640 1 -- 172.21.15.19:0/731625546 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2fec010cb0 con 0x7f2ffc100700 2024-03-18T15:15:30.816 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.814+0000 7f2ff9600640 1 -- 172.21.15.19:0/731625546 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fec018de0 con 0x7f2ffc100700 2024-03-18T15:15:30.816 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.814+0000 7f3003a00640 1 -- 172.21.15.19:0/731625546 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ffc19fef0 con 0x7f2ffc100700 2024-03-18T15:15:30.816 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.815+0000 7f3003a00640 1 -- 172.21.15.19:0/731625546 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ffc068a20 con 0x7f2ffc100700 2024-03-18T15:15:30.817 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.817+0000 7f2ff9600640 1 -- 172.21.15.19:0/731625546 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f2fec01f020 con 0x7f2ffc100700 2024-03-18T15:15:30.818 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.818+0000 7f2ff9600640 1 --2- 172.21.15.19:0/731625546 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f2fe00a2800 0x7f2fe00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:30.818 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.818+0000 7f2ff9600640 1 -- 172.21.15.19:0/731625546 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f2fec0cf240 con 0x7f2ffc100700 2024-03-18T15:15:30.818 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.819+0000 7f2ffbe00640 1 --2- 172.21.15.19:0/731625546 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f2fe00a2800 0x7f2fe00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:30.819 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.819+0000 7f2ffbe00640 1 --2- 172.21.15.19:0/731625546 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f2fe00a2800 0x7f2fe00a4cc0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f2ffc198fa0 tx=0x7f2fe8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:30.822 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:30.822+0000 7f2ff9600640 1 -- 172.21.15.19:0/731625546 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2fec09c040 con 0x7f2ffc100700 2024-03-18T15:15:31.039 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.039+0000 7f3003a00640 1 -- 172.21.15.19:0/731625546 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f2ffc1a01c0 con 0x7f2ffc100700 2024-03-18T15:15:31.040 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.040+0000 7f2ff9600640 1 -- 172.21.15.19:0/731625546 <== mon.2 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v89) ==== 81+0+591 (secure 0 0 0) 0x7f2ffc1a01c0 con 0x7f2ffc100700 2024-03-18T15:15:31.040 INFO:teuthology.orchestra.run.smithi019.stdout:172.21.15.19:0/3860783274 2024-03-19T15:07:14.150983+0000 2024-03-18T15:15:31.040 INFO:teuthology.orchestra.run.smithi019.stdout:172.21.15.19:0/2737223213 2024-03-19T15:07:14.150983+0000 2024-03-18T15:15:31.040 INFO:teuthology.orchestra.run.smithi019.stdout:172.21.15.19:0/176699025 2024-03-19T15:06:38.688405+0000 2024-03-18T15:15:31.040 INFO:teuthology.orchestra.run.smithi019.stdout:172.21.15.19:0/2316969005 2024-03-19T15:07:14.150983+0000 2024-03-18T15:15:31.040 INFO:teuthology.orchestra.run.smithi019.stdout:172.21.15.19:6801/2776008962 2024-03-19T15:07:14.150983+0000 2024-03-18T15:15:31.040 INFO:teuthology.orchestra.run.smithi019.stdout:172.21.15.19:0/2793980440 2024-03-19T15:06:38.688405+0000 2024-03-18T15:15:31.041 INFO:teuthology.orchestra.run.smithi019.stdout:172.21.15.19:6800/3264337803 2024-03-19T15:06:38.688405+0000 2024-03-18T15:15:31.041 INFO:teuthology.orchestra.run.smithi019.stdout:172.21.15.19:6801/3264337803 2024-03-19T15:06:38.688405+0000 2024-03-18T15:15:31.041 INFO:teuthology.orchestra.run.smithi019.stdout:172.21.15.19:0/3166705295 2024-03-19T15:06:38.688405+0000 2024-03-18T15:15:31.041 INFO:teuthology.orchestra.run.smithi019.stdout:172.21.15.19:6800/2776008962 2024-03-19T15:07:14.150983+0000 2024-03-18T15:15:31.041 INFO:teuthology.orchestra.run.smithi019.stderr:listed 10 entries 2024-03-18T15:15:31.043 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.043+0000 7f3003a00640 1 -- 172.21.15.19:0/731625546 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f2fe00a2800 msgr2=0x7f2fe00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:31.043 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.043+0000 7f3003a00640 1 --2- 172.21.15.19:0/731625546 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f2fe00a2800 0x7f2fe00a4cc0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f2ffc198fa0 tx=0x7f2fe8005f90 comp rx=0 tx=0).stop 2024-03-18T15:15:31.043 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.043+0000 7f3003a00640 1 -- 172.21.15.19:0/731625546 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2ffc100700 msgr2=0x7f2ffc197a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:31.043 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.043+0000 7f3003a00640 1 --2- 172.21.15.19:0/731625546 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2ffc100700 0x7f2ffc197a90 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f2fec009d20 tx=0x7f2fec00c6a0 comp rx=0 tx=0).stop 2024-03-18T15:15:31.044 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.044+0000 7f3003a00640 1 -- 172.21.15.19:0/731625546 shutdown_connections 2024-03-18T15:15:31.044 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.044+0000 7f3003a00640 1 --2- 172.21.15.19:0/731625546 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2ffc1022b0 0x7f2ffc19f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:31.044 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.044+0000 7f3003a00640 1 --2- 172.21.15.19:0/731625546 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2ffc1018f0 0x7f2ffc197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:31.044 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.044+0000 7f3003a00640 1 --2- 172.21.15.19:0/731625546 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f2fe00a2800 0x7f2fe00a4cc0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:31.044 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.044+0000 7f3003a00640 1 --2- 172.21.15.19:0/731625546 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2ffc100700 0x7f2ffc197a90 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:31.044 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.044+0000 7f3003a00640 1 -- 172.21.15.19:0/731625546 >> 172.21.15.19:0/731625546 conn(0x7f2ffc0fbc90 msgr2=0x7f2ffc0fd760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:31.045 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.044+0000 7f3003a00640 1 -- 172.21.15.19:0/731625546 shutdown_connections 2024-03-18T15:15:31.045 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.044+0000 7f3003a00640 1 -- 172.21.15.19:0/731625546 wait complete. 2024-03-18T15:15:31.064 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-03-18T15:15:31.064 DEBUG:teuthology.orchestra.run.smithi019:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T15:15:31.114 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T15:15:31.386 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.386+0000 7f97d6e00640 1 -- 172.21.15.19:0/2128357442 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f97d0104bf0 msgr2=0x7f97d0105070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:31.386 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.386+0000 7f97d6e00640 1 --2- 172.21.15.19:0/2128357442 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f97d0104bf0 0x7f97d0105070 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f97bc009900 tx=0x7f97bc02bc30 comp rx=0 tx=0).stop 2024-03-18T15:15:31.386 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.386+0000 7f97d6e00640 1 -- 172.21.15.19:0/2128357442 shutdown_connections 2024-03-18T15:15:31.386 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.386+0000 7f97d6e00640 1 --2- 172.21.15.19:0/2128357442 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f97d01055b0 0x7f97d010be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:31.387 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.386+0000 7f97d6e00640 1 --2- 172.21.15.19:0/2128357442 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f97d0104bf0 0x7f97d0105070 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:31.387 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.386+0000 7f97d6e00640 1 --2- 172.21.15.19:0/2128357442 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f97d0103a00 0x7f97d0103e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:31.387 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.386+0000 7f97d6e00640 1 -- 172.21.15.19:0/2128357442 >> 172.21.15.19:0/2128357442 conn(0x7f97d00fef90 msgr2=0x7f97d01013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:31.387 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.386+0000 7f97d6e00640 1 -- 172.21.15.19:0/2128357442 shutdown_connections 2024-03-18T15:15:31.387 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.386+0000 7f97d6e00640 1 -- 172.21.15.19:0/2128357442 wait complete. 2024-03-18T15:15:31.388 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.387+0000 7f97d6e00640 1 Processor -- start 2024-03-18T15:15:31.388 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.387+0000 7f97d6e00640 1 -- start start 2024-03-18T15:15:31.388 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.388+0000 7f97d6e00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f97d0103a00 0x7f97d0199cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:31.388 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.388+0000 7f97d6e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f97d0104bf0 0x7f97d019a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:31.388 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.388+0000 7f97d6e00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f97d01055b0 0x7f97d01a1250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:31.388 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.388+0000 7f97d6e00640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f97d010f6d0 con 0x7f97d0103a00 2024-03-18T15:15:31.388 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.388+0000 7f97d6e00640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f97d010f550 con 0x7f97d0104bf0 2024-03-18T15:15:31.388 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.388+0000 7f97d6e00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f97d010f850 con 0x7f97d01055b0 2024-03-18T15:15:31.389 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.388+0000 7f97cf400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f97d0104bf0 0x7f97d019a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:31.389 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.388+0000 7f97cfe00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f97d0103a00 0x7f97d0199cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:31.389 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.388+0000 7f97cf400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f97d0104bf0 0x7f97d019a1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:33994/0 (socket says 172.21.15.19:33994) 2024-03-18T15:15:31.389 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.388+0000 7f97cf400640 1 -- 172.21.15.19:0/3585168 learned_addr learned my addr 172.21.15.19:0/3585168 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:15:31.389 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.389+0000 7f97cf400640 1 -- 172.21.15.19:0/3585168 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f97d01055b0 msgr2=0x7f97d01a1250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T15:15:31.389 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.389+0000 7f97cf400640 1 --2- 172.21.15.19:0/3585168 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f97d01055b0 0x7f97d01a1250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:31.390 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.389+0000 7f97cf400640 1 -- 172.21.15.19:0/3585168 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f97d0103a00 msgr2=0x7f97d0199cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:31.390 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.389+0000 7f97cf400640 1 --2- 172.21.15.19:0/3585168 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f97d0103a00 0x7f97d0199cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:31.390 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.389+0000 7f97cf400640 1 -- 172.21.15.19:0/3585168 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97d01a18f0 con 0x7f97d0104bf0 2024-03-18T15:15:31.390 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.389+0000 7f97cfe00640 1 --2- 172.21.15.19:0/3585168 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f97d0103a00 0x7f97d0199cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T15:15:31.390 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.389+0000 7f97cf400640 1 --2- 172.21.15.19:0/3585168 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f97d0104bf0 0x7f97d019a1f0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f97bc005ab0 tx=0x7f97bc02c6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:31.393 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.392+0000 7f97ccc00640 1 -- 172.21.15.19:0/3585168 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97bc043020 con 0x7f97d0104bf0 2024-03-18T15:15:31.393 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.392+0000 7f97ccc00640 1 -- 172.21.15.19:0/3585168 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f97bc0054e0 con 0x7f97d0104bf0 2024-03-18T15:15:31.393 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.392+0000 7f97ccc00640 1 -- 172.21.15.19:0/3585168 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97bc003ce0 con 0x7f97d0104bf0 2024-03-18T15:15:31.393 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.393+0000 7f97d6e00640 1 -- 172.21.15.19:0/3585168 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97d01a1b80 con 0x7f97d0104bf0 2024-03-18T15:15:31.393 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.393+0000 7f97d6e00640 1 -- 172.21.15.19:0/3585168 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97d01a2080 con 0x7f97d0104bf0 2024-03-18T15:15:31.393 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.394+0000 7f97d6e00640 1 -- 172.21.15.19:0/3585168 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97d010ad20 con 0x7f97d0104bf0 2024-03-18T15:15:31.395 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.395+0000 7f97ccc00640 1 -- 172.21.15.19:0/3585168 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f97bc046020 con 0x7f97d0104bf0 2024-03-18T15:15:31.397 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.396+0000 7f97ccc00640 1 --2- 172.21.15.19:0/3585168 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f97b40a2880 0x7f97b40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:31.397 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.397+0000 7f97ccc00640 1 -- 172.21.15.19:0/3585168 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f97bc0f0ec0 con 0x7f97d0104bf0 2024-03-18T15:15:31.397 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.397+0000 7f97cfe00640 1 --2- 172.21.15.19:0/3585168 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f97b40a2880 0x7f97b40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:31.397 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.398+0000 7f97cfe00640 1 --2- 172.21.15.19:0/3585168 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f97b40a2880 0x7f97b40a4d40 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f97d019ab80 tx=0x7f97c000b6b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:31.400 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.400+0000 7f97ccc00640 1 -- 172.21.15.19:0/3585168 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f97bc04d020 con 0x7f97d0104bf0 2024-03-18T15:15:31.605 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.605+0000 7f97d6e00640 1 -- 172.21.15.19:0/3585168 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f97d00623e0 con 0x7f97d0104bf0 2024-03-18T15:15:31.607 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.607+0000 7f97ccc00640 1 -- 172.21.15.19:0/3585168 <== mon.2 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24093 (secure 0 0 0) 0x7f97bc0bdcc0 con 0x7f97d0104bf0 2024-03-18T15:15:31.607 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-18T15:15:31.607 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":89,"fsid":"d4dc2ba6-e538-11ee-95c9-87774f69a715","created":"2024-03-18T15:05:57.538551+0000","modified":"2024-03-18T15:15:15.582418+0000","last_up_change":"2024-03-18T15:14:12.009522+0000","last_in_change":"2024-03-18T15:13:52.555387+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T15:09:56.843349+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T15:15:08.941047+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T15:15:10.044671+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T15:15:13.621916+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"37b5a5cd-27ae-4e77-9b05-774b817b2ee9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6802","nonce":402333948},{"type":"v1","addr":"172.21.15.19:6803","nonce":402333948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6804","nonce":402333948},{"type":"v1","addr":"172.21.15.19:6805","nonce":402333948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6808","nonce":402333948},{"type":"v1","addr":"172.21.15.19:6809","nonce":402333948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6806","nonce":402333948},{"type":"v1","addr":"172.21.15.19:6807","nonce":402333948}]},"public_addr":"172.21.15.19:6803/402333948","cluster_addr":"172.21.15.19:6805/402333948","heartbeat_back_addr":"172.21.15.19:6809/402333948","heartbeat_front_addr":"172.21.15.19:6807/402333948","state":["exists","up"]},{"osd":1,"uuid":"3a1134e8-ede7-4dbd-9be1-eefd2cbb43bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1951880113},{"type":"v1","addr":"172.21.15.33:6801","nonce":1951880113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1951880113},{"type":"v1","addr":"172.21.15.33:6803","nonce":1951880113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1951880113},{"type":"v1","addr":"172.21.15.33:6807","nonce":1951880113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1951880113},{"type":"v1","addr":"172.21.15.33:6805","nonce":1951880113}]},"public_addr":"172.21.15.33:6801/1951880113","cluster_addr":"172.21.15.33:6803/1951880113","heartbeat_back_addr":"172.21.15.33:6807/1951880113","heartbeat_front_addr":"172.21.15.33:6805/1951880113","state":["exists","up"]},{"osd":2,"uuid":"f383a164-a3a0-49fd-a9e8-9c2178d8cfb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":1004727949},{"type":"v1","addr":"172.21.15.104:6801","nonce":1004727949}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":1004727949},{"type":"v1","addr":"172.21.15.104:6803","nonce":1004727949}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":1004727949},{"type":"v1","addr":"172.21.15.104:6807","nonce":1004727949}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":1004727949},{"type":"v1","addr":"172.21.15.104:6805","nonce":1004727949}]},"public_addr":"172.21.15.104:6801/1004727949","cluster_addr":"172.21.15.104:6803/1004727949","heartbeat_back_addr":"172.21.15.104:6807/1004727949","heartbeat_front_addr":"172.21.15.104:6805/1004727949","state":["exists","up"]},{"osd":3,"uuid":"c409e116-28bf-45dd-a791-539af22abe1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6810","nonce":3571525854},{"type":"v1","addr":"172.21.15.19:6811","nonce":3571525854}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6812","nonce":3571525854},{"type":"v1","addr":"172.21.15.19:6813","nonce":3571525854}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6816","nonce":3571525854},{"type":"v1","addr":"172.21.15.19:6817","nonce":3571525854}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6814","nonce":3571525854},{"type":"v1","addr":"172.21.15.19:6815","nonce":3571525854}]},"public_addr":"172.21.15.19:6811/3571525854","cluster_addr":"172.21.15.19:6813/3571525854","heartbeat_back_addr":"172.21.15.19:6817/3571525854","heartbeat_front_addr":"172.21.15.19:6815/3571525854","state":["exists","up"]},{"osd":4,"uuid":"696eb719-0b3e-43ae-b5f1-ccd73a3b2843","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":624901749},{"type":"v1","addr":"172.21.15.33:6809","nonce":624901749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":624901749},{"type":"v1","addr":"172.21.15.33:6811","nonce":624901749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":624901749},{"type":"v1","addr":"172.21.15.33:6815","nonce":624901749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":624901749},{"type":"v1","addr":"172.21.15.33:6813","nonce":624901749}]},"public_addr":"172.21.15.33:6809/624901749","cluster_addr":"172.21.15.33:6811/624901749","heartbeat_back_addr":"172.21.15.33:6815/624901749","heartbeat_front_addr":"172.21.15.33:6813/624901749","state":["exists","up"]},{"osd":5,"uuid":"21789779-8bb9-42c7-b9ee-437985f11e3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":1258208297},{"type":"v1","addr":"172.21.15.104:6809","nonce":1258208297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":1258208297},{"type":"v1","addr":"172.21.15.104:6811","nonce":1258208297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":1258208297},{"type":"v1","addr":"172.21.15.104:6815","nonce":1258208297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":1258208297},{"type":"v1","addr":"172.21.15.104:6813","nonce":1258208297}]},"public_addr":"172.21.15.104:6809/1258208297","cluster_addr":"172.21.15.104:6811/1258208297","heartbeat_back_addr":"172.21.15.104:6815/1258208297","heartbeat_front_addr":"172.21.15.104:6813/1258208297","state":["exists","up"]},{"osd":6,"uuid":"0e92a2ef-43cd-487c-9203-e219158aef8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6818","nonce":2782482301},{"type":"v1","addr":"172.21.15.19:6819","nonce":2782482301}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6820","nonce":2782482301},{"type":"v1","addr":"172.21.15.19:6821","nonce":2782482301}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6824","nonce":2782482301},{"type":"v1","addr":"172.21.15.19:6825","nonce":2782482301}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6822","nonce":2782482301},{"type":"v1","addr":"172.21.15.19:6823","nonce":2782482301}]},"public_addr":"172.21.15.19:6819/2782482301","cluster_addr":"172.21.15.19:6821/2782482301","heartbeat_back_addr":"172.21.15.19:6825/2782482301","heartbeat_front_addr":"172.21.15.19:6823/2782482301","state":["exists","up"]},{"osd":7,"uuid":"a16e39a1-3685-491d-b376-3e80a2084072","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":4238160988},{"type":"v1","addr":"172.21.15.33:6817","nonce":4238160988}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":4238160988},{"type":"v1","addr":"172.21.15.33:6819","nonce":4238160988}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":4238160988},{"type":"v1","addr":"172.21.15.33:6823","nonce":4238160988}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":4238160988},{"type":"v1","addr":"172.21.15.33:6821","nonce":4238160988}]},"public_addr":"172.21.15.33:6817/4238160988","cluster_addr":"172.21.15.33:6819/4238160988","heartbeat_back_addr":"172.21.15.33:6823/4238160988","heartbeat_front_addr":"172.21.15.33:6821/4238160988","state":["exists","up"]},{"osd":8,"uuid":"d5c3fa92-cd5d-4349-9c7d-d5666bacde16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":1014779813},{"type":"v1","addr":"172.21.15.104:6817","nonce":1014779813}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":1014779813},{"type":"v1","addr":"172.21.15.104:6819","nonce":1014779813}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":1014779813},{"type":"v1","addr":"172.21.15.104:6823","nonce":1014779813}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":1014779813},{"type":"v1","addr":"172.21.15.104:6821","nonce":1014779813}]},"public_addr":"172.21.15.104:6817/1014779813","cluster_addr":"172.21.15.104:6819/1014779813","heartbeat_back_addr":"172.21.15.104:6823/1014779813","heartbeat_front_addr":"172.21.15.104:6821/1014779813","state":["exists","up"]},{"osd":9,"uuid":"eed2e9b2-26d3-4c46-88ec-34b5596fe991","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6826","nonce":2641151009},{"type":"v1","addr":"172.21.15.19:6827","nonce":2641151009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6828","nonce":2641151009},{"type":"v1","addr":"172.21.15.19:6829","nonce":2641151009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6832","nonce":2641151009},{"type":"v1","addr":"172.21.15.19:6833","nonce":2641151009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6830","nonce":2641151009},{"type":"v1","addr":"172.21.15.19:6831","nonce":2641151009}]},"public_addr":"172.21.15.19:6827/2641151009","cluster_addr":"172.21.15.19:6829/2641151009","heartbeat_back_addr":"172.21.15.19:6833/2641151009","heartbeat_front_addr":"172.21.15.19:6831/2641151009","state":["exists","up"]},{"osd":10,"uuid":"f3c35ea5-fbe7-4dc4-93c5-b92d1a714db3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":1346058869},{"type":"v1","addr":"172.21.15.33:6825","nonce":1346058869}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":1346058869},{"type":"v1","addr":"172.21.15.33:6827","nonce":1346058869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":1346058869},{"type":"v1","addr":"172.21.15.33:6831","nonce":1346058869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":1346058869},{"type":"v1","addr":"172.21.15.33:6829","nonce":1346058869}]},"public_addr":"172.21.15.33:6825/1346058869","cluster_addr":"172.21.15.33:6827/1346058869","heartbeat_back_addr":"172.21.15.33:6831/1346058869","heartbeat_front_addr":"172.21.15.33:6829/1346058869","state":["exists","up"]},{"osd":11,"uuid":"c8b51571-f807-4b62-90b6-7529e09e59da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":3759498281},{"type":"v1","addr":"172.21.15.104:6825","nonce":3759498281}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":3759498281},{"type":"v1","addr":"172.21.15.104:6827","nonce":3759498281}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":3759498281},{"type":"v1","addr":"172.21.15.104:6831","nonce":3759498281}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":3759498281},{"type":"v1","addr":"172.21.15.104:6829","nonce":3759498281}]},"public_addr":"172.21.15.104:6825/3759498281","cluster_addr":"172.21.15.104:6827/3759498281","heartbeat_back_addr":"172.21.15.104:6831/3759498281","heartbeat_front_addr":"172.21.15.104:6829/3759498281","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:09:05.267795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:09:30.234068+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:09:53.870232+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:10:26.471296+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:10:53.180341+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:11:17.801361+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:11:51.364816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:12:18.306408+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:12:43.978744+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:13:18.213964+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:13:45.437544+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:14:10.148580+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.19:0/3860783274":"2024-03-19T15:07:14.150983+0000","172.21.15.19:0/2737223213":"2024-03-19T15:07:14.150983+0000","172.21.15.19:0/176699025":"2024-03-19T15:06:38.688405+0000","172.21.15.19:0/2316969005":"2024-03-19T15:07:14.150983+0000","172.21.15.19:6801/2776008962":"2024-03-19T15:07:14.150983+0000","172.21.15.19:0/2793980440":"2024-03-19T15:06:38.688405+0000","172.21.15.19:6800/3264337803":"2024-03-19T15:06:38.688405+0000","172.21.15.19:6801/3264337803":"2024-03-19T15:06:38.688405+0000","172.21.15.19:0/3166705295":"2024-03-19T15:06:38.688405+0000","172.21.15.19:6800/2776008962":"2024-03-19T15:07:14.150983+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T15:15:31.610 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.610+0000 7f97d6e00640 1 -- 172.21.15.19:0/3585168 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f97b40a2880 msgr2=0x7f97b40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:31.610 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.610+0000 7f97d6e00640 1 --2- 172.21.15.19:0/3585168 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f97b40a2880 0x7f97b40a4d40 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f97d019ab80 tx=0x7f97c000b6b0 comp rx=0 tx=0).stop 2024-03-18T15:15:31.610 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.610+0000 7f97d6e00640 1 -- 172.21.15.19:0/3585168 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f97d0104bf0 msgr2=0x7f97d019a1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:31.610 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.610+0000 7f97d6e00640 1 --2- 172.21.15.19:0/3585168 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f97d0104bf0 0x7f97d019a1f0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f97bc005ab0 tx=0x7f97bc02c6c0 comp rx=0 tx=0).stop 2024-03-18T15:15:31.610 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.610+0000 7f97d6e00640 1 -- 172.21.15.19:0/3585168 shutdown_connections 2024-03-18T15:15:31.610 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.610+0000 7f97d6e00640 1 --2- 172.21.15.19:0/3585168 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f97d01055b0 0x7f97d01a1250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:31.611 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.610+0000 7f97d6e00640 1 --2- 172.21.15.19:0/3585168 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f97b40a2880 0x7f97b40a4d40 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:31.611 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.610+0000 7f97d6e00640 1 --2- 172.21.15.19:0/3585168 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f97d0104bf0 0x7f97d019a1f0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:31.611 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.610+0000 7f97d6e00640 1 --2- 172.21.15.19:0/3585168 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f97d0103a00 0x7f97d0199cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:31.611 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.610+0000 7f97d6e00640 1 -- 172.21.15.19:0/3585168 >> 172.21.15.19:0/3585168 conn(0x7f97d00fef90 msgr2=0x7f97d0107ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:31.611 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.610+0000 7f97d6e00640 1 -- 172.21.15.19:0/3585168 shutdown_connections 2024-03-18T15:15:31.611 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.610+0000 7f97d6e00640 1 -- 172.21.15.19:0/3585168 wait complete. 2024-03-18T15:15:31.631 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T15:09:56.843349+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T15:15:08.941047+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T15:15:10.044671+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-18T15:15:13.621916+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-18T15:15:31.632 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T15:15:31.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:31 smithi104 ceph-mon[36070]: pgmap v327: 129 pgs: 129 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-03-18T15:15:31.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:31 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/10578497' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T15:15:31.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:31 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/731625546' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-18T15:15:31.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:31 smithi019 ceph-mon[31522]: pgmap v327: 129 pgs: 129 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-03-18T15:15:31.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:31 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/10578497' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T15:15:31.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:31 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/731625546' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-18T15:15:31.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:31 smithi033 ceph-mon[36648]: pgmap v327: 129 pgs: 129 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-03-18T15:15:31.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:31 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/10578497' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T15:15:31.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:31 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/731625546' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-18T15:15:31.861 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.861+0000 7f9aab400640 1 -- 172.21.15.19:0/1231883998 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9aa4104bf0 msgr2=0x7f9aa4105070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:31.861 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.861+0000 7f9aab400640 1 --2- 172.21.15.19:0/1231883998 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9aa4104bf0 0x7f9aa4105070 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f9a94009900 tx=0x7f9a9402bc50 comp rx=0 tx=0).stop 2024-03-18T15:15:31.861 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.861+0000 7f9aab400640 1 -- 172.21.15.19:0/1231883998 shutdown_connections 2024-03-18T15:15:31.862 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.861+0000 7f9aab400640 1 --2- 172.21.15.19:0/1231883998 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9aa41055b0 0x7f9aa410be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:31.862 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.861+0000 7f9aab400640 1 --2- 172.21.15.19:0/1231883998 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9aa4104bf0 0x7f9aa4105070 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:31.862 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.861+0000 7f9aab400640 1 --2- 172.21.15.19:0/1231883998 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9aa4103a00 0x7f9aa4103e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:31.862 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.861+0000 7f9aab400640 1 -- 172.21.15.19:0/1231883998 >> 172.21.15.19:0/1231883998 conn(0x7f9aa40fef90 msgr2=0x7f9aa41013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:31.862 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.861+0000 7f9aab400640 1 -- 172.21.15.19:0/1231883998 shutdown_connections 2024-03-18T15:15:31.862 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.861+0000 7f9aab400640 1 -- 172.21.15.19:0/1231883998 wait complete. 2024-03-18T15:15:31.864 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.862+0000 7f9aab400640 1 Processor -- start 2024-03-18T15:15:31.864 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.862+0000 7f9aab400640 1 -- start start 2024-03-18T15:15:31.864 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.863+0000 7f9aab400640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9aa4103a00 0x7f9aa4199d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:31.864 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.863+0000 7f9aab400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9aa4104bf0 0x7f9aa419a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:31.864 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.863+0000 7f9aab400640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9aa41055b0 0x7f9aa41a1300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:31.864 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.863+0000 7f9aab400640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f9aa410f660 con 0x7f9aa41055b0 2024-03-18T15:15:31.864 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.863+0000 7f9aab400640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9aa410f4e0 con 0x7f9aa4104bf0 2024-03-18T15:15:31.864 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.863+0000 7f9aab400640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f9aa410f7e0 con 0x7f9aa4103a00 2024-03-18T15:15:31.864 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.863+0000 7f9a9be00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9aa4104bf0 0x7f9aa419a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:31.865 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.863+0000 7f9a9be00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9aa4104bf0 0x7f9aa419a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:34022/0 (socket says 172.21.15.19:34022) 2024-03-18T15:15:31.865 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.863+0000 7f9a9be00640 1 -- 172.21.15.19:0/1464720723 learned_addr learned my addr 172.21.15.19:0/1464720723 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:15:31.865 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.863+0000 7f9aa8a00640 1 --2- 172.21.15.19:0/1464720723 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9aa4103a00 0x7f9aa4199d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:31.865 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.863+0000 7f9aa9400640 1 --2- 172.21.15.19:0/1464720723 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9aa41055b0 0x7f9aa41a1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:31.865 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.863+0000 7f9a9be00640 1 -- 172.21.15.19:0/1464720723 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9aa4103a00 msgr2=0x7f9aa4199d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:31.865 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.863+0000 7f9a9be00640 1 --2- 172.21.15.19:0/1464720723 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9aa4103a00 0x7f9aa4199d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:31.865 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.863+0000 7f9a9be00640 1 -- 172.21.15.19:0/1464720723 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9aa41055b0 msgr2=0x7f9aa41a1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:31.865 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.863+0000 7f9a9be00640 1 --2- 172.21.15.19:0/1464720723 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9aa41055b0 0x7f9aa41a1300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:31.865 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.863+0000 7f9a9be00640 1 -- 172.21.15.19:0/1464720723 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9aa41a1970 con 0x7f9aa4104bf0 2024-03-18T15:15:31.865 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.864+0000 7f9a9be00640 1 --2- 172.21.15.19:0/1464720723 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9aa4104bf0 0x7f9aa419a290 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f9a94009230 tx=0x7f9a9402c670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:31.865 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.864+0000 7f9aa2000640 1 -- 172.21.15.19:0/1464720723 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a94042020 con 0x7f9aa4104bf0 2024-03-18T15:15:31.865 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.864+0000 7f9aa2000640 1 -- 172.21.15.19:0/1464720723 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a94004450 con 0x7f9aa4104bf0 2024-03-18T15:15:31.865 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.864+0000 7f9aa2000640 1 -- 172.21.15.19:0/1464720723 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a94003e20 con 0x7f9aa4104bf0 2024-03-18T15:15:31.865 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.864+0000 7f9aab400640 1 -- 172.21.15.19:0/1464720723 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9aa41a1c00 con 0x7f9aa4104bf0 2024-03-18T15:15:31.866 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.864+0000 7f9aab400640 1 -- 172.21.15.19:0/1464720723 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9aa41a21a0 con 0x7f9aa4104bf0 2024-03-18T15:15:31.866 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.865+0000 7f9aab400640 1 -- 172.21.15.19:0/1464720723 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9aa410ad20 con 0x7f9aa4104bf0 2024-03-18T15:15:31.867 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.867+0000 7f9aa2000640 1 -- 172.21.15.19:0/1464720723 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f9a94045020 con 0x7f9aa4104bf0 2024-03-18T15:15:31.868 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.868+0000 7f9aa2000640 1 --2- 172.21.15.19:0/1464720723 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f9a840a2880 0x7f9a840a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:31.868 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.868+0000 7f9aa2000640 1 -- 172.21.15.19:0/1464720723 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f9a940f0c40 con 0x7f9aa4104bf0 2024-03-18T15:15:31.868 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.868+0000 7f9aa8a00640 1 --2- 172.21.15.19:0/1464720723 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f9a840a2880 0x7f9a840a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:31.869 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.869+0000 7f9aa8a00640 1 --2- 172.21.15.19:0/1464720723 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f9a840a2880 0x7f9a840a4d40 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c0095f0 tx=0x7f9a8c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:31.872 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:31.872+0000 7f9aa2000640 1 -- 172.21.15.19:0/1464720723 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9a9404c020 con 0x7f9aa4104bf0 2024-03-18T15:15:32.073 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.073+0000 7f9aab400640 1 -- 172.21.15.19:0/1464720723 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f9aa402d1b0 con 0x7f9aa4104bf0 2024-03-18T15:15:32.074 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.074+0000 7f9aa2000640 1 -- 172.21.15.19:0/1464720723 <== mon.2 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) ==== 93+0+10 (secure 0 0 0) 0x7f9a940bda40 con 0x7f9aa4104bf0 2024-03-18T15:15:32.074 INFO:teuthology.orchestra.run.smithi019.stdout:pg_num: 1 2024-03-18T15:15:32.077 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.077+0000 7f9aab400640 1 -- 172.21.15.19:0/1464720723 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f9a840a2880 msgr2=0x7f9a840a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:32.078 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.077+0000 7f9aab400640 1 --2- 172.21.15.19:0/1464720723 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f9a840a2880 0x7f9a840a4d40 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c0095f0 tx=0x7f9a8c005f90 comp rx=0 tx=0).stop 2024-03-18T15:15:32.078 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.077+0000 7f9aab400640 1 -- 172.21.15.19:0/1464720723 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9aa4104bf0 msgr2=0x7f9aa419a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:32.078 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.077+0000 7f9aab400640 1 --2- 172.21.15.19:0/1464720723 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9aa4104bf0 0x7f9aa419a290 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f9a94009230 tx=0x7f9a9402c670 comp rx=0 tx=0).stop 2024-03-18T15:15:32.078 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.077+0000 7f9aab400640 1 -- 172.21.15.19:0/1464720723 shutdown_connections 2024-03-18T15:15:32.078 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.077+0000 7f9aab400640 1 --2- 172.21.15.19:0/1464720723 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9aa41055b0 0x7f9aa41a1300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:32.078 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.077+0000 7f9aab400640 1 --2- 172.21.15.19:0/1464720723 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f9a840a2880 0x7f9a840a4d40 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:32.078 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.077+0000 7f9aab400640 1 --2- 172.21.15.19:0/1464720723 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9aa4104bf0 0x7f9aa419a290 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:32.078 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.077+0000 7f9aab400640 1 --2- 172.21.15.19:0/1464720723 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9aa4103a00 0x7f9aa4199d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:32.078 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.077+0000 7f9aab400640 1 -- 172.21.15.19:0/1464720723 >> 172.21.15.19:0/1464720723 conn(0x7f9aa40fef90 msgr2=0x7f9aa4107ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:32.078 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.077+0000 7f9aab400640 1 -- 172.21.15.19:0/1464720723 shutdown_connections 2024-03-18T15:15:32.078 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.077+0000 7f9aab400640 1 -- 172.21.15.19:0/1464720723 wait complete. 2024-03-18T15:15:32.097 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T15:15:32.330 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.330+0000 7f02a6a00640 1 -- 172.21.15.19:0/3646194942 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f02a0074240 msgr2=0x7f02a00746c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:32.331 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.330+0000 7f02a6a00640 1 --2- 172.21.15.19:0/3646194942 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f02a0074240 0x7f02a00746c0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f028c009ca0 tx=0x7f028c02c0f0 comp rx=0 tx=0).stop 2024-03-18T15:15:32.331 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.330+0000 7f02a6a00640 1 -- 172.21.15.19:0/3646194942 shutdown_connections 2024-03-18T15:15:32.331 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.330+0000 7f02a6a00640 1 --2- 172.21.15.19:0/3646194942 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f02a01095a0 0x7f02a010ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:32.331 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.330+0000 7f02a6a00640 1 --2- 172.21.15.19:0/3646194942 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f02a0074240 0x7f02a00746c0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:32.331 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.330+0000 7f02a6a00640 1 --2- 172.21.15.19:0/3646194942 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f02a0075820 0x7f02a0073c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:32.331 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.330+0000 7f02a6a00640 1 -- 172.21.15.19:0/3646194942 >> 172.21.15.19:0/3646194942 conn(0x7f02a00fcc50 msgr2=0x7f02a00ff090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:32.331 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.330+0000 7f02a6a00640 1 -- 172.21.15.19:0/3646194942 shutdown_connections 2024-03-18T15:15:32.331 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.330+0000 7f02a6a00640 1 -- 172.21.15.19:0/3646194942 wait complete. 2024-03-18T15:15:32.333 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.331+0000 7f02a6a00640 1 Processor -- start 2024-03-18T15:15:32.333 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.331+0000 7f02a6a00640 1 -- start start 2024-03-18T15:15:32.333 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.332+0000 7f02a6a00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f02a0074240 0x7f02a0199c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:32.333 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.332+0000 7f02a6a00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f02a0075820 0x7f02a019a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:32.333 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.332+0000 7f02a6a00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f02a01095a0 0x7f02a01a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:32.334 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.332+0000 7f02a6a00640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f02a010f540 con 0x7f02a01095a0 2024-03-18T15:15:32.334 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.332+0000 7f02a6a00640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f02a010f3c0 con 0x7f02a0075820 2024-03-18T15:15:32.334 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.332+0000 7f02a6a00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f02a010f6c0 con 0x7f02a0074240 2024-03-18T15:15:32.334 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.332+0000 7f02a4c00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f02a0075820 0x7f02a019a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:32.334 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.332+0000 7f02a6000640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f02a01095a0 0x7f02a01a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:32.334 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.332+0000 7f02a4c00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f02a0075820 0x7f02a019a170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:60970/0 (socket says 172.21.15.19:60970) 2024-03-18T15:15:32.334 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.332+0000 7f02a4c00640 1 -- 172.21.15.19:0/1388505458 learned_addr learned my addr 172.21.15.19:0/1388505458 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:15:32.334 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.332+0000 7f02a5600640 1 --2- 172.21.15.19:0/1388505458 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f02a0074240 0x7f02a0199c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:32.334 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.332+0000 7f02a4c00640 1 -- 172.21.15.19:0/1388505458 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f02a0074240 msgr2=0x7f02a0199c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:32.334 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.332+0000 7f02a4c00640 1 --2- 172.21.15.19:0/1388505458 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f02a0074240 0x7f02a0199c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:32.334 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.332+0000 7f02a4c00640 1 -- 172.21.15.19:0/1388505458 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f02a01095a0 msgr2=0x7f02a01a1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:32.334 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.332+0000 7f02a4c00640 1 --2- 172.21.15.19:0/1388505458 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f02a01095a0 0x7f02a01a1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:32.334 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.332+0000 7f02a4c00640 1 -- 172.21.15.19:0/1388505458 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02a01a1800 con 0x7f02a0075820 2024-03-18T15:15:32.335 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.332+0000 7f02a5600640 1 --2- 172.21.15.19:0/1388505458 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f02a0074240 0x7f02a0199c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T15:15:32.335 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.332+0000 7f02a4c00640 1 --2- 172.21.15.19:0/1388505458 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f02a0075820 0x7f02a019a170 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f028c009dc0 tx=0x7f028c032f20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:32.335 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.333+0000 7f029e000640 1 -- 172.21.15.19:0/1388505458 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f028c041020 con 0x7f02a0075820 2024-03-18T15:15:32.335 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.333+0000 7f029e000640 1 -- 172.21.15.19:0/1388505458 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f028c034cd0 con 0x7f02a0075820 2024-03-18T15:15:32.335 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.333+0000 7f02a6a00640 1 -- 172.21.15.19:0/1388505458 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02a01a1a90 con 0x7f02a0075820 2024-03-18T15:15:32.335 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.333+0000 7f029e000640 1 -- 172.21.15.19:0/1388505458 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f028c039040 con 0x7f02a0075820 2024-03-18T15:15:32.335 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.333+0000 7f02a6a00640 1 -- 172.21.15.19:0/1388505458 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02a01a1fd0 con 0x7f02a0075820 2024-03-18T15:15:32.336 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.334+0000 7f02a6a00640 1 -- 172.21.15.19:0/1388505458 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02a01082d0 con 0x7f02a0075820 2024-03-18T15:15:32.336 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.336+0000 7f029e000640 1 -- 172.21.15.19:0/1388505458 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f028c044020 con 0x7f02a0075820 2024-03-18T15:15:32.337 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.337+0000 7f029e000640 1 --2- 172.21.15.19:0/1388505458 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f02840a2880 0x7f02840a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:32.337 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.337+0000 7f029e000640 1 -- 172.21.15.19:0/1388505458 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f028c0f0b00 con 0x7f02a0075820 2024-03-18T15:15:32.337 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.337+0000 7f02a5600640 1 --2- 172.21.15.19:0/1388505458 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f02840a2880 0x7f02840a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:32.338 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.338+0000 7f02a5600640 1 --2- 172.21.15.19:0/1388505458 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f02840a2880 0x7f02840a4d40 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f0298009c20 tx=0x7f0298005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:32.341 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.341+0000 7f029e000640 1 -- 172.21.15.19:0/1388505458 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f028c04b020 con 0x7f02a0075820 2024-03-18T15:15:32.549 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.549+0000 7f02a6a00640 1 -- 172.21.15.19:0/1388505458 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f02a00623e0 con 0x7f02a0075820 2024-03-18T15:15:32.550 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.550+0000 7f029e000640 1 -- 172.21.15.19:0/1388505458 <== mon.2 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) ==== 104+0+11 (secure 0 0 0) 0x7f028c0bd960 con 0x7f02a0075820 2024-03-18T15:15:32.550 INFO:teuthology.orchestra.run.smithi019.stdout:pg_num: 32 2024-03-18T15:15:32.553 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.552+0000 7f02a6a00640 1 -- 172.21.15.19:0/1388505458 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f02840a2880 msgr2=0x7f02840a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:32.553 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.552+0000 7f02a6a00640 1 --2- 172.21.15.19:0/1388505458 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f02840a2880 0x7f02840a4d40 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f0298009c20 tx=0x7f0298005f90 comp rx=0 tx=0).stop 2024-03-18T15:15:32.553 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.552+0000 7f02a6a00640 1 -- 172.21.15.19:0/1388505458 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f02a0075820 msgr2=0x7f02a019a170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:32.553 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.552+0000 7f02a6a00640 1 --2- 172.21.15.19:0/1388505458 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f02a0075820 0x7f02a019a170 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f028c009dc0 tx=0x7f028c032f20 comp rx=0 tx=0).stop 2024-03-18T15:15:32.553 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.553+0000 7f02a6a00640 1 -- 172.21.15.19:0/1388505458 shutdown_connections 2024-03-18T15:15:32.553 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.553+0000 7f02a6a00640 1 --2- 172.21.15.19:0/1388505458 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f02a01095a0 0x7f02a01a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:32.553 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.553+0000 7f02a6a00640 1 --2- 172.21.15.19:0/1388505458 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f02840a2880 0x7f02840a4d40 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:32.553 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.553+0000 7f02a6a00640 1 --2- 172.21.15.19:0/1388505458 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f02a0075820 0x7f02a019a170 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:32.553 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.553+0000 7f02a6a00640 1 --2- 172.21.15.19:0/1388505458 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f02a0074240 0x7f02a0199c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:32.553 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.553+0000 7f02a6a00640 1 -- 172.21.15.19:0/1388505458 >> 172.21.15.19:0/1388505458 conn(0x7f02a00fcc50 msgr2=0x7f02a01054a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:32.554 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.553+0000 7f02a6a00640 1 -- 172.21.15.19:0/1388505458 shutdown_connections 2024-03-18T15:15:32.554 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.553+0000 7f02a6a00640 1 -- 172.21.15.19:0/1388505458 wait complete. 2024-03-18T15:15:32.571 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T15:15:32.654 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:32 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/3585168' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:15:32.654 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:32 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/1464720723' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T15:15:32.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:32 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/3585168' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:15:32.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:32 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/1464720723' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T15:15:32.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:32 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/3585168' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:15:32.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:32 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/1464720723' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T15:15:32.799 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.799+0000 7efd3ec00640 1 -- 172.21.15.19:0/3278293486 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efd38103290 msgr2=0x7efd38109b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:32.799 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.799+0000 7efd3ec00640 1 --2- 172.21.15.19:0/3278293486 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efd38103290 0x7efd38109b20 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7efd30009ca0 tx=0x7efd3002c100 comp rx=0 tx=0).stop 2024-03-18T15:15:32.799 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.799+0000 7efd3ec00640 1 -- 172.21.15.19:0/3278293486 shutdown_connections 2024-03-18T15:15:32.800 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.799+0000 7efd3ec00640 1 --2- 172.21.15.19:0/3278293486 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efd38103290 0x7efd38109b20 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:32.800 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.799+0000 7efd3ec00640 1 --2- 172.21.15.19:0/3278293486 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efd381028d0 0x7efd38102d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:32.800 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.799+0000 7efd3ec00640 1 --2- 172.21.15.19:0/3278293486 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efd381016e0 0x7efd38101b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:32.800 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.799+0000 7efd3ec00640 1 -- 172.21.15.19:0/3278293486 >> 172.21.15.19:0/3278293486 conn(0x7efd380fcc50 msgr2=0x7efd380ff090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:32.800 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.799+0000 7efd3ec00640 1 -- 172.21.15.19:0/3278293486 shutdown_connections 2024-03-18T15:15:32.800 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.799+0000 7efd3ec00640 1 -- 172.21.15.19:0/3278293486 wait complete. 2024-03-18T15:15:32.802 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.800+0000 7efd3ec00640 1 Processor -- start 2024-03-18T15:15:32.802 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.800+0000 7efd3ec00640 1 -- start start 2024-03-18T15:15:32.802 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.800+0000 7efd3ec00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efd381016e0 0x7efd38197980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:32.802 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.800+0000 7efd3ec00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efd381028d0 0x7efd38197ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:32.803 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.800+0000 7efd3ec00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efd38103290 0x7efd3819ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:32.803 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.800+0000 7efd3ec00640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7efd3810d360 con 0x7efd38103290 2024-03-18T15:15:32.803 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.800+0000 7efd3ec00640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7efd3810d1e0 con 0x7efd381016e0 2024-03-18T15:15:32.803 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.800+0000 7efd3ec00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7efd3810d4e0 con 0x7efd381028d0 2024-03-18T15:15:32.803 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.801+0000 7efd3d800640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efd381016e0 0x7efd38197980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:32.803 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.801+0000 7efd3ce00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efd381028d0 0x7efd38197ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:32.803 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.801+0000 7efd3d800640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efd381016e0 0x7efd38197980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:60978/0 (socket says 172.21.15.19:60978) 2024-03-18T15:15:32.803 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.801+0000 7efd3d800640 1 -- 172.21.15.19:0/3161294969 learned_addr learned my addr 172.21.15.19:0/3161294969 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:15:32.803 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.801+0000 7efd3e200640 1 --2- 172.21.15.19:0/3161294969 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efd38103290 0x7efd3819ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:32.803 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.801+0000 7efd3d800640 1 -- 172.21.15.19:0/3161294969 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efd381028d0 msgr2=0x7efd38197ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:32.803 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.801+0000 7efd3d800640 1 --2- 172.21.15.19:0/3161294969 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efd381028d0 0x7efd38197ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:32.803 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.801+0000 7efd3d800640 1 -- 172.21.15.19:0/3161294969 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efd38103290 msgr2=0x7efd3819ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:32.803 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.801+0000 7efd3d800640 1 --2- 172.21.15.19:0/3161294969 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efd38103290 0x7efd3819ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:32.804 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.801+0000 7efd3d800640 1 -- 172.21.15.19:0/3161294969 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd3819f5a0 con 0x7efd381016e0 2024-03-18T15:15:32.804 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.802+0000 7efd3d800640 1 --2- 172.21.15.19:0/3161294969 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efd381016e0 0x7efd38197980 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7efd2800e9c0 tx=0x7efd2800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:32.804 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.802+0000 7efd36000640 1 -- 172.21.15.19:0/3161294969 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd2800cd80 con 0x7efd381016e0 2024-03-18T15:15:32.804 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.802+0000 7efd3ec00640 1 -- 172.21.15.19:0/3161294969 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd3819f890 con 0x7efd381016e0 2024-03-18T15:15:32.804 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.802+0000 7efd36000640 1 -- 172.21.15.19:0/3161294969 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efd280035f0 con 0x7efd381016e0 2024-03-18T15:15:32.804 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.802+0000 7efd36000640 1 -- 172.21.15.19:0/3161294969 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd28010690 con 0x7efd381016e0 2024-03-18T15:15:32.804 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.802+0000 7efd3ec00640 1 -- 172.21.15.19:0/3161294969 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd3819fd90 con 0x7efd381016e0 2024-03-18T15:15:32.804 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.803+0000 7efd3ec00640 1 -- 172.21.15.19:0/3161294969 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd38108a00 con 0x7efd381016e0 2024-03-18T15:15:32.805 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.805+0000 7efd36000640 1 -- 172.21.15.19:0/3161294969 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7efd280037a0 con 0x7efd381016e0 2024-03-18T15:15:32.807 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.807+0000 7efd36000640 1 --2- 172.21.15.19:0/3161294969 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7efd1c0a2800 0x7efd1c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:32.807 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.807+0000 7efd36000640 1 -- 172.21.15.19:0/3161294969 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7efd280cf060 con 0x7efd381016e0 2024-03-18T15:15:32.807 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.807+0000 7efd3ce00640 1 --2- 172.21.15.19:0/3161294969 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7efd1c0a2800 0x7efd1c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:32.808 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.808+0000 7efd3ce00640 1 --2- 172.21.15.19:0/3161294969 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7efd1c0a2800 0x7efd1c0a4cc0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7efd38198e90 tx=0x7efd24005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:32.811 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:32.811+0000 7efd36000640 1 -- 172.21.15.19:0/3161294969 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7efd2809be60 con 0x7efd381016e0 2024-03-18T15:15:33.017 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.017+0000 7efd3ec00640 1 -- 172.21.15.19:0/3161294969 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7efd380623e0 con 0x7efd381016e0 2024-03-18T15:15:33.018 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.018+0000 7efd36000640 1 -- 172.21.15.19:0/3161294969 <== mon.2 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) ==== 100+0+11 (secure 0 0 0) 0x7efd2809b760 con 0x7efd381016e0 2024-03-18T15:15:33.018 INFO:teuthology.orchestra.run.smithi019.stdout:pg_num: 64 2024-03-18T15:15:33.021 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.020+0000 7efd3ec00640 1 -- 172.21.15.19:0/3161294969 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7efd1c0a2800 msgr2=0x7efd1c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:33.021 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.021+0000 7efd3ec00640 1 --2- 172.21.15.19:0/3161294969 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7efd1c0a2800 0x7efd1c0a4cc0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7efd38198e90 tx=0x7efd24005f90 comp rx=0 tx=0).stop 2024-03-18T15:15:33.021 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.021+0000 7efd3ec00640 1 -- 172.21.15.19:0/3161294969 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efd381016e0 msgr2=0x7efd38197980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:33.021 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.021+0000 7efd3ec00640 1 --2- 172.21.15.19:0/3161294969 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efd381016e0 0x7efd38197980 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7efd2800e9c0 tx=0x7efd2800ee80 comp rx=0 tx=0).stop 2024-03-18T15:15:33.022 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.021+0000 7efd3ec00640 1 -- 172.21.15.19:0/3161294969 shutdown_connections 2024-03-18T15:15:33.022 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.021+0000 7efd3ec00640 1 --2- 172.21.15.19:0/3161294969 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efd38103290 0x7efd3819ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:33.022 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.021+0000 7efd3ec00640 1 --2- 172.21.15.19:0/3161294969 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efd381028d0 0x7efd38197ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:33.022 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.021+0000 7efd3ec00640 1 --2- 172.21.15.19:0/3161294969 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7efd1c0a2800 0x7efd1c0a4cc0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:33.022 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.021+0000 7efd3ec00640 1 --2- 172.21.15.19:0/3161294969 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efd381016e0 0x7efd38197980 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:33.022 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.021+0000 7efd3ec00640 1 -- 172.21.15.19:0/3161294969 >> 172.21.15.19:0/3161294969 conn(0x7efd380fcc50 msgr2=0x7efd38105bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:33.022 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.021+0000 7efd3ec00640 1 -- 172.21.15.19:0/3161294969 shutdown_connections 2024-03-18T15:15:33.022 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.021+0000 7efd3ec00640 1 -- 172.21.15.19:0/3161294969 wait complete. 2024-03-18T15:15:33.039 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-18T15:15:33.269 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.268+0000 7f0bd7000640 1 -- 172.21.15.19:0/3313319167 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0bd0107d00 msgr2=0x7f0bd0108120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:33.269 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.268+0000 7f0bd7000640 1 --2- 172.21.15.19:0/3313319167 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0bd0107d00 0x7f0bd0108120 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f0bc4009900 tx=0x7f0bc402bc30 comp rx=0 tx=0).stop 2024-03-18T15:15:33.269 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.268+0000 7f0bd7000640 1 -- 172.21.15.19:0/3313319167 shutdown_connections 2024-03-18T15:15:33.269 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.268+0000 7f0bd7000640 1 --2- 172.21.15.19:0/3313319167 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0bd01098b0 0x7f0bd0110140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:33.269 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.268+0000 7f0bd7000640 1 --2- 172.21.15.19:0/3313319167 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0bd0108ef0 0x7f0bd0109370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:33.269 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.268+0000 7f0bd7000640 1 --2- 172.21.15.19:0/3313319167 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0bd0107d00 0x7f0bd0108120 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:33.269 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.268+0000 7f0bd7000640 1 -- 172.21.15.19:0/3313319167 >> 172.21.15.19:0/3313319167 conn(0x7f0bd0076cb0 msgr2=0x7f0bd00790d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:33.269 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.269+0000 7f0bd7000640 1 -- 172.21.15.19:0/3313319167 shutdown_connections 2024-03-18T15:15:33.270 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.269+0000 7f0bd7000640 1 -- 172.21.15.19:0/3313319167 wait complete. 2024-03-18T15:15:33.272 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.270+0000 7f0bd7000640 1 Processor -- start 2024-03-18T15:15:33.272 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.270+0000 7f0bd7000640 1 -- start start 2024-03-18T15:15:33.272 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.270+0000 7f0bd7000640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0bd0107d00 0x7f0bd019e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:33.272 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.270+0000 7f0bd7000640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0bd0108ef0 0x7f0bd019e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:33.272 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.270+0000 7f0bd7000640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0bd01098b0 0x7f0bd01a55c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:33.272 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.270+0000 7f0bd7000640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f0bd0113970 con 0x7f0bd0108ef0 2024-03-18T15:15:33.272 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.270+0000 7f0bd7000640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f0bd01137f0 con 0x7f0bd01098b0 2024-03-18T15:15:33.272 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.270+0000 7f0bd7000640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f0bd0113af0 con 0x7f0bd0107d00 2024-03-18T15:15:33.272 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.271+0000 7f0bd5c00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0bd0107d00 0x7f0bd019e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:33.272 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.271+0000 7f0bd6600640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0bd01098b0 0x7f0bd01a55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:33.273 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.271+0000 7f0bd5c00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0bd0107d00 0x7f0bd019e010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.19:38182/0 (socket says 172.21.15.19:38182) 2024-03-18T15:15:33.273 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.271+0000 7f0bd6600640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0bd01098b0 0x7f0bd01a55c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:60998/0 (socket says 172.21.15.19:60998) 2024-03-18T15:15:33.273 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.271+0000 7f0bd5c00640 1 -- 172.21.15.19:0/433459453 learned_addr learned my addr 172.21.15.19:0/433459453 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:15:33.273 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.271+0000 7f0bd5200640 1 --2- 172.21.15.19:0/433459453 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0bd0108ef0 0x7f0bd019e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:33.273 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.271+0000 7f0bd6600640 1 -- 172.21.15.19:0/433459453 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0bd0107d00 msgr2=0x7f0bd019e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:33.273 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.271+0000 7f0bd6600640 1 --2- 172.21.15.19:0/433459453 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0bd0107d00 0x7f0bd019e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:33.273 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.271+0000 7f0bd6600640 1 -- 172.21.15.19:0/433459453 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0bd0108ef0 msgr2=0x7f0bd019e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:33.273 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.271+0000 7f0bd6600640 1 --2- 172.21.15.19:0/433459453 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0bd0108ef0 0x7f0bd019e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:33.273 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.271+0000 7f0bd6600640 1 -- 172.21.15.19:0/433459453 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bd01a5c30 con 0x7f0bd01098b0 2024-03-18T15:15:33.273 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.271+0000 7f0bd6600640 1 --2- 172.21.15.19:0/433459453 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0bd01098b0 0x7f0bd01a55c0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f0bc8009fd0 tx=0x7f0bc800ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:33.273 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.272+0000 7f0bce000640 1 -- 172.21.15.19:0/433459453 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bc800ce20 con 0x7f0bd01098b0 2024-03-18T15:15:33.273 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.272+0000 7f0bce000640 1 -- 172.21.15.19:0/433459453 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0bc8005020 con 0x7f0bd01098b0 2024-03-18T15:15:33.273 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.272+0000 7f0bce000640 1 -- 172.21.15.19:0/433459453 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bc8010720 con 0x7f0bd01098b0 2024-03-18T15:15:33.274 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.272+0000 7f0bd7000640 1 -- 172.21.15.19:0/433459453 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0bd01a5f20 con 0x7f0bd01098b0 2024-03-18T15:15:33.274 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.272+0000 7f0bd7000640 1 -- 172.21.15.19:0/433459453 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0bd01a6390 con 0x7f0bd01098b0 2024-03-18T15:15:33.274 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.273+0000 7f0bd7000640 1 -- 172.21.15.19:0/433459453 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0bd010f020 con 0x7f0bd01098b0 2024-03-18T15:15:33.275 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.275+0000 7f0bce000640 1 -- 172.21.15.19:0/433459453 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f0bc8005540 con 0x7f0bd01098b0 2024-03-18T15:15:33.276 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.276+0000 7f0bce000640 1 --2- 172.21.15.19:0/433459453 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f0bb80a2880 0x7f0bb80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:33.276 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.276+0000 7f0bce000640 1 -- 172.21.15.19:0/433459453 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f0bc80cf3f0 con 0x7f0bd01098b0 2024-03-18T15:15:33.276 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.276+0000 7f0bd5c00640 1 --2- 172.21.15.19:0/433459453 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f0bb80a2880 0x7f0bb80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:33.277 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.277+0000 7f0bd5c00640 1 --2- 172.21.15.19:0/433459453 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f0bb80a2880 0x7f0bb80a4d40 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f0bc402c8e0 tx=0x7f0bc4005b50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:33.280 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.280+0000 7f0bce000640 1 -- 172.21.15.19:0/433459453 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0bc809c1f0 con 0x7f0bd01098b0 2024-03-18T15:15:33.482 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.482+0000 7f0bd7000640 1 -- 172.21.15.19:0/433459453 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f0bd00623e0 con 0x7f0bd01098b0 2024-03-18T15:15:33.483 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.483+0000 7f0bce000640 1 -- 172.21.15.19:0/433459453 <== mon.2 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v89) ==== 103+0+11 (secure 0 0 0) 0x7f0bc809baf0 con 0x7f0bd01098b0 2024-03-18T15:15:33.483 INFO:teuthology.orchestra.run.smithi019.stdout:pg_num: 32 2024-03-18T15:15:33.486 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.486+0000 7f0bd7000640 1 -- 172.21.15.19:0/433459453 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f0bb80a2880 msgr2=0x7f0bb80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:33.486 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.486+0000 7f0bd7000640 1 --2- 172.21.15.19:0/433459453 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f0bb80a2880 0x7f0bb80a4d40 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f0bc402c8e0 tx=0x7f0bc4005b50 comp rx=0 tx=0).stop 2024-03-18T15:15:33.486 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.486+0000 7f0bd7000640 1 -- 172.21.15.19:0/433459453 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0bd01098b0 msgr2=0x7f0bd01a55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:33.486 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.486+0000 7f0bd7000640 1 --2- 172.21.15.19:0/433459453 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0bd01098b0 0x7f0bd01a55c0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f0bc8009fd0 tx=0x7f0bc800ef10 comp rx=0 tx=0).stop 2024-03-18T15:15:33.486 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.486+0000 7f0bd7000640 1 -- 172.21.15.19:0/433459453 shutdown_connections 2024-03-18T15:15:33.487 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.486+0000 7f0bd7000640 1 --2- 172.21.15.19:0/433459453 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f0bb80a2880 0x7f0bb80a4d40 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:33.487 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.486+0000 7f0bd7000640 1 --2- 172.21.15.19:0/433459453 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0bd01098b0 0x7f0bd01a55c0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:33.487 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.486+0000 7f0bd7000640 1 --2- 172.21.15.19:0/433459453 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0bd0108ef0 0x7f0bd019e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:33.487 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.486+0000 7f0bd7000640 1 --2- 172.21.15.19:0/433459453 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0bd0107d00 0x7f0bd019e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:33.487 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.486+0000 7f0bd7000640 1 -- 172.21.15.19:0/433459453 >> 172.21.15.19:0/433459453 conn(0x7f0bd0076cb0 msgr2=0x7f0bd010c1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:33.487 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.486+0000 7f0bd7000640 1 -- 172.21.15.19:0/433459453 shutdown_connections 2024-03-18T15:15:33.487 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.486+0000 7f0bd7000640 1 -- 172.21.15.19:0/433459453 wait complete. 2024-03-18T15:15:33.505 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-18T15:15:33.687 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:33 smithi019 ceph-mon[31522]: pgmap v328: 129 pgs: 129 active+clean; 581 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-18T15:15:33.688 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:33 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/1388505458' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T15:15:33.688 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:33 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/3161294969' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T15:15:33.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:33 smithi104 ceph-mon[36070]: pgmap v328: 129 pgs: 129 active+clean; 581 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-18T15:15:33.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:33 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/1388505458' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T15:15:33.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:33 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/3161294969' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T15:15:33.723 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.722+0000 7f7eefc00640 1 -- 172.21.15.19:0/1737427977 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7ee8108ff0 msgr2=0x7f7ee8109470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:33.723 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.722+0000 7f7eefc00640 1 --2- 172.21.15.19:0/1737427977 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7ee8108ff0 0x7f7ee8109470 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f7ed0009900 tx=0x7f7ed002bc30 comp rx=0 tx=0).stop 2024-03-18T15:15:33.723 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.722+0000 7f7eefc00640 1 -- 172.21.15.19:0/1737427977 shutdown_connections 2024-03-18T15:15:33.723 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.722+0000 7f7eefc00640 1 --2- 172.21.15.19:0/1737427977 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7ee81099b0 0x7f7ee8110240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:33.723 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.722+0000 7f7eefc00640 1 --2- 172.21.15.19:0/1737427977 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7ee8108ff0 0x7f7ee8109470 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:33.723 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.722+0000 7f7eefc00640 1 --2- 172.21.15.19:0/1737427977 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7ee8107e00 0x7f7ee8108220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:33.723 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.722+0000 7f7eefc00640 1 -- 172.21.15.19:0/1737427977 >> 172.21.15.19:0/1737427977 conn(0x7f7ee8076da0 msgr2=0x7f7ee80791e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:33.724 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.722+0000 7f7eefc00640 1 -- 172.21.15.19:0/1737427977 shutdown_connections 2024-03-18T15:15:33.724 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.722+0000 7f7eefc00640 1 -- 172.21.15.19:0/1737427977 wait complete. 2024-03-18T15:15:33.726 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.723+0000 7f7eefc00640 1 Processor -- start 2024-03-18T15:15:33.726 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.723+0000 7f7eefc00640 1 -- start start 2024-03-18T15:15:33.726 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.724+0000 7f7eefc00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7ee8107e00 0x7f7ee819e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:33.726 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.724+0000 7f7eefc00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7ee8108ff0 0x7f7ee819e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:33.726 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.724+0000 7f7eefc00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7ee81099b0 0x7f7ee81a5690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:33.726 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.724+0000 7f7eefc00640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f7ee8113ad0 con 0x7f7ee81099b0 2024-03-18T15:15:33.726 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.724+0000 7f7eefc00640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7ee8113950 con 0x7f7ee8107e00 2024-03-18T15:15:33.726 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.724+0000 7f7eefc00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f7ee8113c50 con 0x7f7ee8108ff0 2024-03-18T15:15:33.726 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.724+0000 7f7eed200640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7ee8107e00 0x7f7ee819e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:33.727 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.724+0000 7f7eed200640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7ee8107e00 0x7f7ee819e0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:32778/0 (socket says 172.21.15.19:32778) 2024-03-18T15:15:33.727 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.724+0000 7f7eed200640 1 -- 172.21.15.19:0/3952527702 learned_addr learned my addr 172.21.15.19:0/3952527702 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:15:33.727 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.724+0000 7f7eedc00640 1 --2- 172.21.15.19:0/3952527702 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7ee81099b0 0x7f7ee81a5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:33.727 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.724+0000 7f7ee7e00640 1 --2- 172.21.15.19:0/3952527702 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7ee8108ff0 0x7f7ee819e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:33.727 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.724+0000 7f7eed200640 1 -- 172.21.15.19:0/3952527702 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7ee8108ff0 msgr2=0x7f7ee819e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:33.727 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.724+0000 7f7eed200640 1 --2- 172.21.15.19:0/3952527702 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7ee8108ff0 0x7f7ee819e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:33.727 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.725+0000 7f7eed200640 1 -- 172.21.15.19:0/3952527702 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7ee81099b0 msgr2=0x7f7ee81a5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:33.727 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.725+0000 7f7eed200640 1 --2- 172.21.15.19:0/3952527702 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7ee81099b0 0x7f7ee81a5690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:33.727 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.725+0000 7f7eed200640 1 -- 172.21.15.19:0/3952527702 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ee81a5d00 con 0x7f7ee8107e00 2024-03-18T15:15:33.727 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.725+0000 7f7ee7e00640 1 --2- 172.21.15.19:0/3952527702 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7ee8108ff0 0x7f7ee819e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T15:15:33.727 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.725+0000 7f7eed200640 1 --2- 172.21.15.19:0/3952527702 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7ee8107e00 0x7f7ee819e0e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f7ed800e9c0 tx=0x7f7ed800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:33.727 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.725+0000 7f7ee5600640 1 -- 172.21.15.19:0/3952527702 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ed800cd80 con 0x7f7ee8107e00 2024-03-18T15:15:33.727 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.725+0000 7f7ee5600640 1 -- 172.21.15.19:0/3952527702 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ed80035f0 con 0x7f7ee8107e00 2024-03-18T15:15:33.728 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.725+0000 7f7ee5600640 1 -- 172.21.15.19:0/3952527702 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ed8010690 con 0x7f7ee8107e00 2024-03-18T15:15:33.728 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.725+0000 7f7eefc00640 1 -- 172.21.15.19:0/3952527702 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ee81a5ff0 con 0x7f7ee8107e00 2024-03-18T15:15:33.728 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.725+0000 7f7eefc00640 1 -- 172.21.15.19:0/3952527702 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ee81a6460 con 0x7f7ee8107e00 2024-03-18T15:15:33.728 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.726+0000 7f7eefc00640 1 -- 172.21.15.19:0/3952527702 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ee810f0c0 con 0x7f7ee8107e00 2024-03-18T15:15:33.728 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.728+0000 7f7ee5600640 1 -- 172.21.15.19:0/3952527702 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f7ed80037a0 con 0x7f7ee8107e00 2024-03-18T15:15:33.729 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.729+0000 7f7ee5600640 1 --2- 172.21.15.19:0/3952527702 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f7ecc0a2880 0x7f7ecc0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:33.729 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.729+0000 7f7ee5600640 1 -- 172.21.15.19:0/3952527702 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f7ed80cf0e0 con 0x7f7ee8107e00 2024-03-18T15:15:33.729 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.730+0000 7f7ee7e00640 1 --2- 172.21.15.19:0/3952527702 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f7ecc0a2880 0x7f7ecc0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:33.730 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.731+0000 7f7ee7e00640 1 --2- 172.21.15.19:0/3952527702 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f7ecc0a2880 0x7f7ecc0a4d40 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f7ed002c240 tx=0x7f7ed002df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:33.733 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.733+0000 7f7ee5600640 1 -- 172.21.15.19:0/3952527702 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7ed809bee0 con 0x7f7ee8107e00 2024-03-18T15:15:33.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:33 smithi033 ceph-mon[36648]: pgmap v328: 129 pgs: 129 active+clean; 581 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-18T15:15:33.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:33 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/1388505458' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T15:15:33.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:33 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/3161294969' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T15:15:33.948 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.948+0000 7f7eefc00640 1 -- 172.21.15.19:0/3952527702 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f7ee802d1b0 con 0x7f7ee8107e00 2024-03-18T15:15:33.949 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.949+0000 7f7ee5600640 1 -- 172.21.15.19:0/3952527702 <== mon.2 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v89) ==== 81+0+591 (secure 0 0 0) 0x7f7ed809b7e0 con 0x7f7ee8107e00 2024-03-18T15:15:33.949 INFO:teuthology.orchestra.run.smithi019.stdout:172.21.15.19:0/3860783274 2024-03-19T15:07:14.150983+0000 2024-03-18T15:15:33.949 INFO:teuthology.orchestra.run.smithi019.stdout:172.21.15.19:0/2737223213 2024-03-19T15:07:14.150983+0000 2024-03-18T15:15:33.949 INFO:teuthology.orchestra.run.smithi019.stdout:172.21.15.19:0/176699025 2024-03-19T15:06:38.688405+0000 2024-03-18T15:15:33.949 INFO:teuthology.orchestra.run.smithi019.stdout:172.21.15.19:0/2316969005 2024-03-19T15:07:14.150983+0000 2024-03-18T15:15:33.949 INFO:teuthology.orchestra.run.smithi019.stdout:172.21.15.19:6801/2776008962 2024-03-19T15:07:14.150983+0000 2024-03-18T15:15:33.949 INFO:teuthology.orchestra.run.smithi019.stdout:172.21.15.19:0/2793980440 2024-03-19T15:06:38.688405+0000 2024-03-18T15:15:33.950 INFO:teuthology.orchestra.run.smithi019.stdout:172.21.15.19:6800/3264337803 2024-03-19T15:06:38.688405+0000 2024-03-18T15:15:33.950 INFO:teuthology.orchestra.run.smithi019.stdout:172.21.15.19:6801/3264337803 2024-03-19T15:06:38.688405+0000 2024-03-18T15:15:33.950 INFO:teuthology.orchestra.run.smithi019.stdout:172.21.15.19:0/3166705295 2024-03-19T15:06:38.688405+0000 2024-03-18T15:15:33.950 INFO:teuthology.orchestra.run.smithi019.stdout:172.21.15.19:6800/2776008962 2024-03-19T15:07:14.150983+0000 2024-03-18T15:15:33.950 INFO:teuthology.orchestra.run.smithi019.stderr:listed 10 entries 2024-03-18T15:15:33.951 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.951+0000 7f7eefc00640 1 -- 172.21.15.19:0/3952527702 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f7ecc0a2880 msgr2=0x7f7ecc0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:33.951 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.951+0000 7f7eefc00640 1 --2- 172.21.15.19:0/3952527702 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f7ecc0a2880 0x7f7ecc0a4d40 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f7ed002c240 tx=0x7f7ed002df90 comp rx=0 tx=0).stop 2024-03-18T15:15:33.951 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.951+0000 7f7eefc00640 1 -- 172.21.15.19:0/3952527702 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7ee8107e00 msgr2=0x7f7ee819e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:33.951 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.951+0000 7f7eefc00640 1 --2- 172.21.15.19:0/3952527702 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7ee8107e00 0x7f7ee819e0e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f7ed800e9c0 tx=0x7f7ed800ee80 comp rx=0 tx=0).stop 2024-03-18T15:15:33.952 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.951+0000 7f7eefc00640 1 -- 172.21.15.19:0/3952527702 shutdown_connections 2024-03-18T15:15:33.952 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.951+0000 7f7eefc00640 1 --2- 172.21.15.19:0/3952527702 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7ee81099b0 0x7f7ee81a5690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:33.952 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.951+0000 7f7eefc00640 1 --2- 172.21.15.19:0/3952527702 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7ee8108ff0 0x7f7ee819e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:33.952 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.951+0000 7f7eefc00640 1 --2- 172.21.15.19:0/3952527702 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f7ecc0a2880 0x7f7ecc0a4d40 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:33.953 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.951+0000 7f7eefc00640 1 --2- 172.21.15.19:0/3952527702 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7ee8107e00 0x7f7ee819e0e0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:33.953 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.951+0000 7f7eefc00640 1 -- 172.21.15.19:0/3952527702 >> 172.21.15.19:0/3952527702 conn(0x7f7ee8076da0 msgr2=0x7f7ee810c290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:33.953 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.952+0000 7f7eefc00640 1 -- 172.21.15.19:0/3952527702 shutdown_connections 2024-03-18T15:15:33.953 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:33.952+0000 7f7eefc00640 1 -- 172.21.15.19:0/3952527702 wait complete. 2024-03-18T15:15:33.970 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi019.front.sepia.ceph.com... 2024-03-18T15:15:33.971 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T15:15:33.971 DEBUG:teuthology.orchestra.run.smithi019:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-03-18T15:15:34.055 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi019.stderr:ceph-fuse[75442]: fuse finished with error 0 and tester_r 0 2024-03-18T15:15:34.062 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T15:15:34.658 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:34 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/433459453' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T15:15:34.658 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:34 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/3952527702' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-18T15:15:34.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:34 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/433459453' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T15:15:34.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:34 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/3952527702' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-18T15:15:34.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:34 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/433459453' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T15:15:34.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:34 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/3952527702' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-18T15:15:35.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:35 smithi104 ceph-mon[36070]: pgmap v329: 129 pgs: 129 active+clean; 581 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-18T15:15:35.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:35 smithi019 ceph-mon[31522]: pgmap v329: 129 pgs: 129 active+clean; 581 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-18T15:15:35.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:35 smithi033 ceph-mon[36648]: pgmap v329: 129 pgs: 129 active+clean; 581 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-18T15:15:37.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:37 smithi104 ceph-mon[36070]: pgmap v330: 129 pgs: 129 active+clean; 581 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-18T15:15:37.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:37 smithi019 ceph-mon[31522]: pgmap v330: 129 pgs: 129 active+clean; 581 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-18T15:15:37.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:37 smithi033 ceph-mon[36648]: pgmap v330: 129 pgs: 129 active+clean; 581 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-18T15:15:39.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:39 smithi104 ceph-mon[36070]: pgmap v331: 129 pgs: 129 active+clean; 581 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-18T15:15:39.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:39 smithi019 ceph-mon[31522]: pgmap v331: 129 pgs: 129 active+clean; 581 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-18T15:15:39.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:39 smithi033 ceph-mon[36648]: pgmap v331: 129 pgs: 129 active+clean; 581 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-18T15:15:40.164 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi019.front.sepia.ceph.com 2024-03-18T15:15:40.164 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T15:15:40.165 DEBUG:teuthology.orchestra.run.smithi019:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-03-18T15:15:40.235 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T15:15:40.463 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:40.462+0000 7f0c35c00640 1 -- 172.21.15.19:0/2103409053 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0c30104c00 msgr2=0x7f0c30106ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:40.463 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:40.462+0000 7f0c35c00640 1 --2- 172.21.15.19:0/2103409053 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0c30104c00 0x7f0c30106ff0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f0c1c009900 tx=0x7f0c1c02bc40 comp rx=0 tx=0).stop 2024-03-18T15:15:40.463 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:40.462+0000 7f0c35c00640 1 -- 172.21.15.19:0/2103409053 shutdown_connections 2024-03-18T15:15:40.463 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:40.462+0000 7f0c35c00640 1 --2- 172.21.15.19:0/2103409053 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0c30107530 0x7f0c30109920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:40.463 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:40.462+0000 7f0c35c00640 1 --2- 172.21.15.19:0/2103409053 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0c30104c00 0x7f0c30106ff0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:40.463 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:40.462+0000 7f0c35c00640 1 --2- 172.21.15.19:0/2103409053 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0c301022d0 0x7f0c301046c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:40.463 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:40.462+0000 7f0c35c00640 1 -- 172.21.15.19:0/2103409053 >> 172.21.15.19:0/2103409053 conn(0x7f0c300fbc40 msgr2=0x7f0c300fe080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:40.463 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:40.462+0000 7f0c35c00640 1 -- 172.21.15.19:0/2103409053 shutdown_connections 2024-03-18T15:15:40.463 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:40.462+0000 7f0c35c00640 1 -- 172.21.15.19:0/2103409053 wait complete. 2024-03-18T15:15:40.465 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:40.463+0000 7f0c35c00640 1 Processor -- start 2024-03-18T15:15:40.466 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:40.463+0000 7f0c35c00640 1 -- start start 2024-03-18T15:15:40.466 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:40.464+0000 7f0c35c00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0c301022d0 0x7f0c30101c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:40.466 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:40.464+0000 7f0c35c00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0c30104c00 0x7f0c30100280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:40.466 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:40.464+0000 7f0c35c00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0c30107530 0x7f0c301007c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:40.466 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:40.464+0000 7f0c35c00640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f0c3010d470 con 0x7f0c30107530 2024-03-18T15:15:40.466 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:40.464+0000 7f0c35c00640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f0c3010d2f0 con 0x7f0c30104c00 2024-03-18T15:15:40.466 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:40.464+0000 7f0c35c00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f0c3010d5f0 con 0x7f0c301022d0 2024-03-18T15:15:40.466 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:40.464+0000 7f0c2f400640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0c30107530 0x7f0c301007c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:40.466 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:40.464+0000 7f0c2e000640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0c30104c00 0x7f0c30100280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:40.466 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:40.464+0000 7f0c2e000640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0c30104c00 0x7f0c30100280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:32790/0 (socket says 172.21.15.19:32790) 2024-03-18T15:15:40.466 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:40.464+0000 7f0c2e000640 1 -- 172.21.15.19:0/4151512721 learned_addr learned my addr 172.21.15.19:0/4151512721 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:15:40.466 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:40.464+0000 7f0c2ea00640 1 --2- 172.21.15.19:0/4151512721 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0c301022d0 0x7f0c30101c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:40.467 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:40.464+0000 7f0c2e000640 1 -- 172.21.15.19:0/4151512721 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0c301022d0 msgr2=0x7f0c30101c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:40.467 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:40.465+0000 7f0c2e000640 1 --2- 172.21.15.19:0/4151512721 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0c301022d0 0x7f0c30101c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:40.467 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:40.465+0000 7f0c2e000640 1 -- 172.21.15.19:0/4151512721 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0c30107530 msgr2=0x7f0c301007c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:40.467 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:40.465+0000 7f0c2e000640 1 --2- 172.21.15.19:0/4151512721 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0c30107530 0x7f0c301007c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:40.467 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:40.465+0000 7f0c2e000640 1 -- 172.21.15.19:0/4151512721 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c30101050 con 0x7f0c30104c00 2024-03-18T15:15:40.467 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:40.465+0000 7f0c2f400640 1 --2- 172.21.15.19:0/4151512721 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0c30107530 0x7f0c301007c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:15:40.467 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:40.465+0000 7f0c2ea00640 1 --2- 172.21.15.19:0/4151512721 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0c301022d0 0x7f0c30101c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:15:40.467 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:40.465+0000 7f0c2e000640 1 --2- 172.21.15.19:0/4151512721 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0c30104c00 0x7f0c30100280 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f0c1c02efd0 tx=0x7f0c1c02c660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:40.467 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:40.465+0000 7f0c27400640 1 -- 172.21.15.19:0/4151512721 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c1c041070 con 0x7f0c30104c00 2024-03-18T15:15:40.467 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:40.465+0000 7f0c27400640 1 -- 172.21.15.19:0/4151512721 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0c1c004450 con 0x7f0c30104c00 2024-03-18T15:15:40.467 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:40.465+0000 7f0c27400640 1 -- 172.21.15.19:0/4151512721 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c1c003e50 con 0x7f0c30104c00 2024-03-18T15:15:40.467 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:40.465+0000 7f0c35c00640 1 -- 172.21.15.19:0/4151512721 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c301a1bd0 con 0x7f0c30104c00 2024-03-18T15:15:40.467 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:40.465+0000 7f0c35c00640 1 -- 172.21.15.19:0/4151512721 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c301a20a0 con 0x7f0c30104c00 2024-03-18T15:15:40.468 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:40.467+0000 7f0c35c00640 1 -- 172.21.15.19:0/4151512721 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c300fd730 con 0x7f0c30104c00 2024-03-18T15:15:40.468 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:40.469+0000 7f0c27400640 1 -- 172.21.15.19:0/4151512721 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f0c1c044070 con 0x7f0c30104c00 2024-03-18T15:15:40.470 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:40.470+0000 7f0c27400640 1 --2- 172.21.15.19:0/4151512721 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f0c140a2a20 0x7f0c140a4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:40.470 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:40.470+0000 7f0c27400640 1 -- 172.21.15.19:0/4151512721 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f0c1c0f07f0 con 0x7f0c30104c00 2024-03-18T15:15:40.470 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:40.470+0000 7f0c2ea00640 1 --2- 172.21.15.19:0/4151512721 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f0c140a2a20 0x7f0c140a4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:40.471 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:40.471+0000 7f0c2ea00640 1 --2- 172.21.15.19:0/4151512721 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f0c140a2a20 0x7f0c140a4ee0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f0c20009fc0 tx=0x7f0c20005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:40.474 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:40.474+0000 7f0c27400640 1 -- 172.21.15.19:0/4151512721 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0c1c0bcfa0 con 0x7f0c30104c00 2024-03-18T15:15:40.681 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:40.681+0000 7f0c35c00640 1 -- 172.21.15.19:0/4151512721 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0c30062480 con 0x7f0c30104c00 2024-03-18T15:15:40.683 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:40.683+0000 7f0c27400640 1 -- 172.21.15.19:0/4151512721 <== mon.2 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24093 (secure 0 0 0) 0x7f0c1c0c1040 con 0x7f0c30104c00 2024-03-18T15:15:40.684 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-18T15:15:40.684 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":89,"fsid":"d4dc2ba6-e538-11ee-95c9-87774f69a715","created":"2024-03-18T15:05:57.538551+0000","modified":"2024-03-18T15:15:15.582418+0000","last_up_change":"2024-03-18T15:14:12.009522+0000","last_in_change":"2024-03-18T15:13:52.555387+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T15:09:56.843349+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T15:15:08.941047+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T15:15:10.044671+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T15:15:13.621916+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"37b5a5cd-27ae-4e77-9b05-774b817b2ee9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6802","nonce":402333948},{"type":"v1","addr":"172.21.15.19:6803","nonce":402333948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6804","nonce":402333948},{"type":"v1","addr":"172.21.15.19:6805","nonce":402333948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6808","nonce":402333948},{"type":"v1","addr":"172.21.15.19:6809","nonce":402333948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6806","nonce":402333948},{"type":"v1","addr":"172.21.15.19:6807","nonce":402333948}]},"public_addr":"172.21.15.19:6803/402333948","cluster_addr":"172.21.15.19:6805/402333948","heartbeat_back_addr":"172.21.15.19:6809/402333948","heartbeat_front_addr":"172.21.15.19:6807/402333948","state":["exists","up"]},{"osd":1,"uuid":"3a1134e8-ede7-4dbd-9be1-eefd2cbb43bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1951880113},{"type":"v1","addr":"172.21.15.33:6801","nonce":1951880113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1951880113},{"type":"v1","addr":"172.21.15.33:6803","nonce":1951880113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1951880113},{"type":"v1","addr":"172.21.15.33:6807","nonce":1951880113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1951880113},{"type":"v1","addr":"172.21.15.33:6805","nonce":1951880113}]},"public_addr":"172.21.15.33:6801/1951880113","cluster_addr":"172.21.15.33:6803/1951880113","heartbeat_back_addr":"172.21.15.33:6807/1951880113","heartbeat_front_addr":"172.21.15.33:6805/1951880113","state":["exists","up"]},{"osd":2,"uuid":"f383a164-a3a0-49fd-a9e8-9c2178d8cfb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":1004727949},{"type":"v1","addr":"172.21.15.104:6801","nonce":1004727949}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":1004727949},{"type":"v1","addr":"172.21.15.104:6803","nonce":1004727949}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":1004727949},{"type":"v1","addr":"172.21.15.104:6807","nonce":1004727949}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":1004727949},{"type":"v1","addr":"172.21.15.104:6805","nonce":1004727949}]},"public_addr":"172.21.15.104:6801/1004727949","cluster_addr":"172.21.15.104:6803/1004727949","heartbeat_back_addr":"172.21.15.104:6807/1004727949","heartbeat_front_addr":"172.21.15.104:6805/1004727949","state":["exists","up"]},{"osd":3,"uuid":"c409e116-28bf-45dd-a791-539af22abe1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6810","nonce":3571525854},{"type":"v1","addr":"172.21.15.19:6811","nonce":3571525854}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6812","nonce":3571525854},{"type":"v1","addr":"172.21.15.19:6813","nonce":3571525854}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6816","nonce":3571525854},{"type":"v1","addr":"172.21.15.19:6817","nonce":3571525854}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6814","nonce":3571525854},{"type":"v1","addr":"172.21.15.19:6815","nonce":3571525854}]},"public_addr":"172.21.15.19:6811/3571525854","cluster_addr":"172.21.15.19:6813/3571525854","heartbeat_back_addr":"172.21.15.19:6817/3571525854","heartbeat_front_addr":"172.21.15.19:6815/3571525854","state":["exists","up"]},{"osd":4,"uuid":"696eb719-0b3e-43ae-b5f1-ccd73a3b2843","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":624901749},{"type":"v1","addr":"172.21.15.33:6809","nonce":624901749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":624901749},{"type":"v1","addr":"172.21.15.33:6811","nonce":624901749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":624901749},{"type":"v1","addr":"172.21.15.33:6815","nonce":624901749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":624901749},{"type":"v1","addr":"172.21.15.33:6813","nonce":624901749}]},"public_addr":"172.21.15.33:6809/624901749","cluster_addr":"172.21.15.33:6811/624901749","heartbeat_back_addr":"172.21.15.33:6815/624901749","heartbeat_front_addr":"172.21.15.33:6813/624901749","state":["exists","up"]},{"osd":5,"uuid":"21789779-8bb9-42c7-b9ee-437985f11e3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":1258208297},{"type":"v1","addr":"172.21.15.104:6809","nonce":1258208297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":1258208297},{"type":"v1","addr":"172.21.15.104:6811","nonce":1258208297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":1258208297},{"type":"v1","addr":"172.21.15.104:6815","nonce":1258208297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":1258208297},{"type":"v1","addr":"172.21.15.104:6813","nonce":1258208297}]},"public_addr":"172.21.15.104:6809/1258208297","cluster_addr":"172.21.15.104:6811/1258208297","heartbeat_back_addr":"172.21.15.104:6815/1258208297","heartbeat_front_addr":"172.21.15.104:6813/1258208297","state":["exists","up"]},{"osd":6,"uuid":"0e92a2ef-43cd-487c-9203-e219158aef8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6818","nonce":2782482301},{"type":"v1","addr":"172.21.15.19:6819","nonce":2782482301}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6820","nonce":2782482301},{"type":"v1","addr":"172.21.15.19:6821","nonce":2782482301}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6824","nonce":2782482301},{"type":"v1","addr":"172.21.15.19:6825","nonce":2782482301}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6822","nonce":2782482301},{"type":"v1","addr":"172.21.15.19:6823","nonce":2782482301}]},"public_addr":"172.21.15.19:6819/2782482301","cluster_addr":"172.21.15.19:6821/2782482301","heartbeat_back_addr":"172.21.15.19:6825/2782482301","heartbeat_front_addr":"172.21.15.19:6823/2782482301","state":["exists","up"]},{"osd":7,"uuid":"a16e39a1-3685-491d-b376-3e80a2084072","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":4238160988},{"type":"v1","addr":"172.21.15.33:6817","nonce":4238160988}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":4238160988},{"type":"v1","addr":"172.21.15.33:6819","nonce":4238160988}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":4238160988},{"type":"v1","addr":"172.21.15.33:6823","nonce":4238160988}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":4238160988},{"type":"v1","addr":"172.21.15.33:6821","nonce":4238160988}]},"public_addr":"172.21.15.33:6817/4238160988","cluster_addr":"172.21.15.33:6819/4238160988","heartbeat_back_addr":"172.21.15.33:6823/4238160988","heartbeat_front_addr":"172.21.15.33:6821/4238160988","state":["exists","up"]},{"osd":8,"uuid":"d5c3fa92-cd5d-4349-9c7d-d5666bacde16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":1014779813},{"type":"v1","addr":"172.21.15.104:6817","nonce":1014779813}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":1014779813},{"type":"v1","addr":"172.21.15.104:6819","nonce":1014779813}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":1014779813},{"type":"v1","addr":"172.21.15.104:6823","nonce":1014779813}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":1014779813},{"type":"v1","addr":"172.21.15.104:6821","nonce":1014779813}]},"public_addr":"172.21.15.104:6817/1014779813","cluster_addr":"172.21.15.104:6819/1014779813","heartbeat_back_addr":"172.21.15.104:6823/1014779813","heartbeat_front_addr":"172.21.15.104:6821/1014779813","state":["exists","up"]},{"osd":9,"uuid":"eed2e9b2-26d3-4c46-88ec-34b5596fe991","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6826","nonce":2641151009},{"type":"v1","addr":"172.21.15.19:6827","nonce":2641151009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6828","nonce":2641151009},{"type":"v1","addr":"172.21.15.19:6829","nonce":2641151009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6832","nonce":2641151009},{"type":"v1","addr":"172.21.15.19:6833","nonce":2641151009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6830","nonce":2641151009},{"type":"v1","addr":"172.21.15.19:6831","nonce":2641151009}]},"public_addr":"172.21.15.19:6827/2641151009","cluster_addr":"172.21.15.19:6829/2641151009","heartbeat_back_addr":"172.21.15.19:6833/2641151009","heartbeat_front_addr":"172.21.15.19:6831/2641151009","state":["exists","up"]},{"osd":10,"uuid":"f3c35ea5-fbe7-4dc4-93c5-b92d1a714db3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":1346058869},{"type":"v1","addr":"172.21.15.33:6825","nonce":1346058869}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":1346058869},{"type":"v1","addr":"172.21.15.33:6827","nonce":1346058869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":1346058869},{"type":"v1","addr":"172.21.15.33:6831","nonce":1346058869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":1346058869},{"type":"v1","addr":"172.21.15.33:6829","nonce":1346058869}]},"public_addr":"172.21.15.33:6825/1346058869","cluster_addr":"172.21.15.33:6827/1346058869","heartbeat_back_addr":"172.21.15.33:6831/1346058869","heartbeat_front_addr":"172.21.15.33:6829/1346058869","state":["exists","up"]},{"osd":11,"uuid":"c8b51571-f807-4b62-90b6-7529e09e59da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":3759498281},{"type":"v1","addr":"172.21.15.104:6825","nonce":3759498281}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":3759498281},{"type":"v1","addr":"172.21.15.104:6827","nonce":3759498281}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":3759498281},{"type":"v1","addr":"172.21.15.104:6831","nonce":3759498281}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":3759498281},{"type":"v1","addr":"172.21.15.104:6829","nonce":3759498281}]},"public_addr":"172.21.15.104:6825/3759498281","cluster_addr":"172.21.15.104:6827/3759498281","heartbeat_back_addr":"172.21.15.104:6831/3759498281","heartbeat_front_addr":"172.21.15.104:6829/3759498281","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:09:05.267795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:09:30.234068+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:09:53.870232+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:10:26.471296+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:10:53.180341+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:11:17.801361+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:11:51.364816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:12:18.306408+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:12:43.978744+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:13:18.213964+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:13:45.437544+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:14:10.148580+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.19:0/3860783274":"2024-03-19T15:07:14.150983+0000","172.21.15.19:0/2737223213":"2024-03-19T15:07:14.150983+0000","172.21.15.19:0/176699025":"2024-03-19T15:06:38.688405+0000","172.21.15.19:0/2316969005":"2024-03-19T15:07:14.150983+0000","172.21.15.19:6801/2776008962":"2024-03-19T15:07:14.150983+0000","172.21.15.19:0/2793980440":"2024-03-19T15:06:38.688405+0000","172.21.15.19:6800/3264337803":"2024-03-19T15:06:38.688405+0000","172.21.15.19:6801/3264337803":"2024-03-19T15:06:38.688405+0000","172.21.15.19:0/3166705295":"2024-03-19T15:06:38.688405+0000","172.21.15.19:6800/2776008962":"2024-03-19T15:07:14.150983+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T15:15:40.687 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:40.686+0000 7f0c35c00640 1 -- 172.21.15.19:0/4151512721 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f0c140a2a20 msgr2=0x7f0c140a4ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:40.687 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:40.686+0000 7f0c35c00640 1 --2- 172.21.15.19:0/4151512721 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f0c140a2a20 0x7f0c140a4ee0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f0c20009fc0 tx=0x7f0c20005f90 comp rx=0 tx=0).stop 2024-03-18T15:15:40.687 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:40.686+0000 7f0c35c00640 1 -- 172.21.15.19:0/4151512721 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0c30104c00 msgr2=0x7f0c30100280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:40.687 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:40.686+0000 7f0c35c00640 1 --2- 172.21.15.19:0/4151512721 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0c30104c00 0x7f0c30100280 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f0c1c02efd0 tx=0x7f0c1c02c660 comp rx=0 tx=0).stop 2024-03-18T15:15:40.687 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:40.686+0000 7f0c35c00640 1 -- 172.21.15.19:0/4151512721 shutdown_connections 2024-03-18T15:15:40.687 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:40.686+0000 7f0c35c00640 1 --2- 172.21.15.19:0/4151512721 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0c30107530 0x7f0c301007c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:40.687 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:40.686+0000 7f0c35c00640 1 --2- 172.21.15.19:0/4151512721 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f0c140a2a20 0x7f0c140a4ee0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:40.687 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:40.686+0000 7f0c35c00640 1 --2- 172.21.15.19:0/4151512721 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0c30104c00 0x7f0c30100280 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:40.687 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:40.686+0000 7f0c35c00640 1 --2- 172.21.15.19:0/4151512721 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0c301022d0 0x7f0c30101c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:40.687 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:40.686+0000 7f0c35c00640 1 -- 172.21.15.19:0/4151512721 >> 172.21.15.19:0/4151512721 conn(0x7f0c300fbc40 msgr2=0x7f0c30102b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:40.687 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:40.687+0000 7f0c35c00640 1 -- 172.21.15.19:0/4151512721 shutdown_connections 2024-03-18T15:15:40.687 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:40.687+0000 7f0c35c00640 1 -- 172.21.15.19:0/4151512721 wait complete. 2024-03-18T15:15:40.709 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T15:15:40.931 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:40.931+0000 7fbe2fe00640 1 -- 172.21.15.19:0/625940097 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbe30108e10 msgr2=0x7fbe30109290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:40.931 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:40.931+0000 7fbe2fe00640 1 --2- 172.21.15.19:0/625940097 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbe30108e10 0x7fbe30109290 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fbe18009900 tx=0x7fbe1802bc30 comp rx=0 tx=0).stop 2024-03-18T15:15:40.932 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:40.931+0000 7fbe2fe00640 1 -- 172.21.15.19:0/625940097 shutdown_connections 2024-03-18T15:15:40.932 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:40.931+0000 7fbe2fe00640 1 --2- 172.21.15.19:0/625940097 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbe301097d0 0x7fbe30110060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:40.932 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:40.931+0000 7fbe2fe00640 1 --2- 172.21.15.19:0/625940097 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbe30108e10 0x7fbe30109290 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:40.932 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:40.931+0000 7fbe2fe00640 1 --2- 172.21.15.19:0/625940097 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbe30107c20 0x7fbe30108040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:40.932 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:40.931+0000 7fbe2fe00640 1 -- 172.21.15.19:0/625940097 >> 172.21.15.19:0/625940097 conn(0x7fbe30076bb0 msgr2=0x7fbe30078ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:40.932 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:40.931+0000 7fbe2fe00640 1 -- 172.21.15.19:0/625940097 shutdown_connections 2024-03-18T15:15:40.932 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:40.931+0000 7fbe2fe00640 1 -- 172.21.15.19:0/625940097 wait complete. 2024-03-18T15:15:40.932 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:40.933+0000 7fbe2fe00640 1 Processor -- start 2024-03-18T15:15:40.933 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:40.933+0000 7fbe2fe00640 1 -- start start 2024-03-18T15:15:40.935 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:40.933+0000 7fbe2fe00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbe30107c20 0x7fbe3019dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:40.935 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:40.933+0000 7fbe2fe00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbe30108e10 0x7fbe3019e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:40.935 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:40.933+0000 7fbe2fe00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbe301097d0 0x7fbe301a5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:40.935 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:40.933+0000 7fbe2fe00640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fbe301138f0 con 0x7fbe301097d0 2024-03-18T15:15:40.935 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:40.933+0000 7fbe2fe00640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fbe30113770 con 0x7fbe30108e10 2024-03-18T15:15:40.935 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:40.933+0000 7fbe2fe00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fbe30113a70 con 0x7fbe30107c20 2024-03-18T15:15:40.935 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:40.934+0000 7fbe26000640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbe30108e10 0x7fbe3019e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:40.936 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:40.934+0000 7fbe26000640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbe30108e10 0x7fbe3019e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:32806/0 (socket says 172.21.15.19:32806) 2024-03-18T15:15:40.936 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:40.934+0000 7fbe2f400640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbe301097d0 0x7fbe301a5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:40.936 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:40.934+0000 7fbe2ea00640 1 --2- 172.21.15.19:0/814407610 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbe30107c20 0x7fbe3019dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:40.936 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:40.934+0000 7fbe26000640 1 -- 172.21.15.19:0/814407610 learned_addr learned my addr 172.21.15.19:0/814407610 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:15:40.936 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:40.934+0000 7fbe26000640 1 -- 172.21.15.19:0/814407610 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbe30107c20 msgr2=0x7fbe3019dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:40.936 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:40.934+0000 7fbe26000640 1 --2- 172.21.15.19:0/814407610 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbe30107c20 0x7fbe3019dee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:40.936 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:40.934+0000 7fbe26000640 1 -- 172.21.15.19:0/814407610 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbe301097d0 msgr2=0x7fbe301a5490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:40.936 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:40.934+0000 7fbe26000640 1 --2- 172.21.15.19:0/814407610 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbe301097d0 0x7fbe301a5490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:40.936 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:40.934+0000 7fbe26000640 1 -- 172.21.15.19:0/814407610 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe301a5b00 con 0x7fbe30108e10 2024-03-18T15:15:40.936 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:40.934+0000 7fbe2f400640 1 --2- 172.21.15.19:0/814407610 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbe301097d0 0x7fbe301a5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:15:40.936 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:40.935+0000 7fbe26000640 1 --2- 172.21.15.19:0/814407610 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbe30108e10 0x7fbe3019e420 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fbe18000c00 tx=0x7fbe1802c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:40.936 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:40.935+0000 7fbe27e00640 1 -- 172.21.15.19:0/814407610 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe18043020 con 0x7fbe30108e10 2024-03-18T15:15:40.936 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:40.935+0000 7fbe27e00640 1 -- 172.21.15.19:0/814407610 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbe180054e0 con 0x7fbe30108e10 2024-03-18T15:15:40.936 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:40.935+0000 7fbe27e00640 1 -- 172.21.15.19:0/814407610 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe18003c40 con 0x7fbe30108e10 2024-03-18T15:15:40.937 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:40.935+0000 7fbe2fe00640 1 -- 172.21.15.19:0/814407610 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe301a5d90 con 0x7fbe30108e10 2024-03-18T15:15:40.937 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:40.935+0000 7fbe2fe00640 1 -- 172.21.15.19:0/814407610 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe301a6280 con 0x7fbe30108e10 2024-03-18T15:15:40.937 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:40.936+0000 7fbe2fe00640 1 -- 172.21.15.19:0/814407610 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe3010ef40 con 0x7fbe30108e10 2024-03-18T15:15:40.938 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:40.938+0000 7fbe27e00640 1 -- 172.21.15.19:0/814407610 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fbe18046020 con 0x7fbe30108e10 2024-03-18T15:15:40.940 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:40.940+0000 7fbe27e00640 1 --2- 172.21.15.19:0/814407610 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fbe140a2950 0x7fbe140a4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:40.940 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:40.940+0000 7fbe27e00640 1 -- 172.21.15.19:0/814407610 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fbe180f1180 con 0x7fbe30108e10 2024-03-18T15:15:40.940 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:40.940+0000 7fbe2ea00640 1 --2- 172.21.15.19:0/814407610 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fbe140a2950 0x7fbe140a4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:40.940 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:40.941+0000 7fbe2ea00640 1 --2- 172.21.15.19:0/814407610 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fbe140a2950 0x7fbe140a4e10 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fbe30108c70 tx=0x7fbe20009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:40.942 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:40.942+0000 7fbe27e00640 1 -- 172.21.15.19:0/814407610 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fbe1804d020 con 0x7fbe30108e10 2024-03-18T15:15:41.147 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:41.147+0000 7fbe2fe00640 1 -- 172.21.15.19:0/814407610 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbe3002d380 con 0x7fbe30108e10 2024-03-18T15:15:41.149 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:41.149+0000 7fbe27e00640 1 -- 172.21.15.19:0/814407610 <== mon.2 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24093 (secure 0 0 0) 0x7fbe180be000 con 0x7fbe30108e10 2024-03-18T15:15:41.150 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-18T15:15:41.150 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":89,"fsid":"d4dc2ba6-e538-11ee-95c9-87774f69a715","created":"2024-03-18T15:05:57.538551+0000","modified":"2024-03-18T15:15:15.582418+0000","last_up_change":"2024-03-18T15:14:12.009522+0000","last_in_change":"2024-03-18T15:13:52.555387+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T15:09:56.843349+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T15:15:08.941047+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T15:15:10.044671+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T15:15:13.621916+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"37b5a5cd-27ae-4e77-9b05-774b817b2ee9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6802","nonce":402333948},{"type":"v1","addr":"172.21.15.19:6803","nonce":402333948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6804","nonce":402333948},{"type":"v1","addr":"172.21.15.19:6805","nonce":402333948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6808","nonce":402333948},{"type":"v1","addr":"172.21.15.19:6809","nonce":402333948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6806","nonce":402333948},{"type":"v1","addr":"172.21.15.19:6807","nonce":402333948}]},"public_addr":"172.21.15.19:6803/402333948","cluster_addr":"172.21.15.19:6805/402333948","heartbeat_back_addr":"172.21.15.19:6809/402333948","heartbeat_front_addr":"172.21.15.19:6807/402333948","state":["exists","up"]},{"osd":1,"uuid":"3a1134e8-ede7-4dbd-9be1-eefd2cbb43bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1951880113},{"type":"v1","addr":"172.21.15.33:6801","nonce":1951880113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1951880113},{"type":"v1","addr":"172.21.15.33:6803","nonce":1951880113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1951880113},{"type":"v1","addr":"172.21.15.33:6807","nonce":1951880113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1951880113},{"type":"v1","addr":"172.21.15.33:6805","nonce":1951880113}]},"public_addr":"172.21.15.33:6801/1951880113","cluster_addr":"172.21.15.33:6803/1951880113","heartbeat_back_addr":"172.21.15.33:6807/1951880113","heartbeat_front_addr":"172.21.15.33:6805/1951880113","state":["exists","up"]},{"osd":2,"uuid":"f383a164-a3a0-49fd-a9e8-9c2178d8cfb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":1004727949},{"type":"v1","addr":"172.21.15.104:6801","nonce":1004727949}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":1004727949},{"type":"v1","addr":"172.21.15.104:6803","nonce":1004727949}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":1004727949},{"type":"v1","addr":"172.21.15.104:6807","nonce":1004727949}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":1004727949},{"type":"v1","addr":"172.21.15.104:6805","nonce":1004727949}]},"public_addr":"172.21.15.104:6801/1004727949","cluster_addr":"172.21.15.104:6803/1004727949","heartbeat_back_addr":"172.21.15.104:6807/1004727949","heartbeat_front_addr":"172.21.15.104:6805/1004727949","state":["exists","up"]},{"osd":3,"uuid":"c409e116-28bf-45dd-a791-539af22abe1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6810","nonce":3571525854},{"type":"v1","addr":"172.21.15.19:6811","nonce":3571525854}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6812","nonce":3571525854},{"type":"v1","addr":"172.21.15.19:6813","nonce":3571525854}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6816","nonce":3571525854},{"type":"v1","addr":"172.21.15.19:6817","nonce":3571525854}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6814","nonce":3571525854},{"type":"v1","addr":"172.21.15.19:6815","nonce":3571525854}]},"public_addr":"172.21.15.19:6811/3571525854","cluster_addr":"172.21.15.19:6813/3571525854","heartbeat_back_addr":"172.21.15.19:6817/3571525854","heartbeat_front_addr":"172.21.15.19:6815/3571525854","state":["exists","up"]},{"osd":4,"uuid":"696eb719-0b3e-43ae-b5f1-ccd73a3b2843","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":624901749},{"type":"v1","addr":"172.21.15.33:6809","nonce":624901749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":624901749},{"type":"v1","addr":"172.21.15.33:6811","nonce":624901749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":624901749},{"type":"v1","addr":"172.21.15.33:6815","nonce":624901749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":624901749},{"type":"v1","addr":"172.21.15.33:6813","nonce":624901749}]},"public_addr":"172.21.15.33:6809/624901749","cluster_addr":"172.21.15.33:6811/624901749","heartbeat_back_addr":"172.21.15.33:6815/624901749","heartbeat_front_addr":"172.21.15.33:6813/624901749","state":["exists","up"]},{"osd":5,"uuid":"21789779-8bb9-42c7-b9ee-437985f11e3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":1258208297},{"type":"v1","addr":"172.21.15.104:6809","nonce":1258208297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":1258208297},{"type":"v1","addr":"172.21.15.104:6811","nonce":1258208297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":1258208297},{"type":"v1","addr":"172.21.15.104:6815","nonce":1258208297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":1258208297},{"type":"v1","addr":"172.21.15.104:6813","nonce":1258208297}]},"public_addr":"172.21.15.104:6809/1258208297","cluster_addr":"172.21.15.104:6811/1258208297","heartbeat_back_addr":"172.21.15.104:6815/1258208297","heartbeat_front_addr":"172.21.15.104:6813/1258208297","state":["exists","up"]},{"osd":6,"uuid":"0e92a2ef-43cd-487c-9203-e219158aef8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6818","nonce":2782482301},{"type":"v1","addr":"172.21.15.19:6819","nonce":2782482301}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6820","nonce":2782482301},{"type":"v1","addr":"172.21.15.19:6821","nonce":2782482301}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6824","nonce":2782482301},{"type":"v1","addr":"172.21.15.19:6825","nonce":2782482301}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6822","nonce":2782482301},{"type":"v1","addr":"172.21.15.19:6823","nonce":2782482301}]},"public_addr":"172.21.15.19:6819/2782482301","cluster_addr":"172.21.15.19:6821/2782482301","heartbeat_back_addr":"172.21.15.19:6825/2782482301","heartbeat_front_addr":"172.21.15.19:6823/2782482301","state":["exists","up"]},{"osd":7,"uuid":"a16e39a1-3685-491d-b376-3e80a2084072","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":4238160988},{"type":"v1","addr":"172.21.15.33:6817","nonce":4238160988}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":4238160988},{"type":"v1","addr":"172.21.15.33:6819","nonce":4238160988}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":4238160988},{"type":"v1","addr":"172.21.15.33:6823","nonce":4238160988}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":4238160988},{"type":"v1","addr":"172.21.15.33:6821","nonce":4238160988}]},"public_addr":"172.21.15.33:6817/4238160988","cluster_addr":"172.21.15.33:6819/4238160988","heartbeat_back_addr":"172.21.15.33:6823/4238160988","heartbeat_front_addr":"172.21.15.33:6821/4238160988","state":["exists","up"]},{"osd":8,"uuid":"d5c3fa92-cd5d-4349-9c7d-d5666bacde16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":1014779813},{"type":"v1","addr":"172.21.15.104:6817","nonce":1014779813}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":1014779813},{"type":"v1","addr":"172.21.15.104:6819","nonce":1014779813}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":1014779813},{"type":"v1","addr":"172.21.15.104:6823","nonce":1014779813}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":1014779813},{"type":"v1","addr":"172.21.15.104:6821","nonce":1014779813}]},"public_addr":"172.21.15.104:6817/1014779813","cluster_addr":"172.21.15.104:6819/1014779813","heartbeat_back_addr":"172.21.15.104:6823/1014779813","heartbeat_front_addr":"172.21.15.104:6821/1014779813","state":["exists","up"]},{"osd":9,"uuid":"eed2e9b2-26d3-4c46-88ec-34b5596fe991","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6826","nonce":2641151009},{"type":"v1","addr":"172.21.15.19:6827","nonce":2641151009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6828","nonce":2641151009},{"type":"v1","addr":"172.21.15.19:6829","nonce":2641151009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6832","nonce":2641151009},{"type":"v1","addr":"172.21.15.19:6833","nonce":2641151009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6830","nonce":2641151009},{"type":"v1","addr":"172.21.15.19:6831","nonce":2641151009}]},"public_addr":"172.21.15.19:6827/2641151009","cluster_addr":"172.21.15.19:6829/2641151009","heartbeat_back_addr":"172.21.15.19:6833/2641151009","heartbeat_front_addr":"172.21.15.19:6831/2641151009","state":["exists","up"]},{"osd":10,"uuid":"f3c35ea5-fbe7-4dc4-93c5-b92d1a714db3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":1346058869},{"type":"v1","addr":"172.21.15.33:6825","nonce":1346058869}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":1346058869},{"type":"v1","addr":"172.21.15.33:6827","nonce":1346058869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":1346058869},{"type":"v1","addr":"172.21.15.33:6831","nonce":1346058869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":1346058869},{"type":"v1","addr":"172.21.15.33:6829","nonce":1346058869}]},"public_addr":"172.21.15.33:6825/1346058869","cluster_addr":"172.21.15.33:6827/1346058869","heartbeat_back_addr":"172.21.15.33:6831/1346058869","heartbeat_front_addr":"172.21.15.33:6829/1346058869","state":["exists","up"]},{"osd":11,"uuid":"c8b51571-f807-4b62-90b6-7529e09e59da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":3759498281},{"type":"v1","addr":"172.21.15.104:6825","nonce":3759498281}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":3759498281},{"type":"v1","addr":"172.21.15.104:6827","nonce":3759498281}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":3759498281},{"type":"v1","addr":"172.21.15.104:6831","nonce":3759498281}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":3759498281},{"type":"v1","addr":"172.21.15.104:6829","nonce":3759498281}]},"public_addr":"172.21.15.104:6825/3759498281","cluster_addr":"172.21.15.104:6827/3759498281","heartbeat_back_addr":"172.21.15.104:6831/3759498281","heartbeat_front_addr":"172.21.15.104:6829/3759498281","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:09:05.267795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:09:30.234068+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:09:53.870232+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:10:26.471296+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:10:53.180341+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:11:17.801361+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:11:51.364816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:12:18.306408+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:12:43.978744+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:13:18.213964+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:13:45.437544+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:14:10.148580+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.19:0/3860783274":"2024-03-19T15:07:14.150983+0000","172.21.15.19:0/2737223213":"2024-03-19T15:07:14.150983+0000","172.21.15.19:0/176699025":"2024-03-19T15:06:38.688405+0000","172.21.15.19:0/2316969005":"2024-03-19T15:07:14.150983+0000","172.21.15.19:6801/2776008962":"2024-03-19T15:07:14.150983+0000","172.21.15.19:0/2793980440":"2024-03-19T15:06:38.688405+0000","172.21.15.19:6800/3264337803":"2024-03-19T15:06:38.688405+0000","172.21.15.19:6801/3264337803":"2024-03-19T15:06:38.688405+0000","172.21.15.19:0/3166705295":"2024-03-19T15:06:38.688405+0000","172.21.15.19:6800/2776008962":"2024-03-19T15:07:14.150983+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T15:15:41.152 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:41.152+0000 7fbe2fe00640 1 -- 172.21.15.19:0/814407610 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fbe140a2950 msgr2=0x7fbe140a4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:41.152 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:41.152+0000 7fbe2fe00640 1 --2- 172.21.15.19:0/814407610 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fbe140a2950 0x7fbe140a4e10 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fbe30108c70 tx=0x7fbe20009500 comp rx=0 tx=0).stop 2024-03-18T15:15:41.152 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:41.152+0000 7fbe2fe00640 1 -- 172.21.15.19:0/814407610 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbe30108e10 msgr2=0x7fbe3019e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:41.152 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:41.152+0000 7fbe2fe00640 1 --2- 172.21.15.19:0/814407610 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbe30108e10 0x7fbe3019e420 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fbe18000c00 tx=0x7fbe1802c650 comp rx=0 tx=0).stop 2024-03-18T15:15:41.153 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:41.152+0000 7fbe2fe00640 1 -- 172.21.15.19:0/814407610 shutdown_connections 2024-03-18T15:15:41.153 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:41.152+0000 7fbe2fe00640 1 --2- 172.21.15.19:0/814407610 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbe301097d0 0x7fbe301a5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:41.153 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:41.152+0000 7fbe2fe00640 1 --2- 172.21.15.19:0/814407610 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fbe140a2950 0x7fbe140a4e10 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:41.153 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:41.152+0000 7fbe2fe00640 1 --2- 172.21.15.19:0/814407610 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbe30108e10 0x7fbe3019e420 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:41.153 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:41.152+0000 7fbe2fe00640 1 --2- 172.21.15.19:0/814407610 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbe30107c20 0x7fbe3019dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:41.153 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:41.152+0000 7fbe2fe00640 1 -- 172.21.15.19:0/814407610 >> 172.21.15.19:0/814407610 conn(0x7fbe30076bb0 msgr2=0x7fbe3010c110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:41.153 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:41.152+0000 7fbe2fe00640 1 -- 172.21.15.19:0/814407610 shutdown_connections 2024-03-18T15:15:41.153 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:41.152+0000 7fbe2fe00640 1 -- 172.21.15.19:0/814407610 wait complete. 2024-03-18T15:15:41.173 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-18T15:15:41.392 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:41.392+0000 7f04ed000640 1 -- 172.21.15.19:0/795351365 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f04e810ddc0 msgr2=0x7f04e81101b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:41.393 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:41.392+0000 7f04ed000640 1 --2- 172.21.15.19:0/795351365 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f04e810ddc0 0x7f04e81101b0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f04e0009930 tx=0x7f04e002bc30 comp rx=0 tx=0).stop 2024-03-18T15:15:41.393 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:41.392+0000 7f04ed000640 1 -- 172.21.15.19:0/795351365 shutdown_connections 2024-03-18T15:15:41.393 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:41.392+0000 7f04ed000640 1 --2- 172.21.15.19:0/795351365 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f04e810ddc0 0x7f04e81101b0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:41.393 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:41.392+0000 7f04ed000640 1 --2- 172.21.15.19:0/795351365 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f04e8074240 0x7f04e80746c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:41.393 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:41.392+0000 7f04ed000640 1 --2- 172.21.15.19:0/795351365 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f04e8075820 0x7f04e8073c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:41.393 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:41.392+0000 7f04ed000640 1 -- 172.21.15.19:0/795351365 >> 172.21.15.19:0/795351365 conn(0x7f04e80fcd60 msgr2=0x7f04e80ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:41.393 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:41.392+0000 7f04ed000640 1 -- 172.21.15.19:0/795351365 shutdown_connections 2024-03-18T15:15:41.393 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:41.392+0000 7f04ed000640 1 -- 172.21.15.19:0/795351365 wait complete. 2024-03-18T15:15:41.395 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:41.393+0000 7f04ed000640 1 Processor -- start 2024-03-18T15:15:41.395 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:41.393+0000 7f04ed000640 1 -- start start 2024-03-18T15:15:41.395 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:41.394+0000 7f04ed000640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f04e8074240 0x7f04e819e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:41.396 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:41.394+0000 7f04ed000640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f04e8075820 0x7f04e819e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:41.396 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:41.394+0000 7f04ed000640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f04e810ddc0 0x7f04e81a56b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:41.396 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:41.394+0000 7f04ed000640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f04e8113ab0 con 0x7f04e810ddc0 2024-03-18T15:15:41.396 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:41.394+0000 7f04ed000640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f04e8113930 con 0x7f04e8074240 2024-03-18T15:15:41.396 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:41.394+0000 7f04ed000640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f04e8113c30 con 0x7f04e8075820 2024-03-18T15:15:41.396 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:41.394+0000 7f04e6800640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f04e810ddc0 0x7f04e81a56b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:41.396 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:41.394+0000 7f04e5e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f04e8074240 0x7f04e819e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:41.396 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:41.394+0000 7f04e6800640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f04e810ddc0 0x7f04e81a56b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:48646/0 (socket says 172.21.15.19:48646) 2024-03-18T15:15:41.396 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:41.394+0000 7f04e5e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f04e8074240 0x7f04e819e100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:32824/0 (socket says 172.21.15.19:32824) 2024-03-18T15:15:41.396 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:41.394+0000 7f04e6800640 1 -- 172.21.15.19:0/1520724927 learned_addr learned my addr 172.21.15.19:0/1520724927 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:15:41.396 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:41.394+0000 7f04dd400640 1 --2- 172.21.15.19:0/1520724927 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f04e8075820 0x7f04e819e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:41.396 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:41.394+0000 7f04e6800640 1 -- 172.21.15.19:0/1520724927 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f04e8075820 msgr2=0x7f04e819e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:41.396 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:41.394+0000 7f04e6800640 1 --2- 172.21.15.19:0/1520724927 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f04e8075820 0x7f04e819e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:41.397 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:41.394+0000 7f04e6800640 1 -- 172.21.15.19:0/1520724927 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f04e8074240 msgr2=0x7f04e819e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:41.397 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:41.394+0000 7f04e6800640 1 --2- 172.21.15.19:0/1520724927 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f04e8074240 0x7f04e819e100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:41.397 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:41.394+0000 7f04e6800640 1 -- 172.21.15.19:0/1520724927 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04e81a5d20 con 0x7f04e810ddc0 2024-03-18T15:15:41.397 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:41.394+0000 7f04e5e00640 1 --2- 172.21.15.19:0/1520724927 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f04e8074240 0x7f04e819e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:15:41.397 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:41.394+0000 7f04dd400640 1 --2- 172.21.15.19:0/1520724927 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f04e8075820 0x7f04e819e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T15:15:41.397 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:41.395+0000 7f04e6800640 1 --2- 172.21.15.19:0/1520724927 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f04e810ddc0 0x7f04e81a56b0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f04e002c8e0 tx=0x7f04e002c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:41.397 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:41.395+0000 7f04df400640 1 -- 172.21.15.19:0/1520724927 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04e0043020 con 0x7f04e810ddc0 2024-03-18T15:15:41.397 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:41.395+0000 7f04df400640 1 -- 172.21.15.19:0/1520724927 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f04e00054e0 con 0x7f04e810ddc0 2024-03-18T15:15:41.397 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:41.395+0000 7f04ed000640 1 -- 172.21.15.19:0/1520724927 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04e81a5fb0 con 0x7f04e810ddc0 2024-03-18T15:15:41.397 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:41.395+0000 7f04df400640 1 -- 172.21.15.19:0/1520724927 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04e0003cb0 con 0x7f04e810ddc0 2024-03-18T15:15:41.397 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:41.395+0000 7f04ed000640 1 -- 172.21.15.19:0/1520724927 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04e81a6470 con 0x7f04e810ddc0 2024-03-18T15:15:41.398 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:41.397+0000 7f04ed000640 1 -- 172.21.15.19:0/1520724927 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04e8103f90 con 0x7f04e810ddc0 2024-03-18T15:15:41.398 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:41.398+0000 7f04df400640 1 -- 172.21.15.19:0/1520724927 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f04e0046020 con 0x7f04e810ddc0 2024-03-18T15:15:41.400 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:41.400+0000 7f04df400640 1 --2- 172.21.15.19:0/1520724927 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f04cc0a2800 0x7f04cc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:41.400 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:41.400+0000 7f04df400640 1 -- 172.21.15.19:0/1520724927 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f04e00f0e70 con 0x7f04e810ddc0 2024-03-18T15:15:41.400 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:41.400+0000 7f04e5e00640 1 --2- 172.21.15.19:0/1520724927 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f04cc0a2800 0x7f04cc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:41.401 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:41.401+0000 7f04e5e00640 1 --2- 172.21.15.19:0/1520724927 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f04cc0a2800 0x7f04cc0a4cc0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f04d8005fd0 tx=0x7f04d8009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:41.404 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:41.404+0000 7f04df400640 1 -- 172.21.15.19:0/1520724927 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f04e004d020 con 0x7f04e810ddc0 2024-03-18T15:15:41.655 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:41.655+0000 7f04ed000640 1 -- 172.21.15.19:0/1520724927 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f04e8062480 con 0x7f04e810ddc0 2024-03-18T15:15:41.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:41 smithi104 ceph-mon[36070]: pgmap v332: 129 pgs: 129 active+clean; 581 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-18T15:15:41.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:41 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/4151512721' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:15:41.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:41 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/814407610' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:15:41.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:41 smithi019 ceph-mon[31522]: pgmap v332: 129 pgs: 129 active+clean; 581 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-18T15:15:41.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:41 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/4151512721' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:15:41.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:41 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/814407610' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:15:41.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:41 smithi033 ceph-mon[36648]: pgmap v332: 129 pgs: 129 active+clean; 581 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-18T15:15:41.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:41 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/4151512721' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:15:41.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:41 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/814407610' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:15:42.443 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:42.443+0000 7f04df400640 1 -- 172.21.15.19:0/1520724927 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v17) ==== 118+0+0 (secure 0 0 0) 0x7f04e00bdc70 con 0x7f04e810ddc0 2024-03-18T15:15:42.446 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:42.446+0000 7f04ed000640 1 -- 172.21.15.19:0/1520724927 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f04cc0a2800 msgr2=0x7f04cc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:42.447 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:42.446+0000 7f04ed000640 1 --2- 172.21.15.19:0/1520724927 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f04cc0a2800 0x7f04cc0a4cc0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f04d8005fd0 tx=0x7f04d8009500 comp rx=0 tx=0).stop 2024-03-18T15:15:42.447 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:42.446+0000 7f04ed000640 1 -- 172.21.15.19:0/1520724927 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f04e810ddc0 msgr2=0x7f04e81a56b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:42.447 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:42.446+0000 7f04ed000640 1 --2- 172.21.15.19:0/1520724927 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f04e810ddc0 0x7f04e81a56b0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f04e002c8e0 tx=0x7f04e002c6a0 comp rx=0 tx=0).stop 2024-03-18T15:15:42.447 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:42.447+0000 7f04ed000640 1 -- 172.21.15.19:0/1520724927 shutdown_connections 2024-03-18T15:15:42.447 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:42.447+0000 7f04ed000640 1 --2- 172.21.15.19:0/1520724927 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f04e810ddc0 0x7f04e81a56b0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:42.448 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:42.447+0000 7f04ed000640 1 --2- 172.21.15.19:0/1520724927 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f04e8075820 0x7f04e819e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:42.448 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:42.447+0000 7f04ed000640 1 --2- 172.21.15.19:0/1520724927 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f04cc0a2800 0x7f04cc0a4cc0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:42.448 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:42.447+0000 7f04ed000640 1 --2- 172.21.15.19:0/1520724927 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f04e8074240 0x7f04e819e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:42.448 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:42.447+0000 7f04ed000640 1 -- 172.21.15.19:0/1520724927 >> 172.21.15.19:0/1520724927 conn(0x7f04e80fcd60 msgr2=0x7f04e8101160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:42.448 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:42.447+0000 7f04ed000640 1 -- 172.21.15.19:0/1520724927 shutdown_connections 2024-03-18T15:15:42.448 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:42.447+0000 7f04ed000640 1 -- 172.21.15.19:0/1520724927 wait complete. 2024-03-18T15:15:42.469 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2024-03-18T15:15:42.469 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 5 2024-03-18T15:15:42.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:42 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/1520724927' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-18T15:15:42.703 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:42.703+0000 7f2367200640 1 -- 172.21.15.19:0/4237891273 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2368102160 msgr2=0x7f2368104550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:42.703 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:42.703+0000 7f2367200640 1 --2- 172.21.15.19:0/4237891273 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2368102160 0x7f2368104550 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f2358009860 tx=0x7f235802bb70 comp rx=0 tx=0).stop 2024-03-18T15:15:42.704 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:42.703+0000 7f2367200640 1 -- 172.21.15.19:0/4237891273 shutdown_connections 2024-03-18T15:15:42.704 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:42.703+0000 7f2367200640 1 --2- 172.21.15.19:0/4237891273 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f23681073c0 0x7f2368109810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:42.704 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:42.703+0000 7f2367200640 1 --2- 172.21.15.19:0/4237891273 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2368104a90 0x7f2368106e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:42.704 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:42.703+0000 7f2367200640 1 --2- 172.21.15.19:0/4237891273 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2368102160 0x7f2368104550 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:42.704 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:42.703+0000 7f2367200640 1 -- 172.21.15.19:0/4237891273 >> 172.21.15.19:0/4237891273 conn(0x7f23680fbb50 msgr2=0x7f23680fdf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:42.704 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:42.703+0000 7f2367200640 1 -- 172.21.15.19:0/4237891273 shutdown_connections 2024-03-18T15:15:42.704 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:42.703+0000 7f2367200640 1 -- 172.21.15.19:0/4237891273 wait complete. 2024-03-18T15:15:42.707 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:42.704+0000 7f2367200640 1 Processor -- start 2024-03-18T15:15:42.707 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:42.704+0000 7f2367200640 1 -- start start 2024-03-18T15:15:42.707 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:42.705+0000 7f2367200640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2368102160 0x7f2368101ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:42.707 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:42.705+0000 7f2367200640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2368104a90 0x7f2368100110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:42.707 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:42.705+0000 7f2367200640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f23681073c0 0x7f2368100650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:42.707 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:42.705+0000 7f2367200640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f236810d370 con 0x7f2368104a90 2024-03-18T15:15:42.707 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:42.705+0000 7f2367200640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f236810d1f0 con 0x7f23681073c0 2024-03-18T15:15:42.707 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:42.705+0000 7f2367200640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f236810d4f0 con 0x7f2368102160 2024-03-18T15:15:42.707 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:42.705+0000 7f2366800640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f23681073c0 0x7f2368100650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:42.707 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:42.705+0000 7f235fe00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2368104a90 0x7f2368100110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:42.708 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:42.705+0000 7f2366800640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f23681073c0 0x7f2368100650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:35582/0 (socket says 172.21.15.19:35582) 2024-03-18T15:15:42.708 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:42.705+0000 7f2366800640 1 -- 172.21.15.19:0/3689037481 learned_addr learned my addr 172.21.15.19:0/3689037481 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:15:42.708 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:42.705+0000 7f2365e00640 1 --2- 172.21.15.19:0/3689037481 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2368102160 0x7f2368101ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:42.708 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:42.705+0000 7f2366800640 1 -- 172.21.15.19:0/3689037481 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2368102160 msgr2=0x7f2368101ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:42.708 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:42.705+0000 7f2366800640 1 --2- 172.21.15.19:0/3689037481 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2368102160 0x7f2368101ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:42.708 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:42.706+0000 7f2366800640 1 -- 172.21.15.19:0/3689037481 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2368104a90 msgr2=0x7f2368100110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:42.708 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:42.706+0000 7f2366800640 1 --2- 172.21.15.19:0/3689037481 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2368104a90 0x7f2368100110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:42.708 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:42.706+0000 7f2366800640 1 -- 172.21.15.19:0/3689037481 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2368100ee0 con 0x7f23681073c0 2024-03-18T15:15:42.708 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:42.706+0000 7f235fe00640 1 --2- 172.21.15.19:0/3689037481 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2368104a90 0x7f2368100110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:15:42.708 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:42.706+0000 7f2366800640 1 --2- 172.21.15.19:0/3689037481 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f23681073c0 0x7f2368100650 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f236000efc0 tx=0x7f236000c800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:42.708 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:42.706+0000 7f235ea00640 1 -- 172.21.15.19:0/3689037481 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2360003970 con 0x7f23681073c0 2024-03-18T15:15:42.708 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:42.706+0000 7f235ea00640 1 -- 172.21.15.19:0/3689037481 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2360004020 con 0x7f23681073c0 2024-03-18T15:15:42.708 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:42.706+0000 7f235ea00640 1 -- 172.21.15.19:0/3689037481 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f236000b9f0 con 0x7f23681073c0 2024-03-18T15:15:42.709 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:42.706+0000 7f2367200640 1 -- 172.21.15.19:0/3689037481 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23681a1b30 con 0x7f23681073c0 2024-03-18T15:15:42.709 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:42.706+0000 7f2367200640 1 -- 172.21.15.19:0/3689037481 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23681a2080 con 0x7f23681073c0 2024-03-18T15:15:42.709 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:42.707+0000 7f2367200640 1 -- 172.21.15.19:0/3689037481 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f23680fd640 con 0x7f23681073c0 2024-03-18T15:15:42.709 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:42.709+0000 7f235ea00640 1 -- 172.21.15.19:0/3689037481 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f23600041d0 con 0x7f23681073c0 2024-03-18T15:15:42.710 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:42.710+0000 7f235ea00640 1 --2- 172.21.15.19:0/3689037481 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f234c0a2880 0x7f234c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:42.710 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:42.711+0000 7f235ea00640 1 -- 172.21.15.19:0/3689037481 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f23600ceec0 con 0x7f23681073c0 2024-03-18T15:15:42.711 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:42.711+0000 7f2365e00640 1 --2- 172.21.15.19:0/3689037481 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f234c0a2880 0x7f234c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:42.711 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:42.711+0000 7f2365e00640 1 --2- 172.21.15.19:0/3689037481 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f234c0a2880 0x7f234c0a4d40 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f235802c180 tx=0x7f2358034ad0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:42.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:42 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/1520724927' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-18T15:15:42.715 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:42.715+0000 7f235ea00640 1 -- 172.21.15.19:0/3689037481 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f236009bcc0 con 0x7f23681073c0 2024-03-18T15:15:42.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:42 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/1520724927' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-18T15:15:42.962 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:42.962+0000 7f2367200640 1 -- 172.21.15.19:0/3689037481 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) -- 0x7f23680623e0 con 0x7f23681073c0 2024-03-18T15:15:43.450 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:43.450+0000 7f235ea00640 1 -- 172.21.15.19:0/3689037481 <== mon.2 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v18) ==== 105+0+0 (secure 0 0 0) 0x7f236009b5c0 con 0x7f23681073c0 2024-03-18T15:15:43.454 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:43.454+0000 7f2367200640 1 -- 172.21.15.19:0/3689037481 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f234c0a2880 msgr2=0x7f234c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:43.454 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:43.454+0000 7f2367200640 1 --2- 172.21.15.19:0/3689037481 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f234c0a2880 0x7f234c0a4d40 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f235802c180 tx=0x7f2358034ad0 comp rx=0 tx=0).stop 2024-03-18T15:15:43.454 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:43.454+0000 7f2367200640 1 -- 172.21.15.19:0/3689037481 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f23681073c0 msgr2=0x7f2368100650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:43.454 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:43.454+0000 7f2367200640 1 --2- 172.21.15.19:0/3689037481 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f23681073c0 0x7f2368100650 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f236000efc0 tx=0x7f236000c800 comp rx=0 tx=0).stop 2024-03-18T15:15:43.455 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:43.454+0000 7f2367200640 1 -- 172.21.15.19:0/3689037481 shutdown_connections 2024-03-18T15:15:43.455 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:43.454+0000 7f2367200640 1 --2- 172.21.15.19:0/3689037481 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f234c0a2880 0x7f234c0a4d40 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:43.455 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:43.454+0000 7f2367200640 1 --2- 172.21.15.19:0/3689037481 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f23681073c0 0x7f2368100650 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:43.455 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:43.454+0000 7f2367200640 1 --2- 172.21.15.19:0/3689037481 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2368104a90 0x7f2368100110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:43.455 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:43.454+0000 7f2367200640 1 --2- 172.21.15.19:0/3689037481 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2368102160 0x7f2368101ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:43.455 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:43.454+0000 7f2367200640 1 -- 172.21.15.19:0/3689037481 >> 172.21.15.19:0/3689037481 conn(0x7f23680fbb50 msgr2=0x7f2368102aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:43.455 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:43.455+0000 7f2367200640 1 -- 172.21.15.19:0/3689037481 shutdown_connections 2024-03-18T15:15:43.455 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:43.455+0000 7f2367200640 1 -- 172.21.15.19:0/3689037481 wait complete. 2024-03-18T15:15:43.481 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-03-18T15:15:43.687 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:43.686+0000 7f67a7600640 1 -- 172.21.15.19:0/2693629938 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f67a0104bf0 msgr2=0x7f67a0105070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:43.687 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:43.686+0000 7f67a7600640 1 --2- 172.21.15.19:0/2693629938 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f67a0104bf0 0x7f67a0105070 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f678c009900 tx=0x7f678c02bc40 comp rx=0 tx=0).stop 2024-03-18T15:15:43.687 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:43.687+0000 7f67a7600640 1 -- 172.21.15.19:0/2693629938 shutdown_connections 2024-03-18T15:15:43.687 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:43.687+0000 7f67a7600640 1 --2- 172.21.15.19:0/2693629938 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f67a01055b0 0x7f67a010be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:43.687 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:43.687+0000 7f67a7600640 1 --2- 172.21.15.19:0/2693629938 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f67a0104bf0 0x7f67a0105070 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:43.687 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:43.687+0000 7f67a7600640 1 --2- 172.21.15.19:0/2693629938 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f67a0103a00 0x7f67a0103e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:43.687 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:43.687+0000 7f67a7600640 1 -- 172.21.15.19:0/2693629938 >> 172.21.15.19:0/2693629938 conn(0x7f67a00fef90 msgr2=0x7f67a01013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:43.688 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:43.687+0000 7f67a7600640 1 -- 172.21.15.19:0/2693629938 shutdown_connections 2024-03-18T15:15:43.688 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:43.687+0000 7f67a7600640 1 -- 172.21.15.19:0/2693629938 wait complete. 2024-03-18T15:15:43.690 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:43.688+0000 7f67a7600640 1 Processor -- start 2024-03-18T15:15:43.690 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:43.688+0000 7f67a7600640 1 -- start start 2024-03-18T15:15:43.690 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:43.688+0000 7f67a7600640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f67a0103a00 0x7f67a0199d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:43.690 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:43.688+0000 7f67a7600640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f67a0104bf0 0x7f67a019a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:43.690 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:43.688+0000 7f67a7600640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f67a01055b0 0x7f67a01a12f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:43.690 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:43.688+0000 7f67a7600640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f67a010f6d0 con 0x7f67a0103a00 2024-03-18T15:15:43.690 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:43.688+0000 7f67a7600640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f67a010f550 con 0x7f67a01055b0 2024-03-18T15:15:43.690 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:43.688+0000 7f67a7600640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f67a010f850 con 0x7f67a0104bf0 2024-03-18T15:15:43.690 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:43.688+0000 7f67a5600640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f67a01055b0 0x7f67a01a12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:43.690 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:43.688+0000 7f67a4c00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f67a0103a00 0x7f67a0199d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:43.690 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:43.688+0000 7f67a5600640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f67a01055b0 0x7f67a01a12f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:35598/0 (socket says 172.21.15.19:35598) 2024-03-18T15:15:43.690 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:43.688+0000 7f67a5600640 1 -- 172.21.15.19:0/4053246804 learned_addr learned my addr 172.21.15.19:0/4053246804 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:15:43.691 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:43.688+0000 7f679fe00640 1 --2- 172.21.15.19:0/4053246804 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f67a0104bf0 0x7f67a019a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:43.691 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:43.689+0000 7f67a5600640 1 -- 172.21.15.19:0/4053246804 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f67a0104bf0 msgr2=0x7f67a019a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:43.691 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:43.689+0000 7f67a5600640 1 --2- 172.21.15.19:0/4053246804 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f67a0104bf0 0x7f67a019a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:43.691 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:43.689+0000 7f67a5600640 1 -- 172.21.15.19:0/4053246804 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f67a0103a00 msgr2=0x7f67a0199d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:43.691 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:43.689+0000 7f67a5600640 1 --2- 172.21.15.19:0/4053246804 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f67a0103a00 0x7f67a0199d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:43.691 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:43.689+0000 7f67a5600640 1 -- 172.21.15.19:0/4053246804 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67a01a1960 con 0x7f67a01055b0 2024-03-18T15:15:43.691 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:43.689+0000 7f679fe00640 1 --2- 172.21.15.19:0/4053246804 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f67a0104bf0 0x7f67a019a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T15:15:43.691 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:43.689+0000 7f67a5600640 1 --2- 172.21.15.19:0/4053246804 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f67a01055b0 0x7f67a01a12f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f679800efd0 tx=0x7f679800c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:43.691 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:43.689+0000 7f679d600640 1 -- 172.21.15.19:0/4053246804 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6798003b70 con 0x7f67a01055b0 2024-03-18T15:15:43.691 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:43.689+0000 7f679d600640 1 -- 172.21.15.19:0/4053246804 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6798003d10 con 0x7f67a01055b0 2024-03-18T15:15:43.691 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:43.689+0000 7f679d600640 1 -- 172.21.15.19:0/4053246804 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67980109f0 con 0x7f67a01055b0 2024-03-18T15:15:43.692 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:43.689+0000 7f67a7600640 1 -- 172.21.15.19:0/4053246804 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67a01a1c50 con 0x7f67a01055b0 2024-03-18T15:15:43.692 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:43.689+0000 7f67a7600640 1 -- 172.21.15.19:0/4053246804 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67a01a2120 con 0x7f67a01055b0 2024-03-18T15:15:43.692 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:43.690+0000 7f67a7600640 1 -- 172.21.15.19:0/4053246804 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67a010ad20 con 0x7f67a01055b0 2024-03-18T15:15:43.692 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:43.693+0000 7f679d600640 1 -- 172.21.15.19:0/4053246804 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f6798010b90 con 0x7f67a01055b0 2024-03-18T15:15:43.694 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:43.694+0000 7f679d600640 1 --2- 172.21.15.19:0/4053246804 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f67840a2880 0x7f67840a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:43.694 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:43.694+0000 7f679d600640 1 -- 172.21.15.19:0/4053246804 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f67980cf4d0 con 0x7f67a01055b0 2024-03-18T15:15:43.694 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:43.694+0000 7f67a4c00640 1 --2- 172.21.15.19:0/4053246804 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f67840a2880 0x7f67840a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:43.695 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:43.695+0000 7f67a4c00640 1 --2- 172.21.15.19:0/4053246804 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f67840a2880 0x7f67840a4d40 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f6790005fd0 tx=0x7f679000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:43.696 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:43.696+0000 7f679d600640 1 -- 172.21.15.19:0/4053246804 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f679809c2d0 con 0x7f67a01055b0 2024-03-18T15:15:43.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:43 smithi019 ceph-mon[31522]: pgmap v333: 129 pgs: 129 active+clean; 581 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-18T15:15:43.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:43 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/1520724927' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-18T15:15:43.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:43 smithi019 ceph-mon[31522]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-18T15:15:43.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:43 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/3689037481' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-03-18T15:15:43.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:43 smithi019 ceph-mon[31522]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-03-18T15:15:43.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:43 smithi033 ceph-mon[36648]: pgmap v333: 129 pgs: 129 active+clean; 581 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-18T15:15:43.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:43 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/1520724927' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-18T15:15:43.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:43 smithi033 ceph-mon[36648]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-18T15:15:43.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:43 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/3689037481' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-03-18T15:15:43.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:43 smithi033 ceph-mon[36648]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-03-18T15:15:43.941 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:43.940+0000 7f67a7600640 1 -- 172.21.15.19:0/4053246804 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7f67a002d3d0 con 0x7f67a01055b0 2024-03-18T15:15:43.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:43 smithi104 ceph-mon[36070]: pgmap v333: 129 pgs: 129 active+clean; 581 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-18T15:15:43.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:43 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/1520724927' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-18T15:15:43.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:43 smithi104 ceph-mon[36070]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-18T15:15:43.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:43 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/3689037481' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-03-18T15:15:43.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:43 smithi104 ceph-mon[36070]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-03-18T15:15:44.464 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:44.464+0000 7f679d600640 1 -- 172.21.15.19:0/4053246804 <== mon.2 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v20) ==== 121+0+0 (secure 0 0 0) 0x7f679809bbd0 con 0x7f67a01055b0 2024-03-18T15:15:44.468 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:44.468+0000 7f67a7600640 1 -- 172.21.15.19:0/4053246804 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f67840a2880 msgr2=0x7f67840a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:44.468 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:44.468+0000 7f67a7600640 1 --2- 172.21.15.19:0/4053246804 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f67840a2880 0x7f67840a4d40 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f6790005fd0 tx=0x7f679000b390 comp rx=0 tx=0).stop 2024-03-18T15:15:44.469 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:44.468+0000 7f67a7600640 1 -- 172.21.15.19:0/4053246804 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f67a01055b0 msgr2=0x7f67a01a12f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:44.469 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:44.468+0000 7f67a7600640 1 --2- 172.21.15.19:0/4053246804 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f67a01055b0 0x7f67a01a12f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f679800efd0 tx=0x7f679800c6a0 comp rx=0 tx=0).stop 2024-03-18T15:15:44.469 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:44.468+0000 7f67a7600640 1 -- 172.21.15.19:0/4053246804 shutdown_connections 2024-03-18T15:15:44.469 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:44.468+0000 7f67a7600640 1 --2- 172.21.15.19:0/4053246804 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f67840a2880 0x7f67840a4d40 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:44.469 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:44.468+0000 7f67a7600640 1 --2- 172.21.15.19:0/4053246804 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f67a01055b0 0x7f67a01a12f0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:44.469 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:44.468+0000 7f67a7600640 1 --2- 172.21.15.19:0/4053246804 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f67a0104bf0 0x7f67a019a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:44.469 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:44.468+0000 7f67a7600640 1 --2- 172.21.15.19:0/4053246804 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f67a0103a00 0x7f67a0199d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:44.469 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:44.468+0000 7f67a7600640 1 -- 172.21.15.19:0/4053246804 >> 172.21.15.19:0/4053246804 conn(0x7f67a00fef90 msgr2=0x7f67a0107ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:44.469 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:44.469+0000 7f67a7600640 1 -- 172.21.15.19:0/4053246804 shutdown_connections 2024-03-18T15:15:44.469 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:44.469+0000 7f67a7600640 1 -- 172.21.15.19:0/4053246804 wait complete. 2024-03-18T15:15:44.497 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-03-18T15:15:44.698 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:44.698+0000 7fba51a00640 1 -- 172.21.15.19:0/2628027722 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fba4c1099b0 msgr2=0x7fba4c110240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:44.699 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:44.698+0000 7fba51a00640 1 --2- 172.21.15.19:0/2628027722 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fba4c1099b0 0x7fba4c110240 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fba44009ca0 tx=0x7fba4402c100 comp rx=0 tx=0).stop 2024-03-18T15:15:44.699 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:44.698+0000 7fba51a00640 1 -- 172.21.15.19:0/2628027722 shutdown_connections 2024-03-18T15:15:44.699 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:44.698+0000 7fba51a00640 1 --2- 172.21.15.19:0/2628027722 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fba4c1099b0 0x7fba4c110240 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:44.699 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:44.698+0000 7fba51a00640 1 --2- 172.21.15.19:0/2628027722 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fba4c108ff0 0x7fba4c109470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:44.699 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:44.698+0000 7fba51a00640 1 --2- 172.21.15.19:0/2628027722 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba4c107e00 0x7fba4c108220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:44.699 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:44.698+0000 7fba51a00640 1 -- 172.21.15.19:0/2628027722 >> 172.21.15.19:0/2628027722 conn(0x7fba4c076da0 msgr2=0x7fba4c0791e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:44.699 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:44.698+0000 7fba51a00640 1 -- 172.21.15.19:0/2628027722 shutdown_connections 2024-03-18T15:15:44.699 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:44.698+0000 7fba51a00640 1 -- 172.21.15.19:0/2628027722 wait complete. 2024-03-18T15:15:44.701 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:44.699+0000 7fba51a00640 1 Processor -- start 2024-03-18T15:15:44.701 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:44.699+0000 7fba51a00640 1 -- start start 2024-03-18T15:15:44.701 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:44.699+0000 7fba51a00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fba4c107e00 0x7fba4c19e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:44.701 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:44.699+0000 7fba51a00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba4c108ff0 0x7fba4c19e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:44.701 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:44.699+0000 7fba51a00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fba4c1099b0 0x7fba4c1a5650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:44.701 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:44.699+0000 7fba51a00640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fba4c113a60 con 0x7fba4c107e00 2024-03-18T15:15:44.701 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:44.699+0000 7fba51a00640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fba4c1138e0 con 0x7fba4c1099b0 2024-03-18T15:15:44.701 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:44.699+0000 7fba51a00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fba4c113be0 con 0x7fba4c108ff0 2024-03-18T15:15:44.701 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:44.700+0000 7fba4b400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fba4c1099b0 0x7fba4c1a5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:44.702 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:44.700+0000 7fba4b400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fba4c1099b0 0x7fba4c1a5650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:35620/0 (socket says 172.21.15.19:35620) 2024-03-18T15:15:44.702 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:44.700+0000 7fba4b400640 1 -- 172.21.15.19:0/660631835 learned_addr learned my addr 172.21.15.19:0/660631835 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:15:44.702 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:44.700+0000 7fba4a000640 1 --2- 172.21.15.19:0/660631835 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba4c108ff0 0x7fba4c19e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:44.702 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:44.700+0000 7fba4aa00640 1 --2- 172.21.15.19:0/660631835 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fba4c107e00 0x7fba4c19e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:44.702 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:44.700+0000 7fba4b400640 1 -- 172.21.15.19:0/660631835 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba4c108ff0 msgr2=0x7fba4c19e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:44.702 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:44.700+0000 7fba4b400640 1 --2- 172.21.15.19:0/660631835 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba4c108ff0 0x7fba4c19e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:44.702 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:44.700+0000 7fba4b400640 1 -- 172.21.15.19:0/660631835 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fba4c107e00 msgr2=0x7fba4c19e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:44.702 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:44.700+0000 7fba4b400640 1 --2- 172.21.15.19:0/660631835 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fba4c107e00 0x7fba4c19e0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:44.702 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:44.700+0000 7fba4b400640 1 -- 172.21.15.19:0/660631835 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba4c1a5cc0 con 0x7fba4c1099b0 2024-03-18T15:15:44.702 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:44.700+0000 7fba4b400640 1 --2- 172.21.15.19:0/660631835 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fba4c1099b0 0x7fba4c1a5650 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fba4402c640 tx=0x7fba44035d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:44.702 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:44.701+0000 7fba43400640 1 -- 172.21.15.19:0/660631835 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba44039040 con 0x7fba4c1099b0 2024-03-18T15:15:44.702 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:44.701+0000 7fba43400640 1 -- 172.21.15.19:0/660631835 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fba44044070 con 0x7fba4c1099b0 2024-03-18T15:15:44.702 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:44.701+0000 7fba51a00640 1 -- 172.21.15.19:0/660631835 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba4c1a5f50 con 0x7fba4c1099b0 2024-03-18T15:15:44.703 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:44.701+0000 7fba43400640 1 -- 172.21.15.19:0/660631835 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba440043c0 con 0x7fba4c1099b0 2024-03-18T15:15:44.703 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:44.701+0000 7fba51a00640 1 -- 172.21.15.19:0/660631835 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba4c1a6410 con 0x7fba4c1099b0 2024-03-18T15:15:44.703 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:44.702+0000 7fba51a00640 1 -- 172.21.15.19:0/660631835 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba4c10f0c0 con 0x7fba4c1099b0 2024-03-18T15:15:44.703 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:44.704+0000 7fba43400640 1 -- 172.21.15.19:0/660631835 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fba44004560 con 0x7fba4c1099b0 2024-03-18T15:15:44.705 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:44.705+0000 7fba43400640 1 --2- 172.21.15.19:0/660631835 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fba300a2880 0x7fba300a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:44.705 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:44.705+0000 7fba43400640 1 -- 172.21.15.19:0/660631835 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fba440f11d0 con 0x7fba4c1099b0 2024-03-18T15:15:44.705 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:44.705+0000 7fba4aa00640 1 --2- 172.21.15.19:0/660631835 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fba300a2880 0x7fba300a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:44.706 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:44.706+0000 7fba4aa00640 1 --2- 172.21.15.19:0/660631835 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fba300a2880 0x7fba300a4d40 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fba4c108e50 tx=0x7fba3c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:44.709 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:44.709+0000 7fba43400640 1 -- 172.21.15.19:0/660631835 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fba440be030 con 0x7fba4c1099b0 2024-03-18T15:15:44.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:44 smithi033 ceph-mon[36648]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-18T15:15:44.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:44 smithi033 ceph-mon[36648]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-03-18T15:15:44.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:44 smithi033 ceph-mon[36648]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-18T15:15:44.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:44 smithi033 ceph-mon[36648]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-18T15:15:44.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:44 smithi033 ceph-mon[36648]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-03-18T15:15:44.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:44 smithi033 ceph-mon[36648]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-03-18T15:15:44.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:44 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/4053246804' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-18T15:15:44.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:44 smithi033 ceph-mon[36648]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-18T15:15:44.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:44 smithi104 ceph-mon[36070]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-18T15:15:44.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:44 smithi104 ceph-mon[36070]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-03-18T15:15:44.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:44 smithi104 ceph-mon[36070]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-18T15:15:44.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:44 smithi104 ceph-mon[36070]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-18T15:15:44.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:44 smithi104 ceph-mon[36070]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-03-18T15:15:44.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:44 smithi104 ceph-mon[36070]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-03-18T15:15:44.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:44 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/4053246804' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-18T15:15:44.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:44 smithi104 ceph-mon[36070]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-18T15:15:44.954 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:44.954+0000 7fba51a00640 1 -- 172.21.15.19:0/660631835 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7fba4c062480 con 0x7fba4c1099b0 2024-03-18T15:15:44.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:44 smithi019 ceph-mon[31522]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-18T15:15:44.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:44 smithi019 ceph-mon[31522]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-03-18T15:15:44.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:44 smithi019 ceph-mon[31522]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-18T15:15:44.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:44 smithi019 ceph-mon[31522]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-18T15:15:44.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:44 smithi019 ceph-mon[31522]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-03-18T15:15:44.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:44 smithi019 ceph-mon[31522]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-03-18T15:15:44.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:44 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/4053246804' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-18T15:15:44.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:44 smithi019 ceph-mon[31522]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-18T15:15:45.479 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:45.478+0000 7fba43400640 1 -- 172.21.15.19:0/660631835 <== mon.2 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v22) ==== 115+0+0 (secure 0 0 0) 0x7fba440098b0 con 0x7fba4c1099b0 2024-03-18T15:15:45.482 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:45.482+0000 7fba51a00640 1 -- 172.21.15.19:0/660631835 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fba300a2880 msgr2=0x7fba300a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:45.482 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:45.482+0000 7fba51a00640 1 --2- 172.21.15.19:0/660631835 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fba300a2880 0x7fba300a4d40 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fba4c108e50 tx=0x7fba3c005f90 comp rx=0 tx=0).stop 2024-03-18T15:15:45.483 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:45.482+0000 7fba51a00640 1 -- 172.21.15.19:0/660631835 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fba4c1099b0 msgr2=0x7fba4c1a5650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:45.483 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:45.482+0000 7fba51a00640 1 --2- 172.21.15.19:0/660631835 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fba4c1099b0 0x7fba4c1a5650 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fba4402c640 tx=0x7fba44035d10 comp rx=0 tx=0).stop 2024-03-18T15:15:45.483 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:45.482+0000 7fba51a00640 1 -- 172.21.15.19:0/660631835 shutdown_connections 2024-03-18T15:15:45.483 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:45.482+0000 7fba51a00640 1 --2- 172.21.15.19:0/660631835 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fba300a2880 0x7fba300a4d40 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:45.483 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:45.483+0000 7fba51a00640 1 --2- 172.21.15.19:0/660631835 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fba4c1099b0 0x7fba4c1a5650 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:45.483 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:45.483+0000 7fba51a00640 1 --2- 172.21.15.19:0/660631835 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba4c108ff0 0x7fba4c19e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:45.483 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:45.483+0000 7fba51a00640 1 --2- 172.21.15.19:0/660631835 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fba4c107e00 0x7fba4c19e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:45.483 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:45.483+0000 7fba51a00640 1 -- 172.21.15.19:0/660631835 >> 172.21.15.19:0/660631835 conn(0x7fba4c076da0 msgr2=0x7fba4c10c290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:45.483 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:45.483+0000 7fba51a00640 1 -- 172.21.15.19:0/660631835 shutdown_connections 2024-03-18T15:15:45.484 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:45.483+0000 7fba51a00640 1 -- 172.21.15.19:0/660631835 wait complete. 2024-03-18T15:15:45.511 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--size 25000000000'} subvols for filesystem 'cephfs' 2024-03-18T15:15:45.511 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-03-18T15:15:45.715 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:45.714+0000 7f9bff200640 1 -- 172.21.15.19:0/1109800514 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c001018b0 msgr2=0x7f9c00108140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:45.715 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:45.714+0000 7f9bff200640 1 --2- 172.21.15.19:0/1109800514 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c001018b0 0x7f9c00108140 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f9bf8009ca0 tx=0x7f9bf802c0f0 comp rx=0 tx=0).stop 2024-03-18T15:15:45.715 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:45.715+0000 7f9bff200640 1 -- 172.21.15.19:0/1109800514 shutdown_connections 2024-03-18T15:15:45.715 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:45.715+0000 7f9bff200640 1 --2- 172.21.15.19:0/1109800514 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c001018b0 0x7f9c00108140 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:45.715 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:45.715+0000 7f9bff200640 1 --2- 172.21.15.19:0/1109800514 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9c00100ef0 0x7f9c00101370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:45.715 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:45.715+0000 7f9bff200640 1 --2- 172.21.15.19:0/1109800514 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9c00100050 0x7f9c00100470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:45.715 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:45.715+0000 7f9bff200640 1 -- 172.21.15.19:0/1109800514 >> 172.21.15.19:0/1109800514 conn(0x7f9c000fae60 msgr2=0x7f9c000fd2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:45.715 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:45.715+0000 7f9bff200640 1 -- 172.21.15.19:0/1109800514 shutdown_connections 2024-03-18T15:15:45.715 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:45.715+0000 7f9bff200640 1 -- 172.21.15.19:0/1109800514 wait complete. 2024-03-18T15:15:45.717 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:45.716+0000 7f9bff200640 1 Processor -- start 2024-03-18T15:15:45.717 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:45.716+0000 7f9bff200640 1 -- start start 2024-03-18T15:15:45.717 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:45.716+0000 7f9bff200640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9c00100050 0x7f9c00199b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:45.717 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:45.716+0000 7f9bff200640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9c00100ef0 0x7f9c0019a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:45.717 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:45.716+0000 7f9bff200640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c001018b0 0x7f9c001a10f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:45.717 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:45.716+0000 7f9bff200640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f9c0010b110 con 0x7f9c00100050 2024-03-18T15:15:45.717 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:45.716+0000 7f9bff200640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9c0010af90 con 0x7f9c00100ef0 2024-03-18T15:15:45.718 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:45.716+0000 7f9bff200640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f9c0010b290 con 0x7f9c001018b0 2024-03-18T15:15:45.718 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:45.716+0000 7f9bf5400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9c00100ef0 0x7f9c0019a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:45.718 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:45.716+0000 7f9bfde00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9c00100050 0x7f9c00199b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:45.718 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:45.716+0000 7f9bf5400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9c00100ef0 0x7f9c0019a080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:35642/0 (socket says 172.21.15.19:35642) 2024-03-18T15:15:45.718 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:45.716+0000 7f9bf5400640 1 -- 172.21.15.19:0/852830746 learned_addr learned my addr 172.21.15.19:0/852830746 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:15:45.718 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:45.716+0000 7f9bfe800640 1 --2- 172.21.15.19:0/852830746 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c001018b0 0x7f9c001a10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:45.718 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:45.717+0000 7f9bf5400640 1 -- 172.21.15.19:0/852830746 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c001018b0 msgr2=0x7f9c001a10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:45.718 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:45.717+0000 7f9bf5400640 1 --2- 172.21.15.19:0/852830746 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c001018b0 0x7f9c001a10f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:45.718 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:45.717+0000 7f9bf5400640 1 -- 172.21.15.19:0/852830746 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9c00100050 msgr2=0x7f9c00199b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:45.718 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:45.717+0000 7f9bf5400640 1 --2- 172.21.15.19:0/852830746 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9c00100050 0x7f9c00199b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:45.718 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:45.717+0000 7f9bf5400640 1 -- 172.21.15.19:0/852830746 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c001a1760 con 0x7f9c00100ef0 2024-03-18T15:15:45.718 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:45.717+0000 7f9bf5400640 1 --2- 172.21.15.19:0/852830746 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9c00100ef0 0x7f9c0019a080 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f9be800e9f0 tx=0x7f9be800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:45.720 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:45.717+0000 7f9bf7400640 1 -- 172.21.15.19:0/852830746 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9be800cd80 con 0x7f9c00100ef0 2024-03-18T15:15:45.720 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:45.717+0000 7f9bf7400640 1 -- 172.21.15.19:0/852830746 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9be8010910 con 0x7f9c00100ef0 2024-03-18T15:15:45.720 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:45.717+0000 7f9bf7400640 1 -- 172.21.15.19:0/852830746 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9be8018990 con 0x7f9c00100ef0 2024-03-18T15:15:45.721 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:45.717+0000 7f9bff200640 1 -- 172.21.15.19:0/852830746 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c001a1a50 con 0x7f9c00100ef0 2024-03-18T15:15:45.721 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:45.718+0000 7f9bff200640 1 -- 172.21.15.19:0/852830746 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c001a1f50 con 0x7f9c00100ef0 2024-03-18T15:15:45.721 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:45.719+0000 7f9bff200640 1 -- 172.21.15.19:0/852830746 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c00106fc0 con 0x7f9c00100ef0 2024-03-18T15:15:45.721 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:45.721+0000 7f9bf7400640 1 -- 172.21.15.19:0/852830746 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f9be8010ac0 con 0x7f9c00100ef0 2024-03-18T15:15:45.721 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:45.722+0000 7f9bf7400640 1 --2- 172.21.15.19:0/852830746 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f9be40a2880 0x7f9be40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:45.722 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:45.722+0000 7f9bf7400640 1 -- 172.21.15.19:0/852830746 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f9be80cefb0 con 0x7f9c00100ef0 2024-03-18T15:15:45.722 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:45.722+0000 7f9bfde00640 1 --2- 172.21.15.19:0/852830746 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f9be40a2880 0x7f9be40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:45.723 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:45.723+0000 7f9bfde00640 1 --2- 172.21.15.19:0/852830746 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f9be40a2880 0x7f9be40a4d40 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f9c00100dc0 tx=0x7f9bf0010040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:45.726 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:45.726+0000 7f9bf7400640 1 -- 172.21.15.19:0/852830746 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9be809b760 con 0x7f9c00100ef0 2024-03-18T15:15:45.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:45 smithi033 ceph-mon[36648]: pgmap v334: 129 pgs: 129 active+clean; 582 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-18T15:15:45.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:45 smithi033 ceph-mon[36648]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-18T15:15:45.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:45 smithi033 ceph-mon[36648]: mds.? [v2:172.21.15.104:6838/805341436,v1:172.21.15.104:6839/805341436] up:active 2024-03-18T15:15:45.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:45 smithi033 ceph-mon[36648]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-18T15:15:45.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:45 smithi033 ceph-mon[36648]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-03-18T15:15:45.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:45 smithi033 ceph-mon[36648]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:creating} 9 up:standby 2024-03-18T15:15:45.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:45 smithi033 ceph-mon[36648]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-03-18T15:15:45.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:45 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/660631835' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-18T15:15:45.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:45 smithi033 ceph-mon[36648]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-18T15:15:45.868 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:45 smithi019 ceph-mon[31522]: pgmap v334: 129 pgs: 129 active+clean; 582 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-18T15:15:45.868 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:45 smithi019 ceph-mon[31522]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-18T15:15:45.868 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:45 smithi019 ceph-mon[31522]: mds.? [v2:172.21.15.104:6838/805341436,v1:172.21.15.104:6839/805341436] up:active 2024-03-18T15:15:45.868 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:45 smithi019 ceph-mon[31522]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-18T15:15:45.868 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:45 smithi019 ceph-mon[31522]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-03-18T15:15:45.868 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:45 smithi019 ceph-mon[31522]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:creating} 9 up:standby 2024-03-18T15:15:45.868 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:45 smithi019 ceph-mon[31522]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-03-18T15:15:45.868 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:45 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/660631835' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-18T15:15:45.868 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:45 smithi019 ceph-mon[31522]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-18T15:15:45.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:45 smithi104 ceph-mon[36070]: pgmap v334: 129 pgs: 129 active+clean; 582 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-18T15:15:45.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:45 smithi104 ceph-mon[36070]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-18T15:15:45.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:45 smithi104 ceph-mon[36070]: mds.? [v2:172.21.15.104:6838/805341436,v1:172.21.15.104:6839/805341436] up:active 2024-03-18T15:15:45.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:45 smithi104 ceph-mon[36070]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-18T15:15:45.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:45 smithi104 ceph-mon[36070]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-03-18T15:15:45.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:45 smithi104 ceph-mon[36070]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:creating} 9 up:standby 2024-03-18T15:15:45.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:45 smithi104 ceph-mon[36070]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-03-18T15:15:45.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:45 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/660631835' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-18T15:15:45.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:45 smithi104 ceph-mon[36070]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-18T15:15:45.962 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:45.961+0000 7f9bff200640 1 -- 172.21.15.19:0/852830746 --> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f9c00061c20 con 0x7f9be40a2880 2024-03-18T15:15:45.994 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:45.994+0000 7f9bf7400640 1 -- 172.21.15.19:0/852830746 <== mgr.14152 v2:172.21.15.19:6800/2017714236 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f9c00061c20 con 0x7f9be40a2880 2024-03-18T15:15:45.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:45.996+0000 7f9bff200640 1 -- 172.21.15.19:0/852830746 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f9be40a2880 msgr2=0x7f9be40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:45.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:45.996+0000 7f9bff200640 1 --2- 172.21.15.19:0/852830746 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f9be40a2880 0x7f9be40a4d40 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f9c00100dc0 tx=0x7f9bf0010040 comp rx=0 tx=0).stop 2024-03-18T15:15:45.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:45.996+0000 7f9bff200640 1 -- 172.21.15.19:0/852830746 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9c00100ef0 msgr2=0x7f9c0019a080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:45.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:45.996+0000 7f9bff200640 1 --2- 172.21.15.19:0/852830746 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9c00100ef0 0x7f9c0019a080 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f9be800e9f0 tx=0x7f9be800eeb0 comp rx=0 tx=0).stop 2024-03-18T15:15:45.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:45.997+0000 7f9bff200640 1 -- 172.21.15.19:0/852830746 shutdown_connections 2024-03-18T15:15:45.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:45.997+0000 7f9bff200640 1 --2- 172.21.15.19:0/852830746 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c001018b0 0x7f9c001a10f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:45.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:45.997+0000 7f9bff200640 1 --2- 172.21.15.19:0/852830746 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f9be40a2880 0x7f9be40a4d40 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:45.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:45.997+0000 7f9bff200640 1 --2- 172.21.15.19:0/852830746 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9c00100ef0 0x7f9c0019a080 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:45.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:45.997+0000 7f9bff200640 1 --2- 172.21.15.19:0/852830746 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9c00100050 0x7f9c00199b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:45.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:45.997+0000 7f9bff200640 1 -- 172.21.15.19:0/852830746 >> 172.21.15.19:0/852830746 conn(0x7f9c000fae60 msgr2=0x7f9c00104190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:45.998 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:45.997+0000 7f9bff200640 1 -- 172.21.15.19:0/852830746 shutdown_connections 2024-03-18T15:15:45.998 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:45.997+0000 7f9bff200640 1 -- 172.21.15.19:0/852830746 wait complete. 2024-03-18T15:15:46.017 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-03-18T15:15:46.215 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:15:45 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:15:45.975+0000 7f4486a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T15:15:46.215 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:15:45 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:15:45.975+0000 7f4486a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T15:15:46.216 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:15:45 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:15:45.975+0000 7f4486a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T15:15:46.216 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:15:45 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:15:45.975+0000 7f4486a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T15:15:46.216 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:15:45 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:15:45.975+0000 7f4486a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T15:15:46.235 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:46.235+0000 7fd6acc00640 1 -- 172.21.15.19:0/2490788280 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd6a8102300 msgr2=0x7fd6a81046f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:46.235 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:46.235+0000 7fd6acc00640 1 --2- 172.21.15.19:0/2490788280 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd6a8102300 0x7fd6a81046f0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fd698009ca0 tx=0x7fd69802c0f0 comp rx=0 tx=0).stop 2024-03-18T15:15:46.235 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:46.235+0000 7fd6acc00640 1 -- 172.21.15.19:0/2490788280 shutdown_connections 2024-03-18T15:15:46.235 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:46.235+0000 7fd6acc00640 1 --2- 172.21.15.19:0/2490788280 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd6a8107560 0x7fd6a8109950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:46.235 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:46.235+0000 7fd6acc00640 1 --2- 172.21.15.19:0/2490788280 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd6a8104c30 0x7fd6a8107020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:46.236 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:46.235+0000 7fd6acc00640 1 --2- 172.21.15.19:0/2490788280 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd6a8102300 0x7fd6a81046f0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:46.236 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:46.235+0000 7fd6acc00640 1 -- 172.21.15.19:0/2490788280 >> 172.21.15.19:0/2490788280 conn(0x7fd6a80fbc70 msgr2=0x7fd6a80fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:46.236 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:46.235+0000 7fd6acc00640 1 -- 172.21.15.19:0/2490788280 shutdown_connections 2024-03-18T15:15:46.236 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:46.235+0000 7fd6acc00640 1 -- 172.21.15.19:0/2490788280 wait complete. 2024-03-18T15:15:46.236 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:46.236+0000 7fd6acc00640 1 Processor -- start 2024-03-18T15:15:46.236 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:46.237+0000 7fd6acc00640 1 -- start start 2024-03-18T15:15:46.238 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:46.237+0000 7fd6acc00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd6a8102300 0x7fd6a8101c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:46.238 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:46.237+0000 7fd6acc00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd6a8104c30 0x7fd6a81002b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:46.238 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:46.237+0000 7fd6acc00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd6a8107560 0x7fd6a81007f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:46.238 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:46.237+0000 7fd6acc00640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fd6a810d450 con 0x7fd6a8107560 2024-03-18T15:15:46.238 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:46.237+0000 7fd6acc00640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd6a810d2d0 con 0x7fd6a8104c30 2024-03-18T15:15:46.238 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:46.237+0000 7fd6acc00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fd6a810d5d0 con 0x7fd6a8102300 2024-03-18T15:15:46.238 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:46.237+0000 7fd6a5400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd6a8104c30 0x7fd6a81002b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:46.238 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:46.237+0000 7fd6a5e00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd6a8102300 0x7fd6a8101c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:46.238 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:46.237+0000 7fd6a5400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd6a8104c30 0x7fd6a81002b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:35682/0 (socket says 172.21.15.19:35682) 2024-03-18T15:15:46.238 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:46.237+0000 7fd6a5400640 1 -- 172.21.15.19:0/1159158811 learned_addr learned my addr 172.21.15.19:0/1159158811 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:15:46.238 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:46.237+0000 7fd6a6800640 1 --2- 172.21.15.19:0/1159158811 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd6a8107560 0x7fd6a81007f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:46.239 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:46.238+0000 7fd6a5400640 1 -- 172.21.15.19:0/1159158811 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd6a8102300 msgr2=0x7fd6a8101c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:46.239 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:46.238+0000 7fd6a5400640 1 --2- 172.21.15.19:0/1159158811 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd6a8102300 0x7fd6a8101c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:46.239 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:46.238+0000 7fd6a5400640 1 -- 172.21.15.19:0/1159158811 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd6a8107560 msgr2=0x7fd6a81007f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:46.239 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:46.238+0000 7fd6a5400640 1 --2- 172.21.15.19:0/1159158811 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd6a8107560 0x7fd6a81007f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:46.239 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:46.238+0000 7fd6a5400640 1 -- 172.21.15.19:0/1159158811 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6a8101080 con 0x7fd6a8104c30 2024-03-18T15:15:46.239 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:46.238+0000 7fd6a6800640 1 --2- 172.21.15.19:0/1159158811 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd6a8107560 0x7fd6a81007f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T15:15:46.239 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:46.238+0000 7fd6a5400640 1 --2- 172.21.15.19:0/1159158811 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd6a8104c30 0x7fd6a81002b0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fd69000e9c0 tx=0x7fd69000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:46.240 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:46.238+0000 7fd69ea00640 1 -- 172.21.15.19:0/1159158811 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd69000cd80 con 0x7fd6a8104c30 2024-03-18T15:15:46.240 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:46.238+0000 7fd69ea00640 1 -- 172.21.15.19:0/1159158811 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd690005500 con 0x7fd6a8104c30 2024-03-18T15:15:46.240 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:46.238+0000 7fd69ea00640 1 -- 172.21.15.19:0/1159158811 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd690010690 con 0x7fd6a8104c30 2024-03-18T15:15:46.240 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:46.238+0000 7fd6acc00640 1 -- 172.21.15.19:0/1159158811 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6a81a1ca0 con 0x7fd6a8104c30 2024-03-18T15:15:46.240 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:46.238+0000 7fd6acc00640 1 -- 172.21.15.19:0/1159158811 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6a81a2190 con 0x7fd6a8104c30 2024-03-18T15:15:46.240 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:46.240+0000 7fd6acc00640 1 -- 172.21.15.19:0/1159158811 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6a80fd760 con 0x7fd6a8104c30 2024-03-18T15:15:46.242 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:46.242+0000 7fd69ea00640 1 -- 172.21.15.19:0/1159158811 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fd6900046c0 con 0x7fd6a8104c30 2024-03-18T15:15:46.244 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:46.243+0000 7fd69ea00640 1 --2- 172.21.15.19:0/1159158811 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fd68c0a2880 0x7fd68c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:46.244 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:46.244+0000 7fd69ea00640 1 -- 172.21.15.19:0/1159158811 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fd6900cf0c0 con 0x7fd6a8104c30 2024-03-18T15:15:46.244 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:46.244+0000 7fd6a5e00640 1 --2- 172.21.15.19:0/1159158811 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fd68c0a2880 0x7fd68c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:46.244 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:46.245+0000 7fd6a5e00640 1 --2- 172.21.15.19:0/1159158811 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fd68c0a2880 0x7fd68c0a4d40 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fd69802c700 tx=0x7fd6980323c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:46.247 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:46.247+0000 7fd69ea00640 1 -- 172.21.15.19:0/1159158811 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd69009bec0 con 0x7fd6a8104c30 2024-03-18T15:15:46.480 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:46.480+0000 7fd6acc00640 1 -- 172.21.15.19:0/1159158811 --> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fd6a8061cb0 con 0x7fd68c0a2880 2024-03-18T15:15:46.513 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:46.512+0000 7fd69ea00640 1 -- 172.21.15.19:0/1159158811 <== mgr.14152 v2:172.21.15.19:6800/2017714236 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fd6a8061cb0 con 0x7fd68c0a2880 2024-03-18T15:15:46.516 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:46.516+0000 7fd6acc00640 1 -- 172.21.15.19:0/1159158811 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fd68c0a2880 msgr2=0x7fd68c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:46.516 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:46.516+0000 7fd6acc00640 1 --2- 172.21.15.19:0/1159158811 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fd68c0a2880 0x7fd68c0a4d40 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fd69802c700 tx=0x7fd6980323c0 comp rx=0 tx=0).stop 2024-03-18T15:15:46.516 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:46.516+0000 7fd6acc00640 1 -- 172.21.15.19:0/1159158811 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd6a8104c30 msgr2=0x7fd6a81002b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:46.517 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:46.517+0000 7fd69ea00640 1 -- 172.21.15.19:0/1159158811 <== mon.2 v2:172.21.15.33:3300/0 7 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fd690097e30 con 0x7fd6a8104c30 2024-03-18T15:15:46.517 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:46.517+0000 7fd6acc00640 1 --2- 172.21.15.19:0/1159158811 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd6a8104c30 0x7fd6a81002b0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fd69000e9c0 tx=0x7fd69000ee80 comp rx=0 tx=0).stop 2024-03-18T15:15:46.518 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:46.518+0000 7fd6acc00640 1 -- 172.21.15.19:0/1159158811 shutdown_connections 2024-03-18T15:15:46.518 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:46.518+0000 7fd6acc00640 1 --2- 172.21.15.19:0/1159158811 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd6a8107560 0x7fd6a81007f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:46.518 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:46.518+0000 7fd6acc00640 1 --2- 172.21.15.19:0/1159158811 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fd68c0a2880 0x7fd68c0a4d40 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:46.518 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:46.518+0000 7fd6acc00640 1 --2- 172.21.15.19:0/1159158811 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd6a8104c30 0x7fd6a81002b0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:46.518 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:46.518+0000 7fd6acc00640 1 --2- 172.21.15.19:0/1159158811 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd6a8102300 0x7fd6a8101c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:46.518 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:46.518+0000 7fd6acc00640 1 -- 172.21.15.19:0/1159158811 >> 172.21.15.19:0/1159158811 conn(0x7fd6a80fbc70 msgr2=0x7fd6a8102d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:46.518 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:46.518+0000 7fd6acc00640 1 -- 172.21.15.19:0/1159158811 shutdown_connections 2024-03-18T15:15:46.519 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:46.518+0000 7fd6acc00640 1 -- 172.21.15.19:0/1159158811 wait complete. 2024-03-18T15:15:46.549 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-03-18T15:15:46.753 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:46.753+0000 7f62efe00640 1 -- 172.21.15.19:0/3487428890 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f62f0103900 msgr2=0x7f62f0103d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:46.754 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:46.753+0000 7f62efe00640 1 --2- 172.21.15.19:0/3487428890 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f62f0103900 0x7f62f0103d20 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f62e0009900 tx=0x7f62e002bc40 comp rx=0 tx=0).stop 2024-03-18T15:15:46.754 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:46.753+0000 7f62efe00640 1 -- 172.21.15.19:0/3487428890 shutdown_connections 2024-03-18T15:15:46.754 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:46.753+0000 7f62efe00640 1 --2- 172.21.15.19:0/3487428890 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f62f01054b0 0x7f62f010bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:46.754 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:46.753+0000 7f62efe00640 1 --2- 172.21.15.19:0/3487428890 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62f0104af0 0x7f62f0104f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:46.754 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:46.753+0000 7f62efe00640 1 --2- 172.21.15.19:0/3487428890 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f62f0103900 0x7f62f0103d20 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:46.754 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:46.753+0000 7f62efe00640 1 -- 172.21.15.19:0/3487428890 >> 172.21.15.19:0/3487428890 conn(0x7f62f00fee90 msgr2=0x7f62f01012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:46.754 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:46.753+0000 7f62efe00640 1 -- 172.21.15.19:0/3487428890 shutdown_connections 2024-03-18T15:15:46.754 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:46.754+0000 7f62efe00640 1 -- 172.21.15.19:0/3487428890 wait complete. 2024-03-18T15:15:46.755 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:46.755+0000 7f62efe00640 1 Processor -- start 2024-03-18T15:15:46.755 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:46.755+0000 7f62efe00640 1 -- start start 2024-03-18T15:15:46.756 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:46.755+0000 7f62efe00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f62f0103900 0x7f62f0079590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:46.756 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:46.755+0000 7f62efe00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62f0104af0 0x7f62f0079ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:46.756 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:46.756+0000 7f62efe00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f62f01054b0 0x7f62f0076100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:46.756 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:46.756+0000 7f62efe00640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f62f010f570 con 0x7f62f0103900 2024-03-18T15:15:46.757 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:46.756+0000 7f62eea00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f62f0103900 0x7f62f0079590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:46.757 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:46.756+0000 7f62efe00640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f62f010f3f0 con 0x7f62f01054b0 2024-03-18T15:15:46.757 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:46.756+0000 7f62efe00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f62f010f6f0 con 0x7f62f0104af0 2024-03-18T15:15:46.757 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:46.756+0000 7f62e6000640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62f0104af0 0x7f62f0079ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:46.757 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:46.756+0000 7f62eea00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f62f0103900 0x7f62f0079590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:49208/0 (socket says 172.21.15.19:49208) 2024-03-18T15:15:46.757 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:46.756+0000 7f62eea00640 1 -- 172.21.15.19:0/1608590986 learned_addr learned my addr 172.21.15.19:0/1608590986 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:15:46.757 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:46.756+0000 7f62ef400640 1 --2- 172.21.15.19:0/1608590986 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f62f01054b0 0x7f62f0076100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:46.757 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:46.756+0000 7f62e6000640 1 -- 172.21.15.19:0/1608590986 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f62f01054b0 msgr2=0x7f62f0076100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:46.757 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:46.756+0000 7f62e6000640 1 --2- 172.21.15.19:0/1608590986 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f62f01054b0 0x7f62f0076100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:46.757 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:46.756+0000 7f62e6000640 1 -- 172.21.15.19:0/1608590986 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f62f0103900 msgr2=0x7f62f0079590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:46.757 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:46.756+0000 7f62e6000640 1 --2- 172.21.15.19:0/1608590986 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f62f0103900 0x7f62f0079590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:46.757 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:46.756+0000 7f62e6000640 1 -- 172.21.15.19:0/1608590986 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62f00769c0 con 0x7f62f0104af0 2024-03-18T15:15:46.757 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:46.756+0000 7f62eea00640 1 --2- 172.21.15.19:0/1608590986 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f62f0103900 0x7f62f0079590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:15:46.757 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:46.756+0000 7f62e6000640 1 --2- 172.21.15.19:0/1608590986 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62f0104af0 0x7f62f0079ad0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f62d800e9f0 tx=0x7f62d800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:46.758 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:46.757+0000 7f62e7e00640 1 -- 172.21.15.19:0/1608590986 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62d800ccc0 con 0x7f62f0104af0 2024-03-18T15:15:46.758 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:46.757+0000 7f62e7e00640 1 -- 172.21.15.19:0/1608590986 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f62d8005020 con 0x7f62f0104af0 2024-03-18T15:15:46.758 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:46.757+0000 7f62e7e00640 1 -- 172.21.15.19:0/1608590986 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62d80049e0 con 0x7f62f0104af0 2024-03-18T15:15:46.758 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:46.757+0000 7f62efe00640 1 -- 172.21.15.19:0/1608590986 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62f01a6090 con 0x7f62f0104af0 2024-03-18T15:15:46.758 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:46.757+0000 7f62efe00640 1 -- 172.21.15.19:0/1608590986 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62f01a64a0 con 0x7f62f0104af0 2024-03-18T15:15:46.759 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:46.758+0000 7f62efe00640 1 -- 172.21.15.19:0/1608590986 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62f010ac70 con 0x7f62f0104af0 2024-03-18T15:15:46.760 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:46.760+0000 7f62e7e00640 1 -- 172.21.15.19:0/1608590986 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f62d8005560 con 0x7f62f0104af0 2024-03-18T15:15:46.761 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:46.761+0000 7f62e7e00640 1 --2- 172.21.15.19:0/1608590986 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f62d00a2890 0x7f62d00a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:46.761 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:46.761+0000 7f62e7e00640 1 -- 172.21.15.19:0/1608590986 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f62d80cee80 con 0x7f62f0104af0 2024-03-18T15:15:46.762 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:46.762+0000 7f62eea00640 1 --2- 172.21.15.19:0/1608590986 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f62d00a2890 0x7f62d00a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:46.762 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:46.762+0000 7f62eea00640 1 --2- 172.21.15.19:0/1608590986 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f62d00a2890 0x7f62d00a4d50 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f62e002c8f0 tx=0x7f62e0005b50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:46.770 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:46.770+0000 7f62e7e00640 1 -- 172.21.15.19:0/1608590986 <== mon.1 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f62d80051d0 con 0x7f62f0104af0 2024-03-18T15:15:46.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:46 smithi104 ceph-mon[36070]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-18T15:15:46.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:46 smithi104 ceph-mon[36070]: mds.? [v2:172.21.15.104:6836/260984674,v1:172.21.15.104:6837/260984674] up:active 2024-03-18T15:15:46.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:46 smithi104 ceph-mon[36070]: daemon mds.h assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-03-18T15:15:46.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:46 smithi104 ceph-mon[36070]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 9 up:standby 2024-03-18T15:15:46.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:46 smithi104 ceph-mon[36070]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=c=up:active,3=h=up:creating} 8 up:standby 2024-03-18T15:15:46.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:46 smithi104 ceph-mon[36070]: daemon mds.h is now active in filesystem cephfs as rank 3 2024-03-18T15:15:46.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:46 smithi104 ceph-mon[36070]: from='client.24800 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:15:46.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:46 smithi019 ceph-mon[31522]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-18T15:15:46.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:46 smithi019 ceph-mon[31522]: mds.? [v2:172.21.15.104:6836/260984674,v1:172.21.15.104:6837/260984674] up:active 2024-03-18T15:15:46.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:46 smithi019 ceph-mon[31522]: daemon mds.h assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-03-18T15:15:46.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:46 smithi019 ceph-mon[31522]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 9 up:standby 2024-03-18T15:15:46.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:46 smithi019 ceph-mon[31522]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=c=up:active,3=h=up:creating} 8 up:standby 2024-03-18T15:15:46.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:46 smithi019 ceph-mon[31522]: daemon mds.h is now active in filesystem cephfs as rank 3 2024-03-18T15:15:46.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:46 smithi019 ceph-mon[31522]: from='client.24800 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:15:47.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:46 smithi033 ceph-mon[36648]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-18T15:15:47.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:46 smithi033 ceph-mon[36648]: mds.? [v2:172.21.15.104:6836/260984674,v1:172.21.15.104:6837/260984674] up:active 2024-03-18T15:15:47.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:46 smithi033 ceph-mon[36648]: daemon mds.h assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-03-18T15:15:47.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:46 smithi033 ceph-mon[36648]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 9 up:standby 2024-03-18T15:15:47.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:46 smithi033 ceph-mon[36648]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=c=up:active,3=h=up:creating} 8 up:standby 2024-03-18T15:15:47.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:46 smithi033 ceph-mon[36648]: daemon mds.h is now active in filesystem cephfs as rank 3 2024-03-18T15:15:47.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:46 smithi033 ceph-mon[36648]: from='client.24800 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:15:47.014 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.014+0000 7f62efe00640 1 -- 172.21.15.19:0/1608590986 --> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7f62f0061c20 con 0x7f62d00a2890 2024-03-18T15:15:47.019 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.019+0000 7f62e7e00640 1 -- 172.21.15.19:0/1608590986 <== mgr.14152 v2:172.21.15.19:6800/2017714236 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f62f0061c20 con 0x7f62d00a2890 2024-03-18T15:15:47.019 INFO:teuthology.orchestra.run.smithi019.stdout:/volumes/qa/sv_0/c7e9a40c-1dc2-46ba-8e54-92384c423055 2024-03-18T15:15:47.022 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.021+0000 7f62efe00640 1 -- 172.21.15.19:0/1608590986 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f62d00a2890 msgr2=0x7f62d00a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:47.022 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.021+0000 7f62efe00640 1 --2- 172.21.15.19:0/1608590986 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f62d00a2890 0x7f62d00a4d50 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f62e002c8f0 tx=0x7f62e0005b50 comp rx=0 tx=0).stop 2024-03-18T15:15:47.022 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.021+0000 7f62efe00640 1 -- 172.21.15.19:0/1608590986 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62f0104af0 msgr2=0x7f62f0079ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:47.022 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.021+0000 7f62efe00640 1 --2- 172.21.15.19:0/1608590986 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62f0104af0 0x7f62f0079ad0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f62d800e9f0 tx=0x7f62d800eeb0 comp rx=0 tx=0).stop 2024-03-18T15:15:47.022 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.021+0000 7f62efe00640 1 -- 172.21.15.19:0/1608590986 shutdown_connections 2024-03-18T15:15:47.022 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.021+0000 7f62efe00640 1 --2- 172.21.15.19:0/1608590986 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f62d00a2890 0x7f62d00a4d50 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:47.022 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.021+0000 7f62efe00640 1 --2- 172.21.15.19:0/1608590986 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f62f01054b0 0x7f62f0076100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:47.022 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.021+0000 7f62efe00640 1 --2- 172.21.15.19:0/1608590986 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62f0104af0 0x7f62f0079ad0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:47.022 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.021+0000 7f62efe00640 1 --2- 172.21.15.19:0/1608590986 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f62f0103900 0x7f62f0079590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:47.023 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.021+0000 7f62efe00640 1 -- 172.21.15.19:0/1608590986 >> 172.21.15.19:0/1608590986 conn(0x7f62f00fee90 msgr2=0x7f62f0107e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:47.023 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.022+0000 7f62efe00640 1 -- 172.21.15.19:0/1608590986 shutdown_connections 2024-03-18T15:15:47.023 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.022+0000 7f62efe00640 1 -- 172.21.15.19:0/1608590986 wait complete. 2024-03-18T15:15:47.040 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-03-18T15:15:47.245 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.244+0000 7fc57b400640 1 -- 172.21.15.19:0/3536132016 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc574104c10 msgr2=0x7fc574105090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:47.245 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.244+0000 7fc57b400640 1 --2- 172.21.15.19:0/3536132016 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc574104c10 0x7fc574105090 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fc560009860 tx=0x7fc56002bb70 comp rx=0 tx=0).stop 2024-03-18T15:15:47.245 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.245+0000 7fc57b400640 1 -- 172.21.15.19:0/3536132016 shutdown_connections 2024-03-18T15:15:47.245 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.245+0000 7fc57b400640 1 --2- 172.21.15.19:0/3536132016 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc5741055d0 0x7fc57410be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:47.246 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.245+0000 7fc57b400640 1 --2- 172.21.15.19:0/3536132016 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc574104c10 0x7fc574105090 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:47.246 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.245+0000 7fc57b400640 1 --2- 172.21.15.19:0/3536132016 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc574103a20 0x7fc574103e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:47.246 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.245+0000 7fc57b400640 1 -- 172.21.15.19:0/3536132016 >> 172.21.15.19:0/3536132016 conn(0x7fc5740fef90 msgr2=0x7fc5741013d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:47.246 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.245+0000 7fc57b400640 1 -- 172.21.15.19:0/3536132016 shutdown_connections 2024-03-18T15:15:47.246 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.245+0000 7fc57b400640 1 -- 172.21.15.19:0/3536132016 wait complete. 2024-03-18T15:15:47.248 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.246+0000 7fc57b400640 1 Processor -- start 2024-03-18T15:15:47.248 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.246+0000 7fc57b400640 1 -- start start 2024-03-18T15:15:47.248 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.247+0000 7fc57b400640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc574103a20 0x7fc574199d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:47.248 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.247+0000 7fc57b400640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc574104c10 0x7fc57419a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:47.248 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.247+0000 7fc57b400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc5741055d0 0x7fc5741a12a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:47.248 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.247+0000 7fc57b400640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fc57410f6a0 con 0x7fc574104c10 2024-03-18T15:15:47.248 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.247+0000 7fc57b400640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc57410f520 con 0x7fc5741055d0 2024-03-18T15:15:47.249 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.247+0000 7fc57b400640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fc57410f820 con 0x7fc574103a20 2024-03-18T15:15:47.249 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.247+0000 7fc578a00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc574103a20 0x7fc574199d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:47.249 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.247+0000 7fc578a00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc574103a20 0x7fc574199d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.19:36472/0 (socket says 172.21.15.19:36472) 2024-03-18T15:15:47.249 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.247+0000 7fc578a00640 1 -- 172.21.15.19:0/4165629863 learned_addr learned my addr 172.21.15.19:0/4165629863 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:15:47.249 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.247+0000 7fc573e00640 1 --2- 172.21.15.19:0/4165629863 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc574104c10 0x7fc57419a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:47.249 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.247+0000 7fc579400640 1 --2- 172.21.15.19:0/4165629863 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc5741055d0 0x7fc5741a12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:47.249 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.247+0000 7fc578a00640 1 -- 172.21.15.19:0/4165629863 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc5741055d0 msgr2=0x7fc5741a12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:47.249 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.247+0000 7fc578a00640 1 --2- 172.21.15.19:0/4165629863 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc5741055d0 0x7fc5741a12a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:47.249 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.247+0000 7fc578a00640 1 -- 172.21.15.19:0/4165629863 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc574104c10 msgr2=0x7fc57419a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:47.249 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.247+0000 7fc578a00640 1 --2- 172.21.15.19:0/4165629863 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc574104c10 0x7fc57419a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:47.249 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.247+0000 7fc578a00640 1 -- 172.21.15.19:0/4165629863 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5741a1940 con 0x7fc574103a20 2024-03-18T15:15:47.249 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.248+0000 7fc578a00640 1 --2- 172.21.15.19:0/4165629863 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc574103a20 0x7fc574199d30 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fc56400e9c0 tx=0x7fc56400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:47.249 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.248+0000 7fc571600640 1 -- 172.21.15.19:0/4165629863 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc56400cd80 con 0x7fc574103a20 2024-03-18T15:15:47.249 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.248+0000 7fc571600640 1 -- 172.21.15.19:0/4165629863 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc5640035f0 con 0x7fc574103a20 2024-03-18T15:15:47.249 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.248+0000 7fc57b400640 1 -- 172.21.15.19:0/4165629863 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5741a1c30 con 0x7fc574103a20 2024-03-18T15:15:47.250 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.248+0000 7fc571600640 1 -- 172.21.15.19:0/4165629863 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc564010690 con 0x7fc574103a20 2024-03-18T15:15:47.250 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.248+0000 7fc57b400640 1 -- 172.21.15.19:0/4165629863 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5741a2180 con 0x7fc574103a20 2024-03-18T15:15:47.250 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.250+0000 7fc57b400640 1 -- 172.21.15.19:0/4165629863 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc57410ad90 con 0x7fc574103a20 2024-03-18T15:15:47.251 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.251+0000 7fc571600640 1 -- 172.21.15.19:0/4165629863 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fc5640037a0 con 0x7fc574103a20 2024-03-18T15:15:47.252 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.252+0000 7fc571600640 1 --2- 172.21.15.19:0/4165629863 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fc5580a2890 0x7fc5580a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:47.253 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.253+0000 7fc571600640 1 -- 172.21.15.19:0/4165629863 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fc5640cf2b0 con 0x7fc574103a20 2024-03-18T15:15:47.253 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.253+0000 7fc573e00640 1 --2- 172.21.15.19:0/4165629863 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fc5580a2890 0x7fc5580a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:47.253 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.253+0000 7fc573e00640 1 --2- 172.21.15.19:0/4165629863 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fc5580a2890 0x7fc5580a4d50 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fc560009830 tx=0x7fc560034ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:47.256 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.256+0000 7fc571600640 1 -- 172.21.15.19:0/4165629863 <== mon.1 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc56409bf50 con 0x7fc574103a20 2024-03-18T15:15:47.495 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.495+0000 7fc57b400640 1 -- 172.21.15.19:0/4165629863 --> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fc574061c20 con 0x7fc5580a2890 2024-03-18T15:15:47.526 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.526+0000 7fc571600640 1 -- 172.21.15.19:0/4165629863 <== mgr.14152 v2:172.21.15.19:6800/2017714236 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fc574061c20 con 0x7fc5580a2890 2024-03-18T15:15:47.529 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.529+0000 7fc57b400640 1 -- 172.21.15.19:0/4165629863 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fc5580a2890 msgr2=0x7fc5580a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:47.530 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.529+0000 7fc57b400640 1 --2- 172.21.15.19:0/4165629863 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fc5580a2890 0x7fc5580a4d50 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fc560009830 tx=0x7fc560034ab0 comp rx=0 tx=0).stop 2024-03-18T15:15:47.530 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.529+0000 7fc57b400640 1 -- 172.21.15.19:0/4165629863 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc574103a20 msgr2=0x7fc574199d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:47.530 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.529+0000 7fc57b400640 1 --2- 172.21.15.19:0/4165629863 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc574103a20 0x7fc574199d30 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fc56400e9c0 tx=0x7fc56400ee80 comp rx=0 tx=0).stop 2024-03-18T15:15:47.530 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.529+0000 7fc57b400640 1 -- 172.21.15.19:0/4165629863 shutdown_connections 2024-03-18T15:15:47.530 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.529+0000 7fc57b400640 1 --2- 172.21.15.19:0/4165629863 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fc5580a2890 0x7fc5580a4d50 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:47.530 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.529+0000 7fc57b400640 1 --2- 172.21.15.19:0/4165629863 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc5741055d0 0x7fc5741a12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:47.530 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.529+0000 7fc57b400640 1 --2- 172.21.15.19:0/4165629863 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc574104c10 0x7fc57419a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:47.530 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.529+0000 7fc57b400640 1 --2- 172.21.15.19:0/4165629863 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc574103a20 0x7fc574199d30 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:47.530 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.529+0000 7fc57b400640 1 -- 172.21.15.19:0/4165629863 >> 172.21.15.19:0/4165629863 conn(0x7fc5740fef90 msgr2=0x7fc574107f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:47.530 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.529+0000 7fc57b400640 1 -- 172.21.15.19:0/4165629863 shutdown_connections 2024-03-18T15:15:47.530 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.529+0000 7fc57b400640 1 -- 172.21.15.19:0/4165629863 wait complete. 2024-03-18T15:15:47.549 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-03-18T15:15:47.753 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.752+0000 7ff814e00640 1 -- 172.21.15.19:0/3269485496 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff810107e00 msgr2=0x7ff810108220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:47.753 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.752+0000 7ff814e00640 1 --2- 172.21.15.19:0/3269485496 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff810107e00 0x7ff810108220 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7ff800009900 tx=0x7ff80002bc50 comp rx=0 tx=0).stop 2024-03-18T15:15:47.753 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.752+0000 7ff814e00640 1 -- 172.21.15.19:0/3269485496 shutdown_connections 2024-03-18T15:15:47.753 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.752+0000 7ff814e00640 1 --2- 172.21.15.19:0/3269485496 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff8101099b0 0x7ff810110240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:47.753 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.752+0000 7ff814e00640 1 --2- 172.21.15.19:0/3269485496 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff810108ff0 0x7ff810109470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:47.753 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.752+0000 7ff814e00640 1 --2- 172.21.15.19:0/3269485496 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff810107e00 0x7ff810108220 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:47.753 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.752+0000 7ff814e00640 1 -- 172.21.15.19:0/3269485496 >> 172.21.15.19:0/3269485496 conn(0x7ff810076da0 msgr2=0x7ff8100791e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:47.753 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.752+0000 7ff814e00640 1 -- 172.21.15.19:0/3269485496 shutdown_connections 2024-03-18T15:15:47.753 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.752+0000 7ff814e00640 1 -- 172.21.15.19:0/3269485496 wait complete. 2024-03-18T15:15:47.756 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.754+0000 7ff814e00640 1 Processor -- start 2024-03-18T15:15:47.757 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.754+0000 7ff814e00640 1 -- start start 2024-03-18T15:15:47.757 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.754+0000 7ff814e00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff810107e00 0x7ff81019e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:47.757 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.754+0000 7ff814e00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff810108ff0 0x7ff81019e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:47.757 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.754+0000 7ff814e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff8101099b0 0x7ff8101a56a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:47.757 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.754+0000 7ff814e00640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7ff810113aa0 con 0x7ff810108ff0 2024-03-18T15:15:47.757 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.754+0000 7ff814e00640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff810113920 con 0x7ff8101099b0 2024-03-18T15:15:47.757 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.754+0000 7ff814e00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7ff810113c20 con 0x7ff810107e00 2024-03-18T15:15:47.757 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.754+0000 7ff80de00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff810107e00 0x7ff81019e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:47.757 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.755+0000 7ff80e800640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff8101099b0 0x7ff8101a56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:47.758 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.755+0000 7ff80de00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff810107e00 0x7ff81019e0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.19:36490/0 (socket says 172.21.15.19:36490) 2024-03-18T15:15:47.758 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.755+0000 7ff80de00640 1 -- 172.21.15.19:0/82878934 learned_addr learned my addr 172.21.15.19:0/82878934 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:15:47.758 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.755+0000 7ff80d400640 1 --2- 172.21.15.19:0/82878934 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff810108ff0 0x7ff81019e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:47.758 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.755+0000 7ff80e800640 1 -- 172.21.15.19:0/82878934 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff810107e00 msgr2=0x7ff81019e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:47.758 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.755+0000 7ff80e800640 1 --2- 172.21.15.19:0/82878934 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff810107e00 0x7ff81019e0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:47.758 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.755+0000 7ff80e800640 1 -- 172.21.15.19:0/82878934 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff810108ff0 msgr2=0x7ff81019e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:47.758 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.755+0000 7ff80e800640 1 --2- 172.21.15.19:0/82878934 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff810108ff0 0x7ff81019e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:47.758 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.755+0000 7ff80e800640 1 -- 172.21.15.19:0/82878934 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8101a5d10 con 0x7ff8101099b0 2024-03-18T15:15:47.758 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.755+0000 7ff80de00640 1 --2- 172.21.15.19:0/82878934 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff810107e00 0x7ff81019e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:15:47.758 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.755+0000 7ff80e800640 1 --2- 172.21.15.19:0/82878934 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff8101099b0 0x7ff8101a56a0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7ff808009fd0 tx=0x7ff80800ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:47.758 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.755+0000 7ff806a00640 1 -- 172.21.15.19:0/82878934 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff80800ce20 con 0x7ff8101099b0 2024-03-18T15:15:47.758 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.756+0000 7ff806a00640 1 -- 172.21.15.19:0/82878934 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff808005020 con 0x7ff8101099b0 2024-03-18T15:15:47.758 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.756+0000 7ff814e00640 1 -- 172.21.15.19:0/82878934 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8101a5fa0 con 0x7ff8101099b0 2024-03-18T15:15:47.759 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.756+0000 7ff806a00640 1 -- 172.21.15.19:0/82878934 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff808010720 con 0x7ff8101099b0 2024-03-18T15:15:47.759 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.756+0000 7ff814e00640 1 -- 172.21.15.19:0/82878934 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8101a6470 con 0x7ff8101099b0 2024-03-18T15:15:47.759 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.757+0000 7ff814e00640 1 -- 172.21.15.19:0/82878934 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff81010f120 con 0x7ff8101099b0 2024-03-18T15:15:47.759 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.759+0000 7ff806a00640 1 -- 172.21.15.19:0/82878934 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7ff808005540 con 0x7ff8101099b0 2024-03-18T15:15:47.760 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.760+0000 7ff806a00640 1 --2- 172.21.15.19:0/82878934 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7ff7f40a2890 0x7ff7f40a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:47.760 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.760+0000 7ff806a00640 1 -- 172.21.15.19:0/82878934 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7ff8080cf570 con 0x7ff8101099b0 2024-03-18T15:15:47.760 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.760+0000 7ff80de00640 1 --2- 172.21.15.19:0/82878934 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7ff7f40a2890 0x7ff7f40a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:47.761 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.761+0000 7ff80de00640 1 --2- 172.21.15.19:0/82878934 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7ff7f40a2890 0x7ff7f40a4d50 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7ff80002c900 tx=0x7ff800000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:47.764 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:47.764+0000 7ff806a00640 1 -- 172.21.15.19:0/82878934 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff80809c210 con 0x7ff8101099b0 2024-03-18T15:15:47.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:47 smithi104 ceph-mon[36070]: pgmap v335: 129 pgs: 129 active+clean; 583 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s wr, 4 op/s 2024-03-18T15:15:47.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:47 smithi104 ceph-mon[36070]: from='client.24812 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:15:47.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:47 smithi104 ceph-mon[36070]: mds.? [v2:172.21.15.33:6836/160187378,v1:172.21.15.33:6837/160187378] up:active 2024-03-18T15:15:47.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:47 smithi104 ceph-mon[36070]: daemon mds.f assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-03-18T15:15:47.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:47 smithi104 ceph-mon[36070]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-18T15:15:47.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:47 smithi104 ceph-mon[36070]: Cluster is now healthy 2024-03-18T15:15:47.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:47 smithi104 ceph-mon[36070]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=c=up:active,3=h=up:active} 8 up:standby 2024-03-18T15:15:47.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:47 smithi104 ceph-mon[36070]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-03-18T15:15:47.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:47 smithi104 ceph-mon[36070]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-03-18T15:15:47.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:47 smithi104 ceph-mon[36070]: daemon mds.f is now active in filesystem cephfs as rank 4 2024-03-18T15:15:47.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:47 smithi104 ceph-mon[36070]: mds.? [v2:172.21.15.104:6834/3788595957,v1:172.21.15.104:6835/3788595957] up:active 2024-03-18T15:15:47.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:47 smithi104 ceph-mon[36070]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T15:15:47.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:47 smithi104 ceph-mon[36070]: from='client.24829 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:15:47.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:47 smithi019 ceph-mon[31522]: pgmap v335: 129 pgs: 129 active+clean; 583 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s wr, 4 op/s 2024-03-18T15:15:47.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:47 smithi019 ceph-mon[31522]: from='client.24812 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:15:47.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:47 smithi019 ceph-mon[31522]: mds.? [v2:172.21.15.33:6836/160187378,v1:172.21.15.33:6837/160187378] up:active 2024-03-18T15:15:47.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:47 smithi019 ceph-mon[31522]: daemon mds.f assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-03-18T15:15:47.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:47 smithi019 ceph-mon[31522]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-18T15:15:47.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:47 smithi019 ceph-mon[31522]: Cluster is now healthy 2024-03-18T15:15:47.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:47 smithi019 ceph-mon[31522]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=c=up:active,3=h=up:active} 8 up:standby 2024-03-18T15:15:47.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:47 smithi019 ceph-mon[31522]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-03-18T15:15:47.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:47 smithi019 ceph-mon[31522]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-03-18T15:15:47.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:47 smithi019 ceph-mon[31522]: daemon mds.f is now active in filesystem cephfs as rank 4 2024-03-18T15:15:47.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:47 smithi019 ceph-mon[31522]: mds.? [v2:172.21.15.104:6834/3788595957,v1:172.21.15.104:6835/3788595957] up:active 2024-03-18T15:15:47.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:47 smithi019 ceph-mon[31522]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T15:15:47.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:47 smithi019 ceph-mon[31522]: from='client.24829 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:15:48.002 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.002+0000 7ff814e00640 1 -- 172.21.15.19:0/82878934 --> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7ff810061c20 con 0x7ff7f40a2890 2024-03-18T15:15:48.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:47 smithi033 ceph-mon[36648]: pgmap v335: 129 pgs: 129 active+clean; 583 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s wr, 4 op/s 2024-03-18T15:15:48.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:47 smithi033 ceph-mon[36648]: from='client.24812 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:15:48.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:47 smithi033 ceph-mon[36648]: mds.? [v2:172.21.15.33:6836/160187378,v1:172.21.15.33:6837/160187378] up:active 2024-03-18T15:15:48.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:47 smithi033 ceph-mon[36648]: daemon mds.f assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-03-18T15:15:48.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:47 smithi033 ceph-mon[36648]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-18T15:15:48.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:47 smithi033 ceph-mon[36648]: Cluster is now healthy 2024-03-18T15:15:48.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:47 smithi033 ceph-mon[36648]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=c=up:active,3=h=up:active} 8 up:standby 2024-03-18T15:15:48.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:47 smithi033 ceph-mon[36648]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-03-18T15:15:48.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:47 smithi033 ceph-mon[36648]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-03-18T15:15:48.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:47 smithi033 ceph-mon[36648]: daemon mds.f is now active in filesystem cephfs as rank 4 2024-03-18T15:15:48.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:47 smithi033 ceph-mon[36648]: mds.? [v2:172.21.15.104:6834/3788595957,v1:172.21.15.104:6835/3788595957] up:active 2024-03-18T15:15:48.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:47 smithi033 ceph-mon[36648]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T15:15:48.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:47 smithi033 ceph-mon[36648]: from='client.24829 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:15:48.008 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.007+0000 7ff806a00640 1 -- 172.21.15.19:0/82878934 <== mgr.14152 v2:172.21.15.19:6800/2017714236 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7ff810061c20 con 0x7ff7f40a2890 2024-03-18T15:15:48.008 INFO:teuthology.orchestra.run.smithi019.stdout:/volumes/qa/sv_1/6e5aa47e-98f1-4468-a9ee-8887701ea7d4 2024-03-18T15:15:48.010 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.010+0000 7ff814e00640 1 -- 172.21.15.19:0/82878934 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7ff7f40a2890 msgr2=0x7ff7f40a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:48.010 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.010+0000 7ff814e00640 1 --2- 172.21.15.19:0/82878934 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7ff7f40a2890 0x7ff7f40a4d50 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7ff80002c900 tx=0x7ff800000bc0 comp rx=0 tx=0).stop 2024-03-18T15:15:48.010 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.010+0000 7ff814e00640 1 -- 172.21.15.19:0/82878934 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff8101099b0 msgr2=0x7ff8101a56a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:48.010 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.010+0000 7ff814e00640 1 --2- 172.21.15.19:0/82878934 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff8101099b0 0x7ff8101a56a0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7ff808009fd0 tx=0x7ff80800ef10 comp rx=0 tx=0).stop 2024-03-18T15:15:48.011 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.010+0000 7ff814e00640 1 -- 172.21.15.19:0/82878934 shutdown_connections 2024-03-18T15:15:48.011 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.010+0000 7ff814e00640 1 --2- 172.21.15.19:0/82878934 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7ff7f40a2890 0x7ff7f40a4d50 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:48.011 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.010+0000 7ff814e00640 1 --2- 172.21.15.19:0/82878934 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff8101099b0 0x7ff8101a56a0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:48.011 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.010+0000 7ff814e00640 1 --2- 172.21.15.19:0/82878934 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff810108ff0 0x7ff81019e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:48.011 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.010+0000 7ff814e00640 1 --2- 172.21.15.19:0/82878934 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff810107e00 0x7ff81019e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:48.011 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.010+0000 7ff814e00640 1 -- 172.21.15.19:0/82878934 >> 172.21.15.19:0/82878934 conn(0x7ff810076da0 msgr2=0x7ff81010c2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:48.011 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.010+0000 7ff814e00640 1 -- 172.21.15.19:0/82878934 shutdown_connections 2024-03-18T15:15:48.011 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.010+0000 7ff814e00640 1 -- 172.21.15.19:0/82878934 wait complete. 2024-03-18T15:15:48.030 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T15:15:48.250 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.249+0000 7f11c0e00640 1 -- 172.21.15.19:0/695933855 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f11bc107c20 msgr2=0x7f11bc108040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:48.250 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.249+0000 7f11c0e00640 1 --2- 172.21.15.19:0/695933855 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f11bc107c20 0x7f11bc108040 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f11ac009900 tx=0x7f11ac02bc30 comp rx=0 tx=0).stop 2024-03-18T15:15:48.250 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.249+0000 7f11c0e00640 1 -- 172.21.15.19:0/695933855 shutdown_connections 2024-03-18T15:15:48.250 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.249+0000 7f11c0e00640 1 --2- 172.21.15.19:0/695933855 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f11bc1097d0 0x7f11bc110060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:48.250 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.249+0000 7f11c0e00640 1 --2- 172.21.15.19:0/695933855 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f11bc108e10 0x7f11bc109290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:48.250 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.249+0000 7f11c0e00640 1 --2- 172.21.15.19:0/695933855 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f11bc107c20 0x7f11bc108040 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:48.250 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.249+0000 7f11c0e00640 1 -- 172.21.15.19:0/695933855 >> 172.21.15.19:0/695933855 conn(0x7f11bc076bb0 msgr2=0x7f11bc078ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:48.250 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.250+0000 7f11c0e00640 1 -- 172.21.15.19:0/695933855 shutdown_connections 2024-03-18T15:15:48.251 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.250+0000 7f11c0e00640 1 -- 172.21.15.19:0/695933855 wait complete. 2024-03-18T15:15:48.253 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.250+0000 7f11c0e00640 1 Processor -- start 2024-03-18T15:15:48.253 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.251+0000 7f11c0e00640 1 -- start start 2024-03-18T15:15:48.253 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.251+0000 7f11c0e00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f11bc107c20 0x7f11bc19df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:48.253 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.251+0000 7f11c0e00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f11bc108e10 0x7f11bc19e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:48.253 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.251+0000 7f11c0e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f11bc1097d0 0x7f11bc1a54e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:48.253 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.251+0000 7f11c0e00640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f11bc1138c0 con 0x7f11bc108e10 2024-03-18T15:15:48.253 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.251+0000 7f11c0e00640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f11bc113740 con 0x7f11bc1097d0 2024-03-18T15:15:48.253 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.251+0000 7f11c0e00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f11bc113a40 con 0x7f11bc107c20 2024-03-18T15:15:48.253 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.251+0000 7f11bb400640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f11bc107c20 0x7f11bc19df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:48.253 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.251+0000 7f11bb400640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f11bc107c20 0x7f11bc19df30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.19:36506/0 (socket says 172.21.15.19:36506) 2024-03-18T15:15:48.253 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.251+0000 7f11bb400640 1 -- 172.21.15.19:0/1191952242 learned_addr learned my addr 172.21.15.19:0/1191952242 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:15:48.253 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.251+0000 7f11baa00640 1 --2- 172.21.15.19:0/1191952242 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f11bc108e10 0x7f11bc19e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:48.254 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.252+0000 7f11bbe00640 1 --2- 172.21.15.19:0/1191952242 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f11bc1097d0 0x7f11bc1a54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:48.254 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.252+0000 7f11bb400640 1 -- 172.21.15.19:0/1191952242 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f11bc1097d0 msgr2=0x7f11bc1a54e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:48.254 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.252+0000 7f11bb400640 1 --2- 172.21.15.19:0/1191952242 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f11bc1097d0 0x7f11bc1a54e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:48.254 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.252+0000 7f11bb400640 1 -- 172.21.15.19:0/1191952242 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f11bc108e10 msgr2=0x7f11bc19e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:48.254 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.252+0000 7f11bb400640 1 --2- 172.21.15.19:0/1191952242 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f11bc108e10 0x7f11bc19e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:48.254 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.252+0000 7f11bb400640 1 -- 172.21.15.19:0/1191952242 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11bc1a5b50 con 0x7f11bc107c20 2024-03-18T15:15:48.254 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.252+0000 7f11bbe00640 1 --2- 172.21.15.19:0/1191952242 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f11bc1097d0 0x7f11bc1a54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T15:15:48.254 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.252+0000 7f11bb400640 1 --2- 172.21.15.19:0/1191952242 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f11bc107c20 0x7f11bc19df30 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f11ac000c00 tx=0x7f11ac005dd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:48.254 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.252+0000 7f11b3e00640 1 -- 172.21.15.19:0/1191952242 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11ac043020 con 0x7f11bc107c20 2024-03-18T15:15:48.254 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.252+0000 7f11b3e00640 1 -- 172.21.15.19:0/1191952242 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f11ac0054e0 con 0x7f11bc107c20 2024-03-18T15:15:48.254 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.252+0000 7f11b3e00640 1 -- 172.21.15.19:0/1191952242 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11ac003e60 con 0x7f11bc107c20 2024-03-18T15:15:48.254 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.252+0000 7f11c0e00640 1 -- 172.21.15.19:0/1191952242 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11bc1a5de0 con 0x7f11bc107c20 2024-03-18T15:15:48.254 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.253+0000 7f11c0e00640 1 -- 172.21.15.19:0/1191952242 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11bc1a61f0 con 0x7f11bc107c20 2024-03-18T15:15:48.255 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.254+0000 7f11c0e00640 1 -- 172.21.15.19:0/1191952242 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11bc10ef90 con 0x7f11bc107c20 2024-03-18T15:15:48.255 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.255+0000 7f11b3e00640 1 -- 172.21.15.19:0/1191952242 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f11ac046020 con 0x7f11bc107c20 2024-03-18T15:15:48.257 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.257+0000 7f11b3e00640 1 --2- 172.21.15.19:0/1191952242 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f11a00a2890 0x7f11a00a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:48.257 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.257+0000 7f11b3e00640 1 -- 172.21.15.19:0/1191952242 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f11ac0f0f80 con 0x7f11bc107c20 2024-03-18T15:15:48.257 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.257+0000 7f11baa00640 1 --2- 172.21.15.19:0/1191952242 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f11a00a2890 0x7f11a00a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:48.257 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.258+0000 7f11baa00640 1 --2- 172.21.15.19:0/1191952242 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f11a00a2890 0x7f11a00a4d50 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f11bc19f440 tx=0x7f11a800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:48.265 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.265+0000 7f11b3e00640 1 -- 172.21.15.19:0/1191952242 <== mon.1 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f11ac04d020 con 0x7f11bc107c20 2024-03-18T15:15:48.511 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.511+0000 7f11c0e00640 1 -- 172.21.15.19:0/1191952242 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f11bc02d420 con 0x7f11bc107c20 2024-03-18T15:15:48.516 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.516+0000 7f11b3e00640 1 -- 172.21.15.19:0/1191952242 <== mon.1 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11927 (secure 0 0 0) 0x7f11ac0bdd30 con 0x7f11bc107c20 2024-03-18T15:15:48.516 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-18T15:15:48.517 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6835/1135329545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6834","nonce":1135329545},{"type":"v1","addr":"172.21.15.19:6835","nonce":1135329545}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14457,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6839/3833518557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6838","nonce":3833518557},{"type":"v1","addr":"172.21.15.19:6839","nonce":3833518557}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T15:15:10.589737+0000","modified":"2024-03-18T15:15:46.979999+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24560,"mds_1":24529,"mds_2":24511,"mds_3":24506,"mds_4":24499},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14475":{"gid":14475,"name":"a","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.19:6841/3176583319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6840","nonce":3176583319},{"type":"v1","addr":"172.21.15.19:6841","nonce":3176583319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24433":{"gid":24433,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6833/482725809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":482725809},{"type":"v1","addr":"172.21.15.104:6833","nonce":482725809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24434":{"gid":24434,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/2272279159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":2272279159},{"type":"v1","addr":"172.21.15.33:6833","nonce":2272279159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24469":{"gid":24469,"name":"d","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.19:6837/1778019859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6836","nonce":1778019859},{"type":"v1","addr":"172.21.15.19:6837","nonce":1778019859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6835/4051613151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":4051613151},{"type":"v1","addr":"172.21.15.33:6835","nonce":4051613151}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":17,"addr":"172.21.15.104:6835/3788595957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3788595957},{"type":"v1","addr":"172.21.15.104:6835","nonce":3788595957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":15,"addr":"172.21.15.33:6837/160187378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":160187378},{"type":"v1","addr":"172.21.15.33:6837","nonce":160187378}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24511":{"gid":24511,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":16,"addr":"172.21.15.104:6837/260984674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":260984674},{"type":"v1","addr":"172.21.15.104:6837","nonce":260984674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24529":{"gid":24529,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.104:6839/805341436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6838","nonce":805341436},{"type":"v1","addr":"172.21.15.104:6839","nonce":805341436}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.33:6839/1903591535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":1903591535},{"type":"v1","addr":"172.21.15.33:6839","nonce":1903591535}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24560,"qdb_cluster":[24506,24499,24529,24511,24560]},"id":1}]} 2024-03-18T15:15:48.517 INFO:teuthology.orchestra.run.smithi019.stderr:dumped fsmap epoch 26 2024-03-18T15:15:48.519 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.519+0000 7f11c0e00640 1 -- 172.21.15.19:0/1191952242 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f11a00a2890 msgr2=0x7f11a00a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:48.519 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.519+0000 7f11c0e00640 1 --2- 172.21.15.19:0/1191952242 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f11a00a2890 0x7f11a00a4d50 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f11bc19f440 tx=0x7f11a800b410 comp rx=0 tx=0).stop 2024-03-18T15:15:48.520 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.519+0000 7f11c0e00640 1 -- 172.21.15.19:0/1191952242 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f11bc107c20 msgr2=0x7f11bc19df30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:48.520 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.519+0000 7f11c0e00640 1 --2- 172.21.15.19:0/1191952242 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f11bc107c20 0x7f11bc19df30 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f11ac000c00 tx=0x7f11ac005dd0 comp rx=0 tx=0).stop 2024-03-18T15:15:48.520 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.519+0000 7f11c0e00640 1 -- 172.21.15.19:0/1191952242 shutdown_connections 2024-03-18T15:15:48.520 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.519+0000 7f11c0e00640 1 --2- 172.21.15.19:0/1191952242 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f11a00a2890 0x7f11a00a4d50 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:48.520 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.519+0000 7f11c0e00640 1 --2- 172.21.15.19:0/1191952242 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f11bc1097d0 0x7f11bc1a54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:48.520 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.519+0000 7f11c0e00640 1 --2- 172.21.15.19:0/1191952242 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f11bc108e10 0x7f11bc19e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:48.520 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.519+0000 7f11c0e00640 1 --2- 172.21.15.19:0/1191952242 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f11bc107c20 0x7f11bc19df30 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:48.520 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.519+0000 7f11c0e00640 1 -- 172.21.15.19:0/1191952242 >> 172.21.15.19:0/1191952242 conn(0x7f11bc076bb0 msgr2=0x7f11bc10c160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:48.520 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.519+0000 7f11c0e00640 1 -- 172.21.15.19:0/1191952242 shutdown_connections 2024-03-18T15:15:48.520 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.520+0000 7f11c0e00640 1 -- 172.21.15.19:0/1191952242 wait complete. 2024-03-18T15:15:48.538 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T15:15:48.753 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.752+0000 7f08c3200640 1 -- 172.21.15.19:0/4271425765 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f08c4102820 msgr2=0x7f08c4102ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:48.753 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.752+0000 7f08c3200640 1 --2- 172.21.15.19:0/4271425765 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f08c4102820 0x7f08c4102ca0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f08ac009900 tx=0x7f08ac02bc40 comp rx=0 tx=0).stop 2024-03-18T15:15:48.753 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.752+0000 7f08c3200640 1 -- 172.21.15.19:0/4271425765 shutdown_connections 2024-03-18T15:15:48.753 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.752+0000 7f08c3200640 1 --2- 172.21.15.19:0/4271425765 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f08c41031e0 0x7f08c4109a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:48.753 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.752+0000 7f08c3200640 1 --2- 172.21.15.19:0/4271425765 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f08c4102820 0x7f08c4102ca0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:48.753 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.752+0000 7f08c3200640 1 --2- 172.21.15.19:0/4271425765 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f08c4101630 0x7f08c4101a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:48.753 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.752+0000 7f08c3200640 1 -- 172.21.15.19:0/4271425765 >> 172.21.15.19:0/4271425765 conn(0x7f08c40fcbc0 msgr2=0x7f08c40fefe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:48.753 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.752+0000 7f08c3200640 1 -- 172.21.15.19:0/4271425765 shutdown_connections 2024-03-18T15:15:48.753 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.752+0000 7f08c3200640 1 -- 172.21.15.19:0/4271425765 wait complete. 2024-03-18T15:15:48.756 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.754+0000 7f08c3200640 1 Processor -- start 2024-03-18T15:15:48.756 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.754+0000 7f08c3200640 1 -- start start 2024-03-18T15:15:48.756 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.754+0000 7f08c3200640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f08c4101630 0x7f08c4075590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:48.756 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.754+0000 7f08c3200640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f08c4102820 0x7f08c4073bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:48.756 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.754+0000 7f08c3200640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f08c41031e0 0x7f08c4074110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:48.756 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.754+0000 7f08c3200640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f08c410d2e0 con 0x7f08c41031e0 2024-03-18T15:15:48.756 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.754+0000 7f08c3200640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f08c410d160 con 0x7f08c4102820 2024-03-18T15:15:48.756 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.754+0000 7f08c3200640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f08c410d460 con 0x7f08c4101630 2024-03-18T15:15:48.756 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.754+0000 7f08c1e00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f08c4101630 0x7f08c4075590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:48.756 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.754+0000 7f08c1400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f08c4102820 0x7f08c4073bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:48.756 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.755+0000 7f08c1e00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f08c4101630 0x7f08c4075590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.19:36512/0 (socket says 172.21.15.19:36512) 2024-03-18T15:15:48.756 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.755+0000 7f08c1400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f08c4102820 0x7f08c4073bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:35750/0 (socket says 172.21.15.19:35750) 2024-03-18T15:15:48.756 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.755+0000 7f08c1400640 1 -- 172.21.15.19:0/2278289990 learned_addr learned my addr 172.21.15.19:0/2278289990 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:15:48.757 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.755+0000 7f08c2800640 1 --2- 172.21.15.19:0/2278289990 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f08c41031e0 0x7f08c4074110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:48.757 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.755+0000 7f08c1400640 1 -- 172.21.15.19:0/2278289990 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f08c4101630 msgr2=0x7f08c4075590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:48.757 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.755+0000 7f08c1400640 1 --2- 172.21.15.19:0/2278289990 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f08c4101630 0x7f08c4075590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:48.757 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.755+0000 7f08c1400640 1 -- 172.21.15.19:0/2278289990 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f08c41031e0 msgr2=0x7f08c4074110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:48.757 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.755+0000 7f08c1400640 1 --2- 172.21.15.19:0/2278289990 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f08c41031e0 0x7f08c4074110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:48.757 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.755+0000 7f08c1400640 1 -- 172.21.15.19:0/2278289990 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08c40749a0 con 0x7f08c4102820 2024-03-18T15:15:48.757 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.755+0000 7f08c2800640 1 --2- 172.21.15.19:0/2278289990 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f08c41031e0 0x7f08c4074110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T15:15:48.757 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.755+0000 7f08c1400640 1 --2- 172.21.15.19:0/2278289990 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f08c4102820 0x7f08c4073bd0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f08ac02c8f0 tx=0x7f08ac005d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:48.757 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.755+0000 7f08baa00640 1 -- 172.21.15.19:0/2278289990 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08ac041070 con 0x7f08c4102820 2024-03-18T15:15:48.757 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.756+0000 7f08c3200640 1 -- 172.21.15.19:0/2278289990 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08c41a1930 con 0x7f08c4102820 2024-03-18T15:15:48.757 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.756+0000 7f08baa00640 1 -- 172.21.15.19:0/2278289990 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f08ac0044d0 con 0x7f08c4102820 2024-03-18T15:15:48.757 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.756+0000 7f08baa00640 1 -- 172.21.15.19:0/2278289990 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08ac034440 con 0x7f08c4102820 2024-03-18T15:15:48.757 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.756+0000 7f08c3200640 1 -- 172.21.15.19:0/2278289990 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08c41a1e80 con 0x7f08c4102820 2024-03-18T15:15:48.758 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.757+0000 7f08c3200640 1 -- 172.21.15.19:0/2278289990 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08c41088f0 con 0x7f08c4102820 2024-03-18T15:15:48.758 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.759+0000 7f08baa00640 1 -- 172.21.15.19:0/2278289990 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f08ac044070 con 0x7f08c4102820 2024-03-18T15:15:48.760 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.760+0000 7f08baa00640 1 --2- 172.21.15.19:0/2278289990 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f08a80a2890 0x7f08a80a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:48.760 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.760+0000 7f08baa00640 1 -- 172.21.15.19:0/2278289990 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f08ac0f0990 con 0x7f08c4102820 2024-03-18T15:15:48.760 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.760+0000 7f08c1e00640 1 --2- 172.21.15.19:0/2278289990 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f08a80a2890 0x7f08a80a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:48.761 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.761+0000 7f08c1e00640 1 --2- 172.21.15.19:0/2278289990 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f08a80a2890 0x7f08a80a4d50 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f08c4102680 tx=0x7f08b4009670 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:48.764 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.764+0000 7f08baa00640 1 -- 172.21.15.19:0/2278289990 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f08ac0bd740 con 0x7f08c4102820 2024-03-18T15:15:48.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:48 smithi104 ceph-mon[36070]: from='client.24835 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:15:48.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:48 smithi104 ceph-mon[36070]: from='client.24830 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:15:48.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:48 smithi019 ceph-mon[31522]: from='client.24835 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:15:48.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:48 smithi019 ceph-mon[31522]: from='client.24830 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:15:48.970 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.970+0000 7f08c3200640 1 -- 172.21.15.19:0/2278289990 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f08c402d400 con 0x7f08c4102820 2024-03-18T15:15:48.972 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.972+0000 7f08baa00640 1 -- 172.21.15.19:0/2278289990 <== mon.2 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24093 (secure 0 0 0) 0x7f08ac0bd040 con 0x7f08c4102820 2024-03-18T15:15:48.973 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-18T15:15:48.973 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":89,"fsid":"d4dc2ba6-e538-11ee-95c9-87774f69a715","created":"2024-03-18T15:05:57.538551+0000","modified":"2024-03-18T15:15:15.582418+0000","last_up_change":"2024-03-18T15:14:12.009522+0000","last_in_change":"2024-03-18T15:13:52.555387+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T15:09:56.843349+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T15:15:08.941047+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T15:15:10.044671+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T15:15:13.621916+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"37b5a5cd-27ae-4e77-9b05-774b817b2ee9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6802","nonce":402333948},{"type":"v1","addr":"172.21.15.19:6803","nonce":402333948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6804","nonce":402333948},{"type":"v1","addr":"172.21.15.19:6805","nonce":402333948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6808","nonce":402333948},{"type":"v1","addr":"172.21.15.19:6809","nonce":402333948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6806","nonce":402333948},{"type":"v1","addr":"172.21.15.19:6807","nonce":402333948}]},"public_addr":"172.21.15.19:6803/402333948","cluster_addr":"172.21.15.19:6805/402333948","heartbeat_back_addr":"172.21.15.19:6809/402333948","heartbeat_front_addr":"172.21.15.19:6807/402333948","state":["exists","up"]},{"osd":1,"uuid":"3a1134e8-ede7-4dbd-9be1-eefd2cbb43bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1951880113},{"type":"v1","addr":"172.21.15.33:6801","nonce":1951880113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1951880113},{"type":"v1","addr":"172.21.15.33:6803","nonce":1951880113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1951880113},{"type":"v1","addr":"172.21.15.33:6807","nonce":1951880113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1951880113},{"type":"v1","addr":"172.21.15.33:6805","nonce":1951880113}]},"public_addr":"172.21.15.33:6801/1951880113","cluster_addr":"172.21.15.33:6803/1951880113","heartbeat_back_addr":"172.21.15.33:6807/1951880113","heartbeat_front_addr":"172.21.15.33:6805/1951880113","state":["exists","up"]},{"osd":2,"uuid":"f383a164-a3a0-49fd-a9e8-9c2178d8cfb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":1004727949},{"type":"v1","addr":"172.21.15.104:6801","nonce":1004727949}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":1004727949},{"type":"v1","addr":"172.21.15.104:6803","nonce":1004727949}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":1004727949},{"type":"v1","addr":"172.21.15.104:6807","nonce":1004727949}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":1004727949},{"type":"v1","addr":"172.21.15.104:6805","nonce":1004727949}]},"public_addr":"172.21.15.104:6801/1004727949","cluster_addr":"172.21.15.104:6803/1004727949","heartbeat_back_addr":"172.21.15.104:6807/1004727949","heartbeat_front_addr":"172.21.15.104:6805/1004727949","state":["exists","up"]},{"osd":3,"uuid":"c409e116-28bf-45dd-a791-539af22abe1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6810","nonce":3571525854},{"type":"v1","addr":"172.21.15.19:6811","nonce":3571525854}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6812","nonce":3571525854},{"type":"v1","addr":"172.21.15.19:6813","nonce":3571525854}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6816","nonce":3571525854},{"type":"v1","addr":"172.21.15.19:6817","nonce":3571525854}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6814","nonce":3571525854},{"type":"v1","addr":"172.21.15.19:6815","nonce":3571525854}]},"public_addr":"172.21.15.19:6811/3571525854","cluster_addr":"172.21.15.19:6813/3571525854","heartbeat_back_addr":"172.21.15.19:6817/3571525854","heartbeat_front_addr":"172.21.15.19:6815/3571525854","state":["exists","up"]},{"osd":4,"uuid":"696eb719-0b3e-43ae-b5f1-ccd73a3b2843","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":624901749},{"type":"v1","addr":"172.21.15.33:6809","nonce":624901749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":624901749},{"type":"v1","addr":"172.21.15.33:6811","nonce":624901749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":624901749},{"type":"v1","addr":"172.21.15.33:6815","nonce":624901749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":624901749},{"type":"v1","addr":"172.21.15.33:6813","nonce":624901749}]},"public_addr":"172.21.15.33:6809/624901749","cluster_addr":"172.21.15.33:6811/624901749","heartbeat_back_addr":"172.21.15.33:6815/624901749","heartbeat_front_addr":"172.21.15.33:6813/624901749","state":["exists","up"]},{"osd":5,"uuid":"21789779-8bb9-42c7-b9ee-437985f11e3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":1258208297},{"type":"v1","addr":"172.21.15.104:6809","nonce":1258208297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":1258208297},{"type":"v1","addr":"172.21.15.104:6811","nonce":1258208297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":1258208297},{"type":"v1","addr":"172.21.15.104:6815","nonce":1258208297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":1258208297},{"type":"v1","addr":"172.21.15.104:6813","nonce":1258208297}]},"public_addr":"172.21.15.104:6809/1258208297","cluster_addr":"172.21.15.104:6811/1258208297","heartbeat_back_addr":"172.21.15.104:6815/1258208297","heartbeat_front_addr":"172.21.15.104:6813/1258208297","state":["exists","up"]},{"osd":6,"uuid":"0e92a2ef-43cd-487c-9203-e219158aef8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6818","nonce":2782482301},{"type":"v1","addr":"172.21.15.19:6819","nonce":2782482301}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6820","nonce":2782482301},{"type":"v1","addr":"172.21.15.19:6821","nonce":2782482301}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6824","nonce":2782482301},{"type":"v1","addr":"172.21.15.19:6825","nonce":2782482301}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6822","nonce":2782482301},{"type":"v1","addr":"172.21.15.19:6823","nonce":2782482301}]},"public_addr":"172.21.15.19:6819/2782482301","cluster_addr":"172.21.15.19:6821/2782482301","heartbeat_back_addr":"172.21.15.19:6825/2782482301","heartbeat_front_addr":"172.21.15.19:6823/2782482301","state":["exists","up"]},{"osd":7,"uuid":"a16e39a1-3685-491d-b376-3e80a2084072","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":4238160988},{"type":"v1","addr":"172.21.15.33:6817","nonce":4238160988}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":4238160988},{"type":"v1","addr":"172.21.15.33:6819","nonce":4238160988}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":4238160988},{"type":"v1","addr":"172.21.15.33:6823","nonce":4238160988}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":4238160988},{"type":"v1","addr":"172.21.15.33:6821","nonce":4238160988}]},"public_addr":"172.21.15.33:6817/4238160988","cluster_addr":"172.21.15.33:6819/4238160988","heartbeat_back_addr":"172.21.15.33:6823/4238160988","heartbeat_front_addr":"172.21.15.33:6821/4238160988","state":["exists","up"]},{"osd":8,"uuid":"d5c3fa92-cd5d-4349-9c7d-d5666bacde16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":1014779813},{"type":"v1","addr":"172.21.15.104:6817","nonce":1014779813}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":1014779813},{"type":"v1","addr":"172.21.15.104:6819","nonce":1014779813}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":1014779813},{"type":"v1","addr":"172.21.15.104:6823","nonce":1014779813}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":1014779813},{"type":"v1","addr":"172.21.15.104:6821","nonce":1014779813}]},"public_addr":"172.21.15.104:6817/1014779813","cluster_addr":"172.21.15.104:6819/1014779813","heartbeat_back_addr":"172.21.15.104:6823/1014779813","heartbeat_front_addr":"172.21.15.104:6821/1014779813","state":["exists","up"]},{"osd":9,"uuid":"eed2e9b2-26d3-4c46-88ec-34b5596fe991","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6826","nonce":2641151009},{"type":"v1","addr":"172.21.15.19:6827","nonce":2641151009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6828","nonce":2641151009},{"type":"v1","addr":"172.21.15.19:6829","nonce":2641151009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6832","nonce":2641151009},{"type":"v1","addr":"172.21.15.19:6833","nonce":2641151009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6830","nonce":2641151009},{"type":"v1","addr":"172.21.15.19:6831","nonce":2641151009}]},"public_addr":"172.21.15.19:6827/2641151009","cluster_addr":"172.21.15.19:6829/2641151009","heartbeat_back_addr":"172.21.15.19:6833/2641151009","heartbeat_front_addr":"172.21.15.19:6831/2641151009","state":["exists","up"]},{"osd":10,"uuid":"f3c35ea5-fbe7-4dc4-93c5-b92d1a714db3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":1346058869},{"type":"v1","addr":"172.21.15.33:6825","nonce":1346058869}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":1346058869},{"type":"v1","addr":"172.21.15.33:6827","nonce":1346058869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":1346058869},{"type":"v1","addr":"172.21.15.33:6831","nonce":1346058869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":1346058869},{"type":"v1","addr":"172.21.15.33:6829","nonce":1346058869}]},"public_addr":"172.21.15.33:6825/1346058869","cluster_addr":"172.21.15.33:6827/1346058869","heartbeat_back_addr":"172.21.15.33:6831/1346058869","heartbeat_front_addr":"172.21.15.33:6829/1346058869","state":["exists","up"]},{"osd":11,"uuid":"c8b51571-f807-4b62-90b6-7529e09e59da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":3759498281},{"type":"v1","addr":"172.21.15.104:6825","nonce":3759498281}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":3759498281},{"type":"v1","addr":"172.21.15.104:6827","nonce":3759498281}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":3759498281},{"type":"v1","addr":"172.21.15.104:6831","nonce":3759498281}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":3759498281},{"type":"v1","addr":"172.21.15.104:6829","nonce":3759498281}]},"public_addr":"172.21.15.104:6825/3759498281","cluster_addr":"172.21.15.104:6827/3759498281","heartbeat_back_addr":"172.21.15.104:6831/3759498281","heartbeat_front_addr":"172.21.15.104:6829/3759498281","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:09:05.267795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:09:30.234068+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:09:53.870232+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:10:26.471296+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:10:53.180341+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:11:17.801361+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:11:51.364816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:12:18.306408+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:12:43.978744+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:13:18.213964+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:13:45.437544+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:14:10.148580+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.19:0/3860783274":"2024-03-19T15:07:14.150983+0000","172.21.15.19:0/2737223213":"2024-03-19T15:07:14.150983+0000","172.21.15.19:0/176699025":"2024-03-19T15:06:38.688405+0000","172.21.15.19:0/2316969005":"2024-03-19T15:07:14.150983+0000","172.21.15.19:6801/2776008962":"2024-03-19T15:07:14.150983+0000","172.21.15.19:0/2793980440":"2024-03-19T15:06:38.688405+0000","172.21.15.19:6800/3264337803":"2024-03-19T15:06:38.688405+0000","172.21.15.19:6801/3264337803":"2024-03-19T15:06:38.688405+0000","172.21.15.19:0/3166705295":"2024-03-19T15:06:38.688405+0000","172.21.15.19:6800/2776008962":"2024-03-19T15:07:14.150983+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T15:15:48.976 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.975+0000 7f08c3200640 1 -- 172.21.15.19:0/2278289990 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f08a80a2890 msgr2=0x7f08a80a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:48.976 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.975+0000 7f08c3200640 1 --2- 172.21.15.19:0/2278289990 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f08a80a2890 0x7f08a80a4d50 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f08c4102680 tx=0x7f08b4009670 comp rx=0 tx=0).stop 2024-03-18T15:15:48.976 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.975+0000 7f08c3200640 1 -- 172.21.15.19:0/2278289990 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f08c4102820 msgr2=0x7f08c4073bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:48.976 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.975+0000 7f08c3200640 1 --2- 172.21.15.19:0/2278289990 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f08c4102820 0x7f08c4073bd0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f08ac02c8f0 tx=0x7f08ac005d50 comp rx=0 tx=0).stop 2024-03-18T15:15:48.976 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.976+0000 7f08c3200640 1 -- 172.21.15.19:0/2278289990 shutdown_connections 2024-03-18T15:15:48.976 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.976+0000 7f08c3200640 1 --2- 172.21.15.19:0/2278289990 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f08c41031e0 0x7f08c4074110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:48.976 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.976+0000 7f08c3200640 1 --2- 172.21.15.19:0/2278289990 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f08a80a2890 0x7f08a80a4d50 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:48.976 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.976+0000 7f08c3200640 1 --2- 172.21.15.19:0/2278289990 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f08c4102820 0x7f08c4073bd0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:48.976 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.976+0000 7f08c3200640 1 --2- 172.21.15.19:0/2278289990 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f08c4101630 0x7f08c4075590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:48.977 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.976+0000 7f08c3200640 1 -- 172.21.15.19:0/2278289990 >> 172.21.15.19:0/2278289990 conn(0x7f08c40fcbc0 msgr2=0x7f08c4105ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:48.977 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.976+0000 7f08c3200640 1 -- 172.21.15.19:0/2278289990 shutdown_connections 2024-03-18T15:15:48.977 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:48.976+0000 7f08c3200640 1 -- 172.21.15.19:0/2278289990 wait complete. 2024-03-18T15:15:48.995 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-18T15:15:48.995 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T15:15:49.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:48 smithi033 ceph-mon[36648]: from='client.24835 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:15:49.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:48 smithi033 ceph-mon[36648]: from='client.24830 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:15:49.220 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:49.220+0000 7fe4f3200640 1 -- 172.21.15.19:0/931106030 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe4ec0757c0 msgr2=0x7fe4ec073c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:49.220 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:49.220+0000 7fe4f3200640 1 --2- 172.21.15.19:0/931106030 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe4ec0757c0 0x7fe4ec073c10 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fe4dc009ca0 tx=0x7fe4dc02c190 comp rx=0 tx=0).stop 2024-03-18T15:15:49.221 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:49.220+0000 7fe4f3200640 1 -- 172.21.15.19:0/931106030 shutdown_connections 2024-03-18T15:15:49.221 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:49.220+0000 7fe4f3200640 1 --2- 172.21.15.19:0/931106030 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe4ec109680 0x7fe4ec10bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:49.221 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:49.220+0000 7fe4f3200640 1 --2- 172.21.15.19:0/931106030 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe4ec0741e0 0x7fe4ec074640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:49.221 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:49.220+0000 7fe4f3200640 1 --2- 172.21.15.19:0/931106030 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe4ec0757c0 0x7fe4ec073c10 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:49.221 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:49.220+0000 7fe4f3200640 1 -- 172.21.15.19:0/931106030 >> 172.21.15.19:0/931106030 conn(0x7fe4ec0fcd80 msgr2=0x7fe4ec0ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:49.221 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:49.220+0000 7fe4f3200640 1 -- 172.21.15.19:0/931106030 shutdown_connections 2024-03-18T15:15:49.221 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:49.220+0000 7fe4f3200640 1 -- 172.21.15.19:0/931106030 wait complete. 2024-03-18T15:15:49.223 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:49.221+0000 7fe4f3200640 1 Processor -- start 2024-03-18T15:15:49.223 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:49.222+0000 7fe4f3200640 1 -- start start 2024-03-18T15:15:49.223 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:49.222+0000 7fe4f3200640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe4ec0741e0 0x7fe4ec199cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:49.223 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:49.222+0000 7fe4f3200640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe4ec0757c0 0x7fe4ec19a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:49.224 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:49.222+0000 7fe4f3200640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe4ec109680 0x7fe4ec1a1260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:49.224 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:49.222+0000 7fe4f3200640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fe4ec10f660 con 0x7fe4ec109680 2024-03-18T15:15:49.224 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:49.222+0000 7fe4f3200640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe4ec10f4e0 con 0x7fe4ec0741e0 2024-03-18T15:15:49.224 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:49.222+0000 7fe4f3200640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fe4ec10f7e0 con 0x7fe4ec0757c0 2024-03-18T15:15:49.224 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:49.222+0000 7fe4ebe00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe4ec0741e0 0x7fe4ec199cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:49.224 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:49.222+0000 7fe4eb400640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe4ec0757c0 0x7fe4ec19a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:49.224 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:49.222+0000 7fe4ebe00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe4ec0741e0 0x7fe4ec199cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:35768/0 (socket says 172.21.15.19:35768) 2024-03-18T15:15:49.224 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:49.222+0000 7fe4ebe00640 1 -- 172.21.15.19:0/2758824204 learned_addr learned my addr 172.21.15.19:0/2758824204 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:15:49.224 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:49.222+0000 7fe4eb400640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe4ec0757c0 0x7fe4ec19a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.19:36528/0 (socket says 172.21.15.19:36528) 2024-03-18T15:15:49.224 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:49.222+0000 7fe4f1200640 1 --2- 172.21.15.19:0/2758824204 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe4ec109680 0x7fe4ec1a1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:49.224 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:49.223+0000 7fe4ebe00640 1 -- 172.21.15.19:0/2758824204 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe4ec0757c0 msgr2=0x7fe4ec19a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:49.224 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:49.223+0000 7fe4ebe00640 1 --2- 172.21.15.19:0/2758824204 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe4ec0757c0 0x7fe4ec19a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:49.224 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:49.223+0000 7fe4ebe00640 1 -- 172.21.15.19:0/2758824204 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe4ec109680 msgr2=0x7fe4ec1a1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:49.224 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:49.223+0000 7fe4ebe00640 1 --2- 172.21.15.19:0/2758824204 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe4ec109680 0x7fe4ec1a1260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:49.225 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:49.223+0000 7fe4ebe00640 1 -- 172.21.15.19:0/2758824204 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4ec1a1900 con 0x7fe4ec0741e0 2024-03-18T15:15:49.225 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:49.223+0000 7fe4f1200640 1 --2- 172.21.15.19:0/2758824204 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe4ec109680 0x7fe4ec1a1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T15:15:49.225 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:49.223+0000 7fe4eb400640 1 --2- 172.21.15.19:0/2758824204 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe4ec0757c0 0x7fe4ec19a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:15:49.225 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:49.223+0000 7fe4ebe00640 1 --2- 172.21.15.19:0/2758824204 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe4ec0741e0 0x7fe4ec199cc0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe4dc0326d0 tx=0x7fe4dc032360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:49.225 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:49.223+0000 7fe4e8c00640 1 -- 172.21.15.19:0/2758824204 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4dc039040 con 0x7fe4ec0741e0 2024-03-18T15:15:49.225 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:49.223+0000 7fe4e8c00640 1 -- 172.21.15.19:0/2758824204 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe4dc044070 con 0x7fe4ec0741e0 2024-03-18T15:15:49.225 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:49.223+0000 7fe4e8c00640 1 -- 172.21.15.19:0/2758824204 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4dc004500 con 0x7fe4ec0741e0 2024-03-18T15:15:49.225 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:49.223+0000 7fe4f3200640 1 -- 172.21.15.19:0/2758824204 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4ec1a1b90 con 0x7fe4ec0741e0 2024-03-18T15:15:49.225 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:49.223+0000 7fe4f3200640 1 -- 172.21.15.19:0/2758824204 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4ec1a2000 con 0x7fe4ec0741e0 2024-03-18T15:15:49.225 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:49.225+0000 7fe4f3200640 1 -- 172.21.15.19:0/2758824204 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4ec108440 con 0x7fe4ec0741e0 2024-03-18T15:15:49.226 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:49.226+0000 7fe4e8c00640 1 -- 172.21.15.19:0/2758824204 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fe4dc003bf0 con 0x7fe4ec0741e0 2024-03-18T15:15:49.228 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:49.228+0000 7fe4e8c00640 1 --2- 172.21.15.19:0/2758824204 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fe4d00a2910 0x7fe4d00a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:49.228 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:49.228+0000 7fe4e8c00640 1 -- 172.21.15.19:0/2758824204 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fe4dc0f10f0 con 0x7fe4ec0741e0 2024-03-18T15:15:49.228 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:49.228+0000 7fe4eb400640 1 --2- 172.21.15.19:0/2758824204 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fe4d00a2910 0x7fe4d00a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:49.229 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:49.229+0000 7fe4eb400640 1 --2- 172.21.15.19:0/2758824204 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fe4d00a2910 0x7fe4d00a4dd0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fe4ec19b1c0 tx=0x7fe4d4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:49.231 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:49.231+0000 7fe4e8c00640 1 -- 172.21.15.19:0/2758824204 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe4dc04e020 con 0x7fe4ec0741e0 2024-03-18T15:15:49.437 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:49.437+0000 7fe4f3200640 1 -- 172.21.15.19:0/2758824204 --> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe4ec061c20 con 0x7fe4d00a2910 2024-03-18T15:15:49.473 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:49.473+0000 7fe4e8c00640 1 -- 172.21.15.19:0/2758824204 <== mgr.14152 v2:172.21.15.19:6800/2017714236 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340596 (secure 0 0 0) 0x7fe4ec061c20 con 0x7fe4d00a2910 2024-03-18T15:15:49.479 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-18T15:15:49.481 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-03-18T15:15:49.483 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:49.481+0000 7fe4f3200640 1 -- 172.21.15.19:0/2758824204 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fe4d00a2910 msgr2=0x7fe4d00a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:49.483 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:49.481+0000 7fe4f3200640 1 --2- 172.21.15.19:0/2758824204 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fe4d00a2910 0x7fe4d00a4dd0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fe4ec19b1c0 tx=0x7fe4d4005f90 comp rx=0 tx=0).stop 2024-03-18T15:15:49.483 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:49.481+0000 7fe4f3200640 1 -- 172.21.15.19:0/2758824204 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe4ec0741e0 msgr2=0x7fe4ec199cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:49.483 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:49.481+0000 7fe4f3200640 1 --2- 172.21.15.19:0/2758824204 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe4ec0741e0 0x7fe4ec199cc0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe4dc0326d0 tx=0x7fe4dc032360 comp rx=0 tx=0).stop 2024-03-18T15:15:49.483 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:49.481+0000 7fe4f3200640 1 -- 172.21.15.19:0/2758824204 shutdown_connections 2024-03-18T15:15:49.483 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:49.481+0000 7fe4f3200640 1 --2- 172.21.15.19:0/2758824204 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe4ec109680 0x7fe4ec1a1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:49.483 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:49.481+0000 7fe4f3200640 1 --2- 172.21.15.19:0/2758824204 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe4ec0757c0 0x7fe4ec19a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:49.483 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:49.481+0000 7fe4f3200640 1 --2- 172.21.15.19:0/2758824204 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fe4d00a2910 0x7fe4d00a4dd0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:49.484 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:49.481+0000 7fe4f3200640 1 --2- 172.21.15.19:0/2758824204 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe4ec0741e0 0x7fe4ec199cc0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:49.484 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:49.481+0000 7fe4f3200640 1 -- 172.21.15.19:0/2758824204 >> 172.21.15.19:0/2758824204 conn(0x7fe4ec0fcd80 msgr2=0x7fe4ec105610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:49.484 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:49.481+0000 7fe4f3200640 1 -- 172.21.15.19:0/2758824204 shutdown_connections 2024-03-18T15:15:49.484 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:49.481+0000 7fe4f3200640 1 -- 172.21.15.19:0/2758824204 wait complete. 2024-03-18T15:15:49.506 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":336,"stamp":"2024-03-18T15:15:48.309522+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":617763,"num_objects":72,"num_object_clones":0,"num_object_copies":216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72,"num_whiteouts":0,"num_read":142,"num_read_kb":145,"num_write":342,"num_write_kb":3007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":47,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":278,"ondisk_log_size":278,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":356740,"kb_used_data":11744,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124585084,"statfs":{"total":1151940427776,"available":1151575126016,"internally_reserved":0,"allocated":12025856,"data_stored":7559205,"data_compressed":283005,"data_compressed_allocated":380928,"data_compressed_original":2260992,"omap_allocated":19143,"internal_metadata":353154361},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22896,"num_objects":48,"num_object_clones":0,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":6,"num_read_kb":26,"num_write":113,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":33,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006416"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.024181+0000","last_change":"2024-03-18T15:15:10.682660+0000","last_active":"2024-03-18T15:15:18.024181+0000","last_peered":"2024-03-18T15:15:18.024181+0000","last_clean":"2024-03-18T15:15:18.024181+0000","last_became_active":"2024-03-18T15:15:10.682433+0000","last_became_peered":"2024-03-18T15:15:10.682433+0000","last_unstale":"2024-03-18T15:15:18.024181+0000","last_undegraded":"2024-03-18T15:15:18.024181+0000","last_fullsized":"2024-03-18T15:15:18.024181+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:48:55.154555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:17.578802+0000","last_change":"2024-03-18T15:15:10.678100+0000","last_active":"2024-03-18T15:15:17.578802+0000","last_peered":"2024-03-18T15:15:17.578802+0000","last_clean":"2024-03-18T15:15:17.578802+0000","last_became_active":"2024-03-18T15:15:10.677543+0000","last_became_peered":"2024-03-18T15:15:10.677543+0000","last_unstale":"2024-03-18T15:15:17.578802+0000","last_undegraded":"2024-03-18T15:15:17.578802+0000","last_fullsized":"2024-03-18T15:15:17.578802+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:57:50.087837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:17.578411+0000","last_change":"2024-03-18T15:15:10.674672+0000","last_active":"2024-03-18T15:15:17.578411+0000","last_peered":"2024-03-18T15:15:17.578411+0000","last_clean":"2024-03-18T15:15:17.578411+0000","last_became_active":"2024-03-18T15:15:10.674346+0000","last_became_peered":"2024-03-18T15:15:10.674346+0000","last_unstale":"2024-03-18T15:15:17.578411+0000","last_undegraded":"2024-03-18T15:15:17.578411+0000","last_fullsized":"2024-03-18T15:15:17.578411+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:21:40.054636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:16.415207+0000","last_change":"2024-03-18T15:15:10.660022+0000","last_active":"2024-03-18T15:15:16.415207+0000","last_peered":"2024-03-18T15:15:16.415207+0000","last_clean":"2024-03-18T15:15:16.415207+0000","last_became_active":"2024-03-18T15:15:10.659622+0000","last_became_peered":"2024-03-18T15:15:10.659622+0000","last_unstale":"2024-03-18T15:15:16.415207+0000","last_undegraded":"2024-03-18T15:15:16.415207+0000","last_fullsized":"2024-03-18T15:15:16.415207+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:55:33.777539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.024343+0000","last_change":"2024-03-18T15:15:10.681499+0000","last_active":"2024-03-18T15:15:18.024343+0000","last_peered":"2024-03-18T15:15:18.024343+0000","last_clean":"2024-03-18T15:15:18.024343+0000","last_became_active":"2024-03-18T15:15:10.681034+0000","last_became_peered":"2024-03-18T15:15:10.681034+0000","last_unstale":"2024-03-18T15:15:18.024343+0000","last_undegraded":"2024-03-18T15:15:18.024343+0000","last_fullsized":"2024-03-18T15:15:18.024343+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:16:43.171924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.023583+0000","last_change":"2024-03-18T15:15:10.669968+0000","last_active":"2024-03-18T15:15:18.023583+0000","last_peered":"2024-03-18T15:15:18.023583+0000","last_clean":"2024-03-18T15:15:18.023583+0000","last_became_active":"2024-03-18T15:15:10.668477+0000","last_became_peered":"2024-03-18T15:15:10.668477+0000","last_unstale":"2024-03-18T15:15:18.023583+0000","last_undegraded":"2024-03-18T15:15:18.023583+0000","last_fullsized":"2024-03-18T15:15:18.023583+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:14:46.364453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:16.104236+0000","last_change":"2024-03-18T15:15:10.669433+0000","last_active":"2024-03-18T15:15:16.104236+0000","last_peered":"2024-03-18T15:15:16.104236+0000","last_clean":"2024-03-18T15:15:16.104236+0000","last_became_active":"2024-03-18T15:15:10.669099+0000","last_became_peered":"2024-03-18T15:15:10.669099+0000","last_unstale":"2024-03-18T15:15:16.104236+0000","last_undegraded":"2024-03-18T15:15:16.104236+0000","last_fullsized":"2024-03-18T15:15:16.104236+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:37:20.697605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.023327+0000","last_change":"2024-03-18T15:15:10.681679+0000","last_active":"2024-03-18T15:15:18.023327+0000","last_peered":"2024-03-18T15:15:18.023327+0000","last_clean":"2024-03-18T15:15:18.023327+0000","last_became_active":"2024-03-18T15:15:10.681079+0000","last_became_peered":"2024-03-18T15:15:10.681079+0000","last_unstale":"2024-03-18T15:15:18.023327+0000","last_undegraded":"2024-03-18T15:15:18.023327+0000","last_fullsized":"2024-03-18T15:15:18.023327+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:24:39.639451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.023248+0000","last_change":"2024-03-18T15:15:10.664111+0000","last_active":"2024-03-18T15:15:18.023248+0000","last_peered":"2024-03-18T15:15:18.023248+0000","last_clean":"2024-03-18T15:15:18.023248+0000","last_became_active":"2024-03-18T15:15:10.663482+0000","last_became_peered":"2024-03-18T15:15:10.663482+0000","last_unstale":"2024-03-18T15:15:18.023248+0000","last_undegraded":"2024-03-18T15:15:18.023248+0000","last_fullsized":"2024-03-18T15:15:18.023248+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:49:50.810014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.023620+0000","last_change":"2024-03-18T15:15:10.657799+0000","last_active":"2024-03-18T15:15:18.023620+0000","last_peered":"2024-03-18T15:15:18.023620+0000","last_clean":"2024-03-18T15:15:18.023620+0000","last_became_active":"2024-03-18T15:15:10.657468+0000","last_became_peered":"2024-03-18T15:15:10.657468+0000","last_unstale":"2024-03-18T15:15:18.023620+0000","last_undegraded":"2024-03-18T15:15:18.023620+0000","last_fullsized":"2024-03-18T15:15:18.023620+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:22:46.025516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:16.104690+0000","last_change":"2024-03-18T15:15:10.665040+0000","last_active":"2024-03-18T15:15:16.104690+0000","last_peered":"2024-03-18T15:15:16.104690+0000","last_clean":"2024-03-18T15:15:16.104690+0000","last_became_active":"2024-03-18T15:15:10.663929+0000","last_became_peered":"2024-03-18T15:15:10.663929+0000","last_unstale":"2024-03-18T15:15:16.104690+0000","last_undegraded":"2024-03-18T15:15:16.104690+0000","last_fullsized":"2024-03-18T15:15:16.104690+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:05:29.954918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:16.415185+0000","last_change":"2024-03-18T15:15:10.662301+0000","last_active":"2024-03-18T15:15:16.415185+0000","last_peered":"2024-03-18T15:15:16.415185+0000","last_clean":"2024-03-18T15:15:16.415185+0000","last_became_active":"2024-03-18T15:15:10.661956+0000","last_became_peered":"2024-03-18T15:15:10.661956+0000","last_unstale":"2024-03-18T15:15:16.415185+0000","last_undegraded":"2024-03-18T15:15:16.415185+0000","last_fullsized":"2024-03-18T15:15:16.415185+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:30:36.699858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:17.599156+0000","last_change":"2024-03-18T15:15:10.650795+0000","last_active":"2024-03-18T15:15:17.599156+0000","last_peered":"2024-03-18T15:15:17.599156+0000","last_clean":"2024-03-18T15:15:17.599156+0000","last_became_active":"2024-03-18T15:15:10.649499+0000","last_became_peered":"2024-03-18T15:15:10.649499+0000","last_unstale":"2024-03-18T15:15:17.599156+0000","last_undegraded":"2024-03-18T15:15:17.599156+0000","last_fullsized":"2024-03-18T15:15:17.599156+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:30:19.094619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:17.577874+0000","last_change":"2024-03-18T15:15:10.677030+0000","last_active":"2024-03-18T15:15:17.577874+0000","last_peered":"2024-03-18T15:15:17.577874+0000","last_clean":"2024-03-18T15:15:17.577874+0000","last_became_active":"2024-03-18T15:15:10.676680+0000","last_became_peered":"2024-03-18T15:15:10.676680+0000","last_unstale":"2024-03-18T15:15:17.577874+0000","last_undegraded":"2024-03-18T15:15:17.577874+0000","last_fullsized":"2024-03-18T15:15:17.577874+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:32:16.382003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:17.578068+0000","last_change":"2024-03-18T15:15:10.677512+0000","last_active":"2024-03-18T15:15:17.578068+0000","last_peered":"2024-03-18T15:15:17.578068+0000","last_clean":"2024-03-18T15:15:17.578068+0000","last_became_active":"2024-03-18T15:15:10.670557+0000","last_became_peered":"2024-03-18T15:15:10.670557+0000","last_unstale":"2024-03-18T15:15:17.578068+0000","last_undegraded":"2024-03-18T15:15:17.578068+0000","last_fullsized":"2024-03-18T15:15:17.578068+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:06:58.330728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.022783+0000","last_change":"2024-03-18T15:15:10.675859+0000","last_active":"2024-03-18T15:15:18.022783+0000","last_peered":"2024-03-18T15:15:18.022783+0000","last_clean":"2024-03-18T15:15:18.022783+0000","last_became_active":"2024-03-18T15:15:10.675458+0000","last_became_peered":"2024-03-18T15:15:10.675458+0000","last_unstale":"2024-03-18T15:15:18.022783+0000","last_undegraded":"2024-03-18T15:15:18.022783+0000","last_fullsized":"2024-03-18T15:15:18.022783+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:07:43.306884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.024088+0000","last_change":"2024-03-18T15:15:10.677093+0000","last_active":"2024-03-18T15:15:18.024088+0000","last_peered":"2024-03-18T15:15:18.024088+0000","last_clean":"2024-03-18T15:15:18.024088+0000","last_became_active":"2024-03-18T15:15:10.676878+0000","last_became_peered":"2024-03-18T15:15:10.676878+0000","last_unstale":"2024-03-18T15:15:18.024088+0000","last_undegraded":"2024-03-18T15:15:18.024088+0000","last_fullsized":"2024-03-18T15:15:18.024088+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:18:23.517133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:16.108248+0000","last_change":"2024-03-18T15:15:10.674793+0000","last_active":"2024-03-18T15:15:16.108248+0000","last_peered":"2024-03-18T15:15:16.108248+0000","last_clean":"2024-03-18T15:15:16.108248+0000","last_became_active":"2024-03-18T15:15:10.674433+0000","last_became_peered":"2024-03-18T15:15:10.674433+0000","last_unstale":"2024-03-18T15:15:16.108248+0000","last_undegraded":"2024-03-18T15:15:16.108248+0000","last_fullsized":"2024-03-18T15:15:16.108248+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:17:24.774598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:16.105553+0000","last_change":"2024-03-18T15:15:10.670757+0000","last_active":"2024-03-18T15:15:16.105553+0000","last_peered":"2024-03-18T15:15:16.105553+0000","last_clean":"2024-03-18T15:15:16.105553+0000","last_became_active":"2024-03-18T15:15:10.670483+0000","last_became_peered":"2024-03-18T15:15:10.670483+0000","last_unstale":"2024-03-18T15:15:16.105553+0000","last_undegraded":"2024-03-18T15:15:16.105553+0000","last_fullsized":"2024-03-18T15:15:16.105553+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:59:20.070597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:16.105358+0000","last_change":"2024-03-18T15:15:15.115252+0000","last_active":"2024-03-18T15:15:16.105358+0000","last_peered":"2024-03-18T15:15:16.105358+0000","last_clean":"2024-03-18T15:15:16.105358+0000","last_became_active":"2024-03-18T15:15:15.114812+0000","last_became_peered":"2024-03-18T15:15:15.114812+0000","last_unstale":"2024-03-18T15:15:16.105358+0000","last_undegraded":"2024-03-18T15:15:16.105358+0000","last_fullsized":"2024-03-18T15:15:16.105358+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:04:15.657483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'9","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:44.495561+0000","last_change":"2024-03-18T15:15:10.092873+0000","last_active":"2024-03-18T15:15:44.495561+0000","last_peered":"2024-03-18T15:15:44.495561+0000","last_clean":"2024-03-18T15:15:44.495561+0000","last_became_active":"2024-03-18T15:15:10.092456+0000","last_became_peered":"2024-03-18T15:15:10.092456+0000","last_unstale":"2024-03-18T15:15:44.495561+0000","last_undegraded":"2024-03-18T15:15:44.495561+0000","last_fullsized":"2024-03-18T15:15:44.495561+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:01:53.596264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.024152+0000","last_change":"2024-03-18T15:15:10.682119+0000","last_active":"2024-03-18T15:15:18.024152+0000","last_peered":"2024-03-18T15:15:18.024152+0000","last_clean":"2024-03-18T15:15:18.024152+0000","last_became_active":"2024-03-18T15:15:10.680489+0000","last_became_peered":"2024-03-18T15:15:10.680489+0000","last_unstale":"2024-03-18T15:15:18.024152+0000","last_undegraded":"2024-03-18T15:15:18.024152+0000","last_fullsized":"2024-03-18T15:15:18.024152+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:52:13.477107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:16.414854+0000","last_change":"2024-03-18T15:15:15.114645+0000","last_active":"2024-03-18T15:15:16.414854+0000","last_peered":"2024-03-18T15:15:16.414854+0000","last_clean":"2024-03-18T15:15:16.414854+0000","last_became_active":"2024-03-18T15:15:15.114245+0000","last_became_peered":"2024-03-18T15:15:15.114245+0000","last_unstale":"2024-03-18T15:15:16.414854+0000","last_undegraded":"2024-03-18T15:15:16.414854+0000","last_fullsized":"2024-03-18T15:15:16.414854+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:08:31.429216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.024632+0000","last_change":"2024-03-18T15:15:10.100265+0000","last_active":"2024-03-18T15:15:18.024632+0000","last_peered":"2024-03-18T15:15:18.024632+0000","last_clean":"2024-03-18T15:15:18.024632+0000","last_became_active":"2024-03-18T15:15:10.099861+0000","last_became_peered":"2024-03-18T15:15:10.099861+0000","last_unstale":"2024-03-18T15:15:18.024632+0000","last_undegraded":"2024-03-18T15:15:18.024632+0000","last_fullsized":"2024-03-18T15:15:18.024632+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:52:47.863313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:17.579240+0000","last_change":"2024-03-18T15:15:10.682009+0000","last_active":"2024-03-18T15:15:17.579240+0000","last_peered":"2024-03-18T15:15:17.579240+0000","last_clean":"2024-03-18T15:15:17.579240+0000","last_became_active":"2024-03-18T15:15:10.680026+0000","last_became_peered":"2024-03-18T15:15:10.680026+0000","last_unstale":"2024-03-18T15:15:17.579240+0000","last_undegraded":"2024-03-18T15:15:17.579240+0000","last_fullsized":"2024-03-18T15:15:17.579240+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:00:27.869376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:16.415059+0000","last_change":"2024-03-18T15:15:15.124914+0000","last_active":"2024-03-18T15:15:16.415059+0000","last_peered":"2024-03-18T15:15:16.415059+0000","last_clean":"2024-03-18T15:15:16.415059+0000","last_became_active":"2024-03-18T15:15:15.124477+0000","last_became_peered":"2024-03-18T15:15:15.124477+0000","last_unstale":"2024-03-18T15:15:16.415059+0000","last_undegraded":"2024-03-18T15:15:16.415059+0000","last_fullsized":"2024-03-18T15:15:16.415059+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:58:40.596613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'7","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:45.495439+0000","last_change":"2024-03-18T15:15:10.083718+0000","last_active":"2024-03-18T15:15:45.495439+0000","last_peered":"2024-03-18T15:15:45.495439+0000","last_clean":"2024-03-18T15:15:45.495439+0000","last_became_active":"2024-03-18T15:15:10.083350+0000","last_became_peered":"2024-03-18T15:15:10.083350+0000","last_unstale":"2024-03-18T15:15:45.495439+0000","last_undegraded":"2024-03-18T15:15:45.495439+0000","last_fullsized":"2024-03-18T15:15:45.495439+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:59:52.912657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.023026+0000","last_change":"2024-03-18T15:15:10.669514+0000","last_active":"2024-03-18T15:15:18.023026+0000","last_peered":"2024-03-18T15:15:18.023026+0000","last_clean":"2024-03-18T15:15:18.023026+0000","last_became_active":"2024-03-18T15:15:10.669154+0000","last_became_peered":"2024-03-18T15:15:10.669154+0000","last_unstale":"2024-03-18T15:15:18.023026+0000","last_undegraded":"2024-03-18T15:15:18.023026+0000","last_fullsized":"2024-03-18T15:15:18.023026+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:10:49.068869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:16.414871+0000","last_change":"2024-03-18T15:15:15.112079+0000","last_active":"2024-03-18T15:15:16.414871+0000","last_peered":"2024-03-18T15:15:16.414871+0000","last_clean":"2024-03-18T15:15:16.414871+0000","last_became_active":"2024-03-18T15:15:15.111652+0000","last_became_peered":"2024-03-18T15:15:15.111652+0000","last_unstale":"2024-03-18T15:15:16.414871+0000","last_undegraded":"2024-03-18T15:15:16.414871+0000","last_fullsized":"2024-03-18T15:15:16.414871+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:15:55.542490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.025108+0000","last_change":"2024-03-18T15:15:10.077975+0000","last_active":"2024-03-18T15:15:18.025108+0000","last_peered":"2024-03-18T15:15:18.025108+0000","last_clean":"2024-03-18T15:15:18.025108+0000","last_became_active":"2024-03-18T15:15:10.077583+0000","last_became_peered":"2024-03-18T15:15:10.077583+0000","last_unstale":"2024-03-18T15:15:18.025108+0000","last_undegraded":"2024-03-18T15:15:18.025108+0000","last_fullsized":"2024-03-18T15:15:18.025108+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:41:43.741623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.024106+0000","last_change":"2024-03-18T15:15:10.665107+0000","last_active":"2024-03-18T15:15:18.024106+0000","last_peered":"2024-03-18T15:15:18.024106+0000","last_clean":"2024-03-18T15:15:18.024106+0000","last_became_active":"2024-03-18T15:15:10.664786+0000","last_became_peered":"2024-03-18T15:15:10.664786+0000","last_unstale":"2024-03-18T15:15:18.024106+0000","last_undegraded":"2024-03-18T15:15:18.024106+0000","last_fullsized":"2024-03-18T15:15:18.024106+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:58:40.439564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:16.105527+0000","last_change":"2024-03-18T15:15:15.100480+0000","last_active":"2024-03-18T15:15:16.105527+0000","last_peered":"2024-03-18T15:15:16.105527+0000","last_clean":"2024-03-18T15:15:16.105527+0000","last_became_active":"2024-03-18T15:15:15.100015+0000","last_became_peered":"2024-03-18T15:15:15.100015+0000","last_unstale":"2024-03-18T15:15:16.105527+0000","last_undegraded":"2024-03-18T15:15:16.105527+0000","last_fullsized":"2024-03-18T15:15:16.105527+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:50:08.723714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:43.469397+0000","last_change":"2024-03-18T15:15:10.082592+0000","last_active":"2024-03-18T15:15:43.469397+0000","last_peered":"2024-03-18T15:15:43.469397+0000","last_clean":"2024-03-18T15:15:43.469397+0000","last_became_active":"2024-03-18T15:15:10.082269+0000","last_became_peered":"2024-03-18T15:15:10.082269+0000","last_unstale":"2024-03-18T15:15:43.469397+0000","last_undegraded":"2024-03-18T15:15:43.469397+0000","last_fullsized":"2024-03-18T15:15:43.469397+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:18:57.751331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.024033+0000","last_change":"2024-03-18T15:15:10.667803+0000","last_active":"2024-03-18T15:15:18.024033+0000","last_peered":"2024-03-18T15:15:18.024033+0000","last_clean":"2024-03-18T15:15:18.024033+0000","last_became_active":"2024-03-18T15:15:10.667478+0000","last_became_peered":"2024-03-18T15:15:10.667478+0000","last_unstale":"2024-03-18T15:15:18.024033+0000","last_undegraded":"2024-03-18T15:15:18.024033+0000","last_fullsized":"2024-03-18T15:15:18.024033+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:22:48.340642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:16.105523+0000","last_change":"2024-03-18T15:15:15.116404+0000","last_active":"2024-03-18T15:15:16.105523+0000","last_peered":"2024-03-18T15:15:16.105523+0000","last_clean":"2024-03-18T15:15:16.105523+0000","last_became_active":"2024-03-18T15:15:15.115952+0000","last_became_peered":"2024-03-18T15:15:15.115952+0000","last_unstale":"2024-03-18T15:15:16.105523+0000","last_undegraded":"2024-03-18T15:15:16.105523+0000","last_fullsized":"2024-03-18T15:15:16.105523+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:38:44.283538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'7","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:45.502553+0000","last_change":"2024-03-18T15:15:10.080879+0000","last_active":"2024-03-18T15:15:45.502553+0000","last_peered":"2024-03-18T15:15:45.502553+0000","last_clean":"2024-03-18T15:15:45.502553+0000","last_became_active":"2024-03-18T15:15:10.080478+0000","last_became_peered":"2024-03-18T15:15:10.080478+0000","last_unstale":"2024-03-18T15:15:45.502553+0000","last_undegraded":"2024-03-18T15:15:45.502553+0000","last_fullsized":"2024-03-18T15:15:45.502553+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:28:36.511689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.024722+0000","last_change":"2024-03-18T15:15:10.681145+0000","last_active":"2024-03-18T15:15:18.024722+0000","last_peered":"2024-03-18T15:15:18.024722+0000","last_clean":"2024-03-18T15:15:18.024722+0000","last_became_active":"2024-03-18T15:15:10.680816+0000","last_became_peered":"2024-03-18T15:15:10.680816+0000","last_unstale":"2024-03-18T15:15:18.024722+0000","last_undegraded":"2024-03-18T15:15:18.024722+0000","last_fullsized":"2024-03-18T15:15:18.024722+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:15:56.665596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.024082+0000","last_change":"2024-03-18T15:15:15.111563+0000","last_active":"2024-03-18T15:15:18.024082+0000","last_peered":"2024-03-18T15:15:18.024082+0000","last_clean":"2024-03-18T15:15:18.024082+0000","last_became_active":"2024-03-18T15:15:15.111169+0000","last_became_peered":"2024-03-18T15:15:15.111169+0000","last_unstale":"2024-03-18T15:15:18.024082+0000","last_undegraded":"2024-03-18T15:15:18.024082+0000","last_fullsized":"2024-03-18T15:15:18.024082+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:44:05.491883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:17.598765+0000","last_change":"2024-03-18T15:15:10.091753+0000","last_active":"2024-03-18T15:15:17.598765+0000","last_peered":"2024-03-18T15:15:17.598765+0000","last_clean":"2024-03-18T15:15:17.598765+0000","last_became_active":"2024-03-18T15:15:10.091393+0000","last_became_peered":"2024-03-18T15:15:10.091393+0000","last_unstale":"2024-03-18T15:15:17.598765+0000","last_undegraded":"2024-03-18T15:15:17.598765+0000","last_fullsized":"2024-03-18T15:15:17.598765+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:24:28.068621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.023592+0000","last_change":"2024-03-18T15:15:10.672785+0000","last_active":"2024-03-18T15:15:18.023592+0000","last_peered":"2024-03-18T15:15:18.023592+0000","last_clean":"2024-03-18T15:15:18.023592+0000","last_became_active":"2024-03-18T15:15:10.672143+0000","last_became_peered":"2024-03-18T15:15:10.672143+0000","last_unstale":"2024-03-18T15:15:18.023592+0000","last_undegraded":"2024-03-18T15:15:18.023592+0000","last_fullsized":"2024-03-18T15:15:18.023592+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:26:40.034989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:17.578797+0000","last_change":"2024-03-18T15:15:15.114790+0000","last_active":"2024-03-18T15:15:17.578797+0000","last_peered":"2024-03-18T15:15:17.578797+0000","last_clean":"2024-03-18T15:15:17.578797+0000","last_became_active":"2024-03-18T15:15:15.112646+0000","last_became_peered":"2024-03-18T15:15:15.112646+0000","last_unstale":"2024-03-18T15:15:17.578797+0000","last_undegraded":"2024-03-18T15:15:17.578797+0000","last_fullsized":"2024-03-18T15:15:17.578797+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:59:56.686888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.023794+0000","last_change":"2024-03-18T15:15:10.082566+0000","last_active":"2024-03-18T15:15:18.023794+0000","last_peered":"2024-03-18T15:15:18.023794+0000","last_clean":"2024-03-18T15:15:18.023794+0000","last_became_active":"2024-03-18T15:15:10.082256+0000","last_became_peered":"2024-03-18T15:15:10.082256+0000","last_unstale":"2024-03-18T15:15:18.023794+0000","last_undegraded":"2024-03-18T15:15:18.023794+0000","last_fullsized":"2024-03-18T15:15:18.023794+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:04:47.361932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:16.107351+0000","last_change":"2024-03-18T15:15:10.683148+0000","last_active":"2024-03-18T15:15:16.107351+0000","last_peered":"2024-03-18T15:15:16.107351+0000","last_clean":"2024-03-18T15:15:16.107351+0000","last_became_active":"2024-03-18T15:15:10.682671+0000","last_became_peered":"2024-03-18T15:15:10.682671+0000","last_unstale":"2024-03-18T15:15:16.107351+0000","last_undegraded":"2024-03-18T15:15:16.107351+0000","last_fullsized":"2024-03-18T15:15:16.107351+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:52:15.763400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.023476+0000","last_change":"2024-03-18T15:15:15.115995+0000","last_active":"2024-03-18T15:15:18.023476+0000","last_peered":"2024-03-18T15:15:18.023476+0000","last_clean":"2024-03-18T15:15:18.023476+0000","last_became_active":"2024-03-18T15:15:15.115560+0000","last_became_peered":"2024-03-18T15:15:15.115560+0000","last_unstale":"2024-03-18T15:15:18.023476+0000","last_undegraded":"2024-03-18T15:15:18.023476+0000","last_fullsized":"2024-03-18T15:15:18.023476+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:20:36.765286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'4","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:46.991418+0000","last_change":"2024-03-18T15:15:10.095122+0000","last_active":"2024-03-18T15:15:46.991418+0000","last_peered":"2024-03-18T15:15:46.991418+0000","last_clean":"2024-03-18T15:15:46.991418+0000","last_became_active":"2024-03-18T15:15:10.094797+0000","last_became_peered":"2024-03-18T15:15:10.094797+0000","last_unstale":"2024-03-18T15:15:46.991418+0000","last_undegraded":"2024-03-18T15:15:46.991418+0000","last_fullsized":"2024-03-18T15:15:46.991418+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:16:15.124369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.023429+0000","last_change":"2024-03-18T15:15:10.673297+0000","last_active":"2024-03-18T15:15:18.023429+0000","last_peered":"2024-03-18T15:15:18.023429+0000","last_clean":"2024-03-18T15:15:18.023429+0000","last_became_active":"2024-03-18T15:15:10.672890+0000","last_became_peered":"2024-03-18T15:15:10.672890+0000","last_unstale":"2024-03-18T15:15:18.023429+0000","last_undegraded":"2024-03-18T15:15:18.023429+0000","last_fullsized":"2024-03-18T15:15:18.023429+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:10:45.958639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:16.107137+0000","last_change":"2024-03-18T15:15:15.123200+0000","last_active":"2024-03-18T15:15:16.107137+0000","last_peered":"2024-03-18T15:15:16.107137+0000","last_clean":"2024-03-18T15:15:16.107137+0000","last_became_active":"2024-03-18T15:15:15.114554+0000","last_became_peered":"2024-03-18T15:15:15.114554+0000","last_unstale":"2024-03-18T15:15:16.107137+0000","last_undegraded":"2024-03-18T15:15:16.107137+0000","last_fullsized":"2024-03-18T15:15:16.107137+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:35:25.380377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:43.467710+0000","last_change":"2024-03-18T15:15:10.085330+0000","last_active":"2024-03-18T15:15:43.467710+0000","last_peered":"2024-03-18T15:15:43.467710+0000","last_clean":"2024-03-18T15:15:43.467710+0000","last_became_active":"2024-03-18T15:15:10.083179+0000","last_became_peered":"2024-03-18T15:15:10.083179+0000","last_unstale":"2024-03-18T15:15:43.467710+0000","last_undegraded":"2024-03-18T15:15:43.467710+0000","last_fullsized":"2024-03-18T15:15:43.467710+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:56:33.843757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:17.579017+0000","last_change":"2024-03-18T15:15:10.679397+0000","last_active":"2024-03-18T15:15:17.579017+0000","last_peered":"2024-03-18T15:15:17.579017+0000","last_clean":"2024-03-18T15:15:17.579017+0000","last_became_active":"2024-03-18T15:15:10.679006+0000","last_became_peered":"2024-03-18T15:15:10.679006+0000","last_unstale":"2024-03-18T15:15:17.579017+0000","last_undegraded":"2024-03-18T15:15:17.579017+0000","last_fullsized":"2024-03-18T15:15:17.579017+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:38:58.251600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:16.415258+0000","last_change":"2024-03-18T15:15:15.124936+0000","last_active":"2024-03-18T15:15:16.415258+0000","last_peered":"2024-03-18T15:15:16.415258+0000","last_clean":"2024-03-18T15:15:16.415258+0000","last_became_active":"2024-03-18T15:15:15.124501+0000","last_became_peered":"2024-03-18T15:15:15.124501+0000","last_unstale":"2024-03-18T15:15:16.415258+0000","last_undegraded":"2024-03-18T15:15:16.415258+0000","last_fullsized":"2024-03-18T15:15:16.415258+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:31:47.221428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.024042+0000","last_change":"2024-03-18T15:15:10.098885+0000","last_active":"2024-03-18T15:15:18.024042+0000","last_peered":"2024-03-18T15:15:18.024042+0000","last_clean":"2024-03-18T15:15:18.024042+0000","last_became_active":"2024-03-18T15:15:10.098604+0000","last_became_peered":"2024-03-18T15:15:10.098604+0000","last_unstale":"2024-03-18T15:15:18.024042+0000","last_undegraded":"2024-03-18T15:15:18.024042+0000","last_fullsized":"2024-03-18T15:15:18.024042+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:59:08.716855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.023208+0000","last_change":"2024-03-18T15:15:10.662297+0000","last_active":"2024-03-18T15:15:18.023208+0000","last_peered":"2024-03-18T15:15:18.023208+0000","last_clean":"2024-03-18T15:15:18.023208+0000","last_became_active":"2024-03-18T15:15:10.661910+0000","last_became_peered":"2024-03-18T15:15:10.661910+0000","last_unstale":"2024-03-18T15:15:18.023208+0000","last_undegraded":"2024-03-18T15:15:18.023208+0000","last_fullsized":"2024-03-18T15:15:18.023208+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:26:38.043543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:16.107603+0000","last_change":"2024-03-18T15:15:15.107352+0000","last_active":"2024-03-18T15:15:16.107603+0000","last_peered":"2024-03-18T15:15:16.107603+0000","last_clean":"2024-03-18T15:15:16.107603+0000","last_became_active":"2024-03-18T15:15:15.106851+0000","last_became_peered":"2024-03-18T15:15:15.106851+0000","last_unstale":"2024-03-18T15:15:16.107603+0000","last_undegraded":"2024-03-18T15:15:16.107603+0000","last_fullsized":"2024-03-18T15:15:16.107603+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:56:35.695188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'22","reported_seq":49,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:46.992087+0000","last_change":"2024-03-18T15:15:10.083226+0000","last_active":"2024-03-18T15:15:46.992087+0000","last_peered":"2024-03-18T15:15:46.992087+0000","last_clean":"2024-03-18T15:15:46.992087+0000","last_became_active":"2024-03-18T15:15:10.082829+0000","last_became_peered":"2024-03-18T15:15:10.082829+0000","last_unstale":"2024-03-18T15:15:46.992087+0000","last_undegraded":"2024-03-18T15:15:46.992087+0000","last_fullsized":"2024-03-18T15:15:46.992087+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:30:34.091815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23141,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":24,"num_write":24,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.024552+0000","last_change":"2024-03-18T15:15:10.659322+0000","last_active":"2024-03-18T15:15:18.024552+0000","last_peered":"2024-03-18T15:15:18.024552+0000","last_clean":"2024-03-18T15:15:18.024552+0000","last_became_active":"2024-03-18T15:15:10.658997+0000","last_became_peered":"2024-03-18T15:15:10.658997+0000","last_unstale":"2024-03-18T15:15:18.024552+0000","last_undegraded":"2024-03-18T15:15:18.024552+0000","last_fullsized":"2024-03-18T15:15:18.024552+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:04:08.934683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.025229+0000","last_change":"2024-03-18T15:15:15.110658+0000","last_active":"2024-03-18T15:15:18.025229+0000","last_peered":"2024-03-18T15:15:18.025229+0000","last_clean":"2024-03-18T15:15:18.025229+0000","last_became_active":"2024-03-18T15:15:15.110205+0000","last_became_peered":"2024-03-18T15:15:15.110205+0000","last_unstale":"2024-03-18T15:15:18.025229+0000","last_undegraded":"2024-03-18T15:15:18.025229+0000","last_fullsized":"2024-03-18T15:15:18.025229+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:09:10.014743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'7","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:44.489491+0000","last_change":"2024-03-18T15:15:10.080305+0000","last_active":"2024-03-18T15:15:44.489491+0000","last_peered":"2024-03-18T15:15:44.489491+0000","last_clean":"2024-03-18T15:15:44.489491+0000","last_became_active":"2024-03-18T15:15:10.079990+0000","last_became_peered":"2024-03-18T15:15:10.079990+0000","last_unstale":"2024-03-18T15:15:44.489491+0000","last_undegraded":"2024-03-18T15:15:44.489491+0000","last_fullsized":"2024-03-18T15:15:44.489491+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:10:08.312462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:17.598866+0000","last_change":"2024-03-18T15:15:10.671178+0000","last_active":"2024-03-18T15:15:17.598866+0000","last_peered":"2024-03-18T15:15:17.598866+0000","last_clean":"2024-03-18T15:15:17.598866+0000","last_became_active":"2024-03-18T15:15:10.670862+0000","last_became_peered":"2024-03-18T15:15:10.670862+0000","last_unstale":"2024-03-18T15:15:17.598866+0000","last_undegraded":"2024-03-18T15:15:17.598866+0000","last_fullsized":"2024-03-18T15:15:17.598866+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:53:34.916473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.023580+0000","last_change":"2024-03-18T15:15:15.107792+0000","last_active":"2024-03-18T15:15:18.023580+0000","last_peered":"2024-03-18T15:15:18.023580+0000","last_clean":"2024-03-18T15:15:18.023580+0000","last_became_active":"2024-03-18T15:15:15.107327+0000","last_became_peered":"2024-03-18T15:15:15.107327+0000","last_unstale":"2024-03-18T15:15:18.023580+0000","last_undegraded":"2024-03-18T15:15:18.023580+0000","last_fullsized":"2024-03-18T15:15:18.023580+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:42:47.837146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:43.472011+0000","last_change":"2024-03-18T15:15:10.079128+0000","last_active":"2024-03-18T15:15:43.472011+0000","last_peered":"2024-03-18T15:15:43.472011+0000","last_clean":"2024-03-18T15:15:43.472011+0000","last_became_active":"2024-03-18T15:15:10.078812+0000","last_became_peered":"2024-03-18T15:15:10.078812+0000","last_unstale":"2024-03-18T15:15:43.472011+0000","last_undegraded":"2024-03-18T15:15:43.472011+0000","last_fullsized":"2024-03-18T15:15:43.472011+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:12:11.473273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:16.104672+0000","last_change":"2024-03-18T15:15:10.668240+0000","last_active":"2024-03-18T15:15:16.104672+0000","last_peered":"2024-03-18T15:15:16.104672+0000","last_clean":"2024-03-18T15:15:16.104672+0000","last_became_active":"2024-03-18T15:15:10.667885+0000","last_became_peered":"2024-03-18T15:15:10.667885+0000","last_unstale":"2024-03-18T15:15:16.104672+0000","last_undegraded":"2024-03-18T15:15:16.104672+0000","last_fullsized":"2024-03-18T15:15:16.104672+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:17:23.998018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:16.108323+0000","last_change":"2024-03-18T15:15:15.119067+0000","last_active":"2024-03-18T15:15:16.108323+0000","last_peered":"2024-03-18T15:15:16.108323+0000","last_clean":"2024-03-18T15:15:16.108323+0000","last_became_active":"2024-03-18T15:15:15.118566+0000","last_became_peered":"2024-03-18T15:15:15.118566+0000","last_unstale":"2024-03-18T15:15:16.108323+0000","last_undegraded":"2024-03-18T15:15:16.108323+0000","last_fullsized":"2024-03-18T15:15:16.108323+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:21:00.868724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:45.496062+0000","last_change":"2024-03-18T15:15:10.080791+0000","last_active":"2024-03-18T15:15:45.496062+0000","last_peered":"2024-03-18T15:15:45.496062+0000","last_clean":"2024-03-18T15:15:45.496062+0000","last_became_active":"2024-03-18T15:15:10.080368+0000","last_became_peered":"2024-03-18T15:15:10.080368+0000","last_unstale":"2024-03-18T15:15:45.496062+0000","last_undegraded":"2024-03-18T15:15:45.496062+0000","last_fullsized":"2024-03-18T15:15:45.496062+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:10:55.575331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:16.104498+0000","last_change":"2024-03-18T15:15:10.666395+0000","last_active":"2024-03-18T15:15:16.104498+0000","last_peered":"2024-03-18T15:15:16.104498+0000","last_clean":"2024-03-18T15:15:16.104498+0000","last_became_active":"2024-03-18T15:15:10.666057+0000","last_became_peered":"2024-03-18T15:15:10.666057+0000","last_unstale":"2024-03-18T15:15:16.104498+0000","last_undegraded":"2024-03-18T15:15:16.104498+0000","last_fullsized":"2024-03-18T15:15:16.104498+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:04:50.639732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:17.599906+0000","last_change":"2024-03-18T15:15:15.125605+0000","last_active":"2024-03-18T15:15:17.599906+0000","last_peered":"2024-03-18T15:15:17.599906+0000","last_clean":"2024-03-18T15:15:17.599906+0000","last_became_active":"2024-03-18T15:15:15.125041+0000","last_became_peered":"2024-03-18T15:15:15.125041+0000","last_unstale":"2024-03-18T15:15:17.599906+0000","last_undegraded":"2024-03-18T15:15:17.599906+0000","last_fullsized":"2024-03-18T15:15:17.599906+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:27:25.901710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'16","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:44.492188+0000","last_change":"2024-03-18T15:15:10.079653+0000","last_active":"2024-03-18T15:15:44.492188+0000","last_peered":"2024-03-18T15:15:44.492188+0000","last_clean":"2024-03-18T15:15:44.492188+0000","last_became_active":"2024-03-18T15:15:10.079343+0000","last_became_peered":"2024-03-18T15:15:10.079343+0000","last_unstale":"2024-03-18T15:15:44.492188+0000","last_undegraded":"2024-03-18T15:15:44.492188+0000","last_fullsized":"2024-03-18T15:15:44.492188+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:33:58.746118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.024499+0000","last_change":"2024-03-18T15:15:10.661241+0000","last_active":"2024-03-18T15:15:18.024499+0000","last_peered":"2024-03-18T15:15:18.024499+0000","last_clean":"2024-03-18T15:15:18.024499+0000","last_became_active":"2024-03-18T15:15:10.660821+0000","last_became_peered":"2024-03-18T15:15:10.660821+0000","last_unstale":"2024-03-18T15:15:18.024499+0000","last_undegraded":"2024-03-18T15:15:18.024499+0000","last_fullsized":"2024-03-18T15:15:18.024499+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:13:59.973615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.024445+0000","last_change":"2024-03-18T15:15:15.118314+0000","last_active":"2024-03-18T15:15:18.024445+0000","last_peered":"2024-03-18T15:15:18.024445+0000","last_clean":"2024-03-18T15:15:18.024445+0000","last_became_active":"2024-03-18T15:15:15.118001+0000","last_became_peered":"2024-03-18T15:15:15.118001+0000","last_unstale":"2024-03-18T15:15:18.024445+0000","last_undegraded":"2024-03-18T15:15:18.024445+0000","last_fullsized":"2024-03-18T15:15:18.024445+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:15:58.320283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.024756+0000","last_change":"2024-03-18T15:15:10.077394+0000","last_active":"2024-03-18T15:15:18.024756+0000","last_peered":"2024-03-18T15:15:18.024756+0000","last_clean":"2024-03-18T15:15:18.024756+0000","last_became_active":"2024-03-18T15:15:10.077027+0000","last_became_peered":"2024-03-18T15:15:10.077027+0000","last_unstale":"2024-03-18T15:15:18.024756+0000","last_undegraded":"2024-03-18T15:15:18.024756+0000","last_fullsized":"2024-03-18T15:15:18.024756+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:34:07.524849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.023346+0000","last_change":"2024-03-18T15:15:10.659723+0000","last_active":"2024-03-18T15:15:18.023346+0000","last_peered":"2024-03-18T15:15:18.023346+0000","last_clean":"2024-03-18T15:15:18.023346+0000","last_became_active":"2024-03-18T15:15:10.659388+0000","last_became_peered":"2024-03-18T15:15:10.659388+0000","last_unstale":"2024-03-18T15:15:18.023346+0000","last_undegraded":"2024-03-18T15:15:18.023346+0000","last_fullsized":"2024-03-18T15:15:18.023346+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:21:52.605870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.023894+0000","last_change":"2024-03-18T15:15:15.115459+0000","last_active":"2024-03-18T15:15:18.023894+0000","last_peered":"2024-03-18T15:15:18.023894+0000","last_clean":"2024-03-18T15:15:18.023894+0000","last_became_active":"2024-03-18T15:15:15.115048+0000","last_became_peered":"2024-03-18T15:15:15.115048+0000","last_unstale":"2024-03-18T15:15:18.023894+0000","last_undegraded":"2024-03-18T15:15:18.023894+0000","last_fullsized":"2024-03-18T15:15:18.023894+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:04:23.118904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:16.107977+0000","last_change":"2024-03-18T15:15:10.096698+0000","last_active":"2024-03-18T15:15:16.107977+0000","last_peered":"2024-03-18T15:15:16.107977+0000","last_clean":"2024-03-18T15:15:16.107977+0000","last_became_active":"2024-03-18T15:15:10.096212+0000","last_became_peered":"2024-03-18T15:15:10.096212+0000","last_unstale":"2024-03-18T15:15:16.107977+0000","last_undegraded":"2024-03-18T15:15:16.107977+0000","last_fullsized":"2024-03-18T15:15:16.107977+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:02:07.627327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.023743+0000","last_change":"2024-03-18T15:15:10.672954+0000","last_active":"2024-03-18T15:15:18.023743+0000","last_peered":"2024-03-18T15:15:18.023743+0000","last_clean":"2024-03-18T15:15:18.023743+0000","last_became_active":"2024-03-18T15:15:10.672415+0000","last_became_peered":"2024-03-18T15:15:10.672415+0000","last_unstale":"2024-03-18T15:15:18.023743+0000","last_undegraded":"2024-03-18T15:15:18.023743+0000","last_fullsized":"2024-03-18T15:15:18.023743+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:47:48.421411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:17.599329+0000","last_change":"2024-03-18T15:15:15.121334+0000","last_active":"2024-03-18T15:15:17.599329+0000","last_peered":"2024-03-18T15:15:17.599329+0000","last_clean":"2024-03-18T15:15:17.599329+0000","last_became_active":"2024-03-18T15:15:15.119797+0000","last_became_peered":"2024-03-18T15:15:15.119797+0000","last_unstale":"2024-03-18T15:15:17.599329+0000","last_undegraded":"2024-03-18T15:15:17.599329+0000","last_fullsized":"2024-03-18T15:15:17.599329+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:28:39.072485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:16.104354+0000","last_change":"2024-03-18T15:15:10.082589+0000","last_active":"2024-03-18T15:15:16.104354+0000","last_peered":"2024-03-18T15:15:16.104354+0000","last_clean":"2024-03-18T15:15:16.104354+0000","last_became_active":"2024-03-18T15:15:10.082190+0000","last_became_peered":"2024-03-18T15:15:10.082190+0000","last_unstale":"2024-03-18T15:15:16.104354+0000","last_undegraded":"2024-03-18T15:15:16.104354+0000","last_fullsized":"2024-03-18T15:15:16.104354+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:55:15.631914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.023456+0000","last_change":"2024-03-18T15:15:10.672145+0000","last_active":"2024-03-18T15:15:18.023456+0000","last_peered":"2024-03-18T15:15:18.023456+0000","last_clean":"2024-03-18T15:15:18.023456+0000","last_became_active":"2024-03-18T15:15:10.670303+0000","last_became_peered":"2024-03-18T15:15:10.670303+0000","last_unstale":"2024-03-18T15:15:18.023456+0000","last_undegraded":"2024-03-18T15:15:18.023456+0000","last_fullsized":"2024-03-18T15:15:18.023456+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:22:50.528794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:17.599875+0000","last_change":"2024-03-18T15:15:15.113675+0000","last_active":"2024-03-18T15:15:17.599875+0000","last_peered":"2024-03-18T15:15:17.599875+0000","last_clean":"2024-03-18T15:15:17.599875+0000","last_became_active":"2024-03-18T15:15:15.113067+0000","last_became_peered":"2024-03-18T15:15:15.113067+0000","last_unstale":"2024-03-18T15:15:17.599875+0000","last_undegraded":"2024-03-18T15:15:17.599875+0000","last_fullsized":"2024-03-18T15:15:17.599875+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:37:41.751216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.023951+0000","last_change":"2024-03-18T15:15:10.658032+0000","last_active":"2024-03-18T15:15:18.023951+0000","last_peered":"2024-03-18T15:15:18.023951+0000","last_clean":"2024-03-18T15:15:18.023951+0000","last_became_active":"2024-03-18T15:15:10.657707+0000","last_became_peered":"2024-03-18T15:15:10.657707+0000","last_unstale":"2024-03-18T15:15:18.023951+0000","last_undegraded":"2024-03-18T15:15:18.023951+0000","last_fullsized":"2024-03-18T15:15:18.023951+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:44:24.137349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:17.598690+0000","last_change":"2024-03-18T15:15:10.085330+0000","last_active":"2024-03-18T15:15:17.598690+0000","last_peered":"2024-03-18T15:15:17.598690+0000","last_clean":"2024-03-18T15:15:17.598690+0000","last_became_active":"2024-03-18T15:15:10.080303+0000","last_became_peered":"2024-03-18T15:15:10.080303+0000","last_unstale":"2024-03-18T15:15:17.598690+0000","last_undegraded":"2024-03-18T15:15:17.598690+0000","last_fullsized":"2024-03-18T15:15:17.598690+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:26:39.293028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:16.104581+0000","last_change":"2024-03-18T15:15:15.125172+0000","last_active":"2024-03-18T15:15:16.104581+0000","last_peered":"2024-03-18T15:15:16.104581+0000","last_clean":"2024-03-18T15:15:16.104581+0000","last_became_active":"2024-03-18T15:15:15.124591+0000","last_became_peered":"2024-03-18T15:15:15.124591+0000","last_unstale":"2024-03-18T15:15:16.104581+0000","last_undegraded":"2024-03-18T15:15:16.104581+0000","last_fullsized":"2024-03-18T15:15:16.104581+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:50:16.339041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.023809+0000","last_change":"2024-03-18T15:15:10.685343+0000","last_active":"2024-03-18T15:15:18.023809+0000","last_peered":"2024-03-18T15:15:18.023809+0000","last_clean":"2024-03-18T15:15:18.023809+0000","last_became_active":"2024-03-18T15:15:10.684395+0000","last_became_peered":"2024-03-18T15:15:10.684395+0000","last_unstale":"2024-03-18T15:15:18.023809+0000","last_undegraded":"2024-03-18T15:15:18.023809+0000","last_fullsized":"2024-03-18T15:15:18.023809+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:11:26.982047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:45.500018+0000","last_change":"2024-03-18T15:15:10.085503+0000","last_active":"2024-03-18T15:15:45.500018+0000","last_peered":"2024-03-18T15:15:45.500018+0000","last_clean":"2024-03-18T15:15:45.500018+0000","last_became_active":"2024-03-18T15:15:10.085125+0000","last_became_peered":"2024-03-18T15:15:10.085125+0000","last_unstale":"2024-03-18T15:15:45.500018+0000","last_undegraded":"2024-03-18T15:15:45.500018+0000","last_fullsized":"2024-03-18T15:15:45.500018+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:17:13.091428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":355,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:16.104198+0000","last_change":"2024-03-18T15:13:50.952630+0000","last_active":"2024-03-18T15:15:16.104198+0000","last_peered":"2024-03-18T15:15:16.104198+0000","last_clean":"2024-03-18T15:15:16.104198+0000","last_became_active":"2024-03-18T15:13:50.952144+0000","last_became_peered":"2024-03-18T15:13:50.952144+0000","last_unstale":"2024-03-18T15:15:16.104198+0000","last_undegraded":"2024-03-18T15:15:16.104198+0000","last_fullsized":"2024-03-18T15:15:16.104198+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:09:56.845161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:09:56.845161+0000","last_clean_scrub_stamp":"2024-03-18T15:09:56.845161+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:53:27.959711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.024727+0000","last_change":"2024-03-18T15:15:15.115418+0000","last_active":"2024-03-18T15:15:18.024727+0000","last_peered":"2024-03-18T15:15:18.024727+0000","last_clean":"2024-03-18T15:15:18.024727+0000","last_became_active":"2024-03-18T15:15:15.114715+0000","last_became_peered":"2024-03-18T15:15:15.114715+0000","last_unstale":"2024-03-18T15:15:18.024727+0000","last_undegraded":"2024-03-18T15:15:18.024727+0000","last_fullsized":"2024-03-18T15:15:18.024727+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:37:51.553099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:17.598587+0000","last_change":"2024-03-18T15:15:10.665492+0000","last_active":"2024-03-18T15:15:17.598587+0000","last_peered":"2024-03-18T15:15:17.598587+0000","last_clean":"2024-03-18T15:15:17.598587+0000","last_became_active":"2024-03-18T15:15:10.665097+0000","last_became_peered":"2024-03-18T15:15:10.665097+0000","last_unstale":"2024-03-18T15:15:17.598587+0000","last_undegraded":"2024-03-18T15:15:17.598587+0000","last_fullsized":"2024-03-18T15:15:17.598587+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:27:06.864459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:44.484810+0000","last_change":"2024-03-18T15:15:10.085322+0000","last_active":"2024-03-18T15:15:44.484810+0000","last_peered":"2024-03-18T15:15:44.484810+0000","last_clean":"2024-03-18T15:15:44.484810+0000","last_became_active":"2024-03-18T15:15:10.083114+0000","last_became_peered":"2024-03-18T15:15:10.083114+0000","last_unstale":"2024-03-18T15:15:44.484810+0000","last_undegraded":"2024-03-18T15:15:44.484810+0000","last_fullsized":"2024-03-18T15:15:44.484810+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:13:31.878513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.023259+0000","last_change":"2024-03-18T15:15:15.105432+0000","last_active":"2024-03-18T15:15:18.023259+0000","last_peered":"2024-03-18T15:15:18.023259+0000","last_clean":"2024-03-18T15:15:18.023259+0000","last_became_active":"2024-03-18T15:15:15.104929+0000","last_became_peered":"2024-03-18T15:15:15.104929+0000","last_unstale":"2024-03-18T15:15:18.023259+0000","last_undegraded":"2024-03-18T15:15:18.023259+0000","last_fullsized":"2024-03-18T15:15:18.023259+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:50:37.334639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:45.500978+0000","last_change":"2024-03-18T15:15:10.099722+0000","last_active":"2024-03-18T15:15:45.500978+0000","last_peered":"2024-03-18T15:15:45.500978+0000","last_clean":"2024-03-18T15:15:45.500978+0000","last_became_active":"2024-03-18T15:15:10.099410+0000","last_became_peered":"2024-03-18T15:15:10.099410+0000","last_unstale":"2024-03-18T15:15:45.500978+0000","last_undegraded":"2024-03-18T15:15:45.500978+0000","last_fullsized":"2024-03-18T15:15:45.500978+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:19:24.877473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:17.599504+0000","last_change":"2024-03-18T15:15:10.669235+0000","last_active":"2024-03-18T15:15:17.599504+0000","last_peered":"2024-03-18T15:15:17.599504+0000","last_clean":"2024-03-18T15:15:17.599504+0000","last_became_active":"2024-03-18T15:15:10.667369+0000","last_became_peered":"2024-03-18T15:15:10.667369+0000","last_unstale":"2024-03-18T15:15:17.599504+0000","last_undegraded":"2024-03-18T15:15:17.599504+0000","last_fullsized":"2024-03-18T15:15:17.599504+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:36:08.961435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.023835+0000","last_change":"2024-03-18T15:15:15.119228+0000","last_active":"2024-03-18T15:15:18.023835+0000","last_peered":"2024-03-18T15:15:18.023835+0000","last_clean":"2024-03-18T15:15:18.023835+0000","last_became_active":"2024-03-18T15:15:15.115585+0000","last_became_peered":"2024-03-18T15:15:15.115585+0000","last_unstale":"2024-03-18T15:15:18.023835+0000","last_undegraded":"2024-03-18T15:15:18.023835+0000","last_fullsized":"2024-03-18T15:15:18.023835+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:36:43.263907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:44.493312+0000","last_change":"2024-03-18T15:15:10.105259+0000","last_active":"2024-03-18T15:15:44.493312+0000","last_peered":"2024-03-18T15:15:44.493312+0000","last_clean":"2024-03-18T15:15:44.493312+0000","last_became_active":"2024-03-18T15:15:10.104315+0000","last_became_peered":"2024-03-18T15:15:10.104315+0000","last_unstale":"2024-03-18T15:15:44.493312+0000","last_undegraded":"2024-03-18T15:15:44.493312+0000","last_fullsized":"2024-03-18T15:15:44.493312+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:27:05.565134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.024161+0000","last_change":"2024-03-18T15:15:10.673615+0000","last_active":"2024-03-18T15:15:18.024161+0000","last_peered":"2024-03-18T15:15:18.024161+0000","last_clean":"2024-03-18T15:15:18.024161+0000","last_became_active":"2024-03-18T15:15:10.672308+0000","last_became_peered":"2024-03-18T15:15:10.672308+0000","last_unstale":"2024-03-18T15:15:18.024161+0000","last_undegraded":"2024-03-18T15:15:18.024161+0000","last_fullsized":"2024-03-18T15:15:18.024161+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:32:13.889281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:16.105119+0000","last_change":"2024-03-18T15:15:15.099447+0000","last_active":"2024-03-18T15:15:16.105119+0000","last_peered":"2024-03-18T15:15:16.105119+0000","last_clean":"2024-03-18T15:15:16.105119+0000","last_became_active":"2024-03-18T15:15:15.098857+0000","last_became_peered":"2024-03-18T15:15:15.098857+0000","last_unstale":"2024-03-18T15:15:16.105119+0000","last_undegraded":"2024-03-18T15:15:16.105119+0000","last_fullsized":"2024-03-18T15:15:16.105119+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:51:22.115359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:43.468516+0000","last_change":"2024-03-18T15:15:10.084281+0000","last_active":"2024-03-18T15:15:43.468516+0000","last_peered":"2024-03-18T15:15:43.468516+0000","last_clean":"2024-03-18T15:15:43.468516+0000","last_became_active":"2024-03-18T15:15:10.083791+0000","last_became_peered":"2024-03-18T15:15:10.083791+0000","last_unstale":"2024-03-18T15:15:43.468516+0000","last_undegraded":"2024-03-18T15:15:43.468516+0000","last_fullsized":"2024-03-18T15:15:43.468516+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:08:33.066629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:17.578906+0000","last_change":"2024-03-18T15:15:10.682823+0000","last_active":"2024-03-18T15:15:17.578906+0000","last_peered":"2024-03-18T15:15:17.578906+0000","last_clean":"2024-03-18T15:15:17.578906+0000","last_became_active":"2024-03-18T15:15:10.681887+0000","last_became_peered":"2024-03-18T15:15:10.681887+0000","last_unstale":"2024-03-18T15:15:17.578906+0000","last_undegraded":"2024-03-18T15:15:17.578906+0000","last_fullsized":"2024-03-18T15:15:17.578906+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:33:29.918824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:16.415170+0000","last_change":"2024-03-18T15:15:15.114952+0000","last_active":"2024-03-18T15:15:16.415170+0000","last_peered":"2024-03-18T15:15:16.415170+0000","last_clean":"2024-03-18T15:15:16.415170+0000","last_became_active":"2024-03-18T15:15:15.114331+0000","last_became_peered":"2024-03-18T15:15:15.114331+0000","last_unstale":"2024-03-18T15:15:16.415170+0000","last_undegraded":"2024-03-18T15:15:16.415170+0000","last_fullsized":"2024-03-18T15:15:16.415170+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:22:08.637152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:43.471482+0000","last_change":"2024-03-18T15:15:10.080291+0000","last_active":"2024-03-18T15:15:43.471482+0000","last_peered":"2024-03-18T15:15:43.471482+0000","last_clean":"2024-03-18T15:15:43.471482+0000","last_became_active":"2024-03-18T15:15:10.079634+0000","last_became_peered":"2024-03-18T15:15:10.079634+0000","last_unstale":"2024-03-18T15:15:43.471482+0000","last_undegraded":"2024-03-18T15:15:43.471482+0000","last_fullsized":"2024-03-18T15:15:43.471482+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:12:10.280094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.023550+0000","last_change":"2024-03-18T15:15:10.657245+0000","last_active":"2024-03-18T15:15:18.023550+0000","last_peered":"2024-03-18T15:15:18.023550+0000","last_clean":"2024-03-18T15:15:18.023550+0000","last_became_active":"2024-03-18T15:15:10.655766+0000","last_became_peered":"2024-03-18T15:15:10.655766+0000","last_unstale":"2024-03-18T15:15:18.023550+0000","last_undegraded":"2024-03-18T15:15:18.023550+0000","last_fullsized":"2024-03-18T15:15:18.023550+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:44:26.340539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:16.108571+0000","last_change":"2024-03-18T15:15:15.109103+0000","last_active":"2024-03-18T15:15:16.108571+0000","last_peered":"2024-03-18T15:15:16.108571+0000","last_clean":"2024-03-18T15:15:16.108571+0000","last_became_active":"2024-03-18T15:15:15.108693+0000","last_became_peered":"2024-03-18T15:15:15.108693+0000","last_unstale":"2024-03-18T15:15:16.108571+0000","last_undegraded":"2024-03-18T15:15:16.108571+0000","last_fullsized":"2024-03-18T15:15:16.108571+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:40:57.841960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'7","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:46.547283+0000","last_change":"2024-03-18T15:15:10.103183+0000","last_active":"2024-03-18T15:15:46.547283+0000","last_peered":"2024-03-18T15:15:46.547283+0000","last_clean":"2024-03-18T15:15:46.547283+0000","last_became_active":"2024-03-18T15:15:10.101948+0000","last_became_peered":"2024-03-18T15:15:10.101948+0000","last_unstale":"2024-03-18T15:15:46.547283+0000","last_undegraded":"2024-03-18T15:15:46.547283+0000","last_fullsized":"2024-03-18T15:15:46.547283+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:04:51.718433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.023104+0000","last_change":"2024-03-18T15:15:10.666940+0000","last_active":"2024-03-18T15:15:18.023104+0000","last_peered":"2024-03-18T15:15:18.023104+0000","last_clean":"2024-03-18T15:15:18.023104+0000","last_became_active":"2024-03-18T15:15:10.666639+0000","last_became_peered":"2024-03-18T15:15:10.666639+0000","last_unstale":"2024-03-18T15:15:18.023104+0000","last_undegraded":"2024-03-18T15:15:18.023104+0000","last_fullsized":"2024-03-18T15:15:18.023104+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:04:45.945023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:16.415012+0000","last_change":"2024-03-18T15:15:15.120541+0000","last_active":"2024-03-18T15:15:16.415012+0000","last_peered":"2024-03-18T15:15:16.415012+0000","last_clean":"2024-03-18T15:15:16.415012+0000","last_became_active":"2024-03-18T15:15:15.120097+0000","last_became_peered":"2024-03-18T15:15:15.120097+0000","last_unstale":"2024-03-18T15:15:16.415012+0000","last_undegraded":"2024-03-18T15:15:16.415012+0000","last_fullsized":"2024-03-18T15:15:16.415012+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:12:08.932598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:45.464197+0000","last_change":"2024-03-18T15:15:10.079199+0000","last_active":"2024-03-18T15:15:45.464197+0000","last_peered":"2024-03-18T15:15:45.464197+0000","last_clean":"2024-03-18T15:15:45.464197+0000","last_became_active":"2024-03-18T15:15:10.078872+0000","last_became_peered":"2024-03-18T15:15:10.078872+0000","last_unstale":"2024-03-18T15:15:45.464197+0000","last_undegraded":"2024-03-18T15:15:45.464197+0000","last_fullsized":"2024-03-18T15:15:45.464197+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:31:44.722779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.023818+0000","last_change":"2024-03-18T15:15:10.686304+0000","last_active":"2024-03-18T15:15:18.023818+0000","last_peered":"2024-03-18T15:15:18.023818+0000","last_clean":"2024-03-18T15:15:18.023818+0000","last_became_active":"2024-03-18T15:15:10.685636+0000","last_became_peered":"2024-03-18T15:15:10.685636+0000","last_unstale":"2024-03-18T15:15:18.023818+0000","last_undegraded":"2024-03-18T15:15:18.023818+0000","last_fullsized":"2024-03-18T15:15:18.023818+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:31:03.315971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:16.415113+0000","last_change":"2024-03-18T15:15:15.108254+0000","last_active":"2024-03-18T15:15:16.415113+0000","last_peered":"2024-03-18T15:15:16.415113+0000","last_clean":"2024-03-18T15:15:16.415113+0000","last_became_active":"2024-03-18T15:15:15.107806+0000","last_became_peered":"2024-03-18T15:15:15.107806+0000","last_unstale":"2024-03-18T15:15:16.415113+0000","last_undegraded":"2024-03-18T15:15:16.415113+0000","last_fullsized":"2024-03-18T15:15:16.415113+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:55:53.427426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:43.467841+0000","last_change":"2024-03-18T15:15:10.082574+0000","last_active":"2024-03-18T15:15:43.467841+0000","last_peered":"2024-03-18T15:15:43.467841+0000","last_clean":"2024-03-18T15:15:43.467841+0000","last_became_active":"2024-03-18T15:15:10.082252+0000","last_became_peered":"2024-03-18T15:15:10.082252+0000","last_unstale":"2024-03-18T15:15:43.467841+0000","last_undegraded":"2024-03-18T15:15:43.467841+0000","last_fullsized":"2024-03-18T15:15:43.467841+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:25:17.475023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:16.104324+0000","last_change":"2024-03-18T15:15:10.667304+0000","last_active":"2024-03-18T15:15:16.104324+0000","last_peered":"2024-03-18T15:15:16.104324+0000","last_clean":"2024-03-18T15:15:16.104324+0000","last_became_active":"2024-03-18T15:15:10.666932+0000","last_became_peered":"2024-03-18T15:15:10.666932+0000","last_unstale":"2024-03-18T15:15:16.104324+0000","last_undegraded":"2024-03-18T15:15:16.104324+0000","last_fullsized":"2024-03-18T15:15:16.104324+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:22:33.817618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.022728+0000","last_change":"2024-03-18T15:15:15.113499+0000","last_active":"2024-03-18T15:15:18.022728+0000","last_peered":"2024-03-18T15:15:18.022728+0000","last_clean":"2024-03-18T15:15:18.022728+0000","last_became_active":"2024-03-18T15:15:15.113095+0000","last_became_peered":"2024-03-18T15:15:15.113095+0000","last_unstale":"2024-03-18T15:15:18.022728+0000","last_undegraded":"2024-03-18T15:15:18.022728+0000","last_fullsized":"2024-03-18T15:15:18.022728+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:58:28.255634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:45.500316+0000","last_change":"2024-03-18T15:15:10.078129+0000","last_active":"2024-03-18T15:15:45.500316+0000","last_peered":"2024-03-18T15:15:45.500316+0000","last_clean":"2024-03-18T15:15:45.500316+0000","last_became_active":"2024-03-18T15:15:10.076202+0000","last_became_peered":"2024-03-18T15:15:10.076202+0000","last_unstale":"2024-03-18T15:15:45.500316+0000","last_undegraded":"2024-03-18T15:15:45.500316+0000","last_fullsized":"2024-03-18T15:15:45.500316+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:24:58.240937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:16.104721+0000","last_change":"2024-03-18T15:15:10.665036+0000","last_active":"2024-03-18T15:15:16.104721+0000","last_peered":"2024-03-18T15:15:16.104721+0000","last_clean":"2024-03-18T15:15:16.104721+0000","last_became_active":"2024-03-18T15:15:10.664614+0000","last_became_peered":"2024-03-18T15:15:10.664614+0000","last_unstale":"2024-03-18T15:15:16.104721+0000","last_undegraded":"2024-03-18T15:15:16.104721+0000","last_fullsized":"2024-03-18T15:15:16.104721+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:31:42.454317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:17.578249+0000","last_change":"2024-03-18T15:15:15.127812+0000","last_active":"2024-03-18T15:15:17.578249+0000","last_peered":"2024-03-18T15:15:17.578249+0000","last_clean":"2024-03-18T15:15:17.578249+0000","last_became_active":"2024-03-18T15:15:15.127300+0000","last_became_peered":"2024-03-18T15:15:15.127300+0000","last_unstale":"2024-03-18T15:15:17.578249+0000","last_undegraded":"2024-03-18T15:15:17.578249+0000","last_fullsized":"2024-03-18T15:15:17.578249+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:41:25.292884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.024223+0000","last_change":"2024-03-18T15:15:10.093866+0000","last_active":"2024-03-18T15:15:18.024223+0000","last_peered":"2024-03-18T15:15:18.024223+0000","last_clean":"2024-03-18T15:15:18.024223+0000","last_became_active":"2024-03-18T15:15:10.093568+0000","last_became_peered":"2024-03-18T15:15:10.093568+0000","last_unstale":"2024-03-18T15:15:18.024223+0000","last_undegraded":"2024-03-18T15:15:18.024223+0000","last_fullsized":"2024-03-18T15:15:18.024223+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:00:03.533705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:16.104362+0000","last_change":"2024-03-18T15:15:10.679506+0000","last_active":"2024-03-18T15:15:16.104362+0000","last_peered":"2024-03-18T15:15:16.104362+0000","last_clean":"2024-03-18T15:15:16.104362+0000","last_became_active":"2024-03-18T15:15:10.679164+0000","last_became_peered":"2024-03-18T15:15:10.679164+0000","last_unstale":"2024-03-18T15:15:16.104362+0000","last_undegraded":"2024-03-18T15:15:16.104362+0000","last_fullsized":"2024-03-18T15:15:16.104362+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:04:55.266273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.023611+0000","last_change":"2024-03-18T15:15:15.099072+0000","last_active":"2024-03-18T15:15:18.023611+0000","last_peered":"2024-03-18T15:15:18.023611+0000","last_clean":"2024-03-18T15:15:18.023611+0000","last_became_active":"2024-03-18T15:15:15.098165+0000","last_became_peered":"2024-03-18T15:15:15.098165+0000","last_unstale":"2024-03-18T15:15:18.023611+0000","last_undegraded":"2024-03-18T15:15:18.023611+0000","last_fullsized":"2024-03-18T15:15:18.023611+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:47:47.377924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:17.599824+0000","last_change":"2024-03-18T15:15:10.102562+0000","last_active":"2024-03-18T15:15:17.599824+0000","last_peered":"2024-03-18T15:15:17.599824+0000","last_clean":"2024-03-18T15:15:17.599824+0000","last_became_active":"2024-03-18T15:15:10.099318+0000","last_became_peered":"2024-03-18T15:15:10.099318+0000","last_unstale":"2024-03-18T15:15:17.599824+0000","last_undegraded":"2024-03-18T15:15:17.599824+0000","last_fullsized":"2024-03-18T15:15:17.599824+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:55:15.601925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:16.104687+0000","last_change":"2024-03-18T15:15:10.662645+0000","last_active":"2024-03-18T15:15:16.104687+0000","last_peered":"2024-03-18T15:15:16.104687+0000","last_clean":"2024-03-18T15:15:16.104687+0000","last_became_active":"2024-03-18T15:15:10.662280+0000","last_became_peered":"2024-03-18T15:15:10.662280+0000","last_unstale":"2024-03-18T15:15:16.104687+0000","last_undegraded":"2024-03-18T15:15:16.104687+0000","last_fullsized":"2024-03-18T15:15:16.104687+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:11:22.598169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.023182+0000","last_change":"2024-03-18T15:15:10.661839+0000","last_active":"2024-03-18T15:15:18.023182+0000","last_peered":"2024-03-18T15:15:18.023182+0000","last_clean":"2024-03-18T15:15:18.023182+0000","last_became_active":"2024-03-18T15:15:10.661356+0000","last_became_peered":"2024-03-18T15:15:10.661356+0000","last_unstale":"2024-03-18T15:15:18.023182+0000","last_undegraded":"2024-03-18T15:15:18.023182+0000","last_fullsized":"2024-03-18T15:15:18.023182+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:13:33.637281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:16.105099+0000","last_change":"2024-03-18T15:15:10.661946+0000","last_active":"2024-03-18T15:15:16.105099+0000","last_peered":"2024-03-18T15:15:16.105099+0000","last_clean":"2024-03-18T15:15:16.105099+0000","last_became_active":"2024-03-18T15:15:10.661574+0000","last_became_peered":"2024-03-18T15:15:10.661574+0000","last_unstale":"2024-03-18T15:15:16.105099+0000","last_undegraded":"2024-03-18T15:15:16.105099+0000","last_fullsized":"2024-03-18T15:15:16.105099+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:53:50.284156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:16.104978+0000","last_change":"2024-03-18T15:15:10.674064+0000","last_active":"2024-03-18T15:15:16.104978+0000","last_peered":"2024-03-18T15:15:16.104978+0000","last_clean":"2024-03-18T15:15:16.104978+0000","last_became_active":"2024-03-18T15:15:10.673820+0000","last_became_peered":"2024-03-18T15:15:10.673820+0000","last_unstale":"2024-03-18T15:15:16.104978+0000","last_undegraded":"2024-03-18T15:15:16.104978+0000","last_fullsized":"2024-03-18T15:15:16.104978+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:29:28.977824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:17.578512+0000","last_change":"2024-03-18T15:15:10.671276+0000","last_active":"2024-03-18T15:15:17.578512+0000","last_peered":"2024-03-18T15:15:17.578512+0000","last_clean":"2024-03-18T15:15:17.578512+0000","last_became_active":"2024-03-18T15:15:10.670926+0000","last_became_peered":"2024-03-18T15:15:10.670926+0000","last_unstale":"2024-03-18T15:15:17.578512+0000","last_undegraded":"2024-03-18T15:15:17.578512+0000","last_fullsized":"2024-03-18T15:15:17.578512+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:15:43.972815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:17.599478+0000","last_change":"2024-03-18T15:15:10.680165+0000","last_active":"2024-03-18T15:15:17.599478+0000","last_peered":"2024-03-18T15:15:17.599478+0000","last_clean":"2024-03-18T15:15:17.599478+0000","last_became_active":"2024-03-18T15:15:10.679849+0000","last_became_peered":"2024-03-18T15:15:10.679849+0000","last_unstale":"2024-03-18T15:15:17.599478+0000","last_undegraded":"2024-03-18T15:15:17.599478+0000","last_fullsized":"2024-03-18T15:15:17.599478+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:15:00.352175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.023964+0000","last_change":"2024-03-18T15:15:10.658966+0000","last_active":"2024-03-18T15:15:18.023964+0000","last_peered":"2024-03-18T15:15:18.023964+0000","last_clean":"2024-03-18T15:15:18.023964+0000","last_became_active":"2024-03-18T15:15:10.658654+0000","last_became_peered":"2024-03-18T15:15:10.658654+0000","last_unstale":"2024-03-18T15:15:18.023964+0000","last_undegraded":"2024-03-18T15:15:18.023964+0000","last_fullsized":"2024-03-18T15:15:18.023964+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:42:25.730735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:16.105653+0000","last_change":"2024-03-18T15:15:10.665694+0000","last_active":"2024-03-18T15:15:16.105653+0000","last_peered":"2024-03-18T15:15:16.105653+0000","last_clean":"2024-03-18T15:15:16.105653+0000","last_became_active":"2024-03-18T15:15:10.664688+0000","last_became_peered":"2024-03-18T15:15:10.664688+0000","last_unstale":"2024-03-18T15:15:16.105653+0000","last_undegraded":"2024-03-18T15:15:16.105653+0000","last_fullsized":"2024-03-18T15:15:16.105653+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:32:23.108060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.024267+0000","last_change":"2024-03-18T15:15:10.676129+0000","last_active":"2024-03-18T15:15:18.024267+0000","last_peered":"2024-03-18T15:15:18.024267+0000","last_clean":"2024-03-18T15:15:18.024267+0000","last_became_active":"2024-03-18T15:15:10.675819+0000","last_became_peered":"2024-03-18T15:15:10.675819+0000","last_unstale":"2024-03-18T15:15:18.024267+0000","last_undegraded":"2024-03-18T15:15:18.024267+0000","last_fullsized":"2024-03-18T15:15:18.024267+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:35:28.578802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.023850+0000","last_change":"2024-03-18T15:15:10.668833+0000","last_active":"2024-03-18T15:15:18.023850+0000","last_peered":"2024-03-18T15:15:18.023850+0000","last_clean":"2024-03-18T15:15:18.023850+0000","last_became_active":"2024-03-18T15:15:10.668572+0000","last_became_peered":"2024-03-18T15:15:10.668572+0000","last_unstale":"2024-03-18T15:15:18.023850+0000","last_undegraded":"2024-03-18T15:15:18.023850+0000","last_fullsized":"2024-03-18T15:15:18.023850+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:12:44.333515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.023703+0000","last_change":"2024-03-18T15:15:10.677496+0000","last_active":"2024-03-18T15:15:18.023703+0000","last_peered":"2024-03-18T15:15:18.023703+0000","last_clean":"2024-03-18T15:15:18.023703+0000","last_became_active":"2024-03-18T15:15:10.674869+0000","last_became_peered":"2024-03-18T15:15:10.674869+0000","last_unstale":"2024-03-18T15:15:18.023703+0000","last_undegraded":"2024-03-18T15:15:18.023703+0000","last_fullsized":"2024-03-18T15:15:18.023703+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:03:10.531488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.022863+0000","last_change":"2024-03-18T15:15:10.673488+0000","last_active":"2024-03-18T15:15:18.022863+0000","last_peered":"2024-03-18T15:15:18.022863+0000","last_clean":"2024-03-18T15:15:18.022863+0000","last_became_active":"2024-03-18T15:15:10.673117+0000","last_became_peered":"2024-03-18T15:15:10.673117+0000","last_unstale":"2024-03-18T15:15:18.022863+0000","last_undegraded":"2024-03-18T15:15:18.022863+0000","last_fullsized":"2024-03-18T15:15:18.022863+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:29:22.823829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:17.599210+0000","last_change":"2024-03-18T15:15:10.678916+0000","last_active":"2024-03-18T15:15:17.599210+0000","last_peered":"2024-03-18T15:15:17.599210+0000","last_clean":"2024-03-18T15:15:17.599210+0000","last_became_active":"2024-03-18T15:15:10.678545+0000","last_became_peered":"2024-03-18T15:15:10.678545+0000","last_unstale":"2024-03-18T15:15:17.599210+0000","last_undegraded":"2024-03-18T15:15:17.599210+0000","last_fullsized":"2024-03-18T15:15:17.599210+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:13:52.714537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.023710+0000","last_change":"2024-03-18T15:15:10.660397+0000","last_active":"2024-03-18T15:15:18.023710+0000","last_peered":"2024-03-18T15:15:18.023710+0000","last_clean":"2024-03-18T15:15:18.023710+0000","last_became_active":"2024-03-18T15:15:10.660085+0000","last_became_peered":"2024-03-18T15:15:10.660085+0000","last_unstale":"2024-03-18T15:15:18.023710+0000","last_undegraded":"2024-03-18T15:15:18.023710+0000","last_fullsized":"2024-03-18T15:15:18.023710+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:13:04.591062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3542208,"data_compressed":210616,"data_compressed_allocated":270336,"data_compressed_original":3506176,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":27395,"num_objects":70,"num_object_clones":0,"num_object_copies":210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70,"num_whiteouts":0,"num_read":6,"num_read_kb":26,"num_write":161,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":47,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":88804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":141,"ondisk_log_size":141,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383701,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":978944,"data_stored":482503,"data_compressed":14671,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645337,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28064,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717088,"statfs":{"total":95995035648,"available":95966298112,"internally_reserved":0,"allocated":1073152,"data_stored":1079727,"data_compressed":50322,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808862,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32136,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713016,"statfs":{"total":95995035648,"available":95962128384,"internally_reserved":0,"allocated":983040,"data_stored":478033,"data_compressed":14671,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070502,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32136,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713016,"statfs":{"total":95995035648,"available":95962128384,"internally_reserved":0,"allocated":983040,"data_stored":477953,"data_compressed":14671,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332138,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28020,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717132,"statfs":{"total":95995035648,"available":95966343168,"internally_reserved":0,"allocated":1028096,"data_stored":1069375,"data_compressed":50322,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495664,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32116,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713036,"statfs":{"total":95995035648,"available":95962148864,"internally_reserved":0,"allocated":962560,"data_stored":477215,"data_compressed":14671,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574925,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32160,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712992,"statfs":{"total":95995035648,"available":95962103808,"internally_reserved":0,"allocated":1007616,"data_stored":502108,"data_compressed":14671,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738450,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":1003520,"data_stored":479393,"data_compressed":14671,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411400,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1015808,"data_stored":489629,"data_compressed":14671,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182466,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":991232,"data_stored":477645,"data_compressed":14671,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920829,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32128,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713024,"statfs":{"total":95995035648,"available":95962136576,"internally_reserved":0,"allocated":974848,"data_stored":477859,"data_compressed":14671,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757302,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28020,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717132,"statfs":{"total":95995035648,"available":95966343168,"internally_reserved":0,"allocated":1024000,"data_stored":1067765,"data_compressed":50322,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34006,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34006,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35651,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35651,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35651,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35651,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":42223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":12414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":24877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T15:15:49.512 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T15:15:49.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:49 smithi033 ceph-mon[36648]: pgmap v336: 129 pgs: 129 active+clean; 603 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4.2 KiB/s wr, 9 op/s 2024-03-18T15:15:49.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:49 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/1191952242' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:15:49.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:49 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/2278289990' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:15:49.757 INFO:journalctl@ceph.mds.b.smithi033.stdout:Mar 18 15:15:49 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-b[67599]: 2024-03-18T15:15:49.380+0000 7f90acfba700 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2024-03-18T15:15:49.782 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:49.781+0000 7f3999a00640 1 -- 172.21.15.19:0/3143817649 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f39941055b0 msgr2=0x7f399410be40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:49.782 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:49.781+0000 7f3999a00640 1 --2- 172.21.15.19:0/3143817649 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f39941055b0 0x7f399410be40 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f398c009860 tx=0x7f398c02bb70 comp rx=0 tx=0).stop 2024-03-18T15:15:49.782 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:49.782+0000 7f3999a00640 1 -- 172.21.15.19:0/3143817649 shutdown_connections 2024-03-18T15:15:49.782 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:49.782+0000 7f3999a00640 1 --2- 172.21.15.19:0/3143817649 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f39941055b0 0x7f399410be40 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:49.783 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:49.782+0000 7f3999a00640 1 --2- 172.21.15.19:0/3143817649 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3994104bf0 0x7f3994105070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:49.783 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:49.782+0000 7f3999a00640 1 --2- 172.21.15.19:0/3143817649 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3994103a00 0x7f3994103e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:49.783 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:49.782+0000 7f3999a00640 1 -- 172.21.15.19:0/3143817649 >> 172.21.15.19:0/3143817649 conn(0x7f39940fef90 msgr2=0x7f39941013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:49.783 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:49.782+0000 7f3999a00640 1 -- 172.21.15.19:0/3143817649 shutdown_connections 2024-03-18T15:15:49.783 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:49.782+0000 7f3999a00640 1 -- 172.21.15.19:0/3143817649 wait complete. 2024-03-18T15:15:49.784 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:49.783+0000 7f3999a00640 1 Processor -- start 2024-03-18T15:15:49.784 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:49.783+0000 7f3999a00640 1 -- start start 2024-03-18T15:15:49.784 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:49.784+0000 7f3999a00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3994103a00 0x7f3994199c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:49.784 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:49.784+0000 7f3999a00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3994104bf0 0x7f399419a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:49.785 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:49.784+0000 7f3999a00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f39941055b0 0x7f39941a1210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:49.785 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:49.784+0000 7f3999a00640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f399410f660 con 0x7f3994104bf0 2024-03-18T15:15:49.785 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:49.784+0000 7f3999a00640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f399410f4e0 con 0x7f39941055b0 2024-03-18T15:15:49.785 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:49.784+0000 7f3999a00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f399410f7e0 con 0x7f3994103a00 2024-03-18T15:15:49.785 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:49.784+0000 7f3992a00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3994103a00 0x7f3994199c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:49.785 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:49.784+0000 7f3992a00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3994103a00 0x7f3994199c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.19:36536/0 (socket says 172.21.15.19:36536) 2024-03-18T15:15:49.785 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:49.784+0000 7f3993400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f39941055b0 0x7f39941a1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:49.785 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:49.784+0000 7f3992a00640 1 -- 172.21.15.19:0/3232804118 learned_addr learned my addr 172.21.15.19:0/3232804118 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:15:49.785 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:49.784+0000 7f3992000640 1 --2- 172.21.15.19:0/3232804118 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3994104bf0 0x7f399419a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:49.785 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:49.784+0000 7f3992a00640 1 -- 172.21.15.19:0/3232804118 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f39941055b0 msgr2=0x7f39941a1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:49.785 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:49.784+0000 7f3992a00640 1 --2- 172.21.15.19:0/3232804118 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f39941055b0 0x7f39941a1210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:49.785 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:49.784+0000 7f3992a00640 1 -- 172.21.15.19:0/3232804118 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3994104bf0 msgr2=0x7f399419a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:49.785 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:49.784+0000 7f3992a00640 1 --2- 172.21.15.19:0/3232804118 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3994104bf0 0x7f399419a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:49.785 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:49.784+0000 7f3992a00640 1 -- 172.21.15.19:0/3232804118 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39941a18b0 con 0x7f3994103a00 2024-03-18T15:15:49.785 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:49.784+0000 7f3993400640 1 --2- 172.21.15.19:0/3232804118 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f39941055b0 0x7f39941a1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:15:49.786 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:49.785+0000 7f3992a00640 1 --2- 172.21.15.19:0/3232804118 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3994103a00 0x7f3994199c70 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f398400e9c0 tx=0x7f398400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:49.786 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:49.785+0000 7f398b400640 1 -- 172.21.15.19:0/3232804118 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f398400cd80 con 0x7f3994103a00 2024-03-18T15:15:49.786 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:49.785+0000 7f398b400640 1 -- 172.21.15.19:0/3232804118 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f39840035f0 con 0x7f3994103a00 2024-03-18T15:15:49.787 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:49.785+0000 7f398b400640 1 -- 172.21.15.19:0/3232804118 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3984010690 con 0x7f3994103a00 2024-03-18T15:15:49.787 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:49.785+0000 7f3999a00640 1 -- 172.21.15.19:0/3232804118 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39941a1ba0 con 0x7f3994103a00 2024-03-18T15:15:49.787 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:49.785+0000 7f3999a00640 1 -- 172.21.15.19:0/3232804118 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39941a20a0 con 0x7f3994103a00 2024-03-18T15:15:49.787 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:49.786+0000 7f3999a00640 1 -- 172.21.15.19:0/3232804118 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f399410ad20 con 0x7f3994103a00 2024-03-18T15:15:49.788 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:49.788+0000 7f398b400640 1 -- 172.21.15.19:0/3232804118 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f39840037a0 con 0x7f3994103a00 2024-03-18T15:15:49.789 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:49.789+0000 7f398b400640 1 --2- 172.21.15.19:0/3232804118 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f39780a2910 0x7f39780a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:49.789 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:49.789+0000 7f398b400640 1 -- 172.21.15.19:0/3232804118 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f39840cf270 con 0x7f3994103a00 2024-03-18T15:15:49.789 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:49.790+0000 7f3992000640 1 --2- 172.21.15.19:0/3232804118 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f39780a2910 0x7f39780a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:49.790 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:49.790+0000 7f3992000640 1 --2- 172.21.15.19:0/3232804118 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f39780a2910 0x7f39780a4dd0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f399419b170 tx=0x7f397c0095c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:49.793 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:49.793+0000 7f398b400640 1 -- 172.21.15.19:0/3232804118 <== mon.1 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f398409bf10 con 0x7f3994103a00 2024-03-18T15:15:49.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:49 smithi104 ceph-mon[36070]: pgmap v336: 129 pgs: 129 active+clean; 603 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4.2 KiB/s wr, 9 op/s 2024-03-18T15:15:49.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:49 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/1191952242' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:15:49.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:49 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/2278289990' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:15:49.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:49 smithi019 ceph-mon[31522]: pgmap v336: 129 pgs: 129 active+clean; 603 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4.2 KiB/s wr, 9 op/s 2024-03-18T15:15:49.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:49 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/1191952242' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:15:49.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:49 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/2278289990' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:15:49.994 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:49.994+0000 7f3999a00640 1 -- 172.21.15.19:0/3232804118 --> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f3994061c20 con 0x7f39780a2910 2024-03-18T15:15:50.023 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:50.023+0000 7f398b400640 1 -- 172.21.15.19:0/3232804118 <== mgr.14152 v2:172.21.15.19:6800/2017714236 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340596 (secure 0 0 0) 0x7f3994061c20 con 0x7f39780a2910 2024-03-18T15:15:50.028 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-18T15:15:50.030 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-03-18T15:15:50.032 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:50.029+0000 7f3999a00640 1 -- 172.21.15.19:0/3232804118 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f39780a2910 msgr2=0x7f39780a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:50.032 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:50.029+0000 7f3999a00640 1 --2- 172.21.15.19:0/3232804118 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f39780a2910 0x7f39780a4dd0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f399419b170 tx=0x7f397c0095c0 comp rx=0 tx=0).stop 2024-03-18T15:15:50.032 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:50.029+0000 7f3999a00640 1 -- 172.21.15.19:0/3232804118 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3994103a00 msgr2=0x7f3994199c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:50.032 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:50.029+0000 7f3999a00640 1 --2- 172.21.15.19:0/3232804118 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3994103a00 0x7f3994199c70 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f398400e9c0 tx=0x7f398400ee80 comp rx=0 tx=0).stop 2024-03-18T15:15:50.032 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:50.029+0000 7f3999a00640 1 -- 172.21.15.19:0/3232804118 shutdown_connections 2024-03-18T15:15:50.032 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:50.029+0000 7f3999a00640 1 --2- 172.21.15.19:0/3232804118 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f39780a2910 0x7f39780a4dd0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:50.032 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:50.029+0000 7f3999a00640 1 --2- 172.21.15.19:0/3232804118 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f39941055b0 0x7f39941a1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:50.033 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:50.029+0000 7f3999a00640 1 --2- 172.21.15.19:0/3232804118 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3994104bf0 0x7f399419a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:50.033 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:50.029+0000 7f3999a00640 1 --2- 172.21.15.19:0/3232804118 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3994103a00 0x7f3994199c70 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:50.033 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:50.029+0000 7f3999a00640 1 -- 172.21.15.19:0/3232804118 >> 172.21.15.19:0/3232804118 conn(0x7f39940fef90 msgr2=0x7f3994107ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:50.033 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:50.030+0000 7f3999a00640 1 -- 172.21.15.19:0/3232804118 shutdown_connections 2024-03-18T15:15:50.033 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:50.030+0000 7f3999a00640 1 -- 172.21.15.19:0/3232804118 wait complete. 2024-03-18T15:15:50.049 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":336,"stamp":"2024-03-18T15:15:48.309522+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":617763,"num_objects":72,"num_object_clones":0,"num_object_copies":216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72,"num_whiteouts":0,"num_read":142,"num_read_kb":145,"num_write":342,"num_write_kb":3007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":47,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":278,"ondisk_log_size":278,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":356740,"kb_used_data":11744,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124585084,"statfs":{"total":1151940427776,"available":1151575126016,"internally_reserved":0,"allocated":12025856,"data_stored":7559205,"data_compressed":283005,"data_compressed_allocated":380928,"data_compressed_original":2260992,"omap_allocated":19143,"internal_metadata":353154361},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22896,"num_objects":48,"num_object_clones":0,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":6,"num_read_kb":26,"num_write":113,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":33,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006416"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.024181+0000","last_change":"2024-03-18T15:15:10.682660+0000","last_active":"2024-03-18T15:15:18.024181+0000","last_peered":"2024-03-18T15:15:18.024181+0000","last_clean":"2024-03-18T15:15:18.024181+0000","last_became_active":"2024-03-18T15:15:10.682433+0000","last_became_peered":"2024-03-18T15:15:10.682433+0000","last_unstale":"2024-03-18T15:15:18.024181+0000","last_undegraded":"2024-03-18T15:15:18.024181+0000","last_fullsized":"2024-03-18T15:15:18.024181+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:48:55.154555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:17.578802+0000","last_change":"2024-03-18T15:15:10.678100+0000","last_active":"2024-03-18T15:15:17.578802+0000","last_peered":"2024-03-18T15:15:17.578802+0000","last_clean":"2024-03-18T15:15:17.578802+0000","last_became_active":"2024-03-18T15:15:10.677543+0000","last_became_peered":"2024-03-18T15:15:10.677543+0000","last_unstale":"2024-03-18T15:15:17.578802+0000","last_undegraded":"2024-03-18T15:15:17.578802+0000","last_fullsized":"2024-03-18T15:15:17.578802+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:57:50.087837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:17.578411+0000","last_change":"2024-03-18T15:15:10.674672+0000","last_active":"2024-03-18T15:15:17.578411+0000","last_peered":"2024-03-18T15:15:17.578411+0000","last_clean":"2024-03-18T15:15:17.578411+0000","last_became_active":"2024-03-18T15:15:10.674346+0000","last_became_peered":"2024-03-18T15:15:10.674346+0000","last_unstale":"2024-03-18T15:15:17.578411+0000","last_undegraded":"2024-03-18T15:15:17.578411+0000","last_fullsized":"2024-03-18T15:15:17.578411+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:21:40.054636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:16.415207+0000","last_change":"2024-03-18T15:15:10.660022+0000","last_active":"2024-03-18T15:15:16.415207+0000","last_peered":"2024-03-18T15:15:16.415207+0000","last_clean":"2024-03-18T15:15:16.415207+0000","last_became_active":"2024-03-18T15:15:10.659622+0000","last_became_peered":"2024-03-18T15:15:10.659622+0000","last_unstale":"2024-03-18T15:15:16.415207+0000","last_undegraded":"2024-03-18T15:15:16.415207+0000","last_fullsized":"2024-03-18T15:15:16.415207+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:55:33.777539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.024343+0000","last_change":"2024-03-18T15:15:10.681499+0000","last_active":"2024-03-18T15:15:18.024343+0000","last_peered":"2024-03-18T15:15:18.024343+0000","last_clean":"2024-03-18T15:15:18.024343+0000","last_became_active":"2024-03-18T15:15:10.681034+0000","last_became_peered":"2024-03-18T15:15:10.681034+0000","last_unstale":"2024-03-18T15:15:18.024343+0000","last_undegraded":"2024-03-18T15:15:18.024343+0000","last_fullsized":"2024-03-18T15:15:18.024343+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:16:43.171924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.023583+0000","last_change":"2024-03-18T15:15:10.669968+0000","last_active":"2024-03-18T15:15:18.023583+0000","last_peered":"2024-03-18T15:15:18.023583+0000","last_clean":"2024-03-18T15:15:18.023583+0000","last_became_active":"2024-03-18T15:15:10.668477+0000","last_became_peered":"2024-03-18T15:15:10.668477+0000","last_unstale":"2024-03-18T15:15:18.023583+0000","last_undegraded":"2024-03-18T15:15:18.023583+0000","last_fullsized":"2024-03-18T15:15:18.023583+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:14:46.364453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:16.104236+0000","last_change":"2024-03-18T15:15:10.669433+0000","last_active":"2024-03-18T15:15:16.104236+0000","last_peered":"2024-03-18T15:15:16.104236+0000","last_clean":"2024-03-18T15:15:16.104236+0000","last_became_active":"2024-03-18T15:15:10.669099+0000","last_became_peered":"2024-03-18T15:15:10.669099+0000","last_unstale":"2024-03-18T15:15:16.104236+0000","last_undegraded":"2024-03-18T15:15:16.104236+0000","last_fullsized":"2024-03-18T15:15:16.104236+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:37:20.697605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.023327+0000","last_change":"2024-03-18T15:15:10.681679+0000","last_active":"2024-03-18T15:15:18.023327+0000","last_peered":"2024-03-18T15:15:18.023327+0000","last_clean":"2024-03-18T15:15:18.023327+0000","last_became_active":"2024-03-18T15:15:10.681079+0000","last_became_peered":"2024-03-18T15:15:10.681079+0000","last_unstale":"2024-03-18T15:15:18.023327+0000","last_undegraded":"2024-03-18T15:15:18.023327+0000","last_fullsized":"2024-03-18T15:15:18.023327+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:24:39.639451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.023248+0000","last_change":"2024-03-18T15:15:10.664111+0000","last_active":"2024-03-18T15:15:18.023248+0000","last_peered":"2024-03-18T15:15:18.023248+0000","last_clean":"2024-03-18T15:15:18.023248+0000","last_became_active":"2024-03-18T15:15:10.663482+0000","last_became_peered":"2024-03-18T15:15:10.663482+0000","last_unstale":"2024-03-18T15:15:18.023248+0000","last_undegraded":"2024-03-18T15:15:18.023248+0000","last_fullsized":"2024-03-18T15:15:18.023248+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:49:50.810014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.023620+0000","last_change":"2024-03-18T15:15:10.657799+0000","last_active":"2024-03-18T15:15:18.023620+0000","last_peered":"2024-03-18T15:15:18.023620+0000","last_clean":"2024-03-18T15:15:18.023620+0000","last_became_active":"2024-03-18T15:15:10.657468+0000","last_became_peered":"2024-03-18T15:15:10.657468+0000","last_unstale":"2024-03-18T15:15:18.023620+0000","last_undegraded":"2024-03-18T15:15:18.023620+0000","last_fullsized":"2024-03-18T15:15:18.023620+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:22:46.025516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:16.104690+0000","last_change":"2024-03-18T15:15:10.665040+0000","last_active":"2024-03-18T15:15:16.104690+0000","last_peered":"2024-03-18T15:15:16.104690+0000","last_clean":"2024-03-18T15:15:16.104690+0000","last_became_active":"2024-03-18T15:15:10.663929+0000","last_became_peered":"2024-03-18T15:15:10.663929+0000","last_unstale":"2024-03-18T15:15:16.104690+0000","last_undegraded":"2024-03-18T15:15:16.104690+0000","last_fullsized":"2024-03-18T15:15:16.104690+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:05:29.954918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:16.415185+0000","last_change":"2024-03-18T15:15:10.662301+0000","last_active":"2024-03-18T15:15:16.415185+0000","last_peered":"2024-03-18T15:15:16.415185+0000","last_clean":"2024-03-18T15:15:16.415185+0000","last_became_active":"2024-03-18T15:15:10.661956+0000","last_became_peered":"2024-03-18T15:15:10.661956+0000","last_unstale":"2024-03-18T15:15:16.415185+0000","last_undegraded":"2024-03-18T15:15:16.415185+0000","last_fullsized":"2024-03-18T15:15:16.415185+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:30:36.699858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:17.599156+0000","last_change":"2024-03-18T15:15:10.650795+0000","last_active":"2024-03-18T15:15:17.599156+0000","last_peered":"2024-03-18T15:15:17.599156+0000","last_clean":"2024-03-18T15:15:17.599156+0000","last_became_active":"2024-03-18T15:15:10.649499+0000","last_became_peered":"2024-03-18T15:15:10.649499+0000","last_unstale":"2024-03-18T15:15:17.599156+0000","last_undegraded":"2024-03-18T15:15:17.599156+0000","last_fullsized":"2024-03-18T15:15:17.599156+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:30:19.094619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:17.577874+0000","last_change":"2024-03-18T15:15:10.677030+0000","last_active":"2024-03-18T15:15:17.577874+0000","last_peered":"2024-03-18T15:15:17.577874+0000","last_clean":"2024-03-18T15:15:17.577874+0000","last_became_active":"2024-03-18T15:15:10.676680+0000","last_became_peered":"2024-03-18T15:15:10.676680+0000","last_unstale":"2024-03-18T15:15:17.577874+0000","last_undegraded":"2024-03-18T15:15:17.577874+0000","last_fullsized":"2024-03-18T15:15:17.577874+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:32:16.382003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:17.578068+0000","last_change":"2024-03-18T15:15:10.677512+0000","last_active":"2024-03-18T15:15:17.578068+0000","last_peered":"2024-03-18T15:15:17.578068+0000","last_clean":"2024-03-18T15:15:17.578068+0000","last_became_active":"2024-03-18T15:15:10.670557+0000","last_became_peered":"2024-03-18T15:15:10.670557+0000","last_unstale":"2024-03-18T15:15:17.578068+0000","last_undegraded":"2024-03-18T15:15:17.578068+0000","last_fullsized":"2024-03-18T15:15:17.578068+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:06:58.330728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.022783+0000","last_change":"2024-03-18T15:15:10.675859+0000","last_active":"2024-03-18T15:15:18.022783+0000","last_peered":"2024-03-18T15:15:18.022783+0000","last_clean":"2024-03-18T15:15:18.022783+0000","last_became_active":"2024-03-18T15:15:10.675458+0000","last_became_peered":"2024-03-18T15:15:10.675458+0000","last_unstale":"2024-03-18T15:15:18.022783+0000","last_undegraded":"2024-03-18T15:15:18.022783+0000","last_fullsized":"2024-03-18T15:15:18.022783+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:07:43.306884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.024088+0000","last_change":"2024-03-18T15:15:10.677093+0000","last_active":"2024-03-18T15:15:18.024088+0000","last_peered":"2024-03-18T15:15:18.024088+0000","last_clean":"2024-03-18T15:15:18.024088+0000","last_became_active":"2024-03-18T15:15:10.676878+0000","last_became_peered":"2024-03-18T15:15:10.676878+0000","last_unstale":"2024-03-18T15:15:18.024088+0000","last_undegraded":"2024-03-18T15:15:18.024088+0000","last_fullsized":"2024-03-18T15:15:18.024088+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:18:23.517133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:16.108248+0000","last_change":"2024-03-18T15:15:10.674793+0000","last_active":"2024-03-18T15:15:16.108248+0000","last_peered":"2024-03-18T15:15:16.108248+0000","last_clean":"2024-03-18T15:15:16.108248+0000","last_became_active":"2024-03-18T15:15:10.674433+0000","last_became_peered":"2024-03-18T15:15:10.674433+0000","last_unstale":"2024-03-18T15:15:16.108248+0000","last_undegraded":"2024-03-18T15:15:16.108248+0000","last_fullsized":"2024-03-18T15:15:16.108248+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:17:24.774598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:16.105553+0000","last_change":"2024-03-18T15:15:10.670757+0000","last_active":"2024-03-18T15:15:16.105553+0000","last_peered":"2024-03-18T15:15:16.105553+0000","last_clean":"2024-03-18T15:15:16.105553+0000","last_became_active":"2024-03-18T15:15:10.670483+0000","last_became_peered":"2024-03-18T15:15:10.670483+0000","last_unstale":"2024-03-18T15:15:16.105553+0000","last_undegraded":"2024-03-18T15:15:16.105553+0000","last_fullsized":"2024-03-18T15:15:16.105553+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:59:20.070597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:16.105358+0000","last_change":"2024-03-18T15:15:15.115252+0000","last_active":"2024-03-18T15:15:16.105358+0000","last_peered":"2024-03-18T15:15:16.105358+0000","last_clean":"2024-03-18T15:15:16.105358+0000","last_became_active":"2024-03-18T15:15:15.114812+0000","last_became_peered":"2024-03-18T15:15:15.114812+0000","last_unstale":"2024-03-18T15:15:16.105358+0000","last_undegraded":"2024-03-18T15:15:16.105358+0000","last_fullsized":"2024-03-18T15:15:16.105358+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:04:15.657483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'9","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:44.495561+0000","last_change":"2024-03-18T15:15:10.092873+0000","last_active":"2024-03-18T15:15:44.495561+0000","last_peered":"2024-03-18T15:15:44.495561+0000","last_clean":"2024-03-18T15:15:44.495561+0000","last_became_active":"2024-03-18T15:15:10.092456+0000","last_became_peered":"2024-03-18T15:15:10.092456+0000","last_unstale":"2024-03-18T15:15:44.495561+0000","last_undegraded":"2024-03-18T15:15:44.495561+0000","last_fullsized":"2024-03-18T15:15:44.495561+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:01:53.596264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.024152+0000","last_change":"2024-03-18T15:15:10.682119+0000","last_active":"2024-03-18T15:15:18.024152+0000","last_peered":"2024-03-18T15:15:18.024152+0000","last_clean":"2024-03-18T15:15:18.024152+0000","last_became_active":"2024-03-18T15:15:10.680489+0000","last_became_peered":"2024-03-18T15:15:10.680489+0000","last_unstale":"2024-03-18T15:15:18.024152+0000","last_undegraded":"2024-03-18T15:15:18.024152+0000","last_fullsized":"2024-03-18T15:15:18.024152+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:52:13.477107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:16.414854+0000","last_change":"2024-03-18T15:15:15.114645+0000","last_active":"2024-03-18T15:15:16.414854+0000","last_peered":"2024-03-18T15:15:16.414854+0000","last_clean":"2024-03-18T15:15:16.414854+0000","last_became_active":"2024-03-18T15:15:15.114245+0000","last_became_peered":"2024-03-18T15:15:15.114245+0000","last_unstale":"2024-03-18T15:15:16.414854+0000","last_undegraded":"2024-03-18T15:15:16.414854+0000","last_fullsized":"2024-03-18T15:15:16.414854+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:08:31.429216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.024632+0000","last_change":"2024-03-18T15:15:10.100265+0000","last_active":"2024-03-18T15:15:18.024632+0000","last_peered":"2024-03-18T15:15:18.024632+0000","last_clean":"2024-03-18T15:15:18.024632+0000","last_became_active":"2024-03-18T15:15:10.099861+0000","last_became_peered":"2024-03-18T15:15:10.099861+0000","last_unstale":"2024-03-18T15:15:18.024632+0000","last_undegraded":"2024-03-18T15:15:18.024632+0000","last_fullsized":"2024-03-18T15:15:18.024632+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:52:47.863313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:17.579240+0000","last_change":"2024-03-18T15:15:10.682009+0000","last_active":"2024-03-18T15:15:17.579240+0000","last_peered":"2024-03-18T15:15:17.579240+0000","last_clean":"2024-03-18T15:15:17.579240+0000","last_became_active":"2024-03-18T15:15:10.680026+0000","last_became_peered":"2024-03-18T15:15:10.680026+0000","last_unstale":"2024-03-18T15:15:17.579240+0000","last_undegraded":"2024-03-18T15:15:17.579240+0000","last_fullsized":"2024-03-18T15:15:17.579240+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:00:27.869376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:16.415059+0000","last_change":"2024-03-18T15:15:15.124914+0000","last_active":"2024-03-18T15:15:16.415059+0000","last_peered":"2024-03-18T15:15:16.415059+0000","last_clean":"2024-03-18T15:15:16.415059+0000","last_became_active":"2024-03-18T15:15:15.124477+0000","last_became_peered":"2024-03-18T15:15:15.124477+0000","last_unstale":"2024-03-18T15:15:16.415059+0000","last_undegraded":"2024-03-18T15:15:16.415059+0000","last_fullsized":"2024-03-18T15:15:16.415059+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:58:40.596613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'7","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:45.495439+0000","last_change":"2024-03-18T15:15:10.083718+0000","last_active":"2024-03-18T15:15:45.495439+0000","last_peered":"2024-03-18T15:15:45.495439+0000","last_clean":"2024-03-18T15:15:45.495439+0000","last_became_active":"2024-03-18T15:15:10.083350+0000","last_became_peered":"2024-03-18T15:15:10.083350+0000","last_unstale":"2024-03-18T15:15:45.495439+0000","last_undegraded":"2024-03-18T15:15:45.495439+0000","last_fullsized":"2024-03-18T15:15:45.495439+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:59:52.912657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.023026+0000","last_change":"2024-03-18T15:15:10.669514+0000","last_active":"2024-03-18T15:15:18.023026+0000","last_peered":"2024-03-18T15:15:18.023026+0000","last_clean":"2024-03-18T15:15:18.023026+0000","last_became_active":"2024-03-18T15:15:10.669154+0000","last_became_peered":"2024-03-18T15:15:10.669154+0000","last_unstale":"2024-03-18T15:15:18.023026+0000","last_undegraded":"2024-03-18T15:15:18.023026+0000","last_fullsized":"2024-03-18T15:15:18.023026+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:10:49.068869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:16.414871+0000","last_change":"2024-03-18T15:15:15.112079+0000","last_active":"2024-03-18T15:15:16.414871+0000","last_peered":"2024-03-18T15:15:16.414871+0000","last_clean":"2024-03-18T15:15:16.414871+0000","last_became_active":"2024-03-18T15:15:15.111652+0000","last_became_peered":"2024-03-18T15:15:15.111652+0000","last_unstale":"2024-03-18T15:15:16.414871+0000","last_undegraded":"2024-03-18T15:15:16.414871+0000","last_fullsized":"2024-03-18T15:15:16.414871+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:15:55.542490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.025108+0000","last_change":"2024-03-18T15:15:10.077975+0000","last_active":"2024-03-18T15:15:18.025108+0000","last_peered":"2024-03-18T15:15:18.025108+0000","last_clean":"2024-03-18T15:15:18.025108+0000","last_became_active":"2024-03-18T15:15:10.077583+0000","last_became_peered":"2024-03-18T15:15:10.077583+0000","last_unstale":"2024-03-18T15:15:18.025108+0000","last_undegraded":"2024-03-18T15:15:18.025108+0000","last_fullsized":"2024-03-18T15:15:18.025108+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:41:43.741623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.024106+0000","last_change":"2024-03-18T15:15:10.665107+0000","last_active":"2024-03-18T15:15:18.024106+0000","last_peered":"2024-03-18T15:15:18.024106+0000","last_clean":"2024-03-18T15:15:18.024106+0000","last_became_active":"2024-03-18T15:15:10.664786+0000","last_became_peered":"2024-03-18T15:15:10.664786+0000","last_unstale":"2024-03-18T15:15:18.024106+0000","last_undegraded":"2024-03-18T15:15:18.024106+0000","last_fullsized":"2024-03-18T15:15:18.024106+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:58:40.439564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:16.105527+0000","last_change":"2024-03-18T15:15:15.100480+0000","last_active":"2024-03-18T15:15:16.105527+0000","last_peered":"2024-03-18T15:15:16.105527+0000","last_clean":"2024-03-18T15:15:16.105527+0000","last_became_active":"2024-03-18T15:15:15.100015+0000","last_became_peered":"2024-03-18T15:15:15.100015+0000","last_unstale":"2024-03-18T15:15:16.105527+0000","last_undegraded":"2024-03-18T15:15:16.105527+0000","last_fullsized":"2024-03-18T15:15:16.105527+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:50:08.723714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:43.469397+0000","last_change":"2024-03-18T15:15:10.082592+0000","last_active":"2024-03-18T15:15:43.469397+0000","last_peered":"2024-03-18T15:15:43.469397+0000","last_clean":"2024-03-18T15:15:43.469397+0000","last_became_active":"2024-03-18T15:15:10.082269+0000","last_became_peered":"2024-03-18T15:15:10.082269+0000","last_unstale":"2024-03-18T15:15:43.469397+0000","last_undegraded":"2024-03-18T15:15:43.469397+0000","last_fullsized":"2024-03-18T15:15:43.469397+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:18:57.751331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.024033+0000","last_change":"2024-03-18T15:15:10.667803+0000","last_active":"2024-03-18T15:15:18.024033+0000","last_peered":"2024-03-18T15:15:18.024033+0000","last_clean":"2024-03-18T15:15:18.024033+0000","last_became_active":"2024-03-18T15:15:10.667478+0000","last_became_peered":"2024-03-18T15:15:10.667478+0000","last_unstale":"2024-03-18T15:15:18.024033+0000","last_undegraded":"2024-03-18T15:15:18.024033+0000","last_fullsized":"2024-03-18T15:15:18.024033+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:22:48.340642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:16.105523+0000","last_change":"2024-03-18T15:15:15.116404+0000","last_active":"2024-03-18T15:15:16.105523+0000","last_peered":"2024-03-18T15:15:16.105523+0000","last_clean":"2024-03-18T15:15:16.105523+0000","last_became_active":"2024-03-18T15:15:15.115952+0000","last_became_peered":"2024-03-18T15:15:15.115952+0000","last_unstale":"2024-03-18T15:15:16.105523+0000","last_undegraded":"2024-03-18T15:15:16.105523+0000","last_fullsized":"2024-03-18T15:15:16.105523+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:38:44.283538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'7","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:45.502553+0000","last_change":"2024-03-18T15:15:10.080879+0000","last_active":"2024-03-18T15:15:45.502553+0000","last_peered":"2024-03-18T15:15:45.502553+0000","last_clean":"2024-03-18T15:15:45.502553+0000","last_became_active":"2024-03-18T15:15:10.080478+0000","last_became_peered":"2024-03-18T15:15:10.080478+0000","last_unstale":"2024-03-18T15:15:45.502553+0000","last_undegraded":"2024-03-18T15:15:45.502553+0000","last_fullsized":"2024-03-18T15:15:45.502553+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:28:36.511689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.024722+0000","last_change":"2024-03-18T15:15:10.681145+0000","last_active":"2024-03-18T15:15:18.024722+0000","last_peered":"2024-03-18T15:15:18.024722+0000","last_clean":"2024-03-18T15:15:18.024722+0000","last_became_active":"2024-03-18T15:15:10.680816+0000","last_became_peered":"2024-03-18T15:15:10.680816+0000","last_unstale":"2024-03-18T15:15:18.024722+0000","last_undegraded":"2024-03-18T15:15:18.024722+0000","last_fullsized":"2024-03-18T15:15:18.024722+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:15:56.665596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.024082+0000","last_change":"2024-03-18T15:15:15.111563+0000","last_active":"2024-03-18T15:15:18.024082+0000","last_peered":"2024-03-18T15:15:18.024082+0000","last_clean":"2024-03-18T15:15:18.024082+0000","last_became_active":"2024-03-18T15:15:15.111169+0000","last_became_peered":"2024-03-18T15:15:15.111169+0000","last_unstale":"2024-03-18T15:15:18.024082+0000","last_undegraded":"2024-03-18T15:15:18.024082+0000","last_fullsized":"2024-03-18T15:15:18.024082+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:44:05.491883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:17.598765+0000","last_change":"2024-03-18T15:15:10.091753+0000","last_active":"2024-03-18T15:15:17.598765+0000","last_peered":"2024-03-18T15:15:17.598765+0000","last_clean":"2024-03-18T15:15:17.598765+0000","last_became_active":"2024-03-18T15:15:10.091393+0000","last_became_peered":"2024-03-18T15:15:10.091393+0000","last_unstale":"2024-03-18T15:15:17.598765+0000","last_undegraded":"2024-03-18T15:15:17.598765+0000","last_fullsized":"2024-03-18T15:15:17.598765+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:24:28.068621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.023592+0000","last_change":"2024-03-18T15:15:10.672785+0000","last_active":"2024-03-18T15:15:18.023592+0000","last_peered":"2024-03-18T15:15:18.023592+0000","last_clean":"2024-03-18T15:15:18.023592+0000","last_became_active":"2024-03-18T15:15:10.672143+0000","last_became_peered":"2024-03-18T15:15:10.672143+0000","last_unstale":"2024-03-18T15:15:18.023592+0000","last_undegraded":"2024-03-18T15:15:18.023592+0000","last_fullsized":"2024-03-18T15:15:18.023592+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:26:40.034989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:17.578797+0000","last_change":"2024-03-18T15:15:15.114790+0000","last_active":"2024-03-18T15:15:17.578797+0000","last_peered":"2024-03-18T15:15:17.578797+0000","last_clean":"2024-03-18T15:15:17.578797+0000","last_became_active":"2024-03-18T15:15:15.112646+0000","last_became_peered":"2024-03-18T15:15:15.112646+0000","last_unstale":"2024-03-18T15:15:17.578797+0000","last_undegraded":"2024-03-18T15:15:17.578797+0000","last_fullsized":"2024-03-18T15:15:17.578797+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:59:56.686888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.023794+0000","last_change":"2024-03-18T15:15:10.082566+0000","last_active":"2024-03-18T15:15:18.023794+0000","last_peered":"2024-03-18T15:15:18.023794+0000","last_clean":"2024-03-18T15:15:18.023794+0000","last_became_active":"2024-03-18T15:15:10.082256+0000","last_became_peered":"2024-03-18T15:15:10.082256+0000","last_unstale":"2024-03-18T15:15:18.023794+0000","last_undegraded":"2024-03-18T15:15:18.023794+0000","last_fullsized":"2024-03-18T15:15:18.023794+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:04:47.361932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:16.107351+0000","last_change":"2024-03-18T15:15:10.683148+0000","last_active":"2024-03-18T15:15:16.107351+0000","last_peered":"2024-03-18T15:15:16.107351+0000","last_clean":"2024-03-18T15:15:16.107351+0000","last_became_active":"2024-03-18T15:15:10.682671+0000","last_became_peered":"2024-03-18T15:15:10.682671+0000","last_unstale":"2024-03-18T15:15:16.107351+0000","last_undegraded":"2024-03-18T15:15:16.107351+0000","last_fullsized":"2024-03-18T15:15:16.107351+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:52:15.763400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.023476+0000","last_change":"2024-03-18T15:15:15.115995+0000","last_active":"2024-03-18T15:15:18.023476+0000","last_peered":"2024-03-18T15:15:18.023476+0000","last_clean":"2024-03-18T15:15:18.023476+0000","last_became_active":"2024-03-18T15:15:15.115560+0000","last_became_peered":"2024-03-18T15:15:15.115560+0000","last_unstale":"2024-03-18T15:15:18.023476+0000","last_undegraded":"2024-03-18T15:15:18.023476+0000","last_fullsized":"2024-03-18T15:15:18.023476+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:20:36.765286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'4","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:46.991418+0000","last_change":"2024-03-18T15:15:10.095122+0000","last_active":"2024-03-18T15:15:46.991418+0000","last_peered":"2024-03-18T15:15:46.991418+0000","last_clean":"2024-03-18T15:15:46.991418+0000","last_became_active":"2024-03-18T15:15:10.094797+0000","last_became_peered":"2024-03-18T15:15:10.094797+0000","last_unstale":"2024-03-18T15:15:46.991418+0000","last_undegraded":"2024-03-18T15:15:46.991418+0000","last_fullsized":"2024-03-18T15:15:46.991418+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:16:15.124369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.023429+0000","last_change":"2024-03-18T15:15:10.673297+0000","last_active":"2024-03-18T15:15:18.023429+0000","last_peered":"2024-03-18T15:15:18.023429+0000","last_clean":"2024-03-18T15:15:18.023429+0000","last_became_active":"2024-03-18T15:15:10.672890+0000","last_became_peered":"2024-03-18T15:15:10.672890+0000","last_unstale":"2024-03-18T15:15:18.023429+0000","last_undegraded":"2024-03-18T15:15:18.023429+0000","last_fullsized":"2024-03-18T15:15:18.023429+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:10:45.958639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:16.107137+0000","last_change":"2024-03-18T15:15:15.123200+0000","last_active":"2024-03-18T15:15:16.107137+0000","last_peered":"2024-03-18T15:15:16.107137+0000","last_clean":"2024-03-18T15:15:16.107137+0000","last_became_active":"2024-03-18T15:15:15.114554+0000","last_became_peered":"2024-03-18T15:15:15.114554+0000","last_unstale":"2024-03-18T15:15:16.107137+0000","last_undegraded":"2024-03-18T15:15:16.107137+0000","last_fullsized":"2024-03-18T15:15:16.107137+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:35:25.380377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:43.467710+0000","last_change":"2024-03-18T15:15:10.085330+0000","last_active":"2024-03-18T15:15:43.467710+0000","last_peered":"2024-03-18T15:15:43.467710+0000","last_clean":"2024-03-18T15:15:43.467710+0000","last_became_active":"2024-03-18T15:15:10.083179+0000","last_became_peered":"2024-03-18T15:15:10.083179+0000","last_unstale":"2024-03-18T15:15:43.467710+0000","last_undegraded":"2024-03-18T15:15:43.467710+0000","last_fullsized":"2024-03-18T15:15:43.467710+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:56:33.843757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:17.579017+0000","last_change":"2024-03-18T15:15:10.679397+0000","last_active":"2024-03-18T15:15:17.579017+0000","last_peered":"2024-03-18T15:15:17.579017+0000","last_clean":"2024-03-18T15:15:17.579017+0000","last_became_active":"2024-03-18T15:15:10.679006+0000","last_became_peered":"2024-03-18T15:15:10.679006+0000","last_unstale":"2024-03-18T15:15:17.579017+0000","last_undegraded":"2024-03-18T15:15:17.579017+0000","last_fullsized":"2024-03-18T15:15:17.579017+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:38:58.251600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:16.415258+0000","last_change":"2024-03-18T15:15:15.124936+0000","last_active":"2024-03-18T15:15:16.415258+0000","last_peered":"2024-03-18T15:15:16.415258+0000","last_clean":"2024-03-18T15:15:16.415258+0000","last_became_active":"2024-03-18T15:15:15.124501+0000","last_became_peered":"2024-03-18T15:15:15.124501+0000","last_unstale":"2024-03-18T15:15:16.415258+0000","last_undegraded":"2024-03-18T15:15:16.415258+0000","last_fullsized":"2024-03-18T15:15:16.415258+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:31:47.221428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.024042+0000","last_change":"2024-03-18T15:15:10.098885+0000","last_active":"2024-03-18T15:15:18.024042+0000","last_peered":"2024-03-18T15:15:18.024042+0000","last_clean":"2024-03-18T15:15:18.024042+0000","last_became_active":"2024-03-18T15:15:10.098604+0000","last_became_peered":"2024-03-18T15:15:10.098604+0000","last_unstale":"2024-03-18T15:15:18.024042+0000","last_undegraded":"2024-03-18T15:15:18.024042+0000","last_fullsized":"2024-03-18T15:15:18.024042+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:59:08.716855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.023208+0000","last_change":"2024-03-18T15:15:10.662297+0000","last_active":"2024-03-18T15:15:18.023208+0000","last_peered":"2024-03-18T15:15:18.023208+0000","last_clean":"2024-03-18T15:15:18.023208+0000","last_became_active":"2024-03-18T15:15:10.661910+0000","last_became_peered":"2024-03-18T15:15:10.661910+0000","last_unstale":"2024-03-18T15:15:18.023208+0000","last_undegraded":"2024-03-18T15:15:18.023208+0000","last_fullsized":"2024-03-18T15:15:18.023208+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:26:38.043543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:16.107603+0000","last_change":"2024-03-18T15:15:15.107352+0000","last_active":"2024-03-18T15:15:16.107603+0000","last_peered":"2024-03-18T15:15:16.107603+0000","last_clean":"2024-03-18T15:15:16.107603+0000","last_became_active":"2024-03-18T15:15:15.106851+0000","last_became_peered":"2024-03-18T15:15:15.106851+0000","last_unstale":"2024-03-18T15:15:16.107603+0000","last_undegraded":"2024-03-18T15:15:16.107603+0000","last_fullsized":"2024-03-18T15:15:16.107603+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:56:35.695188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'22","reported_seq":49,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:46.992087+0000","last_change":"2024-03-18T15:15:10.083226+0000","last_active":"2024-03-18T15:15:46.992087+0000","last_peered":"2024-03-18T15:15:46.992087+0000","last_clean":"2024-03-18T15:15:46.992087+0000","last_became_active":"2024-03-18T15:15:10.082829+0000","last_became_peered":"2024-03-18T15:15:10.082829+0000","last_unstale":"2024-03-18T15:15:46.992087+0000","last_undegraded":"2024-03-18T15:15:46.992087+0000","last_fullsized":"2024-03-18T15:15:46.992087+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:30:34.091815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23141,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":24,"num_write":24,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.024552+0000","last_change":"2024-03-18T15:15:10.659322+0000","last_active":"2024-03-18T15:15:18.024552+0000","last_peered":"2024-03-18T15:15:18.024552+0000","last_clean":"2024-03-18T15:15:18.024552+0000","last_became_active":"2024-03-18T15:15:10.658997+0000","last_became_peered":"2024-03-18T15:15:10.658997+0000","last_unstale":"2024-03-18T15:15:18.024552+0000","last_undegraded":"2024-03-18T15:15:18.024552+0000","last_fullsized":"2024-03-18T15:15:18.024552+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:04:08.934683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.025229+0000","last_change":"2024-03-18T15:15:15.110658+0000","last_active":"2024-03-18T15:15:18.025229+0000","last_peered":"2024-03-18T15:15:18.025229+0000","last_clean":"2024-03-18T15:15:18.025229+0000","last_became_active":"2024-03-18T15:15:15.110205+0000","last_became_peered":"2024-03-18T15:15:15.110205+0000","last_unstale":"2024-03-18T15:15:18.025229+0000","last_undegraded":"2024-03-18T15:15:18.025229+0000","last_fullsized":"2024-03-18T15:15:18.025229+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:09:10.014743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'7","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:44.489491+0000","last_change":"2024-03-18T15:15:10.080305+0000","last_active":"2024-03-18T15:15:44.489491+0000","last_peered":"2024-03-18T15:15:44.489491+0000","last_clean":"2024-03-18T15:15:44.489491+0000","last_became_active":"2024-03-18T15:15:10.079990+0000","last_became_peered":"2024-03-18T15:15:10.079990+0000","last_unstale":"2024-03-18T15:15:44.489491+0000","last_undegraded":"2024-03-18T15:15:44.489491+0000","last_fullsized":"2024-03-18T15:15:44.489491+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:10:08.312462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:17.598866+0000","last_change":"2024-03-18T15:15:10.671178+0000","last_active":"2024-03-18T15:15:17.598866+0000","last_peered":"2024-03-18T15:15:17.598866+0000","last_clean":"2024-03-18T15:15:17.598866+0000","last_became_active":"2024-03-18T15:15:10.670862+0000","last_became_peered":"2024-03-18T15:15:10.670862+0000","last_unstale":"2024-03-18T15:15:17.598866+0000","last_undegraded":"2024-03-18T15:15:17.598866+0000","last_fullsized":"2024-03-18T15:15:17.598866+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:53:34.916473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.023580+0000","last_change":"2024-03-18T15:15:15.107792+0000","last_active":"2024-03-18T15:15:18.023580+0000","last_peered":"2024-03-18T15:15:18.023580+0000","last_clean":"2024-03-18T15:15:18.023580+0000","last_became_active":"2024-03-18T15:15:15.107327+0000","last_became_peered":"2024-03-18T15:15:15.107327+0000","last_unstale":"2024-03-18T15:15:18.023580+0000","last_undegraded":"2024-03-18T15:15:18.023580+0000","last_fullsized":"2024-03-18T15:15:18.023580+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:42:47.837146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:43.472011+0000","last_change":"2024-03-18T15:15:10.079128+0000","last_active":"2024-03-18T15:15:43.472011+0000","last_peered":"2024-03-18T15:15:43.472011+0000","last_clean":"2024-03-18T15:15:43.472011+0000","last_became_active":"2024-03-18T15:15:10.078812+0000","last_became_peered":"2024-03-18T15:15:10.078812+0000","last_unstale":"2024-03-18T15:15:43.472011+0000","last_undegraded":"2024-03-18T15:15:43.472011+0000","last_fullsized":"2024-03-18T15:15:43.472011+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:12:11.473273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:16.104672+0000","last_change":"2024-03-18T15:15:10.668240+0000","last_active":"2024-03-18T15:15:16.104672+0000","last_peered":"2024-03-18T15:15:16.104672+0000","last_clean":"2024-03-18T15:15:16.104672+0000","last_became_active":"2024-03-18T15:15:10.667885+0000","last_became_peered":"2024-03-18T15:15:10.667885+0000","last_unstale":"2024-03-18T15:15:16.104672+0000","last_undegraded":"2024-03-18T15:15:16.104672+0000","last_fullsized":"2024-03-18T15:15:16.104672+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:17:23.998018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:16.108323+0000","last_change":"2024-03-18T15:15:15.119067+0000","last_active":"2024-03-18T15:15:16.108323+0000","last_peered":"2024-03-18T15:15:16.108323+0000","last_clean":"2024-03-18T15:15:16.108323+0000","last_became_active":"2024-03-18T15:15:15.118566+0000","last_became_peered":"2024-03-18T15:15:15.118566+0000","last_unstale":"2024-03-18T15:15:16.108323+0000","last_undegraded":"2024-03-18T15:15:16.108323+0000","last_fullsized":"2024-03-18T15:15:16.108323+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:21:00.868724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:45.496062+0000","last_change":"2024-03-18T15:15:10.080791+0000","last_active":"2024-03-18T15:15:45.496062+0000","last_peered":"2024-03-18T15:15:45.496062+0000","last_clean":"2024-03-18T15:15:45.496062+0000","last_became_active":"2024-03-18T15:15:10.080368+0000","last_became_peered":"2024-03-18T15:15:10.080368+0000","last_unstale":"2024-03-18T15:15:45.496062+0000","last_undegraded":"2024-03-18T15:15:45.496062+0000","last_fullsized":"2024-03-18T15:15:45.496062+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:10:55.575331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:16.104498+0000","last_change":"2024-03-18T15:15:10.666395+0000","last_active":"2024-03-18T15:15:16.104498+0000","last_peered":"2024-03-18T15:15:16.104498+0000","last_clean":"2024-03-18T15:15:16.104498+0000","last_became_active":"2024-03-18T15:15:10.666057+0000","last_became_peered":"2024-03-18T15:15:10.666057+0000","last_unstale":"2024-03-18T15:15:16.104498+0000","last_undegraded":"2024-03-18T15:15:16.104498+0000","last_fullsized":"2024-03-18T15:15:16.104498+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:04:50.639732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:17.599906+0000","last_change":"2024-03-18T15:15:15.125605+0000","last_active":"2024-03-18T15:15:17.599906+0000","last_peered":"2024-03-18T15:15:17.599906+0000","last_clean":"2024-03-18T15:15:17.599906+0000","last_became_active":"2024-03-18T15:15:15.125041+0000","last_became_peered":"2024-03-18T15:15:15.125041+0000","last_unstale":"2024-03-18T15:15:17.599906+0000","last_undegraded":"2024-03-18T15:15:17.599906+0000","last_fullsized":"2024-03-18T15:15:17.599906+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:27:25.901710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'16","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:44.492188+0000","last_change":"2024-03-18T15:15:10.079653+0000","last_active":"2024-03-18T15:15:44.492188+0000","last_peered":"2024-03-18T15:15:44.492188+0000","last_clean":"2024-03-18T15:15:44.492188+0000","last_became_active":"2024-03-18T15:15:10.079343+0000","last_became_peered":"2024-03-18T15:15:10.079343+0000","last_unstale":"2024-03-18T15:15:44.492188+0000","last_undegraded":"2024-03-18T15:15:44.492188+0000","last_fullsized":"2024-03-18T15:15:44.492188+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:33:58.746118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.024499+0000","last_change":"2024-03-18T15:15:10.661241+0000","last_active":"2024-03-18T15:15:18.024499+0000","last_peered":"2024-03-18T15:15:18.024499+0000","last_clean":"2024-03-18T15:15:18.024499+0000","last_became_active":"2024-03-18T15:15:10.660821+0000","last_became_peered":"2024-03-18T15:15:10.660821+0000","last_unstale":"2024-03-18T15:15:18.024499+0000","last_undegraded":"2024-03-18T15:15:18.024499+0000","last_fullsized":"2024-03-18T15:15:18.024499+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:13:59.973615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.024445+0000","last_change":"2024-03-18T15:15:15.118314+0000","last_active":"2024-03-18T15:15:18.024445+0000","last_peered":"2024-03-18T15:15:18.024445+0000","last_clean":"2024-03-18T15:15:18.024445+0000","last_became_active":"2024-03-18T15:15:15.118001+0000","last_became_peered":"2024-03-18T15:15:15.118001+0000","last_unstale":"2024-03-18T15:15:18.024445+0000","last_undegraded":"2024-03-18T15:15:18.024445+0000","last_fullsized":"2024-03-18T15:15:18.024445+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:15:58.320283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.024756+0000","last_change":"2024-03-18T15:15:10.077394+0000","last_active":"2024-03-18T15:15:18.024756+0000","last_peered":"2024-03-18T15:15:18.024756+0000","last_clean":"2024-03-18T15:15:18.024756+0000","last_became_active":"2024-03-18T15:15:10.077027+0000","last_became_peered":"2024-03-18T15:15:10.077027+0000","last_unstale":"2024-03-18T15:15:18.024756+0000","last_undegraded":"2024-03-18T15:15:18.024756+0000","last_fullsized":"2024-03-18T15:15:18.024756+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:34:07.524849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.023346+0000","last_change":"2024-03-18T15:15:10.659723+0000","last_active":"2024-03-18T15:15:18.023346+0000","last_peered":"2024-03-18T15:15:18.023346+0000","last_clean":"2024-03-18T15:15:18.023346+0000","last_became_active":"2024-03-18T15:15:10.659388+0000","last_became_peered":"2024-03-18T15:15:10.659388+0000","last_unstale":"2024-03-18T15:15:18.023346+0000","last_undegraded":"2024-03-18T15:15:18.023346+0000","last_fullsized":"2024-03-18T15:15:18.023346+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:21:52.605870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.023894+0000","last_change":"2024-03-18T15:15:15.115459+0000","last_active":"2024-03-18T15:15:18.023894+0000","last_peered":"2024-03-18T15:15:18.023894+0000","last_clean":"2024-03-18T15:15:18.023894+0000","last_became_active":"2024-03-18T15:15:15.115048+0000","last_became_peered":"2024-03-18T15:15:15.115048+0000","last_unstale":"2024-03-18T15:15:18.023894+0000","last_undegraded":"2024-03-18T15:15:18.023894+0000","last_fullsized":"2024-03-18T15:15:18.023894+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:04:23.118904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:16.107977+0000","last_change":"2024-03-18T15:15:10.096698+0000","last_active":"2024-03-18T15:15:16.107977+0000","last_peered":"2024-03-18T15:15:16.107977+0000","last_clean":"2024-03-18T15:15:16.107977+0000","last_became_active":"2024-03-18T15:15:10.096212+0000","last_became_peered":"2024-03-18T15:15:10.096212+0000","last_unstale":"2024-03-18T15:15:16.107977+0000","last_undegraded":"2024-03-18T15:15:16.107977+0000","last_fullsized":"2024-03-18T15:15:16.107977+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:02:07.627327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.023743+0000","last_change":"2024-03-18T15:15:10.672954+0000","last_active":"2024-03-18T15:15:18.023743+0000","last_peered":"2024-03-18T15:15:18.023743+0000","last_clean":"2024-03-18T15:15:18.023743+0000","last_became_active":"2024-03-18T15:15:10.672415+0000","last_became_peered":"2024-03-18T15:15:10.672415+0000","last_unstale":"2024-03-18T15:15:18.023743+0000","last_undegraded":"2024-03-18T15:15:18.023743+0000","last_fullsized":"2024-03-18T15:15:18.023743+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:47:48.421411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:17.599329+0000","last_change":"2024-03-18T15:15:15.121334+0000","last_active":"2024-03-18T15:15:17.599329+0000","last_peered":"2024-03-18T15:15:17.599329+0000","last_clean":"2024-03-18T15:15:17.599329+0000","last_became_active":"2024-03-18T15:15:15.119797+0000","last_became_peered":"2024-03-18T15:15:15.119797+0000","last_unstale":"2024-03-18T15:15:17.599329+0000","last_undegraded":"2024-03-18T15:15:17.599329+0000","last_fullsized":"2024-03-18T15:15:17.599329+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:28:39.072485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:16.104354+0000","last_change":"2024-03-18T15:15:10.082589+0000","last_active":"2024-03-18T15:15:16.104354+0000","last_peered":"2024-03-18T15:15:16.104354+0000","last_clean":"2024-03-18T15:15:16.104354+0000","last_became_active":"2024-03-18T15:15:10.082190+0000","last_became_peered":"2024-03-18T15:15:10.082190+0000","last_unstale":"2024-03-18T15:15:16.104354+0000","last_undegraded":"2024-03-18T15:15:16.104354+0000","last_fullsized":"2024-03-18T15:15:16.104354+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:55:15.631914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.023456+0000","last_change":"2024-03-18T15:15:10.672145+0000","last_active":"2024-03-18T15:15:18.023456+0000","last_peered":"2024-03-18T15:15:18.023456+0000","last_clean":"2024-03-18T15:15:18.023456+0000","last_became_active":"2024-03-18T15:15:10.670303+0000","last_became_peered":"2024-03-18T15:15:10.670303+0000","last_unstale":"2024-03-18T15:15:18.023456+0000","last_undegraded":"2024-03-18T15:15:18.023456+0000","last_fullsized":"2024-03-18T15:15:18.023456+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:22:50.528794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:17.599875+0000","last_change":"2024-03-18T15:15:15.113675+0000","last_active":"2024-03-18T15:15:17.599875+0000","last_peered":"2024-03-18T15:15:17.599875+0000","last_clean":"2024-03-18T15:15:17.599875+0000","last_became_active":"2024-03-18T15:15:15.113067+0000","last_became_peered":"2024-03-18T15:15:15.113067+0000","last_unstale":"2024-03-18T15:15:17.599875+0000","last_undegraded":"2024-03-18T15:15:17.599875+0000","last_fullsized":"2024-03-18T15:15:17.599875+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:37:41.751216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.023951+0000","last_change":"2024-03-18T15:15:10.658032+0000","last_active":"2024-03-18T15:15:18.023951+0000","last_peered":"2024-03-18T15:15:18.023951+0000","last_clean":"2024-03-18T15:15:18.023951+0000","last_became_active":"2024-03-18T15:15:10.657707+0000","last_became_peered":"2024-03-18T15:15:10.657707+0000","last_unstale":"2024-03-18T15:15:18.023951+0000","last_undegraded":"2024-03-18T15:15:18.023951+0000","last_fullsized":"2024-03-18T15:15:18.023951+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:44:24.137349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:17.598690+0000","last_change":"2024-03-18T15:15:10.085330+0000","last_active":"2024-03-18T15:15:17.598690+0000","last_peered":"2024-03-18T15:15:17.598690+0000","last_clean":"2024-03-18T15:15:17.598690+0000","last_became_active":"2024-03-18T15:15:10.080303+0000","last_became_peered":"2024-03-18T15:15:10.080303+0000","last_unstale":"2024-03-18T15:15:17.598690+0000","last_undegraded":"2024-03-18T15:15:17.598690+0000","last_fullsized":"2024-03-18T15:15:17.598690+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:26:39.293028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:16.104581+0000","last_change":"2024-03-18T15:15:15.125172+0000","last_active":"2024-03-18T15:15:16.104581+0000","last_peered":"2024-03-18T15:15:16.104581+0000","last_clean":"2024-03-18T15:15:16.104581+0000","last_became_active":"2024-03-18T15:15:15.124591+0000","last_became_peered":"2024-03-18T15:15:15.124591+0000","last_unstale":"2024-03-18T15:15:16.104581+0000","last_undegraded":"2024-03-18T15:15:16.104581+0000","last_fullsized":"2024-03-18T15:15:16.104581+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:50:16.339041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.023809+0000","last_change":"2024-03-18T15:15:10.685343+0000","last_active":"2024-03-18T15:15:18.023809+0000","last_peered":"2024-03-18T15:15:18.023809+0000","last_clean":"2024-03-18T15:15:18.023809+0000","last_became_active":"2024-03-18T15:15:10.684395+0000","last_became_peered":"2024-03-18T15:15:10.684395+0000","last_unstale":"2024-03-18T15:15:18.023809+0000","last_undegraded":"2024-03-18T15:15:18.023809+0000","last_fullsized":"2024-03-18T15:15:18.023809+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:11:26.982047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:45.500018+0000","last_change":"2024-03-18T15:15:10.085503+0000","last_active":"2024-03-18T15:15:45.500018+0000","last_peered":"2024-03-18T15:15:45.500018+0000","last_clean":"2024-03-18T15:15:45.500018+0000","last_became_active":"2024-03-18T15:15:10.085125+0000","last_became_peered":"2024-03-18T15:15:10.085125+0000","last_unstale":"2024-03-18T15:15:45.500018+0000","last_undegraded":"2024-03-18T15:15:45.500018+0000","last_fullsized":"2024-03-18T15:15:45.500018+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:17:13.091428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":355,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:16.104198+0000","last_change":"2024-03-18T15:13:50.952630+0000","last_active":"2024-03-18T15:15:16.104198+0000","last_peered":"2024-03-18T15:15:16.104198+0000","last_clean":"2024-03-18T15:15:16.104198+0000","last_became_active":"2024-03-18T15:13:50.952144+0000","last_became_peered":"2024-03-18T15:13:50.952144+0000","last_unstale":"2024-03-18T15:15:16.104198+0000","last_undegraded":"2024-03-18T15:15:16.104198+0000","last_fullsized":"2024-03-18T15:15:16.104198+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:09:56.845161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:09:56.845161+0000","last_clean_scrub_stamp":"2024-03-18T15:09:56.845161+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:53:27.959711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.024727+0000","last_change":"2024-03-18T15:15:15.115418+0000","last_active":"2024-03-18T15:15:18.024727+0000","last_peered":"2024-03-18T15:15:18.024727+0000","last_clean":"2024-03-18T15:15:18.024727+0000","last_became_active":"2024-03-18T15:15:15.114715+0000","last_became_peered":"2024-03-18T15:15:15.114715+0000","last_unstale":"2024-03-18T15:15:18.024727+0000","last_undegraded":"2024-03-18T15:15:18.024727+0000","last_fullsized":"2024-03-18T15:15:18.024727+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:37:51.553099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:17.598587+0000","last_change":"2024-03-18T15:15:10.665492+0000","last_active":"2024-03-18T15:15:17.598587+0000","last_peered":"2024-03-18T15:15:17.598587+0000","last_clean":"2024-03-18T15:15:17.598587+0000","last_became_active":"2024-03-18T15:15:10.665097+0000","last_became_peered":"2024-03-18T15:15:10.665097+0000","last_unstale":"2024-03-18T15:15:17.598587+0000","last_undegraded":"2024-03-18T15:15:17.598587+0000","last_fullsized":"2024-03-18T15:15:17.598587+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:27:06.864459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:44.484810+0000","last_change":"2024-03-18T15:15:10.085322+0000","last_active":"2024-03-18T15:15:44.484810+0000","last_peered":"2024-03-18T15:15:44.484810+0000","last_clean":"2024-03-18T15:15:44.484810+0000","last_became_active":"2024-03-18T15:15:10.083114+0000","last_became_peered":"2024-03-18T15:15:10.083114+0000","last_unstale":"2024-03-18T15:15:44.484810+0000","last_undegraded":"2024-03-18T15:15:44.484810+0000","last_fullsized":"2024-03-18T15:15:44.484810+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:13:31.878513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.023259+0000","last_change":"2024-03-18T15:15:15.105432+0000","last_active":"2024-03-18T15:15:18.023259+0000","last_peered":"2024-03-18T15:15:18.023259+0000","last_clean":"2024-03-18T15:15:18.023259+0000","last_became_active":"2024-03-18T15:15:15.104929+0000","last_became_peered":"2024-03-18T15:15:15.104929+0000","last_unstale":"2024-03-18T15:15:18.023259+0000","last_undegraded":"2024-03-18T15:15:18.023259+0000","last_fullsized":"2024-03-18T15:15:18.023259+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:50:37.334639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:45.500978+0000","last_change":"2024-03-18T15:15:10.099722+0000","last_active":"2024-03-18T15:15:45.500978+0000","last_peered":"2024-03-18T15:15:45.500978+0000","last_clean":"2024-03-18T15:15:45.500978+0000","last_became_active":"2024-03-18T15:15:10.099410+0000","last_became_peered":"2024-03-18T15:15:10.099410+0000","last_unstale":"2024-03-18T15:15:45.500978+0000","last_undegraded":"2024-03-18T15:15:45.500978+0000","last_fullsized":"2024-03-18T15:15:45.500978+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:19:24.877473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:17.599504+0000","last_change":"2024-03-18T15:15:10.669235+0000","last_active":"2024-03-18T15:15:17.599504+0000","last_peered":"2024-03-18T15:15:17.599504+0000","last_clean":"2024-03-18T15:15:17.599504+0000","last_became_active":"2024-03-18T15:15:10.667369+0000","last_became_peered":"2024-03-18T15:15:10.667369+0000","last_unstale":"2024-03-18T15:15:17.599504+0000","last_undegraded":"2024-03-18T15:15:17.599504+0000","last_fullsized":"2024-03-18T15:15:17.599504+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:36:08.961435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.023835+0000","last_change":"2024-03-18T15:15:15.119228+0000","last_active":"2024-03-18T15:15:18.023835+0000","last_peered":"2024-03-18T15:15:18.023835+0000","last_clean":"2024-03-18T15:15:18.023835+0000","last_became_active":"2024-03-18T15:15:15.115585+0000","last_became_peered":"2024-03-18T15:15:15.115585+0000","last_unstale":"2024-03-18T15:15:18.023835+0000","last_undegraded":"2024-03-18T15:15:18.023835+0000","last_fullsized":"2024-03-18T15:15:18.023835+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:36:43.263907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:44.493312+0000","last_change":"2024-03-18T15:15:10.105259+0000","last_active":"2024-03-18T15:15:44.493312+0000","last_peered":"2024-03-18T15:15:44.493312+0000","last_clean":"2024-03-18T15:15:44.493312+0000","last_became_active":"2024-03-18T15:15:10.104315+0000","last_became_peered":"2024-03-18T15:15:10.104315+0000","last_unstale":"2024-03-18T15:15:44.493312+0000","last_undegraded":"2024-03-18T15:15:44.493312+0000","last_fullsized":"2024-03-18T15:15:44.493312+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:27:05.565134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.024161+0000","last_change":"2024-03-18T15:15:10.673615+0000","last_active":"2024-03-18T15:15:18.024161+0000","last_peered":"2024-03-18T15:15:18.024161+0000","last_clean":"2024-03-18T15:15:18.024161+0000","last_became_active":"2024-03-18T15:15:10.672308+0000","last_became_peered":"2024-03-18T15:15:10.672308+0000","last_unstale":"2024-03-18T15:15:18.024161+0000","last_undegraded":"2024-03-18T15:15:18.024161+0000","last_fullsized":"2024-03-18T15:15:18.024161+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:32:13.889281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:16.105119+0000","last_change":"2024-03-18T15:15:15.099447+0000","last_active":"2024-03-18T15:15:16.105119+0000","last_peered":"2024-03-18T15:15:16.105119+0000","last_clean":"2024-03-18T15:15:16.105119+0000","last_became_active":"2024-03-18T15:15:15.098857+0000","last_became_peered":"2024-03-18T15:15:15.098857+0000","last_unstale":"2024-03-18T15:15:16.105119+0000","last_undegraded":"2024-03-18T15:15:16.105119+0000","last_fullsized":"2024-03-18T15:15:16.105119+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:51:22.115359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:43.468516+0000","last_change":"2024-03-18T15:15:10.084281+0000","last_active":"2024-03-18T15:15:43.468516+0000","last_peered":"2024-03-18T15:15:43.468516+0000","last_clean":"2024-03-18T15:15:43.468516+0000","last_became_active":"2024-03-18T15:15:10.083791+0000","last_became_peered":"2024-03-18T15:15:10.083791+0000","last_unstale":"2024-03-18T15:15:43.468516+0000","last_undegraded":"2024-03-18T15:15:43.468516+0000","last_fullsized":"2024-03-18T15:15:43.468516+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:08:33.066629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:17.578906+0000","last_change":"2024-03-18T15:15:10.682823+0000","last_active":"2024-03-18T15:15:17.578906+0000","last_peered":"2024-03-18T15:15:17.578906+0000","last_clean":"2024-03-18T15:15:17.578906+0000","last_became_active":"2024-03-18T15:15:10.681887+0000","last_became_peered":"2024-03-18T15:15:10.681887+0000","last_unstale":"2024-03-18T15:15:17.578906+0000","last_undegraded":"2024-03-18T15:15:17.578906+0000","last_fullsized":"2024-03-18T15:15:17.578906+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:33:29.918824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:16.415170+0000","last_change":"2024-03-18T15:15:15.114952+0000","last_active":"2024-03-18T15:15:16.415170+0000","last_peered":"2024-03-18T15:15:16.415170+0000","last_clean":"2024-03-18T15:15:16.415170+0000","last_became_active":"2024-03-18T15:15:15.114331+0000","last_became_peered":"2024-03-18T15:15:15.114331+0000","last_unstale":"2024-03-18T15:15:16.415170+0000","last_undegraded":"2024-03-18T15:15:16.415170+0000","last_fullsized":"2024-03-18T15:15:16.415170+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:22:08.637152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:43.471482+0000","last_change":"2024-03-18T15:15:10.080291+0000","last_active":"2024-03-18T15:15:43.471482+0000","last_peered":"2024-03-18T15:15:43.471482+0000","last_clean":"2024-03-18T15:15:43.471482+0000","last_became_active":"2024-03-18T15:15:10.079634+0000","last_became_peered":"2024-03-18T15:15:10.079634+0000","last_unstale":"2024-03-18T15:15:43.471482+0000","last_undegraded":"2024-03-18T15:15:43.471482+0000","last_fullsized":"2024-03-18T15:15:43.471482+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:12:10.280094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.023550+0000","last_change":"2024-03-18T15:15:10.657245+0000","last_active":"2024-03-18T15:15:18.023550+0000","last_peered":"2024-03-18T15:15:18.023550+0000","last_clean":"2024-03-18T15:15:18.023550+0000","last_became_active":"2024-03-18T15:15:10.655766+0000","last_became_peered":"2024-03-18T15:15:10.655766+0000","last_unstale":"2024-03-18T15:15:18.023550+0000","last_undegraded":"2024-03-18T15:15:18.023550+0000","last_fullsized":"2024-03-18T15:15:18.023550+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:44:26.340539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:16.108571+0000","last_change":"2024-03-18T15:15:15.109103+0000","last_active":"2024-03-18T15:15:16.108571+0000","last_peered":"2024-03-18T15:15:16.108571+0000","last_clean":"2024-03-18T15:15:16.108571+0000","last_became_active":"2024-03-18T15:15:15.108693+0000","last_became_peered":"2024-03-18T15:15:15.108693+0000","last_unstale":"2024-03-18T15:15:16.108571+0000","last_undegraded":"2024-03-18T15:15:16.108571+0000","last_fullsized":"2024-03-18T15:15:16.108571+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:40:57.841960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'7","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:46.547283+0000","last_change":"2024-03-18T15:15:10.103183+0000","last_active":"2024-03-18T15:15:46.547283+0000","last_peered":"2024-03-18T15:15:46.547283+0000","last_clean":"2024-03-18T15:15:46.547283+0000","last_became_active":"2024-03-18T15:15:10.101948+0000","last_became_peered":"2024-03-18T15:15:10.101948+0000","last_unstale":"2024-03-18T15:15:46.547283+0000","last_undegraded":"2024-03-18T15:15:46.547283+0000","last_fullsized":"2024-03-18T15:15:46.547283+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:04:51.718433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.023104+0000","last_change":"2024-03-18T15:15:10.666940+0000","last_active":"2024-03-18T15:15:18.023104+0000","last_peered":"2024-03-18T15:15:18.023104+0000","last_clean":"2024-03-18T15:15:18.023104+0000","last_became_active":"2024-03-18T15:15:10.666639+0000","last_became_peered":"2024-03-18T15:15:10.666639+0000","last_unstale":"2024-03-18T15:15:18.023104+0000","last_undegraded":"2024-03-18T15:15:18.023104+0000","last_fullsized":"2024-03-18T15:15:18.023104+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:04:45.945023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:16.415012+0000","last_change":"2024-03-18T15:15:15.120541+0000","last_active":"2024-03-18T15:15:16.415012+0000","last_peered":"2024-03-18T15:15:16.415012+0000","last_clean":"2024-03-18T15:15:16.415012+0000","last_became_active":"2024-03-18T15:15:15.120097+0000","last_became_peered":"2024-03-18T15:15:15.120097+0000","last_unstale":"2024-03-18T15:15:16.415012+0000","last_undegraded":"2024-03-18T15:15:16.415012+0000","last_fullsized":"2024-03-18T15:15:16.415012+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:12:08.932598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:45.464197+0000","last_change":"2024-03-18T15:15:10.079199+0000","last_active":"2024-03-18T15:15:45.464197+0000","last_peered":"2024-03-18T15:15:45.464197+0000","last_clean":"2024-03-18T15:15:45.464197+0000","last_became_active":"2024-03-18T15:15:10.078872+0000","last_became_peered":"2024-03-18T15:15:10.078872+0000","last_unstale":"2024-03-18T15:15:45.464197+0000","last_undegraded":"2024-03-18T15:15:45.464197+0000","last_fullsized":"2024-03-18T15:15:45.464197+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:31:44.722779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.023818+0000","last_change":"2024-03-18T15:15:10.686304+0000","last_active":"2024-03-18T15:15:18.023818+0000","last_peered":"2024-03-18T15:15:18.023818+0000","last_clean":"2024-03-18T15:15:18.023818+0000","last_became_active":"2024-03-18T15:15:10.685636+0000","last_became_peered":"2024-03-18T15:15:10.685636+0000","last_unstale":"2024-03-18T15:15:18.023818+0000","last_undegraded":"2024-03-18T15:15:18.023818+0000","last_fullsized":"2024-03-18T15:15:18.023818+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:31:03.315971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:16.415113+0000","last_change":"2024-03-18T15:15:15.108254+0000","last_active":"2024-03-18T15:15:16.415113+0000","last_peered":"2024-03-18T15:15:16.415113+0000","last_clean":"2024-03-18T15:15:16.415113+0000","last_became_active":"2024-03-18T15:15:15.107806+0000","last_became_peered":"2024-03-18T15:15:15.107806+0000","last_unstale":"2024-03-18T15:15:16.415113+0000","last_undegraded":"2024-03-18T15:15:16.415113+0000","last_fullsized":"2024-03-18T15:15:16.415113+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:55:53.427426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:43.467841+0000","last_change":"2024-03-18T15:15:10.082574+0000","last_active":"2024-03-18T15:15:43.467841+0000","last_peered":"2024-03-18T15:15:43.467841+0000","last_clean":"2024-03-18T15:15:43.467841+0000","last_became_active":"2024-03-18T15:15:10.082252+0000","last_became_peered":"2024-03-18T15:15:10.082252+0000","last_unstale":"2024-03-18T15:15:43.467841+0000","last_undegraded":"2024-03-18T15:15:43.467841+0000","last_fullsized":"2024-03-18T15:15:43.467841+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:25:17.475023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:16.104324+0000","last_change":"2024-03-18T15:15:10.667304+0000","last_active":"2024-03-18T15:15:16.104324+0000","last_peered":"2024-03-18T15:15:16.104324+0000","last_clean":"2024-03-18T15:15:16.104324+0000","last_became_active":"2024-03-18T15:15:10.666932+0000","last_became_peered":"2024-03-18T15:15:10.666932+0000","last_unstale":"2024-03-18T15:15:16.104324+0000","last_undegraded":"2024-03-18T15:15:16.104324+0000","last_fullsized":"2024-03-18T15:15:16.104324+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:22:33.817618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.022728+0000","last_change":"2024-03-18T15:15:15.113499+0000","last_active":"2024-03-18T15:15:18.022728+0000","last_peered":"2024-03-18T15:15:18.022728+0000","last_clean":"2024-03-18T15:15:18.022728+0000","last_became_active":"2024-03-18T15:15:15.113095+0000","last_became_peered":"2024-03-18T15:15:15.113095+0000","last_unstale":"2024-03-18T15:15:18.022728+0000","last_undegraded":"2024-03-18T15:15:18.022728+0000","last_fullsized":"2024-03-18T15:15:18.022728+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:58:28.255634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:45.500316+0000","last_change":"2024-03-18T15:15:10.078129+0000","last_active":"2024-03-18T15:15:45.500316+0000","last_peered":"2024-03-18T15:15:45.500316+0000","last_clean":"2024-03-18T15:15:45.500316+0000","last_became_active":"2024-03-18T15:15:10.076202+0000","last_became_peered":"2024-03-18T15:15:10.076202+0000","last_unstale":"2024-03-18T15:15:45.500316+0000","last_undegraded":"2024-03-18T15:15:45.500316+0000","last_fullsized":"2024-03-18T15:15:45.500316+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:24:58.240937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:16.104721+0000","last_change":"2024-03-18T15:15:10.665036+0000","last_active":"2024-03-18T15:15:16.104721+0000","last_peered":"2024-03-18T15:15:16.104721+0000","last_clean":"2024-03-18T15:15:16.104721+0000","last_became_active":"2024-03-18T15:15:10.664614+0000","last_became_peered":"2024-03-18T15:15:10.664614+0000","last_unstale":"2024-03-18T15:15:16.104721+0000","last_undegraded":"2024-03-18T15:15:16.104721+0000","last_fullsized":"2024-03-18T15:15:16.104721+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:31:42.454317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:17.578249+0000","last_change":"2024-03-18T15:15:15.127812+0000","last_active":"2024-03-18T15:15:17.578249+0000","last_peered":"2024-03-18T15:15:17.578249+0000","last_clean":"2024-03-18T15:15:17.578249+0000","last_became_active":"2024-03-18T15:15:15.127300+0000","last_became_peered":"2024-03-18T15:15:15.127300+0000","last_unstale":"2024-03-18T15:15:17.578249+0000","last_undegraded":"2024-03-18T15:15:17.578249+0000","last_fullsized":"2024-03-18T15:15:17.578249+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:41:25.292884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.024223+0000","last_change":"2024-03-18T15:15:10.093866+0000","last_active":"2024-03-18T15:15:18.024223+0000","last_peered":"2024-03-18T15:15:18.024223+0000","last_clean":"2024-03-18T15:15:18.024223+0000","last_became_active":"2024-03-18T15:15:10.093568+0000","last_became_peered":"2024-03-18T15:15:10.093568+0000","last_unstale":"2024-03-18T15:15:18.024223+0000","last_undegraded":"2024-03-18T15:15:18.024223+0000","last_fullsized":"2024-03-18T15:15:18.024223+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:00:03.533705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:16.104362+0000","last_change":"2024-03-18T15:15:10.679506+0000","last_active":"2024-03-18T15:15:16.104362+0000","last_peered":"2024-03-18T15:15:16.104362+0000","last_clean":"2024-03-18T15:15:16.104362+0000","last_became_active":"2024-03-18T15:15:10.679164+0000","last_became_peered":"2024-03-18T15:15:10.679164+0000","last_unstale":"2024-03-18T15:15:16.104362+0000","last_undegraded":"2024-03-18T15:15:16.104362+0000","last_fullsized":"2024-03-18T15:15:16.104362+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:04:55.266273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.023611+0000","last_change":"2024-03-18T15:15:15.099072+0000","last_active":"2024-03-18T15:15:18.023611+0000","last_peered":"2024-03-18T15:15:18.023611+0000","last_clean":"2024-03-18T15:15:18.023611+0000","last_became_active":"2024-03-18T15:15:15.098165+0000","last_became_peered":"2024-03-18T15:15:15.098165+0000","last_unstale":"2024-03-18T15:15:18.023611+0000","last_undegraded":"2024-03-18T15:15:18.023611+0000","last_fullsized":"2024-03-18T15:15:18.023611+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:47:47.377924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:17.599824+0000","last_change":"2024-03-18T15:15:10.102562+0000","last_active":"2024-03-18T15:15:17.599824+0000","last_peered":"2024-03-18T15:15:17.599824+0000","last_clean":"2024-03-18T15:15:17.599824+0000","last_became_active":"2024-03-18T15:15:10.099318+0000","last_became_peered":"2024-03-18T15:15:10.099318+0000","last_unstale":"2024-03-18T15:15:17.599824+0000","last_undegraded":"2024-03-18T15:15:17.599824+0000","last_fullsized":"2024-03-18T15:15:17.599824+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:55:15.601925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:16.104687+0000","last_change":"2024-03-18T15:15:10.662645+0000","last_active":"2024-03-18T15:15:16.104687+0000","last_peered":"2024-03-18T15:15:16.104687+0000","last_clean":"2024-03-18T15:15:16.104687+0000","last_became_active":"2024-03-18T15:15:10.662280+0000","last_became_peered":"2024-03-18T15:15:10.662280+0000","last_unstale":"2024-03-18T15:15:16.104687+0000","last_undegraded":"2024-03-18T15:15:16.104687+0000","last_fullsized":"2024-03-18T15:15:16.104687+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:11:22.598169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.023182+0000","last_change":"2024-03-18T15:15:10.661839+0000","last_active":"2024-03-18T15:15:18.023182+0000","last_peered":"2024-03-18T15:15:18.023182+0000","last_clean":"2024-03-18T15:15:18.023182+0000","last_became_active":"2024-03-18T15:15:10.661356+0000","last_became_peered":"2024-03-18T15:15:10.661356+0000","last_unstale":"2024-03-18T15:15:18.023182+0000","last_undegraded":"2024-03-18T15:15:18.023182+0000","last_fullsized":"2024-03-18T15:15:18.023182+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:13:33.637281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:16.105099+0000","last_change":"2024-03-18T15:15:10.661946+0000","last_active":"2024-03-18T15:15:16.105099+0000","last_peered":"2024-03-18T15:15:16.105099+0000","last_clean":"2024-03-18T15:15:16.105099+0000","last_became_active":"2024-03-18T15:15:10.661574+0000","last_became_peered":"2024-03-18T15:15:10.661574+0000","last_unstale":"2024-03-18T15:15:16.105099+0000","last_undegraded":"2024-03-18T15:15:16.105099+0000","last_fullsized":"2024-03-18T15:15:16.105099+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:53:50.284156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:16.104978+0000","last_change":"2024-03-18T15:15:10.674064+0000","last_active":"2024-03-18T15:15:16.104978+0000","last_peered":"2024-03-18T15:15:16.104978+0000","last_clean":"2024-03-18T15:15:16.104978+0000","last_became_active":"2024-03-18T15:15:10.673820+0000","last_became_peered":"2024-03-18T15:15:10.673820+0000","last_unstale":"2024-03-18T15:15:16.104978+0000","last_undegraded":"2024-03-18T15:15:16.104978+0000","last_fullsized":"2024-03-18T15:15:16.104978+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:29:28.977824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:17.578512+0000","last_change":"2024-03-18T15:15:10.671276+0000","last_active":"2024-03-18T15:15:17.578512+0000","last_peered":"2024-03-18T15:15:17.578512+0000","last_clean":"2024-03-18T15:15:17.578512+0000","last_became_active":"2024-03-18T15:15:10.670926+0000","last_became_peered":"2024-03-18T15:15:10.670926+0000","last_unstale":"2024-03-18T15:15:17.578512+0000","last_undegraded":"2024-03-18T15:15:17.578512+0000","last_fullsized":"2024-03-18T15:15:17.578512+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:15:43.972815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:17.599478+0000","last_change":"2024-03-18T15:15:10.680165+0000","last_active":"2024-03-18T15:15:17.599478+0000","last_peered":"2024-03-18T15:15:17.599478+0000","last_clean":"2024-03-18T15:15:17.599478+0000","last_became_active":"2024-03-18T15:15:10.679849+0000","last_became_peered":"2024-03-18T15:15:10.679849+0000","last_unstale":"2024-03-18T15:15:17.599478+0000","last_undegraded":"2024-03-18T15:15:17.599478+0000","last_fullsized":"2024-03-18T15:15:17.599478+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:15:00.352175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.023964+0000","last_change":"2024-03-18T15:15:10.658966+0000","last_active":"2024-03-18T15:15:18.023964+0000","last_peered":"2024-03-18T15:15:18.023964+0000","last_clean":"2024-03-18T15:15:18.023964+0000","last_became_active":"2024-03-18T15:15:10.658654+0000","last_became_peered":"2024-03-18T15:15:10.658654+0000","last_unstale":"2024-03-18T15:15:18.023964+0000","last_undegraded":"2024-03-18T15:15:18.023964+0000","last_fullsized":"2024-03-18T15:15:18.023964+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:42:25.730735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:16.105653+0000","last_change":"2024-03-18T15:15:10.665694+0000","last_active":"2024-03-18T15:15:16.105653+0000","last_peered":"2024-03-18T15:15:16.105653+0000","last_clean":"2024-03-18T15:15:16.105653+0000","last_became_active":"2024-03-18T15:15:10.664688+0000","last_became_peered":"2024-03-18T15:15:10.664688+0000","last_unstale":"2024-03-18T15:15:16.105653+0000","last_undegraded":"2024-03-18T15:15:16.105653+0000","last_fullsized":"2024-03-18T15:15:16.105653+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:32:23.108060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.024267+0000","last_change":"2024-03-18T15:15:10.676129+0000","last_active":"2024-03-18T15:15:18.024267+0000","last_peered":"2024-03-18T15:15:18.024267+0000","last_clean":"2024-03-18T15:15:18.024267+0000","last_became_active":"2024-03-18T15:15:10.675819+0000","last_became_peered":"2024-03-18T15:15:10.675819+0000","last_unstale":"2024-03-18T15:15:18.024267+0000","last_undegraded":"2024-03-18T15:15:18.024267+0000","last_fullsized":"2024-03-18T15:15:18.024267+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:35:28.578802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.023850+0000","last_change":"2024-03-18T15:15:10.668833+0000","last_active":"2024-03-18T15:15:18.023850+0000","last_peered":"2024-03-18T15:15:18.023850+0000","last_clean":"2024-03-18T15:15:18.023850+0000","last_became_active":"2024-03-18T15:15:10.668572+0000","last_became_peered":"2024-03-18T15:15:10.668572+0000","last_unstale":"2024-03-18T15:15:18.023850+0000","last_undegraded":"2024-03-18T15:15:18.023850+0000","last_fullsized":"2024-03-18T15:15:18.023850+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:12:44.333515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.023703+0000","last_change":"2024-03-18T15:15:10.677496+0000","last_active":"2024-03-18T15:15:18.023703+0000","last_peered":"2024-03-18T15:15:18.023703+0000","last_clean":"2024-03-18T15:15:18.023703+0000","last_became_active":"2024-03-18T15:15:10.674869+0000","last_became_peered":"2024-03-18T15:15:10.674869+0000","last_unstale":"2024-03-18T15:15:18.023703+0000","last_undegraded":"2024-03-18T15:15:18.023703+0000","last_fullsized":"2024-03-18T15:15:18.023703+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:03:10.531488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.022863+0000","last_change":"2024-03-18T15:15:10.673488+0000","last_active":"2024-03-18T15:15:18.022863+0000","last_peered":"2024-03-18T15:15:18.022863+0000","last_clean":"2024-03-18T15:15:18.022863+0000","last_became_active":"2024-03-18T15:15:10.673117+0000","last_became_peered":"2024-03-18T15:15:10.673117+0000","last_unstale":"2024-03-18T15:15:18.022863+0000","last_undegraded":"2024-03-18T15:15:18.022863+0000","last_fullsized":"2024-03-18T15:15:18.022863+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:29:22.823829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:17.599210+0000","last_change":"2024-03-18T15:15:10.678916+0000","last_active":"2024-03-18T15:15:17.599210+0000","last_peered":"2024-03-18T15:15:17.599210+0000","last_clean":"2024-03-18T15:15:17.599210+0000","last_became_active":"2024-03-18T15:15:10.678545+0000","last_became_peered":"2024-03-18T15:15:10.678545+0000","last_unstale":"2024-03-18T15:15:17.599210+0000","last_undegraded":"2024-03-18T15:15:17.599210+0000","last_fullsized":"2024-03-18T15:15:17.599210+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:13:52.714537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T15:15:18.023710+0000","last_change":"2024-03-18T15:15:10.660397+0000","last_active":"2024-03-18T15:15:18.023710+0000","last_peered":"2024-03-18T15:15:18.023710+0000","last_clean":"2024-03-18T15:15:18.023710+0000","last_became_active":"2024-03-18T15:15:10.660085+0000","last_became_peered":"2024-03-18T15:15:10.660085+0000","last_unstale":"2024-03-18T15:15:18.023710+0000","last_undegraded":"2024-03-18T15:15:18.023710+0000","last_fullsized":"2024-03-18T15:15:18.023710+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:13:04.591062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3542208,"data_compressed":210616,"data_compressed_allocated":270336,"data_compressed_original":3506176,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":27395,"num_objects":70,"num_object_clones":0,"num_object_copies":210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70,"num_whiteouts":0,"num_read":6,"num_read_kb":26,"num_write":161,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":47,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":88804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":141,"ondisk_log_size":141,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383701,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":978944,"data_stored":482503,"data_compressed":14671,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645337,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28064,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717088,"statfs":{"total":95995035648,"available":95966298112,"internally_reserved":0,"allocated":1073152,"data_stored":1079727,"data_compressed":50322,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808862,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32136,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713016,"statfs":{"total":95995035648,"available":95962128384,"internally_reserved":0,"allocated":983040,"data_stored":478033,"data_compressed":14671,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070502,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32136,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713016,"statfs":{"total":95995035648,"available":95962128384,"internally_reserved":0,"allocated":983040,"data_stored":477953,"data_compressed":14671,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332138,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28020,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717132,"statfs":{"total":95995035648,"available":95966343168,"internally_reserved":0,"allocated":1028096,"data_stored":1069375,"data_compressed":50322,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495664,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32116,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713036,"statfs":{"total":95995035648,"available":95962148864,"internally_reserved":0,"allocated":962560,"data_stored":477215,"data_compressed":14671,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574925,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32160,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712992,"statfs":{"total":95995035648,"available":95962103808,"internally_reserved":0,"allocated":1007616,"data_stored":502108,"data_compressed":14671,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738450,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":1003520,"data_stored":479393,"data_compressed":14671,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411400,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1015808,"data_stored":489629,"data_compressed":14671,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182466,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":991232,"data_stored":477645,"data_compressed":14671,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920829,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32128,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713024,"statfs":{"total":95995035648,"available":95962136576,"internally_reserved":0,"allocated":974848,"data_stored":477859,"data_compressed":14671,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757302,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28020,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717132,"statfs":{"total":95995035648,"available":95966343168,"internally_reserved":0,"allocated":1024000,"data_stored":1067765,"data_compressed":50322,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34006,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34006,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35651,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35651,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35651,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35651,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":42223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":12414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":24877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T15:15:50.054 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-18T15:15:50.055 INFO:tasks.cephadm:Setting up client nodes... 2024-03-18T15:15:50.055 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-18T15:15:50.569 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:15:50.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:50 smithi104 ceph-mon[36070]: from='client.24845 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T15:15:50.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:50 smithi104 ceph-mon[36070]: from='client.24865 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T15:15:50.958 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:50 smithi019 ceph-mon[31522]: from='client.24845 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T15:15:50.958 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:50 smithi019 ceph-mon[31522]: from='client.24865 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T15:15:51.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:50 smithi033 ceph-mon[36648]: from='client.24845 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T15:15:51.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:50 smithi033 ceph-mon[36648]: from='client.24865 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T15:15:51.774 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:51 smithi019 ceph-mon[31522]: pgmap v337: 129 pgs: 129 active+clean; 604 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5.5 KiB/s wr, 12 op/s 2024-03-18T15:15:51.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:51 smithi104 ceph-mon[36070]: pgmap v337: 129 pgs: 129 active+clean; 604 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5.5 KiB/s wr, 12 op/s 2024-03-18T15:15:52.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:51 smithi033 ceph-mon[36648]: pgmap v337: 129 pgs: 129 active+clean; 604 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5.5 KiB/s wr, 12 op/s 2024-03-18T15:15:52.930 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:52.929+0000 7fa9d3e00700 1 -- 172.21.15.19:0/2182621687 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9cc076300 msgr2=0x7fa9cc074760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:52.930 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:52.929+0000 7fa9d3e00700 1 --2- 172.21.15.19:0/2182621687 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9cc076300 0x7fa9cc074760 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fa9c0009990 tx=0x7fa9c0009ca0 comp rx=0 tx=0).stop 2024-03-18T15:15:52.930 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:52.929+0000 7fa9d3e00700 1 -- 172.21.15.19:0/2182621687 shutdown_connections 2024-03-18T15:15:52.930 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:52.929+0000 7fa9d3e00700 1 --2- 172.21.15.19:0/2182621687 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9cc076300 0x7fa9cc074760 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:52.931 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:52.929+0000 7fa9d3e00700 1 --2- 172.21.15.19:0/2182621687 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa9cc10a1f0 0x7fa9cc10c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:52.931 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:52.929+0000 7fa9d3e00700 1 --2- 172.21.15.19:0/2182621687 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa9cc074d30 0x7fa9cc0751b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:52.931 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:52.929+0000 7fa9d3e00700 1 -- 172.21.15.19:0/2182621687 >> 172.21.15.19:0/2182621687 conn(0x7fa9cc0fd8e0 msgr2=0x7fa9cc0ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:52.931 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:52.929+0000 7fa9d3e00700 1 -- 172.21.15.19:0/2182621687 shutdown_connections 2024-03-18T15:15:52.931 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:52.929+0000 7fa9d3e00700 1 -- 172.21.15.19:0/2182621687 wait complete. 2024-03-18T15:15:52.933 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:52.930+0000 7fa9d3e00700 1 Processor -- start 2024-03-18T15:15:52.933 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:52.931+0000 7fa9d3e00700 1 -- start start 2024-03-18T15:15:52.933 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:52.931+0000 7fa9d3e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa9cc074d30 0x7fa9cc19a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:52.933 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:52.931+0000 7fa9d3e00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa9cc076300 0x7fa9cc19ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:52.933 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:52.931+0000 7fa9d3e00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9cc10a1f0 0x7fa9cc1a1c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:52.933 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:52.931+0000 7fa9d3e00700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fa9cc110160 con 0x7fa9cc076300 2024-03-18T15:15:52.933 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:52.931+0000 7fa9d3e00700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa9cc10ffe0 con 0x7fa9cc074d30 2024-03-18T15:15:52.933 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:52.931+0000 7fa9d3e00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fa9cc1102e0 con 0x7fa9cc10a1f0 2024-03-18T15:15:52.933 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:52.931+0000 7fa9d1400700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa9cc074d30 0x7fa9cc19a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:52.934 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:52.931+0000 7fa9d1400700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa9cc074d30 0x7fa9cc19a6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:53494/0 (socket says 172.21.15.19:53494) 2024-03-18T15:15:52.934 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:52.931+0000 7fa9d1400700 1 -- 172.21.15.19:0/1125907044 learned_addr learned my addr 172.21.15.19:0/1125907044 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:15:52.934 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:52.931+0000 7fa9d1e00700 1 --2- 172.21.15.19:0/1125907044 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9cc10a1f0 0x7fa9cc1a1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:52.934 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:52.931+0000 7fa9d0a00700 1 --2- 172.21.15.19:0/1125907044 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa9cc076300 0x7fa9cc19ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:52.934 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:52.932+0000 7fa9d1400700 1 -- 172.21.15.19:0/1125907044 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9cc10a1f0 msgr2=0x7fa9cc1a1c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:52.934 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:52.932+0000 7fa9d1400700 1 --2- 172.21.15.19:0/1125907044 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9cc10a1f0 0x7fa9cc1a1c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:52.934 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:52.932+0000 7fa9d1400700 1 -- 172.21.15.19:0/1125907044 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa9cc076300 msgr2=0x7fa9cc19ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:52.934 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:52.932+0000 7fa9d1400700 1 --2- 172.21.15.19:0/1125907044 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa9cc076300 0x7fa9cc19ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:52.934 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:52.932+0000 7fa9d1400700 1 -- 172.21.15.19:0/1125907044 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9cc1a2320 con 0x7fa9cc074d30 2024-03-18T15:15:52.934 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:52.932+0000 7fa9d1400700 1 --2- 172.21.15.19:0/1125907044 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa9cc074d30 0x7fa9cc19a6c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fa9c000b8e0 tx=0x7fa9c000bbb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:52.934 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:52.932+0000 7fa9ca000700 1 -- 172.21.15.19:0/1125907044 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9c0026070 con 0x7fa9cc074d30 2024-03-18T15:15:52.934 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:52.932+0000 7fa9ca000700 1 -- 172.21.15.19:0/1125907044 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa9c000f460 con 0x7fa9cc074d30 2024-03-18T15:15:52.934 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:52.932+0000 7fa9d3e00700 1 -- 172.21.15.19:0/1125907044 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9cc1a25b0 con 0x7fa9cc074d30 2024-03-18T15:15:52.935 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:52.932+0000 7fa9ca000700 1 -- 172.21.15.19:0/1125907044 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9c0021670 con 0x7fa9cc074d30 2024-03-18T15:15:52.935 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:52.932+0000 7fa9d3e00700 1 -- 172.21.15.19:0/1125907044 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9cc1a2a60 con 0x7fa9cc074d30 2024-03-18T15:15:52.935 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:52.934+0000 7fa9d3e00700 1 -- 172.21.15.19:0/1125907044 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9cc0507e0 con 0x7fa9cc074d30 2024-03-18T15:15:52.935 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:52.935+0000 7fa9ca000700 1 -- 172.21.15.19:0/1125907044 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fa9c0029070 con 0x7fa9cc074d30 2024-03-18T15:15:52.937 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:52.937+0000 7fa9ca000700 1 --2- 172.21.15.19:0/1125907044 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fa9b40a2a50 0x7fa9b40a4f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:52.937 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:52.937+0000 7fa9ca000700 1 -- 172.21.15.19:0/1125907044 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fa9c00cfe20 con 0x7fa9cc074d30 2024-03-18T15:15:52.937 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:52.937+0000 7fa9d0a00700 1 --2- 172.21.15.19:0/1125907044 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fa9b40a2a50 0x7fa9b40a4f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:52.938 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:52.938+0000 7fa9d0a00700 1 --2- 172.21.15.19:0/1125907044 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fa9b40a2a50 0x7fa9b40a4f10 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa9b8005950 tx=0x7fa9b80058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:52.941 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:52.941+0000 7fa9ca000700 1 -- 172.21.15.19:0/1125907044 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa9c005d740 con 0x7fa9cc074d30 2024-03-18T15:15:53.260 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:53.260+0000 7fa9d3e00700 1 -- 172.21.15.19:0/1125907044 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7fa9cc02dc10 con 0x7fa9cc074d30 2024-03-18T15:15:53.270 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:53.269+0000 7fa9ca000700 1 -- 172.21.15.19:0/1125907044 <== mon.2 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7fa9c0013070 con 0x7fa9cc074d30 2024-03-18T15:15:53.270 INFO:teuthology.orchestra.run.smithi019.stdout:[client.0] 2024-03-18T15:15:53.270 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQCpWvhlAWqiDxAAEruNnJJpUZGmZj6pMB91Gw== 2024-03-18T15:15:53.272 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:53.272+0000 7fa9d3e00700 1 -- 172.21.15.19:0/1125907044 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fa9b40a2a50 msgr2=0x7fa9b40a4f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:53.272 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:53.272+0000 7fa9d3e00700 1 --2- 172.21.15.19:0/1125907044 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fa9b40a2a50 0x7fa9b40a4f10 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa9b8005950 tx=0x7fa9b80058e0 comp rx=0 tx=0).stop 2024-03-18T15:15:53.273 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:53.272+0000 7fa9d3e00700 1 -- 172.21.15.19:0/1125907044 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa9cc074d30 msgr2=0x7fa9cc19a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:53.273 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:53.272+0000 7fa9d3e00700 1 --2- 172.21.15.19:0/1125907044 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa9cc074d30 0x7fa9cc19a6c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fa9c000b8e0 tx=0x7fa9c000bbb0 comp rx=0 tx=0).stop 2024-03-18T15:15:53.273 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:53.272+0000 7fa9d3e00700 1 -- 172.21.15.19:0/1125907044 shutdown_connections 2024-03-18T15:15:53.273 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:53.272+0000 7fa9d3e00700 1 --2- 172.21.15.19:0/1125907044 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fa9b40a2a50 0x7fa9b40a4f10 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:53.273 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:53.272+0000 7fa9d3e00700 1 --2- 172.21.15.19:0/1125907044 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9cc10a1f0 0x7fa9cc1a1c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:53.273 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:53.272+0000 7fa9d3e00700 1 --2- 172.21.15.19:0/1125907044 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa9cc074d30 0x7fa9cc19a6c0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:53.273 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:53.272+0000 7fa9d3e00700 1 --2- 172.21.15.19:0/1125907044 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa9cc076300 0x7fa9cc19ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:53.274 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:53.272+0000 7fa9d3e00700 1 -- 172.21.15.19:0/1125907044 >> 172.21.15.19:0/1125907044 conn(0x7fa9cc0fd8e0 msgr2=0x7fa9cc1060f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:53.274 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:53.273+0000 7fa9d3e00700 1 -- 172.21.15.19:0/1125907044 shutdown_connections 2024-03-18T15:15:53.274 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:53.273+0000 7fa9d3e00700 1 -- 172.21.15.19:0/1125907044 wait complete. 2024-03-18T15:15:53.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:53 smithi104 ceph-mon[36070]: pgmap v338: 129 pgs: 129 active+clean; 628 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 8.4 KiB/s wr, 21 op/s 2024-03-18T15:15:53.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:53 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/1125907044' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-18T15:15:53.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:53 smithi104 ceph-mon[36070]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-18T15:15:53.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:53 smithi104 ceph-mon[36070]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-18T15:15:53.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:53 smithi019 ceph-mon[31522]: pgmap v338: 129 pgs: 129 active+clean; 628 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 8.4 KiB/s wr, 21 op/s 2024-03-18T15:15:53.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:53 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/1125907044' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-18T15:15:53.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:53 smithi019 ceph-mon[31522]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-18T15:15:53.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:53 smithi019 ceph-mon[31522]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-18T15:15:54.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:53 smithi033 ceph-mon[36648]: pgmap v338: 129 pgs: 129 active+clean; 628 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 8.4 KiB/s wr, 21 op/s 2024-03-18T15:15:54.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:53 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/1125907044' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-18T15:15:54.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:53 smithi033 ceph-mon[36648]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-18T15:15:54.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:53 smithi033 ceph-mon[36648]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-18T15:15:54.104 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-03-18T15:15:54.104 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-18T15:15:54.104 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-18T15:15:54.275 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-18T15:15:54.275 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-18T15:15:54.275 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 -- ceph mgr dump --format=json 2024-03-18T15:15:54.787 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:15:55.828 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:55 smithi019 ceph-mon[31522]: pgmap v339: 129 pgs: 129 active+clean; 628 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 8.4 KiB/s wr, 22 op/s 2024-03-18T15:15:55.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:55 smithi104 ceph-mon[36070]: pgmap v339: 129 pgs: 129 active+clean; 628 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 8.4 KiB/s wr, 22 op/s 2024-03-18T15:15:56.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:55 smithi033 ceph-mon[36648]: pgmap v339: 129 pgs: 129 active+clean; 628 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 8.4 KiB/s wr, 22 op/s 2024-03-18T15:15:57.156 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:57.155+0000 7fab9d200700 1 -- 172.21.15.19:0/4160951424 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab98103c90 msgr2=0x7fab9810c180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:57.156 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:57.155+0000 7fab9d200700 1 --2- 172.21.15.19:0/4160951424 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab98103c90 0x7fab9810c180 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fab80009990 tx=0x7fab80009ca0 comp rx=0 tx=0).stop 2024-03-18T15:15:57.156 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:57.155+0000 7fab9d200700 1 -- 172.21.15.19:0/4160951424 shutdown_connections 2024-03-18T15:15:57.156 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:57.155+0000 7fab9d200700 1 --2- 172.21.15.19:0/4160951424 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fab9810c7a0 0x7fab9810ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:57.156 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:57.155+0000 7fab9d200700 1 --2- 172.21.15.19:0/4160951424 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fab98103370 0x7fab98103750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:57.157 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:57.155+0000 7fab9d200700 1 --2- 172.21.15.19:0/4160951424 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab98103c90 0x7fab9810c180 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:57.157 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:57.155+0000 7fab9d200700 1 -- 172.21.15.19:0/4160951424 >> 172.21.15.19:0/4160951424 conn(0x7fab980fcba0 msgr2=0x7fab980fefc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:57.157 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:57.155+0000 7fab9d200700 1 -- 172.21.15.19:0/4160951424 shutdown_connections 2024-03-18T15:15:57.157 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:57.156+0000 7fab9d200700 1 -- 172.21.15.19:0/4160951424 wait complete. 2024-03-18T15:15:57.157 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:57.157+0000 7fab9d200700 1 Processor -- start 2024-03-18T15:15:57.157 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:57.157+0000 7fab9d200700 1 -- start start 2024-03-18T15:15:57.158 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:57.158+0000 7fab9d200700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab98103370 0x7fab981a0830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:57.158 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:57.158+0000 7fab9d200700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fab98103c90 0x7fab981a0d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:57.158 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:57.158+0000 7fab9d200700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fab9810c7a0 0x7fab9819a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:57.158 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:57.158+0000 7fab9d200700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fab98112510 con 0x7fab9810c7a0 2024-03-18T15:15:57.158 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:57.158+0000 7fab9d200700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fab98112390 con 0x7fab98103370 2024-03-18T15:15:57.158 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:57.158+0000 7fab9d200700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fab98112690 con 0x7fab98103c90 2024-03-18T15:15:57.160 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:57.158+0000 7fab8fe00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fab98103c90 0x7fab981a0d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:57.160 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:57.158+0000 7fab97400700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab98103370 0x7fab981a0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:57.160 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:57.158+0000 7fab8fe00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fab98103c90 0x7fab981a0d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.19:53380/0 (socket says 172.21.15.19:53380) 2024-03-18T15:15:57.160 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:57.158+0000 7fab8fe00700 1 -- 172.21.15.19:0/2223338097 learned_addr learned my addr 172.21.15.19:0/2223338097 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:15:57.160 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:57.158+0000 7fab97e00700 1 --2- 172.21.15.19:0/2223338097 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fab9810c7a0 0x7fab9819a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:57.160 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:57.159+0000 7fab97400700 1 -- 172.21.15.19:0/2223338097 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fab98103c90 msgr2=0x7fab981a0d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:57.160 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:57.159+0000 7fab97400700 1 --2- 172.21.15.19:0/2223338097 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fab98103c90 0x7fab981a0d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:57.160 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:57.159+0000 7fab97400700 1 -- 172.21.15.19:0/2223338097 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fab9810c7a0 msgr2=0x7fab9819a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:57.160 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:57.159+0000 7fab97400700 1 --2- 172.21.15.19:0/2223338097 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fab9810c7a0 0x7fab9819a9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:57.160 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:57.159+0000 7fab97400700 1 -- 172.21.15.19:0/2223338097 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab9819b150 con 0x7fab98103370 2024-03-18T15:15:57.160 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:57.159+0000 7fab8fe00700 1 --2- 172.21.15.19:0/2223338097 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fab98103c90 0x7fab981a0d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:15:57.161 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:57.159+0000 7fab97400700 1 --2- 172.21.15.19:0/2223338097 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab98103370 0x7fab981a0830 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fab8800e9f0 tx=0x7fab8800ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:57.161 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:57.159+0000 7fab94c00700 1 -- 172.21.15.19:0/2223338097 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab8800c9c0 con 0x7fab98103370 2024-03-18T15:15:57.161 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:57.159+0000 7fab94c00700 1 -- 172.21.15.19:0/2223338097 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab8800cb60 con 0x7fab98103370 2024-03-18T15:15:57.161 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:57.159+0000 7fab9d200700 1 -- 172.21.15.19:0/2223338097 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab9819b440 con 0x7fab98103370 2024-03-18T15:15:57.161 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:57.159+0000 7fab94c00700 1 -- 172.21.15.19:0/2223338097 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab88018680 con 0x7fab98103370 2024-03-18T15:15:57.161 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:57.159+0000 7fab9d200700 1 -- 172.21.15.19:0/2223338097 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab981a7700 con 0x7fab98103370 2024-03-18T15:15:57.161 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:57.161+0000 7fab9d200700 1 -- 172.21.15.19:0/2223338097 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab98068f90 con 0x7fab98103370 2024-03-18T15:15:57.162 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:57.162+0000 7fab94c00700 1 -- 172.21.15.19:0/2223338097 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fab88010430 con 0x7fab98103370 2024-03-18T15:15:57.164 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:57.164+0000 7fab94c00700 1 --2- 172.21.15.19:0/2223338097 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fab7c0a29a0 0x7fab7c0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:15:57.164 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:57.164+0000 7fab94c00700 1 -- 172.21.15.19:0/2223338097 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fab880cef50 con 0x7fab98103370 2024-03-18T15:15:57.164 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:57.164+0000 7fab8fe00700 1 --2- 172.21.15.19:0/2223338097 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fab7c0a29a0 0x7fab7c0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:15:57.164 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:57.165+0000 7fab8fe00700 1 --2- 172.21.15.19:0/2223338097 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fab7c0a29a0 0x7fab7c0a4e60 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fab9810e000 tx=0x7fab80005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:15:57.172 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:57.172+0000 7fab94c00700 1 -- 172.21.15.19:0/2223338097 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fab8809bbf0 con 0x7fab98103370 2024-03-18T15:15:57.461 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:57.460+0000 7fab9d200700 1 -- 172.21.15.19:0/2223338097 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7fab981a7930 con 0x7fab98103370 2024-03-18T15:15:57.467 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:57.467+0000 7fab94c00700 1 -- 172.21.15.19:0/2223338097 <== mon.2 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+262659 (secure 0 0 0) 0x7fab8809b4f0 con 0x7fab98103370 2024-03-18T15:15:57.473 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-18T15:15:57.475 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:57.474+0000 7fab9d200700 1 -- 172.21.15.19:0/2223338097 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fab7c0a29a0 msgr2=0x7fab7c0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:57.475 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:57.474+0000 7fab9d200700 1 --2- 172.21.15.19:0/2223338097 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fab7c0a29a0 0x7fab7c0a4e60 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fab9810e000 tx=0x7fab80005fd0 comp rx=0 tx=0).stop 2024-03-18T15:15:57.475 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:57.474+0000 7fab9d200700 1 -- 172.21.15.19:0/2223338097 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab98103370 msgr2=0x7fab981a0830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:15:57.475 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:57.474+0000 7fab9d200700 1 --2- 172.21.15.19:0/2223338097 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab98103370 0x7fab981a0830 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fab8800e9f0 tx=0x7fab8800ed00 comp rx=0 tx=0).stop 2024-03-18T15:15:57.475 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:57.474+0000 7fab9d200700 1 -- 172.21.15.19:0/2223338097 shutdown_connections 2024-03-18T15:15:57.475 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:57.474+0000 7fab9d200700 1 --2- 172.21.15.19:0/2223338097 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fab7c0a29a0 0x7fab7c0a4e60 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:57.475 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:57.474+0000 7fab9d200700 1 --2- 172.21.15.19:0/2223338097 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fab9810c7a0 0x7fab9819a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:57.475 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:57.474+0000 7fab9d200700 1 --2- 172.21.15.19:0/2223338097 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fab98103370 0x7fab981a0830 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:57.475 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:57.474+0000 7fab9d200700 1 --2- 172.21.15.19:0/2223338097 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fab98103c90 0x7fab981a0d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:15:57.475 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:57.474+0000 7fab9d200700 1 -- 172.21.15.19:0/2223338097 >> 172.21.15.19:0/2223338097 conn(0x7fab980fcba0 msgr2=0x7fab980fef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:15:57.475 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:57.474+0000 7fab9d200700 1 -- 172.21.15.19:0/2223338097 shutdown_connections 2024-03-18T15:15:57.476 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:15:57.474+0000 7fab9d200700 1 -- 172.21.15.19:0/2223338097 wait complete. 2024-03-18T15:15:57.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:57 smithi019 ceph-mon[31522]: pgmap v340: 129 pgs: 129 active+clean; 628 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 8.2 KiB/s wr, 24 op/s 2024-03-18T15:15:57.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:57 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/2223338097' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-18T15:15:57.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:57 smithi104 ceph-mon[36070]: pgmap v340: 129 pgs: 129 active+clean; 628 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 8.2 KiB/s wr, 24 op/s 2024-03-18T15:15:57.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:57 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/2223338097' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-18T15:15:58.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:57 smithi033 ceph-mon[36648]: pgmap v340: 129 pgs: 129 active+clean; 628 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 8.2 KiB/s wr, 24 op/s 2024-03-18T15:15:58.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:57 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/2223338097' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-18T15:15:58.268 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6800","nonce":2017714236},{"type":"v1","addr":"172.21.15.19:6801","nonce":2017714236}]},"active_addr":"172.21.15.19:6801/2017714236","active_change":"2024-03-18T15:07:14.151203+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24109,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24110,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.19:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.19:0","nonce":1024556841}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.19:0","nonce":84203799}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.19:0","nonce":2353244552}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.19:0","nonce":2730531676}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.19:0","nonce":1059506598}]}]} 2024-03-18T15:15:58.274 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-18T15:15:58.274 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-18T15:15:58.274 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 -- ceph osd dump --format=json 2024-03-18T15:15:58.719 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:15:59.799 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:15:59 smithi019 ceph-mon[31522]: pgmap v341: 129 pgs: 129 active+clean; 640 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.9 KiB/s wr, 22 op/s 2024-03-18T15:15:59.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:15:59 smithi104 ceph-mon[36070]: pgmap v341: 129 pgs: 129 active+clean; 640 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.9 KiB/s wr, 22 op/s 2024-03-18T15:16:00.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:15:59 smithi033 ceph-mon[36648]: pgmap v341: 129 pgs: 129 active+clean; 640 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.9 KiB/s wr, 22 op/s 2024-03-18T15:16:01.015 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:01.014+0000 7f615ee00700 1 -- 172.21.15.19:0/2309277157 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6158104e90 msgr2=0x7f6158111350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:01.015 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:01.014+0000 7f615ee00700 1 --2- 172.21.15.19:0/2309277157 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6158104e90 0x7f6158111350 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f615000b0d0 tx=0x7f615000b3e0 comp rx=0 tx=0).stop 2024-03-18T15:16:01.015 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:01.014+0000 7f615ee00700 1 -- 172.21.15.19:0/2309277157 shutdown_connections 2024-03-18T15:16:01.015 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:01.014+0000 7f615ee00700 1 --2- 172.21.15.19:0/2309277157 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6158104e90 0x7f6158111350 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:01.015 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:01.014+0000 7f615ee00700 1 --2- 172.21.15.19:0/2309277157 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f615810a580 0x7f615810a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:01.015 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:01.014+0000 7f615ee00700 1 --2- 172.21.15.19:0/2309277157 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f61581044d0 0x7f6158104950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:01.015 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:01.014+0000 7f615ee00700 1 -- 172.21.15.19:0/2309277157 >> 172.21.15.19:0/2309277157 conn(0x7f61580fffe0 msgr2=0x7f6158102400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:16:01.015 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:01.014+0000 7f615ee00700 1 -- 172.21.15.19:0/2309277157 shutdown_connections 2024-03-18T15:16:01.015 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:01.014+0000 7f615ee00700 1 -- 172.21.15.19:0/2309277157 wait complete. 2024-03-18T15:16:01.017 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:01.015+0000 7f615ee00700 1 Processor -- start 2024-03-18T15:16:01.017 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:01.015+0000 7f615ee00700 1 -- start start 2024-03-18T15:16:01.017 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:01.016+0000 7f615ee00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f61581044d0 0x7f61581a0810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:16:01.017 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:01.016+0000 7f615ee00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6158104e90 0x7f61581a0d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:16:01.017 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:01.016+0000 7f615ee00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f615810a580 0x7f615819a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:16:01.017 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:01.016+0000 7f615ee00700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f6158114810 con 0x7f615810a580 2024-03-18T15:16:01.017 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:01.016+0000 7f615ee00700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6158114690 con 0x7f61581044d0 2024-03-18T15:16:01.017 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:01.016+0000 7f615ee00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f6158114990 con 0x7f6158104e90 2024-03-18T15:16:01.017 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:01.016+0000 7f6157e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f61581044d0 0x7f61581a0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:16:01.018 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:01.016+0000 7f6157e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f61581044d0 0x7f61581a0810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:53542/0 (socket says 172.21.15.19:53542) 2024-03-18T15:16:01.018 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:01.016+0000 7f6157e00700 1 -- 172.21.15.19:0/1972539963 learned_addr learned my addr 172.21.15.19:0/1972539963 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:16:01.018 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:01.016+0000 7f614f400700 1 --2- 172.21.15.19:0/1972539963 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6158104e90 0x7f61581a0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:16:01.018 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:01.016+0000 7f615ce00700 1 --2- 172.21.15.19:0/1972539963 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f615810a580 0x7f615819a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:16:01.018 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:01.017+0000 7f6157e00700 1 -- 172.21.15.19:0/1972539963 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6158104e90 msgr2=0x7f61581a0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:01.018 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:01.017+0000 7f6157e00700 1 --2- 172.21.15.19:0/1972539963 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6158104e90 0x7f61581a0d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:01.018 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:01.017+0000 7f6157e00700 1 -- 172.21.15.19:0/1972539963 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f615810a580 msgr2=0x7f615819a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:01.018 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:01.017+0000 7f6157e00700 1 --2- 172.21.15.19:0/1972539963 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f615810a580 0x7f615819a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:01.018 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:01.017+0000 7f6157e00700 1 -- 172.21.15.19:0/1972539963 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f615819b130 con 0x7f61581044d0 2024-03-18T15:16:01.018 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:01.017+0000 7f615ce00700 1 --2- 172.21.15.19:0/1972539963 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f615810a580 0x7f615819a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T15:16:01.018 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:01.017+0000 7f6157e00700 1 --2- 172.21.15.19:0/1972539963 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f61581044d0 0x7f61581a0810 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f614800eb30 tx=0x7f614800ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:16:01.019 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:01.017+0000 7f6155600700 1 -- 172.21.15.19:0/1972539963 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f614800cc90 con 0x7f61581044d0 2024-03-18T15:16:01.019 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:01.017+0000 7f6155600700 1 -- 172.21.15.19:0/1972539963 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6148004500 con 0x7f61581044d0 2024-03-18T15:16:01.019 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:01.017+0000 7f6155600700 1 -- 172.21.15.19:0/1972539963 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6148010430 con 0x7f61581044d0 2024-03-18T15:16:01.020 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:01.017+0000 7f615ee00700 1 -- 172.21.15.19:0/1972539963 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f615819b420 con 0x7f61581044d0 2024-03-18T15:16:01.020 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:01.017+0000 7f615ee00700 1 -- 172.21.15.19:0/1972539963 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61581a76e0 con 0x7f61581044d0 2024-03-18T15:16:01.020 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:01.019+0000 7f615ee00700 1 -- 172.21.15.19:0/1972539963 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6158068f90 con 0x7f61581044d0 2024-03-18T15:16:01.020 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:01.021+0000 7f6155600700 1 -- 172.21.15.19:0/1972539963 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f6148003f70 con 0x7f61581044d0 2024-03-18T15:16:01.022 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:01.022+0000 7f6155600700 1 --2- 172.21.15.19:0/1972539963 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f613c0a29a0 0x7f613c0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:16:01.022 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:01.022+0000 7f6155600700 1 -- 172.21.15.19:0/1972539963 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f61480cf150 con 0x7f61581044d0 2024-03-18T15:16:01.022 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:01.022+0000 7f614f400700 1 --2- 172.21.15.19:0/1972539963 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f613c0a29a0 0x7f613c0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:16:01.023 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:01.023+0000 7f614f400700 1 --2- 172.21.15.19:0/1972539963 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f613c0a29a0 0x7f613c0a4e60 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f6158109af0 tx=0x7f6140009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:16:01.026 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:01.026+0000 7f6155600700 1 -- 172.21.15.19:0/1972539963 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f614809bdf0 con 0x7f61581044d0 2024-03-18T15:16:01.256 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:01.255+0000 7f615ee00700 1 -- 172.21.15.19:0/1972539963 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f61581a79c0 con 0x7f61581044d0 2024-03-18T15:16:01.258 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:01.258+0000 7f6155600700 1 -- 172.21.15.19:0/1972539963 <== mon.2 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24093 (secure 0 0 0) 0x7f614809b6f0 con 0x7f61581044d0 2024-03-18T15:16:01.259 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-18T15:16:01.259 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":89,"fsid":"d4dc2ba6-e538-11ee-95c9-87774f69a715","created":"2024-03-18T15:05:57.538551+0000","modified":"2024-03-18T15:15:15.582418+0000","last_up_change":"2024-03-18T15:14:12.009522+0000","last_in_change":"2024-03-18T15:13:52.555387+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T15:09:56.843349+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T15:15:08.941047+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T15:15:10.044671+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T15:15:13.621916+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"37b5a5cd-27ae-4e77-9b05-774b817b2ee9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6802","nonce":402333948},{"type":"v1","addr":"172.21.15.19:6803","nonce":402333948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6804","nonce":402333948},{"type":"v1","addr":"172.21.15.19:6805","nonce":402333948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6808","nonce":402333948},{"type":"v1","addr":"172.21.15.19:6809","nonce":402333948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6806","nonce":402333948},{"type":"v1","addr":"172.21.15.19:6807","nonce":402333948}]},"public_addr":"172.21.15.19:6803/402333948","cluster_addr":"172.21.15.19:6805/402333948","heartbeat_back_addr":"172.21.15.19:6809/402333948","heartbeat_front_addr":"172.21.15.19:6807/402333948","state":["exists","up"]},{"osd":1,"uuid":"3a1134e8-ede7-4dbd-9be1-eefd2cbb43bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1951880113},{"type":"v1","addr":"172.21.15.33:6801","nonce":1951880113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1951880113},{"type":"v1","addr":"172.21.15.33:6803","nonce":1951880113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1951880113},{"type":"v1","addr":"172.21.15.33:6807","nonce":1951880113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1951880113},{"type":"v1","addr":"172.21.15.33:6805","nonce":1951880113}]},"public_addr":"172.21.15.33:6801/1951880113","cluster_addr":"172.21.15.33:6803/1951880113","heartbeat_back_addr":"172.21.15.33:6807/1951880113","heartbeat_front_addr":"172.21.15.33:6805/1951880113","state":["exists","up"]},{"osd":2,"uuid":"f383a164-a3a0-49fd-a9e8-9c2178d8cfb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":1004727949},{"type":"v1","addr":"172.21.15.104:6801","nonce":1004727949}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":1004727949},{"type":"v1","addr":"172.21.15.104:6803","nonce":1004727949}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":1004727949},{"type":"v1","addr":"172.21.15.104:6807","nonce":1004727949}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":1004727949},{"type":"v1","addr":"172.21.15.104:6805","nonce":1004727949}]},"public_addr":"172.21.15.104:6801/1004727949","cluster_addr":"172.21.15.104:6803/1004727949","heartbeat_back_addr":"172.21.15.104:6807/1004727949","heartbeat_front_addr":"172.21.15.104:6805/1004727949","state":["exists","up"]},{"osd":3,"uuid":"c409e116-28bf-45dd-a791-539af22abe1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6810","nonce":3571525854},{"type":"v1","addr":"172.21.15.19:6811","nonce":3571525854}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6812","nonce":3571525854},{"type":"v1","addr":"172.21.15.19:6813","nonce":3571525854}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6816","nonce":3571525854},{"type":"v1","addr":"172.21.15.19:6817","nonce":3571525854}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6814","nonce":3571525854},{"type":"v1","addr":"172.21.15.19:6815","nonce":3571525854}]},"public_addr":"172.21.15.19:6811/3571525854","cluster_addr":"172.21.15.19:6813/3571525854","heartbeat_back_addr":"172.21.15.19:6817/3571525854","heartbeat_front_addr":"172.21.15.19:6815/3571525854","state":["exists","up"]},{"osd":4,"uuid":"696eb719-0b3e-43ae-b5f1-ccd73a3b2843","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":624901749},{"type":"v1","addr":"172.21.15.33:6809","nonce":624901749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":624901749},{"type":"v1","addr":"172.21.15.33:6811","nonce":624901749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":624901749},{"type":"v1","addr":"172.21.15.33:6815","nonce":624901749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":624901749},{"type":"v1","addr":"172.21.15.33:6813","nonce":624901749}]},"public_addr":"172.21.15.33:6809/624901749","cluster_addr":"172.21.15.33:6811/624901749","heartbeat_back_addr":"172.21.15.33:6815/624901749","heartbeat_front_addr":"172.21.15.33:6813/624901749","state":["exists","up"]},{"osd":5,"uuid":"21789779-8bb9-42c7-b9ee-437985f11e3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":1258208297},{"type":"v1","addr":"172.21.15.104:6809","nonce":1258208297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":1258208297},{"type":"v1","addr":"172.21.15.104:6811","nonce":1258208297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":1258208297},{"type":"v1","addr":"172.21.15.104:6815","nonce":1258208297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":1258208297},{"type":"v1","addr":"172.21.15.104:6813","nonce":1258208297}]},"public_addr":"172.21.15.104:6809/1258208297","cluster_addr":"172.21.15.104:6811/1258208297","heartbeat_back_addr":"172.21.15.104:6815/1258208297","heartbeat_front_addr":"172.21.15.104:6813/1258208297","state":["exists","up"]},{"osd":6,"uuid":"0e92a2ef-43cd-487c-9203-e219158aef8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6818","nonce":2782482301},{"type":"v1","addr":"172.21.15.19:6819","nonce":2782482301}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6820","nonce":2782482301},{"type":"v1","addr":"172.21.15.19:6821","nonce":2782482301}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6824","nonce":2782482301},{"type":"v1","addr":"172.21.15.19:6825","nonce":2782482301}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6822","nonce":2782482301},{"type":"v1","addr":"172.21.15.19:6823","nonce":2782482301}]},"public_addr":"172.21.15.19:6819/2782482301","cluster_addr":"172.21.15.19:6821/2782482301","heartbeat_back_addr":"172.21.15.19:6825/2782482301","heartbeat_front_addr":"172.21.15.19:6823/2782482301","state":["exists","up"]},{"osd":7,"uuid":"a16e39a1-3685-491d-b376-3e80a2084072","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":4238160988},{"type":"v1","addr":"172.21.15.33:6817","nonce":4238160988}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":4238160988},{"type":"v1","addr":"172.21.15.33:6819","nonce":4238160988}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":4238160988},{"type":"v1","addr":"172.21.15.33:6823","nonce":4238160988}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":4238160988},{"type":"v1","addr":"172.21.15.33:6821","nonce":4238160988}]},"public_addr":"172.21.15.33:6817/4238160988","cluster_addr":"172.21.15.33:6819/4238160988","heartbeat_back_addr":"172.21.15.33:6823/4238160988","heartbeat_front_addr":"172.21.15.33:6821/4238160988","state":["exists","up"]},{"osd":8,"uuid":"d5c3fa92-cd5d-4349-9c7d-d5666bacde16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":1014779813},{"type":"v1","addr":"172.21.15.104:6817","nonce":1014779813}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":1014779813},{"type":"v1","addr":"172.21.15.104:6819","nonce":1014779813}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":1014779813},{"type":"v1","addr":"172.21.15.104:6823","nonce":1014779813}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":1014779813},{"type":"v1","addr":"172.21.15.104:6821","nonce":1014779813}]},"public_addr":"172.21.15.104:6817/1014779813","cluster_addr":"172.21.15.104:6819/1014779813","heartbeat_back_addr":"172.21.15.104:6823/1014779813","heartbeat_front_addr":"172.21.15.104:6821/1014779813","state":["exists","up"]},{"osd":9,"uuid":"eed2e9b2-26d3-4c46-88ec-34b5596fe991","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6826","nonce":2641151009},{"type":"v1","addr":"172.21.15.19:6827","nonce":2641151009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6828","nonce":2641151009},{"type":"v1","addr":"172.21.15.19:6829","nonce":2641151009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6832","nonce":2641151009},{"type":"v1","addr":"172.21.15.19:6833","nonce":2641151009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6830","nonce":2641151009},{"type":"v1","addr":"172.21.15.19:6831","nonce":2641151009}]},"public_addr":"172.21.15.19:6827/2641151009","cluster_addr":"172.21.15.19:6829/2641151009","heartbeat_back_addr":"172.21.15.19:6833/2641151009","heartbeat_front_addr":"172.21.15.19:6831/2641151009","state":["exists","up"]},{"osd":10,"uuid":"f3c35ea5-fbe7-4dc4-93c5-b92d1a714db3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":1346058869},{"type":"v1","addr":"172.21.15.33:6825","nonce":1346058869}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":1346058869},{"type":"v1","addr":"172.21.15.33:6827","nonce":1346058869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":1346058869},{"type":"v1","addr":"172.21.15.33:6831","nonce":1346058869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":1346058869},{"type":"v1","addr":"172.21.15.33:6829","nonce":1346058869}]},"public_addr":"172.21.15.33:6825/1346058869","cluster_addr":"172.21.15.33:6827/1346058869","heartbeat_back_addr":"172.21.15.33:6831/1346058869","heartbeat_front_addr":"172.21.15.33:6829/1346058869","state":["exists","up"]},{"osd":11,"uuid":"c8b51571-f807-4b62-90b6-7529e09e59da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":3759498281},{"type":"v1","addr":"172.21.15.104:6825","nonce":3759498281}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":3759498281},{"type":"v1","addr":"172.21.15.104:6827","nonce":3759498281}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":3759498281},{"type":"v1","addr":"172.21.15.104:6831","nonce":3759498281}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":3759498281},{"type":"v1","addr":"172.21.15.104:6829","nonce":3759498281}]},"public_addr":"172.21.15.104:6825/3759498281","cluster_addr":"172.21.15.104:6827/3759498281","heartbeat_back_addr":"172.21.15.104:6831/3759498281","heartbeat_front_addr":"172.21.15.104:6829/3759498281","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:09:05.267795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:09:30.234068+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:09:53.870232+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:10:26.471296+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:10:53.180341+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:11:17.801361+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:11:51.364816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:12:18.306408+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:12:43.978744+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:13:18.213964+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:13:45.437544+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:14:10.148580+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.19:0/3860783274":"2024-03-19T15:07:14.150983+0000","172.21.15.19:0/2737223213":"2024-03-19T15:07:14.150983+0000","172.21.15.19:0/176699025":"2024-03-19T15:06:38.688405+0000","172.21.15.19:0/2316969005":"2024-03-19T15:07:14.150983+0000","172.21.15.19:6801/2776008962":"2024-03-19T15:07:14.150983+0000","172.21.15.19:0/2793980440":"2024-03-19T15:06:38.688405+0000","172.21.15.19:6800/3264337803":"2024-03-19T15:06:38.688405+0000","172.21.15.19:6801/3264337803":"2024-03-19T15:06:38.688405+0000","172.21.15.19:0/3166705295":"2024-03-19T15:06:38.688405+0000","172.21.15.19:6800/2776008962":"2024-03-19T15:07:14.150983+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T15:16:01.262 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:01.261+0000 7f615ee00700 1 -- 172.21.15.19:0/1972539963 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f613c0a29a0 msgr2=0x7f613c0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:01.262 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:01.261+0000 7f615ee00700 1 --2- 172.21.15.19:0/1972539963 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f613c0a29a0 0x7f613c0a4e60 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f6158109af0 tx=0x7f6140009260 comp rx=0 tx=0).stop 2024-03-18T15:16:01.262 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:01.261+0000 7f615ee00700 1 -- 172.21.15.19:0/1972539963 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f61581044d0 msgr2=0x7f61581a0810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:01.262 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:01.261+0000 7f615ee00700 1 --2- 172.21.15.19:0/1972539963 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f61581044d0 0x7f61581a0810 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f614800eb30 tx=0x7f614800ee40 comp rx=0 tx=0).stop 2024-03-18T15:16:01.262 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:01.261+0000 7f615ee00700 1 -- 172.21.15.19:0/1972539963 shutdown_connections 2024-03-18T15:16:01.262 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:01.261+0000 7f615ee00700 1 --2- 172.21.15.19:0/1972539963 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f613c0a29a0 0x7f613c0a4e60 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:01.262 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:01.261+0000 7f615ee00700 1 --2- 172.21.15.19:0/1972539963 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f615810a580 0x7f615819a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:01.262 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:01.261+0000 7f615ee00700 1 --2- 172.21.15.19:0/1972539963 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f61581044d0 0x7f61581a0810 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:01.262 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:01.261+0000 7f615ee00700 1 --2- 172.21.15.19:0/1972539963 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6158104e90 0x7f61581a0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:01.262 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:01.261+0000 7f615ee00700 1 -- 172.21.15.19:0/1972539963 >> 172.21.15.19:0/1972539963 conn(0x7f61580fffe0 msgr2=0x7f6158101540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:16:01.262 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:01.262+0000 7f615ee00700 1 -- 172.21.15.19:0/1972539963 shutdown_connections 2024-03-18T15:16:01.262 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:01.262+0000 7f615ee00700 1 -- 172.21.15.19:0/1972539963 wait complete. 2024-03-18T15:16:01.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:16:01 smithi104 ceph-mon[36070]: pgmap v342: 129 pgs: 129 active+clean; 640 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 5.5 KiB/s wr, 18 op/s 2024-03-18T15:16:01.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:16:01 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/1972539963' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:16:01.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:16:01 smithi019 ceph-mon[31522]: pgmap v342: 129 pgs: 129 active+clean; 640 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 5.5 KiB/s wr, 18 op/s 2024-03-18T15:16:01.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:16:01 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/1972539963' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:16:02.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:16:01 smithi033 ceph-mon[36648]: pgmap v342: 129 pgs: 129 active+clean; 640 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 5.5 KiB/s wr, 18 op/s 2024-03-18T15:16:02.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:16:01 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/1972539963' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:16:02.128 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-18T15:16:02.128 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 -- ceph osd dump --format=json 2024-03-18T15:16:02.611 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:16:04.213 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:16:04 smithi019 ceph-mon[31522]: pgmap v343: 129 pgs: 129 active+clean; 640 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 4.2 KiB/s wr, 18 op/s 2024-03-18T15:16:04.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:16:04 smithi033 ceph-mon[36648]: pgmap v343: 129 pgs: 129 active+clean; 640 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 4.2 KiB/s wr, 18 op/s 2024-03-18T15:16:04.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:16:04 smithi104 ceph-mon[36070]: pgmap v343: 129 pgs: 129 active+clean; 640 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 4.2 KiB/s wr, 18 op/s 2024-03-18T15:16:04.720 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:04.719+0000 7f74fb600700 1 -- 172.21.15.19:0/1804794861 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f74f4102350 msgr2=0x7f74f41027d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:04.721 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:04.719+0000 7f74fb600700 1 --2- 172.21.15.19:0/1804794861 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f74f4102350 0x7f74f41027d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f74dc0099c0 tx=0x7f74dc009cd0 comp rx=0 tx=0).stop 2024-03-18T15:16:04.721 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:04.720+0000 7f74fb600700 1 -- 172.21.15.19:0/1804794861 shutdown_connections 2024-03-18T15:16:04.721 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:04.720+0000 7f74fb600700 1 --2- 172.21.15.19:0/1804794861 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f74f4102d10 0x7f74f410f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:04.721 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:04.720+0000 7f74fb600700 1 --2- 172.21.15.19:0/1804794861 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f74f41083a0 0x7f74f4108780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:04.721 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:04.720+0000 7f74fb600700 1 --2- 172.21.15.19:0/1804794861 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f74f4102350 0x7f74f41027d0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:04.721 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:04.720+0000 7f74fb600700 1 -- 172.21.15.19:0/1804794861 >> 172.21.15.19:0/1804794861 conn(0x7f74f4079420 msgr2=0x7f74f4079830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:16:04.721 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:04.720+0000 7f74fb600700 1 -- 172.21.15.19:0/1804794861 shutdown_connections 2024-03-18T15:16:04.721 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:04.720+0000 7f74fb600700 1 -- 172.21.15.19:0/1804794861 wait complete. 2024-03-18T15:16:04.723 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:04.721+0000 7f74fb600700 1 Processor -- start 2024-03-18T15:16:04.723 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:04.721+0000 7f74fb600700 1 -- start start 2024-03-18T15:16:04.723 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:04.722+0000 7f74fb600700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f74f4102350 0x7f74f419e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:16:04.723 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:04.722+0000 7f74fb600700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f74f4102d10 0x7f74f419ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:16:04.724 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:04.722+0000 7f74fb600700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f74f41083a0 0x7f74f4198860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:16:04.724 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:04.722+0000 7f74fb600700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f74f41125c0 con 0x7f74f4102350 2024-03-18T15:16:04.724 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:04.722+0000 7f74fb600700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f74f4112440 con 0x7f74f4102d10 2024-03-18T15:16:04.724 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:04.722+0000 7f74fb600700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f74f4112740 con 0x7f74f41083a0 2024-03-18T15:16:04.724 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:04.722+0000 7f74ebe00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f74f4102d10 0x7f74f419ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:16:04.724 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:04.722+0000 7f74ebe00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f74f4102d10 0x7f74f419ebe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:46850/0 (socket says 172.21.15.19:46850) 2024-03-18T15:16:04.724 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:04.722+0000 7f74ebe00700 1 -- 172.21.15.19:0/2378908030 learned_addr learned my addr 172.21.15.19:0/2378908030 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:16:04.724 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:04.722+0000 7f74f8c00700 1 --2- 172.21.15.19:0/2378908030 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f74f4102350 0x7f74f419e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:16:04.724 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:04.722+0000 7f74f9600700 1 --2- 172.21.15.19:0/2378908030 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f74f41083a0 0x7f74f4198860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:16:04.724 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:04.722+0000 7f74ebe00700 1 -- 172.21.15.19:0/2378908030 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f74f41083a0 msgr2=0x7f74f4198860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:04.724 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:04.722+0000 7f74ebe00700 1 --2- 172.21.15.19:0/2378908030 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f74f41083a0 0x7f74f4198860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:04.724 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:04.722+0000 7f74ebe00700 1 -- 172.21.15.19:0/2378908030 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f74f4102350 msgr2=0x7f74f419e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:04.725 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:04.722+0000 7f74ebe00700 1 --2- 172.21.15.19:0/2378908030 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f74f4102350 0x7f74f419e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:04.725 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:04.722+0000 7f74ebe00700 1 -- 172.21.15.19:0/2378908030 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74f4198fc0 con 0x7f74f4102d10 2024-03-18T15:16:04.725 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:04.723+0000 7f74ebe00700 1 --2- 172.21.15.19:0/2378908030 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f74f4102d10 0x7f74f419ebe0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f74dc006010 tx=0x7f74dc003a90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:16:04.725 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:04.723+0000 7f74f2000700 1 -- 172.21.15.19:0/2378908030 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74dc003680 con 0x7f74f4102d10 2024-03-18T15:16:04.725 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:04.723+0000 7f74f2000700 1 -- 172.21.15.19:0/2378908030 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74dc003e50 con 0x7f74f4102d10 2024-03-18T15:16:04.725 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:04.723+0000 7f74fb600700 1 -- 172.21.15.19:0/2378908030 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74f4199250 con 0x7f74f4102d10 2024-03-18T15:16:04.725 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:04.723+0000 7f74f2000700 1 -- 172.21.15.19:0/2378908030 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74dc00fe10 con 0x7f74f4102d10 2024-03-18T15:16:04.725 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:04.723+0000 7f74fb600700 1 -- 172.21.15.19:0/2378908030 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74f41a5570 con 0x7f74f4102d10 2024-03-18T15:16:04.725 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:04.725+0000 7f74fb600700 1 -- 172.21.15.19:0/2378908030 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74f4103a10 con 0x7f74f4102d10 2024-03-18T15:16:04.726 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:04.727+0000 7f74f2000700 1 -- 172.21.15.19:0/2378908030 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f74dc00f460 con 0x7f74f4102d10 2024-03-18T15:16:04.728 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:04.728+0000 7f74f2000700 1 --2- 172.21.15.19:0/2378908030 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f74d80a2ac0 0x7f74d80a4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:16:04.728 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:04.728+0000 7f74f2000700 1 -- 172.21.15.19:0/2378908030 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f74dc0cfb10 con 0x7f74f4102d10 2024-03-18T15:16:04.728 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:04.728+0000 7f74f8c00700 1 --2- 172.21.15.19:0/2378908030 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f74d80a2ac0 0x7f74d80a4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:16:04.729 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:04.729+0000 7f74f8c00700 1 --2- 172.21.15.19:0/2378908030 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f74d80a2ac0 0x7f74d80a4f80 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f74f41999a0 tx=0x7f74e4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:16:04.730 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:04.730+0000 7f74f2000700 1 -- 172.21.15.19:0/2378908030 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f74dc013070 con 0x7f74f4102d10 2024-03-18T15:16:04.959 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:04.959+0000 7f74fb600700 1 -- 172.21.15.19:0/2378908030 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f74f402db70 con 0x7f74f4102d10 2024-03-18T15:16:04.961 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:04.961+0000 7f74f2000700 1 -- 172.21.15.19:0/2378908030 <== mon.2 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24093 (secure 0 0 0) 0x7f74dc0a2030 con 0x7f74f4102d10 2024-03-18T15:16:04.962 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-18T15:16:04.962 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":89,"fsid":"d4dc2ba6-e538-11ee-95c9-87774f69a715","created":"2024-03-18T15:05:57.538551+0000","modified":"2024-03-18T15:15:15.582418+0000","last_up_change":"2024-03-18T15:14:12.009522+0000","last_in_change":"2024-03-18T15:13:52.555387+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T15:09:56.843349+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T15:15:08.941047+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T15:15:10.044671+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T15:15:13.621916+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"37b5a5cd-27ae-4e77-9b05-774b817b2ee9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6802","nonce":402333948},{"type":"v1","addr":"172.21.15.19:6803","nonce":402333948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6804","nonce":402333948},{"type":"v1","addr":"172.21.15.19:6805","nonce":402333948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6808","nonce":402333948},{"type":"v1","addr":"172.21.15.19:6809","nonce":402333948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6806","nonce":402333948},{"type":"v1","addr":"172.21.15.19:6807","nonce":402333948}]},"public_addr":"172.21.15.19:6803/402333948","cluster_addr":"172.21.15.19:6805/402333948","heartbeat_back_addr":"172.21.15.19:6809/402333948","heartbeat_front_addr":"172.21.15.19:6807/402333948","state":["exists","up"]},{"osd":1,"uuid":"3a1134e8-ede7-4dbd-9be1-eefd2cbb43bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1951880113},{"type":"v1","addr":"172.21.15.33:6801","nonce":1951880113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1951880113},{"type":"v1","addr":"172.21.15.33:6803","nonce":1951880113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1951880113},{"type":"v1","addr":"172.21.15.33:6807","nonce":1951880113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1951880113},{"type":"v1","addr":"172.21.15.33:6805","nonce":1951880113}]},"public_addr":"172.21.15.33:6801/1951880113","cluster_addr":"172.21.15.33:6803/1951880113","heartbeat_back_addr":"172.21.15.33:6807/1951880113","heartbeat_front_addr":"172.21.15.33:6805/1951880113","state":["exists","up"]},{"osd":2,"uuid":"f383a164-a3a0-49fd-a9e8-9c2178d8cfb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":1004727949},{"type":"v1","addr":"172.21.15.104:6801","nonce":1004727949}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":1004727949},{"type":"v1","addr":"172.21.15.104:6803","nonce":1004727949}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":1004727949},{"type":"v1","addr":"172.21.15.104:6807","nonce":1004727949}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":1004727949},{"type":"v1","addr":"172.21.15.104:6805","nonce":1004727949}]},"public_addr":"172.21.15.104:6801/1004727949","cluster_addr":"172.21.15.104:6803/1004727949","heartbeat_back_addr":"172.21.15.104:6807/1004727949","heartbeat_front_addr":"172.21.15.104:6805/1004727949","state":["exists","up"]},{"osd":3,"uuid":"c409e116-28bf-45dd-a791-539af22abe1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6810","nonce":3571525854},{"type":"v1","addr":"172.21.15.19:6811","nonce":3571525854}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6812","nonce":3571525854},{"type":"v1","addr":"172.21.15.19:6813","nonce":3571525854}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6816","nonce":3571525854},{"type":"v1","addr":"172.21.15.19:6817","nonce":3571525854}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6814","nonce":3571525854},{"type":"v1","addr":"172.21.15.19:6815","nonce":3571525854}]},"public_addr":"172.21.15.19:6811/3571525854","cluster_addr":"172.21.15.19:6813/3571525854","heartbeat_back_addr":"172.21.15.19:6817/3571525854","heartbeat_front_addr":"172.21.15.19:6815/3571525854","state":["exists","up"]},{"osd":4,"uuid":"696eb719-0b3e-43ae-b5f1-ccd73a3b2843","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":624901749},{"type":"v1","addr":"172.21.15.33:6809","nonce":624901749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":624901749},{"type":"v1","addr":"172.21.15.33:6811","nonce":624901749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":624901749},{"type":"v1","addr":"172.21.15.33:6815","nonce":624901749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":624901749},{"type":"v1","addr":"172.21.15.33:6813","nonce":624901749}]},"public_addr":"172.21.15.33:6809/624901749","cluster_addr":"172.21.15.33:6811/624901749","heartbeat_back_addr":"172.21.15.33:6815/624901749","heartbeat_front_addr":"172.21.15.33:6813/624901749","state":["exists","up"]},{"osd":5,"uuid":"21789779-8bb9-42c7-b9ee-437985f11e3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":1258208297},{"type":"v1","addr":"172.21.15.104:6809","nonce":1258208297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":1258208297},{"type":"v1","addr":"172.21.15.104:6811","nonce":1258208297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":1258208297},{"type":"v1","addr":"172.21.15.104:6815","nonce":1258208297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":1258208297},{"type":"v1","addr":"172.21.15.104:6813","nonce":1258208297}]},"public_addr":"172.21.15.104:6809/1258208297","cluster_addr":"172.21.15.104:6811/1258208297","heartbeat_back_addr":"172.21.15.104:6815/1258208297","heartbeat_front_addr":"172.21.15.104:6813/1258208297","state":["exists","up"]},{"osd":6,"uuid":"0e92a2ef-43cd-487c-9203-e219158aef8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6818","nonce":2782482301},{"type":"v1","addr":"172.21.15.19:6819","nonce":2782482301}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6820","nonce":2782482301},{"type":"v1","addr":"172.21.15.19:6821","nonce":2782482301}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6824","nonce":2782482301},{"type":"v1","addr":"172.21.15.19:6825","nonce":2782482301}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6822","nonce":2782482301},{"type":"v1","addr":"172.21.15.19:6823","nonce":2782482301}]},"public_addr":"172.21.15.19:6819/2782482301","cluster_addr":"172.21.15.19:6821/2782482301","heartbeat_back_addr":"172.21.15.19:6825/2782482301","heartbeat_front_addr":"172.21.15.19:6823/2782482301","state":["exists","up"]},{"osd":7,"uuid":"a16e39a1-3685-491d-b376-3e80a2084072","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":4238160988},{"type":"v1","addr":"172.21.15.33:6817","nonce":4238160988}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":4238160988},{"type":"v1","addr":"172.21.15.33:6819","nonce":4238160988}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":4238160988},{"type":"v1","addr":"172.21.15.33:6823","nonce":4238160988}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":4238160988},{"type":"v1","addr":"172.21.15.33:6821","nonce":4238160988}]},"public_addr":"172.21.15.33:6817/4238160988","cluster_addr":"172.21.15.33:6819/4238160988","heartbeat_back_addr":"172.21.15.33:6823/4238160988","heartbeat_front_addr":"172.21.15.33:6821/4238160988","state":["exists","up"]},{"osd":8,"uuid":"d5c3fa92-cd5d-4349-9c7d-d5666bacde16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":1014779813},{"type":"v1","addr":"172.21.15.104:6817","nonce":1014779813}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":1014779813},{"type":"v1","addr":"172.21.15.104:6819","nonce":1014779813}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":1014779813},{"type":"v1","addr":"172.21.15.104:6823","nonce":1014779813}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":1014779813},{"type":"v1","addr":"172.21.15.104:6821","nonce":1014779813}]},"public_addr":"172.21.15.104:6817/1014779813","cluster_addr":"172.21.15.104:6819/1014779813","heartbeat_back_addr":"172.21.15.104:6823/1014779813","heartbeat_front_addr":"172.21.15.104:6821/1014779813","state":["exists","up"]},{"osd":9,"uuid":"eed2e9b2-26d3-4c46-88ec-34b5596fe991","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6826","nonce":2641151009},{"type":"v1","addr":"172.21.15.19:6827","nonce":2641151009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6828","nonce":2641151009},{"type":"v1","addr":"172.21.15.19:6829","nonce":2641151009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6832","nonce":2641151009},{"type":"v1","addr":"172.21.15.19:6833","nonce":2641151009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6830","nonce":2641151009},{"type":"v1","addr":"172.21.15.19:6831","nonce":2641151009}]},"public_addr":"172.21.15.19:6827/2641151009","cluster_addr":"172.21.15.19:6829/2641151009","heartbeat_back_addr":"172.21.15.19:6833/2641151009","heartbeat_front_addr":"172.21.15.19:6831/2641151009","state":["exists","up"]},{"osd":10,"uuid":"f3c35ea5-fbe7-4dc4-93c5-b92d1a714db3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":1346058869},{"type":"v1","addr":"172.21.15.33:6825","nonce":1346058869}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":1346058869},{"type":"v1","addr":"172.21.15.33:6827","nonce":1346058869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":1346058869},{"type":"v1","addr":"172.21.15.33:6831","nonce":1346058869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":1346058869},{"type":"v1","addr":"172.21.15.33:6829","nonce":1346058869}]},"public_addr":"172.21.15.33:6825/1346058869","cluster_addr":"172.21.15.33:6827/1346058869","heartbeat_back_addr":"172.21.15.33:6831/1346058869","heartbeat_front_addr":"172.21.15.33:6829/1346058869","state":["exists","up"]},{"osd":11,"uuid":"c8b51571-f807-4b62-90b6-7529e09e59da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":3759498281},{"type":"v1","addr":"172.21.15.104:6825","nonce":3759498281}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":3759498281},{"type":"v1","addr":"172.21.15.104:6827","nonce":3759498281}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":3759498281},{"type":"v1","addr":"172.21.15.104:6831","nonce":3759498281}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":3759498281},{"type":"v1","addr":"172.21.15.104:6829","nonce":3759498281}]},"public_addr":"172.21.15.104:6825/3759498281","cluster_addr":"172.21.15.104:6827/3759498281","heartbeat_back_addr":"172.21.15.104:6831/3759498281","heartbeat_front_addr":"172.21.15.104:6829/3759498281","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:09:05.267795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:09:30.234068+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:09:53.870232+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:10:26.471296+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:10:53.180341+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:11:17.801361+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:11:51.364816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:12:18.306408+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:12:43.978744+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:13:18.213964+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:13:45.437544+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:14:10.148580+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.19:0/3860783274":"2024-03-19T15:07:14.150983+0000","172.21.15.19:0/2737223213":"2024-03-19T15:07:14.150983+0000","172.21.15.19:0/176699025":"2024-03-19T15:06:38.688405+0000","172.21.15.19:0/2316969005":"2024-03-19T15:07:14.150983+0000","172.21.15.19:6801/2776008962":"2024-03-19T15:07:14.150983+0000","172.21.15.19:0/2793980440":"2024-03-19T15:06:38.688405+0000","172.21.15.19:6800/3264337803":"2024-03-19T15:06:38.688405+0000","172.21.15.19:6801/3264337803":"2024-03-19T15:06:38.688405+0000","172.21.15.19:0/3166705295":"2024-03-19T15:06:38.688405+0000","172.21.15.19:6800/2776008962":"2024-03-19T15:07:14.150983+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T15:16:04.964 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:04.964+0000 7f74fb600700 1 -- 172.21.15.19:0/2378908030 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f74d80a2ac0 msgr2=0x7f74d80a4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:04.964 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:04.964+0000 7f74fb600700 1 --2- 172.21.15.19:0/2378908030 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f74d80a2ac0 0x7f74d80a4f80 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f74f41999a0 tx=0x7f74e4009260 comp rx=0 tx=0).stop 2024-03-18T15:16:04.964 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:04.964+0000 7f74fb600700 1 -- 172.21.15.19:0/2378908030 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f74f4102d10 msgr2=0x7f74f419ebe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:04.964 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:04.964+0000 7f74fb600700 1 --2- 172.21.15.19:0/2378908030 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f74f4102d10 0x7f74f419ebe0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f74dc006010 tx=0x7f74dc003a90 comp rx=0 tx=0).stop 2024-03-18T15:16:04.965 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:04.964+0000 7f74fb600700 1 -- 172.21.15.19:0/2378908030 shutdown_connections 2024-03-18T15:16:04.965 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:04.964+0000 7f74fb600700 1 --2- 172.21.15.19:0/2378908030 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f74d80a2ac0 0x7f74d80a4f80 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:04.965 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:04.964+0000 7f74fb600700 1 --2- 172.21.15.19:0/2378908030 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f74f41083a0 0x7f74f4198860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:04.965 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:04.964+0000 7f74fb600700 1 --2- 172.21.15.19:0/2378908030 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f74f4102350 0x7f74f419e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:04.965 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:04.964+0000 7f74fb600700 1 --2- 172.21.15.19:0/2378908030 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f74f4102d10 0x7f74f419ebe0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:04.965 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:04.964+0000 7f74fb600700 1 -- 172.21.15.19:0/2378908030 >> 172.21.15.19:0/2378908030 conn(0x7f74f4079420 msgr2=0x7f74f41012a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:16:04.965 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:04.964+0000 7f74fb600700 1 -- 172.21.15.19:0/2378908030 shutdown_connections 2024-03-18T15:16:04.965 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:04.964+0000 7f74fb600700 1 -- 172.21.15.19:0/2378908030 wait complete. 2024-03-18T15:16:05.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:16:05 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/2378908030' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:16:05.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:16:05 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/2378908030' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:16:05.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:16:05 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/2378908030' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:16:05.746 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-18T15:16:05.747 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-18T15:16:05.747 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-18T15:16:05.748 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-18T15:16:05.748 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-18T15:16:05.748 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-18T15:16:05.749 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-18T15:16:05.749 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-18T15:16:05.750 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 -- ceph tell osd.8 flush_pg_stats 2024-03-18T15:16:05.750 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 -- ceph tell osd.9 flush_pg_stats 2024-03-18T15:16:05.750 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 -- ceph tell osd.10 flush_pg_stats 2024-03-18T15:16:05.751 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 -- ceph tell osd.11 flush_pg_stats 2024-03-18T15:16:06.224 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:16:06 smithi019 ceph-mon[31522]: pgmap v344: 129 pgs: 129 active+clean; 640 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-03-18T15:16:06.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:16:06 smithi104 ceph-mon[36070]: pgmap v344: 129 pgs: 129 active+clean; 640 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-03-18T15:16:06.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:16:06 smithi033 ceph-mon[36648]: pgmap v344: 129 pgs: 129 active+clean; 640 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-03-18T15:16:06.931 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:16:06.933 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:16:06.934 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:16:06.936 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:16:06.940 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:16:06.951 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:16:06.952 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:16:06.952 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:16:06.963 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:16:06.966 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:16:06.966 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:16:07.085 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:16:08.172 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:16:08 smithi019 ceph-mon[31522]: pgmap v345: 129 pgs: 129 active+clean; 640 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-03-18T15:16:08.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:16:08 smithi104 ceph-mon[36070]: pgmap v345: 129 pgs: 129 active+clean; 640 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-03-18T15:16:08.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:16:08 smithi033 ceph-mon[36648]: pgmap v345: 129 pgs: 129 active+clean; 640 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-03-18T15:16:10.397 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:16:10 smithi019 ceph-mon[31522]: pgmap v346: 129 pgs: 129 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-03-18T15:16:10.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:16:10 smithi104 ceph-mon[36070]: pgmap v346: 129 pgs: 129 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-03-18T15:16:10.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:16:10 smithi033 ceph-mon[36648]: pgmap v346: 129 pgs: 129 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-03-18T15:16:11.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:16:11 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:16:11.444 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:16:11 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:16:11.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:16:11 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:16:12.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:16:12 smithi033 ceph-mon[36648]: pgmap v347: 129 pgs: 129 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 426 B/s wr, 9 op/s 2024-03-18T15:16:12.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:16:12 smithi104 ceph-mon[36070]: pgmap v347: 129 pgs: 129 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 426 B/s wr, 9 op/s 2024-03-18T15:16:12.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:16:12 smithi019 ceph-mon[31522]: pgmap v347: 129 pgs: 129 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 426 B/s wr, 9 op/s 2024-03-18T15:16:13.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:16:13 smithi019 ceph-mon[31522]: pgmap v348: 129 pgs: 129 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 426 B/s wr, 11 op/s 2024-03-18T15:16:13.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:16:13 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:16:13.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:16:13 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:16:13.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:16:13 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:16:13.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:16:13 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:16:14.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:16:13 smithi104 ceph-mon[36070]: pgmap v348: 129 pgs: 129 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 426 B/s wr, 11 op/s 2024-03-18T15:16:14.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:16:13 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:16:14.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:16:13 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:16:14.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:16:13 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:16:14.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:16:13 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:16:14.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:16:13 smithi033 ceph-mon[36648]: pgmap v348: 129 pgs: 129 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 426 B/s wr, 11 op/s 2024-03-18T15:16:14.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:16:13 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:16:14.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:16:13 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:16:14.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:16:13 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:16:14.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:16:13 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:16:15.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:16:14 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]: dispatch 2024-03-18T15:16:15.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:16:14 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]: dispatch 2024-03-18T15:16:15.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:16:14 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]: dispatch 2024-03-18T15:16:15.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:16:14 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]: dispatch 2024-03-18T15:16:15.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:16:14 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]: dispatch 2024-03-18T15:16:15.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:16:14 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [3, 6]}]: dispatch 2024-03-18T15:16:15.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:16:14 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]: dispatch 2024-03-18T15:16:15.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:16:14 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [3, 9]}]: dispatch 2024-03-18T15:16:15.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:16:14 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.37", "id": [5, 8]}]: dispatch 2024-03-18T15:16:15.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:16:14 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 7]}]: dispatch 2024-03-18T15:16:15.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:16:14 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]: dispatch 2024-03-18T15:16:15.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:16:14 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]: dispatch 2024-03-18T15:16:15.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:16:14 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]: dispatch 2024-03-18T15:16:15.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:16:14 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]: dispatch 2024-03-18T15:16:15.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:16:14 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]: dispatch 2024-03-18T15:16:15.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:16:14 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [3, 6]}]: dispatch 2024-03-18T15:16:15.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:16:14 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]: dispatch 2024-03-18T15:16:15.217 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:16:14 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [3, 9]}]: dispatch 2024-03-18T15:16:15.217 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:16:14 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.37", "id": [5, 8]}]: dispatch 2024-03-18T15:16:15.217 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:16:14 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 7]}]: dispatch 2024-03-18T15:16:15.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:16:14 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]: dispatch 2024-03-18T15:16:15.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:16:14 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]: dispatch 2024-03-18T15:16:15.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:16:14 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]: dispatch 2024-03-18T15:16:15.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:16:14 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]: dispatch 2024-03-18T15:16:15.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:16:14 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]: dispatch 2024-03-18T15:16:15.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:16:14 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [3, 6]}]: dispatch 2024-03-18T15:16:15.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:16:14 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]: dispatch 2024-03-18T15:16:15.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:16:14 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [3, 9]}]: dispatch 2024-03-18T15:16:15.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:16:14 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.37", "id": [5, 8]}]: dispatch 2024-03-18T15:16:15.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:16:14 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 7]}]: dispatch 2024-03-18T15:16:15.935 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:16:15 smithi019 ceph-mon[31522]: pgmap v349: 129 pgs: 129 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 426 B/s wr, 9 op/s 2024-03-18T15:16:15.935 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:16:15 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]': finished 2024-03-18T15:16:15.935 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:16:15 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]': finished 2024-03-18T15:16:15.935 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:16:15 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]': finished 2024-03-18T15:16:15.935 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:16:15 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]': finished 2024-03-18T15:16:15.935 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:16:15 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]': finished 2024-03-18T15:16:15.935 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:16:15 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [3, 6]}]': finished 2024-03-18T15:16:15.935 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:16:15 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]': finished 2024-03-18T15:16:15.936 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:16:15 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [3, 9]}]': finished 2024-03-18T15:16:15.936 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:16:15 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.37", "id": [5, 8]}]': finished 2024-03-18T15:16:15.936 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:16:15 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 7]}]': finished 2024-03-18T15:16:15.936 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:16:15 smithi019 ceph-mon[31522]: osdmap e90: 12 total, 12 up, 12 in 2024-03-18T15:16:16.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:16:15 smithi104 ceph-mon[36070]: pgmap v349: 129 pgs: 129 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 426 B/s wr, 9 op/s 2024-03-18T15:16:16.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:16:15 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]': finished 2024-03-18T15:16:16.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:16:15 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]': finished 2024-03-18T15:16:16.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:16:15 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]': finished 2024-03-18T15:16:16.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:16:15 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]': finished 2024-03-18T15:16:16.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:16:15 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]': finished 2024-03-18T15:16:16.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:16:15 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [3, 6]}]': finished 2024-03-18T15:16:16.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:16:15 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]': finished 2024-03-18T15:16:16.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:16:15 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [3, 9]}]': finished 2024-03-18T15:16:16.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:16:15 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.37", "id": [5, 8]}]': finished 2024-03-18T15:16:16.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:16:15 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 7]}]': finished 2024-03-18T15:16:16.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:16:15 smithi104 ceph-mon[36070]: osdmap e90: 12 total, 12 up, 12 in 2024-03-18T15:16:16.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:16:15 smithi033 ceph-mon[36648]: pgmap v349: 129 pgs: 129 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 426 B/s wr, 9 op/s 2024-03-18T15:16:16.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:16:15 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]': finished 2024-03-18T15:16:16.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:16:15 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]': finished 2024-03-18T15:16:16.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:16:15 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]': finished 2024-03-18T15:16:16.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:16:15 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]': finished 2024-03-18T15:16:16.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:16:15 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]': finished 2024-03-18T15:16:16.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:16:15 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [3, 6]}]': finished 2024-03-18T15:16:16.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:16:15 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]': finished 2024-03-18T15:16:16.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:16:15 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [3, 9]}]': finished 2024-03-18T15:16:16.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:16:15 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.37", "id": [5, 8]}]': finished 2024-03-18T15:16:16.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:16:15 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 7]}]': finished 2024-03-18T15:16:16.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:16:15 smithi033 ceph-mon[36648]: osdmap e90: 12 total, 12 up, 12 in 2024-03-18T15:16:16.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:16:16 smithi019 ceph-mon[31522]: osdmap e91: 12 total, 12 up, 12 in 2024-03-18T15:16:17.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:16:16 smithi104 ceph-mon[36070]: osdmap e91: 12 total, 12 up, 12 in 2024-03-18T15:16:17.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:16:16 smithi033 ceph-mon[36648]: osdmap e91: 12 total, 12 up, 12 in 2024-03-18T15:16:17.866 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:16:17 smithi019 ceph-mon[31522]: pgmap v352: 129 pgs: 3 peering, 126 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 9 op/s 2024-03-18T15:16:17.866 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:16:17 smithi019 ceph-mon[31522]: Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-03-18T15:16:18.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:16:17 smithi104 ceph-mon[36070]: pgmap v352: 129 pgs: 3 peering, 126 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 9 op/s 2024-03-18T15:16:18.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:16:17 smithi104 ceph-mon[36070]: Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-03-18T15:16:18.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:16:17 smithi033 ceph-mon[36648]: pgmap v352: 129 pgs: 3 peering, 126 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 9 op/s 2024-03-18T15:16:18.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:16:17 smithi033 ceph-mon[36648]: Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-03-18T15:16:20.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:16:19 smithi104 ceph-mon[36070]: pgmap v353: 129 pgs: 3 peering, 126 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 255 B/s wr, 10 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-03-18T15:16:20.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:16:19 smithi019 ceph-mon[31522]: pgmap v353: 129 pgs: 3 peering, 126 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 255 B/s wr, 10 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-03-18T15:16:20.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:16:19 smithi033 ceph-mon[36648]: pgmap v353: 129 pgs: 3 peering, 126 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 255 B/s wr, 10 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-03-18T15:16:20.419 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.054+0000 7fdb2a600700 1 -- 172.21.15.19:0/1112825033 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdb2406b420 msgr2=0x7fdb24106f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:20.420 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.054+0000 7fdb2a600700 1 --2- 172.21.15.19:0/1112825033 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdb2406b420 0x7fdb24106f20 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fdb0c0099c0 tx=0x7fdb0c009cd0 comp rx=0 tx=0).stop 2024-03-18T15:16:20.420 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.054+0000 7fdb2a600700 1 -- 172.21.15.19:0/1112825033 shutdown_connections 2024-03-18T15:16:20.420 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.054+0000 7fdb2a600700 1 --2- 172.21.15.19:0/1112825033 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fdb24107650 0x7fdb2410f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:20.420 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.054+0000 7fdb2a600700 1 --2- 172.21.15.19:0/1112825033 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdb2406aa70 0x7fdb2406ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:20.420 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.054+0000 7fdb2a600700 1 --2- 172.21.15.19:0/1112825033 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdb2406b420 0x7fdb24106f20 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:20.420 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.054+0000 7fdb2a600700 1 -- 172.21.15.19:0/1112825033 >> 172.21.15.19:0/1112825033 conn(0x7fdb24077220 msgr2=0x7fdb24077630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:16:20.420 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.054+0000 7fdb2a600700 1 -- 172.21.15.19:0/1112825033 shutdown_connections 2024-03-18T15:16:20.420 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.054+0000 7fdb2a600700 1 -- 172.21.15.19:0/1112825033 wait complete. 2024-03-18T15:16:20.420 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.055+0000 7fdb2a600700 1 Processor -- start 2024-03-18T15:16:20.420 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.056+0000 7fdb2a600700 1 -- start start 2024-03-18T15:16:20.420 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.056+0000 7fdb2a600700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdb2406aa70 0x7fdb2419e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:16:20.421 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.056+0000 7fdb2a600700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fdb2406b420 0x7fdb2419ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:16:20.421 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.056+0000 7fdb2a600700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdb24107650 0x7fdb241988d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:16:20.421 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.056+0000 7fdb2a600700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fdb241125f0 con 0x7fdb2406b420 2024-03-18T15:16:20.421 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.056+0000 7fdb2a600700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fdb24112470 con 0x7fdb2406aa70 2024-03-18T15:16:20.421 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.056+0000 7fdb2a600700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fdb24112770 con 0x7fdb24107650 2024-03-18T15:16:20.421 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.056+0000 7fdb23400700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdb2406aa70 0x7fdb2419e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:16:20.421 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.056+0000 7fdb23400700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdb2406aa70 0x7fdb2419e710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:55974/0 (socket says 172.21.15.19:55974) 2024-03-18T15:16:20.421 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.056+0000 7fdb23e00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdb24107650 0x7fdb241988d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:16:20.421 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.056+0000 7fdb23400700 1 -- 172.21.15.19:0/2353158303 learned_addr learned my addr 172.21.15.19:0/2353158303 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:16:20.421 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.056+0000 7fdb1aa00700 1 --2- 172.21.15.19:0/2353158303 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fdb2406b420 0x7fdb2419ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:16:20.421 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.057+0000 7fdb23400700 1 -- 172.21.15.19:0/2353158303 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdb24107650 msgr2=0x7fdb241988d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:20.421 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.057+0000 7fdb23400700 1 --2- 172.21.15.19:0/2353158303 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdb24107650 0x7fdb241988d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:20.421 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.057+0000 7fdb23400700 1 -- 172.21.15.19:0/2353158303 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fdb2406b420 msgr2=0x7fdb2419ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:20.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.057+0000 7fdb23400700 1 --2- 172.21.15.19:0/2353158303 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fdb2406b420 0x7fdb2419ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:20.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.057+0000 7fdb23400700 1 -- 172.21.15.19:0/2353158303 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb24199030 con 0x7fdb2406aa70 2024-03-18T15:16:20.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.057+0000 7fdb1aa00700 1 --2- 172.21.15.19:0/2353158303 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fdb2406b420 0x7fdb2419ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T15:16:20.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.057+0000 7fdb23400700 1 --2- 172.21.15.19:0/2353158303 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdb2406aa70 0x7fdb2419e710 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fdb1400c370 tx=0x7fdb1400c680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:16:20.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.057+0000 7fdb20c00700 1 -- 172.21.15.19:0/2353158303 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb1400cea0 con 0x7fdb2406aa70 2024-03-18T15:16:20.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.057+0000 7fdb2a600700 1 -- 172.21.15.19:0/2353158303 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb241992c0 con 0x7fdb2406aa70 2024-03-18T15:16:20.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.057+0000 7fdb20c00700 1 -- 172.21.15.19:0/2353158303 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdb140090d0 con 0x7fdb2406aa70 2024-03-18T15:16:20.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.057+0000 7fdb20c00700 1 -- 172.21.15.19:0/2353158303 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb14004880 con 0x7fdb2406aa70 2024-03-18T15:16:20.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.057+0000 7fdb2a600700 1 -- 172.21.15.19:0/2353158303 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb241a55e0 con 0x7fdb2406aa70 2024-03-18T15:16:20.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.058+0000 7fdb2a600700 1 -- 172.21.15.19:0/2353158303 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fdb240629e0 con 0x7fdb2406aa70 2024-03-18T15:16:20.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.060+0000 7fdb20c00700 1 -- 172.21.15.19:0/2353158303 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fdb1401a020 con 0x7fdb2406aa70 2024-03-18T15:16:20.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.062+0000 7fdb20c00700 1 --2- 172.21.15.19:0/2353158303 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fdb080a2920 0x7fdb080a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:16:20.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.062+0000 7fdb20c00700 1 -- 172.21.15.19:0/2353158303 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fdb140ceb20 con 0x7fdb2406aa70 2024-03-18T15:16:20.423 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.062+0000 7fdb1aa00700 1 --2- 172.21.15.19:0/2353158303 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fdb080a2920 0x7fdb080a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:16:20.423 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.062+0000 7fdb20c00700 1 --2- 172.21.15.19:0/2353158303 >> [v2:172.21.15.104:6800/1004727949,v1:172.21.15.104:6801/1004727949] conn(0x7fdb080ada80 0x7fdb080aff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:16:20.423 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.062+0000 7fdb20c00700 1 -- 172.21.15.19:0/2353158303 --> [v2:172.21.15.104:6800/1004727949,v1:172.21.15.104:6801/1004727949] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdb1401fe50 con 0x7fdb080ada80 2024-03-18T15:16:20.423 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.062+0000 7fdb20c00700 1 -- 172.21.15.19:0/2353158303 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7fdb140ceed0 con 0x7fdb2406aa70 2024-03-18T15:16:20.423 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.062+0000 7fdb23e00700 1 --2- 172.21.15.19:0/2353158303 >> [v2:172.21.15.104:6800/1004727949,v1:172.21.15.104:6801/1004727949] conn(0x7fdb080ada80 0x7fdb080aff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:16:20.423 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.063+0000 7fdb1aa00700 1 --2- 172.21.15.19:0/2353158303 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fdb080a2920 0x7fdb080a4de0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fdb0c009990 tx=0x7fdb0c005ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:16:20.423 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.063+0000 7fdb23e00700 1 --2- 172.21.15.19:0/2353158303 >> [v2:172.21.15.104:6800/1004727949,v1:172.21.15.104:6801/1004727949] conn(0x7fdb080ada80 0x7fdb080aff20 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:16:20.423 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.064+0000 7fdb20c00700 1 -- 172.21.15.19:0/2353158303 <== osd.2 v2:172.21.15.104:6800/1004727949 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fdb1401fe50 con 0x7fdb080ada80 2024-03-18T15:16:20.423 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.083+0000 7fdb2a600700 1 -- 172.21.15.19:0/2353158303 --> [v2:172.21.15.104:6800/1004727949,v1:172.21.15.104:6801/1004727949] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fdb24062e60 con 0x7fdb080ada80 2024-03-18T15:16:20.423 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.084+0000 7fdb20c00700 1 -- 172.21.15.19:0/2353158303 <== osd.2 v2:172.21.15.104:6800/1004727949 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fdb24062e60 con 0x7fdb080ada80 2024-03-18T15:16:20.424 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.084+0000 7fdb2a600700 1 -- 172.21.15.19:0/2353158303 >> [v2:172.21.15.104:6800/1004727949,v1:172.21.15.104:6801/1004727949] conn(0x7fdb080ada80 msgr2=0x7fdb080aff20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:20.424 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.084+0000 7fdb2a600700 1 --2- 172.21.15.19:0/2353158303 >> [v2:172.21.15.104:6800/1004727949,v1:172.21.15.104:6801/1004727949] conn(0x7fdb080ada80 0x7fdb080aff20 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:20.424 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.084+0000 7fdb2a600700 1 -- 172.21.15.19:0/2353158303 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fdb080a2920 msgr2=0x7fdb080a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:20.424 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.084+0000 7fdb2a600700 1 --2- 172.21.15.19:0/2353158303 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fdb080a2920 0x7fdb080a4de0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fdb0c009990 tx=0x7fdb0c005ab0 comp rx=0 tx=0).stop 2024-03-18T15:16:20.424 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.084+0000 7fdb2a600700 1 -- 172.21.15.19:0/2353158303 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdb2406aa70 msgr2=0x7fdb2419e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:20.424 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.084+0000 7fdb2a600700 1 --2- 172.21.15.19:0/2353158303 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdb2406aa70 0x7fdb2419e710 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fdb1400c370 tx=0x7fdb1400c680 comp rx=0 tx=0).stop 2024-03-18T15:16:20.424 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.084+0000 7fdb23e00700 1 -- 172.21.15.19:0/2353158303 reap_dead start 2024-03-18T15:16:20.424 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.084+0000 7fdb2a600700 1 -- 172.21.15.19:0/2353158303 shutdown_connections 2024-03-18T15:16:20.424 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.084+0000 7fdb2a600700 1 -- 172.21.15.19:0/2353158303 >> 172.21.15.19:0/2353158303 conn(0x7fdb24077220 msgr2=0x7fdb240ff5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:16:20.424 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.084+0000 7fdb2a600700 1 -- 172.21.15.19:0/2353158303 shutdown_connections 2024-03-18T15:16:20.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.084+0000 7fdb2a600700 1 -- 172.21.15.19:0/2353158303 wait complete. 2024-03-18T15:16:21.375 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.615+0000 7f46d3c00700 1 -- 172.21.15.19:0/3508929243 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f46cc074e20 msgr2=0x7f46cc10e790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:21.375 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.615+0000 7f46d3c00700 1 --2- 172.21.15.19:0/3508929243 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f46cc074e20 0x7f46cc10e790 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f46b8009990 tx=0x7f46b8009ca0 comp rx=0 tx=0).stop 2024-03-18T15:16:21.375 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.616+0000 7f46d3c00700 1 -- 172.21.15.19:0/3508929243 shutdown_connections 2024-03-18T15:16:21.375 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.616+0000 7f46d3c00700 1 --2- 172.21.15.19:0/3508929243 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f46cc10ecd0 0x7f46cc1110c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:21.375 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.616+0000 7f46d3c00700 1 --2- 172.21.15.19:0/3508929243 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f46cc074500 0x7f46cc0748e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:21.375 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.616+0000 7f46d3c00700 1 --2- 172.21.15.19:0/3508929243 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f46cc074e20 0x7f46cc10e790 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:21.375 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.616+0000 7f46d3c00700 1 -- 172.21.15.19:0/3508929243 >> 172.21.15.19:0/3508929243 conn(0x7f46cc079430 msgr2=0x7f46cc079840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:16:21.375 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.616+0000 7f46d3c00700 1 -- 172.21.15.19:0/3508929243 shutdown_connections 2024-03-18T15:16:21.375 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.616+0000 7f46d3c00700 1 -- 172.21.15.19:0/3508929243 wait complete. 2024-03-18T15:16:21.375 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.617+0000 7f46d3c00700 1 Processor -- start 2024-03-18T15:16:21.375 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.617+0000 7f46d3c00700 1 -- start start 2024-03-18T15:16:21.375 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.617+0000 7f46d3c00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f46cc074500 0x7f46cc19ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:16:21.376 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.617+0000 7f46d3c00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f46cc074e20 0x7f46cc19b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:16:21.376 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.617+0000 7f46d3c00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f46cc10ecd0 0x7f46cc19f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:16:21.376 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.617+0000 7f46d3c00700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f46cc1148f0 con 0x7f46cc074500 2024-03-18T15:16:21.376 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.617+0000 7f46d3c00700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f46cc114770 con 0x7f46cc10ecd0 2024-03-18T15:16:21.376 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.617+0000 7f46d3c00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f46cc114a70 con 0x7f46cc074e20 2024-03-18T15:16:21.376 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.618+0000 7f46cbe00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f46cc074e20 0x7f46cc19b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:16:21.376 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.618+0000 7f46d1200700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f46cc074500 0x7f46cc19ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:16:21.376 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.618+0000 7f46cbe00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f46cc074e20 0x7f46cc19b2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.19:34174/0 (socket says 172.21.15.19:34174) 2024-03-18T15:16:21.376 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.618+0000 7f46cbe00700 1 -- 172.21.15.19:0/272524024 learned_addr learned my addr 172.21.15.19:0/272524024 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:16:21.376 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.618+0000 7f46d1c00700 1 --2- 172.21.15.19:0/272524024 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f46cc10ecd0 0x7f46cc19f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:16:21.376 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.618+0000 7f46cbe00700 1 -- 172.21.15.19:0/272524024 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f46cc10ecd0 msgr2=0x7f46cc19f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:21.376 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.618+0000 7f46cbe00700 1 --2- 172.21.15.19:0/272524024 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f46cc10ecd0 0x7f46cc19f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:21.376 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.618+0000 7f46cbe00700 1 -- 172.21.15.19:0/272524024 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f46cc074500 msgr2=0x7f46cc19ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:21.376 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.618+0000 7f46cbe00700 1 --2- 172.21.15.19:0/272524024 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f46cc074500 0x7f46cc19ad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:21.377 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.618+0000 7f46cbe00700 1 -- 172.21.15.19:0/272524024 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46cc19fd40 con 0x7f46cc074e20 2024-03-18T15:16:21.377 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.618+0000 7f46d1c00700 1 --2- 172.21.15.19:0/272524024 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f46cc10ecd0 0x7f46cc19f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:16:21.377 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.618+0000 7f46cbe00700 1 --2- 172.21.15.19:0/272524024 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f46cc074e20 0x7f46cc19b2c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f46b8005910 tx=0x7f46b800bab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:16:21.377 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.619+0000 7f46c9600700 1 -- 172.21.15.19:0/272524024 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46b8027070 con 0x7f46cc074e20 2024-03-18T15:16:21.377 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.619+0000 7f46c9600700 1 -- 172.21.15.19:0/272524024 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f46b8004650 con 0x7f46cc074e20 2024-03-18T15:16:21.377 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.619+0000 7f46c9600700 1 -- 172.21.15.19:0/272524024 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46b800f460 con 0x7f46cc074e20 2024-03-18T15:16:21.377 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.619+0000 7f46d3c00700 1 -- 172.21.15.19:0/272524024 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46cc19ffd0 con 0x7f46cc074e20 2024-03-18T15:16:21.377 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.619+0000 7f46d3c00700 1 -- 172.21.15.19:0/272524024 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46cc1a78b0 con 0x7f46cc074e20 2024-03-18T15:16:21.377 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.620+0000 7f46d3c00700 1 -- 172.21.15.19:0/272524024 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f46cc0629e0 con 0x7f46cc074e20 2024-03-18T15:16:21.377 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.622+0000 7f46c9600700 1 -- 172.21.15.19:0/272524024 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f46b802a070 con 0x7f46cc074e20 2024-03-18T15:16:21.377 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.623+0000 7f46c9600700 1 --2- 172.21.15.19:0/272524024 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f46b40a29a0 0x7f46b40a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:16:21.377 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.624+0000 7f46c9600700 1 -- 172.21.15.19:0/272524024 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f46b80d0550 con 0x7f46cc074e20 2024-03-18T15:16:21.377 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.624+0000 7f46d1200700 1 --2- 172.21.15.19:0/272524024 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f46b40a29a0 0x7f46b40a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:16:21.377 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.624+0000 7f46c9600700 1 --2- 172.21.15.19:0/272524024 >> [v2:172.21.15.104:6824/3759498281,v1:172.21.15.104:6825/3759498281] conn(0x7f46b40adb00 0x7f46b40affa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:16:21.377 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.624+0000 7f46c9600700 1 -- 172.21.15.19:0/272524024 --> [v2:172.21.15.104:6824/3759498281,v1:172.21.15.104:6825/3759498281] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f46b8010ec0 con 0x7f46b40adb00 2024-03-18T15:16:21.377 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.624+0000 7f46c9600700 1 -- 172.21.15.19:0/272524024 <== mon.1 v2:172.21.15.104:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f46b80d0900 con 0x7f46cc074e20 2024-03-18T15:16:21.378 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.624+0000 7f46d1c00700 1 --2- 172.21.15.19:0/272524024 >> [v2:172.21.15.104:6824/3759498281,v1:172.21.15.104:6825/3759498281] conn(0x7f46b40adb00 0x7f46b40affa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:16:21.378 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.625+0000 7f46d1200700 1 --2- 172.21.15.19:0/272524024 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f46b40a29a0 0x7f46b40a4e60 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f46c0009c60 tx=0x7f46c0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:16:21.378 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.625+0000 7f46d1c00700 1 --2- 172.21.15.19:0/272524024 >> [v2:172.21.15.104:6824/3759498281,v1:172.21.15.104:6825/3759498281] conn(0x7f46b40adb00 0x7f46b40affa0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:16:21.378 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.626+0000 7f46c9600700 1 -- 172.21.15.19:0/272524024 <== osd.11 v2:172.21.15.104:6824/3759498281 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f46b8010ec0 con 0x7f46b40adb00 2024-03-18T15:16:21.378 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.644+0000 7f46d3c00700 1 -- 172.21.15.19:0/272524024 --> [v2:172.21.15.104:6824/3759498281,v1:172.21.15.104:6825/3759498281] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f46cc062de0 con 0x7f46b40adb00 2024-03-18T15:16:21.378 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.644+0000 7f46c9600700 1 -- 172.21.15.19:0/272524024 <== osd.11 v2:172.21.15.104:6824/3759498281 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f46cc062de0 con 0x7f46b40adb00 2024-03-18T15:16:21.378 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.645+0000 7f46d3c00700 1 -- 172.21.15.19:0/272524024 >> [v2:172.21.15.104:6824/3759498281,v1:172.21.15.104:6825/3759498281] conn(0x7f46b40adb00 msgr2=0x7f46b40affa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:21.379 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.645+0000 7f46d3c00700 1 --2- 172.21.15.19:0/272524024 >> [v2:172.21.15.104:6824/3759498281,v1:172.21.15.104:6825/3759498281] conn(0x7f46b40adb00 0x7f46b40affa0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:21.379 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.645+0000 7f46d3c00700 1 -- 172.21.15.19:0/272524024 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f46b40a29a0 msgr2=0x7f46b40a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:21.379 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.645+0000 7f46d3c00700 1 --2- 172.21.15.19:0/272524024 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f46b40a29a0 0x7f46b40a4e60 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f46c0009c60 tx=0x7f46c0009260 comp rx=0 tx=0).stop 2024-03-18T15:16:21.379 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.645+0000 7f46d3c00700 1 -- 172.21.15.19:0/272524024 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f46cc074e20 msgr2=0x7f46cc19b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:21.379 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.645+0000 7f46d3c00700 1 --2- 172.21.15.19:0/272524024 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f46cc074e20 0x7f46cc19b2c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f46b8005910 tx=0x7f46b800bab0 comp rx=0 tx=0).stop 2024-03-18T15:16:21.379 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.645+0000 7f46d1c00700 1 -- 172.21.15.19:0/272524024 reap_dead start 2024-03-18T15:16:21.379 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.645+0000 7f46d3c00700 1 -- 172.21.15.19:0/272524024 shutdown_connections 2024-03-18T15:16:21.379 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.645+0000 7f46d3c00700 1 -- 172.21.15.19:0/272524024 >> 172.21.15.19:0/272524024 conn(0x7f46cc079430 msgr2=0x7f46cc0ff4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:16:21.379 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.646+0000 7f46d3c00700 1 -- 172.21.15.19:0/272524024 shutdown_connections 2024-03-18T15:16:21.379 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:19.646+0000 7f46d3c00700 1 -- 172.21.15.19:0/272524024 wait complete. 2024-03-18T15:16:21.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:16:21 smithi019 ceph-mon[31522]: pgmap v354: 129 pgs: 3 peering, 126 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 7 op/s; 77 B/s, 1 keys/s, 1 objects/s recovering 2024-03-18T15:16:22.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:16:21 smithi104 ceph-mon[36070]: pgmap v354: 129 pgs: 3 peering, 126 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 7 op/s; 77 B/s, 1 keys/s, 1 objects/s recovering 2024-03-18T15:16:22.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:16:21 smithi033 ceph-mon[36648]: pgmap v354: 129 pgs: 3 peering, 126 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 7 op/s; 77 B/s, 1 keys/s, 1 objects/s recovering 2024-03-18T15:16:23.154 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:16:22 smithi019 ceph-mon[31522]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-03-18T15:16:23.154 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:16:22 smithi019 ceph-mon[31522]: Cluster is now healthy 2024-03-18T15:16:23.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:16:22 smithi104 ceph-mon[36070]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-03-18T15:16:23.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:16:22 smithi104 ceph-mon[36070]: Cluster is now healthy 2024-03-18T15:16:23.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:16:22 smithi033 ceph-mon[36648]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-03-18T15:16:23.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:16:22 smithi033 ceph-mon[36648]: Cluster is now healthy 2024-03-18T15:16:23.427 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:23.425+0000 7f964e800700 1 -- 172.21.15.19:0/1028643891 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9648102410 msgr2=0x7f96481027f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:23.427 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:23.425+0000 7f964e800700 1 --2- 172.21.15.19:0/1028643891 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9648102410 0x7f96481027f0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f963c009920 tx=0x7f963c009c30 comp rx=0 tx=0).stop 2024-03-18T15:16:23.427 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:23.425+0000 7f964e800700 1 -- 172.21.15.19:0/1028643891 shutdown_connections 2024-03-18T15:16:23.427 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:23.425+0000 7f964e800700 1 --2- 172.21.15.19:0/1028643891 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9648102410 0x7f96481027f0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:23.427 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:23.425+0000 7f964e800700 1 --2- 172.21.15.19:0/1028643891 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f96481074f0 0x7f964810f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:23.427 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:23.425+0000 7f964e800700 1 --2- 172.21.15.19:0/1028643891 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9648102dc0 0x7f9648106e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:23.427 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:23.425+0000 7f964e800700 1 -- 172.21.15.19:0/1028643891 >> 172.21.15.19:0/1028643891 conn(0x7f96480fdca0 msgr2=0x7f96481000c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:16:23.427 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:23.426+0000 7f964e800700 1 -- 172.21.15.19:0/1028643891 shutdown_connections 2024-03-18T15:16:23.428 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:23.426+0000 7f964e800700 1 -- 172.21.15.19:0/1028643891 wait complete. 2024-03-18T15:16:23.428 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:23.427+0000 7f964e800700 1 Processor -- start 2024-03-18T15:16:23.428 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:23.428+0000 7f964e800700 1 -- start start 2024-03-18T15:16:23.429 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:23.428+0000 7f964e800700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9648102410 0x7f96481a08d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:16:23.429 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:23.428+0000 7f964e800700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9648102dc0 0x7f96481a0e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:16:23.429 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:23.428+0000 7f964e800700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f96481074f0 0x7f964819aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:16:23.430 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:23.428+0000 7f964e800700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f9648112520 con 0x7f9648102410 2024-03-18T15:16:23.430 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:23.428+0000 7f964e800700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f96481123a0 con 0x7f96481074f0 2024-03-18T15:16:23.430 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:23.428+0000 7f964e800700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f96481126a0 con 0x7f9648102dc0 2024-03-18T15:16:23.430 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:23.428+0000 7f964de00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f96481074f0 0x7f964819aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:16:23.430 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:23.428+0000 7f964de00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f96481074f0 0x7f964819aae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:50918/0 (socket says 172.21.15.19:50918) 2024-03-18T15:16:23.430 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:23.428+0000 7f964de00700 1 -- 172.21.15.19:0/3994616909 learned_addr learned my addr 172.21.15.19:0/3994616909 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:16:23.430 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:23.428+0000 7f964ca00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9648102dc0 0x7f96481a0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:16:23.430 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:23.429+0000 7f964d400700 1 --2- 172.21.15.19:0/3994616909 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9648102410 0x7f96481a08d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:16:23.431 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:23.429+0000 7f964de00700 1 -- 172.21.15.19:0/3994616909 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9648102dc0 msgr2=0x7f96481a0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:23.431 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:23.429+0000 7f964de00700 1 --2- 172.21.15.19:0/3994616909 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9648102dc0 0x7f96481a0e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:23.431 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:23.429+0000 7f964de00700 1 -- 172.21.15.19:0/3994616909 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9648102410 msgr2=0x7f96481a08d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:23.431 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:23.429+0000 7f964de00700 1 --2- 172.21.15.19:0/3994616909 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9648102410 0x7f96481a08d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:23.431 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:23.429+0000 7f964de00700 1 -- 172.21.15.19:0/3994616909 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f964819b240 con 0x7f96481074f0 2024-03-18T15:16:23.431 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:23.429+0000 7f964ca00700 1 --2- 172.21.15.19:0/3994616909 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9648102dc0 0x7f96481a0e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:16:23.431 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:23.429+0000 7f964de00700 1 --2- 172.21.15.19:0/3994616909 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f96481074f0 0x7f964819aae0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f9640007ee0 tx=0x7f964000d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:16:23.431 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:23.429+0000 7f9646000700 1 -- 172.21.15.19:0/3994616909 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f964000dbd0 con 0x7f96481074f0 2024-03-18T15:16:23.432 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:23.429+0000 7f9646000700 1 -- 172.21.15.19:0/3994616909 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f964000dd70 con 0x7f96481074f0 2024-03-18T15:16:23.432 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:23.429+0000 7f964e800700 1 -- 172.21.15.19:0/3994616909 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f964819b530 con 0x7f96481074f0 2024-03-18T15:16:23.432 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:23.429+0000 7f9646000700 1 -- 172.21.15.19:0/3994616909 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9640005a60 con 0x7f96481074f0 2024-03-18T15:16:23.432 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:23.430+0000 7f964e800700 1 -- 172.21.15.19:0/3994616909 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96481a77a0 con 0x7f96481074f0 2024-03-18T15:16:23.432 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:23.431+0000 7f963b400700 1 -- 172.21.15.19:0/3994616909 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9648062a60 con 0x7f96481074f0 2024-03-18T15:16:23.435 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:23.434+0000 7f9646000700 1 -- 172.21.15.19:0/3994616909 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f964001a020 con 0x7f96481074f0 2024-03-18T15:16:23.436 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:23.435+0000 7f9646000700 1 --2- 172.21.15.19:0/3994616909 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f96300a29a0 0x7f96300a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:16:23.436 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:23.435+0000 7f9646000700 1 -- 172.21.15.19:0/3994616909 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f96400ceff0 con 0x7f96481074f0 2024-03-18T15:16:23.436 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:23.435+0000 7f964d400700 1 --2- 172.21.15.19:0/3994616909 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f96300a29a0 0x7f96300a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:16:23.437 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:23.436+0000 7f9646000700 1 --2- 172.21.15.19:0/3994616909 >> [v2:172.21.15.33:6808/624901749,v1:172.21.15.33:6809/624901749] conn(0x7f96300adb00 0x7f96300affa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:16:23.437 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:23.436+0000 7f9646000700 1 -- 172.21.15.19:0/3994616909 --> [v2:172.21.15.33:6808/624901749,v1:172.21.15.33:6809/624901749] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9640012ec0 con 0x7f96300adb00 2024-03-18T15:16:23.437 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:23.436+0000 7f9646000700 1 -- 172.21.15.19:0/3994616909 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f96400cf420 con 0x7f96481074f0 2024-03-18T15:16:23.437 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:23.436+0000 7f964ca00700 1 --2- 172.21.15.19:0/3994616909 >> [v2:172.21.15.33:6808/624901749,v1:172.21.15.33:6809/624901749] conn(0x7f96300adb00 0x7f96300affa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:16:23.437 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:23.436+0000 7f964d400700 1 --2- 172.21.15.19:0/3994616909 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f96300a29a0 0x7f96300a4e60 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f963c009680 tx=0x7f963c019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:16:23.438 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:23.437+0000 7f964ca00700 1 --2- 172.21.15.19:0/3994616909 >> [v2:172.21.15.33:6808/624901749,v1:172.21.15.33:6809/624901749] conn(0x7f96300adb00 0x7f96300affa0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:16:23.439 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:23.438+0000 7f9646000700 1 -- 172.21.15.19:0/3994616909 <== osd.4 v2:172.21.15.33:6808/624901749 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9640012ec0 con 0x7f96300adb00 2024-03-18T15:16:23.458 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:23.457+0000 7f963b400700 1 -- 172.21.15.19:0/3994616909 --> [v2:172.21.15.33:6808/624901749,v1:172.21.15.33:6809/624901749] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f9648062ee0 con 0x7f96300adb00 2024-03-18T15:16:23.459 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:23.458+0000 7f9646000700 1 -- 172.21.15.19:0/3994616909 <== osd.4 v2:172.21.15.33:6808/624901749 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f9648062ee0 con 0x7f96300adb00 2024-03-18T15:16:23.460 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:23.458+0000 7f963b400700 1 -- 172.21.15.19:0/3994616909 >> [v2:172.21.15.33:6808/624901749,v1:172.21.15.33:6809/624901749] conn(0x7f96300adb00 msgr2=0x7f96300affa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:23.460 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:23.458+0000 7f963b400700 1 --2- 172.21.15.19:0/3994616909 >> [v2:172.21.15.33:6808/624901749,v1:172.21.15.33:6809/624901749] conn(0x7f96300adb00 0x7f96300affa0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:23.460 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:23.458+0000 7f963b400700 1 -- 172.21.15.19:0/3994616909 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f96300a29a0 msgr2=0x7f96300a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:23.460 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:23.458+0000 7f963b400700 1 --2- 172.21.15.19:0/3994616909 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f96300a29a0 0x7f96300a4e60 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f963c009680 tx=0x7f963c019040 comp rx=0 tx=0).stop 2024-03-18T15:16:23.460 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:23.458+0000 7f963b400700 1 -- 172.21.15.19:0/3994616909 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f96481074f0 msgr2=0x7f964819aae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:23.460 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:23.459+0000 7f963b400700 1 --2- 172.21.15.19:0/3994616909 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f96481074f0 0x7f964819aae0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f9640007ee0 tx=0x7f964000d3b0 comp rx=0 tx=0).stop 2024-03-18T15:16:23.460 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:23.459+0000 7f964de00700 1 -- 172.21.15.19:0/3994616909 reap_dead start 2024-03-18T15:16:23.460 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:23.459+0000 7f963b400700 1 -- 172.21.15.19:0/3994616909 shutdown_connections 2024-03-18T15:16:23.460 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:23.459+0000 7f963b400700 1 -- 172.21.15.19:0/3994616909 >> 172.21.15.19:0/3994616909 conn(0x7f96480fdca0 msgr2=0x7f96480fffd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:16:23.461 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:23.459+0000 7f963b400700 1 -- 172.21.15.19:0/3994616909 shutdown_connections 2024-03-18T15:16:23.461 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:23.459+0000 7f963b400700 1 -- 172.21.15.19:0/3994616909 wait complete. 2024-03-18T15:16:23.725 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:21.449+0000 7f28ada00700 1 -- 172.21.15.19:0/3158980418 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f28a806a940 msgr2=0x7f28a810f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:23.725 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:21.449+0000 7f28ada00700 1 --2- 172.21.15.19:0/3158980418 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f28a806a940 0x7f28a810f120 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f28a000b0d0 tx=0x7f28a000b3e0 comp rx=0 tx=0).stop 2024-03-18T15:16:23.725 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:21.449+0000 7f28ada00700 1 -- 172.21.15.19:0/3158980418 shutdown_connections 2024-03-18T15:16:23.725 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:21.449+0000 7f28ada00700 1 --2- 172.21.15.19:0/3158980418 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f28a8108300 0x7f28a81086e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:23.725 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:21.449+0000 7f28ada00700 1 --2- 172.21.15.19:0/3158980418 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f28a806a940 0x7f28a810f120 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:23.725 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:21.449+0000 7f28ada00700 1 --2- 172.21.15.19:0/3158980418 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f28a8069f80 0x7f28a806a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:23.725 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:21.449+0000 7f28ada00700 1 -- 172.21.15.19:0/3158980418 >> 172.21.15.19:0/3158980418 conn(0x7f28a8077f80 msgr2=0x7f28a8078390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:16:23.725 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:21.449+0000 7f28ada00700 1 -- 172.21.15.19:0/3158980418 shutdown_connections 2024-03-18T15:16:23.725 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:21.450+0000 7f28ada00700 1 -- 172.21.15.19:0/3158980418 wait complete. 2024-03-18T15:16:23.725 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:21.451+0000 7f28ada00700 1 Processor -- start 2024-03-18T15:16:23.726 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:21.452+0000 7f28ada00700 1 -- start start 2024-03-18T15:16:23.726 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:21.452+0000 7f28ada00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f28a8069f80 0x7f28a81015f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:16:23.726 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:21.452+0000 7f28ada00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f28a806a940 0x7f28a8101b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:16:23.726 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:21.452+0000 7f28ada00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f28a8108300 0x7f28a8102070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:16:23.726 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:21.452+0000 7f28ada00700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f28a8112580 con 0x7f28a806a940 2024-03-18T15:16:23.726 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:21.452+0000 7f28ada00700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f28a8112400 con 0x7f28a8069f80 2024-03-18T15:16:23.726 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:21.452+0000 7f28ada00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f28a8112700 con 0x7f28a8108300 2024-03-18T15:16:23.726 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:21.452+0000 7f28a6a00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f28a8069f80 0x7f28a81015f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:16:23.726 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:21.452+0000 7f28a6a00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f28a8069f80 0x7f28a81015f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:56006/0 (socket says 172.21.15.19:56006) 2024-03-18T15:16:23.726 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:21.452+0000 7f28a6a00700 1 -- 172.21.15.19:0/513317740 learned_addr learned my addr 172.21.15.19:0/513317740 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:16:23.726 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:21.452+0000 7f28a7400700 1 --2- 172.21.15.19:0/513317740 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f28a8108300 0x7f28a8102070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:16:23.726 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:21.452+0000 7f28a6000700 1 --2- 172.21.15.19:0/513317740 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f28a806a940 0x7f28a8101b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:16:23.726 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:21.453+0000 7f28a6a00700 1 -- 172.21.15.19:0/513317740 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f28a8108300 msgr2=0x7f28a8102070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:23.726 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:21.453+0000 7f28a6a00700 1 --2- 172.21.15.19:0/513317740 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f28a8108300 0x7f28a8102070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:23.727 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:21.453+0000 7f28a6a00700 1 -- 172.21.15.19:0/513317740 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f28a806a940 msgr2=0x7f28a8101b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:23.727 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:21.453+0000 7f28a6a00700 1 --2- 172.21.15.19:0/513317740 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f28a806a940 0x7f28a8101b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:23.727 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:21.453+0000 7f28a6a00700 1 -- 172.21.15.19:0/513317740 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28a81a6f90 con 0x7f28a8069f80 2024-03-18T15:16:23.727 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:21.453+0000 7f28a7400700 1 --2- 172.21.15.19:0/513317740 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f28a8108300 0x7f28a8102070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:16:23.727 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:21.453+0000 7f28a6a00700 1 --2- 172.21.15.19:0/513317740 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f28a8069f80 0x7f28a81015f0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f289800e9a0 tx=0x7f289800ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:16:23.727 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:21.453+0000 7f289f400700 1 -- 172.21.15.19:0/513317740 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f289800ca10 con 0x7f28a8069f80 2024-03-18T15:16:23.727 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:21.453+0000 7f289f400700 1 -- 172.21.15.19:0/513317740 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f289800cbb0 con 0x7f28a8069f80 2024-03-18T15:16:23.727 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:21.453+0000 7f289f400700 1 -- 172.21.15.19:0/513317740 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2898018680 con 0x7f28a8069f80 2024-03-18T15:16:23.727 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:21.453+0000 7f28ada00700 1 -- 172.21.15.19:0/513317740 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28a81a7280 con 0x7f28a8069f80 2024-03-18T15:16:23.727 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:21.454+0000 7f28ada00700 1 -- 172.21.15.19:0/513317740 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28a81a7810 con 0x7f28a8069f80 2024-03-18T15:16:23.727 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:21.454+0000 7f28ada00700 1 -- 172.21.15.19:0/513317740 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f28a80629e0 con 0x7f28a8069f80 2024-03-18T15:16:23.728 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:21.456+0000 7f289f400700 1 -- 172.21.15.19:0/513317740 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f2898010430 con 0x7f28a8069f80 2024-03-18T15:16:23.728 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:21.458+0000 7f289f400700 1 --2- 172.21.15.19:0/513317740 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f288c0a29a0 0x7f288c0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:16:23.728 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:21.458+0000 7f289f400700 1 -- 172.21.15.19:0/513317740 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f28980cf910 con 0x7f28a8069f80 2024-03-18T15:16:23.728 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:21.458+0000 7f28a6000700 1 --2- 172.21.15.19:0/513317740 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f288c0a29a0 0x7f288c0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:16:23.728 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:21.458+0000 7f289f400700 1 --2- 172.21.15.19:0/513317740 >> [v2:172.21.15.33:6800/1951880113,v1:172.21.15.33:6801/1951880113] conn(0x7f288c0adb00 0x7f288c0affa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:16:23.728 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:21.458+0000 7f289f400700 1 -- 172.21.15.19:0/513317740 --> [v2:172.21.15.33:6800/1951880113,v1:172.21.15.33:6801/1951880113] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f289801bec0 con 0x7f288c0adb00 2024-03-18T15:16:23.728 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:21.458+0000 7f289f400700 1 -- 172.21.15.19:0/513317740 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f28980cfcc0 con 0x7f28a8069f80 2024-03-18T15:16:23.728 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:21.458+0000 7f28a7400700 1 --2- 172.21.15.19:0/513317740 >> [v2:172.21.15.33:6800/1951880113,v1:172.21.15.33:6801/1951880113] conn(0x7f288c0adb00 0x7f288c0affa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:16:23.728 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:21.459+0000 7f28a6000700 1 --2- 172.21.15.19:0/513317740 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f288c0a29a0 0x7f288c0a4e60 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f28a8107180 tx=0x7f2894009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:16:23.729 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:21.459+0000 7f28a7400700 1 --2- 172.21.15.19:0/513317740 >> [v2:172.21.15.33:6800/1951880113,v1:172.21.15.33:6801/1951880113] conn(0x7f288c0adb00 0x7f288c0affa0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:16:23.729 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:21.460+0000 7f289f400700 1 -- 172.21.15.19:0/513317740 <== osd.1 v2:172.21.15.33:6800/1951880113 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f289801bec0 con 0x7f288c0adb00 2024-03-18T15:16:23.729 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:21.478+0000 7f28ada00700 1 -- 172.21.15.19:0/513317740 --> [v2:172.21.15.33:6800/1951880113,v1:172.21.15.33:6801/1951880113] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f28a8062de0 con 0x7f288c0adb00 2024-03-18T15:16:23.729 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:21.479+0000 7f289f400700 1 -- 172.21.15.19:0/513317740 <== osd.1 v2:172.21.15.33:6800/1951880113 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f28a8062de0 con 0x7f288c0adb00 2024-03-18T15:16:23.729 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:21.480+0000 7f28ada00700 1 -- 172.21.15.19:0/513317740 >> [v2:172.21.15.33:6800/1951880113,v1:172.21.15.33:6801/1951880113] conn(0x7f288c0adb00 msgr2=0x7f288c0affa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:23.729 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:21.480+0000 7f28ada00700 1 --2- 172.21.15.19:0/513317740 >> [v2:172.21.15.33:6800/1951880113,v1:172.21.15.33:6801/1951880113] conn(0x7f288c0adb00 0x7f288c0affa0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:23.729 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:21.480+0000 7f28ada00700 1 -- 172.21.15.19:0/513317740 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f288c0a29a0 msgr2=0x7f288c0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:23.729 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:21.480+0000 7f28ada00700 1 --2- 172.21.15.19:0/513317740 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f288c0a29a0 0x7f288c0a4e60 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f28a8107180 tx=0x7f2894009260 comp rx=0 tx=0).stop 2024-03-18T15:16:23.729 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:21.480+0000 7f28ada00700 1 -- 172.21.15.19:0/513317740 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f28a8069f80 msgr2=0x7f28a81015f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:23.729 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:21.480+0000 7f28ada00700 1 --2- 172.21.15.19:0/513317740 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f28a8069f80 0x7f28a81015f0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f289800e9a0 tx=0x7f289800ecb0 comp rx=0 tx=0).stop 2024-03-18T15:16:23.730 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:21.480+0000 7f28a7400700 1 -- 172.21.15.19:0/513317740 reap_dead start 2024-03-18T15:16:23.730 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:21.480+0000 7f28ada00700 1 -- 172.21.15.19:0/513317740 shutdown_connections 2024-03-18T15:16:23.730 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:21.480+0000 7f28ada00700 1 -- 172.21.15.19:0/513317740 >> 172.21.15.19:0/513317740 conn(0x7f28a8077f80 msgr2=0x7f28a8100130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:16:23.730 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:21.480+0000 7f28ada00700 1 -- 172.21.15.19:0/513317740 shutdown_connections 2024-03-18T15:16:23.730 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:21.480+0000 7f28ada00700 1 -- 172.21.15.19:0/513317740 wait complete. 2024-03-18T15:16:24.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:16:23 smithi104 ceph-mon[36070]: pgmap v355: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 255 B/s wr, 10 op/s; 77 B/s, 1 keys/s, 1 objects/s recovering 2024-03-18T15:16:24.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:16:23 smithi019 ceph-mon[31522]: pgmap v355: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 255 B/s wr, 10 op/s; 77 B/s, 1 keys/s, 1 objects/s recovering 2024-03-18T15:16:24.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:16:23 smithi033 ceph-mon[36648]: pgmap v355: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 255 B/s wr, 10 op/s; 77 B/s, 1 keys/s, 1 objects/s recovering 2024-03-18T15:16:24.760 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.759+0000 7f05e0400700 1 -- 172.21.15.19:0/481732077 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f05d8074500 msgr2=0x7f05d80748e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:24.761 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.759+0000 7f05e0400700 1 --2- 172.21.15.19:0/481732077 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f05d8074500 0x7f05d80748e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f05cc009990 tx=0x7f05cc009ca0 comp rx=0 tx=0).stop 2024-03-18T15:16:24.761 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.759+0000 7f05e0400700 1 -- 172.21.15.19:0/481732077 shutdown_connections 2024-03-18T15:16:24.761 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.759+0000 7f05e0400700 1 --2- 172.21.15.19:0/481732077 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f05d810ecd0 0x7f05d81110c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:24.761 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.759+0000 7f05e0400700 1 --2- 172.21.15.19:0/481732077 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f05d8074500 0x7f05d80748e0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:24.761 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.759+0000 7f05e0400700 1 --2- 172.21.15.19:0/481732077 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f05d8074e20 0x7f05d810e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:24.761 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.759+0000 7f05e0400700 1 -- 172.21.15.19:0/481732077 >> 172.21.15.19:0/481732077 conn(0x7f05d8079430 msgr2=0x7f05d8079840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:16:24.761 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.759+0000 7f05e0400700 1 -- 172.21.15.19:0/481732077 shutdown_connections 2024-03-18T15:16:24.761 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.759+0000 7f05e0400700 1 -- 172.21.15.19:0/481732077 wait complete. 2024-03-18T15:16:24.763 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.760+0000 7f05e0400700 1 Processor -- start 2024-03-18T15:16:24.763 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.760+0000 7f05e0400700 1 -- start start 2024-03-18T15:16:24.763 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.761+0000 7f05e0400700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f05d8074500 0x7f05d81a08f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:16:24.763 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.761+0000 7f05e0400700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f05d8074e20 0x7f05d81a0e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:16:24.763 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.761+0000 7f05e0400700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f05d810ecd0 0x7f05d819ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:16:24.763 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.761+0000 7f05e0400700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f05d8114880 con 0x7f05d8074e20 2024-03-18T15:16:24.763 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.761+0000 7f05e0400700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f05d8114700 con 0x7f05d810ecd0 2024-03-18T15:16:24.763 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.761+0000 7f05e0400700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f05d8114a00 con 0x7f05d8074500 2024-03-18T15:16:24.764 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.761+0000 7f05dda00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f05d8074500 0x7f05d81a08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:16:24.764 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.761+0000 7f05de400700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f05d810ecd0 0x7f05d819ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:16:24.764 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.761+0000 7f05dda00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f05d8074500 0x7f05d81a08f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.19:60724/0 (socket says 172.21.15.19:60724) 2024-03-18T15:16:24.764 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.761+0000 7f05dda00700 1 -- 172.21.15.19:0/1497256386 learned_addr learned my addr 172.21.15.19:0/1497256386 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:16:24.764 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.761+0000 7f05dd000700 1 --2- 172.21.15.19:0/1497256386 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f05d8074e20 0x7f05d81a0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:16:24.764 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.762+0000 7f05dda00700 1 -- 172.21.15.19:0/1497256386 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f05d810ecd0 msgr2=0x7f05d819ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:24.764 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.762+0000 7f05dda00700 1 --2- 172.21.15.19:0/1497256386 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f05d810ecd0 0x7f05d819ab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:24.764 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.762+0000 7f05dda00700 1 -- 172.21.15.19:0/1497256386 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f05d8074e20 msgr2=0x7f05d81a0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:24.764 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.762+0000 7f05dda00700 1 --2- 172.21.15.19:0/1497256386 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f05d8074e20 0x7f05d81a0e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:24.764 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.762+0000 7f05dda00700 1 -- 172.21.15.19:0/1497256386 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05d819b260 con 0x7f05d8074500 2024-03-18T15:16:24.764 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.762+0000 7f05dd000700 1 --2- 172.21.15.19:0/1497256386 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f05d8074e20 0x7f05d81a0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T15:16:24.764 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.762+0000 7f05dda00700 1 --2- 172.21.15.19:0/1497256386 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f05d8074500 0x7f05d81a08f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f05cc00bae0 tx=0x7f05cc00bb10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:16:24.765 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.762+0000 7f05d6000700 1 -- 172.21.15.19:0/1497256386 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05cc026070 con 0x7f05d8074500 2024-03-18T15:16:24.765 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.762+0000 7f05d6000700 1 -- 172.21.15.19:0/1497256386 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f05cc003d70 con 0x7f05d8074500 2024-03-18T15:16:24.765 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.762+0000 7f05d6000700 1 -- 172.21.15.19:0/1497256386 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05cc021410 con 0x7f05d8074500 2024-03-18T15:16:24.765 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.762+0000 7f05e0400700 1 -- 172.21.15.19:0/1497256386 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05d819b4f0 con 0x7f05d8074500 2024-03-18T15:16:24.765 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.763+0000 7f05e0400700 1 -- 172.21.15.19:0/1497256386 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05d81a77c0 con 0x7f05d8074500 2024-03-18T15:16:24.765 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.763+0000 7f05e0400700 1 -- 172.21.15.19:0/1497256386 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f05d80629e0 con 0x7f05d8074500 2024-03-18T15:16:24.766 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.766+0000 7f05d6000700 1 -- 172.21.15.19:0/1497256386 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f05cc029070 con 0x7f05d8074500 2024-03-18T15:16:24.768 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.767+0000 7f05d6000700 1 --2- 172.21.15.19:0/1497256386 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f05c00a2920 0x7f05c00a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:16:24.768 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.767+0000 7f05d6000700 1 -- 172.21.15.19:0/1497256386 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f05cc0d0580 con 0x7f05d8074500 2024-03-18T15:16:24.768 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.767+0000 7f05dd000700 1 --2- 172.21.15.19:0/1497256386 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f05c00a2920 0x7f05c00a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:16:24.769 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.767+0000 7f05d6000700 1 --2- 172.21.15.19:0/1497256386 >> [v2:172.21.15.33:6816/4238160988,v1:172.21.15.33:6817/4238160988] conn(0x7f05c00ada80 0x7f05c00aff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:16:24.769 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.767+0000 7f05d6000700 1 -- 172.21.15.19:0/1497256386 --> [v2:172.21.15.33:6816/4238160988,v1:172.21.15.33:6817/4238160988] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f05c00b05f0 con 0x7f05c00ada80 2024-03-18T15:16:24.769 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.767+0000 7f05d6000700 1 -- 172.21.15.19:0/1497256386 <== mon.1 v2:172.21.15.104:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f05cc0d0930 con 0x7f05d8074500 2024-03-18T15:16:24.769 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.768+0000 7f05de400700 1 --2- 172.21.15.19:0/1497256386 >> [v2:172.21.15.33:6816/4238160988,v1:172.21.15.33:6817/4238160988] conn(0x7f05c00ada80 0x7f05c00aff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:16:24.769 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.768+0000 7f05dd000700 1 --2- 172.21.15.19:0/1497256386 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f05c00a2920 0x7f05c00a4de0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f05d8101d70 tx=0x7f05c4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:16:24.769 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.768+0000 7f05de400700 1 --2- 172.21.15.19:0/1497256386 >> [v2:172.21.15.33:6816/4238160988,v1:172.21.15.33:6817/4238160988] conn(0x7f05c00ada80 0x7f05c00aff20 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:16:24.770 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.769+0000 7f05d6000700 1 -- 172.21.15.19:0/1497256386 <== osd.7 v2:172.21.15.33:6816/4238160988 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f05c00b05f0 con 0x7f05c00ada80 2024-03-18T15:16:24.789 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.788+0000 7f05e0400700 1 -- 172.21.15.19:0/1497256386 --> [v2:172.21.15.33:6816/4238160988,v1:172.21.15.33:6817/4238160988] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f05d8062de0 con 0x7f05c00ada80 2024-03-18T15:16:24.789 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.788+0000 7f05d6000700 1 -- 172.21.15.19:0/1497256386 <== osd.7 v2:172.21.15.33:6816/4238160988 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f05d8062de0 con 0x7f05c00ada80 2024-03-18T15:16:24.790 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.789+0000 7f05e0400700 1 -- 172.21.15.19:0/1497256386 >> [v2:172.21.15.33:6816/4238160988,v1:172.21.15.33:6817/4238160988] conn(0x7f05c00ada80 msgr2=0x7f05c00aff20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:24.790 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.789+0000 7f05e0400700 1 --2- 172.21.15.19:0/1497256386 >> [v2:172.21.15.33:6816/4238160988,v1:172.21.15.33:6817/4238160988] conn(0x7f05c00ada80 0x7f05c00aff20 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:24.790 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.789+0000 7f05e0400700 1 -- 172.21.15.19:0/1497256386 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f05c00a2920 msgr2=0x7f05c00a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:24.790 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.789+0000 7f05e0400700 1 --2- 172.21.15.19:0/1497256386 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f05c00a2920 0x7f05c00a4de0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f05d8101d70 tx=0x7f05c4009260 comp rx=0 tx=0).stop 2024-03-18T15:16:24.790 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.789+0000 7f05e0400700 1 -- 172.21.15.19:0/1497256386 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f05d8074500 msgr2=0x7f05d81a08f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:24.790 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.789+0000 7f05e0400700 1 --2- 172.21.15.19:0/1497256386 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f05d8074500 0x7f05d81a08f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f05cc00bae0 tx=0x7f05cc00bb10 comp rx=0 tx=0).stop 2024-03-18T15:16:24.790 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.789+0000 7f05de400700 1 -- 172.21.15.19:0/1497256386 reap_dead start 2024-03-18T15:16:24.791 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.789+0000 7f05e0400700 1 -- 172.21.15.19:0/1497256386 shutdown_connections 2024-03-18T15:16:24.791 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.789+0000 7f05e0400700 1 -- 172.21.15.19:0/1497256386 >> 172.21.15.19:0/1497256386 conn(0x7f05d8079430 msgr2=0x7f05d80ff440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:16:24.791 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.790+0000 7f05e0400700 1 -- 172.21.15.19:0/1497256386 shutdown_connections 2024-03-18T15:16:24.791 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.790+0000 7f05e0400700 1 -- 172.21.15.19:0/1497256386 wait complete. 2024-03-18T15:16:24.865 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.478+0000 7fd985200700 1 -- 172.21.15.19:0/3105368743 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd980107210 msgr2=0x7fd9801075f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:24.865 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.478+0000 7fd985200700 1 --2- 172.21.15.19:0/3105368743 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd980107210 0x7fd9801075f0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fd970009990 tx=0x7fd970009ca0 comp rx=0 tx=0).stop 2024-03-18T15:16:24.865 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.478+0000 7fd985200700 1 -- 172.21.15.19:0/3105368743 shutdown_connections 2024-03-18T15:16:24.865 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.478+0000 7fd985200700 1 --2- 172.21.15.19:0/3105368743 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd980107210 0x7fd9801075f0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:24.865 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.478+0000 7fd985200700 1 --2- 172.21.15.19:0/3105368743 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd980101b70 0x7fd98010e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:24.865 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.478+0000 7fd985200700 1 --2- 172.21.15.19:0/3105368743 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd9801011d0 0x7fd980101630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:24.865 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.478+0000 7fd985200700 1 -- 172.21.15.19:0/3105368743 >> 172.21.15.19:0/3105368743 conn(0x7fd980078100 msgr2=0x7fd980078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:16:24.865 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.478+0000 7fd985200700 1 -- 172.21.15.19:0/3105368743 shutdown_connections 2024-03-18T15:16:24.865 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.478+0000 7fd985200700 1 -- 172.21.15.19:0/3105368743 wait complete. 2024-03-18T15:16:24.865 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.480+0000 7fd985200700 1 Processor -- start 2024-03-18T15:16:24.866 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.480+0000 7fd985200700 1 -- start start 2024-03-18T15:16:24.866 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.480+0000 7fd985200700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd9801011d0 0x7fd98019e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:16:24.866 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.480+0000 7fd985200700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd980101b70 0x7fd98019ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:16:24.866 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.480+0000 7fd985200700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd980107210 0x7fd9801988f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:16:24.866 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.480+0000 7fd985200700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fd9801114a0 con 0x7fd980107210 2024-03-18T15:16:24.866 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.480+0000 7fd985200700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd980111320 con 0x7fd980101b70 2024-03-18T15:16:24.866 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.480+0000 7fd985200700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fd980111620 con 0x7fd9801011d0 2024-03-18T15:16:24.866 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.481+0000 7fd97de00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd9801011d0 0x7fd98019e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:16:24.866 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.481+0000 7fd97d400700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd980101b70 0x7fd98019ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:16:24.866 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.481+0000 7fd97de00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd9801011d0 0x7fd98019e6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.19:60716/0 (socket says 172.21.15.19:60716) 2024-03-18T15:16:24.866 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.481+0000 7fd97de00700 1 -- 172.21.15.19:0/1072056433 learned_addr learned my addr 172.21.15.19:0/1072056433 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:16:24.866 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.481+0000 7fd97e800700 1 --2- 172.21.15.19:0/1072056433 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd980107210 0x7fd9801988f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:16:24.867 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.481+0000 7fd97de00700 1 -- 172.21.15.19:0/1072056433 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd980101b70 msgr2=0x7fd98019ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:24.867 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.481+0000 7fd97de00700 1 --2- 172.21.15.19:0/1072056433 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd980101b70 0x7fd98019ec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:24.867 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.481+0000 7fd97de00700 1 -- 172.21.15.19:0/1072056433 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd980107210 msgr2=0x7fd9801988f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:24.867 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.481+0000 7fd97de00700 1 --2- 172.21.15.19:0/1072056433 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd980107210 0x7fd9801988f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:24.867 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.481+0000 7fd97de00700 1 -- 172.21.15.19:0/1072056433 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd980199050 con 0x7fd9801011d0 2024-03-18T15:16:24.867 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.481+0000 7fd97de00700 1 --2- 172.21.15.19:0/1072056433 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd9801011d0 0x7fd98019e6e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fd970000c00 tx=0x7fd97000bab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:16:24.867 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.482+0000 7fd976a00700 1 -- 172.21.15.19:0/1072056433 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd970027070 con 0x7fd9801011d0 2024-03-18T15:16:24.867 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.482+0000 7fd976a00700 1 -- 172.21.15.19:0/1072056433 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd9700056d0 con 0x7fd9801011d0 2024-03-18T15:16:24.867 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.482+0000 7fd976a00700 1 -- 172.21.15.19:0/1072056433 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd970022e10 con 0x7fd9801011d0 2024-03-18T15:16:24.868 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.482+0000 7fd985200700 1 -- 172.21.15.19:0/1072056433 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9801992e0 con 0x7fd9801011d0 2024-03-18T15:16:24.868 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.482+0000 7fd985200700 1 -- 172.21.15.19:0/1072056433 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9801a55b0 con 0x7fd9801011d0 2024-03-18T15:16:24.868 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.483+0000 7fd985200700 1 -- 172.21.15.19:0/1072056433 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd980105310 con 0x7fd9801011d0 2024-03-18T15:16:24.868 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.485+0000 7fd976a00700 1 -- 172.21.15.19:0/1072056433 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fd97002a070 con 0x7fd9801011d0 2024-03-18T15:16:24.868 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.486+0000 7fd976a00700 1 --2- 172.21.15.19:0/1072056433 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fd9640a29a0 0x7fd9640a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:16:24.868 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.487+0000 7fd976a00700 1 -- 172.21.15.19:0/1072056433 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fd9700d03d0 con 0x7fd9801011d0 2024-03-18T15:16:24.868 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.487+0000 7fd97d400700 1 --2- 172.21.15.19:0/1072056433 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fd9640a29a0 0x7fd9640a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:16:24.868 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.487+0000 7fd976a00700 1 --2- 172.21.15.19:0/1072056433 >> [v2:172.21.15.33:6824/1346058869,v1:172.21.15.33:6825/1346058869] conn(0x7fd9640adb00 0x7fd9640affa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:16:24.868 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.487+0000 7fd976a00700 1 -- 172.21.15.19:0/1072056433 --> [v2:172.21.15.33:6824/1346058869,v1:172.21.15.33:6825/1346058869] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd970025e70 con 0x7fd9640adb00 2024-03-18T15:16:24.868 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.487+0000 7fd976a00700 1 -- 172.21.15.19:0/1072056433 <== mon.1 v2:172.21.15.104:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7fd9700d0820 con 0x7fd9801011d0 2024-03-18T15:16:24.868 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.487+0000 7fd97e800700 1 --2- 172.21.15.19:0/1072056433 >> [v2:172.21.15.33:6824/1346058869,v1:172.21.15.33:6825/1346058869] conn(0x7fd9640adb00 0x7fd9640affa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:16:24.868 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.487+0000 7fd97d400700 1 --2- 172.21.15.19:0/1072056433 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fd9640a29a0 0x7fd9640a4e60 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fd980078700 tx=0x7fd96c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:16:24.868 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.488+0000 7fd97e800700 1 --2- 172.21.15.19:0/1072056433 >> [v2:172.21.15.33:6824/1346058869,v1:172.21.15.33:6825/1346058869] conn(0x7fd9640adb00 0x7fd9640affa0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:16:24.869 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.489+0000 7fd976a00700 1 -- 172.21.15.19:0/1072056433 <== osd.10 v2:172.21.15.33:6824/1346058869 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd970025e70 con 0x7fd9640adb00 2024-03-18T15:16:24.869 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.507+0000 7fd985200700 1 -- 172.21.15.19:0/1072056433 --> [v2:172.21.15.33:6824/1346058869,v1:172.21.15.33:6825/1346058869] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fd9801028c0 con 0x7fd9640adb00 2024-03-18T15:16:24.869 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.508+0000 7fd976a00700 1 -- 172.21.15.19:0/1072056433 <== osd.10 v2:172.21.15.33:6824/1346058869 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fd9801028c0 con 0x7fd9640adb00 2024-03-18T15:16:24.869 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.508+0000 7fd985200700 1 -- 172.21.15.19:0/1072056433 >> [v2:172.21.15.33:6824/1346058869,v1:172.21.15.33:6825/1346058869] conn(0x7fd9640adb00 msgr2=0x7fd9640affa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:24.869 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.508+0000 7fd985200700 1 --2- 172.21.15.19:0/1072056433 >> [v2:172.21.15.33:6824/1346058869,v1:172.21.15.33:6825/1346058869] conn(0x7fd9640adb00 0x7fd9640affa0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:24.869 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.508+0000 7fd985200700 1 -- 172.21.15.19:0/1072056433 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fd9640a29a0 msgr2=0x7fd9640a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:24.869 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.508+0000 7fd985200700 1 --2- 172.21.15.19:0/1072056433 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fd9640a29a0 0x7fd9640a4e60 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fd980078700 tx=0x7fd96c009260 comp rx=0 tx=0).stop 2024-03-18T15:16:24.869 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.508+0000 7fd985200700 1 -- 172.21.15.19:0/1072056433 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd9801011d0 msgr2=0x7fd98019e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:24.869 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.508+0000 7fd985200700 1 --2- 172.21.15.19:0/1072056433 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd9801011d0 0x7fd98019e6e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fd970000c00 tx=0x7fd97000bab0 comp rx=0 tx=0).stop 2024-03-18T15:16:24.869 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.508+0000 7fd97e800700 1 -- 172.21.15.19:0/1072056433 reap_dead start 2024-03-18T15:16:24.869 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.509+0000 7fd985200700 1 -- 172.21.15.19:0/1072056433 shutdown_connections 2024-03-18T15:16:24.869 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.509+0000 7fd985200700 1 -- 172.21.15.19:0/1072056433 >> 172.21.15.19:0/1072056433 conn(0x7fd980078100 msgr2=0x7fd980100550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:16:24.869 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.509+0000 7fd985200700 1 -- 172.21.15.19:0/1072056433 shutdown_connections 2024-03-18T15:16:24.869 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:24.509+0000 7fd985200700 1 -- 172.21.15.19:0/1072056433 wait complete. 2024-03-18T15:16:25.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:16:25 smithi019 ceph-mon[31522]: pgmap v356: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 214 B/s wr, 10 op/s; 65 B/s, 1 keys/s, 0 objects/s recovering 2024-03-18T15:16:26.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:16:25 smithi104 ceph-mon[36070]: pgmap v356: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 214 B/s wr, 10 op/s; 65 B/s, 1 keys/s, 0 objects/s recovering 2024-03-18T15:16:26.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:16:25 smithi033 ceph-mon[36648]: pgmap v356: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 214 B/s wr, 10 op/s; 65 B/s, 1 keys/s, 0 objects/s recovering 2024-03-18T15:16:26.385 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:26.383+0000 7f3bcfc00700 1 -- 172.21.15.19:0/3166244267 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3bc81088e0 msgr2=0x7f3bc8108d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:26.385 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:26.383+0000 7f3bcfc00700 1 --2- 172.21.15.19:0/3166244267 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3bc81088e0 0x7f3bc8108d60 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f3bb0009990 tx=0x7f3bb0009ca0 comp rx=0 tx=0).stop 2024-03-18T15:16:26.385 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:26.383+0000 7f3bcfc00700 1 -- 172.21.15.19:0/3166244267 shutdown_connections 2024-03-18T15:16:26.385 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:26.383+0000 7f3bcfc00700 1 --2- 172.21.15.19:0/3166244267 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3bc81092a0 0x7f3bc81157b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:26.385 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:26.383+0000 7f3bcfc00700 1 --2- 172.21.15.19:0/3166244267 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3bc810e8e0 0x7f3bc810ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:26.385 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:26.383+0000 7f3bcfc00700 1 --2- 172.21.15.19:0/3166244267 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3bc81088e0 0x7f3bc8108d60 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:26.385 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:26.383+0000 7f3bcfc00700 1 -- 172.21.15.19:0/3166244267 >> 172.21.15.19:0/3166244267 conn(0x7f3bc8075aa0 msgr2=0x7f3bc8077ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:16:26.385 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:26.383+0000 7f3bcfc00700 1 -- 172.21.15.19:0/3166244267 shutdown_connections 2024-03-18T15:16:26.385 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:26.383+0000 7f3bcfc00700 1 -- 172.21.15.19:0/3166244267 wait complete. 2024-03-18T15:16:26.386 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:26.385+0000 7f3bcfc00700 1 Processor -- start 2024-03-18T15:16:26.386 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:26.385+0000 7f3bcfc00700 1 -- start start 2024-03-18T15:16:26.387 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:26.385+0000 7f3bcfc00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3bc81088e0 0x7f3bc819f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:16:26.387 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:26.385+0000 7f3bcfc00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3bc81092a0 0x7f3bc819f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:16:26.387 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:26.385+0000 7f3bcfc00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3bc810e8e0 0x7f3bc81a3960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:16:26.387 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:26.385+0000 7f3bcfc00700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f3bc8118cc0 con 0x7f3bc81092a0 2024-03-18T15:16:26.387 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:26.385+0000 7f3bcfc00700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f3bc8118b40 con 0x7f3bc810e8e0 2024-03-18T15:16:26.388 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:26.385+0000 7f3bcfc00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f3bc8118e40 con 0x7f3bc81088e0 2024-03-18T15:16:26.388 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:26.386+0000 7f3bcd200700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3bc81088e0 0x7f3bc819f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:16:26.388 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:26.386+0000 7f3bcd200700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3bc81088e0 0x7f3bc819f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.19:60740/0 (socket says 172.21.15.19:60740) 2024-03-18T15:16:26.388 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:26.386+0000 7f3bcd200700 1 -- 172.21.15.19:0/2679835927 learned_addr learned my addr 172.21.15.19:0/2679835927 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:16:26.388 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:26.386+0000 7f3bbfe00700 1 --2- 172.21.15.19:0/2679835927 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3bc81092a0 0x7f3bc819f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:16:26.388 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:26.386+0000 7f3bcdc00700 1 --2- 172.21.15.19:0/2679835927 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3bc810e8e0 0x7f3bc81a3960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:16:26.388 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:26.386+0000 7f3bcd200700 1 -- 172.21.15.19:0/2679835927 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3bc810e8e0 msgr2=0x7f3bc81a3960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:26.388 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:26.386+0000 7f3bcd200700 1 --2- 172.21.15.19:0/2679835927 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3bc810e8e0 0x7f3bc81a3960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:26.388 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:26.386+0000 7f3bcd200700 1 -- 172.21.15.19:0/2679835927 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3bc81092a0 msgr2=0x7f3bc819f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:26.388 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:26.386+0000 7f3bcd200700 1 --2- 172.21.15.19:0/2679835927 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3bc81092a0 0x7f3bc819f5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:26.388 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:26.386+0000 7f3bcd200700 1 -- 172.21.15.19:0/2679835927 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bc81a40a0 con 0x7f3bc81088e0 2024-03-18T15:16:26.389 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:26.387+0000 7f3bcd200700 1 --2- 172.21.15.19:0/2679835927 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3bc81088e0 0x7f3bc819f090 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f3bb800e9a0 tx=0x7f3bb800ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:16:26.389 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:26.387+0000 7f3bc6000700 1 -- 172.21.15.19:0/2679835927 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bb800c9c0 con 0x7f3bc81088e0 2024-03-18T15:16:26.389 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:26.387+0000 7f3bc6000700 1 -- 172.21.15.19:0/2679835927 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3bb800cb60 con 0x7f3bc81088e0 2024-03-18T15:16:26.389 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:26.387+0000 7f3bc6000700 1 -- 172.21.15.19:0/2679835927 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bb8018660 con 0x7f3bc81088e0 2024-03-18T15:16:26.390 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:26.387+0000 7f3bcfc00700 1 -- 172.21.15.19:0/2679835927 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3bc81a4390 con 0x7f3bc81088e0 2024-03-18T15:16:26.390 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:26.387+0000 7f3bcfc00700 1 -- 172.21.15.19:0/2679835927 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3bc81abc10 con 0x7f3bc81088e0 2024-03-18T15:16:26.390 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:26.388+0000 7f3bcfc00700 1 -- 172.21.15.19:0/2679835927 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3bc8062a60 con 0x7f3bc81088e0 2024-03-18T15:16:26.391 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:26.390+0000 7f3bc6000700 1 -- 172.21.15.19:0/2679835927 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f3bb800cd10 con 0x7f3bc81088e0 2024-03-18T15:16:26.393 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:26.391+0000 7f3bc6000700 1 --2- 172.21.15.19:0/2679835927 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f3bac0a29a0 0x7f3bac0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:16:26.393 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:26.391+0000 7f3bc6000700 1 -- 172.21.15.19:0/2679835927 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f3bb80cefe0 con 0x7f3bc81088e0 2024-03-18T15:16:26.393 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:26.392+0000 7f3bbfe00700 1 --2- 172.21.15.19:0/2679835927 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f3bac0a29a0 0x7f3bac0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:16:26.393 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:26.392+0000 7f3bc6000700 1 --2- 172.21.15.19:0/2679835927 >> [v2:172.21.15.104:6808/1258208297,v1:172.21.15.104:6809/1258208297] conn(0x7f3bac0adb00 0x7f3bac0affa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:16:26.393 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:26.392+0000 7f3bc6000700 1 -- 172.21.15.19:0/2679835927 --> [v2:172.21.15.104:6808/1258208297,v1:172.21.15.104:6809/1258208297] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3bb8012ec0 con 0x7f3bac0adb00 2024-03-18T15:16:26.393 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:26.392+0000 7f3bc6000700 1 -- 172.21.15.19:0/2679835927 <== mon.1 v2:172.21.15.104:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f3bb80d6050 con 0x7f3bc81088e0 2024-03-18T15:16:26.394 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:26.392+0000 7f3bcdc00700 1 --2- 172.21.15.19:0/2679835927 >> [v2:172.21.15.104:6808/1258208297,v1:172.21.15.104:6809/1258208297] conn(0x7f3bac0adb00 0x7f3bac0affa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:16:26.394 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:26.392+0000 7f3bbfe00700 1 --2- 172.21.15.19:0/2679835927 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f3bac0a29a0 0x7f3bac0a4e60 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f3bc80780f0 tx=0x7f3bb0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:16:26.394 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:26.393+0000 7f3bcdc00700 1 --2- 172.21.15.19:0/2679835927 >> [v2:172.21.15.104:6808/1258208297,v1:172.21.15.104:6809/1258208297] conn(0x7f3bac0adb00 0x7f3bac0affa0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:16:26.394 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:26.394+0000 7f3bc6000700 1 -- 172.21.15.19:0/2679835927 <== osd.5 v2:172.21.15.104:6808/1258208297 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3bb8012ec0 con 0x7f3bac0adb00 2024-03-18T15:16:26.414 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:26.412+0000 7f3bcfc00700 1 -- 172.21.15.19:0/2679835927 --> [v2:172.21.15.104:6808/1258208297,v1:172.21.15.104:6809/1258208297] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f3bc8062f30 con 0x7f3bac0adb00 2024-03-18T15:16:26.414 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:26.413+0000 7f3bc6000700 1 -- 172.21.15.19:0/2679835927 <== osd.5 v2:172.21.15.104:6808/1258208297 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f3bc8062f30 con 0x7f3bac0adb00 2024-03-18T15:16:26.415 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:26.413+0000 7f3bcfc00700 1 -- 172.21.15.19:0/2679835927 >> [v2:172.21.15.104:6808/1258208297,v1:172.21.15.104:6809/1258208297] conn(0x7f3bac0adb00 msgr2=0x7f3bac0affa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:26.415 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:26.413+0000 7f3bcfc00700 1 --2- 172.21.15.19:0/2679835927 >> [v2:172.21.15.104:6808/1258208297,v1:172.21.15.104:6809/1258208297] conn(0x7f3bac0adb00 0x7f3bac0affa0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:26.415 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:26.413+0000 7f3bcfc00700 1 -- 172.21.15.19:0/2679835927 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f3bac0a29a0 msgr2=0x7f3bac0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:26.415 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:26.413+0000 7f3bcfc00700 1 --2- 172.21.15.19:0/2679835927 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f3bac0a29a0 0x7f3bac0a4e60 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f3bc80780f0 tx=0x7f3bb0005fd0 comp rx=0 tx=0).stop 2024-03-18T15:16:26.415 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:26.413+0000 7f3bcfc00700 1 -- 172.21.15.19:0/2679835927 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3bc81088e0 msgr2=0x7f3bc819f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:26.415 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:26.413+0000 7f3bcfc00700 1 --2- 172.21.15.19:0/2679835927 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3bc81088e0 0x7f3bc819f090 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f3bb800e9a0 tx=0x7f3bb800ecb0 comp rx=0 tx=0).stop 2024-03-18T15:16:26.415 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:26.413+0000 7f3bcdc00700 1 -- 172.21.15.19:0/2679835927 reap_dead start 2024-03-18T15:16:26.415 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:26.414+0000 7f3bcfc00700 1 -- 172.21.15.19:0/2679835927 shutdown_connections 2024-03-18T15:16:26.415 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:26.414+0000 7f3bcfc00700 1 -- 172.21.15.19:0/2679835927 >> 172.21.15.19:0/2679835927 conn(0x7f3bc8075aa0 msgr2=0x7f3bc8076230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:16:26.415 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:26.414+0000 7f3bcfc00700 1 -- 172.21.15.19:0/2679835927 shutdown_connections 2024-03-18T15:16:26.415 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:26.414+0000 7f3bcfc00700 1 -- 172.21.15.19:0/2679835927 wait complete. 2024-03-18T15:16:27.080 INFO:teuthology.orchestra.run.smithi019.stdout:55834574932 2024-03-18T15:16:27.081 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-18T15:16:28.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:16:27 smithi104 ceph-mon[36070]: pgmap v357: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 194 B/s wr, 10 op/s; 59 B/s, 0 keys/s, 0 objects/s recovering 2024-03-18T15:16:28.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:16:27 smithi019 ceph-mon[31522]: pgmap v357: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 194 B/s wr, 10 op/s; 59 B/s, 0 keys/s, 0 objects/s recovering 2024-03-18T15:16:28.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:16:27 smithi033 ceph-mon[36648]: pgmap v357: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 194 B/s wr, 10 op/s; 59 B/s, 0 keys/s, 0 objects/s recovering 2024-03-18T15:16:28.501 INFO:teuthology.orchestra.run.smithi019.stdout:77309411407 2024-03-18T15:16:28.501 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-18T15:16:28.557 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.796+0000 7f634ea00700 1 -- 172.21.15.19:0/1930030580 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f634810e8e0 msgr2=0x7f634810ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:28.557 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.797+0000 7f634ea00700 1 --2- 172.21.15.19:0/1930030580 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f634810e8e0 0x7f634810ecc0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f63380098f0 tx=0x7f6338009c00 comp rx=0 tx=0).stop 2024-03-18T15:16:28.557 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.797+0000 7f634ea00700 1 -- 172.21.15.19:0/1930030580 shutdown_connections 2024-03-18T15:16:28.557 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.797+0000 7f634ea00700 1 --2- 172.21.15.19:0/1930030580 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f63481092a0 0x7f63481157b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:28.558 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.797+0000 7f634ea00700 1 --2- 172.21.15.19:0/1930030580 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f634810e8e0 0x7f634810ecc0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:28.558 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.797+0000 7f634ea00700 1 --2- 172.21.15.19:0/1930030580 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f63481088e0 0x7f6348108d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:28.558 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.797+0000 7f634ea00700 1 -- 172.21.15.19:0/1930030580 >> 172.21.15.19:0/1930030580 conn(0x7f6348075aa0 msgr2=0x7f6348077ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:16:28.558 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.797+0000 7f634ea00700 1 -- 172.21.15.19:0/1930030580 shutdown_connections 2024-03-18T15:16:28.558 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.797+0000 7f634ea00700 1 -- 172.21.15.19:0/1930030580 wait complete. 2024-03-18T15:16:28.558 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.798+0000 7f634ea00700 1 Processor -- start 2024-03-18T15:16:28.558 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.798+0000 7f634ea00700 1 -- start start 2024-03-18T15:16:28.558 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.799+0000 7f634ea00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f63481088e0 0x7f634819f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:16:28.558 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.799+0000 7f634ea00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f63481092a0 0x7f634819f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:16:28.558 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.799+0000 7f634ea00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f634810e8e0 0x7f63481a39c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:16:28.558 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.799+0000 7f634ea00700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f6348118c90 con 0x7f634810e8e0 2024-03-18T15:16:28.558 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.799+0000 7f634ea00700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6348118b10 con 0x7f63481088e0 2024-03-18T15:16:28.558 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.799+0000 7f634ea00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f6348118e10 con 0x7f63481092a0 2024-03-18T15:16:28.559 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.799+0000 7f6347e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f63481088e0 0x7f634819f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:16:28.559 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.799+0000 7f6347e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f63481088e0 0x7f634819f140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:51008/0 (socket says 172.21.15.19:51008) 2024-03-18T15:16:28.559 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.799+0000 7f6347e00700 1 -- 172.21.15.19:0/1436005879 learned_addr learned my addr 172.21.15.19:0/1436005879 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:16:28.559 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.799+0000 7f633f400700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f63481092a0 0x7f634819f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:16:28.559 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.799+0000 7f634ca00700 1 --2- 172.21.15.19:0/1436005879 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f634810e8e0 0x7f63481a39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:16:28.559 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.800+0000 7f6347e00700 1 -- 172.21.15.19:0/1436005879 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f63481092a0 msgr2=0x7f634819f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:28.559 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.800+0000 7f6347e00700 1 --2- 172.21.15.19:0/1436005879 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f63481092a0 0x7f634819f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:28.559 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.800+0000 7f6347e00700 1 -- 172.21.15.19:0/1436005879 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f634810e8e0 msgr2=0x7f63481a39c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:28.559 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.800+0000 7f6347e00700 1 --2- 172.21.15.19:0/1436005879 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f634810e8e0 0x7f63481a39c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:28.559 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.800+0000 7f6347e00700 1 -- 172.21.15.19:0/1436005879 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63481a4100 con 0x7f63481088e0 2024-03-18T15:16:28.559 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.800+0000 7f633f400700 1 --2- 172.21.15.19:0/1436005879 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f63481092a0 0x7f634819f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:16:28.559 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.800+0000 7f6347e00700 1 --2- 172.21.15.19:0/1436005879 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f63481088e0 0x7f634819f140 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f6338005950 tx=0x7f633800fb20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:16:28.559 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.800+0000 7f6345600700 1 -- 172.21.15.19:0/1436005879 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63380051f0 con 0x7f63481088e0 2024-03-18T15:16:28.559 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.800+0000 7f6345600700 1 -- 172.21.15.19:0/1436005879 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f633800b560 con 0x7f63481088e0 2024-03-18T15:16:28.559 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.800+0000 7f634ea00700 1 -- 172.21.15.19:0/1436005879 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63481a4390 con 0x7f63481088e0 2024-03-18T15:16:28.560 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.800+0000 7f6345600700 1 -- 172.21.15.19:0/1436005879 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6338003a40 con 0x7f63481088e0 2024-03-18T15:16:28.560 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.800+0000 7f634ea00700 1 -- 172.21.15.19:0/1436005879 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63481abc70 con 0x7f63481088e0 2024-03-18T15:16:28.560 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.801+0000 7f634ea00700 1 -- 172.21.15.19:0/1436005879 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6348062a20 con 0x7f63481088e0 2024-03-18T15:16:28.560 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.803+0000 7f6345600700 1 -- 172.21.15.19:0/1436005879 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f633800ba80 con 0x7f63481088e0 2024-03-18T15:16:28.560 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.805+0000 7f6345600700 1 --2- 172.21.15.19:0/1436005879 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f632c0a29a0 0x7f632c0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:16:28.560 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.805+0000 7f6345600700 1 -- 172.21.15.19:0/1436005879 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f63380d0320 con 0x7f63481088e0 2024-03-18T15:16:28.560 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.805+0000 7f633f400700 1 --2- 172.21.15.19:0/1436005879 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f632c0a29a0 0x7f632c0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:16:28.560 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.805+0000 7f6345600700 1 --2- 172.21.15.19:0/1436005879 >> [v2:172.21.15.19:6810/3571525854,v1:172.21.15.19:6811/3571525854] conn(0x7f632c0ad9d0 0x7f632c0afe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:16:28.560 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.805+0000 7f6345600700 1 -- 172.21.15.19:0/1436005879 --> [v2:172.21.15.19:6810/3571525854,v1:172.21.15.19:6811/3571525854] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f633800b0b0 con 0x7f632c0ad9d0 2024-03-18T15:16:28.561 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.805+0000 7f6345600700 1 -- 172.21.15.19:0/1436005879 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f63380d06d0 con 0x7f63481088e0 2024-03-18T15:16:28.561 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.806+0000 7f634ca00700 1 --2- 172.21.15.19:0/1436005879 >> [v2:172.21.15.19:6810/3571525854,v1:172.21.15.19:6811/3571525854] conn(0x7f632c0ad9d0 0x7f632c0afe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:16:28.561 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.806+0000 7f633f400700 1 --2- 172.21.15.19:0/1436005879 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f632c0a29a0 0x7f632c0a4e60 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f634810dee0 tx=0x7f633000b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:16:28.561 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.806+0000 7f634ca00700 1 --2- 172.21.15.19:0/1436005879 >> [v2:172.21.15.19:6810/3571525854,v1:172.21.15.19:6811/3571525854] conn(0x7f632c0ad9d0 0x7f632c0afe70 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:16:28.561 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.807+0000 7f6345600700 1 -- 172.21.15.19:0/1436005879 <== osd.3 v2:172.21.15.19:6810/3571525854 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f633800b0b0 con 0x7f632c0ad9d0 2024-03-18T15:16:28.561 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.825+0000 7f634ea00700 1 -- 172.21.15.19:0/1436005879 --> [v2:172.21.15.19:6810/3571525854,v1:172.21.15.19:6811/3571525854] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f6348062e20 con 0x7f632c0ad9d0 2024-03-18T15:16:28.561 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.826+0000 7f6345600700 1 -- 172.21.15.19:0/1436005879 <== osd.3 v2:172.21.15.19:6810/3571525854 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f6348062e20 con 0x7f632c0ad9d0 2024-03-18T15:16:28.561 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.826+0000 7f634ea00700 1 -- 172.21.15.19:0/1436005879 >> [v2:172.21.15.19:6810/3571525854,v1:172.21.15.19:6811/3571525854] conn(0x7f632c0ad9d0 msgr2=0x7f632c0afe70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:28.561 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.826+0000 7f634ea00700 1 --2- 172.21.15.19:0/1436005879 >> [v2:172.21.15.19:6810/3571525854,v1:172.21.15.19:6811/3571525854] conn(0x7f632c0ad9d0 0x7f632c0afe70 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:28.561 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.826+0000 7f634ea00700 1 -- 172.21.15.19:0/1436005879 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f632c0a29a0 msgr2=0x7f632c0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:28.561 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.826+0000 7f634ea00700 1 --2- 172.21.15.19:0/1436005879 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f632c0a29a0 0x7f632c0a4e60 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f634810dee0 tx=0x7f633000b040 comp rx=0 tx=0).stop 2024-03-18T15:16:28.561 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.826+0000 7f634ea00700 1 -- 172.21.15.19:0/1436005879 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f63481088e0 msgr2=0x7f634819f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:28.561 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.826+0000 7f634ea00700 1 --2- 172.21.15.19:0/1436005879 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f63481088e0 0x7f634819f140 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f6338005950 tx=0x7f633800fb20 comp rx=0 tx=0).stop 2024-03-18T15:16:28.561 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.826+0000 7f634ca00700 1 -- 172.21.15.19:0/1436005879 reap_dead start 2024-03-18T15:16:28.561 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.827+0000 7f634ea00700 1 -- 172.21.15.19:0/1436005879 shutdown_connections 2024-03-18T15:16:28.562 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.827+0000 7f634ea00700 1 -- 172.21.15.19:0/1436005879 >> 172.21.15.19:0/1436005879 conn(0x7f6348075aa0 msgr2=0x7f63480760a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:16:28.562 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.827+0000 7f634ea00700 1 -- 172.21.15.19:0/1436005879 shutdown_connections 2024-03-18T15:16:28.562 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.827+0000 7f634ea00700 1 -- 172.21.15.19:0/1436005879 wait complete. 2024-03-18T15:16:28.826 INFO:teuthology.orchestra.run.smithi019.stdout:343597383708 2024-03-18T15:16:28.826 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.11 2024-03-18T15:16:29.303 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.345+0000 7f7482600700 1 -- 172.21.15.19:0/2671858663 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f747c101280 msgr2=0x7f747c10f1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:29.304 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.345+0000 7f7482600700 1 --2- 172.21.15.19:0/2671858663 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f747c101280 0x7f747c10f1e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f747400b0a0 tx=0x7f747400b3b0 comp rx=0 tx=0).stop 2024-03-18T15:16:29.304 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.346+0000 7f7482600700 1 -- 172.21.15.19:0/2671858663 shutdown_connections 2024-03-18T15:16:29.304 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.346+0000 7f7482600700 1 --2- 172.21.15.19:0/2671858663 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f747c107340 0x7f747c107720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:29.304 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.346+0000 7f7482600700 1 --2- 172.21.15.19:0/2671858663 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f747c101280 0x7f747c10f1e0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:29.304 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.346+0000 7f7482600700 1 --2- 172.21.15.19:0/2671858663 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f747c1008c0 0x7f747c100d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:29.304 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.346+0000 7f7482600700 1 -- 172.21.15.19:0/2671858663 >> 172.21.15.19:0/2671858663 conn(0x7f747c0777d0 msgr2=0x7f747c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:16:29.304 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.346+0000 7f7482600700 1 -- 172.21.15.19:0/2671858663 shutdown_connections 2024-03-18T15:16:29.304 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.346+0000 7f7482600700 1 -- 172.21.15.19:0/2671858663 wait complete. 2024-03-18T15:16:29.304 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.347+0000 7f7482600700 1 Processor -- start 2024-03-18T15:16:29.304 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.347+0000 7f7482600700 1 -- start start 2024-03-18T15:16:29.304 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.347+0000 7f7482600700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f747c1008c0 0x7f747c19ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:16:29.305 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.347+0000 7f7482600700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f747c101280 0x7f747c19b260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:16:29.305 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.347+0000 7f7482600700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f747c107340 0x7f747c19f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:16:29.305 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.347+0000 7f7482600700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f747c112750 con 0x7f747c101280 2024-03-18T15:16:29.305 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.347+0000 7f7482600700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f747c1125d0 con 0x7f747c107340 2024-03-18T15:16:29.305 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.347+0000 7f7482600700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f747c1128d0 con 0x7f747c1008c0 2024-03-18T15:16:29.305 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.348+0000 7f747b400700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f747c1008c0 0x7f747c19ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:16:29.305 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.348+0000 7f747b400700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f747c1008c0 0x7f747c19ad20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.19:60746/0 (socket says 172.21.15.19:60746) 2024-03-18T15:16:29.305 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.348+0000 7f747be00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f747c107340 0x7f747c19f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:16:29.305 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.348+0000 7f747b400700 1 -- 172.21.15.19:0/3444665047 learned_addr learned my addr 172.21.15.19:0/3444665047 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:16:29.305 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.348+0000 7f747aa00700 1 --2- 172.21.15.19:0/3444665047 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f747c101280 0x7f747c19b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:16:29.305 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.348+0000 7f747b400700 1 -- 172.21.15.19:0/3444665047 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f747c107340 msgr2=0x7f747c19f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:29.305 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.348+0000 7f747b400700 1 --2- 172.21.15.19:0/3444665047 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f747c107340 0x7f747c19f5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:29.305 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.348+0000 7f747b400700 1 -- 172.21.15.19:0/3444665047 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f747c101280 msgr2=0x7f747c19b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:29.306 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.348+0000 7f747b400700 1 --2- 172.21.15.19:0/3444665047 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f747c101280 0x7f747c19b260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:29.306 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.348+0000 7f747b400700 1 -- 172.21.15.19:0/3444665047 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f747c19fd30 con 0x7f747c1008c0 2024-03-18T15:16:29.306 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.348+0000 7f747aa00700 1 --2- 172.21.15.19:0/3444665047 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f747c101280 0x7f747c19b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T15:16:29.306 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.348+0000 7f747b400700 1 --2- 172.21.15.19:0/3444665047 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f747c1008c0 0x7f747c19ad20 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f746c00eb30 tx=0x7f746c00ee40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:16:29.306 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.349+0000 7f7473e00700 1 -- 172.21.15.19:0/3444665047 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f746c00cc90 con 0x7f747c1008c0 2024-03-18T15:16:29.306 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.349+0000 7f7482600700 1 -- 172.21.15.19:0/3444665047 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f747c1a0020 con 0x7f747c1008c0 2024-03-18T15:16:29.306 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.349+0000 7f7473e00700 1 -- 172.21.15.19:0/3444665047 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f746c004500 con 0x7f747c1008c0 2024-03-18T15:16:29.306 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.349+0000 7f7473e00700 1 -- 172.21.15.19:0/3444665047 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f746c010430 con 0x7f747c1008c0 2024-03-18T15:16:29.306 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.349+0000 7f7482600700 1 -- 172.21.15.19:0/3444665047 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f747c1a78a0 con 0x7f747c1008c0 2024-03-18T15:16:29.306 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.350+0000 7f7482600700 1 -- 172.21.15.19:0/3444665047 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f747c102cb0 con 0x7f747c1008c0 2024-03-18T15:16:29.306 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.352+0000 7f7473e00700 1 -- 172.21.15.19:0/3444665047 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f746c003f70 con 0x7f747c1008c0 2024-03-18T15:16:29.306 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.353+0000 7f7473e00700 1 --2- 172.21.15.19:0/3444665047 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f74600a2920 0x7f74600a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:16:29.307 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.353+0000 7f7473e00700 1 -- 172.21.15.19:0/3444665047 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f746c0cf310 con 0x7f747c1008c0 2024-03-18T15:16:29.307 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.353+0000 7f747aa00700 1 --2- 172.21.15.19:0/3444665047 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f74600a2920 0x7f74600a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:16:29.307 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.354+0000 7f7473e00700 1 --2- 172.21.15.19:0/3444665047 >> [v2:172.21.15.19:6826/2641151009,v1:172.21.15.19:6827/2641151009] conn(0x7f74600ada80 0x7f74600aff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:16:29.307 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.354+0000 7f7473e00700 1 -- 172.21.15.19:0/3444665047 --> [v2:172.21.15.19:6826/2641151009,v1:172.21.15.19:6827/2641151009] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f746c00be50 con 0x7f74600ada80 2024-03-18T15:16:29.307 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.354+0000 7f7473e00700 1 -- 172.21.15.19:0/3444665047 <== mon.1 v2:172.21.15.104:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f746c0d6050 con 0x7f747c1008c0 2024-03-18T15:16:29.307 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.354+0000 7f747be00700 1 --2- 172.21.15.19:0/3444665047 >> [v2:172.21.15.19:6826/2641151009,v1:172.21.15.19:6827/2641151009] conn(0x7f74600ada80 0x7f74600aff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:16:29.307 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.354+0000 7f747aa00700 1 --2- 172.21.15.19:0/3444665047 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f74600a2920 0x7f74600a4de0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f747c10f600 tx=0x7f7468009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:16:29.307 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.355+0000 7f747be00700 1 --2- 172.21.15.19:0/3444665047 >> [v2:172.21.15.19:6826/2641151009,v1:172.21.15.19:6827/2641151009] conn(0x7f74600ada80 0x7f74600aff20 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:16:29.308 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.356+0000 7f7473e00700 1 -- 172.21.15.19:0/3444665047 <== osd.9 v2:172.21.15.19:6826/2641151009 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f746c00be50 con 0x7f74600ada80 2024-03-18T15:16:29.308 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.379+0000 7f7482600700 1 -- 172.21.15.19:0/3444665047 --> [v2:172.21.15.19:6826/2641151009,v1:172.21.15.19:6827/2641151009] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f747c103be0 con 0x7f74600ada80 2024-03-18T15:16:29.308 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.380+0000 7f7473e00700 1 -- 172.21.15.19:0/3444665047 <== osd.9 v2:172.21.15.19:6826/2641151009 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f747c103be0 con 0x7f74600ada80 2024-03-18T15:16:29.308 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.380+0000 7f7482600700 1 -- 172.21.15.19:0/3444665047 >> [v2:172.21.15.19:6826/2641151009,v1:172.21.15.19:6827/2641151009] conn(0x7f74600ada80 msgr2=0x7f74600aff20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:29.308 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.380+0000 7f7482600700 1 --2- 172.21.15.19:0/3444665047 >> [v2:172.21.15.19:6826/2641151009,v1:172.21.15.19:6827/2641151009] conn(0x7f74600ada80 0x7f74600aff20 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:29.308 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.381+0000 7f7482600700 1 -- 172.21.15.19:0/3444665047 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f74600a2920 msgr2=0x7f74600a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:29.308 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.381+0000 7f7482600700 1 --2- 172.21.15.19:0/3444665047 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f74600a2920 0x7f74600a4de0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f747c10f600 tx=0x7f7468009260 comp rx=0 tx=0).stop 2024-03-18T15:16:29.308 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.381+0000 7f7482600700 1 -- 172.21.15.19:0/3444665047 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f747c1008c0 msgr2=0x7f747c19ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:29.308 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.381+0000 7f7482600700 1 --2- 172.21.15.19:0/3444665047 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f747c1008c0 0x7f747c19ad20 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f746c00eb30 tx=0x7f746c00ee40 comp rx=0 tx=0).stop 2024-03-18T15:16:29.308 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.381+0000 7f747be00700 1 -- 172.21.15.19:0/3444665047 reap_dead start 2024-03-18T15:16:29.308 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.381+0000 7f7482600700 1 -- 172.21.15.19:0/3444665047 shutdown_connections 2024-03-18T15:16:29.308 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.381+0000 7f7482600700 1 -- 172.21.15.19:0/3444665047 >> 172.21.15.19:0/3444665047 conn(0x7f747c0777d0 msgr2=0x7f747c105440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:16:29.308 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.381+0000 7f7482600700 1 -- 172.21.15.19:0/3444665047 shutdown_connections 2024-03-18T15:16:29.308 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:27.381+0000 7f7482600700 1 -- 172.21.15.19:0/3444665047 wait complete. 2024-03-18T15:16:29.902 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:16:29 smithi019 ceph-mon[31522]: pgmap v358: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 170 B/s wr, 10 op/s; 51 B/s, 0 keys/s, 0 objects/s recovering 2024-03-18T15:16:30.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:16:29 smithi104 ceph-mon[36070]: pgmap v358: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 170 B/s wr, 10 op/s; 51 B/s, 0 keys/s, 0 objects/s recovering 2024-03-18T15:16:30.227 INFO:teuthology.orchestra.run.smithi019.stdout:309237645345 2024-03-18T15:16:30.227 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.10 2024-03-18T15:16:30.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:16:29 smithi033 ceph-mon[36648]: pgmap v358: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 170 B/s wr, 10 op/s; 51 B/s, 0 keys/s, 0 objects/s recovering 2024-03-18T15:16:31.170 INFO:teuthology.orchestra.run.smithi019.stdout:287762808871 2024-03-18T15:16:31.170 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.9 2024-03-18T15:16:31.638 INFO:teuthology.orchestra.run.smithi019.stdout:107374182474 2024-03-18T15:16:31.639 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-18T15:16:32.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:16:31 smithi104 ceph-mon[36070]: pgmap v359: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s; 51 B/s, 0 objects/s recovering 2024-03-18T15:16:32.208 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:16:31 smithi019 ceph-mon[31522]: pgmap v359: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s; 51 B/s, 0 objects/s recovering 2024-03-18T15:16:32.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:16:31 smithi033 ceph-mon[36648]: pgmap v359: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s; 51 B/s, 0 objects/s recovering 2024-03-18T15:16:32.900 INFO:teuthology.orchestra.run.smithi019.stdout:141733920837 2024-03-18T15:16:32.900 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-18T15:16:33.079 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:32.718+0000 7f6c4b200700 1 -- 172.21.15.19:0/2043212170 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6c44108320 msgr2=0x7f6c44108700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:33.079 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:32.718+0000 7f6c4b200700 1 --2- 172.21.15.19:0/2043212170 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6c44108320 0x7f6c44108700 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f6c34009990 tx=0x7f6c34009ca0 comp rx=0 tx=0).stop 2024-03-18T15:16:33.079 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:32.718+0000 7f6c4b200700 1 -- 172.21.15.19:0/2043212170 shutdown_connections 2024-03-18T15:16:33.079 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:32.718+0000 7f6c4b200700 1 --2- 172.21.15.19:0/2043212170 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c4406a920 0x7f6c4410f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:33.079 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:32.718+0000 7f6c4b200700 1 --2- 172.21.15.19:0/2043212170 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6c44108320 0x7f6c44108700 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:33.080 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:32.718+0000 7f6c4b200700 1 --2- 172.21.15.19:0/2043212170 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6c44069f80 0x7f6c4406a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:33.080 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:32.718+0000 7f6c4b200700 1 -- 172.21.15.19:0/2043212170 >> 172.21.15.19:0/2043212170 conn(0x7f6c44078100 msgr2=0x7f6c44078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:16:33.080 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:32.718+0000 7f6c4b200700 1 -- 172.21.15.19:0/2043212170 shutdown_connections 2024-03-18T15:16:33.080 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:32.718+0000 7f6c4b200700 1 -- 172.21.15.19:0/2043212170 wait complete. 2024-03-18T15:16:33.080 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:32.720+0000 7f6c4b200700 1 Processor -- start 2024-03-18T15:16:33.080 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:32.720+0000 7f6c4b200700 1 -- start start 2024-03-18T15:16:33.080 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:32.721+0000 7f6c4b200700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c44069f80 0x7f6c4419ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:16:33.080 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:32.721+0000 7f6c4b200700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6c4406a920 0x7f6c4419b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:16:33.080 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:32.721+0000 7f6c4b200700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6c44108320 0x7f6c4419f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:16:33.080 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:32.721+0000 7f6c4b200700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f6c441125b0 con 0x7f6c4406a920 2024-03-18T15:16:33.080 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:32.721+0000 7f6c4b200700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6c44112430 con 0x7f6c44108320 2024-03-18T15:16:33.080 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:32.721+0000 7f6c4b200700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f6c44112730 con 0x7f6c44069f80 2024-03-18T15:16:33.080 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:32.721+0000 7f6c43e00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c44069f80 0x7f6c4419ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:16:33.080 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:32.721+0000 7f6c43e00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c44069f80 0x7f6c4419ad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.19:60484/0 (socket says 172.21.15.19:60484) 2024-03-18T15:16:33.081 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:32.721+0000 7f6c43400700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6c4406a920 0x7f6c4419b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:16:33.081 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:32.721+0000 7f6c49200700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6c44108320 0x7f6c4419f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:16:33.081 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:32.721+0000 7f6c43e00700 1 -- 172.21.15.19:0/1104778038 learned_addr learned my addr 172.21.15.19:0/1104778038 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:16:33.081 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:32.721+0000 7f6c43e00700 1 -- 172.21.15.19:0/1104778038 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6c44108320 msgr2=0x7f6c4419f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:33.081 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:32.721+0000 7f6c43e00700 1 --2- 172.21.15.19:0/1104778038 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6c44108320 0x7f6c4419f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:33.081 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:32.721+0000 7f6c43e00700 1 -- 172.21.15.19:0/1104778038 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6c4406a920 msgr2=0x7f6c4419b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:33.081 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:32.721+0000 7f6c43e00700 1 --2- 172.21.15.19:0/1104778038 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6c4406a920 0x7f6c4419b250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:33.081 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:32.721+0000 7f6c43e00700 1 -- 172.21.15.19:0/1104778038 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c4419fd20 con 0x7f6c44069f80 2024-03-18T15:16:33.081 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:32.721+0000 7f6c49200700 1 --2- 172.21.15.19:0/1104778038 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6c44108320 0x7f6c4419f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:16:33.081 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:32.721+0000 7f6c43400700 1 --2- 172.21.15.19:0/1104778038 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6c4406a920 0x7f6c4419b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:16:33.081 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:32.722+0000 7f6c43e00700 1 --2- 172.21.15.19:0/1104778038 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c44069f80 0x7f6c4419ad10 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f6c34005910 tx=0x7f6c340047b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:16:33.081 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:32.722+0000 7f6c40c00700 1 -- 172.21.15.19:0/1104778038 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c34027070 con 0x7f6c44069f80 2024-03-18T15:16:33.081 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:32.722+0000 7f6c40c00700 1 -- 172.21.15.19:0/1104778038 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6c3400be70 con 0x7f6c44069f80 2024-03-18T15:16:33.082 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:32.722+0000 7f6c40c00700 1 -- 172.21.15.19:0/1104778038 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c340173f0 con 0x7f6c44069f80 2024-03-18T15:16:33.082 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:32.722+0000 7f6c4b200700 1 -- 172.21.15.19:0/1104778038 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c4419ffb0 con 0x7f6c44069f80 2024-03-18T15:16:33.082 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:32.722+0000 7f6c4b200700 1 -- 172.21.15.19:0/1104778038 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c441a7890 con 0x7f6c44069f80 2024-03-18T15:16:33.082 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:32.723+0000 7f6c4b200700 1 -- 172.21.15.19:0/1104778038 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6c4410de90 con 0x7f6c44069f80 2024-03-18T15:16:33.082 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:32.725+0000 7f6c40c00700 1 -- 172.21.15.19:0/1104778038 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f6c3402a070 con 0x7f6c44069f80 2024-03-18T15:16:33.082 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:32.727+0000 7f6c40c00700 1 --2- 172.21.15.19:0/1104778038 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f6c280a2920 0x7f6c280a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:16:33.082 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:32.727+0000 7f6c40c00700 1 -- 172.21.15.19:0/1104778038 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f6c340d02f0 con 0x7f6c44069f80 2024-03-18T15:16:33.082 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:32.727+0000 7f6c43400700 1 --2- 172.21.15.19:0/1104778038 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f6c280a2920 0x7f6c280a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:16:33.082 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:32.727+0000 7f6c40c00700 1 --2- 172.21.15.19:0/1104778038 >> [v2:172.21.15.19:6802/402333948,v1:172.21.15.19:6803/402333948] conn(0x7f6c280ada80 0x7f6c280aff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:16:33.082 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:32.727+0000 7f6c40c00700 1 -- 172.21.15.19:0/1104778038 --> [v2:172.21.15.19:6802/402333948,v1:172.21.15.19:6803/402333948] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6c3400b0b0 con 0x7f6c280ada80 2024-03-18T15:16:33.082 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:32.727+0000 7f6c40c00700 1 -- 172.21.15.19:0/1104778038 <== mon.1 v2:172.21.15.104:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f6c340d06a0 con 0x7f6c44069f80 2024-03-18T15:16:33.082 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:32.728+0000 7f6c49200700 1 --2- 172.21.15.19:0/1104778038 >> [v2:172.21.15.19:6802/402333948,v1:172.21.15.19:6803/402333948] conn(0x7f6c280ada80 0x7f6c280aff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:16:33.082 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:32.728+0000 7f6c43400700 1 --2- 172.21.15.19:0/1104778038 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f6c280a2920 0x7f6c280a4de0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f6c44103210 tx=0x7f6c30009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:16:33.083 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:32.728+0000 7f6c49200700 1 --2- 172.21.15.19:0/1104778038 >> [v2:172.21.15.19:6802/402333948,v1:172.21.15.19:6803/402333948] conn(0x7f6c280ada80 0x7f6c280aff20 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:16:33.083 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:32.729+0000 7f6c40c00700 1 -- 172.21.15.19:0/1104778038 <== osd.0 v2:172.21.15.19:6802/402333948 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6c3400b0b0 con 0x7f6c280ada80 2024-03-18T15:16:33.083 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:32.747+0000 7f6c4b200700 1 -- 172.21.15.19:0/1104778038 --> [v2:172.21.15.19:6802/402333948,v1:172.21.15.19:6803/402333948] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f6c441039d0 con 0x7f6c280ada80 2024-03-18T15:16:33.083 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:32.748+0000 7f6c40c00700 1 -- 172.21.15.19:0/1104778038 <== osd.0 v2:172.21.15.19:6802/402333948 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f6c441039d0 con 0x7f6c280ada80 2024-03-18T15:16:33.083 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:32.749+0000 7f6c4b200700 1 -- 172.21.15.19:0/1104778038 >> [v2:172.21.15.19:6802/402333948,v1:172.21.15.19:6803/402333948] conn(0x7f6c280ada80 msgr2=0x7f6c280aff20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:33.083 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:32.749+0000 7f6c4b200700 1 --2- 172.21.15.19:0/1104778038 >> [v2:172.21.15.19:6802/402333948,v1:172.21.15.19:6803/402333948] conn(0x7f6c280ada80 0x7f6c280aff20 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:33.084 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:32.749+0000 7f6c4b200700 1 -- 172.21.15.19:0/1104778038 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f6c280a2920 msgr2=0x7f6c280a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:33.084 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:32.749+0000 7f6c4b200700 1 --2- 172.21.15.19:0/1104778038 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f6c280a2920 0x7f6c280a4de0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f6c44103210 tx=0x7f6c30009260 comp rx=0 tx=0).stop 2024-03-18T15:16:33.084 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:32.749+0000 7f6c4b200700 1 -- 172.21.15.19:0/1104778038 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c44069f80 msgr2=0x7f6c4419ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:33.084 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:32.749+0000 7f6c4b200700 1 --2- 172.21.15.19:0/1104778038 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c44069f80 0x7f6c4419ad10 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f6c34005910 tx=0x7f6c340047b0 comp rx=0 tx=0).stop 2024-03-18T15:16:33.084 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:32.750+0000 7f6c49200700 1 -- 172.21.15.19:0/1104778038 reap_dead start 2024-03-18T15:16:33.084 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:32.750+0000 7f6c4b200700 1 -- 172.21.15.19:0/1104778038 shutdown_connections 2024-03-18T15:16:33.084 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:32.750+0000 7f6c4b200700 1 -- 172.21.15.19:0/1104778038 >> 172.21.15.19:0/1104778038 conn(0x7f6c44078100 msgr2=0x7f6c4410d4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:16:33.084 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:32.750+0000 7f6c4b200700 1 -- 172.21.15.19:0/1104778038 shutdown_connections 2024-03-18T15:16:33.084 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:32.750+0000 7f6c4b200700 1 -- 172.21.15.19:0/1104778038 wait complete. 2024-03-18T15:16:33.641 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:33.594+0000 7f68dfe00700 1 -- 172.21.15.19:0/3852302144 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f68d81044e0 msgr2=0x7f68d8104960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:33.641 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:33.594+0000 7f68dfe00700 1 --2- 172.21.15.19:0/3852302144 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f68d81044e0 0x7f68d8104960 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f68c40099c0 tx=0x7f68c4009cd0 comp rx=0 tx=0).stop 2024-03-18T15:16:33.642 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:33.594+0000 7f68dfe00700 1 -- 172.21.15.19:0/3852302144 shutdown_connections 2024-03-18T15:16:33.642 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:33.594+0000 7f68dfe00700 1 --2- 172.21.15.19:0/3852302144 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f68d810a590 0x7f68d810a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:33.642 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:33.594+0000 7f68dfe00700 1 --2- 172.21.15.19:0/3852302144 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f68d8104ea0 0x7f68d8111360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:33.642 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:33.594+0000 7f68dfe00700 1 --2- 172.21.15.19:0/3852302144 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f68d81044e0 0x7f68d8104960 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:33.642 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:33.594+0000 7f68dfe00700 1 -- 172.21.15.19:0/3852302144 >> 172.21.15.19:0/3852302144 conn(0x7f68d80fffb0 msgr2=0x7f68d81023d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:16:33.642 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:33.594+0000 7f68dfe00700 1 -- 172.21.15.19:0/3852302144 shutdown_connections 2024-03-18T15:16:33.642 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:33.594+0000 7f68dfe00700 1 -- 172.21.15.19:0/3852302144 wait complete. 2024-03-18T15:16:33.642 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:33.595+0000 7f68dfe00700 1 Processor -- start 2024-03-18T15:16:33.642 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:33.596+0000 7f68dfe00700 1 -- start start 2024-03-18T15:16:33.642 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:33.596+0000 7f68dfe00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f68d81044e0 0x7f68d81a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:16:33.642 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:33.596+0000 7f68dfe00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f68d8104ea0 0x7f68d81a0d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:16:33.642 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:33.596+0000 7f68dfe00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f68d810a590 0x7f68d819a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:16:33.643 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:33.596+0000 7f68dfe00700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f68d8114840 con 0x7f68d81044e0 2024-03-18T15:16:33.643 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:33.596+0000 7f68dfe00700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f68d81146c0 con 0x7f68d810a590 2024-03-18T15:16:33.643 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:33.596+0000 7f68dfe00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f68d81149c0 con 0x7f68d8104ea0 2024-03-18T15:16:33.643 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:33.596+0000 7f68dca00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f68d8104ea0 0x7f68d81a0d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:16:33.643 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:33.596+0000 7f68dde00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f68d810a590 0x7f68d819a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:16:33.643 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:33.596+0000 7f68dca00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f68d8104ea0 0x7f68d81a0d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.19:60504/0 (socket says 172.21.15.19:60504) 2024-03-18T15:16:33.643 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:33.597+0000 7f68dca00700 1 -- 172.21.15.19:0/1427850525 learned_addr learned my addr 172.21.15.19:0/1427850525 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:16:33.643 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:33.597+0000 7f68dd400700 1 --2- 172.21.15.19:0/1427850525 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f68d81044e0 0x7f68d81a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:16:33.643 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:33.597+0000 7f68dde00700 1 -- 172.21.15.19:0/1427850525 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f68d8104ea0 msgr2=0x7f68d81a0d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:33.643 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:33.597+0000 7f68dde00700 1 --2- 172.21.15.19:0/1427850525 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f68d8104ea0 0x7f68d81a0d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:33.643 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:33.597+0000 7f68dde00700 1 -- 172.21.15.19:0/1427850525 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f68d81044e0 msgr2=0x7f68d81a0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:33.643 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:33.597+0000 7f68dde00700 1 --2- 172.21.15.19:0/1427850525 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f68d81044e0 0x7f68d81a0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:33.643 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:33.597+0000 7f68dde00700 1 -- 172.21.15.19:0/1427850525 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68d819b140 con 0x7f68d810a590 2024-03-18T15:16:33.644 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:33.597+0000 7f68dde00700 1 --2- 172.21.15.19:0/1427850525 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f68d810a590 0x7f68d819a9e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f68d000efc0 tx=0x7f68d000c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:16:33.644 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:33.597+0000 7f68d6000700 1 -- 172.21.15.19:0/1427850525 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68d00092e0 con 0x7f68d810a590 2024-03-18T15:16:33.644 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:33.597+0000 7f68d6000700 1 -- 172.21.15.19:0/1427850525 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f68d0004bb0 con 0x7f68d810a590 2024-03-18T15:16:33.644 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:33.597+0000 7f68d6000700 1 -- 172.21.15.19:0/1427850525 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68d0004280 con 0x7f68d810a590 2024-03-18T15:16:33.644 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:33.597+0000 7f68dfe00700 1 -- 172.21.15.19:0/1427850525 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68d819b3d0 con 0x7f68d810a590 2024-03-18T15:16:33.644 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:33.598+0000 7f68dfe00700 1 -- 172.21.15.19:0/1427850525 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68d81a76f0 con 0x7f68d810a590 2024-03-18T15:16:33.644 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:33.598+0000 7f68dfe00700 1 -- 172.21.15.19:0/1427850525 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f68d8101f50 con 0x7f68d810a590 2024-03-18T15:16:33.644 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:33.601+0000 7f68d6000700 1 -- 172.21.15.19:0/1427850525 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f68d001b030 con 0x7f68d810a590 2024-03-18T15:16:33.644 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:33.602+0000 7f68d6000700 1 --2- 172.21.15.19:0/1427850525 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f68c00a29a0 0x7f68c00a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:16:33.644 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:33.602+0000 7f68d6000700 1 -- 172.21.15.19:0/1427850525 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f68d00cf3e0 con 0x7f68d810a590 2024-03-18T15:16:33.644 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:33.602+0000 7f68dd400700 1 --2- 172.21.15.19:0/1427850525 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f68c00a29a0 0x7f68c00a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:16:33.644 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:33.602+0000 7f68d6000700 1 --2- 172.21.15.19:0/1427850525 >> [v2:172.21.15.19:6818/2782482301,v1:172.21.15.19:6819/2782482301] conn(0x7f68c00adb00 0x7f68c00affa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:16:33.644 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:33.602+0000 7f68d6000700 1 -- 172.21.15.19:0/1427850525 --> [v2:172.21.15.19:6818/2782482301,v1:172.21.15.19:6819/2782482301] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f68d000be50 con 0x7f68c00adb00 2024-03-18T15:16:33.645 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:33.602+0000 7f68d6000700 1 -- 172.21.15.19:0/1427850525 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f68d00cf830 con 0x7f68d810a590 2024-03-18T15:16:33.645 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:33.603+0000 7f68dca00700 1 --2- 172.21.15.19:0/1427850525 >> [v2:172.21.15.19:6818/2782482301,v1:172.21.15.19:6819/2782482301] conn(0x7f68c00adb00 0x7f68c00affa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:16:33.645 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:33.603+0000 7f68dd400700 1 --2- 172.21.15.19:0/1427850525 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f68c00a29a0 0x7f68c00a4e60 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f68d819bb20 tx=0x7f68cc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:16:33.645 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:33.603+0000 7f68dca00700 1 --2- 172.21.15.19:0/1427850525 >> [v2:172.21.15.19:6818/2782482301,v1:172.21.15.19:6819/2782482301] conn(0x7f68c00adb00 0x7f68c00affa0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:16:33.645 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:33.604+0000 7f68d6000700 1 -- 172.21.15.19:0/1427850525 <== osd.6 v2:172.21.15.19:6818/2782482301 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f68d000be50 con 0x7f68c00adb00 2024-03-18T15:16:33.645 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:33.623+0000 7f68dfe00700 1 -- 172.21.15.19:0/1427850525 --> [v2:172.21.15.19:6818/2782482301,v1:172.21.15.19:6819/2782482301] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f68d8105c00 con 0x7f68c00adb00 2024-03-18T15:16:33.645 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:33.623+0000 7f68d6000700 1 -- 172.21.15.19:0/1427850525 <== osd.6 v2:172.21.15.19:6818/2782482301 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f68d8105c00 con 0x7f68c00adb00 2024-03-18T15:16:33.645 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:33.624+0000 7f68dfe00700 1 -- 172.21.15.19:0/1427850525 >> [v2:172.21.15.19:6818/2782482301,v1:172.21.15.19:6819/2782482301] conn(0x7f68c00adb00 msgr2=0x7f68c00affa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:33.645 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:33.624+0000 7f68dfe00700 1 --2- 172.21.15.19:0/1427850525 >> [v2:172.21.15.19:6818/2782482301,v1:172.21.15.19:6819/2782482301] conn(0x7f68c00adb00 0x7f68c00affa0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:33.645 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:33.624+0000 7f68dfe00700 1 -- 172.21.15.19:0/1427850525 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f68c00a29a0 msgr2=0x7f68c00a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:33.645 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:33.624+0000 7f68dfe00700 1 --2- 172.21.15.19:0/1427850525 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f68c00a29a0 0x7f68c00a4e60 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f68d819bb20 tx=0x7f68cc009260 comp rx=0 tx=0).stop 2024-03-18T15:16:33.645 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:33.624+0000 7f68dfe00700 1 -- 172.21.15.19:0/1427850525 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f68d810a590 msgr2=0x7f68d819a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:33.645 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:33.624+0000 7f68dfe00700 1 --2- 172.21.15.19:0/1427850525 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f68d810a590 0x7f68d819a9e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f68d000efc0 tx=0x7f68d000c3a0 comp rx=0 tx=0).stop 2024-03-18T15:16:33.645 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:33.624+0000 7f68dde00700 1 -- 172.21.15.19:0/1427850525 reap_dead start 2024-03-18T15:16:33.645 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:33.624+0000 7f68dfe00700 1 -- 172.21.15.19:0/1427850525 shutdown_connections 2024-03-18T15:16:33.646 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:33.624+0000 7f68dfe00700 1 -- 172.21.15.19:0/1427850525 >> 172.21.15.19:0/1427850525 conn(0x7f68d80fffb0 msgr2=0x7f68d81014f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:16:33.646 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:33.625+0000 7f68dfe00700 1 -- 172.21.15.19:0/1427850525 shutdown_connections 2024-03-18T15:16:33.646 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:33.625+0000 7f68dfe00700 1 -- 172.21.15.19:0/1427850525 wait complete. 2024-03-18T15:16:34.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:16:33 smithi104 ceph-mon[36070]: pgmap v360: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-18T15:16:34.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:16:33 smithi019 ceph-mon[31522]: pgmap v360: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-18T15:16:34.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:16:33 smithi033 ceph-mon[36648]: pgmap v360: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-18T15:16:34.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:34.810+0000 7f21e7200700 1 -- 172.21.15.19:0/2088481240 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f21e8103c80 msgr2=0x7f21e810c170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:34.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:34.810+0000 7f21e7200700 1 --2- 172.21.15.19:0/2088481240 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f21e8103c80 0x7f21e810c170 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f21d4009990 tx=0x7f21d4009ca0 comp rx=0 tx=0).stop 2024-03-18T15:16:34.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:34.810+0000 7f21e7200700 1 -- 172.21.15.19:0/2088481240 shutdown_connections 2024-03-18T15:16:34.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:34.810+0000 7f21e7200700 1 --2- 172.21.15.19:0/2088481240 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f21e810c790 0x7f21e810ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:34.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:34.810+0000 7f21e7200700 1 --2- 172.21.15.19:0/2088481240 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f21e8103360 0x7f21e8103740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:34.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:34.810+0000 7f21e7200700 1 --2- 172.21.15.19:0/2088481240 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f21e8103c80 0x7f21e810c170 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:34.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:34.810+0000 7f21e7200700 1 -- 172.21.15.19:0/2088481240 >> 172.21.15.19:0/2088481240 conn(0x7f21e80fcb90 msgr2=0x7f21e80fefb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:16:34.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:34.810+0000 7f21e7200700 1 -- 172.21.15.19:0/2088481240 shutdown_connections 2024-03-18T15:16:34.813 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:34.810+0000 7f21e7200700 1 -- 172.21.15.19:0/2088481240 wait complete. 2024-03-18T15:16:34.813 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:34.812+0000 7f21e7200700 1 Processor -- start 2024-03-18T15:16:34.813 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:34.812+0000 7f21e7200700 1 -- start start 2024-03-18T15:16:34.813 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:34.812+0000 7f21e7200700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f21e8103360 0x7f21e81a0880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:16:34.813 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:34.812+0000 7f21e7200700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f21e8103c80 0x7f21e81a0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:16:34.814 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:34.812+0000 7f21e7200700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f21e810c790 0x7f21e819aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:16:34.814 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:34.813+0000 7f21e7200700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f21e8112520 con 0x7f21e8103c80 2024-03-18T15:16:34.814 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:34.813+0000 7f21e7200700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f21e81123a0 con 0x7f21e810c790 2024-03-18T15:16:34.814 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:34.813+0000 7f21e7200700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f21e81126a0 con 0x7f21e8103360 2024-03-18T15:16:34.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:34.813+0000 7f21e5e00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f21e8103360 0x7f21e81a0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:16:34.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:34.813+0000 7f21e6800700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f21e810c790 0x7f21e819aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:16:34.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:34.813+0000 7f21e5e00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f21e8103360 0x7f21e81a0880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.19:60524/0 (socket says 172.21.15.19:60524) 2024-03-18T15:16:34.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:34.813+0000 7f21e5400700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f21e8103c80 0x7f21e81a0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:16:34.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:34.813+0000 7f21e5e00700 1 -- 172.21.15.19:0/1535129939 learned_addr learned my addr 172.21.15.19:0/1535129939 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:16:34.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:34.813+0000 7f21e6800700 1 -- 172.21.15.19:0/1535129939 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f21e8103360 msgr2=0x7f21e81a0880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:34.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:34.813+0000 7f21e6800700 1 --2- 172.21.15.19:0/1535129939 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f21e8103360 0x7f21e81a0880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:34.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:34.813+0000 7f21e6800700 1 -- 172.21.15.19:0/1535129939 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f21e8103c80 msgr2=0x7f21e81a0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:34.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:34.813+0000 7f21e6800700 1 --2- 172.21.15.19:0/1535129939 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f21e8103c80 0x7f21e81a0dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:34.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:34.813+0000 7f21e6800700 1 -- 172.21.15.19:0/1535129939 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21e819b1a0 con 0x7f21e810c790 2024-03-18T15:16:34.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:34.813+0000 7f21e5e00700 1 --2- 172.21.15.19:0/1535129939 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f21e8103360 0x7f21e81a0880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:16:34.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:34.813+0000 7f21e5400700 1 --2- 172.21.15.19:0/1535129939 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f21e8103c80 0x7f21e81a0dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T15:16:34.816 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:34.814+0000 7f21e6800700 1 --2- 172.21.15.19:0/1535129939 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f21e810c790 0x7f21e819aa40 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f21e000efc0 tx=0x7f21e000c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:16:34.816 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:34.814+0000 7f21dea00700 1 -- 172.21.15.19:0/1535129939 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21e00092e0 con 0x7f21e810c790 2024-03-18T15:16:34.816 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:34.814+0000 7f21e7200700 1 -- 172.21.15.19:0/1535129939 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21e819b430 con 0x7f21e810c790 2024-03-18T15:16:34.816 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:34.814+0000 7f21dea00700 1 -- 172.21.15.19:0/1535129939 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f21e0004bb0 con 0x7f21e810c790 2024-03-18T15:16:34.816 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:34.814+0000 7f21dea00700 1 -- 172.21.15.19:0/1535129939 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21e0004280 con 0x7f21e810c790 2024-03-18T15:16:34.817 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:34.814+0000 7f21e7200700 1 -- 172.21.15.19:0/1535129939 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21e81a7750 con 0x7f21e810c790 2024-03-18T15:16:34.817 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:34.815+0000 7f21e7200700 1 -- 172.21.15.19:0/1535129939 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f21e8102ae0 con 0x7f21e810c790 2024-03-18T15:16:34.818 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:34.817+0000 7f21dea00700 1 -- 172.21.15.19:0/1535129939 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f21e001b030 con 0x7f21e810c790 2024-03-18T15:16:34.820 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:34.819+0000 7f21dea00700 1 --2- 172.21.15.19:0/1535129939 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f21cc0a2920 0x7f21cc0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:16:34.820 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:34.819+0000 7f21dea00700 1 -- 172.21.15.19:0/1535129939 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f21e00cf490 con 0x7f21e810c790 2024-03-18T15:16:34.820 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:34.819+0000 7f21e5e00700 1 --2- 172.21.15.19:0/1535129939 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f21cc0a2920 0x7f21cc0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:16:34.820 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:34.819+0000 7f21dea00700 1 --2- 172.21.15.19:0/1535129939 >> [v2:172.21.15.104:6816/1014779813,v1:172.21.15.104:6817/1014779813] conn(0x7f21cc0ada80 0x7f21cc0aff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:16:34.821 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:34.819+0000 7f21dea00700 1 -- 172.21.15.19:0/1535129939 --> [v2:172.21.15.104:6816/1014779813,v1:172.21.15.104:6817/1014779813] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f21e001fe50 con 0x7f21cc0ada80 2024-03-18T15:16:34.821 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:34.819+0000 7f21dea00700 1 -- 172.21.15.19:0/1535129939 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f21e00cf840 con 0x7f21e810c790 2024-03-18T15:16:34.821 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:34.820+0000 7f21e5400700 1 --2- 172.21.15.19:0/1535129939 >> [v2:172.21.15.104:6816/1014779813,v1:172.21.15.104:6817/1014779813] conn(0x7f21cc0ada80 0x7f21cc0aff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:16:34.821 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:34.820+0000 7f21e5e00700 1 --2- 172.21.15.19:0/1535129939 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f21cc0a2920 0x7f21cc0a4de0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f21e819bb80 tx=0x7f21d8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:16:34.821 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:34.820+0000 7f21e5400700 1 --2- 172.21.15.19:0/1535129939 >> [v2:172.21.15.104:6816/1014779813,v1:172.21.15.104:6817/1014779813] conn(0x7f21cc0ada80 0x7f21cc0aff20 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:16:34.822 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:34.821+0000 7f21dea00700 1 -- 172.21.15.19:0/1535129939 <== osd.8 v2:172.21.15.104:6816/1014779813 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f21e001fe50 con 0x7f21cc0ada80 2024-03-18T15:16:34.841 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:34.840+0000 7f21e7200700 1 -- 172.21.15.19:0/1535129939 --> [v2:172.21.15.104:6816/1014779813,v1:172.21.15.104:6817/1014779813] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f21e806a860 con 0x7f21cc0ada80 2024-03-18T15:16:34.841 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:34.840+0000 7f21dea00700 1 -- 172.21.15.19:0/1535129939 <== osd.8 v2:172.21.15.104:6816/1014779813 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f21e806a860 con 0x7f21cc0ada80 2024-03-18T15:16:34.842 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:34.841+0000 7f21e7200700 1 -- 172.21.15.19:0/1535129939 >> [v2:172.21.15.104:6816/1014779813,v1:172.21.15.104:6817/1014779813] conn(0x7f21cc0ada80 msgr2=0x7f21cc0aff20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:34.842 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:34.841+0000 7f21e7200700 1 --2- 172.21.15.19:0/1535129939 >> [v2:172.21.15.104:6816/1014779813,v1:172.21.15.104:6817/1014779813] conn(0x7f21cc0ada80 0x7f21cc0aff20 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:34.843 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:34.841+0000 7f21e7200700 1 -- 172.21.15.19:0/1535129939 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f21cc0a2920 msgr2=0x7f21cc0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:34.843 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:34.841+0000 7f21e7200700 1 --2- 172.21.15.19:0/1535129939 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f21cc0a2920 0x7f21cc0a4de0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f21e819bb80 tx=0x7f21d8009260 comp rx=0 tx=0).stop 2024-03-18T15:16:34.843 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:34.841+0000 7f21e7200700 1 -- 172.21.15.19:0/1535129939 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f21e810c790 msgr2=0x7f21e819aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:34.843 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:34.841+0000 7f21e7200700 1 --2- 172.21.15.19:0/1535129939 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f21e810c790 0x7f21e819aa40 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f21e000efc0 tx=0x7f21e000c3a0 comp rx=0 tx=0).stop 2024-03-18T15:16:34.843 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:34.841+0000 7f21e6800700 1 -- 172.21.15.19:0/1535129939 reap_dead start 2024-03-18T15:16:34.843 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:34.842+0000 7f21e7200700 1 -- 172.21.15.19:0/1535129939 shutdown_connections 2024-03-18T15:16:34.843 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:34.842+0000 7f21e7200700 1 -- 172.21.15.19:0/1535129939 >> 172.21.15.19:0/1535129939 conn(0x7f21e80fcb90 msgr2=0x7f21e80fef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:16:34.843 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:34.842+0000 7f21e7200700 1 -- 172.21.15.19:0/1535129939 shutdown_connections 2024-03-18T15:16:34.843 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:34.842+0000 7f21e7200700 1 -- 172.21.15.19:0/1535129939 wait complete. 2024-03-18T15:16:35.849 INFO:teuthology.orchestra.run.smithi019.stdout:219043332146 2024-03-18T15:16:35.849 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-18T15:16:35.971 INFO:teuthology.orchestra.run.smithi019.stdout:163208757311 2024-03-18T15:16:35.971 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-18T15:16:36.049 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:16:35 smithi019 ceph-mon[31522]: pgmap v361: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T15:16:36.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:16:35 smithi104 ceph-mon[36070]: pgmap v361: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T15:16:36.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:16:35 smithi033 ceph-mon[36648]: pgmap v361: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T15:16:36.606 INFO:teuthology.orchestra.run.smithi019.stdout:253403070512 2024-03-18T15:16:36.607 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.8 2024-03-18T15:16:36.936 INFO:teuthology.orchestra.run.smithi019.stdout:197568495674 2024-03-18T15:16:36.937 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-18T15:16:37.316 INFO:teuthology.orchestra.run.smithi019.stdout:34359738460 2024-03-18T15:16:37.316 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-18T15:16:37.607 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:16:37.616 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:16:37.623 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:16:37.637 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:16:37.637 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:16:37.649 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:16:37.658 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:16:37.665 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:16:37.690 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:16:37.712 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:16:37.721 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:16:38.109 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:16:37 smithi019 ceph-mon[31522]: pgmap v362: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T15:16:38.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:16:37 smithi104 ceph-mon[36070]: pgmap v362: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T15:16:38.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:16:37 smithi033 ceph-mon[36648]: pgmap v362: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T15:16:40.090 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:16:39 smithi019 ceph-mon[31522]: pgmap v363: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-03-18T15:16:40.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:16:39 smithi104 ceph-mon[36070]: pgmap v363: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-03-18T15:16:40.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:16:39 smithi033 ceph-mon[36648]: pgmap v363: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-03-18T15:16:42.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:16:41 smithi104 ceph-mon[36070]: pgmap v364: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-18T15:16:42.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:16:41 smithi019 ceph-mon[31522]: pgmap v364: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-18T15:16:42.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:16:41 smithi033 ceph-mon[36648]: pgmap v364: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-18T15:16:44.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:16:43 smithi104 ceph-mon[36070]: pgmap v365: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 11 op/s 2024-03-18T15:16:44.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:16:43 smithi019 ceph-mon[31522]: pgmap v365: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 11 op/s 2024-03-18T15:16:44.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:16:43 smithi033 ceph-mon[36648]: pgmap v365: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 11 op/s 2024-03-18T15:16:45.948 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:16:45 smithi019 ceph-mon[31522]: pgmap v366: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-03-18T15:16:46.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:16:45 smithi104 ceph-mon[36070]: pgmap v366: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-03-18T15:16:46.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:16:45 smithi033 ceph-mon[36648]: pgmap v366: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-03-18T15:16:47.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:16:47 smithi019 ceph-mon[31522]: pgmap v367: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-03-18T15:16:48.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:16:47 smithi104 ceph-mon[36070]: pgmap v367: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-03-18T15:16:48.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:16:47 smithi033 ceph-mon[36648]: pgmap v367: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-03-18T15:16:49.657 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.655+0000 7f8874800700 1 -- 172.21.15.19:0/4072087742 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f886c102580 msgr2=0x7f886c102960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:49.657 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.655+0000 7f8874800700 1 --2- 172.21.15.19:0/4072087742 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f886c102580 0x7f886c102960 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f8860009990 tx=0x7f8860009ca0 comp rx=0 tx=0).stop 2024-03-18T15:16:49.657 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.655+0000 7f8874800700 1 -- 172.21.15.19:0/4072087742 shutdown_connections 2024-03-18T15:16:49.657 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.655+0000 7f8874800700 1 --2- 172.21.15.19:0/4072087742 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f886c107650 0x7f886c10f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:49.657 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.655+0000 7f8874800700 1 --2- 172.21.15.19:0/4072087742 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f886c102580 0x7f886c102960 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:49.657 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.655+0000 7f8874800700 1 --2- 172.21.15.19:0/4072087742 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f886c102f30 0x7f886c106f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:49.657 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.655+0000 7f8874800700 1 -- 172.21.15.19:0/4072087742 >> 172.21.15.19:0/4072087742 conn(0x7f886c079430 msgr2=0x7f886c079840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:16:49.657 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.655+0000 7f8874800700 1 -- 172.21.15.19:0/4072087742 shutdown_connections 2024-03-18T15:16:49.657 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.655+0000 7f8874800700 1 -- 172.21.15.19:0/4072087742 wait complete. 2024-03-18T15:16:49.658 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.656+0000 7f8874800700 1 Processor -- start 2024-03-18T15:16:49.658 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.656+0000 7f8874800700 1 -- start start 2024-03-18T15:16:49.658 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.657+0000 7f8874800700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f886c102580 0x7f886c19ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:16:49.658 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.657+0000 7f8874800700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f886c102f30 0x7f886c19b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:16:49.659 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.657+0000 7f8874800700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f886c107650 0x7f886c19f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:16:49.659 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.657+0000 7f8874800700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f886c112680 con 0x7f886c107650 2024-03-18T15:16:49.659 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.657+0000 7f8874800700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f886c112500 con 0x7f886c102580 2024-03-18T15:16:49.659 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.657+0000 7f8874800700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f886c112800 con 0x7f886c102f30 2024-03-18T15:16:49.659 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.657+0000 7f8871e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f886c102580 0x7f886c19ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:16:49.659 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.657+0000 7f8871e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f886c102580 0x7f886c19ad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:60226/0 (socket says 172.21.15.19:60226) 2024-03-18T15:16:49.659 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.657+0000 7f8871e00700 1 -- 172.21.15.19:0/2171359561 learned_addr learned my addr 172.21.15.19:0/2171359561 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:16:49.659 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.657+0000 7f8871400700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f886c102f30 0x7f886c19b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:16:49.659 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.657+0000 7f8872800700 1 --2- 172.21.15.19:0/2171359561 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f886c107650 0x7f886c19f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:16:49.660 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.658+0000 7f8871400700 1 -- 172.21.15.19:0/2171359561 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f886c102580 msgr2=0x7f886c19ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:49.660 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.658+0000 7f8871400700 1 --2- 172.21.15.19:0/2171359561 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f886c102580 0x7f886c19ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:49.660 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.658+0000 7f8871400700 1 -- 172.21.15.19:0/2171359561 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f886c107650 msgr2=0x7f886c19f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:49.660 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.658+0000 7f8871400700 1 --2- 172.21.15.19:0/2171359561 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f886c107650 0x7f886c19f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:49.660 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.658+0000 7f8871400700 1 -- 172.21.15.19:0/2171359561 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f886c19fd60 con 0x7f886c102f30 2024-03-18T15:16:49.661 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.658+0000 7f8871e00700 1 --2- 172.21.15.19:0/2171359561 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f886c102580 0x7f886c19ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:16:49.661 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.658+0000 7f8871400700 1 --2- 172.21.15.19:0/2171359561 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f886c102f30 0x7f886c19b290 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f885c00e9c0 tx=0x7f885c00ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:16:49.661 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.658+0000 7f8866a00700 1 -- 172.21.15.19:0/2171359561 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f885c00c9c0 con 0x7f886c102f30 2024-03-18T15:16:49.661 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.658+0000 7f8866a00700 1 -- 172.21.15.19:0/2171359561 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f885c00cb60 con 0x7f886c102f30 2024-03-18T15:16:49.661 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.658+0000 7f8866a00700 1 -- 172.21.15.19:0/2171359561 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f885c018680 con 0x7f886c102f30 2024-03-18T15:16:49.661 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.658+0000 7f8874800700 1 -- 172.21.15.19:0/2171359561 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f886c1a0050 con 0x7f886c102f30 2024-03-18T15:16:49.661 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.659+0000 7f8874800700 1 -- 172.21.15.19:0/2171359561 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f886c1a78d0 con 0x7f886c102f30 2024-03-18T15:16:49.661 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.660+0000 7f8874800700 1 -- 172.21.15.19:0/2171359561 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f886c068f90 con 0x7f886c102f30 2024-03-18T15:16:49.663 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.662+0000 7f8866a00700 1 -- 172.21.15.19:0/2171359561 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f885c047a90 con 0x7f886c102f30 2024-03-18T15:16:49.664 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.663+0000 7f8866a00700 1 --2- 172.21.15.19:0/2171359561 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f88540a29a0 0x7f88540a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:16:49.665 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.664+0000 7f8866a00700 1 -- 172.21.15.19:0/2171359561 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f885c0ceff0 con 0x7f886c102f30 2024-03-18T15:16:49.665 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.664+0000 7f8871e00700 1 --2- 172.21.15.19:0/2171359561 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f88540a29a0 0x7f88540a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:16:49.665 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.664+0000 7f8871e00700 1 --2- 172.21.15.19:0/2171359561 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f88540a29a0 0x7f88540a4e60 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f8860009750 tx=0x7f886000b280 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:16:49.666 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.665+0000 7f8866a00700 1 -- 172.21.15.19:0/2171359561 <== mon.1 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f885c09bc60 con 0x7f886c102f30 2024-03-18T15:16:49.898 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.896+0000 7f8874800700 1 -- 172.21.15.19:0/2171359561 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f886c1a7b30 con 0x7f886c102f30 2024-03-18T15:16:49.898 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.897+0000 7f8866a00700 1 -- 172.21.15.19:0/2171359561 <== mon.1 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f885c09b560 con 0x7f886c102f30 2024-03-18T15:16:49.899 INFO:teuthology.orchestra.run.smithi019.stdout:107374182479 2024-03-18T15:16:49.901 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.900+0000 7f8874800700 1 -- 172.21.15.19:0/2171359561 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f88540a29a0 msgr2=0x7f88540a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:49.901 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.900+0000 7f8874800700 1 --2- 172.21.15.19:0/2171359561 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f88540a29a0 0x7f88540a4e60 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f8860009750 tx=0x7f886000b280 comp rx=0 tx=0).stop 2024-03-18T15:16:49.902 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.900+0000 7f8874800700 1 -- 172.21.15.19:0/2171359561 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f886c102f30 msgr2=0x7f886c19b290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:49.902 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.900+0000 7f8874800700 1 --2- 172.21.15.19:0/2171359561 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f886c102f30 0x7f886c19b290 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f885c00e9c0 tx=0x7f885c00ecd0 comp rx=0 tx=0).stop 2024-03-18T15:16:49.902 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.901+0000 7f8874800700 1 -- 172.21.15.19:0/2171359561 shutdown_connections 2024-03-18T15:16:49.902 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.901+0000 7f8874800700 1 --2- 172.21.15.19:0/2171359561 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f88540a29a0 0x7f88540a4e60 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:49.903 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.901+0000 7f8874800700 1 --2- 172.21.15.19:0/2171359561 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f886c107650 0x7f886c19f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:49.903 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.901+0000 7f8874800700 1 --2- 172.21.15.19:0/2171359561 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f886c102580 0x7f886c19ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:49.903 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.901+0000 7f8874800700 1 --2- 172.21.15.19:0/2171359561 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f886c102f30 0x7f886c19b290 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:49.903 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.901+0000 7f8874800700 1 -- 172.21.15.19:0/2171359561 >> 172.21.15.19:0/2171359561 conn(0x7f886c079430 msgr2=0x7f886c101ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:16:49.903 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.901+0000 7f8874800700 1 -- 172.21.15.19:0/2171359561 shutdown_connections 2024-03-18T15:16:49.903 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.901+0000 7f8874800700 1 -- 172.21.15.19:0/2171359561 wait complete. 2024-03-18T15:16:50.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:16:49 smithi104 ceph-mon[36070]: pgmap v368: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-03-18T15:16:50.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:16:49 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/2424828889' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-18T15:16:50.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:16:49 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/2171359561' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-18T15:16:50.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:16:49 smithi019 ceph-mon[31522]: pgmap v368: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-03-18T15:16:50.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:16:49 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/2424828889' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-18T15:16:50.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:16:49 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/2171359561' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-18T15:16:50.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:16:49 smithi033 ceph-mon[36648]: pgmap v368: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-03-18T15:16:50.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:16:49 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/2424828889' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-18T15:16:50.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:16:49 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/2171359561' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-18T15:16:50.759 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.257+0000 7f4fe3400700 1 -- 172.21.15.19:0/4240231759 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4fdc076300 msgr2=0x7f4fdc074500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:50.759 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.257+0000 7f4fe3400700 1 --2- 172.21.15.19:0/4240231759 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4fdc076300 0x7f4fdc074500 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f4fcc00b0a0 tx=0x7f4fcc00b3b0 comp rx=0 tx=0).stop 2024-03-18T15:16:50.759 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.257+0000 7f4fe3400700 1 -- 172.21.15.19:0/4240231759 shutdown_connections 2024-03-18T15:16:50.759 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.257+0000 7f4fe3400700 1 --2- 172.21.15.19:0/4240231759 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4fdc076300 0x7f4fdc074500 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:50.759 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.257+0000 7f4fe3400700 1 --2- 172.21.15.19:0/4240231759 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4fdc112f60 0x7f4fdc115350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:50.759 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.257+0000 7f4fe3400700 1 --2- 172.21.15.19:0/4240231759 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4fdc074a40 0x7f4fdc074ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:50.759 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.257+0000 7f4fe3400700 1 -- 172.21.15.19:0/4240231759 >> 172.21.15.19:0/4240231759 conn(0x7f4fdc079400 msgr2=0x7f4fdc079810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:16:50.759 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.258+0000 7f4fe3400700 1 -- 172.21.15.19:0/4240231759 shutdown_connections 2024-03-18T15:16:50.759 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.258+0000 7f4fe3400700 1 -- 172.21.15.19:0/4240231759 wait complete. 2024-03-18T15:16:50.759 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.259+0000 7f4fe3400700 1 Processor -- start 2024-03-18T15:16:50.759 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.259+0000 7f4fe3400700 1 -- start start 2024-03-18T15:16:50.760 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.259+0000 7f4fe3400700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4fdc074a40 0x7f4fdc19f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:16:50.760 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.259+0000 7f4fe3400700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4fdc076300 0x7f4fdc19f560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:16:50.760 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.259+0000 7f4fe3400700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4fdc112f60 0x7f4fdc1a38f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:16:50.760 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.259+0000 7f4fe3400700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f4fdc118b90 con 0x7f4fdc074a40 2024-03-18T15:16:50.760 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.259+0000 7f4fe3400700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4fdc118a10 con 0x7f4fdc112f60 2024-03-18T15:16:50.760 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.259+0000 7f4fe3400700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f4fdc118d10 con 0x7f4fdc076300 2024-03-18T15:16:50.760 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.259+0000 7f4fd3e00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4fdc076300 0x7f4fdc19f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:16:50.760 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.260+0000 7f4fe0a00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4fdc074a40 0x7f4fdc19f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:16:50.760 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.260+0000 7f4fd3e00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4fdc076300 0x7f4fdc19f560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.19:36796/0 (socket says 172.21.15.19:36796) 2024-03-18T15:16:50.760 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.260+0000 7f4fd3e00700 1 -- 172.21.15.19:0/2424828889 learned_addr learned my addr 172.21.15.19:0/2424828889 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:16:50.761 INFO:teuthology.orchestra.run.smithi019.stdout:163208757315 2024-03-18T15:16:50.761 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.260+0000 7f4fe1400700 1 --2- 172.21.15.19:0/2424828889 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4fdc112f60 0x7f4fdc1a38f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:16:50.761 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.260+0000 7f4fd3e00700 1 -- 172.21.15.19:0/2424828889 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4fdc112f60 msgr2=0x7f4fdc1a38f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:50.761 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.260+0000 7f4fd3e00700 1 --2- 172.21.15.19:0/2424828889 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4fdc112f60 0x7f4fdc1a38f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:50.761 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.260+0000 7f4fd3e00700 1 -- 172.21.15.19:0/2424828889 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4fdc074a40 msgr2=0x7f4fdc19f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:50.761 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.260+0000 7f4fd3e00700 1 --2- 172.21.15.19:0/2424828889 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4fdc074a40 0x7f4fdc19f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:50.761 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.260+0000 7f4fd3e00700 1 -- 172.21.15.19:0/2424828889 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fdc1a4030 con 0x7f4fdc076300 2024-03-18T15:16:50.761 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.260+0000 7f4fd3e00700 1 --2- 172.21.15.19:0/2424828889 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4fdc076300 0x7f4fdc19f560 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f4fc400e9a0 tx=0x7f4fc400ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:16:50.761 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.260+0000 7f4fda000700 1 -- 172.21.15.19:0/2424828889 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fc400ca10 con 0x7f4fdc076300 2024-03-18T15:16:50.762 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.260+0000 7f4fda000700 1 -- 172.21.15.19:0/2424828889 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4fc400cbb0 con 0x7f4fdc076300 2024-03-18T15:16:50.762 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.260+0000 7f4fe3400700 1 -- 172.21.15.19:0/2424828889 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4fdc1a4320 con 0x7f4fdc076300 2024-03-18T15:16:50.762 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.261+0000 7f4fda000700 1 -- 172.21.15.19:0/2424828889 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fc4018680 con 0x7f4fdc076300 2024-03-18T15:16:50.762 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.261+0000 7f4fe3400700 1 -- 172.21.15.19:0/2424828889 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4fdc1abba0 con 0x7f4fdc076300 2024-03-18T15:16:50.762 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.262+0000 7f4fe3400700 1 -- 172.21.15.19:0/2424828889 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4fdc068f90 con 0x7f4fdc076300 2024-03-18T15:16:50.762 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.264+0000 7f4fda000700 1 -- 172.21.15.19:0/2424828889 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f4fc4010430 con 0x7f4fdc076300 2024-03-18T15:16:50.762 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.266+0000 7f4fda000700 1 --2- 172.21.15.19:0/2424828889 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f4fc00a29a0 0x7f4fc00a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:16:50.762 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.266+0000 7f4fda000700 1 -- 172.21.15.19:0/2424828889 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f4fc40cf160 con 0x7f4fdc076300 2024-03-18T15:16:50.762 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.266+0000 7f4fe0a00700 1 --2- 172.21.15.19:0/2424828889 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f4fc00a29a0 0x7f4fc00a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:16:50.762 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.267+0000 7f4fe0a00700 1 --2- 172.21.15.19:0/2424828889 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f4fc00a29a0 0x7f4fc00a4e60 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f4fcc009e60 tx=0x7f4fcc00b700 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:16:50.762 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.274+0000 7f4fda000700 1 -- 172.21.15.19:0/2424828889 <== mon.1 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4fc409bce0 con 0x7f4fdc076300 2024-03-18T15:16:50.762 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.509+0000 7f4fe3400700 1 -- 172.21.15.19:0/2424828889 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f4fdc1abe00 con 0x7f4fdc076300 2024-03-18T15:16:50.763 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.509+0000 7f4fda000700 1 -- 172.21.15.19:0/2424828889 <== mon.1 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f4fc409b5e0 con 0x7f4fdc076300 2024-03-18T15:16:50.763 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.512+0000 7f4fe3400700 1 -- 172.21.15.19:0/2424828889 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f4fc00a29a0 msgr2=0x7f4fc00a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:50.763 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.512+0000 7f4fe3400700 1 --2- 172.21.15.19:0/2424828889 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f4fc00a29a0 0x7f4fc00a4e60 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f4fcc009e60 tx=0x7f4fcc00b700 comp rx=0 tx=0).stop 2024-03-18T15:16:50.763 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.512+0000 7f4fe3400700 1 -- 172.21.15.19:0/2424828889 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4fdc076300 msgr2=0x7f4fdc19f560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:50.763 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.512+0000 7f4fe3400700 1 --2- 172.21.15.19:0/2424828889 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4fdc076300 0x7f4fdc19f560 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f4fc400e9a0 tx=0x7f4fc400ecb0 comp rx=0 tx=0).stop 2024-03-18T15:16:50.763 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.512+0000 7f4fe3400700 1 -- 172.21.15.19:0/2424828889 shutdown_connections 2024-03-18T15:16:50.763 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.512+0000 7f4fe3400700 1 --2- 172.21.15.19:0/2424828889 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f4fc00a29a0 0x7f4fc00a4e60 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:50.763 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.512+0000 7f4fe3400700 1 --2- 172.21.15.19:0/2424828889 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4fdc112f60 0x7f4fdc1a38f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:50.763 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.512+0000 7f4fe3400700 1 --2- 172.21.15.19:0/2424828889 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4fdc074a40 0x7f4fdc19f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:50.763 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.512+0000 7f4fe3400700 1 --2- 172.21.15.19:0/2424828889 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4fdc076300 0x7f4fdc19f560 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:50.763 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.512+0000 7f4fe3400700 1 -- 172.21.15.19:0/2424828889 >> 172.21.15.19:0/2424828889 conn(0x7f4fdc079400 msgr2=0x7f4fdc111cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:16:50.763 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.512+0000 7f4fe3400700 1 -- 172.21.15.19:0/2424828889 shutdown_connections 2024-03-18T15:16:50.763 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:49.512+0000 7f4fe3400700 1 -- 172.21.15.19:0/2424828889 wait complete. 2024-03-18T15:16:51.780 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:51.778+0000 7fa8c4800700 1 -- 172.21.15.19:0/4019234400 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa8bc1075f0 msgr2=0x7fa8bc10f1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:51.780 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:51.778+0000 7fa8c4800700 1 --2- 172.21.15.19:0/4019234400 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa8bc1075f0 0x7fa8bc10f1c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fa8b800b0a0 tx=0x7fa8b800b3b0 comp rx=0 tx=0).stop 2024-03-18T15:16:51.780 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:51.778+0000 7fa8c4800700 1 -- 172.21.15.19:0/4019234400 shutdown_connections 2024-03-18T15:16:51.780 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:51.778+0000 7fa8c4800700 1 --2- 172.21.15.19:0/4019234400 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa8bc1075f0 0x7fa8bc10f1c0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:51.780 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:51.779+0000 7fa8c4800700 1 --2- 172.21.15.19:0/4019234400 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa8bc102510 0x7fa8bc1028f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:51.780 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:51.779+0000 7fa8c4800700 1 --2- 172.21.15.19:0/4019234400 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa8bc102ec0 0x7fa8bc106f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:51.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:51.779+0000 7fa8c4800700 1 -- 172.21.15.19:0/4019234400 >> 172.21.15.19:0/4019234400 conn(0x7fa8bc0fdd80 msgr2=0x7fa8bc1001a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:16:51.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:51.779+0000 7fa8c4800700 1 -- 172.21.15.19:0/4019234400 shutdown_connections 2024-03-18T15:16:51.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:51.779+0000 7fa8c4800700 1 -- 172.21.15.19:0/4019234400 wait complete. 2024-03-18T15:16:51.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:51.780+0000 7fa8c4800700 1 Processor -- start 2024-03-18T15:16:51.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:51.781+0000 7fa8c4800700 1 -- start start 2024-03-18T15:16:51.782 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:51.781+0000 7fa8c4800700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa8bc102510 0x7fa8bc1a0880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:16:51.782 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:51.781+0000 7fa8c4800700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa8bc102ec0 0x7fa8bc1a0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:16:51.782 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:51.781+0000 7fa8c4800700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa8bc1075f0 0x7fa8bc19aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:16:51.782 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:51.781+0000 7fa8c4800700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fa8bc112630 con 0x7fa8bc102510 2024-03-18T15:16:51.782 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:51.781+0000 7fa8c4800700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa8bc1124b0 con 0x7fa8bc102ec0 2024-03-18T15:16:51.782 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:51.781+0000 7fa8c4800700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fa8bc1127b0 con 0x7fa8bc1075f0 2024-03-18T15:16:51.782 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:51.781+0000 7fa8c1400700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa8bc102ec0 0x7fa8bc1a0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:16:51.783 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:51.781+0000 7fa8c2800700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa8bc1075f0 0x7fa8bc19aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:16:51.783 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:51.781+0000 7fa8c1400700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa8bc102ec0 0x7fa8bc1a0dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:60272/0 (socket says 172.21.15.19:60272) 2024-03-18T15:16:51.783 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:51.781+0000 7fa8c1400700 1 -- 172.21.15.19:0/1074007697 learned_addr learned my addr 172.21.15.19:0/1074007697 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:16:51.784 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:51.781+0000 7fa8c1e00700 1 --2- 172.21.15.19:0/1074007697 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa8bc102510 0x7fa8bc1a0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:16:51.784 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:51.782+0000 7fa8c1400700 1 -- 172.21.15.19:0/1074007697 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa8bc1075f0 msgr2=0x7fa8bc19aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:51.784 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:51.782+0000 7fa8c1400700 1 --2- 172.21.15.19:0/1074007697 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa8bc1075f0 0x7fa8bc19aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:51.784 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:51.782+0000 7fa8c1400700 1 -- 172.21.15.19:0/1074007697 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa8bc102510 msgr2=0x7fa8bc1a0880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:51.784 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:51.782+0000 7fa8c1400700 1 --2- 172.21.15.19:0/1074007697 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa8bc102510 0x7fa8bc1a0880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:51.784 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:51.782+0000 7fa8c1400700 1 -- 172.21.15.19:0/1074007697 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8bc19b1f0 con 0x7fa8bc102ec0 2024-03-18T15:16:51.784 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:51.782+0000 7fa8c1e00700 1 --2- 172.21.15.19:0/1074007697 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa8bc102510 0x7fa8bc1a0880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T15:16:51.784 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:51.782+0000 7fa8c2800700 1 --2- 172.21.15.19:0/1074007697 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa8bc1075f0 0x7fa8bc19aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:16:51.784 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:51.782+0000 7fa8c1400700 1 --2- 172.21.15.19:0/1074007697 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa8bc102ec0 0x7fa8bc1a0dc0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fa8ac00ea30 tx=0x7fa8ac00ed40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:16:51.784 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:51.782+0000 7fa8b6a00700 1 -- 172.21.15.19:0/1074007697 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8ac00cbd0 con 0x7fa8bc102ec0 2024-03-18T15:16:51.784 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:51.782+0000 7fa8b6a00700 1 -- 172.21.15.19:0/1074007697 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa8ac004500 con 0x7fa8bc102ec0 2024-03-18T15:16:51.784 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:51.782+0000 7fa8b6a00700 1 -- 172.21.15.19:0/1074007697 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8ac010430 con 0x7fa8bc102ec0 2024-03-18T15:16:51.785 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:51.783+0000 7fa8c4800700 1 -- 172.21.15.19:0/1074007697 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8bc19b4e0 con 0x7fa8bc102ec0 2024-03-18T15:16:51.785 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:51.783+0000 7fa8c4800700 1 -- 172.21.15.19:0/1074007697 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8bc1a7750 con 0x7fa8bc102ec0 2024-03-18T15:16:51.785 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:51.784+0000 7fa8c4800700 1 -- 172.21.15.19:0/1074007697 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8bc1039f0 con 0x7fa8bc102ec0 2024-03-18T15:16:51.788 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:51.787+0000 7fa8b6a00700 1 -- 172.21.15.19:0/1074007697 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fa8ac003f50 con 0x7fa8bc102ec0 2024-03-18T15:16:51.790 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:51.789+0000 7fa8b6a00700 1 --2- 172.21.15.19:0/1074007697 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fa8a40a2920 0x7fa8a40a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:16:51.790 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:51.789+0000 7fa8b6a00700 1 -- 172.21.15.19:0/1074007697 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fa8ac0cf1b0 con 0x7fa8bc102ec0 2024-03-18T15:16:51.790 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:51.789+0000 7fa8c1e00700 1 --2- 172.21.15.19:0/1074007697 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fa8a40a2920 0x7fa8a40a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:16:51.790 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:51.790+0000 7fa8c1e00700 1 --2- 172.21.15.19:0/1074007697 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fa8a40a2920 0x7fa8a40a4de0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fa8bc108eb0 tx=0x7fa8b0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:16:51.798 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:51.797+0000 7fa8b6a00700 1 -- 172.21.15.19:0/1074007697 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa8ac09bd30 con 0x7fa8bc102ec0 2024-03-18T15:16:51.849 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:51.689+0000 7f35ace00700 1 -- 172.21.15.19:0/3880399850 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f35a8102f30 msgr2=0x7f35a8106f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:51.849 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:51.689+0000 7f35ace00700 1 --2- 172.21.15.19:0/3880399850 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f35a8102f30 0x7f35a8106f50 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f35940098f0 tx=0x7f3594009c00 comp rx=0 tx=0).stop 2024-03-18T15:16:51.849 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:51.689+0000 7f35ace00700 1 -- 172.21.15.19:0/3880399850 shutdown_connections 2024-03-18T15:16:51.849 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:51.689+0000 7f35ace00700 1 --2- 172.21.15.19:0/3880399850 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f35a8107630 0x7f35a810f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:51.849 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:51.689+0000 7f35ace00700 1 --2- 172.21.15.19:0/3880399850 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f35a8102580 0x7f35a8102960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:51.849 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:51.689+0000 7f35ace00700 1 --2- 172.21.15.19:0/3880399850 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f35a8102f30 0x7f35a8106f50 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:51.849 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:51.690+0000 7f35ace00700 1 -- 172.21.15.19:0/3880399850 >> 172.21.15.19:0/3880399850 conn(0x7f35a80794c0 msgr2=0x7f35a80798d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:16:51.849 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:51.690+0000 7f35ace00700 1 -- 172.21.15.19:0/3880399850 shutdown_connections 2024-03-18T15:16:51.849 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:51.690+0000 7f35ace00700 1 -- 172.21.15.19:0/3880399850 wait complete. 2024-03-18T15:16:51.849 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:51.691+0000 7f35ace00700 1 Processor -- start 2024-03-18T15:16:51.849 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:51.691+0000 7f35ace00700 1 -- start start 2024-03-18T15:16:51.850 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:51.692+0000 7f35ace00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f35a8102580 0x7f35a819e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:16:51.850 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:51.692+0000 7f35ace00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f35a8102f30 0x7f35a819ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:16:51.850 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:51.692+0000 7f35ace00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f35a8107630 0x7f35a8198940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:16:51.850 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:51.692+0000 7f35ace00700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f35a8112620 con 0x7f35a8102f30 2024-03-18T15:16:51.850 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:51.692+0000 7f35ace00700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f35a81124a0 con 0x7f35a8107630 2024-03-18T15:16:51.850 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:51.692+0000 7f35ace00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f35a81127a0 con 0x7f35a8102580 2024-03-18T15:16:51.850 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:51.692+0000 7f35a6800700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f35a8107630 0x7f35a8198940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:16:51.850 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:51.692+0000 7f35a5e00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f35a8102580 0x7f35a819e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:16:51.850 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:51.692+0000 7f35a6800700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f35a8107630 0x7f35a8198940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:60242/0 (socket says 172.21.15.19:60242) 2024-03-18T15:16:51.850 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:51.692+0000 7f35a5e00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f35a8102580 0x7f35a819e780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.19:36826/0 (socket says 172.21.15.19:36826) 2024-03-18T15:16:51.850 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:51.692+0000 7f35a6800700 1 -- 172.21.15.19:0/3517562421 learned_addr learned my addr 172.21.15.19:0/3517562421 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:16:51.850 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:51.692+0000 7f35a5400700 1 --2- 172.21.15.19:0/3517562421 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f35a8102f30 0x7f35a819ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:16:51.850 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:51.692+0000 7f35a6800700 1 -- 172.21.15.19:0/3517562421 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f35a8102580 msgr2=0x7f35a819e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:51.851 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:51.692+0000 7f35a6800700 1 --2- 172.21.15.19:0/3517562421 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f35a8102580 0x7f35a819e780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:51.851 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:51.692+0000 7f35a6800700 1 -- 172.21.15.19:0/3517562421 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f35a8102f30 msgr2=0x7f35a819ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:51.851 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:51.692+0000 7f35a6800700 1 --2- 172.21.15.19:0/3517562421 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f35a8102f30 0x7f35a819ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:51.851 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:51.692+0000 7f35a6800700 1 -- 172.21.15.19:0/3517562421 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35a81990a0 con 0x7f35a8107630 2024-03-18T15:16:51.851 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:51.693+0000 7f35a5e00700 1 --2- 172.21.15.19:0/3517562421 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f35a8102580 0x7f35a819e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:16:51.851 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:51.693+0000 7f35a6800700 1 --2- 172.21.15.19:0/3517562421 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f35a8107630 0x7f35a8198940 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f35a000b470 tx=0x7f35a000b780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:16:51.851 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:51.693+0000 7f359ea00700 1 -- 172.21.15.19:0/3517562421 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35a0004bb0 con 0x7f35a8107630 2024-03-18T15:16:51.851 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:51.693+0000 7f359ea00700 1 -- 172.21.15.19:0/3517562421 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f35a0004d50 con 0x7f35a8107630 2024-03-18T15:16:51.851 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:51.693+0000 7f359ea00700 1 -- 172.21.15.19:0/3517562421 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35a001a470 con 0x7f35a8107630 2024-03-18T15:16:51.851 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:51.693+0000 7f35ace00700 1 -- 172.21.15.19:0/3517562421 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35a8199390 con 0x7f35a8107630 2024-03-18T15:16:51.851 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:51.693+0000 7f35ace00700 1 -- 172.21.15.19:0/3517562421 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35a8074500 con 0x7f35a8107630 2024-03-18T15:16:51.851 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:51.694+0000 7f35ace00700 1 -- 172.21.15.19:0/3517562421 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35a8068f90 con 0x7f35a8107630 2024-03-18T15:16:51.851 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:51.696+0000 7f359ea00700 1 -- 172.21.15.19:0/3517562421 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f35a00076e0 con 0x7f35a8107630 2024-03-18T15:16:51.851 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:51.697+0000 7f359ea00700 1 --2- 172.21.15.19:0/3517562421 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f358c0a29a0 0x7f358c0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:16:51.852 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:51.698+0000 7f359ea00700 1 -- 172.21.15.19:0/3517562421 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f35a00d2970 con 0x7f35a8107630 2024-03-18T15:16:51.852 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:51.698+0000 7f35a5e00700 1 --2- 172.21.15.19:0/3517562421 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f358c0a29a0 0x7f358c0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:16:51.852 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:51.699+0000 7f35a5e00700 1 --2- 172.21.15.19:0/3517562421 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f358c0a29a0 0x7f358c0a4e60 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f35a8199a80 tx=0x7f3598009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:16:51.852 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:51.702+0000 7f359ea00700 1 -- 172.21.15.19:0/3517562421 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f35a009f4f0 con 0x7f35a8107630 2024-03-18T15:16:51.939 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:51.937+0000 7f35ace00700 1 -- 172.21.15.19:0/3517562421 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f35a802db70 con 0x7f35a8107630 2024-03-18T15:16:51.939 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:51.937+0000 7f359ea00700 1 -- 172.21.15.19:0/3517562421 <== mon.2 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f35a009edf0 con 0x7f35a8107630 2024-03-18T15:16:51.939 INFO:teuthology.orchestra.run.smithi019.stdout:77309411413 2024-03-18T15:16:51.941 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:51.940+0000 7f35ace00700 1 -- 172.21.15.19:0/3517562421 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f358c0a29a0 msgr2=0x7f358c0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:51.941 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:51.940+0000 7f35ace00700 1 --2- 172.21.15.19:0/3517562421 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f358c0a29a0 0x7f358c0a4e60 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f35a8199a80 tx=0x7f3598009260 comp rx=0 tx=0).stop 2024-03-18T15:16:51.942 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:51.940+0000 7f35ace00700 1 -- 172.21.15.19:0/3517562421 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f35a8107630 msgr2=0x7f35a8198940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:51.942 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:51.940+0000 7f35ace00700 1 --2- 172.21.15.19:0/3517562421 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f35a8107630 0x7f35a8198940 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f35a000b470 tx=0x7f35a000b780 comp rx=0 tx=0).stop 2024-03-18T15:16:51.942 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:51.940+0000 7f35ace00700 1 -- 172.21.15.19:0/3517562421 shutdown_connections 2024-03-18T15:16:51.942 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:51.940+0000 7f35ace00700 1 --2- 172.21.15.19:0/3517562421 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f358c0a29a0 0x7f358c0a4e60 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:51.942 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:51.940+0000 7f35ace00700 1 --2- 172.21.15.19:0/3517562421 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f35a8102580 0x7f35a819e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:51.942 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:51.940+0000 7f35ace00700 1 --2- 172.21.15.19:0/3517562421 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f35a8107630 0x7f35a8198940 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:51.942 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:51.940+0000 7f35ace00700 1 --2- 172.21.15.19:0/3517562421 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f35a8102f30 0x7f35a819ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:51.942 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:51.940+0000 7f35ace00700 1 -- 172.21.15.19:0/3517562421 >> 172.21.15.19:0/3517562421 conn(0x7f35a80794c0 msgr2=0x7f35a8101af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:16:51.943 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:51.941+0000 7f35ace00700 1 -- 172.21.15.19:0/3517562421 shutdown_connections 2024-03-18T15:16:51.943 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:51.941+0000 7f35ace00700 1 -- 172.21.15.19:0/3517562421 wait complete. 2024-03-18T15:16:52.050 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:52.048+0000 7fa8c4800700 1 -- 172.21.15.19:0/1074007697 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7fa8bc068f90 con 0x7fa8bc102ec0 2024-03-18T15:16:52.050 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:52.048+0000 7fa8b6a00700 1 -- 172.21.15.19:0/1074007697 <== mon.2 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7fa8ac09b630 con 0x7fa8bc102ec0 2024-03-18T15:16:52.050 INFO:teuthology.orchestra.run.smithi019.stdout:343597383714 2024-03-18T15:16:52.052 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:52.051+0000 7fa8c4800700 1 -- 172.21.15.19:0/1074007697 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fa8a40a2920 msgr2=0x7fa8a40a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:52.052 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:52.051+0000 7fa8c4800700 1 --2- 172.21.15.19:0/1074007697 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fa8a40a2920 0x7fa8a40a4de0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fa8bc108eb0 tx=0x7fa8b0009260 comp rx=0 tx=0).stop 2024-03-18T15:16:52.053 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:52.051+0000 7fa8c4800700 1 -- 172.21.15.19:0/1074007697 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa8bc102ec0 msgr2=0x7fa8bc1a0dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:52.053 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:52.051+0000 7fa8c4800700 1 --2- 172.21.15.19:0/1074007697 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa8bc102ec0 0x7fa8bc1a0dc0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fa8ac00ea30 tx=0x7fa8ac00ed40 comp rx=0 tx=0).stop 2024-03-18T15:16:52.053 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:52.051+0000 7fa8c4800700 1 -- 172.21.15.19:0/1074007697 shutdown_connections 2024-03-18T15:16:52.053 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:52.051+0000 7fa8c4800700 1 --2- 172.21.15.19:0/1074007697 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fa8a40a2920 0x7fa8a40a4de0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:52.053 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:52.051+0000 7fa8c4800700 1 --2- 172.21.15.19:0/1074007697 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa8bc1075f0 0x7fa8bc19aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:52.053 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:52.051+0000 7fa8c4800700 1 --2- 172.21.15.19:0/1074007697 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa8bc102510 0x7fa8bc1a0880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:52.053 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:52.051+0000 7fa8c4800700 1 --2- 172.21.15.19:0/1074007697 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa8bc102ec0 0x7fa8bc1a0dc0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:52.053 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:52.051+0000 7fa8c4800700 1 -- 172.21.15.19:0/1074007697 >> 172.21.15.19:0/1074007697 conn(0x7fa8bc0fdd80 msgr2=0x7fa8bc10d650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:16:52.053 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:52.051+0000 7fa8c4800700 1 -- 172.21.15.19:0/1074007697 shutdown_connections 2024-03-18T15:16:52.053 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:52.052+0000 7fa8c4800700 1 -- 172.21.15.19:0/1074007697 wait complete. 2024-03-18T15:16:52.186 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:16:51 smithi019 ceph-mon[31522]: pgmap v369: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T15:16:52.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:16:51 smithi104 ceph-mon[36070]: pgmap v369: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T15:16:52.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:16:51 smithi033 ceph-mon[36648]: pgmap v369: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T15:16:53.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:16:52 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/3517562421' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-18T15:16:53.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:16:52 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/1074007697' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-18T15:16:53.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:16:52 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/3517562421' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-18T15:16:53.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:16:52 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/1074007697' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-18T15:16:53.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:16:52 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/3517562421' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-18T15:16:53.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:16:52 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/1074007697' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-18T15:16:54.057 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:16:53 smithi019 ceph-mon[31522]: pgmap v370: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-18T15:16:54.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:16:53 smithi033 ceph-mon[36648]: pgmap v370: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-18T15:16:54.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:16:53 smithi104 ceph-mon[36070]: pgmap v370: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-18T15:16:55.104 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182474 got 107374182479 for osd.3 2024-03-18T15:16:55.105 DEBUG:teuthology.parallel:result is None 2024-03-18T15:16:55.786 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:55.784+0000 7fea86a00700 1 -- 172.21.15.19:0/2409151550 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fea80069f80 msgr2=0x7fea8006a360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:55.786 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:55.784+0000 7fea86a00700 1 --2- 172.21.15.19:0/2409151550 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fea80069f80 0x7fea8006a360 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fea70009990 tx=0x7fea70009ca0 comp rx=0 tx=0).stop 2024-03-18T15:16:55.786 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:55.784+0000 7fea86a00700 1 -- 172.21.15.19:0/2409151550 shutdown_connections 2024-03-18T15:16:55.786 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:55.784+0000 7fea86a00700 1 --2- 172.21.15.19:0/2409151550 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fea8010ed80 0x7fea80111170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:55.786 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:55.784+0000 7fea86a00700 1 --2- 172.21.15.19:0/2409151550 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fea80069f80 0x7fea8006a360 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:55.786 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:55.784+0000 7fea86a00700 1 --2- 172.21.15.19:0/2409151550 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fea8006a8a0 0x7fea8010e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:55.786 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:55.784+0000 7fea86a00700 1 -- 172.21.15.19:0/2409151550 >> 172.21.15.19:0/2409151550 conn(0x7fea80078100 msgr2=0x7fea80078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:16:55.787 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:55.784+0000 7fea86a00700 1 -- 172.21.15.19:0/2409151550 shutdown_connections 2024-03-18T15:16:55.787 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:55.785+0000 7fea86a00700 1 -- 172.21.15.19:0/2409151550 wait complete. 2024-03-18T15:16:55.788 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:55.786+0000 7fea86a00700 1 Processor -- start 2024-03-18T15:16:55.788 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:55.786+0000 7fea86a00700 1 -- start start 2024-03-18T15:16:55.788 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:55.786+0000 7fea86a00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fea80069f80 0x7fea801a0a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:16:55.788 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:55.786+0000 7fea86a00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fea8006a8a0 0x7fea801a0f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:16:55.788 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:55.786+0000 7fea86a00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fea8010ed80 0x7fea8019ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:16:55.789 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:55.786+0000 7fea86a00700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fea801147e0 con 0x7fea8006a8a0 2024-03-18T15:16:55.789 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:55.786+0000 7fea86a00700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fea80114660 con 0x7fea80069f80 2024-03-18T15:16:55.789 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:55.786+0000 7fea86a00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fea80114960 con 0x7fea8010ed80 2024-03-18T15:16:55.789 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:55.786+0000 7fea7fe00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fea80069f80 0x7fea801a0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:16:55.789 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:55.786+0000 7fea7fe00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fea80069f80 0x7fea801a0a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:50186/0 (socket says 172.21.15.19:50186) 2024-03-18T15:16:55.789 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:55.786+0000 7fea7fe00700 1 -- 172.21.15.19:0/2430935198 learned_addr learned my addr 172.21.15.19:0/2430935198 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:16:55.789 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:55.787+0000 7fea84a00700 1 --2- 172.21.15.19:0/2430935198 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fea8010ed80 0x7fea8019ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:16:55.789 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:55.787+0000 7fea7f400700 1 --2- 172.21.15.19:0/2430935198 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fea8006a8a0 0x7fea801a0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:16:55.789 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:55.787+0000 7fea7fe00700 1 -- 172.21.15.19:0/2430935198 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fea8010ed80 msgr2=0x7fea8019ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:55.789 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:55.787+0000 7fea7fe00700 1 --2- 172.21.15.19:0/2430935198 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fea8010ed80 0x7fea8019ab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:55.789 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:55.787+0000 7fea7fe00700 1 -- 172.21.15.19:0/2430935198 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fea8006a8a0 msgr2=0x7fea801a0f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:55.789 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:55.787+0000 7fea7fe00700 1 --2- 172.21.15.19:0/2430935198 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fea8006a8a0 0x7fea801a0f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:55.789 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:55.787+0000 7fea7fe00700 1 -- 172.21.15.19:0/2430935198 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea8019b440 con 0x7fea80069f80 2024-03-18T15:16:55.790 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:55.787+0000 7fea84a00700 1 --2- 172.21.15.19:0/2430935198 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fea8010ed80 0x7fea8019ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:16:55.790 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:55.787+0000 7fea7f400700 1 --2- 172.21.15.19:0/2430935198 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fea8006a8a0 0x7fea801a0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T15:16:55.790 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:55.787+0000 7fea7fe00700 1 --2- 172.21.15.19:0/2430935198 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fea80069f80 0x7fea801a0a40 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fea70005910 tx=0x7fea7000bab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:16:55.790 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:55.787+0000 7fea7cc00700 1 -- 172.21.15.19:0/2430935198 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea70027070 con 0x7fea80069f80 2024-03-18T15:16:55.791 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:55.788+0000 7fea7cc00700 1 -- 172.21.15.19:0/2430935198 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea70003d70 con 0x7fea80069f80 2024-03-18T15:16:55.791 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:55.788+0000 7fea86a00700 1 -- 172.21.15.19:0/2430935198 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea8019b6d0 con 0x7fea80069f80 2024-03-18T15:16:55.791 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:55.788+0000 7fea7cc00700 1 -- 172.21.15.19:0/2430935198 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea7000f460 con 0x7fea80069f80 2024-03-18T15:16:55.791 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:55.788+0000 7fea86a00700 1 -- 172.21.15.19:0/2430935198 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea801a77f0 con 0x7fea80069f80 2024-03-18T15:16:55.791 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:55.789+0000 7fea86a00700 1 -- 172.21.15.19:0/2430935198 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea801017b0 con 0x7fea80069f80 2024-03-18T15:16:55.792 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:55.791+0000 7fea7cc00700 1 -- 172.21.15.19:0/2430935198 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fea7002a070 con 0x7fea80069f80 2024-03-18T15:16:55.793 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:55.792+0000 7fea7cc00700 1 --2- 172.21.15.19:0/2430935198 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fea640a2920 0x7fea640a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:16:55.794 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:55.793+0000 7fea7cc00700 1 -- 172.21.15.19:0/2430935198 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fea700d0210 con 0x7fea80069f80 2024-03-18T15:16:55.794 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:55.793+0000 7fea7f400700 1 --2- 172.21.15.19:0/2430935198 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fea640a2920 0x7fea640a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:16:55.794 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:55.793+0000 7fea7f400700 1 --2- 172.21.15.19:0/2430935198 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fea640a2920 0x7fea640a4de0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fea8019c260 tx=0x7fea6c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:16:55.800 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:55.799+0000 7fea7cc00700 1 -- 172.21.15.19:0/2430935198 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fea70013070 con 0x7fea80069f80 2024-03-18T15:16:56.041 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:56.039+0000 7fea86a00700 1 -- 172.21.15.19:0/2430935198 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7fea8002db70 con 0x7fea80069f80 2024-03-18T15:16:56.041 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:56.040+0000 7fea7cc00700 1 -- 172.21.15.19:0/2430935198 <== mon.2 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fea7009ce20 con 0x7fea80069f80 2024-03-18T15:16:56.041 INFO:teuthology.orchestra.run.smithi019.stdout:253403070516 2024-03-18T15:16:56.044 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:56.043+0000 7fea86a00700 1 -- 172.21.15.19:0/2430935198 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fea640a2920 msgr2=0x7fea640a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:56.044 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:56.043+0000 7fea86a00700 1 --2- 172.21.15.19:0/2430935198 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fea640a2920 0x7fea640a4de0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fea8019c260 tx=0x7fea6c009260 comp rx=0 tx=0).stop 2024-03-18T15:16:56.044 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:56.043+0000 7fea86a00700 1 -- 172.21.15.19:0/2430935198 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fea80069f80 msgr2=0x7fea801a0a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:56.045 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:56.043+0000 7fea86a00700 1 --2- 172.21.15.19:0/2430935198 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fea80069f80 0x7fea801a0a40 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fea70005910 tx=0x7fea7000bab0 comp rx=0 tx=0).stop 2024-03-18T15:16:56.045 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:56.043+0000 7fea86a00700 1 -- 172.21.15.19:0/2430935198 shutdown_connections 2024-03-18T15:16:56.045 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:56.043+0000 7fea86a00700 1 --2- 172.21.15.19:0/2430935198 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fea640a2920 0x7fea640a4de0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:56.045 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:56.043+0000 7fea86a00700 1 --2- 172.21.15.19:0/2430935198 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fea8010ed80 0x7fea8019ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:56.045 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:56.043+0000 7fea86a00700 1 --2- 172.21.15.19:0/2430935198 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fea80069f80 0x7fea801a0a40 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:56.045 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:56.043+0000 7fea86a00700 1 --2- 172.21.15.19:0/2430935198 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fea8006a8a0 0x7fea801a0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:56.045 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:56.043+0000 7fea86a00700 1 -- 172.21.15.19:0/2430935198 >> 172.21.15.19:0/2430935198 conn(0x7fea80078100 msgr2=0x7fea80110250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:16:56.045 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:56.043+0000 7fea86a00700 1 -- 172.21.15.19:0/2430935198 shutdown_connections 2024-03-18T15:16:56.045 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:56.043+0000 7fea86a00700 1 -- 172.21.15.19:0/2430935198 wait complete. 2024-03-18T15:16:56.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:16:55 smithi019 ceph-mon[31522]: pgmap v371: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T15:16:56.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:16:55 smithi033 ceph-mon[36648]: pgmap v371: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T15:16:56.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:16:55 smithi104 ceph-mon[36070]: pgmap v371: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T15:16:56.724 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:56.250+0000 7f3a9c000700 1 -- 172.21.15.19:0/2643271559 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3a94102320 msgr2=0x7f3a941027a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:56.724 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:56.250+0000 7f3a9c000700 1 --2- 172.21.15.19:0/2643271559 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3a94102320 0x7f3a941027a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f3a80009990 tx=0x7f3a80009ca0 comp rx=0 tx=0).stop 2024-03-18T15:16:56.724 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:56.250+0000 7f3a9c000700 1 -- 172.21.15.19:0/2643271559 shutdown_connections 2024-03-18T15:16:56.724 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:56.250+0000 7f3a9c000700 1 --2- 172.21.15.19:0/2643271559 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3a94102ce0 0x7f3a9410f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:56.724 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:56.250+0000 7f3a9c000700 1 --2- 172.21.15.19:0/2643271559 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3a94108300 0x7f3a941086e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:56.724 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:56.250+0000 7f3a9c000700 1 --2- 172.21.15.19:0/2643271559 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3a94102320 0x7f3a941027a0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:56.724 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:56.250+0000 7f3a9c000700 1 -- 172.21.15.19:0/2643271559 >> 172.21.15.19:0/2643271559 conn(0x7f3a94079400 msgr2=0x7f3a94079810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:16:56.725 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:56.250+0000 7f3a9c000700 1 -- 172.21.15.19:0/2643271559 shutdown_connections 2024-03-18T15:16:56.725 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:56.250+0000 7f3a9c000700 1 -- 172.21.15.19:0/2643271559 wait complete. 2024-03-18T15:16:56.725 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:56.251+0000 7f3a9c000700 1 Processor -- start 2024-03-18T15:16:56.725 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:56.251+0000 7f3a9c000700 1 -- start start 2024-03-18T15:16:56.725 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:56.252+0000 7f3a9c000700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3a94102320 0x7f3a9419e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:16:56.725 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:56.252+0000 7f3a9c000700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3a94102ce0 0x7f3a9419ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:16:56.725 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:56.252+0000 7f3a9c000700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3a94108300 0x7f3a94198850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:16:56.725 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:56.252+0000 7f3a9c000700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f3a941125f0 con 0x7f3a94102ce0 2024-03-18T15:16:56.725 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:56.252+0000 7f3a9c000700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f3a94112470 con 0x7f3a94102320 2024-03-18T15:16:56.725 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:56.252+0000 7f3a9c000700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f3a94112770 con 0x7f3a94108300 2024-03-18T15:16:56.725 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:56.252+0000 7f3a99600700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3a94102320 0x7f3a9419e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:16:56.725 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:56.252+0000 7f3a99600700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3a94102320 0x7f3a9419e6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:50206/0 (socket says 172.21.15.19:50206) 2024-03-18T15:16:56.726 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:56.252+0000 7f3a98c00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3a94102ce0 0x7f3a9419ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:16:56.726 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:56.252+0000 7f3a99600700 1 -- 172.21.15.19:0/47210795 learned_addr learned my addr 172.21.15.19:0/47210795 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:16:56.726 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:56.252+0000 7f3a9a000700 1 --2- 172.21.15.19:0/47210795 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3a94108300 0x7f3a94198850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:16:56.726 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:56.253+0000 7f3a99600700 1 -- 172.21.15.19:0/47210795 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3a94108300 msgr2=0x7f3a94198850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:56.726 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:56.253+0000 7f3a99600700 1 --2- 172.21.15.19:0/47210795 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3a94108300 0x7f3a94198850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:56.726 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:56.253+0000 7f3a99600700 1 -- 172.21.15.19:0/47210795 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3a94102ce0 msgr2=0x7f3a9419ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:56.726 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:56.253+0000 7f3a99600700 1 --2- 172.21.15.19:0/47210795 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3a94102ce0 0x7f3a9419ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:56.726 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:56.253+0000 7f3a99600700 1 -- 172.21.15.19:0/47210795 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a94199030 con 0x7f3a94102320 2024-03-18T15:16:56.726 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:56.253+0000 7f3a98c00700 1 --2- 172.21.15.19:0/47210795 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3a94102ce0 0x7f3a9419ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:16:56.726 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:56.253+0000 7f3a9a000700 1 --2- 172.21.15.19:0/47210795 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3a94108300 0x7f3a94198850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:16:56.726 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:56.253+0000 7f3a99600700 1 --2- 172.21.15.19:0/47210795 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3a94102320 0x7f3a9419e6d0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f3a8800ef30 tx=0x7f3a8800c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:16:56.726 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:56.253+0000 7f3a92000700 1 -- 172.21.15.19:0/47210795 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a880092e0 con 0x7f3a94102320 2024-03-18T15:16:56.726 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:56.253+0000 7f3a92000700 1 -- 172.21.15.19:0/47210795 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a88004bb0 con 0x7f3a94102320 2024-03-18T15:16:56.727 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:56.253+0000 7f3a92000700 1 -- 172.21.15.19:0/47210795 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a88004280 con 0x7f3a94102320 2024-03-18T15:16:56.727 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:56.253+0000 7f3a9c000700 1 -- 172.21.15.19:0/47210795 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a941992c0 con 0x7f3a94102320 2024-03-18T15:16:56.727 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:56.254+0000 7f3a9c000700 1 -- 172.21.15.19:0/47210795 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a941a5530 con 0x7f3a94102320 2024-03-18T15:16:56.727 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:56.255+0000 7f3a9c000700 1 -- 172.21.15.19:0/47210795 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a941039b0 con 0x7f3a94102320 2024-03-18T15:16:56.727 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:56.257+0000 7f3a92000700 1 -- 172.21.15.19:0/47210795 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f3a8801b030 con 0x7f3a94102320 2024-03-18T15:16:56.727 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:56.258+0000 7f3a92000700 1 --2- 172.21.15.19:0/47210795 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f3a7c0a29a0 0x7f3a7c0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:16:56.727 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:56.258+0000 7f3a92000700 1 -- 172.21.15.19:0/47210795 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f3a880cebf0 con 0x7f3a94102320 2024-03-18T15:16:56.727 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:56.259+0000 7f3a98c00700 1 --2- 172.21.15.19:0/47210795 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f3a7c0a29a0 0x7f3a7c0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:16:56.727 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:56.259+0000 7f3a98c00700 1 --2- 172.21.15.19:0/47210795 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f3a7c0a29a0 0x7f3a7c0a4e60 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f3a940799e0 tx=0x7f3a8000b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:16:56.727 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:56.261+0000 7f3a92000700 1 -- 172.21.15.19:0/47210795 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3a88022020 con 0x7f3a94102320 2024-03-18T15:16:56.727 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:56.490+0000 7f3a9c000700 1 -- 172.21.15.19:0/47210795 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f3a9402db70 con 0x7f3a94102320 2024-03-18T15:16:56.727 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:56.490+0000 7f3a92000700 1 -- 172.21.15.19:0/47210795 <== mon.2 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f3a88010040 con 0x7f3a94102320 2024-03-18T15:16:56.728 INFO:teuthology.orchestra.run.smithi019.stdout:219043332152 2024-03-18T15:16:56.728 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:56.493+0000 7f3a9c000700 1 -- 172.21.15.19:0/47210795 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f3a7c0a29a0 msgr2=0x7f3a7c0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:56.728 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:56.493+0000 7f3a9c000700 1 --2- 172.21.15.19:0/47210795 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f3a7c0a29a0 0x7f3a7c0a4e60 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f3a940799e0 tx=0x7f3a8000b260 comp rx=0 tx=0).stop 2024-03-18T15:16:56.728 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:56.493+0000 7f3a9c000700 1 -- 172.21.15.19:0/47210795 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3a94102320 msgr2=0x7f3a9419e6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:56.728 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:56.493+0000 7f3a9c000700 1 --2- 172.21.15.19:0/47210795 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3a94102320 0x7f3a9419e6d0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f3a8800ef30 tx=0x7f3a8800c3a0 comp rx=0 tx=0).stop 2024-03-18T15:16:56.728 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:56.494+0000 7f3a9c000700 1 -- 172.21.15.19:0/47210795 shutdown_connections 2024-03-18T15:16:56.728 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:56.494+0000 7f3a9c000700 1 --2- 172.21.15.19:0/47210795 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f3a7c0a29a0 0x7f3a7c0a4e60 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:56.728 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:56.494+0000 7f3a9c000700 1 --2- 172.21.15.19:0/47210795 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3a94108300 0x7f3a94198850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:56.729 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:56.494+0000 7f3a9c000700 1 --2- 172.21.15.19:0/47210795 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3a94102320 0x7f3a9419e6d0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:56.729 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:56.494+0000 7f3a9c000700 1 --2- 172.21.15.19:0/47210795 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3a94102ce0 0x7f3a9419ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:56.729 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:56.494+0000 7f3a9c000700 1 -- 172.21.15.19:0/47210795 >> 172.21.15.19:0/47210795 conn(0x7f3a94079400 msgr2=0x7f3a94100f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:16:56.729 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:56.494+0000 7f3a9c000700 1 -- 172.21.15.19:0/47210795 shutdown_connections 2024-03-18T15:16:56.729 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:56.494+0000 7f3a9c000700 1 -- 172.21.15.19:0/47210795 wait complete. 2024-03-18T15:16:56.951 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:16:56 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/2430935198' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-18T15:16:56.951 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:16:56 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/47210795' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-18T15:16:57.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:16:56 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/2430935198' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-18T15:16:57.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:16:56 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/47210795' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-18T15:16:57.345 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:16:56 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/2430935198' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-18T15:16:57.345 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:16:56 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/47210795' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-18T15:16:57.711 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411407 got 77309411413 for osd.2 2024-03-18T15:16:57.711 DEBUG:teuthology.parallel:result is None 2024-03-18T15:16:57.712 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757311 got 163208757315 for osd.5 2024-03-18T15:16:57.712 DEBUG:teuthology.parallel:result is None 2024-03-18T15:16:58.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:16:57 smithi019 ceph-mon[31522]: pgmap v372: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T15:16:58.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:16:57 smithi033 ceph-mon[36648]: pgmap v372: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T15:16:58.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:16:57 smithi104 ceph-mon[36070]: pgmap v372: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T15:16:59.337 INFO:tasks.cephadm.ceph_manager.ceph:need seq 343597383708 got 343597383714 for osd.11 2024-03-18T15:16:59.337 DEBUG:teuthology.parallel:result is None 2024-03-18T15:16:59.465 INFO:tasks.cephadm.ceph_manager.ceph:need seq 253403070512 got 253403070516 for osd.8 2024-03-18T15:16:59.466 DEBUG:teuthology.parallel:result is None 2024-03-18T15:16:59.985 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:59.917+0000 7efd34200700 1 -- 172.21.15.19:0/3712725134 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efd2c102580 msgr2=0x7efd2c102960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:59.985 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:59.917+0000 7efd34200700 1 --2- 172.21.15.19:0/3712725134 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efd2c102580 0x7efd2c102960 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7efd20009990 tx=0x7efd20009ca0 comp rx=0 tx=0).stop 2024-03-18T15:16:59.985 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:59.917+0000 7efd34200700 1 -- 172.21.15.19:0/3712725134 shutdown_connections 2024-03-18T15:16:59.985 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:59.917+0000 7efd34200700 1 --2- 172.21.15.19:0/3712725134 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efd2c107650 0x7efd2c10f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:59.985 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:59.917+0000 7efd34200700 1 --2- 172.21.15.19:0/3712725134 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efd2c102580 0x7efd2c102960 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:59.985 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:59.917+0000 7efd34200700 1 --2- 172.21.15.19:0/3712725134 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efd2c102f30 0x7efd2c106f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:59.985 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:59.917+0000 7efd34200700 1 -- 172.21.15.19:0/3712725134 >> 172.21.15.19:0/3712725134 conn(0x7efd2c079430 msgr2=0x7efd2c079840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:16:59.985 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:59.917+0000 7efd34200700 1 -- 172.21.15.19:0/3712725134 shutdown_connections 2024-03-18T15:16:59.986 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:59.917+0000 7efd34200700 1 -- 172.21.15.19:0/3712725134 wait complete. 2024-03-18T15:16:59.986 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:59.919+0000 7efd34200700 1 Processor -- start 2024-03-18T15:16:59.986 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:59.919+0000 7efd34200700 1 -- start start 2024-03-18T15:16:59.986 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:59.919+0000 7efd34200700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efd2c102580 0x7efd2c19e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:16:59.986 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:59.919+0000 7efd34200700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efd2c102f30 0x7efd2c19ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:16:59.986 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:59.919+0000 7efd34200700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efd2c107650 0x7efd2c198980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:16:59.986 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:59.919+0000 7efd34200700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7efd2c112680 con 0x7efd2c102f30 2024-03-18T15:16:59.986 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:59.919+0000 7efd34200700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7efd2c112500 con 0x7efd2c102580 2024-03-18T15:16:59.986 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:59.919+0000 7efd34200700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7efd2c112800 con 0x7efd2c107650 2024-03-18T15:16:59.986 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:59.919+0000 7efd31800700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efd2c102580 0x7efd2c19e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:16:59.986 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:59.919+0000 7efd31800700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efd2c102580 0x7efd2c19e7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:50228/0 (socket says 172.21.15.19:50228) 2024-03-18T15:16:59.986 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:59.920+0000 7efd31800700 1 -- 172.21.15.19:0/17142847 learned_addr learned my addr 172.21.15.19:0/17142847 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:16:59.986 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:59.920+0000 7efd30e00700 1 --2- 172.21.15.19:0/17142847 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efd2c102f30 0x7efd2c19ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:16:59.987 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:59.920+0000 7efd32200700 1 --2- 172.21.15.19:0/17142847 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efd2c107650 0x7efd2c198980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:16:59.987 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:59.920+0000 7efd31800700 1 -- 172.21.15.19:0/17142847 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efd2c107650 msgr2=0x7efd2c198980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:59.987 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:59.920+0000 7efd31800700 1 --2- 172.21.15.19:0/17142847 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efd2c107650 0x7efd2c198980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:59.987 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:59.920+0000 7efd31800700 1 -- 172.21.15.19:0/17142847 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efd2c102f30 msgr2=0x7efd2c19ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:16:59.987 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:59.920+0000 7efd31800700 1 --2- 172.21.15.19:0/17142847 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efd2c102f30 0x7efd2c19ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:16:59.987 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:59.920+0000 7efd31800700 1 -- 172.21.15.19:0/17142847 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd2c1990e0 con 0x7efd2c102580 2024-03-18T15:16:59.987 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:59.920+0000 7efd31800700 1 --2- 172.21.15.19:0/17142847 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efd2c102580 0x7efd2c19e7c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7efd20005910 tx=0x7efd2000b6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:16:59.987 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:59.921+0000 7efd2a000700 1 -- 172.21.15.19:0/17142847 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd20027070 con 0x7efd2c102580 2024-03-18T15:16:59.987 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:59.921+0000 7efd34200700 1 -- 172.21.15.19:0/17142847 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd2c199370 con 0x7efd2c102580 2024-03-18T15:16:59.987 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:59.921+0000 7efd2a000700 1 -- 172.21.15.19:0/17142847 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efd20022470 con 0x7efd2c102580 2024-03-18T15:16:59.987 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:59.921+0000 7efd2a000700 1 -- 172.21.15.19:0/17142847 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd20017650 con 0x7efd2c102580 2024-03-18T15:16:59.987 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:59.921+0000 7efd34200700 1 -- 172.21.15.19:0/17142847 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd2c1a5690 con 0x7efd2c102580 2024-03-18T15:16:59.987 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:59.922+0000 7efd34200700 1 -- 172.21.15.19:0/17142847 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd2c103a50 con 0x7efd2c102580 2024-03-18T15:16:59.987 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:59.925+0000 7efd2a000700 1 -- 172.21.15.19:0/17142847 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7efd2002a070 con 0x7efd2c102580 2024-03-18T15:16:59.988 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:59.927+0000 7efd2a000700 1 --2- 172.21.15.19:0/17142847 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7efd140a2920 0x7efd140a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:16:59.988 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:59.927+0000 7efd2a000700 1 -- 172.21.15.19:0/17142847 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7efd200cfd80 con 0x7efd2c102580 2024-03-18T15:16:59.988 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:59.927+0000 7efd30e00700 1 --2- 172.21.15.19:0/17142847 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7efd140a2920 0x7efd140a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:16:59.988 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:59.928+0000 7efd30e00700 1 --2- 172.21.15.19:0/17142847 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7efd140a2920 0x7efd140a4de0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7efd2c100e10 tx=0x7efd18009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:16:59.988 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:16:59.936+0000 7efd2a000700 1 -- 172.21.15.19:0/17142847 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7efd20013070 con 0x7efd2c102580 2024-03-18T15:16:59.988 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:16:59 smithi019 ceph-mon[31522]: pgmap v373: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T15:17:00.185 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:00.185+0000 7efd34200700 1 -- 172.21.15.19:0/17142847 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7efd2c199eb0 con 0x7efd2c102580 2024-03-18T15:17:00.186 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:00.186+0000 7efd2a000700 1 -- 172.21.15.19:0/17142847 <== mon.2 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7efd2009c990 con 0x7efd2c102580 2024-03-18T15:17:00.186 INFO:teuthology.orchestra.run.smithi019.stdout:287762808877 2024-03-18T15:17:00.189 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:00.189+0000 7efd34200700 1 -- 172.21.15.19:0/17142847 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7efd140a2920 msgr2=0x7efd140a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:00.189 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:00.189+0000 7efd34200700 1 --2- 172.21.15.19:0/17142847 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7efd140a2920 0x7efd140a4de0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7efd2c100e10 tx=0x7efd18009260 comp rx=0 tx=0).stop 2024-03-18T15:17:00.189 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:00.189+0000 7efd34200700 1 -- 172.21.15.19:0/17142847 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efd2c102580 msgr2=0x7efd2c19e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:00.189 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:00.189+0000 7efd34200700 1 --2- 172.21.15.19:0/17142847 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efd2c102580 0x7efd2c19e7c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7efd20005910 tx=0x7efd2000b6c0 comp rx=0 tx=0).stop 2024-03-18T15:17:00.190 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:00.189+0000 7efd34200700 1 -- 172.21.15.19:0/17142847 shutdown_connections 2024-03-18T15:17:00.190 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:00.189+0000 7efd34200700 1 --2- 172.21.15.19:0/17142847 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7efd140a2920 0x7efd140a4de0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:00.190 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:00.189+0000 7efd34200700 1 --2- 172.21.15.19:0/17142847 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efd2c107650 0x7efd2c198980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:00.190 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:00.189+0000 7efd34200700 1 --2- 172.21.15.19:0/17142847 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efd2c102580 0x7efd2c19e7c0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:00.190 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:00.189+0000 7efd34200700 1 --2- 172.21.15.19:0/17142847 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efd2c102f30 0x7efd2c19ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:00.190 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:00.189+0000 7efd34200700 1 -- 172.21.15.19:0/17142847 >> 172.21.15.19:0/17142847 conn(0x7efd2c079430 msgr2=0x7efd2c101ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:17:00.190 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:00.189+0000 7efd34200700 1 -- 172.21.15.19:0/17142847 shutdown_connections 2024-03-18T15:17:00.190 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:00.189+0000 7efd34200700 1 -- 172.21.15.19:0/17142847 wait complete. 2024-03-18T15:17:00.216 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:17:00.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:16:59 smithi033 ceph-mon[36648]: pgmap v373: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T15:17:00.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:16:59 smithi104 ceph-mon[36070]: pgmap v373: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T15:17:00.773 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:00.057+0000 7fc893800700 1 -- 172.21.15.19:0/3525072714 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc88c108320 msgr2=0x7fc88c108700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:00.773 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:00.057+0000 7fc893800700 1 --2- 172.21.15.19:0/3525072714 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc88c108320 0x7fc88c108700 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fc8800098f0 tx=0x7fc880009c00 comp rx=0 tx=0).stop 2024-03-18T15:17:00.773 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:00.057+0000 7fc893800700 1 -- 172.21.15.19:0/3525072714 shutdown_connections 2024-03-18T15:17:00.773 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:00.057+0000 7fc893800700 1 --2- 172.21.15.19:0/3525072714 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc88c06a920 0x7fc88c10f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:00.773 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:00.057+0000 7fc893800700 1 --2- 172.21.15.19:0/3525072714 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc88c108320 0x7fc88c108700 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:00.773 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:00.057+0000 7fc893800700 1 --2- 172.21.15.19:0/3525072714 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc88c069f80 0x7fc88c06a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:00.773 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:00.057+0000 7fc893800700 1 -- 172.21.15.19:0/3525072714 >> 172.21.15.19:0/3525072714 conn(0x7fc88c078100 msgr2=0x7fc88c078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:17:00.773 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:00.058+0000 7fc893800700 1 -- 172.21.15.19:0/3525072714 shutdown_connections 2024-03-18T15:17:00.773 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:00.058+0000 7fc893800700 1 -- 172.21.15.19:0/3525072714 wait complete. 2024-03-18T15:17:00.774 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:00.059+0000 7fc893800700 1 Processor -- start 2024-03-18T15:17:00.774 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:00.060+0000 7fc893800700 1 -- start start 2024-03-18T15:17:00.774 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:00.060+0000 7fc893800700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc88c069f80 0x7fc88c19e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:00.774 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:00.060+0000 7fc893800700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc88c06a920 0x7fc88c19ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:00.774 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:00.060+0000 7fc893800700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc88c108320 0x7fc88c1988b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:00.774 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:00.060+0000 7fc893800700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fc88c1125b0 con 0x7fc88c108320 2024-03-18T15:17:00.774 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:00.060+0000 7fc893800700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc88c112430 con 0x7fc88c06a920 2024-03-18T15:17:00.774 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:00.060+0000 7fc893800700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fc88c112730 con 0x7fc88c069f80 2024-03-18T15:17:00.774 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:00.060+0000 7fc890e00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc88c069f80 0x7fc88c19e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:00.774 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:00.061+0000 7fc890e00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc88c069f80 0x7fc88c19e6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.19:45306/0 (socket says 172.21.15.19:45306) 2024-03-18T15:17:00.774 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:00.061+0000 7fc890e00700 1 -- 172.21.15.19:0/4274613228 learned_addr learned my addr 172.21.15.19:0/4274613228 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:17:00.774 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:00.061+0000 7fc88be00700 1 --2- 172.21.15.19:0/4274613228 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc88c06a920 0x7fc88c19ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:00.775 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:00.061+0000 7fc891800700 1 --2- 172.21.15.19:0/4274613228 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc88c108320 0x7fc88c1988b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:00.775 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:00.061+0000 7fc88be00700 1 -- 172.21.15.19:0/4274613228 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc88c069f80 msgr2=0x7fc88c19e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:00.775 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:00.061+0000 7fc88be00700 1 --2- 172.21.15.19:0/4274613228 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc88c069f80 0x7fc88c19e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:00.775 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:00.061+0000 7fc88be00700 1 -- 172.21.15.19:0/4274613228 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc88c108320 msgr2=0x7fc88c1988b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:00.775 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:00.061+0000 7fc88be00700 1 --2- 172.21.15.19:0/4274613228 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc88c108320 0x7fc88c1988b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:00.775 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:00.061+0000 7fc88be00700 1 -- 172.21.15.19:0/4274613228 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc88c199010 con 0x7fc88c06a920 2024-03-18T15:17:00.775 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:00.062+0000 7fc88be00700 1 --2- 172.21.15.19:0/4274613228 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc88c06a920 0x7fc88c19ec30 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fc87800e9c0 tx=0x7fc87800ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:17:00.775 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:00.062+0000 7fc889600700 1 -- 172.21.15.19:0/4274613228 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc87800c9c0 con 0x7fc88c06a920 2024-03-18T15:17:00.775 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:00.062+0000 7fc889600700 1 -- 172.21.15.19:0/4274613228 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc87800cb60 con 0x7fc88c06a920 2024-03-18T15:17:00.775 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:00.062+0000 7fc889600700 1 -- 172.21.15.19:0/4274613228 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc878018680 con 0x7fc88c06a920 2024-03-18T15:17:00.775 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:00.062+0000 7fc893800700 1 -- 172.21.15.19:0/4274613228 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc88c199300 con 0x7fc88c06a920 2024-03-18T15:17:00.775 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:00.062+0000 7fc893800700 1 -- 172.21.15.19:0/4274613228 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc88c1a55c0 con 0x7fc88c06a920 2024-03-18T15:17:00.775 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:00.064+0000 7fc893800700 1 -- 172.21.15.19:0/4274613228 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc88c068f90 con 0x7fc88c06a920 2024-03-18T15:17:00.776 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:00.067+0000 7fc889600700 1 -- 172.21.15.19:0/4274613228 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fc87800cd10 con 0x7fc88c06a920 2024-03-18T15:17:00.776 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:00.069+0000 7fc889600700 1 --2- 172.21.15.19:0/4274613228 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fc8740a29a0 0x7fc8740a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:00.776 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:00.069+0000 7fc889600700 1 -- 172.21.15.19:0/4274613228 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fc8780cefe0 con 0x7fc88c06a920 2024-03-18T15:17:00.776 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:00.070+0000 7fc890e00700 1 --2- 172.21.15.19:0/4274613228 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fc8740a29a0 0x7fc8740a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:00.776 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:00.071+0000 7fc890e00700 1 --2- 172.21.15.19:0/4274613228 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fc8740a29a0 0x7fc8740a4e60 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fc880009810 tx=0x7fc88001a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:17:00.776 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:00.073+0000 7fc889600700 1 -- 172.21.15.19:0/4274613228 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc87809bb80 con 0x7fc88c06a920 2024-03-18T15:17:00.776 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:00.345+0000 7fc893800700 1 -- 172.21.15.19:0/4274613228 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fc88c02db70 con 0x7fc88c06a920 2024-03-18T15:17:00.776 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:00.345+0000 7fc889600700 1 -- 172.21.15.19:0/4274613228 <== mon.2 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fc87809b480 con 0x7fc88c06a920 2024-03-18T15:17:00.776 INFO:teuthology.orchestra.run.smithi019.stdout:55834574940 2024-03-18T15:17:00.777 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:00.348+0000 7fc893800700 1 -- 172.21.15.19:0/4274613228 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fc8740a29a0 msgr2=0x7fc8740a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:00.777 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:00.348+0000 7fc893800700 1 --2- 172.21.15.19:0/4274613228 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fc8740a29a0 0x7fc8740a4e60 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fc880009810 tx=0x7fc88001a040 comp rx=0 tx=0).stop 2024-03-18T15:17:00.777 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:00.348+0000 7fc893800700 1 -- 172.21.15.19:0/4274613228 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc88c06a920 msgr2=0x7fc88c19ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:00.777 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:00.348+0000 7fc893800700 1 --2- 172.21.15.19:0/4274613228 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc88c06a920 0x7fc88c19ec30 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fc87800e9c0 tx=0x7fc87800ecd0 comp rx=0 tx=0).stop 2024-03-18T15:17:00.777 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:00.348+0000 7fc893800700 1 -- 172.21.15.19:0/4274613228 shutdown_connections 2024-03-18T15:17:00.777 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:00.348+0000 7fc893800700 1 --2- 172.21.15.19:0/4274613228 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fc8740a29a0 0x7fc8740a4e60 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:00.777 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:00.348+0000 7fc893800700 1 --2- 172.21.15.19:0/4274613228 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc88c069f80 0x7fc88c19e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:00.777 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:00.348+0000 7fc893800700 1 --2- 172.21.15.19:0/4274613228 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc88c108320 0x7fc88c1988b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:00.777 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:00.348+0000 7fc893800700 1 --2- 172.21.15.19:0/4274613228 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc88c06a920 0x7fc88c19ec30 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:00.777 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:00.348+0000 7fc893800700 1 -- 172.21.15.19:0/4274613228 >> 172.21.15.19:0/4274613228 conn(0x7fc88c078100 msgr2=0x7fc88c0ffee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:17:00.777 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:00.349+0000 7fc893800700 1 -- 172.21.15.19:0/4274613228 shutdown_connections 2024-03-18T15:17:00.777 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:00.349+0000 7fc893800700 1 -- 172.21.15.19:0/4274613228 wait complete. 2024-03-18T15:17:01.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:17:00 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/17142847' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-18T15:17:01.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:17:00 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/4274613228' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-18T15:17:01.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:17:00 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/17142847' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-18T15:17:01.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:17:00 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/4274613228' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-18T15:17:01.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:17:00 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/17142847' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-18T15:17:01.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:17:00 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/4274613228' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-18T15:17:01.862 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:01.684+0000 7f4f16600700 1 -- 172.21.15.19:0/3161775903 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4f101044e0 msgr2=0x7f4f10104960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:01.862 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:01.684+0000 7f4f16600700 1 --2- 172.21.15.19:0/3161775903 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4f101044e0 0x7f4f10104960 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f4efc0099c0 tx=0x7f4efc009cd0 comp rx=0 tx=0).stop 2024-03-18T15:17:01.862 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:01.684+0000 7f4f16600700 1 -- 172.21.15.19:0/3161775903 shutdown_connections 2024-03-18T15:17:01.862 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:01.684+0000 7f4f16600700 1 --2- 172.21.15.19:0/3161775903 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4f1010a590 0x7f4f1010a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:01.862 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:01.684+0000 7f4f16600700 1 --2- 172.21.15.19:0/3161775903 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4f10104ea0 0x7f4f10111360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:01.862 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:01.684+0000 7f4f16600700 1 --2- 172.21.15.19:0/3161775903 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4f101044e0 0x7f4f10104960 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:01.862 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:01.684+0000 7f4f16600700 1 -- 172.21.15.19:0/3161775903 >> 172.21.15.19:0/3161775903 conn(0x7f4f100fffb0 msgr2=0x7f4f101023d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:17:01.862 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:01.684+0000 7f4f16600700 1 -- 172.21.15.19:0/3161775903 shutdown_connections 2024-03-18T15:17:01.862 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:01.684+0000 7f4f16600700 1 -- 172.21.15.19:0/3161775903 wait complete. 2024-03-18T15:17:01.862 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:01.685+0000 7f4f16600700 1 Processor -- start 2024-03-18T15:17:01.863 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:01.685+0000 7f4f16600700 1 -- start start 2024-03-18T15:17:01.863 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:01.686+0000 7f4f16600700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4f101044e0 0x7f4f101a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:01.863 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:01.686+0000 7f4f16600700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4f10104ea0 0x7f4f101a0d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:01.863 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:01.686+0000 7f4f16600700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4f1010a590 0x7f4f1019a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:01.863 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:01.686+0000 7f4f16600700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f4f10114840 con 0x7f4f101044e0 2024-03-18T15:17:01.863 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:01.686+0000 7f4f16600700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4f101146c0 con 0x7f4f10104ea0 2024-03-18T15:17:01.863 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:01.686+0000 7f4f16600700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f4f101149c0 con 0x7f4f1010a590 2024-03-18T15:17:01.863 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:01.686+0000 7f4f0ea00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4f10104ea0 0x7f4f101a0d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:01.863 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:01.686+0000 7f4f0f400700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4f101044e0 0x7f4f101a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:01.863 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:01.686+0000 7f4f0ea00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4f10104ea0 0x7f4f101a0d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:50272/0 (socket says 172.21.15.19:50272) 2024-03-18T15:17:01.863 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:01.686+0000 7f4f0ea00700 1 -- 172.21.15.19:0/3399785689 learned_addr learned my addr 172.21.15.19:0/3399785689 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:17:01.863 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:01.686+0000 7f4f0fe00700 1 --2- 172.21.15.19:0/3399785689 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4f1010a590 0x7f4f1019a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:01.863 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:01.686+0000 7f4f0ea00700 1 -- 172.21.15.19:0/3399785689 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4f1010a590 msgr2=0x7f4f1019a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:01.863 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:01.686+0000 7f4f0ea00700 1 --2- 172.21.15.19:0/3399785689 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4f1010a590 0x7f4f1019a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:01.863 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:01.686+0000 7f4f0ea00700 1 -- 172.21.15.19:0/3399785689 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4f101044e0 msgr2=0x7f4f101a0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:01.863 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:01.686+0000 7f4f0ea00700 1 --2- 172.21.15.19:0/3399785689 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4f101044e0 0x7f4f101a0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:01.864 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:01.686+0000 7f4f0ea00700 1 -- 172.21.15.19:0/3399785689 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f1019b140 con 0x7f4f10104ea0 2024-03-18T15:17:01.864 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:01.686+0000 7f4f0fe00700 1 --2- 172.21.15.19:0/3399785689 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4f1010a590 0x7f4f1019a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T15:17:01.864 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:01.687+0000 7f4f0ea00700 1 --2- 172.21.15.19:0/3399785689 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4f10104ea0 0x7f4f101a0d60 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f4efc006010 tx=0x7f4efc003a90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:17:01.864 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:01.687+0000 7f4f07e00700 1 -- 172.21.15.19:0/3399785689 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4efc0050c0 con 0x7f4f10104ea0 2024-03-18T15:17:01.864 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:01.687+0000 7f4f07e00700 1 -- 172.21.15.19:0/3399785689 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4efc00f460 con 0x7f4f10104ea0 2024-03-18T15:17:01.864 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:01.687+0000 7f4f07e00700 1 -- 172.21.15.19:0/3399785689 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4efc017650 con 0x7f4f10104ea0 2024-03-18T15:17:01.864 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:01.687+0000 7f4f16600700 1 -- 172.21.15.19:0/3399785689 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f1019b3d0 con 0x7f4f10104ea0 2024-03-18T15:17:01.864 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:01.687+0000 7f4f16600700 1 -- 172.21.15.19:0/3399785689 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f101a76f0 con 0x7f4f10104ea0 2024-03-18T15:17:01.864 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:01.688+0000 7f4f16600700 1 -- 172.21.15.19:0/3399785689 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f10105c00 con 0x7f4f10104ea0 2024-03-18T15:17:01.864 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:01.690+0000 7f4f07e00700 1 -- 172.21.15.19:0/3399785689 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f4efc00f610 con 0x7f4f10104ea0 2024-03-18T15:17:01.864 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:01.691+0000 7f4f07e00700 1 --2- 172.21.15.19:0/3399785689 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f4ef40a29a0 0x7f4ef40a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:01.864 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:01.692+0000 7f4f07e00700 1 -- 172.21.15.19:0/3399785689 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f4efc0cfce0 con 0x7f4f10104ea0 2024-03-18T15:17:01.864 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:01.692+0000 7f4f0f400700 1 --2- 172.21.15.19:0/3399785689 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f4ef40a29a0 0x7f4ef40a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:01.864 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:01.692+0000 7f4f0f400700 1 --2- 172.21.15.19:0/3399785689 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f4ef40a29a0 0x7f4ef40a4e60 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f4f1019bb20 tx=0x7f4f00009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:17:01.864 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:01.695+0000 7f4f07e00700 1 -- 172.21.15.19:0/3399785689 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4efc013070 con 0x7f4f10104ea0 2024-03-18T15:17:01.925 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:01.925+0000 7f4f16600700 1 -- 172.21.15.19:0/3399785689 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f4f10068f90 con 0x7f4f10104ea0 2024-03-18T15:17:01.926 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:01.926+0000 7f4f07e00700 1 -- 172.21.15.19:0/3399785689 <== mon.2 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f4efc020370 con 0x7f4f10104ea0 2024-03-18T15:17:01.926 INFO:teuthology.orchestra.run.smithi019.stdout:197568495680 2024-03-18T15:17:01.930 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:01.930+0000 7f4f16600700 1 -- 172.21.15.19:0/3399785689 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f4ef40a29a0 msgr2=0x7f4ef40a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:01.930 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:01.930+0000 7f4f16600700 1 --2- 172.21.15.19:0/3399785689 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f4ef40a29a0 0x7f4ef40a4e60 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f4f1019bb20 tx=0x7f4f00009260 comp rx=0 tx=0).stop 2024-03-18T15:17:01.930 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:01.930+0000 7f4f16600700 1 -- 172.21.15.19:0/3399785689 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4f10104ea0 msgr2=0x7f4f101a0d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:01.931 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:01.930+0000 7f4f16600700 1 --2- 172.21.15.19:0/3399785689 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4f10104ea0 0x7f4f101a0d60 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f4efc006010 tx=0x7f4efc003a90 comp rx=0 tx=0).stop 2024-03-18T15:17:01.931 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:01.930+0000 7f4f16600700 1 -- 172.21.15.19:0/3399785689 shutdown_connections 2024-03-18T15:17:01.931 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:01.930+0000 7f4f16600700 1 --2- 172.21.15.19:0/3399785689 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f4ef40a29a0 0x7f4ef40a4e60 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:01.931 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:01.930+0000 7f4f16600700 1 --2- 172.21.15.19:0/3399785689 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4f1010a590 0x7f4f1019a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:01.931 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:01.930+0000 7f4f16600700 1 --2- 172.21.15.19:0/3399785689 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4f101044e0 0x7f4f101a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:01.931 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:01.930+0000 7f4f16600700 1 --2- 172.21.15.19:0/3399785689 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4f10104ea0 0x7f4f101a0d60 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:01.931 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:01.930+0000 7f4f16600700 1 -- 172.21.15.19:0/3399785689 >> 172.21.15.19:0/3399785689 conn(0x7f4f100fffb0 msgr2=0x7f4f101014f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:17:01.931 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:01.930+0000 7f4f16600700 1 -- 172.21.15.19:0/3399785689 shutdown_connections 2024-03-18T15:17:01.931 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:01.931+0000 7f4f16600700 1 -- 172.21.15.19:0/3399785689 wait complete. 2024-03-18T15:17:02.199 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:17:01 smithi019 ceph-mon[31522]: pgmap v374: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T15:17:02.199 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:17:01 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/3399785689' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-18T15:17:02.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:17:01 smithi033 ceph-mon[36648]: pgmap v374: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T15:17:02.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:17:01 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/3399785689' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-18T15:17:02.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:17:01 smithi104 ceph-mon[36070]: pgmap v374: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T15:17:02.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:17:01 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/3399785689' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-18T15:17:02.794 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574932 got 55834574940 for osd.1 2024-03-18T15:17:02.794 DEBUG:teuthology.parallel:result is None 2024-03-18T15:17:02.970 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:02.969+0000 7f9154800700 1 -- 172.21.15.19:0/3677334916 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f914c107680 msgr2=0x7f914c10f200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:02.970 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:02.969+0000 7f9154800700 1 --2- 172.21.15.19:0/3677334916 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f914c107680 0x7f914c10f200 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f914800b0d0 tx=0x7f914800b3e0 comp rx=0 tx=0).stop 2024-03-18T15:17:02.970 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:02.969+0000 7f9154800700 1 -- 172.21.15.19:0/3677334916 shutdown_connections 2024-03-18T15:17:02.970 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:02.969+0000 7f9154800700 1 --2- 172.21.15.19:0/3677334916 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f914c107680 0x7f914c10f200 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:02.970 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:02.969+0000 7f9154800700 1 --2- 172.21.15.19:0/3677334916 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f914c06ac30 0x7f914c06b010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:02.971 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:02.969+0000 7f9154800700 1 --2- 172.21.15.19:0/3677334916 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f914c06b5e0 0x7f914c106f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:02.971 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:02.969+0000 7f9154800700 1 -- 172.21.15.19:0/3677334916 >> 172.21.15.19:0/3677334916 conn(0x7f914c077220 msgr2=0x7f914c077630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:17:02.971 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:02.969+0000 7f9154800700 1 -- 172.21.15.19:0/3677334916 shutdown_connections 2024-03-18T15:17:02.971 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:02.970+0000 7f9154800700 1 -- 172.21.15.19:0/3677334916 wait complete. 2024-03-18T15:17:02.972 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:02.971+0000 7f9154800700 1 Processor -- start 2024-03-18T15:17:02.972 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:02.971+0000 7f9154800700 1 -- start start 2024-03-18T15:17:02.972 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:02.971+0000 7f9154800700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f914c06ac30 0x7f914c101300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:02.972 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:02.971+0000 7f9154800700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f914c06b5e0 0x7f914c101840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:02.972 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:02.971+0000 7f9154800700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f914c107680 0x7f914c101d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:02.973 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:02.971+0000 7f9154800700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f914c112620 con 0x7f914c06b5e0 2024-03-18T15:17:02.973 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:02.971+0000 7f9154800700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f914c1124a0 con 0x7f914c107680 2024-03-18T15:17:02.973 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:02.971+0000 7f9154800700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f914c1127a0 con 0x7f914c06ac30 2024-03-18T15:17:02.973 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:02.971+0000 7f9151e00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f914c06ac30 0x7f914c101300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:02.973 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:02.971+0000 7f9151e00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f914c06ac30 0x7f914c101300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.19:44000/0 (socket says 172.21.15.19:44000) 2024-03-18T15:17:02.973 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:02.972+0000 7f9151e00700 1 -- 172.21.15.19:0/3675561951 learned_addr learned my addr 172.21.15.19:0/3675561951 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:17:02.973 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:02.972+0000 7f9152800700 1 --2- 172.21.15.19:0/3675561951 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f914c107680 0x7f914c101d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:02.973 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:02.972+0000 7f9151400700 1 --2- 172.21.15.19:0/3675561951 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f914c06b5e0 0x7f914c101840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:02.973 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:02.972+0000 7f9152800700 1 -- 172.21.15.19:0/3675561951 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f914c06ac30 msgr2=0x7f914c101300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:02.973 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:02.972+0000 7f9152800700 1 --2- 172.21.15.19:0/3675561951 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f914c06ac30 0x7f914c101300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:02.973 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:02.972+0000 7f9152800700 1 -- 172.21.15.19:0/3675561951 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f914c06b5e0 msgr2=0x7f914c101840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:02.973 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:02.972+0000 7f9152800700 1 --2- 172.21.15.19:0/3675561951 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f914c06b5e0 0x7f914c101840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:02.973 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:02.972+0000 7f9152800700 1 -- 172.21.15.19:0/3675561951 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f914c1a6fc0 con 0x7f914c107680 2024-03-18T15:17:02.973 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:02.972+0000 7f9151e00700 1 --2- 172.21.15.19:0/3675561951 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f914c06ac30 0x7f914c101300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:17:02.974 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:02.972+0000 7f9151400700 1 --2- 172.21.15.19:0/3675561951 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f914c06b5e0 0x7f914c101840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T15:17:02.974 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:02.972+0000 7f9152800700 1 --2- 172.21.15.19:0/3675561951 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f914c107680 0x7f914c101d80 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f914800b770 tx=0x7f9148007a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:17:02.974 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:02.973+0000 7f9146a00700 1 -- 172.21.15.19:0/3675561951 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9148021020 con 0x7f914c107680 2024-03-18T15:17:02.975 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:02.973+0000 7f9146a00700 1 -- 172.21.15.19:0/3675561951 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9148004c50 con 0x7f914c107680 2024-03-18T15:17:02.975 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:02.973+0000 7f9146a00700 1 -- 172.21.15.19:0/3675561951 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91480052b0 con 0x7f914c107680 2024-03-18T15:17:02.975 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:02.973+0000 7f9154800700 1 -- 172.21.15.19:0/3675561951 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f914c1a7250 con 0x7f914c107680 2024-03-18T15:17:02.975 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:02.973+0000 7f9154800700 1 -- 172.21.15.19:0/3675561951 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f914c1a76b0 con 0x7f914c107680 2024-03-18T15:17:02.975 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:02.974+0000 7f9154800700 1 -- 172.21.15.19:0/3675561951 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f914c068f90 con 0x7f914c107680 2024-03-18T15:17:02.976 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:02.976+0000 7f9146a00700 1 -- 172.21.15.19:0/3675561951 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f9148024020 con 0x7f914c107680 2024-03-18T15:17:02.978 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:02.978+0000 7f9146a00700 1 --2- 172.21.15.19:0/3675561951 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f91340a2920 0x7f91340a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:02.978 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:02.978+0000 7f9146a00700 1 -- 172.21.15.19:0/3675561951 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f914800f020 con 0x7f914c107680 2024-03-18T15:17:02.978 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:02.978+0000 7f9151e00700 1 --2- 172.21.15.19:0/3675561951 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f91340a2920 0x7f91340a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:02.978 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:02.979+0000 7f9151e00700 1 --2- 172.21.15.19:0/3675561951 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f91340a2920 0x7f91340a4de0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f9140011550 tx=0x7f9140009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:17:02.986 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:02.986+0000 7f9146a00700 1 -- 172.21.15.19:0/3675561951 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f914809cb50 con 0x7f914c107680 2024-03-18T15:17:03.228 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:03.228+0000 7f9154800700 1 -- 172.21.15.19:0/3675561951 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f914c02dd30 con 0x7f914c107680 2024-03-18T15:17:03.229 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:03.228+0000 7f9146a00700 1 -- 172.21.15.19:0/3675561951 <== mon.2 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f914809c450 con 0x7f914c107680 2024-03-18T15:17:03.229 INFO:teuthology.orchestra.run.smithi019.stdout:141733920845 2024-03-18T15:17:03.232 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:03.231+0000 7f9154800700 1 -- 172.21.15.19:0/3675561951 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f91340a2920 msgr2=0x7f91340a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:03.232 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:03.231+0000 7f9154800700 1 --2- 172.21.15.19:0/3675561951 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f91340a2920 0x7f91340a4de0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f9140011550 tx=0x7f9140009260 comp rx=0 tx=0).stop 2024-03-18T15:17:03.232 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:03.231+0000 7f9154800700 1 -- 172.21.15.19:0/3675561951 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f914c107680 msgr2=0x7f914c101d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:03.232 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:03.231+0000 7f9154800700 1 --2- 172.21.15.19:0/3675561951 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f914c107680 0x7f914c101d80 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f914800b770 tx=0x7f9148007a20 comp rx=0 tx=0).stop 2024-03-18T15:17:03.232 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:03.232+0000 7f9154800700 1 -- 172.21.15.19:0/3675561951 shutdown_connections 2024-03-18T15:17:03.232 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:03.232+0000 7f9154800700 1 --2- 172.21.15.19:0/3675561951 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f91340a2920 0x7f91340a4de0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:03.232 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:03.232+0000 7f9154800700 1 --2- 172.21.15.19:0/3675561951 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f914c06ac30 0x7f914c101300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:03.232 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:03.232+0000 7f9154800700 1 --2- 172.21.15.19:0/3675561951 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f914c107680 0x7f914c101d80 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:03.232 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:03.232+0000 7f9154800700 1 --2- 172.21.15.19:0/3675561951 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f914c06b5e0 0x7f914c101840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:03.233 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:03.232+0000 7f9154800700 1 -- 172.21.15.19:0/3675561951 >> 172.21.15.19:0/3675561951 conn(0x7f914c077220 msgr2=0x7f914c0ff280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:17:03.233 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:03.232+0000 7f9154800700 1 -- 172.21.15.19:0/3675561951 shutdown_connections 2024-03-18T15:17:03.233 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:03.232+0000 7f9154800700 1 -- 172.21.15.19:0/3675561951 wait complete. 2024-03-18T15:17:03.665 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:03.520+0000 7f6980c00700 1 -- 172.21.15.19:0/579042935 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f697c10ed00 msgr2=0x7f697c1110f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:03.666 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:03.520+0000 7f6980c00700 1 --2- 172.21.15.19:0/579042935 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f697c10ed00 0x7f697c1110f0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f697400b0a0 tx=0x7f697400b3b0 comp rx=0 tx=0).stop 2024-03-18T15:17:03.666 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:03.521+0000 7f6980c00700 1 -- 172.21.15.19:0/579042935 shutdown_connections 2024-03-18T15:17:03.666 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:03.521+0000 7f6980c00700 1 --2- 172.21.15.19:0/579042935 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f697c10ed00 0x7f697c1110f0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:03.666 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:03.521+0000 7f6980c00700 1 --2- 172.21.15.19:0/579042935 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f697c074500 0x7f697c0748e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:03.666 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:03.521+0000 7f6980c00700 1 --2- 172.21.15.19:0/579042935 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f697c074e20 0x7f697c10e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:03.666 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:03.521+0000 7f6980c00700 1 -- 172.21.15.19:0/579042935 >> 172.21.15.19:0/579042935 conn(0x7f697c079420 msgr2=0x7f697c079830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:17:03.666 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:03.521+0000 7f6980c00700 1 -- 172.21.15.19:0/579042935 shutdown_connections 2024-03-18T15:17:03.666 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:03.521+0000 7f6980c00700 1 -- 172.21.15.19:0/579042935 wait complete. 2024-03-18T15:17:03.666 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:03.522+0000 7f6980c00700 1 Processor -- start 2024-03-18T15:17:03.666 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:03.522+0000 7f6980c00700 1 -- start start 2024-03-18T15:17:03.666 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:03.522+0000 7f6980c00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f697c074500 0x7f697c1a0890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:03.666 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:03.522+0000 7f6980c00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f697c074e20 0x7f697c1a0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:03.666 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:03.522+0000 7f6980c00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f697c10ed00 0x7f697c19aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:03.667 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:03.522+0000 7f6980c00700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f697c114820 con 0x7f697c074500 2024-03-18T15:17:03.667 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:03.522+0000 7f6980c00700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f697c1146a0 con 0x7f697c074e20 2024-03-18T15:17:03.667 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:03.522+0000 7f6980c00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f697c1149a0 con 0x7f697c10ed00 2024-03-18T15:17:03.667 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:03.523+0000 7f6979400700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f697c074e20 0x7f697c1a0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:03.667 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:03.523+0000 7f6979e00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f697c074500 0x7f697c1a0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:03.667 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:03.523+0000 7f697a800700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f697c10ed00 0x7f697c19aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:03.667 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:03.523+0000 7f6979400700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f697c074e20 0x7f697c1a0dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:43618/0 (socket says 172.21.15.19:43618) 2024-03-18T15:17:03.667 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:03.523+0000 7f6979400700 1 -- 172.21.15.19:0/1428043952 learned_addr learned my addr 172.21.15.19:0/1428043952 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:17:03.667 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:03.523+0000 7f6979400700 1 -- 172.21.15.19:0/1428043952 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f697c10ed00 msgr2=0x7f697c19aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:03.667 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:03.523+0000 7f6979400700 1 --2- 172.21.15.19:0/1428043952 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f697c10ed00 0x7f697c19aaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:03.667 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:03.523+0000 7f6979400700 1 -- 172.21.15.19:0/1428043952 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f697c074500 msgr2=0x7f697c1a0890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:03.667 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:03.523+0000 7f6979400700 1 --2- 172.21.15.19:0/1428043952 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f697c074500 0x7f697c1a0890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:03.667 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:03.523+0000 7f6979400700 1 -- 172.21.15.19:0/1428043952 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f697c19b200 con 0x7f697c074e20 2024-03-18T15:17:03.667 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:03.523+0000 7f697a800700 1 --2- 172.21.15.19:0/1428043952 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f697c10ed00 0x7f697c19aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:17:03.668 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:03.523+0000 7f6979400700 1 --2- 172.21.15.19:0/1428043952 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f697c074e20 0x7f697c1a0dd0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f696800e9a0 tx=0x7f696800ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:17:03.668 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:03.524+0000 7f6972a00700 1 -- 172.21.15.19:0/1428043952 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f696800ca10 con 0x7f697c074e20 2024-03-18T15:17:03.668 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:03.524+0000 7f6972a00700 1 -- 172.21.15.19:0/1428043952 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f696800cbb0 con 0x7f697c074e20 2024-03-18T15:17:03.668 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:03.524+0000 7f6980c00700 1 -- 172.21.15.19:0/1428043952 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f697c19b4f0 con 0x7f697c074e20 2024-03-18T15:17:03.668 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:03.524+0000 7f6972a00700 1 -- 172.21.15.19:0/1428043952 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6968018680 con 0x7f697c074e20 2024-03-18T15:17:03.668 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:03.524+0000 7f6980c00700 1 -- 172.21.15.19:0/1428043952 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f697c1a7760 con 0x7f697c074e20 2024-03-18T15:17:03.668 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:03.525+0000 7f6980c00700 1 -- 172.21.15.19:0/1428043952 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f697c068f90 con 0x7f697c074e20 2024-03-18T15:17:03.668 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:03.527+0000 7f6972a00700 1 -- 172.21.15.19:0/1428043952 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f6968010430 con 0x7f697c074e20 2024-03-18T15:17:03.668 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:03.528+0000 7f6972a00700 1 --2- 172.21.15.19:0/1428043952 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f69600a2920 0x7f69600a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:03.668 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:03.528+0000 7f6972a00700 1 -- 172.21.15.19:0/1428043952 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f69680cf170 con 0x7f697c074e20 2024-03-18T15:17:03.668 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:03.528+0000 7f6979e00700 1 --2- 172.21.15.19:0/1428043952 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f69600a2920 0x7f69600a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:03.668 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:03.529+0000 7f6979e00700 1 --2- 172.21.15.19:0/1428043952 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f69600a2920 0x7f69600a4de0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f697c079bc0 tx=0x7f696c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:17:03.668 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:03.532+0000 7f6972a00700 1 -- 172.21.15.19:0/1428043952 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f696809bcf0 con 0x7f697c074e20 2024-03-18T15:17:03.766 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:03.765+0000 7f6980c00700 1 -- 172.21.15.19:0/1428043952 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7f697c1a7990 con 0x7f697c074e20 2024-03-18T15:17:03.766 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:03.766+0000 7f6972a00700 1 -- 172.21.15.19:0/1428043952 <== mon.2 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f696809b5f0 con 0x7f697c074e20 2024-03-18T15:17:03.766 INFO:teuthology.orchestra.run.smithi019.stdout:309237645353 2024-03-18T15:17:03.770 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:03.770+0000 7f6980c00700 1 -- 172.21.15.19:0/1428043952 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f69600a2920 msgr2=0x7f69600a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:03.770 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:03.770+0000 7f6980c00700 1 --2- 172.21.15.19:0/1428043952 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f69600a2920 0x7f69600a4de0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f697c079bc0 tx=0x7f696c009260 comp rx=0 tx=0).stop 2024-03-18T15:17:03.770 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:03.770+0000 7f6980c00700 1 -- 172.21.15.19:0/1428043952 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f697c074e20 msgr2=0x7f697c1a0dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:03.770 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:03.770+0000 7f6980c00700 1 --2- 172.21.15.19:0/1428043952 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f697c074e20 0x7f697c1a0dd0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f696800e9a0 tx=0x7f696800ecb0 comp rx=0 tx=0).stop 2024-03-18T15:17:03.770 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:03.770+0000 7f6980c00700 1 -- 172.21.15.19:0/1428043952 shutdown_connections 2024-03-18T15:17:03.770 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:03.770+0000 7f6980c00700 1 --2- 172.21.15.19:0/1428043952 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f69600a2920 0x7f69600a4de0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:03.770 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:03.770+0000 7f6980c00700 1 --2- 172.21.15.19:0/1428043952 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f697c10ed00 0x7f697c19aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:03.771 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:03.770+0000 7f6980c00700 1 --2- 172.21.15.19:0/1428043952 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f697c074500 0x7f697c1a0890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:03.771 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:03.770+0000 7f6980c00700 1 --2- 172.21.15.19:0/1428043952 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f697c074e20 0x7f697c1a0dd0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:03.771 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:03.770+0000 7f6980c00700 1 -- 172.21.15.19:0/1428043952 >> 172.21.15.19:0/1428043952 conn(0x7f697c079420 msgr2=0x7f697c0ff510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:17:03.771 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:03.770+0000 7f6980c00700 1 -- 172.21.15.19:0/1428043952 shutdown_connections 2024-03-18T15:17:03.771 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:03.770+0000 7f6980c00700 1 -- 172.21.15.19:0/1428043952 wait complete. 2024-03-18T15:17:04.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:17:03 smithi019 ceph-mon[31522]: pgmap v375: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T15:17:04.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:17:03 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/3675561951' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-18T15:17:04.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:17:03 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/1428043952' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-18T15:17:04.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:17:03 smithi033 ceph-mon[36648]: pgmap v375: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T15:17:04.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:17:03 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/3675561951' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-18T15:17:04.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:17:03 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/1428043952' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-18T15:17:04.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:17:03 smithi104 ceph-mon[36070]: pgmap v375: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T15:17:04.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:17:03 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/3675561951' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-18T15:17:04.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:17:03 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/1428043952' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-18T15:17:04.502 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733920837 got 141733920845 for osd.4 2024-03-18T15:17:04.502 DEBUG:teuthology.parallel:result is None 2024-03-18T15:17:04.787 INFO:tasks.cephadm.ceph_manager.ceph:need seq 219043332146 got 219043332152 for osd.7 2024-03-18T15:17:04.788 DEBUG:teuthology.parallel:result is None 2024-03-18T15:17:05.724 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495674 got 197568495680 for osd.6 2024-03-18T15:17:05.725 DEBUG:teuthology.parallel:result is None 2024-03-18T15:17:06.082 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:06.081+0000 7f5b65800700 1 -- 172.21.15.19:0/3318206480 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5b6010a5b0 msgr2=0x7f5b6010a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:06.082 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:06.081+0000 7f5b65800700 1 --2- 172.21.15.19:0/3318206480 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5b6010a5b0 0x7f5b6010a990 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f5b5000b0d0 tx=0x7f5b5000b3e0 comp rx=0 tx=0).stop 2024-03-18T15:17:06.082 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:06.081+0000 7f5b65800700 1 -- 172.21.15.19:0/3318206480 shutdown_connections 2024-03-18T15:17:06.082 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:06.081+0000 7f5b65800700 1 --2- 172.21.15.19:0/3318206480 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5b60104ec0 0x7f5b60111380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:06.082 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:06.081+0000 7f5b65800700 1 --2- 172.21.15.19:0/3318206480 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5b6010a5b0 0x7f5b6010a990 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:06.082 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:06.081+0000 7f5b65800700 1 --2- 172.21.15.19:0/3318206480 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b60104500 0x7f5b60104980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:06.082 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:06.081+0000 7f5b65800700 1 -- 172.21.15.19:0/3318206480 >> 172.21.15.19:0/3318206480 conn(0x7f5b600fffd0 msgr2=0x7f5b601023f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:17:06.082 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:06.081+0000 7f5b65800700 1 -- 172.21.15.19:0/3318206480 shutdown_connections 2024-03-18T15:17:06.082 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:06.081+0000 7f5b65800700 1 -- 172.21.15.19:0/3318206480 wait complete. 2024-03-18T15:17:06.083 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:06.083+0000 7f5b65800700 1 Processor -- start 2024-03-18T15:17:06.083 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:06.083+0000 7f5b65800700 1 -- start start 2024-03-18T15:17:06.084 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:06.084+0000 7f5b65800700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5b60104500 0x7f5b601a0810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:06.084 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:06.084+0000 7f5b65800700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5b60104ec0 0x7f5b601a0d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:06.084 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:06.084+0000 7f5b65800700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b6010a5b0 0x7f5b6019a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:06.084 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:06.084+0000 7f5b65800700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f5b601147f0 con 0x7f5b60104ec0 2024-03-18T15:17:06.084 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:06.084+0000 7f5b65800700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5b60114670 con 0x7f5b60104500 2024-03-18T15:17:06.084 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:06.084+0000 7f5b65800700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f5b60114970 con 0x7f5b6010a5b0 2024-03-18T15:17:06.085 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:06.084+0000 7f5b5ea00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5b60104500 0x7f5b601a0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:06.085 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:06.084+0000 7f5b5ea00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5b60104500 0x7f5b601a0810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:43642/0 (socket says 172.21.15.19:43642) 2024-03-18T15:17:06.085 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:06.084+0000 7f5b5ea00700 1 -- 172.21.15.19:0/206088099 learned_addr learned my addr 172.21.15.19:0/206088099 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:17:06.085 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:06.084+0000 7f5b5f400700 1 --2- 172.21.15.19:0/206088099 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b6010a5b0 0x7f5b6019a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:06.085 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:06.084+0000 7f5b5e000700 1 --2- 172.21.15.19:0/206088099 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5b60104ec0 0x7f5b601a0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:06.086 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:06.084+0000 7f5b5ea00700 1 -- 172.21.15.19:0/206088099 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b6010a5b0 msgr2=0x7f5b6019a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:06.086 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:06.084+0000 7f5b5ea00700 1 --2- 172.21.15.19:0/206088099 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b6010a5b0 0x7f5b6019a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:06.086 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:06.084+0000 7f5b5ea00700 1 -- 172.21.15.19:0/206088099 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5b60104ec0 msgr2=0x7f5b601a0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:06.086 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:06.084+0000 7f5b5ea00700 1 --2- 172.21.15.19:0/206088099 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5b60104ec0 0x7f5b601a0d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:06.086 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:06.084+0000 7f5b5ea00700 1 -- 172.21.15.19:0/206088099 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b6019b130 con 0x7f5b60104500 2024-03-18T15:17:06.086 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:06.084+0000 7f5b5e000700 1 --2- 172.21.15.19:0/206088099 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5b60104ec0 0x7f5b601a0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T15:17:06.086 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:06.085+0000 7f5b5ea00700 1 --2- 172.21.15.19:0/206088099 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5b60104500 0x7f5b601a0810 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f5b5000d010 tx=0x7f5b50007a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:17:06.086 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:06.085+0000 7f5b57400700 1 -- 172.21.15.19:0/206088099 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b50021020 con 0x7f5b60104500 2024-03-18T15:17:06.086 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:06.085+0000 7f5b57400700 1 -- 172.21.15.19:0/206088099 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b50004c50 con 0x7f5b60104500 2024-03-18T15:17:06.086 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:06.085+0000 7f5b65800700 1 -- 172.21.15.19:0/206088099 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b6019b3c0 con 0x7f5b60104500 2024-03-18T15:17:06.086 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:06.085+0000 7f5b57400700 1 -- 172.21.15.19:0/206088099 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b500052b0 con 0x7f5b60104500 2024-03-18T15:17:06.086 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:06.085+0000 7f5b65800700 1 -- 172.21.15.19:0/206088099 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b601a76e0 con 0x7f5b60104500 2024-03-18T15:17:06.087 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:06.086+0000 7f5b65800700 1 -- 172.21.15.19:0/206088099 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b60068f90 con 0x7f5b60104500 2024-03-18T15:17:06.088 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:06.088+0000 7f5b57400700 1 -- 172.21.15.19:0/206088099 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f5b50024020 con 0x7f5b60104500 2024-03-18T15:17:06.089 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:06.089+0000 7f5b57400700 1 --2- 172.21.15.19:0/206088099 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f5b440a29a0 0x7f5b440a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:06.090 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:06.090+0000 7f5b57400700 1 -- 172.21.15.19:0/206088099 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f5b5000f020 con 0x7f5b60104500 2024-03-18T15:17:06.090 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:06.090+0000 7f5b5e000700 1 --2- 172.21.15.19:0/206088099 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f5b440a29a0 0x7f5b440a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:06.090 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:06.090+0000 7f5b5e000700 1 --2- 172.21.15.19:0/206088099 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f5b440a29a0 0x7f5b440a4e60 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f5b60109b60 tx=0x7f5b4c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:17:06.093 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:06.094+0000 7f5b57400700 1 -- 172.21.15.19:0/206088099 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5b5002b070 con 0x7f5b60104500 2024-03-18T15:17:06.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:17:05 smithi019 ceph-mon[31522]: pgmap v376: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T15:17:06.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:17:05 smithi033 ceph-mon[36648]: pgmap v376: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T15:17:06.325 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:06.324+0000 7f5b65800700 1 -- 172.21.15.19:0/206088099 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f5b601a7910 con 0x7f5b60104500 2024-03-18T15:17:06.325 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:06.325+0000 7f5b57400700 1 -- 172.21.15.19:0/206088099 <== mon.2 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f5b5009c920 con 0x7f5b60104500 2024-03-18T15:17:06.325 INFO:teuthology.orchestra.run.smithi019.stdout:34359738466 2024-03-18T15:17:06.327 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:06.328+0000 7f5b65800700 1 -- 172.21.15.19:0/206088099 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f5b440a29a0 msgr2=0x7f5b440a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:06.328 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:06.328+0000 7f5b65800700 1 --2- 172.21.15.19:0/206088099 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f5b440a29a0 0x7f5b440a4e60 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f5b60109b60 tx=0x7f5b4c009260 comp rx=0 tx=0).stop 2024-03-18T15:17:06.328 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:06.328+0000 7f5b65800700 1 -- 172.21.15.19:0/206088099 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5b60104500 msgr2=0x7f5b601a0810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:06.328 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:06.328+0000 7f5b65800700 1 --2- 172.21.15.19:0/206088099 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5b60104500 0x7f5b601a0810 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f5b5000d010 tx=0x7f5b50007a20 comp rx=0 tx=0).stop 2024-03-18T15:17:06.328 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:06.328+0000 7f5b65800700 1 -- 172.21.15.19:0/206088099 shutdown_connections 2024-03-18T15:17:06.328 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:06.328+0000 7f5b65800700 1 --2- 172.21.15.19:0/206088099 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f5b440a29a0 0x7f5b440a4e60 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:06.328 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:06.328+0000 7f5b65800700 1 --2- 172.21.15.19:0/206088099 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b6010a5b0 0x7f5b6019a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:06.328 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:06.328+0000 7f5b65800700 1 --2- 172.21.15.19:0/206088099 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5b60104500 0x7f5b601a0810 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:06.329 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:06.328+0000 7f5b65800700 1 --2- 172.21.15.19:0/206088099 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5b60104ec0 0x7f5b601a0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:06.329 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:06.328+0000 7f5b65800700 1 -- 172.21.15.19:0/206088099 >> 172.21.15.19:0/206088099 conn(0x7f5b600fffd0 msgr2=0x7f5b60101570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:17:06.329 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:06.328+0000 7f5b65800700 1 -- 172.21.15.19:0/206088099 shutdown_connections 2024-03-18T15:17:06.329 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:06.328+0000 7f5b65800700 1 -- 172.21.15.19:0/206088099 wait complete. 2024-03-18T15:17:06.442 INFO:tasks.cephadm.ceph_manager.ceph:need seq 309237645345 got 309237645353 for osd.10 2024-03-18T15:17:06.442 DEBUG:teuthology.parallel:result is None 2024-03-18T15:17:06.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:17:05 smithi104 ceph-mon[36070]: pgmap v376: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T15:17:07.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:17:07 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/206088099' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-18T15:17:07.383 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:17:07 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/206088099' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-18T15:17:07.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:17:07 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/206088099' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-18T15:17:07.455 INFO:tasks.cephadm.ceph_manager.ceph:need seq 287762808871 got 287762808877 for osd.9 2024-03-18T15:17:07.455 DEBUG:teuthology.parallel:result is None 2024-03-18T15:17:07.617 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738460 got 34359738466 for osd.0 2024-03-18T15:17:07.618 DEBUG:teuthology.parallel:result is None 2024-03-18T15:17:07.618 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-18T15:17:07.618 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 -- ceph pg dump --format=json 2024-03-18T15:17:08.071 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:17:08.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:17:08 smithi033 ceph-mon[36648]: pgmap v377: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T15:17:08.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:17:08 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:17:08.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:17:08 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:17:08.382 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:17:08 smithi019 ceph-mon[31522]: pgmap v377: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T15:17:08.383 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:17:08 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:17:08.383 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:17:08 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:17:08.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:17:08 smithi104 ceph-mon[36070]: pgmap v377: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T15:17:08.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:17:08 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:17:08.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:17:08 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:17:10.047 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:17:10 smithi019 ceph-mon[31522]: pgmap v378: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T15:17:10.048 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:17:10 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:17:10.048 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:17:10 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:17:10.048 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:17:10 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:17:10.339 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:10.338+0000 7f29ad800700 1 -- 172.21.15.19:0/2771041898 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f29a81075a0 msgr2=0x7f29a810f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:10.339 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:10.338+0000 7f29ad800700 1 --2- 172.21.15.19:0/2771041898 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f29a81075a0 0x7f29a810f170 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f29a000b0a0 tx=0x7f29a000b3b0 comp rx=0 tx=0).stop 2024-03-18T15:17:10.339 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:10.338+0000 7f29ad800700 1 -- 172.21.15.19:0/2771041898 shutdown_connections 2024-03-18T15:17:10.339 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:10.338+0000 7f29ad800700 1 --2- 172.21.15.19:0/2771041898 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29a8102580 0x7f29a8102960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:10.339 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:10.338+0000 7f29ad800700 1 --2- 172.21.15.19:0/2771041898 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f29a81075a0 0x7f29a810f170 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:10.339 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:10.338+0000 7f29ad800700 1 --2- 172.21.15.19:0/2771041898 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f29a8102f30 0x7f29a8106ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:10.339 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:10.338+0000 7f29ad800700 1 -- 172.21.15.19:0/2771041898 >> 172.21.15.19:0/2771041898 conn(0x7f29a8079490 msgr2=0x7f29a80798a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:17:10.339 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:10.339+0000 7f29ad800700 1 -- 172.21.15.19:0/2771041898 shutdown_connections 2024-03-18T15:17:10.339 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:10.339+0000 7f29ad800700 1 -- 172.21.15.19:0/2771041898 wait complete. 2024-03-18T15:17:10.341 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:10.340+0000 7f29ad800700 1 Processor -- start 2024-03-18T15:17:10.341 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:10.340+0000 7f29ad800700 1 -- start start 2024-03-18T15:17:10.341 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:10.340+0000 7f29ad800700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f29a8102580 0x7f29a81a08a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:10.341 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:10.340+0000 7f29ad800700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f29a8102f30 0x7f29a81a0de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:10.341 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:10.340+0000 7f29ad800700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29a81075a0 0x7f29a819aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:10.341 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:10.340+0000 7f29ad800700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f29a81125e0 con 0x7f29a8102f30 2024-03-18T15:17:10.341 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:10.340+0000 7f29ad800700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f29a8112460 con 0x7f29a8102580 2024-03-18T15:17:10.341 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:10.340+0000 7f29ad800700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f29a8112760 con 0x7f29a81075a0 2024-03-18T15:17:10.341 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:10.341+0000 7f29a6a00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f29a8102580 0x7f29a81a08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:10.341 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:10.341+0000 7f29a6000700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f29a8102f30 0x7f29a81a0de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:10.342 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:10.341+0000 7f29a6a00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f29a8102580 0x7f29a81a08a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:43662/0 (socket says 172.21.15.19:43662) 2024-03-18T15:17:10.342 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:10.341+0000 7f29a6a00700 1 -- 172.21.15.19:0/2286084930 learned_addr learned my addr 172.21.15.19:0/2286084930 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:17:10.342 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:10.341+0000 7f29a7400700 1 --2- 172.21.15.19:0/2286084930 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29a81075a0 0x7f29a819aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:10.342 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:10.341+0000 7f29a6a00700 1 -- 172.21.15.19:0/2286084930 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29a81075a0 msgr2=0x7f29a819aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:10.342 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:10.341+0000 7f29a6a00700 1 --2- 172.21.15.19:0/2286084930 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29a81075a0 0x7f29a819aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:10.342 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:10.341+0000 7f29a6a00700 1 -- 172.21.15.19:0/2286084930 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f29a8102f30 msgr2=0x7f29a81a0de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:10.343 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:10.341+0000 7f29a6a00700 1 --2- 172.21.15.19:0/2286084930 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f29a8102f30 0x7f29a81a0de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:10.343 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:10.341+0000 7f29a6a00700 1 -- 172.21.15.19:0/2286084930 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29a819b1c0 con 0x7f29a8102580 2024-03-18T15:17:10.343 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:10.341+0000 7f29a7400700 1 --2- 172.21.15.19:0/2286084930 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29a81075a0 0x7f29a819aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:17:10.343 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:10.342+0000 7f29a6a00700 1 --2- 172.21.15.19:0/2286084930 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f29a8102580 0x7f29a81a08a0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f299800d950 tx=0x7f299800dc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:17:10.343 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:10.342+0000 7f299f400700 1 -- 172.21.15.19:0/2286084930 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2998016070 con 0x7f29a8102580 2024-03-18T15:17:10.343 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:10.342+0000 7f299f400700 1 -- 172.21.15.19:0/2286084930 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f299800fd70 con 0x7f29a8102580 2024-03-18T15:17:10.343 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:10.342+0000 7f299f400700 1 -- 172.21.15.19:0/2286084930 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f299800e610 con 0x7f29a8102580 2024-03-18T15:17:10.343 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:10.342+0000 7f29ad800700 1 -- 172.21.15.19:0/2286084930 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29a819b4b0 con 0x7f29a8102580 2024-03-18T15:17:10.343 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:10.342+0000 7f29ad800700 1 -- 172.21.15.19:0/2286084930 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29a81a7770 con 0x7f29a8102580 2024-03-18T15:17:10.343 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:10.343+0000 7f29ad800700 1 -- 172.21.15.19:0/2286084930 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29a8068f90 con 0x7f29a8102580 2024-03-18T15:17:10.345 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:10.346+0000 7f299f400700 1 -- 172.21.15.19:0/2286084930 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f2998019070 con 0x7f29a8102580 2024-03-18T15:17:10.347 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:10.347+0000 7f299f400700 1 --2- 172.21.15.19:0/2286084930 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f298c0a2a70 0x7f298c0a4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:10.347 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:10.347+0000 7f299f400700 1 -- 172.21.15.19:0/2286084930 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f29980ce950 con 0x7f29a8102580 2024-03-18T15:17:10.347 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:10.347+0000 7f29a6000700 1 --2- 172.21.15.19:0/2286084930 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f298c0a2a70 0x7f298c0a4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:10.348 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:10.348+0000 7f29a6000700 1 --2- 172.21.15.19:0/2286084930 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f298c0a2a70 0x7f298c0a4f30 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f29a8076560 tx=0x7f2994009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:17:10.349 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:10.349+0000 7f299f400700 1 -- 172.21.15.19:0/2286084930 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f299809b550 con 0x7f29a8102580 2024-03-18T15:17:10.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:17:10 smithi104 ceph-mon[36070]: pgmap v378: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T15:17:10.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:17:10 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:17:10.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:17:10 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:17:10.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:17:10 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:17:10.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:17:10 smithi033 ceph-mon[36648]: pgmap v378: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T15:17:10.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:17:10 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:17:10.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:17:10 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:17:10.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:17:10 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:17:10.574 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:10.574+0000 7f29ad800700 1 -- 172.21.15.19:0/2286084930 --> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f29a81011a0 con 0x7f298c0a2a70 2024-03-18T15:17:10.588 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:10.588+0000 7f299f400700 1 -- 172.21.15.19:0/2286084930 <== mgr.14152 v2:172.21.15.19:6800/2017714236 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340680 (secure 0 0 0) 0x7f29a81011a0 con 0x7f298c0a2a70 2024-03-18T15:17:10.595 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-18T15:17:10.600 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:10.597+0000 7f29ad800700 1 -- 172.21.15.19:0/2286084930 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f298c0a2a70 msgr2=0x7f298c0a4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:10.600 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:10.597+0000 7f29ad800700 1 --2- 172.21.15.19:0/2286084930 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f298c0a2a70 0x7f298c0a4f30 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f29a8076560 tx=0x7f2994009260 comp rx=0 tx=0).stop 2024-03-18T15:17:10.600 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:10.597+0000 7f29ad800700 1 -- 172.21.15.19:0/2286084930 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f29a8102580 msgr2=0x7f29a81a08a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:10.600 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:10.597+0000 7f29ad800700 1 --2- 172.21.15.19:0/2286084930 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f29a8102580 0x7f29a81a08a0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f299800d950 tx=0x7f299800dc60 comp rx=0 tx=0).stop 2024-03-18T15:17:10.600 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:10.597+0000 7f29ad800700 1 -- 172.21.15.19:0/2286084930 shutdown_connections 2024-03-18T15:17:10.600 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:10.597+0000 7f29ad800700 1 --2- 172.21.15.19:0/2286084930 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f298c0a2a70 0x7f298c0a4f30 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:10.600 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:10.597+0000 7f29ad800700 1 --2- 172.21.15.19:0/2286084930 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29a81075a0 0x7f29a819aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:10.600 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:10.597+0000 7f29ad800700 1 --2- 172.21.15.19:0/2286084930 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f29a8102580 0x7f29a81a08a0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:10.600 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:10.597+0000 7f29ad800700 1 --2- 172.21.15.19:0/2286084930 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f29a8102f30 0x7f29a81a0de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:10.600 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:10.597+0000 7f29ad800700 1 -- 172.21.15.19:0/2286084930 >> 172.21.15.19:0/2286084930 conn(0x7f29a8079490 msgr2=0x7f29a810d870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:17:10.601 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:10.598+0000 7f29ad800700 1 -- 172.21.15.19:0/2286084930 shutdown_connections 2024-03-18T15:17:10.601 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:10.598+0000 7f29ad800700 1 -- 172.21.15.19:0/2286084930 wait complete. 2024-03-18T15:17:10.601 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-03-18T15:17:11.448 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":379,"stamp":"2024-03-18T15:17:10.351947+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":660380,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":983,"num_read_kb":619,"num_write":427,"num_write_kb":3080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":1181358,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":350,"ondisk_log_size":350,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":357368,"kb_used_data":12296,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124584456,"statfs":{"total":1151940427776,"available":1151574482944,"internally_reserved":0,"allocated":12591104,"data_stored":7966946,"data_compressed":359394,"data_compressed_allocated":491520,"data_compressed_original":2482176,"omap_allocated":19143,"internal_metadata":353154361},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":120,"num_read_kb":60,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005564"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.799263+0000","last_change":"2024-03-18T15:15:10.682660+0000","last_active":"2024-03-18T15:16:15.799263+0000","last_peered":"2024-03-18T15:16:15.799263+0000","last_clean":"2024-03-18T15:16:15.799263+0000","last_became_active":"2024-03-18T15:15:10.682433+0000","last_became_peered":"2024-03-18T15:15:10.682433+0000","last_unstale":"2024-03-18T15:16:15.799263+0000","last_undegraded":"2024-03-18T15:16:15.799263+0000","last_fullsized":"2024-03-18T15:16:15.799263+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:48:55.154555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.816543+0000","last_change":"2024-03-18T15:15:10.678100+0000","last_active":"2024-03-18T15:16:15.816543+0000","last_peered":"2024-03-18T15:16:15.816543+0000","last_clean":"2024-03-18T15:16:15.816543+0000","last_became_active":"2024-03-18T15:15:10.677543+0000","last_became_peered":"2024-03-18T15:15:10.677543+0000","last_unstale":"2024-03-18T15:16:15.816543+0000","last_undegraded":"2024-03-18T15:16:15.816543+0000","last_fullsized":"2024-03-18T15:16:15.816543+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:57:50.087837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.815393+0000","last_change":"2024-03-18T15:15:10.674672+0000","last_active":"2024-03-18T15:16:15.815393+0000","last_peered":"2024-03-18T15:16:15.815393+0000","last_clean":"2024-03-18T15:16:15.815393+0000","last_became_active":"2024-03-18T15:15:10.674346+0000","last_became_peered":"2024-03-18T15:15:10.674346+0000","last_unstale":"2024-03-18T15:16:15.815393+0000","last_undegraded":"2024-03-18T15:16:15.815393+0000","last_fullsized":"2024-03-18T15:16:15.815393+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:21:40.054636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.799270+0000","last_change":"2024-03-18T15:15:10.660022+0000","last_active":"2024-03-18T15:16:15.799270+0000","last_peered":"2024-03-18T15:16:15.799270+0000","last_clean":"2024-03-18T15:16:15.799270+0000","last_became_active":"2024-03-18T15:15:10.659622+0000","last_became_peered":"2024-03-18T15:15:10.659622+0000","last_unstale":"2024-03-18T15:16:15.799270+0000","last_undegraded":"2024-03-18T15:16:15.799270+0000","last_fullsized":"2024-03-18T15:16:15.799270+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:55:33.777539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.800804+0000","last_change":"2024-03-18T15:15:10.681499+0000","last_active":"2024-03-18T15:16:15.800804+0000","last_peered":"2024-03-18T15:16:15.800804+0000","last_clean":"2024-03-18T15:16:15.800804+0000","last_became_active":"2024-03-18T15:15:10.681034+0000","last_became_peered":"2024-03-18T15:15:10.681034+0000","last_unstale":"2024-03-18T15:16:15.800804+0000","last_undegraded":"2024-03-18T15:16:15.800804+0000","last_fullsized":"2024-03-18T15:16:15.800804+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:16:43.171924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.805600+0000","last_change":"2024-03-18T15:16:15.805600+0000","last_active":"2024-03-18T15:16:15.805600+0000","last_peered":"2024-03-18T15:16:15.805600+0000","last_clean":"2024-03-18T15:16:15.805600+0000","last_became_active":"2024-03-18T15:16:15.805222+0000","last_became_peered":"2024-03-18T15:16:15.805222+0000","last_unstale":"2024-03-18T15:16:15.805600+0000","last_undegraded":"2024-03-18T15:16:15.805600+0000","last_fullsized":"2024-03-18T15:16:15.805600+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:25:41.772461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,7],"acting":[10,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.802571+0000","last_change":"2024-03-18T15:15:10.669433+0000","last_active":"2024-03-18T15:16:15.802571+0000","last_peered":"2024-03-18T15:16:15.802571+0000","last_clean":"2024-03-18T15:16:15.802571+0000","last_became_active":"2024-03-18T15:15:10.669099+0000","last_became_peered":"2024-03-18T15:15:10.669099+0000","last_unstale":"2024-03-18T15:16:15.802571+0000","last_undegraded":"2024-03-18T15:16:15.802571+0000","last_fullsized":"2024-03-18T15:16:15.802571+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:37:20.697605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.930793+0000","last_change":"2024-03-18T15:15:10.681679+0000","last_active":"2024-03-18T15:16:15.930793+0000","last_peered":"2024-03-18T15:16:15.930793+0000","last_clean":"2024-03-18T15:16:15.930793+0000","last_became_active":"2024-03-18T15:15:10.681079+0000","last_became_peered":"2024-03-18T15:15:10.681079+0000","last_unstale":"2024-03-18T15:16:15.930793+0000","last_undegraded":"2024-03-18T15:16:15.930793+0000","last_fullsized":"2024-03-18T15:16:15.930793+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:24:39.639451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.805022+0000","last_change":"2024-03-18T15:16:15.805022+0000","last_active":"2024-03-18T15:16:15.805022+0000","last_peered":"2024-03-18T15:16:15.805022+0000","last_clean":"2024-03-18T15:16:15.805022+0000","last_became_active":"2024-03-18T15:16:15.804623+0000","last_became_peered":"2024-03-18T15:16:15.804623+0000","last_unstale":"2024-03-18T15:16:15.805022+0000","last_undegraded":"2024-03-18T15:16:15.805022+0000","last_fullsized":"2024-03-18T15:16:15.805022+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:06:50.091834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,11],"acting":[1,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.802351+0000","last_change":"2024-03-18T15:15:10.657799+0000","last_active":"2024-03-18T15:16:15.802351+0000","last_peered":"2024-03-18T15:16:15.802351+0000","last_clean":"2024-03-18T15:16:15.802351+0000","last_became_active":"2024-03-18T15:15:10.657468+0000","last_became_peered":"2024-03-18T15:15:10.657468+0000","last_unstale":"2024-03-18T15:16:15.802351+0000","last_undegraded":"2024-03-18T15:16:15.802351+0000","last_fullsized":"2024-03-18T15:16:15.802351+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:22:46.025516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.804320+0000","last_change":"2024-03-18T15:15:10.665040+0000","last_active":"2024-03-18T15:16:15.804320+0000","last_peered":"2024-03-18T15:16:15.804320+0000","last_clean":"2024-03-18T15:16:15.804320+0000","last_became_active":"2024-03-18T15:15:10.663929+0000","last_became_peered":"2024-03-18T15:15:10.663929+0000","last_unstale":"2024-03-18T15:16:15.804320+0000","last_undegraded":"2024-03-18T15:16:15.804320+0000","last_fullsized":"2024-03-18T15:16:15.804320+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:05:29.954918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.799234+0000","last_change":"2024-03-18T15:15:10.662301+0000","last_active":"2024-03-18T15:16:15.799234+0000","last_peered":"2024-03-18T15:16:15.799234+0000","last_clean":"2024-03-18T15:16:15.799234+0000","last_became_active":"2024-03-18T15:15:10.661956+0000","last_became_peered":"2024-03-18T15:15:10.661956+0000","last_unstale":"2024-03-18T15:16:15.799234+0000","last_undegraded":"2024-03-18T15:16:15.799234+0000","last_fullsized":"2024-03-18T15:16:15.799234+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:30:36.699858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.812752+0000","last_change":"2024-03-18T15:15:10.650795+0000","last_active":"2024-03-18T15:16:15.812752+0000","last_peered":"2024-03-18T15:16:15.812752+0000","last_clean":"2024-03-18T15:16:15.812752+0000","last_became_active":"2024-03-18T15:15:10.649499+0000","last_became_peered":"2024-03-18T15:15:10.649499+0000","last_unstale":"2024-03-18T15:16:15.812752+0000","last_undegraded":"2024-03-18T15:16:15.812752+0000","last_fullsized":"2024-03-18T15:16:15.812752+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:30:19.094619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.812341+0000","last_change":"2024-03-18T15:15:10.677030+0000","last_active":"2024-03-18T15:16:15.812341+0000","last_peered":"2024-03-18T15:16:15.812341+0000","last_clean":"2024-03-18T15:16:15.812341+0000","last_became_active":"2024-03-18T15:15:10.676680+0000","last_became_peered":"2024-03-18T15:15:10.676680+0000","last_unstale":"2024-03-18T15:16:15.812341+0000","last_undegraded":"2024-03-18T15:16:15.812341+0000","last_fullsized":"2024-03-18T15:16:15.812341+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:32:16.382003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.811993+0000","last_change":"2024-03-18T15:15:10.677512+0000","last_active":"2024-03-18T15:16:15.811993+0000","last_peered":"2024-03-18T15:16:15.811993+0000","last_clean":"2024-03-18T15:16:15.811993+0000","last_became_active":"2024-03-18T15:15:10.670557+0000","last_became_peered":"2024-03-18T15:15:10.670557+0000","last_unstale":"2024-03-18T15:16:15.811993+0000","last_undegraded":"2024-03-18T15:16:15.811993+0000","last_fullsized":"2024-03-18T15:16:15.811993+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:06:58.330728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.800896+0000","last_change":"2024-03-18T15:15:10.675859+0000","last_active":"2024-03-18T15:16:15.800896+0000","last_peered":"2024-03-18T15:16:15.800896+0000","last_clean":"2024-03-18T15:16:15.800896+0000","last_became_active":"2024-03-18T15:15:10.675458+0000","last_became_peered":"2024-03-18T15:15:10.675458+0000","last_unstale":"2024-03-18T15:16:15.800896+0000","last_undegraded":"2024-03-18T15:16:15.800896+0000","last_fullsized":"2024-03-18T15:16:15.800896+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:07:43.306884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.799196+0000","last_change":"2024-03-18T15:15:10.677093+0000","last_active":"2024-03-18T15:16:15.799196+0000","last_peered":"2024-03-18T15:16:15.799196+0000","last_clean":"2024-03-18T15:16:15.799196+0000","last_became_active":"2024-03-18T15:15:10.676878+0000","last_became_peered":"2024-03-18T15:15:10.676878+0000","last_unstale":"2024-03-18T15:16:15.799196+0000","last_undegraded":"2024-03-18T15:16:15.799196+0000","last_fullsized":"2024-03-18T15:16:15.799196+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:18:23.517133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.803046+0000","last_change":"2024-03-18T15:15:10.674793+0000","last_active":"2024-03-18T15:16:15.803046+0000","last_peered":"2024-03-18T15:16:15.803046+0000","last_clean":"2024-03-18T15:16:15.803046+0000","last_became_active":"2024-03-18T15:15:10.674433+0000","last_became_peered":"2024-03-18T15:15:10.674433+0000","last_unstale":"2024-03-18T15:16:15.803046+0000","last_undegraded":"2024-03-18T15:16:15.803046+0000","last_fullsized":"2024-03-18T15:16:15.803046+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:17:24.774598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.797173+0000","last_change":"2024-03-18T15:15:10.670757+0000","last_active":"2024-03-18T15:16:15.797173+0000","last_peered":"2024-03-18T15:16:15.797173+0000","last_clean":"2024-03-18T15:16:15.797173+0000","last_became_active":"2024-03-18T15:15:10.670483+0000","last_became_peered":"2024-03-18T15:15:10.670483+0000","last_unstale":"2024-03-18T15:16:15.797173+0000","last_undegraded":"2024-03-18T15:16:15.797173+0000","last_fullsized":"2024-03-18T15:16:15.797173+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:59:20.070597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.796937+0000","last_change":"2024-03-18T15:15:15.115252+0000","last_active":"2024-03-18T15:16:15.796937+0000","last_peered":"2024-03-18T15:16:15.796937+0000","last_clean":"2024-03-18T15:16:15.796937+0000","last_became_active":"2024-03-18T15:15:15.114812+0000","last_became_peered":"2024-03-18T15:15:15.114812+0000","last_unstale":"2024-03-18T15:16:15.796937+0000","last_undegraded":"2024-03-18T15:16:15.796937+0000","last_fullsized":"2024-03-18T15:16:15.796937+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:04:15.657483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'13","reported_seq":125,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:17:09.171511+0000","last_change":"2024-03-18T15:15:10.092873+0000","last_active":"2024-03-18T15:17:09.171511+0000","last_peered":"2024-03-18T15:17:09.171511+0000","last_clean":"2024-03-18T15:17:09.171511+0000","last_became_active":"2024-03-18T15:15:10.092456+0000","last_became_peered":"2024-03-18T15:15:10.092456+0000","last_unstale":"2024-03-18T15:17:09.171511+0000","last_undegraded":"2024-03-18T15:17:09.171511+0000","last_fullsized":"2024-03-18T15:17:09.171511+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:01:53.596264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":85,"num_read_kb":85,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.935342+0000","last_change":"2024-03-18T15:15:10.682119+0000","last_active":"2024-03-18T15:16:15.935342+0000","last_peered":"2024-03-18T15:16:15.935342+0000","last_clean":"2024-03-18T15:16:15.935342+0000","last_became_active":"2024-03-18T15:15:10.680489+0000","last_became_peered":"2024-03-18T15:15:10.680489+0000","last_unstale":"2024-03-18T15:16:15.935342+0000","last_undegraded":"2024-03-18T15:16:15.935342+0000","last_fullsized":"2024-03-18T15:16:15.935342+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:52:13.477107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.798715+0000","last_change":"2024-03-18T15:15:15.114645+0000","last_active":"2024-03-18T15:16:15.798715+0000","last_peered":"2024-03-18T15:16:15.798715+0000","last_clean":"2024-03-18T15:16:15.798715+0000","last_became_active":"2024-03-18T15:15:15.114245+0000","last_became_peered":"2024-03-18T15:15:15.114245+0000","last_unstale":"2024-03-18T15:16:15.798715+0000","last_undegraded":"2024-03-18T15:16:15.798715+0000","last_fullsized":"2024-03-18T15:16:15.798715+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:08:31.429216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.935651+0000","last_change":"2024-03-18T15:15:10.100265+0000","last_active":"2024-03-18T15:16:15.935651+0000","last_peered":"2024-03-18T15:16:15.935651+0000","last_clean":"2024-03-18T15:16:15.935651+0000","last_became_active":"2024-03-18T15:15:10.099861+0000","last_became_peered":"2024-03-18T15:15:10.099861+0000","last_unstale":"2024-03-18T15:16:15.935651+0000","last_undegraded":"2024-03-18T15:16:15.935651+0000","last_fullsized":"2024-03-18T15:16:15.935651+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:52:47.863313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.817547+0000","last_change":"2024-03-18T15:15:10.682009+0000","last_active":"2024-03-18T15:16:15.817547+0000","last_peered":"2024-03-18T15:16:15.817547+0000","last_clean":"2024-03-18T15:16:15.817547+0000","last_became_active":"2024-03-18T15:15:10.680026+0000","last_became_peered":"2024-03-18T15:15:10.680026+0000","last_unstale":"2024-03-18T15:16:15.817547+0000","last_undegraded":"2024-03-18T15:16:15.817547+0000","last_fullsized":"2024-03-18T15:16:15.817547+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:00:27.869376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.799082+0000","last_change":"2024-03-18T15:15:15.124914+0000","last_active":"2024-03-18T15:16:15.799082+0000","last_peered":"2024-03-18T15:16:15.799082+0000","last_clean":"2024-03-18T15:16:15.799082+0000","last_became_active":"2024-03-18T15:15:15.124477+0000","last_became_peered":"2024-03-18T15:15:15.124477+0000","last_unstale":"2024-03-18T15:16:15.799082+0000","last_undegraded":"2024-03-18T15:16:15.799082+0000","last_fullsized":"2024-03-18T15:16:15.799082+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:58:40.596613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'12","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.816450+0000","last_change":"2024-03-18T15:16:15.815550+0000","last_active":"2024-03-18T15:16:15.816450+0000","last_peered":"2024-03-18T15:16:15.816450+0000","last_clean":"2024-03-18T15:16:15.816450+0000","last_became_active":"2024-03-18T15:16:15.804910+0000","last_became_peered":"2024-03-18T15:16:15.804910+0000","last_unstale":"2024-03-18T15:16:15.816450+0000","last_undegraded":"2024-03-18T15:16:15.816450+0000","last_fullsized":"2024-03-18T15:16:15.816450+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:35:51.813890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.802816+0000","last_change":"2024-03-18T15:15:10.669514+0000","last_active":"2024-03-18T15:16:15.802816+0000","last_peered":"2024-03-18T15:16:15.802816+0000","last_clean":"2024-03-18T15:16:15.802816+0000","last_became_active":"2024-03-18T15:15:10.669154+0000","last_became_peered":"2024-03-18T15:15:10.669154+0000","last_unstale":"2024-03-18T15:16:15.802816+0000","last_undegraded":"2024-03-18T15:16:15.802816+0000","last_fullsized":"2024-03-18T15:16:15.802816+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:10:49.068869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.798747+0000","last_change":"2024-03-18T15:15:15.112079+0000","last_active":"2024-03-18T15:16:15.798747+0000","last_peered":"2024-03-18T15:16:15.798747+0000","last_clean":"2024-03-18T15:16:15.798747+0000","last_became_active":"2024-03-18T15:15:15.111652+0000","last_became_peered":"2024-03-18T15:15:15.111652+0000","last_unstale":"2024-03-18T15:16:15.798747+0000","last_undegraded":"2024-03-18T15:16:15.798747+0000","last_fullsized":"2024-03-18T15:16:15.798747+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:15:55.542490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"89'5","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.802029+0000","last_change":"2024-03-18T15:15:10.077975+0000","last_active":"2024-03-18T15:16:15.802029+0000","last_peered":"2024-03-18T15:16:15.802029+0000","last_clean":"2024-03-18T15:16:15.802029+0000","last_became_active":"2024-03-18T15:15:10.077583+0000","last_became_peered":"2024-03-18T15:15:10.077583+0000","last_unstale":"2024-03-18T15:16:15.802029+0000","last_undegraded":"2024-03-18T15:16:15.802029+0000","last_fullsized":"2024-03-18T15:16:15.802029+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:41:43.741623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.801416+0000","last_change":"2024-03-18T15:15:10.665107+0000","last_active":"2024-03-18T15:16:15.801416+0000","last_peered":"2024-03-18T15:16:15.801416+0000","last_clean":"2024-03-18T15:16:15.801416+0000","last_became_active":"2024-03-18T15:15:10.664786+0000","last_became_peered":"2024-03-18T15:15:10.664786+0000","last_unstale":"2024-03-18T15:16:15.801416+0000","last_undegraded":"2024-03-18T15:16:15.801416+0000","last_fullsized":"2024-03-18T15:16:15.801416+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:58:40.439564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.797077+0000","last_change":"2024-03-18T15:15:15.100480+0000","last_active":"2024-03-18T15:16:15.797077+0000","last_peered":"2024-03-18T15:16:15.797077+0000","last_clean":"2024-03-18T15:16:15.797077+0000","last_became_active":"2024-03-18T15:15:15.100015+0000","last_became_peered":"2024-03-18T15:15:15.100015+0000","last_unstale":"2024-03-18T15:16:15.797077+0000","last_undegraded":"2024-03-18T15:16:15.797077+0000","last_fullsized":"2024-03-18T15:16:15.797077+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:50:08.723714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'5","reported_seq":128,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:17:07.897494+0000","last_change":"2024-03-18T15:16:15.818058+0000","last_active":"2024-03-18T15:17:07.897494+0000","last_peered":"2024-03-18T15:17:07.897494+0000","last_clean":"2024-03-18T15:17:07.897494+0000","last_became_active":"2024-03-18T15:16:15.805096+0000","last_became_peered":"2024-03-18T15:16:15.805096+0000","last_unstale":"2024-03-18T15:17:07.897494+0000","last_undegraded":"2024-03-18T15:17:07.897494+0000","last_fullsized":"2024-03-18T15:17:07.897494+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:51:48.938314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":84,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":622,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.801377+0000","last_change":"2024-03-18T15:15:10.667803+0000","last_active":"2024-03-18T15:16:15.801377+0000","last_peered":"2024-03-18T15:16:15.801377+0000","last_clean":"2024-03-18T15:16:15.801377+0000","last_became_active":"2024-03-18T15:15:10.667478+0000","last_became_peered":"2024-03-18T15:15:10.667478+0000","last_unstale":"2024-03-18T15:16:15.801377+0000","last_undegraded":"2024-03-18T15:16:15.801377+0000","last_fullsized":"2024-03-18T15:16:15.801377+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:22:48.340642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.797537+0000","last_change":"2024-03-18T15:15:15.116404+0000","last_active":"2024-03-18T15:16:15.797537+0000","last_peered":"2024-03-18T15:16:15.797537+0000","last_clean":"2024-03-18T15:16:15.797537+0000","last_became_active":"2024-03-18T15:15:15.115952+0000","last_became_peered":"2024-03-18T15:15:15.115952+0000","last_unstale":"2024-03-18T15:16:15.797537+0000","last_undegraded":"2024-03-18T15:16:15.797537+0000","last_fullsized":"2024-03-18T15:16:15.797537+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:38:44.283538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'7","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.802493+0000","last_change":"2024-03-18T15:15:10.080879+0000","last_active":"2024-03-18T15:16:15.802493+0000","last_peered":"2024-03-18T15:16:15.802493+0000","last_clean":"2024-03-18T15:16:15.802493+0000","last_became_active":"2024-03-18T15:15:10.080478+0000","last_became_peered":"2024-03-18T15:15:10.080478+0000","last_unstale":"2024-03-18T15:16:15.802493+0000","last_undegraded":"2024-03-18T15:16:15.802493+0000","last_fullsized":"2024-03-18T15:16:15.802493+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:28:36.511689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.815729+0000","last_change":"2024-03-18T15:16:15.815729+0000","last_active":"2024-03-18T15:16:15.815729+0000","last_peered":"2024-03-18T15:16:15.815729+0000","last_clean":"2024-03-18T15:16:15.815729+0000","last_became_active":"2024-03-18T15:16:15.815302+0000","last_became_peered":"2024-03-18T15:16:15.815302+0000","last_unstale":"2024-03-18T15:16:15.815729+0000","last_undegraded":"2024-03-18T15:16:15.815729+0000","last_fullsized":"2024-03-18T15:16:15.815729+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:56:37.560590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.799096+0000","last_change":"2024-03-18T15:15:15.111563+0000","last_active":"2024-03-18T15:16:15.799096+0000","last_peered":"2024-03-18T15:16:15.799096+0000","last_clean":"2024-03-18T15:16:15.799096+0000","last_became_active":"2024-03-18T15:15:15.111169+0000","last_became_peered":"2024-03-18T15:15:15.111169+0000","last_unstale":"2024-03-18T15:16:15.799096+0000","last_undegraded":"2024-03-18T15:16:15.799096+0000","last_fullsized":"2024-03-18T15:16:15.799096+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:44:05.491883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.812438+0000","last_change":"2024-03-18T15:15:10.091753+0000","last_active":"2024-03-18T15:16:15.812438+0000","last_peered":"2024-03-18T15:16:15.812438+0000","last_clean":"2024-03-18T15:16:15.812438+0000","last_became_active":"2024-03-18T15:15:10.091393+0000","last_became_peered":"2024-03-18T15:15:10.091393+0000","last_unstale":"2024-03-18T15:16:15.812438+0000","last_undegraded":"2024-03-18T15:16:15.812438+0000","last_fullsized":"2024-03-18T15:16:15.812438+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:24:28.068621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.931269+0000","last_change":"2024-03-18T15:15:10.672785+0000","last_active":"2024-03-18T15:16:15.931269+0000","last_peered":"2024-03-18T15:16:15.931269+0000","last_clean":"2024-03-18T15:16:15.931269+0000","last_became_active":"2024-03-18T15:15:10.672143+0000","last_became_peered":"2024-03-18T15:15:10.672143+0000","last_unstale":"2024-03-18T15:16:15.931269+0000","last_undegraded":"2024-03-18T15:16:15.931269+0000","last_fullsized":"2024-03-18T15:16:15.931269+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:26:40.034989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.816327+0000","last_change":"2024-03-18T15:15:15.114790+0000","last_active":"2024-03-18T15:16:15.816327+0000","last_peered":"2024-03-18T15:16:15.816327+0000","last_clean":"2024-03-18T15:16:15.816327+0000","last_became_active":"2024-03-18T15:15:15.112646+0000","last_became_peered":"2024-03-18T15:15:15.112646+0000","last_unstale":"2024-03-18T15:16:15.816327+0000","last_undegraded":"2024-03-18T15:16:15.816327+0000","last_fullsized":"2024-03-18T15:16:15.816327+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:59:56.686888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'3","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.799677+0000","last_change":"2024-03-18T15:15:10.082566+0000","last_active":"2024-03-18T15:16:15.799677+0000","last_peered":"2024-03-18T15:16:15.799677+0000","last_clean":"2024-03-18T15:16:15.799677+0000","last_became_active":"2024-03-18T15:15:10.082256+0000","last_became_peered":"2024-03-18T15:15:10.082256+0000","last_unstale":"2024-03-18T15:16:15.799677+0000","last_undegraded":"2024-03-18T15:16:15.799677+0000","last_fullsized":"2024-03-18T15:16:15.799677+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:04:47.361932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.801404+0000","last_change":"2024-03-18T15:15:10.683148+0000","last_active":"2024-03-18T15:16:15.801404+0000","last_peered":"2024-03-18T15:16:15.801404+0000","last_clean":"2024-03-18T15:16:15.801404+0000","last_became_active":"2024-03-18T15:15:10.682671+0000","last_became_peered":"2024-03-18T15:15:10.682671+0000","last_unstale":"2024-03-18T15:16:15.801404+0000","last_undegraded":"2024-03-18T15:16:15.801404+0000","last_fullsized":"2024-03-18T15:16:15.801404+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:52:15.763400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.932162+0000","last_change":"2024-03-18T15:15:15.115995+0000","last_active":"2024-03-18T15:16:15.932162+0000","last_peered":"2024-03-18T15:16:15.932162+0000","last_clean":"2024-03-18T15:16:15.932162+0000","last_became_active":"2024-03-18T15:15:15.115560+0000","last_became_peered":"2024-03-18T15:15:15.115560+0000","last_unstale":"2024-03-18T15:16:15.932162+0000","last_undegraded":"2024-03-18T15:16:15.932162+0000","last_fullsized":"2024-03-18T15:16:15.932162+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:20:36.765286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'4","reported_seq":191,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:17:06.075351+0000","last_change":"2024-03-18T15:15:10.095122+0000","last_active":"2024-03-18T15:17:06.075351+0000","last_peered":"2024-03-18T15:17:06.075351+0000","last_clean":"2024-03-18T15:17:06.075351+0000","last_became_active":"2024-03-18T15:15:10.094797+0000","last_became_peered":"2024-03-18T15:15:10.094797+0000","last_unstale":"2024-03-18T15:17:06.075351+0000","last_undegraded":"2024-03-18T15:17:06.075351+0000","last_fullsized":"2024-03-18T15:17:06.075351+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:16:15.124369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":160,"num_read_kb":160,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.802418+0000","last_change":"2024-03-18T15:15:10.673297+0000","last_active":"2024-03-18T15:16:15.802418+0000","last_peered":"2024-03-18T15:16:15.802418+0000","last_clean":"2024-03-18T15:16:15.802418+0000","last_became_active":"2024-03-18T15:15:10.672890+0000","last_became_peered":"2024-03-18T15:15:10.672890+0000","last_unstale":"2024-03-18T15:16:15.802418+0000","last_undegraded":"2024-03-18T15:16:15.802418+0000","last_fullsized":"2024-03-18T15:16:15.802418+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:10:45.958639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.800316+0000","last_change":"2024-03-18T15:15:15.123200+0000","last_active":"2024-03-18T15:16:15.800316+0000","last_peered":"2024-03-18T15:16:15.800316+0000","last_clean":"2024-03-18T15:16:15.800316+0000","last_became_active":"2024-03-18T15:15:15.114554+0000","last_became_peered":"2024-03-18T15:15:15.114554+0000","last_unstale":"2024-03-18T15:16:15.800316+0000","last_undegraded":"2024-03-18T15:16:15.800316+0000","last_fullsized":"2024-03-18T15:16:15.800316+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:35:25.380377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'4","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.803775+0000","last_change":"2024-03-18T15:15:10.085330+0000","last_active":"2024-03-18T15:16:15.803775+0000","last_peered":"2024-03-18T15:16:15.803775+0000","last_clean":"2024-03-18T15:16:15.803775+0000","last_became_active":"2024-03-18T15:15:10.083179+0000","last_became_peered":"2024-03-18T15:15:10.083179+0000","last_unstale":"2024-03-18T15:16:15.803775+0000","last_undegraded":"2024-03-18T15:16:15.803775+0000","last_fullsized":"2024-03-18T15:16:15.803775+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:56:33.843757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.817356+0000","last_change":"2024-03-18T15:15:10.679397+0000","last_active":"2024-03-18T15:16:15.817356+0000","last_peered":"2024-03-18T15:16:15.817356+0000","last_clean":"2024-03-18T15:16:15.817356+0000","last_became_active":"2024-03-18T15:15:10.679006+0000","last_became_peered":"2024-03-18T15:15:10.679006+0000","last_unstale":"2024-03-18T15:16:15.817356+0000","last_undegraded":"2024-03-18T15:16:15.817356+0000","last_fullsized":"2024-03-18T15:16:15.817356+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:38:58.251600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.799450+0000","last_change":"2024-03-18T15:15:15.124936+0000","last_active":"2024-03-18T15:16:15.799450+0000","last_peered":"2024-03-18T15:16:15.799450+0000","last_clean":"2024-03-18T15:16:15.799450+0000","last_became_active":"2024-03-18T15:15:15.124501+0000","last_became_peered":"2024-03-18T15:15:15.124501+0000","last_unstale":"2024-03-18T15:16:15.799450+0000","last_undegraded":"2024-03-18T15:16:15.799450+0000","last_fullsized":"2024-03-18T15:16:15.799450+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:31:47.221428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'5","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.801299+0000","last_change":"2024-03-18T15:15:10.098885+0000","last_active":"2024-03-18T15:16:15.801299+0000","last_peered":"2024-03-18T15:16:15.801299+0000","last_clean":"2024-03-18T15:16:15.801299+0000","last_became_active":"2024-03-18T15:15:10.098604+0000","last_became_peered":"2024-03-18T15:15:10.098604+0000","last_unstale":"2024-03-18T15:16:15.801299+0000","last_undegraded":"2024-03-18T15:16:15.801299+0000","last_fullsized":"2024-03-18T15:16:15.801299+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:59:08.716855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.816184+0000","last_change":"2024-03-18T15:16:15.816184+0000","last_active":"2024-03-18T15:16:15.816184+0000","last_peered":"2024-03-18T15:16:15.816184+0000","last_clean":"2024-03-18T15:16:15.816184+0000","last_became_active":"2024-03-18T15:16:15.815726+0000","last_became_peered":"2024-03-18T15:16:15.815726+0000","last_unstale":"2024-03-18T15:16:15.816184+0000","last_undegraded":"2024-03-18T15:16:15.816184+0000","last_fullsized":"2024-03-18T15:16:15.816184+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:14:53.264760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.801409+0000","last_change":"2024-03-18T15:15:15.107352+0000","last_active":"2024-03-18T15:16:15.801409+0000","last_peered":"2024-03-18T15:16:15.801409+0000","last_clean":"2024-03-18T15:16:15.801409+0000","last_became_active":"2024-03-18T15:15:15.106851+0000","last_became_peered":"2024-03-18T15:15:15.106851+0000","last_unstale":"2024-03-18T15:16:15.801409+0000","last_undegraded":"2024-03-18T15:16:15.801409+0000","last_fullsized":"2024-03-18T15:16:15.801409+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:56:35.695188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'35","reported_seq":229,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:17:06.855677+0000","last_change":"2024-03-18T15:15:10.083226+0000","last_active":"2024-03-18T15:17:06.855677+0000","last_peered":"2024-03-18T15:17:06.855677+0000","last_clean":"2024-03-18T15:17:06.855677+0000","last_became_active":"2024-03-18T15:15:10.082829+0000","last_became_peered":"2024-03-18T15:15:10.082829+0000","last_unstale":"2024-03-18T15:17:06.855677+0000","last_undegraded":"2024-03-18T15:17:06.855677+0000","last_fullsized":"2024-03-18T15:17:06.855677+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:30:34.091815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63356,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":167,"num_read_kb":145,"num_write":37,"num_write_kb":77,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.801811+0000","last_change":"2024-03-18T15:15:10.659322+0000","last_active":"2024-03-18T15:16:15.801811+0000","last_peered":"2024-03-18T15:16:15.801811+0000","last_clean":"2024-03-18T15:16:15.801811+0000","last_became_active":"2024-03-18T15:15:10.658997+0000","last_became_peered":"2024-03-18T15:15:10.658997+0000","last_unstale":"2024-03-18T15:16:15.801811+0000","last_undegraded":"2024-03-18T15:16:15.801811+0000","last_fullsized":"2024-03-18T15:16:15.801811+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:04:08.934683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.802402+0000","last_change":"2024-03-18T15:15:15.110658+0000","last_active":"2024-03-18T15:16:15.802402+0000","last_peered":"2024-03-18T15:16:15.802402+0000","last_clean":"2024-03-18T15:16:15.802402+0000","last_became_active":"2024-03-18T15:15:15.110205+0000","last_became_peered":"2024-03-18T15:15:15.110205+0000","last_unstale":"2024-03-18T15:16:15.802402+0000","last_undegraded":"2024-03-18T15:16:15.802402+0000","last_fullsized":"2024-03-18T15:16:15.802402+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:09:10.014743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'9","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.797131+0000","last_change":"2024-03-18T15:15:10.080305+0000","last_active":"2024-03-18T15:16:15.797131+0000","last_peered":"2024-03-18T15:16:15.797131+0000","last_clean":"2024-03-18T15:16:15.797131+0000","last_became_active":"2024-03-18T15:15:10.079990+0000","last_became_peered":"2024-03-18T15:15:10.079990+0000","last_unstale":"2024-03-18T15:16:15.797131+0000","last_undegraded":"2024-03-18T15:16:15.797131+0000","last_fullsized":"2024-03-18T15:16:15.797131+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:10:08.312462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.813884+0000","last_change":"2024-03-18T15:15:10.671178+0000","last_active":"2024-03-18T15:16:15.813884+0000","last_peered":"2024-03-18T15:16:15.813884+0000","last_clean":"2024-03-18T15:16:15.813884+0000","last_became_active":"2024-03-18T15:15:10.670862+0000","last_became_peered":"2024-03-18T15:15:10.670862+0000","last_unstale":"2024-03-18T15:16:15.813884+0000","last_undegraded":"2024-03-18T15:16:15.813884+0000","last_fullsized":"2024-03-18T15:16:15.813884+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:53:34.916473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.798741+0000","last_change":"2024-03-18T15:15:15.107792+0000","last_active":"2024-03-18T15:16:15.798741+0000","last_peered":"2024-03-18T15:16:15.798741+0000","last_clean":"2024-03-18T15:16:15.798741+0000","last_became_active":"2024-03-18T15:15:15.107327+0000","last_became_peered":"2024-03-18T15:15:15.107327+0000","last_unstale":"2024-03-18T15:16:15.798741+0000","last_undegraded":"2024-03-18T15:16:15.798741+0000","last_fullsized":"2024-03-18T15:16:15.798741+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:42:47.837146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'5","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.800846+0000","last_change":"2024-03-18T15:15:10.079128+0000","last_active":"2024-03-18T15:16:15.800846+0000","last_peered":"2024-03-18T15:16:15.800846+0000","last_clean":"2024-03-18T15:16:15.800846+0000","last_became_active":"2024-03-18T15:15:10.078812+0000","last_became_peered":"2024-03-18T15:15:10.078812+0000","last_unstale":"2024-03-18T15:16:15.800846+0000","last_undegraded":"2024-03-18T15:16:15.800846+0000","last_fullsized":"2024-03-18T15:16:15.800846+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:12:11.473273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.803031+0000","last_change":"2024-03-18T15:15:10.668240+0000","last_active":"2024-03-18T15:16:15.803031+0000","last_peered":"2024-03-18T15:16:15.803031+0000","last_clean":"2024-03-18T15:16:15.803031+0000","last_became_active":"2024-03-18T15:15:10.667885+0000","last_became_peered":"2024-03-18T15:15:10.667885+0000","last_unstale":"2024-03-18T15:16:15.803031+0000","last_undegraded":"2024-03-18T15:16:15.803031+0000","last_fullsized":"2024-03-18T15:16:15.803031+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:17:23.998018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.803980+0000","last_change":"2024-03-18T15:15:15.119067+0000","last_active":"2024-03-18T15:16:15.803980+0000","last_peered":"2024-03-18T15:16:15.803980+0000","last_clean":"2024-03-18T15:16:15.803980+0000","last_became_active":"2024-03-18T15:15:15.118566+0000","last_became_peered":"2024-03-18T15:15:15.118566+0000","last_unstale":"2024-03-18T15:16:15.803980+0000","last_undegraded":"2024-03-18T15:16:15.803980+0000","last_fullsized":"2024-03-18T15:16:15.803980+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:21:00.868724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"89'5","reported_seq":113,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:17:04.112362+0000","last_change":"2024-03-18T15:15:10.080791+0000","last_active":"2024-03-18T15:17:04.112362+0000","last_peered":"2024-03-18T15:17:04.112362+0000","last_clean":"2024-03-18T15:17:04.112362+0000","last_became_active":"2024-03-18T15:15:10.080368+0000","last_became_peered":"2024-03-18T15:15:10.080368+0000","last_unstale":"2024-03-18T15:17:04.112362+0000","last_undegraded":"2024-03-18T15:17:04.112362+0000","last_fullsized":"2024-03-18T15:17:04.112362+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:10:55.575331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":82,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.812223+0000","last_change":"2024-03-18T15:16:15.812223+0000","last_active":"2024-03-18T15:16:15.812223+0000","last_peered":"2024-03-18T15:16:15.812223+0000","last_clean":"2024-03-18T15:16:15.812223+0000","last_became_active":"2024-03-18T15:16:15.811768+0000","last_became_peered":"2024-03-18T15:16:15.811768+0000","last_unstale":"2024-03-18T15:16:15.812223+0000","last_undegraded":"2024-03-18T15:16:15.812223+0000","last_fullsized":"2024-03-18T15:16:15.812223+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:32:40.179559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,10],"acting":[7,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.808680+0000","last_change":"2024-03-18T15:15:15.125605+0000","last_active":"2024-03-18T15:16:15.808680+0000","last_peered":"2024-03-18T15:16:15.808680+0000","last_clean":"2024-03-18T15:16:15.808680+0000","last_became_active":"2024-03-18T15:15:15.125041+0000","last_became_peered":"2024-03-18T15:15:15.125041+0000","last_unstale":"2024-03-18T15:16:15.808680+0000","last_undegraded":"2024-03-18T15:16:15.808680+0000","last_fullsized":"2024-03-18T15:16:15.808680+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:27:25.901710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'17","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.800898+0000","last_change":"2024-03-18T15:15:10.079653+0000","last_active":"2024-03-18T15:16:15.800898+0000","last_peered":"2024-03-18T15:16:15.800898+0000","last_clean":"2024-03-18T15:16:15.800898+0000","last_became_active":"2024-03-18T15:15:10.079343+0000","last_became_peered":"2024-03-18T15:15:10.079343+0000","last_unstale":"2024-03-18T15:16:15.800898+0000","last_undegraded":"2024-03-18T15:16:15.800898+0000","last_fullsized":"2024-03-18T15:16:15.800898+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:33:58.746118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.800545+0000","last_change":"2024-03-18T15:15:10.661241+0000","last_active":"2024-03-18T15:16:15.800545+0000","last_peered":"2024-03-18T15:16:15.800545+0000","last_clean":"2024-03-18T15:16:15.800545+0000","last_became_active":"2024-03-18T15:15:10.660821+0000","last_became_peered":"2024-03-18T15:15:10.660821+0000","last_unstale":"2024-03-18T15:16:15.800545+0000","last_undegraded":"2024-03-18T15:16:15.800545+0000","last_fullsized":"2024-03-18T15:16:15.800545+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:13:59.973615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.800441+0000","last_change":"2024-03-18T15:15:15.118314+0000","last_active":"2024-03-18T15:16:15.800441+0000","last_peered":"2024-03-18T15:16:15.800441+0000","last_clean":"2024-03-18T15:16:15.800441+0000","last_became_active":"2024-03-18T15:15:15.118001+0000","last_became_peered":"2024-03-18T15:15:15.118001+0000","last_unstale":"2024-03-18T15:16:15.800441+0000","last_undegraded":"2024-03-18T15:16:15.800441+0000","last_fullsized":"2024-03-18T15:16:15.800441+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:15:58.320283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.802532+0000","last_change":"2024-03-18T15:15:10.077394+0000","last_active":"2024-03-18T15:16:15.802532+0000","last_peered":"2024-03-18T15:16:15.802532+0000","last_clean":"2024-03-18T15:16:15.802532+0000","last_became_active":"2024-03-18T15:15:10.077027+0000","last_became_peered":"2024-03-18T15:15:10.077027+0000","last_unstale":"2024-03-18T15:16:15.802532+0000","last_undegraded":"2024-03-18T15:16:15.802532+0000","last_fullsized":"2024-03-18T15:16:15.802532+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:34:07.524849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.798943+0000","last_change":"2024-03-18T15:15:10.659723+0000","last_active":"2024-03-18T15:16:15.798943+0000","last_peered":"2024-03-18T15:16:15.798943+0000","last_clean":"2024-03-18T15:16:15.798943+0000","last_became_active":"2024-03-18T15:15:10.659388+0000","last_became_peered":"2024-03-18T15:15:10.659388+0000","last_unstale":"2024-03-18T15:16:15.798943+0000","last_undegraded":"2024-03-18T15:16:15.798943+0000","last_fullsized":"2024-03-18T15:16:15.798943+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:21:52.605870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.801279+0000","last_change":"2024-03-18T15:15:15.115459+0000","last_active":"2024-03-18T15:16:15.801279+0000","last_peered":"2024-03-18T15:16:15.801279+0000","last_clean":"2024-03-18T15:16:15.801279+0000","last_became_active":"2024-03-18T15:15:15.115048+0000","last_became_peered":"2024-03-18T15:15:15.115048+0000","last_unstale":"2024-03-18T15:16:15.801279+0000","last_undegraded":"2024-03-18T15:16:15.801279+0000","last_fullsized":"2024-03-18T15:16:15.801279+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:04:23.118904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"89'1","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.814760+0000","last_change":"2024-03-18T15:16:15.814760+0000","last_active":"2024-03-18T15:16:15.814760+0000","last_peered":"2024-03-18T15:16:15.814760+0000","last_clean":"2024-03-18T15:16:15.814760+0000","last_became_active":"2024-03-18T15:16:15.814111+0000","last_became_peered":"2024-03-18T15:16:15.814111+0000","last_unstale":"2024-03-18T15:16:15.814760+0000","last_undegraded":"2024-03-18T15:16:15.814760+0000","last_fullsized":"2024-03-18T15:16:15.814760+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:59:51.241047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.802851+0000","last_change":"2024-03-18T15:15:10.672954+0000","last_active":"2024-03-18T15:16:15.802851+0000","last_peered":"2024-03-18T15:16:15.802851+0000","last_clean":"2024-03-18T15:16:15.802851+0000","last_became_active":"2024-03-18T15:15:10.672415+0000","last_became_peered":"2024-03-18T15:15:10.672415+0000","last_unstale":"2024-03-18T15:16:15.802851+0000","last_undegraded":"2024-03-18T15:16:15.802851+0000","last_fullsized":"2024-03-18T15:16:15.802851+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:47:48.421411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.809642+0000","last_change":"2024-03-18T15:15:15.121334+0000","last_active":"2024-03-18T15:16:15.809642+0000","last_peered":"2024-03-18T15:16:15.809642+0000","last_clean":"2024-03-18T15:16:15.809642+0000","last_became_active":"2024-03-18T15:15:15.119797+0000","last_became_peered":"2024-03-18T15:15:15.119797+0000","last_unstale":"2024-03-18T15:16:15.809642+0000","last_undegraded":"2024-03-18T15:16:15.809642+0000","last_fullsized":"2024-03-18T15:16:15.809642+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:28:39.072485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'5","reported_seq":116,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:17:09.172198+0000","last_change":"2024-03-18T15:15:10.082589+0000","last_active":"2024-03-18T15:17:09.172198+0000","last_peered":"2024-03-18T15:17:09.172198+0000","last_clean":"2024-03-18T15:17:09.172198+0000","last_became_active":"2024-03-18T15:15:10.082190+0000","last_became_peered":"2024-03-18T15:15:10.082190+0000","last_unstale":"2024-03-18T15:17:09.172198+0000","last_undegraded":"2024-03-18T15:17:09.172198+0000","last_fullsized":"2024-03-18T15:17:09.172198+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:55:15.631914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":85,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.801883+0000","last_change":"2024-03-18T15:15:10.672145+0000","last_active":"2024-03-18T15:16:15.801883+0000","last_peered":"2024-03-18T15:16:15.801883+0000","last_clean":"2024-03-18T15:16:15.801883+0000","last_became_active":"2024-03-18T15:15:10.670303+0000","last_became_peered":"2024-03-18T15:15:10.670303+0000","last_unstale":"2024-03-18T15:16:15.801883+0000","last_undegraded":"2024-03-18T15:16:15.801883+0000","last_fullsized":"2024-03-18T15:16:15.801883+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:22:50.528794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.813323+0000","last_change":"2024-03-18T15:15:15.113675+0000","last_active":"2024-03-18T15:16:15.813323+0000","last_peered":"2024-03-18T15:16:15.813323+0000","last_clean":"2024-03-18T15:16:15.813323+0000","last_became_active":"2024-03-18T15:15:15.113067+0000","last_became_peered":"2024-03-18T15:15:15.113067+0000","last_unstale":"2024-03-18T15:16:15.813323+0000","last_undegraded":"2024-03-18T15:16:15.813323+0000","last_fullsized":"2024-03-18T15:16:15.813323+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:37:41.751216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.799877+0000","last_change":"2024-03-18T15:15:10.658032+0000","last_active":"2024-03-18T15:16:15.799877+0000","last_peered":"2024-03-18T15:16:15.799877+0000","last_clean":"2024-03-18T15:16:15.799877+0000","last_became_active":"2024-03-18T15:15:10.657707+0000","last_became_peered":"2024-03-18T15:15:10.657707+0000","last_unstale":"2024-03-18T15:16:15.799877+0000","last_undegraded":"2024-03-18T15:16:15.799877+0000","last_fullsized":"2024-03-18T15:16:15.799877+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:44:24.137349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'5","reported_seq":116,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:17:08.118014+0000","last_change":"2024-03-18T15:15:10.085330+0000","last_active":"2024-03-18T15:17:08.118014+0000","last_peered":"2024-03-18T15:17:08.118014+0000","last_clean":"2024-03-18T15:17:08.118014+0000","last_became_active":"2024-03-18T15:15:10.080303+0000","last_became_peered":"2024-03-18T15:15:10.080303+0000","last_unstale":"2024-03-18T15:17:08.118014+0000","last_undegraded":"2024-03-18T15:17:08.118014+0000","last_fullsized":"2024-03-18T15:17:08.118014+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:26:39.293028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":84,"num_read_kb":84,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.804140+0000","last_change":"2024-03-18T15:15:15.125172+0000","last_active":"2024-03-18T15:16:15.804140+0000","last_peered":"2024-03-18T15:16:15.804140+0000","last_clean":"2024-03-18T15:16:15.804140+0000","last_became_active":"2024-03-18T15:15:15.124591+0000","last_became_peered":"2024-03-18T15:15:15.124591+0000","last_unstale":"2024-03-18T15:16:15.804140+0000","last_undegraded":"2024-03-18T15:16:15.804140+0000","last_fullsized":"2024-03-18T15:16:15.804140+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:50:16.339041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.934902+0000","last_change":"2024-03-18T15:15:10.685343+0000","last_active":"2024-03-18T15:16:15.934902+0000","last_peered":"2024-03-18T15:16:15.934902+0000","last_clean":"2024-03-18T15:16:15.934902+0000","last_became_active":"2024-03-18T15:15:10.684395+0000","last_became_peered":"2024-03-18T15:15:10.684395+0000","last_unstale":"2024-03-18T15:16:15.934902+0000","last_undegraded":"2024-03-18T15:16:15.934902+0000","last_fullsized":"2024-03-18T15:16:15.934902+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:11:26.982047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'6","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.799217+0000","last_change":"2024-03-18T15:15:10.085503+0000","last_active":"2024-03-18T15:16:15.799217+0000","last_peered":"2024-03-18T15:16:15.799217+0000","last_clean":"2024-03-18T15:16:15.799217+0000","last_became_active":"2024-03-18T15:15:10.085125+0000","last_became_peered":"2024-03-18T15:15:10.085125+0000","last_unstale":"2024-03-18T15:16:15.799217+0000","last_undegraded":"2024-03-18T15:16:15.799217+0000","last_fullsized":"2024-03-18T15:16:15.799217+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:17:13.091428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":360,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.802205+0000","last_change":"2024-03-18T15:13:50.952630+0000","last_active":"2024-03-18T15:16:15.802205+0000","last_peered":"2024-03-18T15:16:15.802205+0000","last_clean":"2024-03-18T15:16:15.802205+0000","last_became_active":"2024-03-18T15:13:50.952144+0000","last_became_peered":"2024-03-18T15:13:50.952144+0000","last_unstale":"2024-03-18T15:16:15.802205+0000","last_undegraded":"2024-03-18T15:16:15.802205+0000","last_fullsized":"2024-03-18T15:16:15.802205+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:09:56.845161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:09:56.845161+0000","last_clean_scrub_stamp":"2024-03-18T15:09:56.845161+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:53:27.959711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.800857+0000","last_change":"2024-03-18T15:15:15.115418+0000","last_active":"2024-03-18T15:16:15.800857+0000","last_peered":"2024-03-18T15:16:15.800857+0000","last_clean":"2024-03-18T15:16:15.800857+0000","last_became_active":"2024-03-18T15:15:15.114715+0000","last_became_peered":"2024-03-18T15:15:15.114715+0000","last_unstale":"2024-03-18T15:16:15.800857+0000","last_undegraded":"2024-03-18T15:16:15.800857+0000","last_fullsized":"2024-03-18T15:16:15.800857+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:37:51.553099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.812180+0000","last_change":"2024-03-18T15:15:10.665492+0000","last_active":"2024-03-18T15:16:15.812180+0000","last_peered":"2024-03-18T15:16:15.812180+0000","last_clean":"2024-03-18T15:16:15.812180+0000","last_became_active":"2024-03-18T15:15:10.665097+0000","last_became_peered":"2024-03-18T15:15:10.665097+0000","last_unstale":"2024-03-18T15:16:15.812180+0000","last_undegraded":"2024-03-18T15:16:15.812180+0000","last_fullsized":"2024-03-18T15:16:15.812180+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:27:06.864459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.802314+0000","last_change":"2024-03-18T15:15:10.085322+0000","last_active":"2024-03-18T15:16:15.802314+0000","last_peered":"2024-03-18T15:16:15.802314+0000","last_clean":"2024-03-18T15:16:15.802314+0000","last_became_active":"2024-03-18T15:15:10.083114+0000","last_became_peered":"2024-03-18T15:15:10.083114+0000","last_unstale":"2024-03-18T15:16:15.802314+0000","last_undegraded":"2024-03-18T15:16:15.802314+0000","last_fullsized":"2024-03-18T15:16:15.802314+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:13:31.878513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.798597+0000","last_change":"2024-03-18T15:15:15.105432+0000","last_active":"2024-03-18T15:16:15.798597+0000","last_peered":"2024-03-18T15:16:15.798597+0000","last_clean":"2024-03-18T15:16:15.798597+0000","last_became_active":"2024-03-18T15:15:15.104929+0000","last_became_peered":"2024-03-18T15:15:15.104929+0000","last_unstale":"2024-03-18T15:16:15.798597+0000","last_undegraded":"2024-03-18T15:16:15.798597+0000","last_fullsized":"2024-03-18T15:16:15.798597+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:50:37.334639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.802150+0000","last_change":"2024-03-18T15:15:10.099722+0000","last_active":"2024-03-18T15:16:15.802150+0000","last_peered":"2024-03-18T15:16:15.802150+0000","last_clean":"2024-03-18T15:16:15.802150+0000","last_became_active":"2024-03-18T15:15:10.099410+0000","last_became_peered":"2024-03-18T15:15:10.099410+0000","last_unstale":"2024-03-18T15:16:15.802150+0000","last_undegraded":"2024-03-18T15:16:15.802150+0000","last_fullsized":"2024-03-18T15:16:15.802150+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:19:24.877473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.809054+0000","last_change":"2024-03-18T15:15:10.669235+0000","last_active":"2024-03-18T15:16:15.809054+0000","last_peered":"2024-03-18T15:16:15.809054+0000","last_clean":"2024-03-18T15:16:15.809054+0000","last_became_active":"2024-03-18T15:15:10.667369+0000","last_became_peered":"2024-03-18T15:15:10.667369+0000","last_unstale":"2024-03-18T15:16:15.809054+0000","last_undegraded":"2024-03-18T15:16:15.809054+0000","last_fullsized":"2024-03-18T15:16:15.809054+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:36:08.961435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.931053+0000","last_change":"2024-03-18T15:15:15.119228+0000","last_active":"2024-03-18T15:16:15.931053+0000","last_peered":"2024-03-18T15:16:15.931053+0000","last_clean":"2024-03-18T15:16:15.931053+0000","last_became_active":"2024-03-18T15:15:15.115585+0000","last_became_peered":"2024-03-18T15:15:15.115585+0000","last_unstale":"2024-03-18T15:16:15.931053+0000","last_undegraded":"2024-03-18T15:16:15.931053+0000","last_fullsized":"2024-03-18T15:16:15.931053+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:36:43.263907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"89'14","reported_seq":41,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.803147+0000","last_change":"2024-03-18T15:15:10.105259+0000","last_active":"2024-03-18T15:16:15.803147+0000","last_peered":"2024-03-18T15:16:15.803147+0000","last_clean":"2024-03-18T15:16:15.803147+0000","last_became_active":"2024-03-18T15:15:10.104315+0000","last_became_peered":"2024-03-18T15:15:10.104315+0000","last_unstale":"2024-03-18T15:16:15.803147+0000","last_undegraded":"2024-03-18T15:16:15.803147+0000","last_fullsized":"2024-03-18T15:16:15.803147+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:27:05.565134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.930403+0000","last_change":"2024-03-18T15:15:10.673615+0000","last_active":"2024-03-18T15:16:15.930403+0000","last_peered":"2024-03-18T15:16:15.930403+0000","last_clean":"2024-03-18T15:16:15.930403+0000","last_became_active":"2024-03-18T15:15:10.672308+0000","last_became_peered":"2024-03-18T15:15:10.672308+0000","last_unstale":"2024-03-18T15:16:15.930403+0000","last_undegraded":"2024-03-18T15:16:15.930403+0000","last_fullsized":"2024-03-18T15:16:15.930403+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:32:13.889281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.796772+0000","last_change":"2024-03-18T15:15:15.099447+0000","last_active":"2024-03-18T15:16:15.796772+0000","last_peered":"2024-03-18T15:16:15.796772+0000","last_clean":"2024-03-18T15:16:15.796772+0000","last_became_active":"2024-03-18T15:15:15.098857+0000","last_became_peered":"2024-03-18T15:15:15.098857+0000","last_unstale":"2024-03-18T15:16:15.796772+0000","last_undegraded":"2024-03-18T15:16:15.796772+0000","last_fullsized":"2024-03-18T15:16:15.796772+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:51:22.115359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'7","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.799586+0000","last_change":"2024-03-18T15:15:10.084281+0000","last_active":"2024-03-18T15:16:15.799586+0000","last_peered":"2024-03-18T15:16:15.799586+0000","last_clean":"2024-03-18T15:16:15.799586+0000","last_became_active":"2024-03-18T15:15:10.083791+0000","last_became_peered":"2024-03-18T15:15:10.083791+0000","last_unstale":"2024-03-18T15:16:15.799586+0000","last_undegraded":"2024-03-18T15:16:15.799586+0000","last_fullsized":"2024-03-18T15:16:15.799586+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:08:33.066629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.816717+0000","last_change":"2024-03-18T15:15:10.682823+0000","last_active":"2024-03-18T15:16:15.816717+0000","last_peered":"2024-03-18T15:16:15.816717+0000","last_clean":"2024-03-18T15:16:15.816717+0000","last_became_active":"2024-03-18T15:15:10.681887+0000","last_became_peered":"2024-03-18T15:15:10.681887+0000","last_unstale":"2024-03-18T15:16:15.816717+0000","last_undegraded":"2024-03-18T15:16:15.816717+0000","last_fullsized":"2024-03-18T15:16:15.816717+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:33:29.918824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.799079+0000","last_change":"2024-03-18T15:15:15.114952+0000","last_active":"2024-03-18T15:16:15.799079+0000","last_peered":"2024-03-18T15:16:15.799079+0000","last_clean":"2024-03-18T15:16:15.799079+0000","last_became_active":"2024-03-18T15:15:15.114331+0000","last_became_peered":"2024-03-18T15:15:15.114331+0000","last_unstale":"2024-03-18T15:16:15.799079+0000","last_undegraded":"2024-03-18T15:16:15.799079+0000","last_fullsized":"2024-03-18T15:16:15.799079+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:22:08.637152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.801009+0000","last_change":"2024-03-18T15:15:10.080291+0000","last_active":"2024-03-18T15:16:15.801009+0000","last_peered":"2024-03-18T15:16:15.801009+0000","last_clean":"2024-03-18T15:16:15.801009+0000","last_became_active":"2024-03-18T15:15:10.079634+0000","last_became_peered":"2024-03-18T15:15:10.079634+0000","last_unstale":"2024-03-18T15:16:15.801009+0000","last_undegraded":"2024-03-18T15:16:15.801009+0000","last_fullsized":"2024-03-18T15:16:15.801009+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:12:10.280094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.801033+0000","last_change":"2024-03-18T15:15:10.657245+0000","last_active":"2024-03-18T15:16:15.801033+0000","last_peered":"2024-03-18T15:16:15.801033+0000","last_clean":"2024-03-18T15:16:15.801033+0000","last_became_active":"2024-03-18T15:15:10.655766+0000","last_became_peered":"2024-03-18T15:15:10.655766+0000","last_unstale":"2024-03-18T15:16:15.801033+0000","last_undegraded":"2024-03-18T15:16:15.801033+0000","last_fullsized":"2024-03-18T15:16:15.801033+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:44:26.340539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.803861+0000","last_change":"2024-03-18T15:15:15.109103+0000","last_active":"2024-03-18T15:16:15.803861+0000","last_peered":"2024-03-18T15:16:15.803861+0000","last_clean":"2024-03-18T15:16:15.803861+0000","last_became_active":"2024-03-18T15:15:15.108693+0000","last_became_peered":"2024-03-18T15:15:15.108693+0000","last_unstale":"2024-03-18T15:16:15.803861+0000","last_undegraded":"2024-03-18T15:16:15.803861+0000","last_fullsized":"2024-03-18T15:16:15.803861+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:40:57.841960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'7","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.932435+0000","last_change":"2024-03-18T15:15:10.103183+0000","last_active":"2024-03-18T15:16:15.932435+0000","last_peered":"2024-03-18T15:16:15.932435+0000","last_clean":"2024-03-18T15:16:15.932435+0000","last_became_active":"2024-03-18T15:15:10.101948+0000","last_became_peered":"2024-03-18T15:15:10.101948+0000","last_unstale":"2024-03-18T15:16:15.932435+0000","last_undegraded":"2024-03-18T15:16:15.932435+0000","last_fullsized":"2024-03-18T15:16:15.932435+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:04:51.718433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.798849+0000","last_change":"2024-03-18T15:15:10.666940+0000","last_active":"2024-03-18T15:16:15.798849+0000","last_peered":"2024-03-18T15:16:15.798849+0000","last_clean":"2024-03-18T15:16:15.798849+0000","last_became_active":"2024-03-18T15:15:10.666639+0000","last_became_peered":"2024-03-18T15:15:10.666639+0000","last_unstale":"2024-03-18T15:16:15.798849+0000","last_undegraded":"2024-03-18T15:16:15.798849+0000","last_fullsized":"2024-03-18T15:16:15.798849+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:04:45.945023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.798951+0000","last_change":"2024-03-18T15:15:15.120541+0000","last_active":"2024-03-18T15:16:15.798951+0000","last_peered":"2024-03-18T15:16:15.798951+0000","last_clean":"2024-03-18T15:16:15.798951+0000","last_became_active":"2024-03-18T15:15:15.120097+0000","last_became_peered":"2024-03-18T15:15:15.120097+0000","last_unstale":"2024-03-18T15:16:15.798951+0000","last_undegraded":"2024-03-18T15:16:15.798951+0000","last_fullsized":"2024-03-18T15:16:15.798951+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:12:08.932598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.797301+0000","last_change":"2024-03-18T15:15:10.079199+0000","last_active":"2024-03-18T15:16:15.797301+0000","last_peered":"2024-03-18T15:16:15.797301+0000","last_clean":"2024-03-18T15:16:15.797301+0000","last_became_active":"2024-03-18T15:15:10.078872+0000","last_became_peered":"2024-03-18T15:15:10.078872+0000","last_unstale":"2024-03-18T15:16:15.797301+0000","last_undegraded":"2024-03-18T15:16:15.797301+0000","last_fullsized":"2024-03-18T15:16:15.797301+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:31:44.722779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.930429+0000","last_change":"2024-03-18T15:15:10.686304+0000","last_active":"2024-03-18T15:16:15.930429+0000","last_peered":"2024-03-18T15:16:15.930429+0000","last_clean":"2024-03-18T15:16:15.930429+0000","last_became_active":"2024-03-18T15:15:10.685636+0000","last_became_peered":"2024-03-18T15:15:10.685636+0000","last_unstale":"2024-03-18T15:16:15.930429+0000","last_undegraded":"2024-03-18T15:16:15.930429+0000","last_fullsized":"2024-03-18T15:16:15.930429+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:31:03.315971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.798983+0000","last_change":"2024-03-18T15:15:15.108254+0000","last_active":"2024-03-18T15:16:15.798983+0000","last_peered":"2024-03-18T15:16:15.798983+0000","last_clean":"2024-03-18T15:16:15.798983+0000","last_became_active":"2024-03-18T15:15:15.107806+0000","last_became_peered":"2024-03-18T15:15:15.107806+0000","last_unstale":"2024-03-18T15:16:15.798983+0000","last_undegraded":"2024-03-18T15:16:15.798983+0000","last_fullsized":"2024-03-18T15:16:15.798983+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:55:53.427426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.800920+0000","last_change":"2024-03-18T15:15:10.082574+0000","last_active":"2024-03-18T15:16:15.800920+0000","last_peered":"2024-03-18T15:16:15.800920+0000","last_clean":"2024-03-18T15:16:15.800920+0000","last_became_active":"2024-03-18T15:15:10.082252+0000","last_became_peered":"2024-03-18T15:15:10.082252+0000","last_unstale":"2024-03-18T15:16:15.800920+0000","last_undegraded":"2024-03-18T15:16:15.800920+0000","last_fullsized":"2024-03-18T15:16:15.800920+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:25:17.475023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.803242+0000","last_change":"2024-03-18T15:15:10.667304+0000","last_active":"2024-03-18T15:16:15.803242+0000","last_peered":"2024-03-18T15:16:15.803242+0000","last_clean":"2024-03-18T15:16:15.803242+0000","last_became_active":"2024-03-18T15:15:10.666932+0000","last_became_peered":"2024-03-18T15:15:10.666932+0000","last_unstale":"2024-03-18T15:16:15.803242+0000","last_undegraded":"2024-03-18T15:16:15.803242+0000","last_fullsized":"2024-03-18T15:16:15.803242+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:22:33.817618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.801175+0000","last_change":"2024-03-18T15:15:15.113499+0000","last_active":"2024-03-18T15:16:15.801175+0000","last_peered":"2024-03-18T15:16:15.801175+0000","last_clean":"2024-03-18T15:16:15.801175+0000","last_became_active":"2024-03-18T15:15:15.113095+0000","last_became_peered":"2024-03-18T15:15:15.113095+0000","last_unstale":"2024-03-18T15:16:15.801175+0000","last_undegraded":"2024-03-18T15:16:15.801175+0000","last_fullsized":"2024-03-18T15:16:15.801175+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:58:28.255634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'5","reported_seq":114,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:17:05.074730+0000","last_change":"2024-03-18T15:15:10.078129+0000","last_active":"2024-03-18T15:17:05.074730+0000","last_peered":"2024-03-18T15:17:05.074730+0000","last_clean":"2024-03-18T15:17:05.074730+0000","last_became_active":"2024-03-18T15:15:10.076202+0000","last_became_peered":"2024-03-18T15:15:10.076202+0000","last_unstale":"2024-03-18T15:17:05.074730+0000","last_undegraded":"2024-03-18T15:17:05.074730+0000","last_fullsized":"2024-03-18T15:17:05.074730+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:24:58.240937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":82,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.804294+0000","last_change":"2024-03-18T15:15:10.665036+0000","last_active":"2024-03-18T15:16:15.804294+0000","last_peered":"2024-03-18T15:16:15.804294+0000","last_clean":"2024-03-18T15:16:15.804294+0000","last_became_active":"2024-03-18T15:15:10.664614+0000","last_became_peered":"2024-03-18T15:15:10.664614+0000","last_unstale":"2024-03-18T15:16:15.804294+0000","last_undegraded":"2024-03-18T15:16:15.804294+0000","last_fullsized":"2024-03-18T15:16:15.804294+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:31:42.454317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.810628+0000","last_change":"2024-03-18T15:15:15.127812+0000","last_active":"2024-03-18T15:16:15.810628+0000","last_peered":"2024-03-18T15:16:15.810628+0000","last_clean":"2024-03-18T15:16:15.810628+0000","last_became_active":"2024-03-18T15:15:15.127300+0000","last_became_peered":"2024-03-18T15:15:15.127300+0000","last_unstale":"2024-03-18T15:16:15.810628+0000","last_undegraded":"2024-03-18T15:16:15.810628+0000","last_fullsized":"2024-03-18T15:16:15.810628+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:41:25.292884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"89'3","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.801532+0000","last_change":"2024-03-18T15:15:10.093866+0000","last_active":"2024-03-18T15:16:15.801532+0000","last_peered":"2024-03-18T15:16:15.801532+0000","last_clean":"2024-03-18T15:16:15.801532+0000","last_became_active":"2024-03-18T15:15:10.093568+0000","last_became_peered":"2024-03-18T15:15:10.093568+0000","last_unstale":"2024-03-18T15:16:15.801532+0000","last_undegraded":"2024-03-18T15:16:15.801532+0000","last_fullsized":"2024-03-18T15:16:15.801532+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:00:03.533705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.802657+0000","last_change":"2024-03-18T15:15:10.679506+0000","last_active":"2024-03-18T15:16:15.802657+0000","last_peered":"2024-03-18T15:16:15.802657+0000","last_clean":"2024-03-18T15:16:15.802657+0000","last_became_active":"2024-03-18T15:15:10.679164+0000","last_became_peered":"2024-03-18T15:15:10.679164+0000","last_unstale":"2024-03-18T15:16:15.802657+0000","last_undegraded":"2024-03-18T15:16:15.802657+0000","last_fullsized":"2024-03-18T15:16:15.802657+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:04:55.266273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.799435+0000","last_change":"2024-03-18T15:15:15.099072+0000","last_active":"2024-03-18T15:16:15.799435+0000","last_peered":"2024-03-18T15:16:15.799435+0000","last_clean":"2024-03-18T15:16:15.799435+0000","last_became_active":"2024-03-18T15:15:15.098165+0000","last_became_peered":"2024-03-18T15:15:15.098165+0000","last_unstale":"2024-03-18T15:16:15.799435+0000","last_undegraded":"2024-03-18T15:16:15.799435+0000","last_fullsized":"2024-03-18T15:16:15.799435+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:47:47.377924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'5","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.808997+0000","last_change":"2024-03-18T15:15:10.102562+0000","last_active":"2024-03-18T15:16:15.808997+0000","last_peered":"2024-03-18T15:16:15.808997+0000","last_clean":"2024-03-18T15:16:15.808997+0000","last_became_active":"2024-03-18T15:15:10.099318+0000","last_became_peered":"2024-03-18T15:15:10.099318+0000","last_unstale":"2024-03-18T15:16:15.808997+0000","last_undegraded":"2024-03-18T15:16:15.808997+0000","last_fullsized":"2024-03-18T15:16:15.808997+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:55:15.601925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.804293+0000","last_change":"2024-03-18T15:15:10.662645+0000","last_active":"2024-03-18T15:16:15.804293+0000","last_peered":"2024-03-18T15:16:15.804293+0000","last_clean":"2024-03-18T15:16:15.804293+0000","last_became_active":"2024-03-18T15:15:10.662280+0000","last_became_peered":"2024-03-18T15:15:10.662280+0000","last_unstale":"2024-03-18T15:16:15.804293+0000","last_undegraded":"2024-03-18T15:16:15.804293+0000","last_fullsized":"2024-03-18T15:16:15.804293+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:11:22.598169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.798952+0000","last_change":"2024-03-18T15:15:10.661839+0000","last_active":"2024-03-18T15:16:15.798952+0000","last_peered":"2024-03-18T15:16:15.798952+0000","last_clean":"2024-03-18T15:16:15.798952+0000","last_became_active":"2024-03-18T15:15:10.661356+0000","last_became_peered":"2024-03-18T15:15:10.661356+0000","last_unstale":"2024-03-18T15:16:15.798952+0000","last_undegraded":"2024-03-18T15:16:15.798952+0000","last_fullsized":"2024-03-18T15:16:15.798952+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:13:33.637281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.796879+0000","last_change":"2024-03-18T15:15:10.661946+0000","last_active":"2024-03-18T15:16:15.796879+0000","last_peered":"2024-03-18T15:16:15.796879+0000","last_clean":"2024-03-18T15:16:15.796879+0000","last_became_active":"2024-03-18T15:15:10.661574+0000","last_became_peered":"2024-03-18T15:15:10.661574+0000","last_unstale":"2024-03-18T15:16:15.796879+0000","last_undegraded":"2024-03-18T15:16:15.796879+0000","last_fullsized":"2024-03-18T15:16:15.796879+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:53:50.284156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.796979+0000","last_change":"2024-03-18T15:15:10.674064+0000","last_active":"2024-03-18T15:16:15.796979+0000","last_peered":"2024-03-18T15:16:15.796979+0000","last_clean":"2024-03-18T15:16:15.796979+0000","last_became_active":"2024-03-18T15:15:10.673820+0000","last_became_peered":"2024-03-18T15:15:10.673820+0000","last_unstale":"2024-03-18T15:16:15.796979+0000","last_undegraded":"2024-03-18T15:16:15.796979+0000","last_fullsized":"2024-03-18T15:16:15.796979+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:29:28.977824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.813370+0000","last_change":"2024-03-18T15:15:10.671276+0000","last_active":"2024-03-18T15:16:15.813370+0000","last_peered":"2024-03-18T15:16:15.813370+0000","last_clean":"2024-03-18T15:16:15.813370+0000","last_became_active":"2024-03-18T15:15:10.670926+0000","last_became_peered":"2024-03-18T15:15:10.670926+0000","last_unstale":"2024-03-18T15:16:15.813370+0000","last_undegraded":"2024-03-18T15:16:15.813370+0000","last_fullsized":"2024-03-18T15:16:15.813370+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:15:43.972815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.820673+0000","last_change":"2024-03-18T15:16:15.820673+0000","last_active":"2024-03-18T15:16:15.820673+0000","last_peered":"2024-03-18T15:16:15.820673+0000","last_clean":"2024-03-18T15:16:15.820673+0000","last_became_active":"2024-03-18T15:16:15.819923+0000","last_became_peered":"2024-03-18T15:16:15.819923+0000","last_unstale":"2024-03-18T15:16:15.820673+0000","last_undegraded":"2024-03-18T15:16:15.820673+0000","last_fullsized":"2024-03-18T15:16:15.820673+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:28:51.498407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.799171+0000","last_change":"2024-03-18T15:15:10.658966+0000","last_active":"2024-03-18T15:16:15.799171+0000","last_peered":"2024-03-18T15:16:15.799171+0000","last_clean":"2024-03-18T15:16:15.799171+0000","last_became_active":"2024-03-18T15:15:10.658654+0000","last_became_peered":"2024-03-18T15:15:10.658654+0000","last_unstale":"2024-03-18T15:16:15.799171+0000","last_undegraded":"2024-03-18T15:16:15.799171+0000","last_fullsized":"2024-03-18T15:16:15.799171+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:42:25.730735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.797274+0000","last_change":"2024-03-18T15:15:10.665694+0000","last_active":"2024-03-18T15:16:15.797274+0000","last_peered":"2024-03-18T15:16:15.797274+0000","last_clean":"2024-03-18T15:16:15.797274+0000","last_became_active":"2024-03-18T15:15:10.664688+0000","last_became_peered":"2024-03-18T15:15:10.664688+0000","last_unstale":"2024-03-18T15:16:15.797274+0000","last_undegraded":"2024-03-18T15:16:15.797274+0000","last_fullsized":"2024-03-18T15:16:15.797274+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:32:23.108060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.807899+0000","last_change":"2024-03-18T15:16:15.807899+0000","last_active":"2024-03-18T15:16:15.807899+0000","last_peered":"2024-03-18T15:16:15.807899+0000","last_clean":"2024-03-18T15:16:15.807899+0000","last_became_active":"2024-03-18T15:16:15.807358+0000","last_became_peered":"2024-03-18T15:16:15.807358+0000","last_unstale":"2024-03-18T15:16:15.807899+0000","last_undegraded":"2024-03-18T15:16:15.807899+0000","last_fullsized":"2024-03-18T15:16:15.807899+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:12:55.012985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,2],"acting":[9,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.799781+0000","last_change":"2024-03-18T15:15:10.668833+0000","last_active":"2024-03-18T15:16:15.799781+0000","last_peered":"2024-03-18T15:16:15.799781+0000","last_clean":"2024-03-18T15:16:15.799781+0000","last_became_active":"2024-03-18T15:15:10.668572+0000","last_became_peered":"2024-03-18T15:15:10.668572+0000","last_unstale":"2024-03-18T15:16:15.799781+0000","last_undegraded":"2024-03-18T15:16:15.799781+0000","last_fullsized":"2024-03-18T15:16:15.799781+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:12:44.333515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.936318+0000","last_change":"2024-03-18T15:15:10.677496+0000","last_active":"2024-03-18T15:16:15.936318+0000","last_peered":"2024-03-18T15:16:15.936318+0000","last_clean":"2024-03-18T15:16:15.936318+0000","last_became_active":"2024-03-18T15:15:10.674869+0000","last_became_peered":"2024-03-18T15:15:10.674869+0000","last_unstale":"2024-03-18T15:16:15.936318+0000","last_undegraded":"2024-03-18T15:16:15.936318+0000","last_fullsized":"2024-03-18T15:16:15.936318+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:03:10.531488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.802024+0000","last_change":"2024-03-18T15:15:10.673488+0000","last_active":"2024-03-18T15:16:15.802024+0000","last_peered":"2024-03-18T15:16:15.802024+0000","last_clean":"2024-03-18T15:16:15.802024+0000","last_became_active":"2024-03-18T15:15:10.673117+0000","last_became_peered":"2024-03-18T15:15:10.673117+0000","last_unstale":"2024-03-18T15:16:15.802024+0000","last_undegraded":"2024-03-18T15:16:15.802024+0000","last_fullsized":"2024-03-18T15:16:15.802024+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:29:22.823829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.812634+0000","last_change":"2024-03-18T15:15:10.678916+0000","last_active":"2024-03-18T15:16:15.812634+0000","last_peered":"2024-03-18T15:16:15.812634+0000","last_clean":"2024-03-18T15:16:15.812634+0000","last_became_active":"2024-03-18T15:15:10.678545+0000","last_became_peered":"2024-03-18T15:15:10.678545+0000","last_unstale":"2024-03-18T15:16:15.812634+0000","last_undegraded":"2024-03-18T15:16:15.812634+0000","last_fullsized":"2024-03-18T15:16:15.812634+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:13:52.714537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.798924+0000","last_change":"2024-03-18T15:15:10.660397+0000","last_active":"2024-03-18T15:16:15.798924+0000","last_peered":"2024-03-18T15:16:15.798924+0000","last_clean":"2024-03-18T15:16:15.798924+0000","last_became_active":"2024-03-18T15:15:10.660085+0000","last_became_peered":"2024-03-18T15:15:10.660085+0000","last_unstale":"2024-03-18T15:16:15.798924+0000","last_undegraded":"2024-03-18T15:16:15.798924+0000","last_fullsized":"2024-03-18T15:16:15.798924+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:13:04.591062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3542208,"data_compressed":210616,"data_compressed_allocated":270336,"data_compressed_original":3506176,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":69778,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":847,"num_read_kb":500,"num_write":244,"num_write_kb":140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":622,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":209334,"data_compressed":4773,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":211,"ondisk_log_size":211,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383718,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1003520,"data_stored":501433,"data_compressed":20639,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645354,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28148,"kb_used_data":1124,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717004,"statfs":{"total":95995035648,"available":95966212096,"internally_reserved":0,"allocated":1150976,"data_stored":1151605,"data_compressed":57881,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808880,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32192,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712960,"statfs":{"total":95995035648,"available":95962071040,"internally_reserved":0,"allocated":1032192,"data_stored":505459,"data_compressed":20639,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070519,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32180,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712972,"statfs":{"total":95995035648,"available":95962083328,"internally_reserved":0,"allocated":1024000,"data_stored":501741,"data_compressed":20639,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332156,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28072,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717080,"statfs":{"total":95995035648,"available":95966289920,"internally_reserved":0,"allocated":1077248,"data_stored":1097847,"data_compressed":56290,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495682,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32160,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712992,"statfs":{"total":95995035648,"available":95962103808,"internally_reserved":0,"allocated":1003520,"data_stored":496867,"data_compressed":20639,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574942,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32228,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712924,"statfs":{"total":95995035648,"available":95962034176,"internally_reserved":0,"allocated":1069056,"data_stored":561253,"data_compressed":22230,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738467,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1028096,"data_stored":498323,"data_compressed":20639,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411417,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28080,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717072,"statfs":{"total":95995035648,"available":95966281728,"internally_reserved":0,"allocated":1081344,"data_stored":560851,"data_compressed":22230,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182483,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1015808,"data_stored":496575,"data_compressed":20639,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920846,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32180,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712972,"statfs":{"total":95995035648,"available":95962083328,"internally_reserved":0,"allocated":1019904,"data_stored":501563,"data_compressed":20639,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757320,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28084,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717068,"statfs":{"total":95995035648,"available":95966277632,"internally_reserved":0,"allocated":1085440,"data_stored":1093429,"data_compressed":56290,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34006,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34006,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35651,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35651,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35651,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35651,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":65108,"data_compressed":1591,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":64706,"data_compressed":1591,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":65092,"data_compressed":1591,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T15:17:11.454 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 -- ceph pg dump --format=json 2024-03-18T15:17:11.977 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:17:12.337 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:17:12 smithi019 ceph-mon[31522]: pgmap v379: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T15:17:12.337 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:17:12 smithi019 ceph-mon[31522]: from='client.25025 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T15:17:12.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:17:12 smithi104 ceph-mon[36070]: pgmap v379: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T15:17:12.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:17:12 smithi104 ceph-mon[36070]: from='client.25025 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T15:17:12.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:17:12 smithi033 ceph-mon[36648]: pgmap v379: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T15:17:12.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:17:12 smithi033 ceph-mon[36648]: from='client.25025 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T15:17:14.200 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:14.199+0000 7f9e83e00700 1 -- 172.21.15.19:0/2020259849 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9e84102dc0 msgr2=0x7f9e84106e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:14.213 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:14.199+0000 7f9e83e00700 1 --2- 172.21.15.19:0/2020259849 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9e84102dc0 0x7f9e84106e10 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f9e6c0099c0 tx=0x7f9e6c009cd0 comp rx=0 tx=0).stop 2024-03-18T15:17:14.213 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:14.199+0000 7f9e83e00700 1 -- 172.21.15.19:0/2020259849 shutdown_connections 2024-03-18T15:17:14.213 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:14.199+0000 7f9e83e00700 1 --2- 172.21.15.19:0/2020259849 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9e841074f0 0x7f9e8410f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:14.213 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:14.199+0000 7f9e83e00700 1 --2- 172.21.15.19:0/2020259849 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9e84102410 0x7f9e841027f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:14.213 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:14.199+0000 7f9e83e00700 1 --2- 172.21.15.19:0/2020259849 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9e84102dc0 0x7f9e84106e10 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:14.213 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:14.199+0000 7f9e83e00700 1 -- 172.21.15.19:0/2020259849 >> 172.21.15.19:0/2020259849 conn(0x7f9e840fdca0 msgr2=0x7f9e841000c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:17:14.213 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:14.199+0000 7f9e83e00700 1 -- 172.21.15.19:0/2020259849 shutdown_connections 2024-03-18T15:17:14.214 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:14.199+0000 7f9e83e00700 1 -- 172.21.15.19:0/2020259849 wait complete. 2024-03-18T15:17:14.214 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:14.200+0000 7f9e83e00700 1 Processor -- start 2024-03-18T15:17:14.214 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:14.201+0000 7f9e83e00700 1 -- start start 2024-03-18T15:17:14.216 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:14.201+0000 7f9e83e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9e84102410 0x7f9e841a0810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:14.216 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:14.201+0000 7f9e83e00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9e84102dc0 0x7f9e841a0d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:14.216 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:14.201+0000 7f9e83e00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9e841074f0 0x7f9e8419a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:14.216 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:14.201+0000 7f9e83e00700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f9e841124b0 con 0x7f9e841074f0 2024-03-18T15:17:14.216 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:14.201+0000 7f9e83e00700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9e84112330 con 0x7f9e84102410 2024-03-18T15:17:14.216 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:14.201+0000 7f9e83e00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f9e84112630 con 0x7f9e84102dc0 2024-03-18T15:17:14.216 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:14.201+0000 7f9e82a00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9e84102410 0x7f9e841a0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:14.216 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:14.201+0000 7f9e7a000700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9e84102dc0 0x7f9e841a0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:14.216 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:14.201+0000 7f9e82a00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9e84102410 0x7f9e841a0810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:56028/0 (socket says 172.21.15.19:56028) 2024-03-18T15:17:14.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:14.201+0000 7f9e82a00700 1 -- 172.21.15.19:0/3309104084 learned_addr learned my addr 172.21.15.19:0/3309104084 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:17:14.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:14.202+0000 7f9e83400700 1 --2- 172.21.15.19:0/3309104084 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9e841074f0 0x7f9e8419a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:14.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:14.202+0000 7f9e82a00700 1 -- 172.21.15.19:0/3309104084 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9e84102dc0 msgr2=0x7f9e841a0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:14.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:14.202+0000 7f9e82a00700 1 --2- 172.21.15.19:0/3309104084 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9e84102dc0 0x7f9e841a0d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:14.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:14.202+0000 7f9e82a00700 1 -- 172.21.15.19:0/3309104084 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9e841074f0 msgr2=0x7f9e8419a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:14.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:14.202+0000 7f9e82a00700 1 --2- 172.21.15.19:0/3309104084 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9e841074f0 0x7f9e8419a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:14.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:14.202+0000 7f9e82a00700 1 -- 172.21.15.19:0/3309104084 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e8419b130 con 0x7f9e84102410 2024-03-18T15:17:14.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:14.202+0000 7f9e83400700 1 --2- 172.21.15.19:0/3309104084 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9e841074f0 0x7f9e8419a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T15:17:14.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:14.202+0000 7f9e7a000700 1 --2- 172.21.15.19:0/3309104084 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9e84102dc0 0x7f9e841a0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:17:14.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:14.202+0000 7f9e82a00700 1 --2- 172.21.15.19:0/3309104084 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9e84102410 0x7f9e841a0810 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f9e7400eb30 tx=0x7f9e7400ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:17:14.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:14.203+0000 7f9e7be00700 1 -- 172.21.15.19:0/3309104084 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e7400cc90 con 0x7f9e84102410 2024-03-18T15:17:14.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:14.203+0000 7f9e7be00700 1 -- 172.21.15.19:0/3309104084 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e74004500 con 0x7f9e84102410 2024-03-18T15:17:14.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:14.203+0000 7f9e83e00700 1 -- 172.21.15.19:0/3309104084 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e8419b420 con 0x7f9e84102410 2024-03-18T15:17:14.218 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:14.203+0000 7f9e7be00700 1 -- 172.21.15.19:0/3309104084 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e74010430 con 0x7f9e84102410 2024-03-18T15:17:14.218 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:14.203+0000 7f9e83e00700 1 -- 172.21.15.19:0/3309104084 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e841a76e0 con 0x7f9e84102410 2024-03-18T15:17:14.218 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:14.204+0000 7f9e83e00700 1 -- 172.21.15.19:0/3309104084 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e84068f90 con 0x7f9e84102410 2024-03-18T15:17:14.218 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:14.206+0000 7f9e7be00700 1 -- 172.21.15.19:0/3309104084 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f9e74003f70 con 0x7f9e84102410 2024-03-18T15:17:14.218 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:14.207+0000 7f9e7be00700 1 --2- 172.21.15.19:0/3309104084 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f9e680a29a0 0x7f9e680a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:14.218 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:14.207+0000 7f9e7be00700 1 -- 172.21.15.19:0/3309104084 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f9e740cf270 con 0x7f9e84102410 2024-03-18T15:17:14.218 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:14.208+0000 7f9e7a000700 1 --2- 172.21.15.19:0/3309104084 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f9e680a29a0 0x7f9e680a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:14.218 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:14.208+0000 7f9e7a000700 1 --2- 172.21.15.19:0/3309104084 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f9e680a29a0 0x7f9e680a4e60 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f9e6c000c00 tx=0x7f9e6c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:17:14.218 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:14.211+0000 7f9e7be00700 1 -- 172.21.15.19:0/3309104084 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9e7409bdf0 con 0x7f9e84102410 2024-03-18T15:17:14.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:17:14 smithi104 ceph-mon[36070]: pgmap v380: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T15:17:14.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:17:14 smithi019 ceph-mon[31522]: pgmap v380: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T15:17:14.470 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:14.470+0000 7f9e83e00700 1 -- 172.21.15.19:0/3309104084 --> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f9e84075510 con 0x7f9e680a29a0 2024-03-18T15:17:14.501 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:14.501+0000 7f9e7be00700 1 -- 172.21.15.19:0/3309104084 <== mgr.14152 v2:172.21.15.19:6800/2017714236 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340718 (secure 0 0 0) 0x7f9e84075510 con 0x7f9e680a29a0 2024-03-18T15:17:14.502 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-18T15:17:14.511 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:17:14 smithi033 ceph-mon[36648]: pgmap v380: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T15:17:14.511 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:14.509+0000 7f9e83e00700 1 -- 172.21.15.19:0/3309104084 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f9e680a29a0 msgr2=0x7f9e680a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:14.511 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:14.509+0000 7f9e83e00700 1 --2- 172.21.15.19:0/3309104084 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f9e680a29a0 0x7f9e680a4e60 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f9e6c000c00 tx=0x7f9e6c005fb0 comp rx=0 tx=0).stop 2024-03-18T15:17:14.511 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:14.509+0000 7f9e83e00700 1 -- 172.21.15.19:0/3309104084 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9e84102410 msgr2=0x7f9e841a0810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:14.511 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:14.509+0000 7f9e83e00700 1 --2- 172.21.15.19:0/3309104084 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9e84102410 0x7f9e841a0810 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f9e7400eb30 tx=0x7f9e7400ee40 comp rx=0 tx=0).stop 2024-03-18T15:17:14.511 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:14.509+0000 7f9e83e00700 1 -- 172.21.15.19:0/3309104084 shutdown_connections 2024-03-18T15:17:14.511 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:14.509+0000 7f9e83e00700 1 --2- 172.21.15.19:0/3309104084 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f9e680a29a0 0x7f9e680a4e60 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:14.511 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:14.509+0000 7f9e83e00700 1 --2- 172.21.15.19:0/3309104084 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9e841074f0 0x7f9e8419a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:14.512 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:14.509+0000 7f9e83e00700 1 --2- 172.21.15.19:0/3309104084 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9e84102410 0x7f9e841a0810 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:14.512 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:14.509+0000 7f9e83e00700 1 --2- 172.21.15.19:0/3309104084 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9e84102dc0 0x7f9e841a0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:14.512 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:14.509+0000 7f9e83e00700 1 -- 172.21.15.19:0/3309104084 >> 172.21.15.19:0/3309104084 conn(0x7f9e840fdca0 msgr2=0x7f9e8410d8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:17:14.512 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:14.509+0000 7f9e83e00700 1 -- 172.21.15.19:0/3309104084 shutdown_connections 2024-03-18T15:17:14.512 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:14.509+0000 7f9e83e00700 1 -- 172.21.15.19:0/3309104084 wait complete. 2024-03-18T15:17:14.512 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-03-18T15:17:15.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:17:15 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.8", "id": [5, 8]}]: dispatch 2024-03-18T15:17:15.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:17:15 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 10]}]: dispatch 2024-03-18T15:17:15.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:17:15 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 8]}]: dispatch 2024-03-18T15:17:15.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:17:15 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]: dispatch 2024-03-18T15:17:15.311 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":381,"stamp":"2024-03-18T15:17:14.353863+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":660380,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1023,"num_read_kb":639,"num_write":427,"num_write_kb":3080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":1181358,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":389,"ondisk_log_size":389,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":357368,"kb_used_data":12296,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124584456,"statfs":{"total":1151940427776,"available":1151574482944,"internally_reserved":0,"allocated":12591104,"data_stored":7966946,"data_compressed":359394,"data_compressed_allocated":491520,"data_compressed_original":2482176,"omap_allocated":19143,"internal_metadata":353154361},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":110,"num_read_kb":55,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005767"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.799263+0000","last_change":"2024-03-18T15:15:10.682660+0000","last_active":"2024-03-18T15:16:15.799263+0000","last_peered":"2024-03-18T15:16:15.799263+0000","last_clean":"2024-03-18T15:16:15.799263+0000","last_became_active":"2024-03-18T15:15:10.682433+0000","last_became_peered":"2024-03-18T15:15:10.682433+0000","last_unstale":"2024-03-18T15:16:15.799263+0000","last_undegraded":"2024-03-18T15:16:15.799263+0000","last_fullsized":"2024-03-18T15:16:15.799263+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:48:55.154555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.816543+0000","last_change":"2024-03-18T15:15:10.678100+0000","last_active":"2024-03-18T15:16:15.816543+0000","last_peered":"2024-03-18T15:16:15.816543+0000","last_clean":"2024-03-18T15:16:15.816543+0000","last_became_active":"2024-03-18T15:15:10.677543+0000","last_became_peered":"2024-03-18T15:15:10.677543+0000","last_unstale":"2024-03-18T15:16:15.816543+0000","last_undegraded":"2024-03-18T15:16:15.816543+0000","last_fullsized":"2024-03-18T15:16:15.816543+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:57:50.087837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.815393+0000","last_change":"2024-03-18T15:15:10.674672+0000","last_active":"2024-03-18T15:16:15.815393+0000","last_peered":"2024-03-18T15:16:15.815393+0000","last_clean":"2024-03-18T15:16:15.815393+0000","last_became_active":"2024-03-18T15:15:10.674346+0000","last_became_peered":"2024-03-18T15:15:10.674346+0000","last_unstale":"2024-03-18T15:16:15.815393+0000","last_undegraded":"2024-03-18T15:16:15.815393+0000","last_fullsized":"2024-03-18T15:16:15.815393+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:21:40.054636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.799329+0000","last_change":"2024-03-18T15:15:10.660022+0000","last_active":"2024-03-18T15:16:15.799329+0000","last_peered":"2024-03-18T15:16:15.799329+0000","last_clean":"2024-03-18T15:16:15.799329+0000","last_became_active":"2024-03-18T15:15:10.659622+0000","last_became_peered":"2024-03-18T15:15:10.659622+0000","last_unstale":"2024-03-18T15:16:15.799329+0000","last_undegraded":"2024-03-18T15:16:15.799329+0000","last_fullsized":"2024-03-18T15:16:15.799329+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:55:33.777539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.800804+0000","last_change":"2024-03-18T15:15:10.681499+0000","last_active":"2024-03-18T15:16:15.800804+0000","last_peered":"2024-03-18T15:16:15.800804+0000","last_clean":"2024-03-18T15:16:15.800804+0000","last_became_active":"2024-03-18T15:15:10.681034+0000","last_became_peered":"2024-03-18T15:15:10.681034+0000","last_unstale":"2024-03-18T15:16:15.800804+0000","last_undegraded":"2024-03-18T15:16:15.800804+0000","last_fullsized":"2024-03-18T15:16:15.800804+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:16:43.171924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.805600+0000","last_change":"2024-03-18T15:16:15.805600+0000","last_active":"2024-03-18T15:16:15.805600+0000","last_peered":"2024-03-18T15:16:15.805600+0000","last_clean":"2024-03-18T15:16:15.805600+0000","last_became_active":"2024-03-18T15:16:15.805222+0000","last_became_peered":"2024-03-18T15:16:15.805222+0000","last_unstale":"2024-03-18T15:16:15.805600+0000","last_undegraded":"2024-03-18T15:16:15.805600+0000","last_fullsized":"2024-03-18T15:16:15.805600+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:25:41.772461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,7],"acting":[10,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.802571+0000","last_change":"2024-03-18T15:15:10.669433+0000","last_active":"2024-03-18T15:16:15.802571+0000","last_peered":"2024-03-18T15:16:15.802571+0000","last_clean":"2024-03-18T15:16:15.802571+0000","last_became_active":"2024-03-18T15:15:10.669099+0000","last_became_peered":"2024-03-18T15:15:10.669099+0000","last_unstale":"2024-03-18T15:16:15.802571+0000","last_undegraded":"2024-03-18T15:16:15.802571+0000","last_fullsized":"2024-03-18T15:16:15.802571+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:37:20.697605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.930793+0000","last_change":"2024-03-18T15:15:10.681679+0000","last_active":"2024-03-18T15:16:15.930793+0000","last_peered":"2024-03-18T15:16:15.930793+0000","last_clean":"2024-03-18T15:16:15.930793+0000","last_became_active":"2024-03-18T15:15:10.681079+0000","last_became_peered":"2024-03-18T15:15:10.681079+0000","last_unstale":"2024-03-18T15:16:15.930793+0000","last_undegraded":"2024-03-18T15:16:15.930793+0000","last_fullsized":"2024-03-18T15:16:15.930793+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:24:39.639451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.805022+0000","last_change":"2024-03-18T15:16:15.805022+0000","last_active":"2024-03-18T15:16:15.805022+0000","last_peered":"2024-03-18T15:16:15.805022+0000","last_clean":"2024-03-18T15:16:15.805022+0000","last_became_active":"2024-03-18T15:16:15.804623+0000","last_became_peered":"2024-03-18T15:16:15.804623+0000","last_unstale":"2024-03-18T15:16:15.805022+0000","last_undegraded":"2024-03-18T15:16:15.805022+0000","last_fullsized":"2024-03-18T15:16:15.805022+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:06:50.091834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,11],"acting":[1,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.802351+0000","last_change":"2024-03-18T15:15:10.657799+0000","last_active":"2024-03-18T15:16:15.802351+0000","last_peered":"2024-03-18T15:16:15.802351+0000","last_clean":"2024-03-18T15:16:15.802351+0000","last_became_active":"2024-03-18T15:15:10.657468+0000","last_became_peered":"2024-03-18T15:15:10.657468+0000","last_unstale":"2024-03-18T15:16:15.802351+0000","last_undegraded":"2024-03-18T15:16:15.802351+0000","last_fullsized":"2024-03-18T15:16:15.802351+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:22:46.025516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.804320+0000","last_change":"2024-03-18T15:15:10.665040+0000","last_active":"2024-03-18T15:16:15.804320+0000","last_peered":"2024-03-18T15:16:15.804320+0000","last_clean":"2024-03-18T15:16:15.804320+0000","last_became_active":"2024-03-18T15:15:10.663929+0000","last_became_peered":"2024-03-18T15:15:10.663929+0000","last_unstale":"2024-03-18T15:16:15.804320+0000","last_undegraded":"2024-03-18T15:16:15.804320+0000","last_fullsized":"2024-03-18T15:16:15.804320+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:05:29.954918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.799234+0000","last_change":"2024-03-18T15:15:10.662301+0000","last_active":"2024-03-18T15:16:15.799234+0000","last_peered":"2024-03-18T15:16:15.799234+0000","last_clean":"2024-03-18T15:16:15.799234+0000","last_became_active":"2024-03-18T15:15:10.661956+0000","last_became_peered":"2024-03-18T15:15:10.661956+0000","last_unstale":"2024-03-18T15:16:15.799234+0000","last_undegraded":"2024-03-18T15:16:15.799234+0000","last_fullsized":"2024-03-18T15:16:15.799234+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:30:36.699858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:17:14.312674+0000","last_change":"2024-03-18T15:15:10.650795+0000","last_active":"2024-03-18T15:17:14.312674+0000","last_peered":"2024-03-18T15:17:14.312674+0000","last_clean":"2024-03-18T15:17:14.312674+0000","last_became_active":"2024-03-18T15:15:10.649499+0000","last_became_peered":"2024-03-18T15:15:10.649499+0000","last_unstale":"2024-03-18T15:17:14.312674+0000","last_undegraded":"2024-03-18T15:17:14.312674+0000","last_fullsized":"2024-03-18T15:17:14.312674+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:30:19.094619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.812341+0000","last_change":"2024-03-18T15:15:10.677030+0000","last_active":"2024-03-18T15:16:15.812341+0000","last_peered":"2024-03-18T15:16:15.812341+0000","last_clean":"2024-03-18T15:16:15.812341+0000","last_became_active":"2024-03-18T15:15:10.676680+0000","last_became_peered":"2024-03-18T15:15:10.676680+0000","last_unstale":"2024-03-18T15:16:15.812341+0000","last_undegraded":"2024-03-18T15:16:15.812341+0000","last_fullsized":"2024-03-18T15:16:15.812341+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:32:16.382003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.811993+0000","last_change":"2024-03-18T15:15:10.677512+0000","last_active":"2024-03-18T15:16:15.811993+0000","last_peered":"2024-03-18T15:16:15.811993+0000","last_clean":"2024-03-18T15:16:15.811993+0000","last_became_active":"2024-03-18T15:15:10.670557+0000","last_became_peered":"2024-03-18T15:15:10.670557+0000","last_unstale":"2024-03-18T15:16:15.811993+0000","last_undegraded":"2024-03-18T15:16:15.811993+0000","last_fullsized":"2024-03-18T15:16:15.811993+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:06:58.330728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.800896+0000","last_change":"2024-03-18T15:15:10.675859+0000","last_active":"2024-03-18T15:16:15.800896+0000","last_peered":"2024-03-18T15:16:15.800896+0000","last_clean":"2024-03-18T15:16:15.800896+0000","last_became_active":"2024-03-18T15:15:10.675458+0000","last_became_peered":"2024-03-18T15:15:10.675458+0000","last_unstale":"2024-03-18T15:16:15.800896+0000","last_undegraded":"2024-03-18T15:16:15.800896+0000","last_fullsized":"2024-03-18T15:16:15.800896+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:07:43.306884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.799196+0000","last_change":"2024-03-18T15:15:10.677093+0000","last_active":"2024-03-18T15:16:15.799196+0000","last_peered":"2024-03-18T15:16:15.799196+0000","last_clean":"2024-03-18T15:16:15.799196+0000","last_became_active":"2024-03-18T15:15:10.676878+0000","last_became_peered":"2024-03-18T15:15:10.676878+0000","last_unstale":"2024-03-18T15:16:15.799196+0000","last_undegraded":"2024-03-18T15:16:15.799196+0000","last_fullsized":"2024-03-18T15:16:15.799196+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:18:23.517133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.803046+0000","last_change":"2024-03-18T15:15:10.674793+0000","last_active":"2024-03-18T15:16:15.803046+0000","last_peered":"2024-03-18T15:16:15.803046+0000","last_clean":"2024-03-18T15:16:15.803046+0000","last_became_active":"2024-03-18T15:15:10.674433+0000","last_became_peered":"2024-03-18T15:15:10.674433+0000","last_unstale":"2024-03-18T15:16:15.803046+0000","last_undegraded":"2024-03-18T15:16:15.803046+0000","last_fullsized":"2024-03-18T15:16:15.803046+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:17:24.774598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.797173+0000","last_change":"2024-03-18T15:15:10.670757+0000","last_active":"2024-03-18T15:16:15.797173+0000","last_peered":"2024-03-18T15:16:15.797173+0000","last_clean":"2024-03-18T15:16:15.797173+0000","last_became_active":"2024-03-18T15:15:10.670483+0000","last_became_peered":"2024-03-18T15:15:10.670483+0000","last_unstale":"2024-03-18T15:16:15.797173+0000","last_undegraded":"2024-03-18T15:16:15.797173+0000","last_fullsized":"2024-03-18T15:16:15.797173+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:59:20.070597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.796937+0000","last_change":"2024-03-18T15:15:15.115252+0000","last_active":"2024-03-18T15:16:15.796937+0000","last_peered":"2024-03-18T15:16:15.796937+0000","last_clean":"2024-03-18T15:16:15.796937+0000","last_became_active":"2024-03-18T15:15:15.114812+0000","last_became_peered":"2024-03-18T15:15:15.114812+0000","last_unstale":"2024-03-18T15:16:15.796937+0000","last_undegraded":"2024-03-18T15:16:15.796937+0000","last_fullsized":"2024-03-18T15:16:15.796937+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:04:15.657483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'13","reported_seq":125,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:17:09.171511+0000","last_change":"2024-03-18T15:15:10.092873+0000","last_active":"2024-03-18T15:17:09.171511+0000","last_peered":"2024-03-18T15:17:09.171511+0000","last_clean":"2024-03-18T15:17:09.171511+0000","last_became_active":"2024-03-18T15:15:10.092456+0000","last_became_peered":"2024-03-18T15:15:10.092456+0000","last_unstale":"2024-03-18T15:17:09.171511+0000","last_undegraded":"2024-03-18T15:17:09.171511+0000","last_fullsized":"2024-03-18T15:17:09.171511+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:01:53.596264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":85,"num_read_kb":85,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.935386+0000","last_change":"2024-03-18T15:15:10.682119+0000","last_active":"2024-03-18T15:16:15.935386+0000","last_peered":"2024-03-18T15:16:15.935386+0000","last_clean":"2024-03-18T15:16:15.935386+0000","last_became_active":"2024-03-18T15:15:10.680489+0000","last_became_peered":"2024-03-18T15:15:10.680489+0000","last_unstale":"2024-03-18T15:16:15.935386+0000","last_undegraded":"2024-03-18T15:16:15.935386+0000","last_fullsized":"2024-03-18T15:16:15.935386+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:52:13.477107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.798715+0000","last_change":"2024-03-18T15:15:15.114645+0000","last_active":"2024-03-18T15:16:15.798715+0000","last_peered":"2024-03-18T15:16:15.798715+0000","last_clean":"2024-03-18T15:16:15.798715+0000","last_became_active":"2024-03-18T15:15:15.114245+0000","last_became_peered":"2024-03-18T15:15:15.114245+0000","last_unstale":"2024-03-18T15:16:15.798715+0000","last_undegraded":"2024-03-18T15:16:15.798715+0000","last_fullsized":"2024-03-18T15:16:15.798715+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:08:31.429216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.935705+0000","last_change":"2024-03-18T15:15:10.100265+0000","last_active":"2024-03-18T15:16:15.935705+0000","last_peered":"2024-03-18T15:16:15.935705+0000","last_clean":"2024-03-18T15:16:15.935705+0000","last_became_active":"2024-03-18T15:15:10.099861+0000","last_became_peered":"2024-03-18T15:15:10.099861+0000","last_unstale":"2024-03-18T15:16:15.935705+0000","last_undegraded":"2024-03-18T15:16:15.935705+0000","last_fullsized":"2024-03-18T15:16:15.935705+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:52:47.863313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.817547+0000","last_change":"2024-03-18T15:15:10.682009+0000","last_active":"2024-03-18T15:16:15.817547+0000","last_peered":"2024-03-18T15:16:15.817547+0000","last_clean":"2024-03-18T15:16:15.817547+0000","last_became_active":"2024-03-18T15:15:10.680026+0000","last_became_peered":"2024-03-18T15:15:10.680026+0000","last_unstale":"2024-03-18T15:16:15.817547+0000","last_undegraded":"2024-03-18T15:16:15.817547+0000","last_fullsized":"2024-03-18T15:16:15.817547+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:00:27.869376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.799082+0000","last_change":"2024-03-18T15:15:15.124914+0000","last_active":"2024-03-18T15:16:15.799082+0000","last_peered":"2024-03-18T15:16:15.799082+0000","last_clean":"2024-03-18T15:16:15.799082+0000","last_became_active":"2024-03-18T15:15:15.124477+0000","last_became_peered":"2024-03-18T15:15:15.124477+0000","last_unstale":"2024-03-18T15:16:15.799082+0000","last_undegraded":"2024-03-18T15:16:15.799082+0000","last_fullsized":"2024-03-18T15:16:15.799082+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:58:40.596613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'12","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.816450+0000","last_change":"2024-03-18T15:16:15.815550+0000","last_active":"2024-03-18T15:16:15.816450+0000","last_peered":"2024-03-18T15:16:15.816450+0000","last_clean":"2024-03-18T15:16:15.816450+0000","last_became_active":"2024-03-18T15:16:15.804910+0000","last_became_peered":"2024-03-18T15:16:15.804910+0000","last_unstale":"2024-03-18T15:16:15.816450+0000","last_undegraded":"2024-03-18T15:16:15.816450+0000","last_fullsized":"2024-03-18T15:16:15.816450+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:35:51.813890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.802816+0000","last_change":"2024-03-18T15:15:10.669514+0000","last_active":"2024-03-18T15:16:15.802816+0000","last_peered":"2024-03-18T15:16:15.802816+0000","last_clean":"2024-03-18T15:16:15.802816+0000","last_became_active":"2024-03-18T15:15:10.669154+0000","last_became_peered":"2024-03-18T15:15:10.669154+0000","last_unstale":"2024-03-18T15:16:15.802816+0000","last_undegraded":"2024-03-18T15:16:15.802816+0000","last_fullsized":"2024-03-18T15:16:15.802816+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:10:49.068869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.798747+0000","last_change":"2024-03-18T15:15:15.112079+0000","last_active":"2024-03-18T15:16:15.798747+0000","last_peered":"2024-03-18T15:16:15.798747+0000","last_clean":"2024-03-18T15:16:15.798747+0000","last_became_active":"2024-03-18T15:15:15.111652+0000","last_became_peered":"2024-03-18T15:15:15.111652+0000","last_unstale":"2024-03-18T15:16:15.798747+0000","last_undegraded":"2024-03-18T15:16:15.798747+0000","last_fullsized":"2024-03-18T15:16:15.798747+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:15:55.542490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"89'5","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.802029+0000","last_change":"2024-03-18T15:15:10.077975+0000","last_active":"2024-03-18T15:16:15.802029+0000","last_peered":"2024-03-18T15:16:15.802029+0000","last_clean":"2024-03-18T15:16:15.802029+0000","last_became_active":"2024-03-18T15:15:10.077583+0000","last_became_peered":"2024-03-18T15:15:10.077583+0000","last_unstale":"2024-03-18T15:16:15.802029+0000","last_undegraded":"2024-03-18T15:16:15.802029+0000","last_fullsized":"2024-03-18T15:16:15.802029+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:41:43.741623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.801416+0000","last_change":"2024-03-18T15:15:10.665107+0000","last_active":"2024-03-18T15:16:15.801416+0000","last_peered":"2024-03-18T15:16:15.801416+0000","last_clean":"2024-03-18T15:16:15.801416+0000","last_became_active":"2024-03-18T15:15:10.664786+0000","last_became_peered":"2024-03-18T15:15:10.664786+0000","last_unstale":"2024-03-18T15:16:15.801416+0000","last_undegraded":"2024-03-18T15:16:15.801416+0000","last_fullsized":"2024-03-18T15:16:15.801416+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:58:40.439564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.797077+0000","last_change":"2024-03-18T15:15:15.100480+0000","last_active":"2024-03-18T15:16:15.797077+0000","last_peered":"2024-03-18T15:16:15.797077+0000","last_clean":"2024-03-18T15:16:15.797077+0000","last_became_active":"2024-03-18T15:15:15.100015+0000","last_became_peered":"2024-03-18T15:15:15.100015+0000","last_unstale":"2024-03-18T15:16:15.797077+0000","last_undegraded":"2024-03-18T15:16:15.797077+0000","last_fullsized":"2024-03-18T15:16:15.797077+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:50:08.723714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'5","reported_seq":133,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:17:12.906093+0000","last_change":"2024-03-18T15:16:15.818058+0000","last_active":"2024-03-18T15:17:12.906093+0000","last_peered":"2024-03-18T15:17:12.906093+0000","last_clean":"2024-03-18T15:17:12.906093+0000","last_became_active":"2024-03-18T15:16:15.805096+0000","last_became_peered":"2024-03-18T15:16:15.805096+0000","last_unstale":"2024-03-18T15:17:12.906093+0000","last_undegraded":"2024-03-18T15:17:12.906093+0000","last_fullsized":"2024-03-18T15:17:12.906093+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:51:48.938314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":89,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":622,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.801377+0000","last_change":"2024-03-18T15:15:10.667803+0000","last_active":"2024-03-18T15:16:15.801377+0000","last_peered":"2024-03-18T15:16:15.801377+0000","last_clean":"2024-03-18T15:16:15.801377+0000","last_became_active":"2024-03-18T15:15:10.667478+0000","last_became_peered":"2024-03-18T15:15:10.667478+0000","last_unstale":"2024-03-18T15:16:15.801377+0000","last_undegraded":"2024-03-18T15:16:15.801377+0000","last_fullsized":"2024-03-18T15:16:15.801377+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:22:48.340642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.797537+0000","last_change":"2024-03-18T15:15:15.116404+0000","last_active":"2024-03-18T15:16:15.797537+0000","last_peered":"2024-03-18T15:16:15.797537+0000","last_clean":"2024-03-18T15:16:15.797537+0000","last_became_active":"2024-03-18T15:15:15.115952+0000","last_became_peered":"2024-03-18T15:15:15.115952+0000","last_unstale":"2024-03-18T15:16:15.797537+0000","last_undegraded":"2024-03-18T15:16:15.797537+0000","last_fullsized":"2024-03-18T15:16:15.797537+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:38:44.283538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'7","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.802493+0000","last_change":"2024-03-18T15:15:10.080879+0000","last_active":"2024-03-18T15:16:15.802493+0000","last_peered":"2024-03-18T15:16:15.802493+0000","last_clean":"2024-03-18T15:16:15.802493+0000","last_became_active":"2024-03-18T15:15:10.080478+0000","last_became_peered":"2024-03-18T15:15:10.080478+0000","last_unstale":"2024-03-18T15:16:15.802493+0000","last_undegraded":"2024-03-18T15:16:15.802493+0000","last_fullsized":"2024-03-18T15:16:15.802493+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:28:36.511689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.815729+0000","last_change":"2024-03-18T15:16:15.815729+0000","last_active":"2024-03-18T15:16:15.815729+0000","last_peered":"2024-03-18T15:16:15.815729+0000","last_clean":"2024-03-18T15:16:15.815729+0000","last_became_active":"2024-03-18T15:16:15.815302+0000","last_became_peered":"2024-03-18T15:16:15.815302+0000","last_unstale":"2024-03-18T15:16:15.815729+0000","last_undegraded":"2024-03-18T15:16:15.815729+0000","last_fullsized":"2024-03-18T15:16:15.815729+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:56:37.560590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.799096+0000","last_change":"2024-03-18T15:15:15.111563+0000","last_active":"2024-03-18T15:16:15.799096+0000","last_peered":"2024-03-18T15:16:15.799096+0000","last_clean":"2024-03-18T15:16:15.799096+0000","last_became_active":"2024-03-18T15:15:15.111169+0000","last_became_peered":"2024-03-18T15:15:15.111169+0000","last_unstale":"2024-03-18T15:16:15.799096+0000","last_undegraded":"2024-03-18T15:16:15.799096+0000","last_fullsized":"2024-03-18T15:16:15.799096+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:44:05.491883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.812438+0000","last_change":"2024-03-18T15:15:10.091753+0000","last_active":"2024-03-18T15:16:15.812438+0000","last_peered":"2024-03-18T15:16:15.812438+0000","last_clean":"2024-03-18T15:16:15.812438+0000","last_became_active":"2024-03-18T15:15:10.091393+0000","last_became_peered":"2024-03-18T15:15:10.091393+0000","last_unstale":"2024-03-18T15:16:15.812438+0000","last_undegraded":"2024-03-18T15:16:15.812438+0000","last_fullsized":"2024-03-18T15:16:15.812438+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:24:28.068621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.931269+0000","last_change":"2024-03-18T15:15:10.672785+0000","last_active":"2024-03-18T15:16:15.931269+0000","last_peered":"2024-03-18T15:16:15.931269+0000","last_clean":"2024-03-18T15:16:15.931269+0000","last_became_active":"2024-03-18T15:15:10.672143+0000","last_became_peered":"2024-03-18T15:15:10.672143+0000","last_unstale":"2024-03-18T15:16:15.931269+0000","last_undegraded":"2024-03-18T15:16:15.931269+0000","last_fullsized":"2024-03-18T15:16:15.931269+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:26:40.034989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.816327+0000","last_change":"2024-03-18T15:15:15.114790+0000","last_active":"2024-03-18T15:16:15.816327+0000","last_peered":"2024-03-18T15:16:15.816327+0000","last_clean":"2024-03-18T15:16:15.816327+0000","last_became_active":"2024-03-18T15:15:15.112646+0000","last_became_peered":"2024-03-18T15:15:15.112646+0000","last_unstale":"2024-03-18T15:16:15.816327+0000","last_undegraded":"2024-03-18T15:16:15.816327+0000","last_fullsized":"2024-03-18T15:16:15.816327+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:59:56.686888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'3","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.799677+0000","last_change":"2024-03-18T15:15:10.082566+0000","last_active":"2024-03-18T15:16:15.799677+0000","last_peered":"2024-03-18T15:16:15.799677+0000","last_clean":"2024-03-18T15:16:15.799677+0000","last_became_active":"2024-03-18T15:15:10.082256+0000","last_became_peered":"2024-03-18T15:15:10.082256+0000","last_unstale":"2024-03-18T15:16:15.799677+0000","last_undegraded":"2024-03-18T15:16:15.799677+0000","last_fullsized":"2024-03-18T15:16:15.799677+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:04:47.361932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.801404+0000","last_change":"2024-03-18T15:15:10.683148+0000","last_active":"2024-03-18T15:16:15.801404+0000","last_peered":"2024-03-18T15:16:15.801404+0000","last_clean":"2024-03-18T15:16:15.801404+0000","last_became_active":"2024-03-18T15:15:10.682671+0000","last_became_peered":"2024-03-18T15:15:10.682671+0000","last_unstale":"2024-03-18T15:16:15.801404+0000","last_undegraded":"2024-03-18T15:16:15.801404+0000","last_fullsized":"2024-03-18T15:16:15.801404+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:52:15.763400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.932162+0000","last_change":"2024-03-18T15:15:15.115995+0000","last_active":"2024-03-18T15:16:15.932162+0000","last_peered":"2024-03-18T15:16:15.932162+0000","last_clean":"2024-03-18T15:16:15.932162+0000","last_became_active":"2024-03-18T15:15:15.115560+0000","last_became_peered":"2024-03-18T15:15:15.115560+0000","last_unstale":"2024-03-18T15:16:15.932162+0000","last_undegraded":"2024-03-18T15:16:15.932162+0000","last_fullsized":"2024-03-18T15:16:15.932162+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:20:36.765286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'4","reported_seq":201,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:17:11.080237+0000","last_change":"2024-03-18T15:15:10.095122+0000","last_active":"2024-03-18T15:17:11.080237+0000","last_peered":"2024-03-18T15:17:11.080237+0000","last_clean":"2024-03-18T15:17:11.080237+0000","last_became_active":"2024-03-18T15:15:10.094797+0000","last_became_peered":"2024-03-18T15:15:10.094797+0000","last_unstale":"2024-03-18T15:17:11.080237+0000","last_undegraded":"2024-03-18T15:17:11.080237+0000","last_fullsized":"2024-03-18T15:17:11.080237+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:16:15.124369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":170,"num_read_kb":170,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.802418+0000","last_change":"2024-03-18T15:15:10.673297+0000","last_active":"2024-03-18T15:16:15.802418+0000","last_peered":"2024-03-18T15:16:15.802418+0000","last_clean":"2024-03-18T15:16:15.802418+0000","last_became_active":"2024-03-18T15:15:10.672890+0000","last_became_peered":"2024-03-18T15:15:10.672890+0000","last_unstale":"2024-03-18T15:16:15.802418+0000","last_undegraded":"2024-03-18T15:16:15.802418+0000","last_fullsized":"2024-03-18T15:16:15.802418+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:10:45.958639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.800316+0000","last_change":"2024-03-18T15:15:15.123200+0000","last_active":"2024-03-18T15:16:15.800316+0000","last_peered":"2024-03-18T15:16:15.800316+0000","last_clean":"2024-03-18T15:16:15.800316+0000","last_became_active":"2024-03-18T15:15:15.114554+0000","last_became_peered":"2024-03-18T15:15:15.114554+0000","last_unstale":"2024-03-18T15:16:15.800316+0000","last_undegraded":"2024-03-18T15:16:15.800316+0000","last_fullsized":"2024-03-18T15:16:15.800316+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:35:25.380377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'4","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.803775+0000","last_change":"2024-03-18T15:15:10.085330+0000","last_active":"2024-03-18T15:16:15.803775+0000","last_peered":"2024-03-18T15:16:15.803775+0000","last_clean":"2024-03-18T15:16:15.803775+0000","last_became_active":"2024-03-18T15:15:10.083179+0000","last_became_peered":"2024-03-18T15:15:10.083179+0000","last_unstale":"2024-03-18T15:16:15.803775+0000","last_undegraded":"2024-03-18T15:16:15.803775+0000","last_fullsized":"2024-03-18T15:16:15.803775+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:56:33.843757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.817356+0000","last_change":"2024-03-18T15:15:10.679397+0000","last_active":"2024-03-18T15:16:15.817356+0000","last_peered":"2024-03-18T15:16:15.817356+0000","last_clean":"2024-03-18T15:16:15.817356+0000","last_became_active":"2024-03-18T15:15:10.679006+0000","last_became_peered":"2024-03-18T15:15:10.679006+0000","last_unstale":"2024-03-18T15:16:15.817356+0000","last_undegraded":"2024-03-18T15:16:15.817356+0000","last_fullsized":"2024-03-18T15:16:15.817356+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:38:58.251600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.799450+0000","last_change":"2024-03-18T15:15:15.124936+0000","last_active":"2024-03-18T15:16:15.799450+0000","last_peered":"2024-03-18T15:16:15.799450+0000","last_clean":"2024-03-18T15:16:15.799450+0000","last_became_active":"2024-03-18T15:15:15.124501+0000","last_became_peered":"2024-03-18T15:15:15.124501+0000","last_unstale":"2024-03-18T15:16:15.799450+0000","last_undegraded":"2024-03-18T15:16:15.799450+0000","last_fullsized":"2024-03-18T15:16:15.799450+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:31:47.221428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'5","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.801299+0000","last_change":"2024-03-18T15:15:10.098885+0000","last_active":"2024-03-18T15:16:15.801299+0000","last_peered":"2024-03-18T15:16:15.801299+0000","last_clean":"2024-03-18T15:16:15.801299+0000","last_became_active":"2024-03-18T15:15:10.098604+0000","last_became_peered":"2024-03-18T15:15:10.098604+0000","last_unstale":"2024-03-18T15:16:15.801299+0000","last_undegraded":"2024-03-18T15:16:15.801299+0000","last_fullsized":"2024-03-18T15:16:15.801299+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:59:08.716855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.816184+0000","last_change":"2024-03-18T15:16:15.816184+0000","last_active":"2024-03-18T15:16:15.816184+0000","last_peered":"2024-03-18T15:16:15.816184+0000","last_clean":"2024-03-18T15:16:15.816184+0000","last_became_active":"2024-03-18T15:16:15.815726+0000","last_became_peered":"2024-03-18T15:16:15.815726+0000","last_unstale":"2024-03-18T15:16:15.816184+0000","last_undegraded":"2024-03-18T15:16:15.816184+0000","last_fullsized":"2024-03-18T15:16:15.816184+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:14:53.264760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.801409+0000","last_change":"2024-03-18T15:15:15.107352+0000","last_active":"2024-03-18T15:16:15.801409+0000","last_peered":"2024-03-18T15:16:15.801409+0000","last_clean":"2024-03-18T15:16:15.801409+0000","last_became_active":"2024-03-18T15:15:15.106851+0000","last_became_peered":"2024-03-18T15:15:15.106851+0000","last_unstale":"2024-03-18T15:16:15.801409+0000","last_undegraded":"2024-03-18T15:16:15.801409+0000","last_fullsized":"2024-03-18T15:16:15.801409+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:56:35.695188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'35","reported_seq":239,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:17:11.860652+0000","last_change":"2024-03-18T15:15:10.083226+0000","last_active":"2024-03-18T15:17:11.860652+0000","last_peered":"2024-03-18T15:17:11.860652+0000","last_clean":"2024-03-18T15:17:11.860652+0000","last_became_active":"2024-03-18T15:15:10.082829+0000","last_became_peered":"2024-03-18T15:15:10.082829+0000","last_unstale":"2024-03-18T15:17:11.860652+0000","last_undegraded":"2024-03-18T15:17:11.860652+0000","last_fullsized":"2024-03-18T15:17:11.860652+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:30:34.091815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63356,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":177,"num_read_kb":150,"num_write":37,"num_write_kb":77,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.801811+0000","last_change":"2024-03-18T15:15:10.659322+0000","last_active":"2024-03-18T15:16:15.801811+0000","last_peered":"2024-03-18T15:16:15.801811+0000","last_clean":"2024-03-18T15:16:15.801811+0000","last_became_active":"2024-03-18T15:15:10.658997+0000","last_became_peered":"2024-03-18T15:15:10.658997+0000","last_unstale":"2024-03-18T15:16:15.801811+0000","last_undegraded":"2024-03-18T15:16:15.801811+0000","last_fullsized":"2024-03-18T15:16:15.801811+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:04:08.934683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.802402+0000","last_change":"2024-03-18T15:15:15.110658+0000","last_active":"2024-03-18T15:16:15.802402+0000","last_peered":"2024-03-18T15:16:15.802402+0000","last_clean":"2024-03-18T15:16:15.802402+0000","last_became_active":"2024-03-18T15:15:15.110205+0000","last_became_peered":"2024-03-18T15:15:15.110205+0000","last_unstale":"2024-03-18T15:16:15.802402+0000","last_undegraded":"2024-03-18T15:16:15.802402+0000","last_fullsized":"2024-03-18T15:16:15.802402+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:09:10.014743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'9","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.797131+0000","last_change":"2024-03-18T15:15:10.080305+0000","last_active":"2024-03-18T15:16:15.797131+0000","last_peered":"2024-03-18T15:16:15.797131+0000","last_clean":"2024-03-18T15:16:15.797131+0000","last_became_active":"2024-03-18T15:15:10.079990+0000","last_became_peered":"2024-03-18T15:15:10.079990+0000","last_unstale":"2024-03-18T15:16:15.797131+0000","last_undegraded":"2024-03-18T15:16:15.797131+0000","last_fullsized":"2024-03-18T15:16:15.797131+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:10:08.312462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.813884+0000","last_change":"2024-03-18T15:15:10.671178+0000","last_active":"2024-03-18T15:16:15.813884+0000","last_peered":"2024-03-18T15:16:15.813884+0000","last_clean":"2024-03-18T15:16:15.813884+0000","last_became_active":"2024-03-18T15:15:10.670862+0000","last_became_peered":"2024-03-18T15:15:10.670862+0000","last_unstale":"2024-03-18T15:16:15.813884+0000","last_undegraded":"2024-03-18T15:16:15.813884+0000","last_fullsized":"2024-03-18T15:16:15.813884+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:53:34.916473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.798741+0000","last_change":"2024-03-18T15:15:15.107792+0000","last_active":"2024-03-18T15:16:15.798741+0000","last_peered":"2024-03-18T15:16:15.798741+0000","last_clean":"2024-03-18T15:16:15.798741+0000","last_became_active":"2024-03-18T15:15:15.107327+0000","last_became_peered":"2024-03-18T15:15:15.107327+0000","last_unstale":"2024-03-18T15:16:15.798741+0000","last_undegraded":"2024-03-18T15:16:15.798741+0000","last_fullsized":"2024-03-18T15:16:15.798741+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:42:47.837146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'5","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.800846+0000","last_change":"2024-03-18T15:15:10.079128+0000","last_active":"2024-03-18T15:16:15.800846+0000","last_peered":"2024-03-18T15:16:15.800846+0000","last_clean":"2024-03-18T15:16:15.800846+0000","last_became_active":"2024-03-18T15:15:10.078812+0000","last_became_peered":"2024-03-18T15:15:10.078812+0000","last_unstale":"2024-03-18T15:16:15.800846+0000","last_undegraded":"2024-03-18T15:16:15.800846+0000","last_fullsized":"2024-03-18T15:16:15.800846+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:12:11.473273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.803031+0000","last_change":"2024-03-18T15:15:10.668240+0000","last_active":"2024-03-18T15:16:15.803031+0000","last_peered":"2024-03-18T15:16:15.803031+0000","last_clean":"2024-03-18T15:16:15.803031+0000","last_became_active":"2024-03-18T15:15:10.667885+0000","last_became_peered":"2024-03-18T15:15:10.667885+0000","last_unstale":"2024-03-18T15:16:15.803031+0000","last_undegraded":"2024-03-18T15:16:15.803031+0000","last_fullsized":"2024-03-18T15:16:15.803031+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:17:23.998018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.803980+0000","last_change":"2024-03-18T15:15:15.119067+0000","last_active":"2024-03-18T15:16:15.803980+0000","last_peered":"2024-03-18T15:16:15.803980+0000","last_clean":"2024-03-18T15:16:15.803980+0000","last_became_active":"2024-03-18T15:15:15.118566+0000","last_became_peered":"2024-03-18T15:15:15.118566+0000","last_unstale":"2024-03-18T15:16:15.803980+0000","last_undegraded":"2024-03-18T15:16:15.803980+0000","last_fullsized":"2024-03-18T15:16:15.803980+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:21:00.868724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"89'5","reported_seq":118,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:17:09.120129+0000","last_change":"2024-03-18T15:15:10.080791+0000","last_active":"2024-03-18T15:17:09.120129+0000","last_peered":"2024-03-18T15:17:09.120129+0000","last_clean":"2024-03-18T15:17:09.120129+0000","last_became_active":"2024-03-18T15:15:10.080368+0000","last_became_peered":"2024-03-18T15:15:10.080368+0000","last_unstale":"2024-03-18T15:17:09.120129+0000","last_undegraded":"2024-03-18T15:17:09.120129+0000","last_fullsized":"2024-03-18T15:17:09.120129+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:10:55.575331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":87,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.812223+0000","last_change":"2024-03-18T15:16:15.812223+0000","last_active":"2024-03-18T15:16:15.812223+0000","last_peered":"2024-03-18T15:16:15.812223+0000","last_clean":"2024-03-18T15:16:15.812223+0000","last_became_active":"2024-03-18T15:16:15.811768+0000","last_became_peered":"2024-03-18T15:16:15.811768+0000","last_unstale":"2024-03-18T15:16:15.812223+0000","last_undegraded":"2024-03-18T15:16:15.812223+0000","last_fullsized":"2024-03-18T15:16:15.812223+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:32:40.179559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,10],"acting":[7,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.808680+0000","last_change":"2024-03-18T15:15:15.125605+0000","last_active":"2024-03-18T15:16:15.808680+0000","last_peered":"2024-03-18T15:16:15.808680+0000","last_clean":"2024-03-18T15:16:15.808680+0000","last_became_active":"2024-03-18T15:15:15.125041+0000","last_became_peered":"2024-03-18T15:15:15.125041+0000","last_unstale":"2024-03-18T15:16:15.808680+0000","last_undegraded":"2024-03-18T15:16:15.808680+0000","last_fullsized":"2024-03-18T15:16:15.808680+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:27:25.901710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'17","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.800898+0000","last_change":"2024-03-18T15:15:10.079653+0000","last_active":"2024-03-18T15:16:15.800898+0000","last_peered":"2024-03-18T15:16:15.800898+0000","last_clean":"2024-03-18T15:16:15.800898+0000","last_became_active":"2024-03-18T15:15:10.079343+0000","last_became_peered":"2024-03-18T15:15:10.079343+0000","last_unstale":"2024-03-18T15:16:15.800898+0000","last_undegraded":"2024-03-18T15:16:15.800898+0000","last_fullsized":"2024-03-18T15:16:15.800898+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:33:58.746118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.800545+0000","last_change":"2024-03-18T15:15:10.661241+0000","last_active":"2024-03-18T15:16:15.800545+0000","last_peered":"2024-03-18T15:16:15.800545+0000","last_clean":"2024-03-18T15:16:15.800545+0000","last_became_active":"2024-03-18T15:15:10.660821+0000","last_became_peered":"2024-03-18T15:15:10.660821+0000","last_unstale":"2024-03-18T15:16:15.800545+0000","last_undegraded":"2024-03-18T15:16:15.800545+0000","last_fullsized":"2024-03-18T15:16:15.800545+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:13:59.973615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.800441+0000","last_change":"2024-03-18T15:15:15.118314+0000","last_active":"2024-03-18T15:16:15.800441+0000","last_peered":"2024-03-18T15:16:15.800441+0000","last_clean":"2024-03-18T15:16:15.800441+0000","last_became_active":"2024-03-18T15:15:15.118001+0000","last_became_peered":"2024-03-18T15:15:15.118001+0000","last_unstale":"2024-03-18T15:16:15.800441+0000","last_undegraded":"2024-03-18T15:16:15.800441+0000","last_fullsized":"2024-03-18T15:16:15.800441+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:15:58.320283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.802532+0000","last_change":"2024-03-18T15:15:10.077394+0000","last_active":"2024-03-18T15:16:15.802532+0000","last_peered":"2024-03-18T15:16:15.802532+0000","last_clean":"2024-03-18T15:16:15.802532+0000","last_became_active":"2024-03-18T15:15:10.077027+0000","last_became_peered":"2024-03-18T15:15:10.077027+0000","last_unstale":"2024-03-18T15:16:15.802532+0000","last_undegraded":"2024-03-18T15:16:15.802532+0000","last_fullsized":"2024-03-18T15:16:15.802532+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:34:07.524849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.798943+0000","last_change":"2024-03-18T15:15:10.659723+0000","last_active":"2024-03-18T15:16:15.798943+0000","last_peered":"2024-03-18T15:16:15.798943+0000","last_clean":"2024-03-18T15:16:15.798943+0000","last_became_active":"2024-03-18T15:15:10.659388+0000","last_became_peered":"2024-03-18T15:15:10.659388+0000","last_unstale":"2024-03-18T15:16:15.798943+0000","last_undegraded":"2024-03-18T15:16:15.798943+0000","last_fullsized":"2024-03-18T15:16:15.798943+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:21:52.605870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.801279+0000","last_change":"2024-03-18T15:15:15.115459+0000","last_active":"2024-03-18T15:16:15.801279+0000","last_peered":"2024-03-18T15:16:15.801279+0000","last_clean":"2024-03-18T15:16:15.801279+0000","last_became_active":"2024-03-18T15:15:15.115048+0000","last_became_peered":"2024-03-18T15:15:15.115048+0000","last_unstale":"2024-03-18T15:16:15.801279+0000","last_undegraded":"2024-03-18T15:16:15.801279+0000","last_fullsized":"2024-03-18T15:16:15.801279+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:04:23.118904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"89'1","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.814760+0000","last_change":"2024-03-18T15:16:15.814760+0000","last_active":"2024-03-18T15:16:15.814760+0000","last_peered":"2024-03-18T15:16:15.814760+0000","last_clean":"2024-03-18T15:16:15.814760+0000","last_became_active":"2024-03-18T15:16:15.814111+0000","last_became_peered":"2024-03-18T15:16:15.814111+0000","last_unstale":"2024-03-18T15:16:15.814760+0000","last_undegraded":"2024-03-18T15:16:15.814760+0000","last_fullsized":"2024-03-18T15:16:15.814760+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:59:51.241047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.802851+0000","last_change":"2024-03-18T15:15:10.672954+0000","last_active":"2024-03-18T15:16:15.802851+0000","last_peered":"2024-03-18T15:16:15.802851+0000","last_clean":"2024-03-18T15:16:15.802851+0000","last_became_active":"2024-03-18T15:15:10.672415+0000","last_became_peered":"2024-03-18T15:15:10.672415+0000","last_unstale":"2024-03-18T15:16:15.802851+0000","last_undegraded":"2024-03-18T15:16:15.802851+0000","last_fullsized":"2024-03-18T15:16:15.802851+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:47:48.421411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.809642+0000","last_change":"2024-03-18T15:15:15.121334+0000","last_active":"2024-03-18T15:16:15.809642+0000","last_peered":"2024-03-18T15:16:15.809642+0000","last_clean":"2024-03-18T15:16:15.809642+0000","last_became_active":"2024-03-18T15:15:15.119797+0000","last_became_peered":"2024-03-18T15:15:15.119797+0000","last_unstale":"2024-03-18T15:16:15.809642+0000","last_undegraded":"2024-03-18T15:16:15.809642+0000","last_fullsized":"2024-03-18T15:16:15.809642+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:28:39.072485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'5","reported_seq":116,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:17:09.172198+0000","last_change":"2024-03-18T15:15:10.082589+0000","last_active":"2024-03-18T15:17:09.172198+0000","last_peered":"2024-03-18T15:17:09.172198+0000","last_clean":"2024-03-18T15:17:09.172198+0000","last_became_active":"2024-03-18T15:15:10.082190+0000","last_became_peered":"2024-03-18T15:15:10.082190+0000","last_unstale":"2024-03-18T15:17:09.172198+0000","last_undegraded":"2024-03-18T15:17:09.172198+0000","last_fullsized":"2024-03-18T15:17:09.172198+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:55:15.631914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":85,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.801883+0000","last_change":"2024-03-18T15:15:10.672145+0000","last_active":"2024-03-18T15:16:15.801883+0000","last_peered":"2024-03-18T15:16:15.801883+0000","last_clean":"2024-03-18T15:16:15.801883+0000","last_became_active":"2024-03-18T15:15:10.670303+0000","last_became_peered":"2024-03-18T15:15:10.670303+0000","last_unstale":"2024-03-18T15:16:15.801883+0000","last_undegraded":"2024-03-18T15:16:15.801883+0000","last_fullsized":"2024-03-18T15:16:15.801883+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:22:50.528794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.813323+0000","last_change":"2024-03-18T15:15:15.113675+0000","last_active":"2024-03-18T15:16:15.813323+0000","last_peered":"2024-03-18T15:16:15.813323+0000","last_clean":"2024-03-18T15:16:15.813323+0000","last_became_active":"2024-03-18T15:15:15.113067+0000","last_became_peered":"2024-03-18T15:15:15.113067+0000","last_unstale":"2024-03-18T15:16:15.813323+0000","last_undegraded":"2024-03-18T15:16:15.813323+0000","last_fullsized":"2024-03-18T15:16:15.813323+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:37:41.751216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.799919+0000","last_change":"2024-03-18T15:15:10.658032+0000","last_active":"2024-03-18T15:16:15.799919+0000","last_peered":"2024-03-18T15:16:15.799919+0000","last_clean":"2024-03-18T15:16:15.799919+0000","last_became_active":"2024-03-18T15:15:10.657707+0000","last_became_peered":"2024-03-18T15:15:10.657707+0000","last_unstale":"2024-03-18T15:16:15.799919+0000","last_undegraded":"2024-03-18T15:16:15.799919+0000","last_fullsized":"2024-03-18T15:16:15.799919+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:44:24.137349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'5","reported_seq":122,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:17:14.127426+0000","last_change":"2024-03-18T15:15:10.085330+0000","last_active":"2024-03-18T15:17:14.127426+0000","last_peered":"2024-03-18T15:17:14.127426+0000","last_clean":"2024-03-18T15:17:14.127426+0000","last_became_active":"2024-03-18T15:15:10.080303+0000","last_became_peered":"2024-03-18T15:15:10.080303+0000","last_unstale":"2024-03-18T15:17:14.127426+0000","last_undegraded":"2024-03-18T15:17:14.127426+0000","last_fullsized":"2024-03-18T15:17:14.127426+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:26:39.293028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":89,"num_read_kb":89,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.804140+0000","last_change":"2024-03-18T15:15:15.125172+0000","last_active":"2024-03-18T15:16:15.804140+0000","last_peered":"2024-03-18T15:16:15.804140+0000","last_clean":"2024-03-18T15:16:15.804140+0000","last_became_active":"2024-03-18T15:15:15.124591+0000","last_became_peered":"2024-03-18T15:15:15.124591+0000","last_unstale":"2024-03-18T15:16:15.804140+0000","last_undegraded":"2024-03-18T15:16:15.804140+0000","last_fullsized":"2024-03-18T15:16:15.804140+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:50:16.339041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.934902+0000","last_change":"2024-03-18T15:15:10.685343+0000","last_active":"2024-03-18T15:16:15.934902+0000","last_peered":"2024-03-18T15:16:15.934902+0000","last_clean":"2024-03-18T15:16:15.934902+0000","last_became_active":"2024-03-18T15:15:10.684395+0000","last_became_peered":"2024-03-18T15:15:10.684395+0000","last_unstale":"2024-03-18T15:16:15.934902+0000","last_undegraded":"2024-03-18T15:16:15.934902+0000","last_fullsized":"2024-03-18T15:16:15.934902+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:11:26.982047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'6","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.799217+0000","last_change":"2024-03-18T15:15:10.085503+0000","last_active":"2024-03-18T15:16:15.799217+0000","last_peered":"2024-03-18T15:16:15.799217+0000","last_clean":"2024-03-18T15:16:15.799217+0000","last_became_active":"2024-03-18T15:15:10.085125+0000","last_became_peered":"2024-03-18T15:15:10.085125+0000","last_unstale":"2024-03-18T15:16:15.799217+0000","last_undegraded":"2024-03-18T15:16:15.799217+0000","last_fullsized":"2024-03-18T15:16:15.799217+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:17:13.091428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":360,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.802205+0000","last_change":"2024-03-18T15:13:50.952630+0000","last_active":"2024-03-18T15:16:15.802205+0000","last_peered":"2024-03-18T15:16:15.802205+0000","last_clean":"2024-03-18T15:16:15.802205+0000","last_became_active":"2024-03-18T15:13:50.952144+0000","last_became_peered":"2024-03-18T15:13:50.952144+0000","last_unstale":"2024-03-18T15:16:15.802205+0000","last_undegraded":"2024-03-18T15:16:15.802205+0000","last_fullsized":"2024-03-18T15:16:15.802205+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:09:56.845161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:09:56.845161+0000","last_clean_scrub_stamp":"2024-03-18T15:09:56.845161+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:53:27.959711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.800857+0000","last_change":"2024-03-18T15:15:15.115418+0000","last_active":"2024-03-18T15:16:15.800857+0000","last_peered":"2024-03-18T15:16:15.800857+0000","last_clean":"2024-03-18T15:16:15.800857+0000","last_became_active":"2024-03-18T15:15:15.114715+0000","last_became_peered":"2024-03-18T15:15:15.114715+0000","last_unstale":"2024-03-18T15:16:15.800857+0000","last_undegraded":"2024-03-18T15:16:15.800857+0000","last_fullsized":"2024-03-18T15:16:15.800857+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:37:51.553099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:17:14.301713+0000","last_change":"2024-03-18T15:15:10.665492+0000","last_active":"2024-03-18T15:17:14.301713+0000","last_peered":"2024-03-18T15:17:14.301713+0000","last_clean":"2024-03-18T15:17:14.301713+0000","last_became_active":"2024-03-18T15:15:10.665097+0000","last_became_peered":"2024-03-18T15:15:10.665097+0000","last_unstale":"2024-03-18T15:17:14.301713+0000","last_undegraded":"2024-03-18T15:17:14.301713+0000","last_fullsized":"2024-03-18T15:17:14.301713+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:27:06.864459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.802314+0000","last_change":"2024-03-18T15:15:10.085322+0000","last_active":"2024-03-18T15:16:15.802314+0000","last_peered":"2024-03-18T15:16:15.802314+0000","last_clean":"2024-03-18T15:16:15.802314+0000","last_became_active":"2024-03-18T15:15:10.083114+0000","last_became_peered":"2024-03-18T15:15:10.083114+0000","last_unstale":"2024-03-18T15:16:15.802314+0000","last_undegraded":"2024-03-18T15:16:15.802314+0000","last_fullsized":"2024-03-18T15:16:15.802314+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:13:31.878513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.798597+0000","last_change":"2024-03-18T15:15:15.105432+0000","last_active":"2024-03-18T15:16:15.798597+0000","last_peered":"2024-03-18T15:16:15.798597+0000","last_clean":"2024-03-18T15:16:15.798597+0000","last_became_active":"2024-03-18T15:15:15.104929+0000","last_became_peered":"2024-03-18T15:15:15.104929+0000","last_unstale":"2024-03-18T15:16:15.798597+0000","last_undegraded":"2024-03-18T15:16:15.798597+0000","last_fullsized":"2024-03-18T15:16:15.798597+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:50:37.334639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.802150+0000","last_change":"2024-03-18T15:15:10.099722+0000","last_active":"2024-03-18T15:16:15.802150+0000","last_peered":"2024-03-18T15:16:15.802150+0000","last_clean":"2024-03-18T15:16:15.802150+0000","last_became_active":"2024-03-18T15:15:10.099410+0000","last_became_peered":"2024-03-18T15:15:10.099410+0000","last_unstale":"2024-03-18T15:16:15.802150+0000","last_undegraded":"2024-03-18T15:16:15.802150+0000","last_fullsized":"2024-03-18T15:16:15.802150+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:19:24.877473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:17:14.306227+0000","last_change":"2024-03-18T15:15:10.669235+0000","last_active":"2024-03-18T15:17:14.306227+0000","last_peered":"2024-03-18T15:17:14.306227+0000","last_clean":"2024-03-18T15:17:14.306227+0000","last_became_active":"2024-03-18T15:15:10.667369+0000","last_became_peered":"2024-03-18T15:15:10.667369+0000","last_unstale":"2024-03-18T15:17:14.306227+0000","last_undegraded":"2024-03-18T15:17:14.306227+0000","last_fullsized":"2024-03-18T15:17:14.306227+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:36:08.961435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.931053+0000","last_change":"2024-03-18T15:15:15.119228+0000","last_active":"2024-03-18T15:16:15.931053+0000","last_peered":"2024-03-18T15:16:15.931053+0000","last_clean":"2024-03-18T15:16:15.931053+0000","last_became_active":"2024-03-18T15:15:15.115585+0000","last_became_peered":"2024-03-18T15:15:15.115585+0000","last_unstale":"2024-03-18T15:16:15.931053+0000","last_undegraded":"2024-03-18T15:16:15.931053+0000","last_fullsized":"2024-03-18T15:16:15.931053+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:36:43.263907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"89'14","reported_seq":41,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.803147+0000","last_change":"2024-03-18T15:15:10.105259+0000","last_active":"2024-03-18T15:16:15.803147+0000","last_peered":"2024-03-18T15:16:15.803147+0000","last_clean":"2024-03-18T15:16:15.803147+0000","last_became_active":"2024-03-18T15:15:10.104315+0000","last_became_peered":"2024-03-18T15:15:10.104315+0000","last_unstale":"2024-03-18T15:16:15.803147+0000","last_undegraded":"2024-03-18T15:16:15.803147+0000","last_fullsized":"2024-03-18T15:16:15.803147+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:27:05.565134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.930403+0000","last_change":"2024-03-18T15:15:10.673615+0000","last_active":"2024-03-18T15:16:15.930403+0000","last_peered":"2024-03-18T15:16:15.930403+0000","last_clean":"2024-03-18T15:16:15.930403+0000","last_became_active":"2024-03-18T15:15:10.672308+0000","last_became_peered":"2024-03-18T15:15:10.672308+0000","last_unstale":"2024-03-18T15:16:15.930403+0000","last_undegraded":"2024-03-18T15:16:15.930403+0000","last_fullsized":"2024-03-18T15:16:15.930403+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:32:13.889281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.796772+0000","last_change":"2024-03-18T15:15:15.099447+0000","last_active":"2024-03-18T15:16:15.796772+0000","last_peered":"2024-03-18T15:16:15.796772+0000","last_clean":"2024-03-18T15:16:15.796772+0000","last_became_active":"2024-03-18T15:15:15.098857+0000","last_became_peered":"2024-03-18T15:15:15.098857+0000","last_unstale":"2024-03-18T15:16:15.796772+0000","last_undegraded":"2024-03-18T15:16:15.796772+0000","last_fullsized":"2024-03-18T15:16:15.796772+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:51:22.115359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'7","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.799586+0000","last_change":"2024-03-18T15:15:10.084281+0000","last_active":"2024-03-18T15:16:15.799586+0000","last_peered":"2024-03-18T15:16:15.799586+0000","last_clean":"2024-03-18T15:16:15.799586+0000","last_became_active":"2024-03-18T15:15:10.083791+0000","last_became_peered":"2024-03-18T15:15:10.083791+0000","last_unstale":"2024-03-18T15:16:15.799586+0000","last_undegraded":"2024-03-18T15:16:15.799586+0000","last_fullsized":"2024-03-18T15:16:15.799586+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:08:33.066629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.816717+0000","last_change":"2024-03-18T15:15:10.682823+0000","last_active":"2024-03-18T15:16:15.816717+0000","last_peered":"2024-03-18T15:16:15.816717+0000","last_clean":"2024-03-18T15:16:15.816717+0000","last_became_active":"2024-03-18T15:15:10.681887+0000","last_became_peered":"2024-03-18T15:15:10.681887+0000","last_unstale":"2024-03-18T15:16:15.816717+0000","last_undegraded":"2024-03-18T15:16:15.816717+0000","last_fullsized":"2024-03-18T15:16:15.816717+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:33:29.918824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.799079+0000","last_change":"2024-03-18T15:15:15.114952+0000","last_active":"2024-03-18T15:16:15.799079+0000","last_peered":"2024-03-18T15:16:15.799079+0000","last_clean":"2024-03-18T15:16:15.799079+0000","last_became_active":"2024-03-18T15:15:15.114331+0000","last_became_peered":"2024-03-18T15:15:15.114331+0000","last_unstale":"2024-03-18T15:16:15.799079+0000","last_undegraded":"2024-03-18T15:16:15.799079+0000","last_fullsized":"2024-03-18T15:16:15.799079+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:22:08.637152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.801009+0000","last_change":"2024-03-18T15:15:10.080291+0000","last_active":"2024-03-18T15:16:15.801009+0000","last_peered":"2024-03-18T15:16:15.801009+0000","last_clean":"2024-03-18T15:16:15.801009+0000","last_became_active":"2024-03-18T15:15:10.079634+0000","last_became_peered":"2024-03-18T15:15:10.079634+0000","last_unstale":"2024-03-18T15:16:15.801009+0000","last_undegraded":"2024-03-18T15:16:15.801009+0000","last_fullsized":"2024-03-18T15:16:15.801009+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:12:10.280094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.801033+0000","last_change":"2024-03-18T15:15:10.657245+0000","last_active":"2024-03-18T15:16:15.801033+0000","last_peered":"2024-03-18T15:16:15.801033+0000","last_clean":"2024-03-18T15:16:15.801033+0000","last_became_active":"2024-03-18T15:15:10.655766+0000","last_became_peered":"2024-03-18T15:15:10.655766+0000","last_unstale":"2024-03-18T15:16:15.801033+0000","last_undegraded":"2024-03-18T15:16:15.801033+0000","last_fullsized":"2024-03-18T15:16:15.801033+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:44:26.340539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.803861+0000","last_change":"2024-03-18T15:15:15.109103+0000","last_active":"2024-03-18T15:16:15.803861+0000","last_peered":"2024-03-18T15:16:15.803861+0000","last_clean":"2024-03-18T15:16:15.803861+0000","last_became_active":"2024-03-18T15:15:15.108693+0000","last_became_peered":"2024-03-18T15:15:15.108693+0000","last_unstale":"2024-03-18T15:16:15.803861+0000","last_undegraded":"2024-03-18T15:16:15.803861+0000","last_fullsized":"2024-03-18T15:16:15.803861+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:40:57.841960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'7","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.932435+0000","last_change":"2024-03-18T15:15:10.103183+0000","last_active":"2024-03-18T15:16:15.932435+0000","last_peered":"2024-03-18T15:16:15.932435+0000","last_clean":"2024-03-18T15:16:15.932435+0000","last_became_active":"2024-03-18T15:15:10.101948+0000","last_became_peered":"2024-03-18T15:15:10.101948+0000","last_unstale":"2024-03-18T15:16:15.932435+0000","last_undegraded":"2024-03-18T15:16:15.932435+0000","last_fullsized":"2024-03-18T15:16:15.932435+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:04:51.718433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.798849+0000","last_change":"2024-03-18T15:15:10.666940+0000","last_active":"2024-03-18T15:16:15.798849+0000","last_peered":"2024-03-18T15:16:15.798849+0000","last_clean":"2024-03-18T15:16:15.798849+0000","last_became_active":"2024-03-18T15:15:10.666639+0000","last_became_peered":"2024-03-18T15:15:10.666639+0000","last_unstale":"2024-03-18T15:16:15.798849+0000","last_undegraded":"2024-03-18T15:16:15.798849+0000","last_fullsized":"2024-03-18T15:16:15.798849+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:04:45.945023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.798951+0000","last_change":"2024-03-18T15:15:15.120541+0000","last_active":"2024-03-18T15:16:15.798951+0000","last_peered":"2024-03-18T15:16:15.798951+0000","last_clean":"2024-03-18T15:16:15.798951+0000","last_became_active":"2024-03-18T15:15:15.120097+0000","last_became_peered":"2024-03-18T15:15:15.120097+0000","last_unstale":"2024-03-18T15:16:15.798951+0000","last_undegraded":"2024-03-18T15:16:15.798951+0000","last_fullsized":"2024-03-18T15:16:15.798951+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:12:08.932598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.797301+0000","last_change":"2024-03-18T15:15:10.079199+0000","last_active":"2024-03-18T15:16:15.797301+0000","last_peered":"2024-03-18T15:16:15.797301+0000","last_clean":"2024-03-18T15:16:15.797301+0000","last_became_active":"2024-03-18T15:15:10.078872+0000","last_became_peered":"2024-03-18T15:15:10.078872+0000","last_unstale":"2024-03-18T15:16:15.797301+0000","last_undegraded":"2024-03-18T15:16:15.797301+0000","last_fullsized":"2024-03-18T15:16:15.797301+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:31:44.722779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.930429+0000","last_change":"2024-03-18T15:15:10.686304+0000","last_active":"2024-03-18T15:16:15.930429+0000","last_peered":"2024-03-18T15:16:15.930429+0000","last_clean":"2024-03-18T15:16:15.930429+0000","last_became_active":"2024-03-18T15:15:10.685636+0000","last_became_peered":"2024-03-18T15:15:10.685636+0000","last_unstale":"2024-03-18T15:16:15.930429+0000","last_undegraded":"2024-03-18T15:16:15.930429+0000","last_fullsized":"2024-03-18T15:16:15.930429+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:31:03.315971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.798983+0000","last_change":"2024-03-18T15:15:15.108254+0000","last_active":"2024-03-18T15:16:15.798983+0000","last_peered":"2024-03-18T15:16:15.798983+0000","last_clean":"2024-03-18T15:16:15.798983+0000","last_became_active":"2024-03-18T15:15:15.107806+0000","last_became_peered":"2024-03-18T15:15:15.107806+0000","last_unstale":"2024-03-18T15:16:15.798983+0000","last_undegraded":"2024-03-18T15:16:15.798983+0000","last_fullsized":"2024-03-18T15:16:15.798983+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:55:53.427426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.800920+0000","last_change":"2024-03-18T15:15:10.082574+0000","last_active":"2024-03-18T15:16:15.800920+0000","last_peered":"2024-03-18T15:16:15.800920+0000","last_clean":"2024-03-18T15:16:15.800920+0000","last_became_active":"2024-03-18T15:15:10.082252+0000","last_became_peered":"2024-03-18T15:15:10.082252+0000","last_unstale":"2024-03-18T15:16:15.800920+0000","last_undegraded":"2024-03-18T15:16:15.800920+0000","last_fullsized":"2024-03-18T15:16:15.800920+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:25:17.475023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.803242+0000","last_change":"2024-03-18T15:15:10.667304+0000","last_active":"2024-03-18T15:16:15.803242+0000","last_peered":"2024-03-18T15:16:15.803242+0000","last_clean":"2024-03-18T15:16:15.803242+0000","last_became_active":"2024-03-18T15:15:10.666932+0000","last_became_peered":"2024-03-18T15:15:10.666932+0000","last_unstale":"2024-03-18T15:16:15.803242+0000","last_undegraded":"2024-03-18T15:16:15.803242+0000","last_fullsized":"2024-03-18T15:16:15.803242+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:22:33.817618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.801175+0000","last_change":"2024-03-18T15:15:15.113499+0000","last_active":"2024-03-18T15:16:15.801175+0000","last_peered":"2024-03-18T15:16:15.801175+0000","last_clean":"2024-03-18T15:16:15.801175+0000","last_became_active":"2024-03-18T15:15:15.113095+0000","last_became_peered":"2024-03-18T15:15:15.113095+0000","last_unstale":"2024-03-18T15:16:15.801175+0000","last_undegraded":"2024-03-18T15:16:15.801175+0000","last_fullsized":"2024-03-18T15:16:15.801175+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:58:28.255634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'5","reported_seq":119,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:17:10.079490+0000","last_change":"2024-03-18T15:15:10.078129+0000","last_active":"2024-03-18T15:17:10.079490+0000","last_peered":"2024-03-18T15:17:10.079490+0000","last_clean":"2024-03-18T15:17:10.079490+0000","last_became_active":"2024-03-18T15:15:10.076202+0000","last_became_peered":"2024-03-18T15:15:10.076202+0000","last_unstale":"2024-03-18T15:17:10.079490+0000","last_undegraded":"2024-03-18T15:17:10.079490+0000","last_fullsized":"2024-03-18T15:17:10.079490+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:24:58.240937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":87,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.804294+0000","last_change":"2024-03-18T15:15:10.665036+0000","last_active":"2024-03-18T15:16:15.804294+0000","last_peered":"2024-03-18T15:16:15.804294+0000","last_clean":"2024-03-18T15:16:15.804294+0000","last_became_active":"2024-03-18T15:15:10.664614+0000","last_became_peered":"2024-03-18T15:15:10.664614+0000","last_unstale":"2024-03-18T15:16:15.804294+0000","last_undegraded":"2024-03-18T15:16:15.804294+0000","last_fullsized":"2024-03-18T15:16:15.804294+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:31:42.454317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.810628+0000","last_change":"2024-03-18T15:15:15.127812+0000","last_active":"2024-03-18T15:16:15.810628+0000","last_peered":"2024-03-18T15:16:15.810628+0000","last_clean":"2024-03-18T15:16:15.810628+0000","last_became_active":"2024-03-18T15:15:15.127300+0000","last_became_peered":"2024-03-18T15:15:15.127300+0000","last_unstale":"2024-03-18T15:16:15.810628+0000","last_undegraded":"2024-03-18T15:16:15.810628+0000","last_fullsized":"2024-03-18T15:16:15.810628+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:41:25.292884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"89'3","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.801532+0000","last_change":"2024-03-18T15:15:10.093866+0000","last_active":"2024-03-18T15:16:15.801532+0000","last_peered":"2024-03-18T15:16:15.801532+0000","last_clean":"2024-03-18T15:16:15.801532+0000","last_became_active":"2024-03-18T15:15:10.093568+0000","last_became_peered":"2024-03-18T15:15:10.093568+0000","last_unstale":"2024-03-18T15:16:15.801532+0000","last_undegraded":"2024-03-18T15:16:15.801532+0000","last_fullsized":"2024-03-18T15:16:15.801532+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:00:03.533705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.802657+0000","last_change":"2024-03-18T15:15:10.679506+0000","last_active":"2024-03-18T15:16:15.802657+0000","last_peered":"2024-03-18T15:16:15.802657+0000","last_clean":"2024-03-18T15:16:15.802657+0000","last_became_active":"2024-03-18T15:15:10.679164+0000","last_became_peered":"2024-03-18T15:15:10.679164+0000","last_unstale":"2024-03-18T15:16:15.802657+0000","last_undegraded":"2024-03-18T15:16:15.802657+0000","last_fullsized":"2024-03-18T15:16:15.802657+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:04:55.266273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.799435+0000","last_change":"2024-03-18T15:15:15.099072+0000","last_active":"2024-03-18T15:16:15.799435+0000","last_peered":"2024-03-18T15:16:15.799435+0000","last_clean":"2024-03-18T15:16:15.799435+0000","last_became_active":"2024-03-18T15:15:15.098165+0000","last_became_peered":"2024-03-18T15:15:15.098165+0000","last_unstale":"2024-03-18T15:16:15.799435+0000","last_undegraded":"2024-03-18T15:16:15.799435+0000","last_fullsized":"2024-03-18T15:16:15.799435+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:14.072142+0000","last_clean_scrub_stamp":"2024-03-18T15:15:14.072142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:47:47.377924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'5","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.808997+0000","last_change":"2024-03-18T15:15:10.102562+0000","last_active":"2024-03-18T15:16:15.808997+0000","last_peered":"2024-03-18T15:16:15.808997+0000","last_clean":"2024-03-18T15:16:15.808997+0000","last_became_active":"2024-03-18T15:15:10.099318+0000","last_became_peered":"2024-03-18T15:15:10.099318+0000","last_unstale":"2024-03-18T15:16:15.808997+0000","last_undegraded":"2024-03-18T15:16:15.808997+0000","last_fullsized":"2024-03-18T15:16:15.808997+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:08.991307+0000","last_clean_scrub_stamp":"2024-03-18T15:15:08.991307+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:55:15.601925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.804293+0000","last_change":"2024-03-18T15:15:10.662645+0000","last_active":"2024-03-18T15:16:15.804293+0000","last_peered":"2024-03-18T15:16:15.804293+0000","last_clean":"2024-03-18T15:16:15.804293+0000","last_became_active":"2024-03-18T15:15:10.662280+0000","last_became_peered":"2024-03-18T15:15:10.662280+0000","last_unstale":"2024-03-18T15:16:15.804293+0000","last_undegraded":"2024-03-18T15:16:15.804293+0000","last_fullsized":"2024-03-18T15:16:15.804293+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:11:22.598169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.798952+0000","last_change":"2024-03-18T15:15:10.661839+0000","last_active":"2024-03-18T15:16:15.798952+0000","last_peered":"2024-03-18T15:16:15.798952+0000","last_clean":"2024-03-18T15:16:15.798952+0000","last_became_active":"2024-03-18T15:15:10.661356+0000","last_became_peered":"2024-03-18T15:15:10.661356+0000","last_unstale":"2024-03-18T15:16:15.798952+0000","last_undegraded":"2024-03-18T15:16:15.798952+0000","last_fullsized":"2024-03-18T15:16:15.798952+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:13:33.637281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.796879+0000","last_change":"2024-03-18T15:15:10.661946+0000","last_active":"2024-03-18T15:16:15.796879+0000","last_peered":"2024-03-18T15:16:15.796879+0000","last_clean":"2024-03-18T15:16:15.796879+0000","last_became_active":"2024-03-18T15:15:10.661574+0000","last_became_peered":"2024-03-18T15:15:10.661574+0000","last_unstale":"2024-03-18T15:16:15.796879+0000","last_undegraded":"2024-03-18T15:16:15.796879+0000","last_fullsized":"2024-03-18T15:16:15.796879+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:53:50.284156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.796979+0000","last_change":"2024-03-18T15:15:10.674064+0000","last_active":"2024-03-18T15:16:15.796979+0000","last_peered":"2024-03-18T15:16:15.796979+0000","last_clean":"2024-03-18T15:16:15.796979+0000","last_became_active":"2024-03-18T15:15:10.673820+0000","last_became_peered":"2024-03-18T15:15:10.673820+0000","last_unstale":"2024-03-18T15:16:15.796979+0000","last_undegraded":"2024-03-18T15:16:15.796979+0000","last_fullsized":"2024-03-18T15:16:15.796979+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:29:28.977824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.813370+0000","last_change":"2024-03-18T15:15:10.671276+0000","last_active":"2024-03-18T15:16:15.813370+0000","last_peered":"2024-03-18T15:16:15.813370+0000","last_clean":"2024-03-18T15:16:15.813370+0000","last_became_active":"2024-03-18T15:15:10.670926+0000","last_became_peered":"2024-03-18T15:15:10.670926+0000","last_unstale":"2024-03-18T15:16:15.813370+0000","last_undegraded":"2024-03-18T15:16:15.813370+0000","last_fullsized":"2024-03-18T15:16:15.813370+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T15:15:43.972815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:17:14.306538+0000","last_change":"2024-03-18T15:16:15.820673+0000","last_active":"2024-03-18T15:17:14.306538+0000","last_peered":"2024-03-18T15:17:14.306538+0000","last_clean":"2024-03-18T15:17:14.306538+0000","last_became_active":"2024-03-18T15:16:15.819923+0000","last_became_peered":"2024-03-18T15:16:15.819923+0000","last_unstale":"2024-03-18T15:17:14.306538+0000","last_undegraded":"2024-03-18T15:17:14.306538+0000","last_fullsized":"2024-03-18T15:17:14.306538+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:28:51.498407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.799171+0000","last_change":"2024-03-18T15:15:10.658966+0000","last_active":"2024-03-18T15:16:15.799171+0000","last_peered":"2024-03-18T15:16:15.799171+0000","last_clean":"2024-03-18T15:16:15.799171+0000","last_became_active":"2024-03-18T15:15:10.658654+0000","last_became_peered":"2024-03-18T15:15:10.658654+0000","last_unstale":"2024-03-18T15:16:15.799171+0000","last_undegraded":"2024-03-18T15:16:15.799171+0000","last_fullsized":"2024-03-18T15:16:15.799171+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:42:25.730735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.797274+0000","last_change":"2024-03-18T15:15:10.665694+0000","last_active":"2024-03-18T15:16:15.797274+0000","last_peered":"2024-03-18T15:16:15.797274+0000","last_clean":"2024-03-18T15:16:15.797274+0000","last_became_active":"2024-03-18T15:15:10.664688+0000","last_became_peered":"2024-03-18T15:15:10.664688+0000","last_unstale":"2024-03-18T15:16:15.797274+0000","last_undegraded":"2024-03-18T15:16:15.797274+0000","last_fullsized":"2024-03-18T15:16:15.797274+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:32:23.108060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.807899+0000","last_change":"2024-03-18T15:16:15.807899+0000","last_active":"2024-03-18T15:16:15.807899+0000","last_peered":"2024-03-18T15:16:15.807899+0000","last_clean":"2024-03-18T15:16:15.807899+0000","last_became_active":"2024-03-18T15:16:15.807358+0000","last_became_peered":"2024-03-18T15:16:15.807358+0000","last_unstale":"2024-03-18T15:16:15.807899+0000","last_undegraded":"2024-03-18T15:16:15.807899+0000","last_fullsized":"2024-03-18T15:16:15.807899+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:12:55.012985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,2],"acting":[9,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.799781+0000","last_change":"2024-03-18T15:15:10.668833+0000","last_active":"2024-03-18T15:16:15.799781+0000","last_peered":"2024-03-18T15:16:15.799781+0000","last_clean":"2024-03-18T15:16:15.799781+0000","last_became_active":"2024-03-18T15:15:10.668572+0000","last_became_peered":"2024-03-18T15:15:10.668572+0000","last_unstale":"2024-03-18T15:16:15.799781+0000","last_undegraded":"2024-03-18T15:16:15.799781+0000","last_fullsized":"2024-03-18T15:16:15.799781+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:12:44.333515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.936318+0000","last_change":"2024-03-18T15:15:10.677496+0000","last_active":"2024-03-18T15:16:15.936318+0000","last_peered":"2024-03-18T15:16:15.936318+0000","last_clean":"2024-03-18T15:16:15.936318+0000","last_became_active":"2024-03-18T15:15:10.674869+0000","last_became_peered":"2024-03-18T15:15:10.674869+0000","last_unstale":"2024-03-18T15:16:15.936318+0000","last_undegraded":"2024-03-18T15:16:15.936318+0000","last_fullsized":"2024-03-18T15:16:15.936318+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:03:10.531488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.802024+0000","last_change":"2024-03-18T15:15:10.673488+0000","last_active":"2024-03-18T15:16:15.802024+0000","last_peered":"2024-03-18T15:16:15.802024+0000","last_clean":"2024-03-18T15:16:15.802024+0000","last_became_active":"2024-03-18T15:15:10.673117+0000","last_became_peered":"2024-03-18T15:15:10.673117+0000","last_unstale":"2024-03-18T15:16:15.802024+0000","last_undegraded":"2024-03-18T15:16:15.802024+0000","last_fullsized":"2024-03-18T15:16:15.802024+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:29:22.823829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:17:14.316102+0000","last_change":"2024-03-18T15:15:10.678916+0000","last_active":"2024-03-18T15:17:14.316102+0000","last_peered":"2024-03-18T15:17:14.316102+0000","last_clean":"2024-03-18T15:17:14.316102+0000","last_became_active":"2024-03-18T15:15:10.678545+0000","last_became_peered":"2024-03-18T15:15:10.678545+0000","last_unstale":"2024-03-18T15:17:14.316102+0000","last_undegraded":"2024-03-18T15:17:14.316102+0000","last_fullsized":"2024-03-18T15:17:14.316102+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:13:52.714537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T15:16:15.798924+0000","last_change":"2024-03-18T15:15:10.660397+0000","last_active":"2024-03-18T15:16:15.798924+0000","last_peered":"2024-03-18T15:16:15.798924+0000","last_clean":"2024-03-18T15:16:15.798924+0000","last_became_active":"2024-03-18T15:15:10.660085+0000","last_became_peered":"2024-03-18T15:15:10.660085+0000","last_unstale":"2024-03-18T15:16:15.798924+0000","last_undegraded":"2024-03-18T15:16:15.798924+0000","last_fullsized":"2024-03-18T15:16:15.798924+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:15:10.045358+0000","last_clean_scrub_stamp":"2024-03-18T15:15:10.045358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:13:04.591062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3542208,"data_compressed":210616,"data_compressed_allocated":270336,"data_compressed_original":3506176,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":69778,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":887,"num_read_kb":520,"num_write":244,"num_write_kb":140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":622,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":209334,"data_compressed":4773,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":211,"ondisk_log_size":211,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383719,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1003520,"data_stored":501433,"data_compressed":20639,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645355,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28148,"kb_used_data":1124,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717004,"statfs":{"total":95995035648,"available":95966212096,"internally_reserved":0,"allocated":1150976,"data_stored":1151605,"data_compressed":57881,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808880,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32192,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712960,"statfs":{"total":95995035648,"available":95962071040,"internally_reserved":0,"allocated":1032192,"data_stored":505459,"data_compressed":20639,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070520,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32180,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712972,"statfs":{"total":95995035648,"available":95962083328,"internally_reserved":0,"allocated":1024000,"data_stored":501741,"data_compressed":20639,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332156,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28072,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717080,"statfs":{"total":95995035648,"available":95966289920,"internally_reserved":0,"allocated":1077248,"data_stored":1097847,"data_compressed":56290,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495683,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32160,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712992,"statfs":{"total":95995035648,"available":95962103808,"internally_reserved":0,"allocated":1003520,"data_stored":496867,"data_compressed":20639,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574943,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32228,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712924,"statfs":{"total":95995035648,"available":95962034176,"internally_reserved":0,"allocated":1069056,"data_stored":561253,"data_compressed":22230,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738468,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1028096,"data_stored":498323,"data_compressed":20639,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411418,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28080,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717072,"statfs":{"total":95995035648,"available":95966281728,"internally_reserved":0,"allocated":1081344,"data_stored":560851,"data_compressed":22230,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182484,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1015808,"data_stored":496575,"data_compressed":20639,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920847,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32180,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712972,"statfs":{"total":95995035648,"available":95962083328,"internally_reserved":0,"allocated":1019904,"data_stored":501563,"data_compressed":20639,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757321,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28084,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717068,"statfs":{"total":95995035648,"available":95966277632,"internally_reserved":0,"allocated":1085440,"data_stored":1093429,"data_compressed":56290,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34006,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34006,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35651,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35651,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35651,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35651,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":65108,"data_compressed":1591,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":64706,"data_compressed":1591,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":65092,"data_compressed":1591,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T15:17:15.316 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-18T15:17:15.316 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-18T15:17:15.316 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-18T15:17:15.316 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 -- ceph health --format=json 2024-03-18T15:17:15.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:17:15 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.8", "id": [5, 8]}]: dispatch 2024-03-18T15:17:15.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:17:15 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 10]}]: dispatch 2024-03-18T15:17:15.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:17:15 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 8]}]: dispatch 2024-03-18T15:17:15.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:17:15 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]: dispatch 2024-03-18T15:17:15.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:17:15 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.8", "id": [5, 8]}]: dispatch 2024-03-18T15:17:15.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:17:15 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 10]}]: dispatch 2024-03-18T15:17:15.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:17:15 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 8]}]: dispatch 2024-03-18T15:17:15.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:17:15 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]: dispatch 2024-03-18T15:17:15.840 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:17:16.241 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:17:16 smithi019 ceph-mon[31522]: pgmap v381: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T15:17:16.241 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:17:16 smithi019 ceph-mon[31522]: from='client.25031 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T15:17:16.241 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:17:16 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.8", "id": [5, 8]}]': finished 2024-03-18T15:17:16.241 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:17:16 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 10]}]': finished 2024-03-18T15:17:16.242 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:17:16 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 8]}]': finished 2024-03-18T15:17:16.242 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:17:16 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]': finished 2024-03-18T15:17:16.242 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:17:16 smithi019 ceph-mon[31522]: mgrmap e18: x(active, since 10m), standbys: z, y 2024-03-18T15:17:16.242 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:17:16 smithi019 ceph-mon[31522]: osdmap e92: 12 total, 12 up, 12 in 2024-03-18T15:17:16.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:17:16 smithi104 ceph-mon[36070]: pgmap v381: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T15:17:16.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:17:16 smithi104 ceph-mon[36070]: from='client.25031 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T15:17:16.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:17:16 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.8", "id": [5, 8]}]': finished 2024-03-18T15:17:16.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:17:16 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 10]}]': finished 2024-03-18T15:17:16.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:17:16 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 8]}]': finished 2024-03-18T15:17:16.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:17:16 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]': finished 2024-03-18T15:17:16.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:17:16 smithi104 ceph-mon[36070]: mgrmap e18: x(active, since 10m), standbys: z, y 2024-03-18T15:17:16.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:17:16 smithi104 ceph-mon[36070]: osdmap e92: 12 total, 12 up, 12 in 2024-03-18T15:17:16.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:17:16 smithi033 ceph-mon[36648]: pgmap v381: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T15:17:16.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:17:16 smithi033 ceph-mon[36648]: from='client.25031 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T15:17:16.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:17:16 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.8", "id": [5, 8]}]': finished 2024-03-18T15:17:16.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:17:16 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 10]}]': finished 2024-03-18T15:17:16.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:17:16 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 8]}]': finished 2024-03-18T15:17:16.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:17:16 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]': finished 2024-03-18T15:17:16.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:17:16 smithi033 ceph-mon[36648]: mgrmap e18: x(active, since 10m), standbys: z, y 2024-03-18T15:17:16.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:17:16 smithi033 ceph-mon[36648]: osdmap e92: 12 total, 12 up, 12 in 2024-03-18T15:17:17.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:17:17 smithi019 ceph-mon[31522]: osdmap e93: 12 total, 12 up, 12 in 2024-03-18T15:17:17.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:17:17 smithi104 ceph-mon[36070]: osdmap e93: 12 total, 12 up, 12 in 2024-03-18T15:17:17.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:17:17 smithi033 ceph-mon[36648]: osdmap e93: 12 total, 12 up, 12 in 2024-03-18T15:17:18.211 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:18.209+0000 7f9ada400700 1 -- 172.21.15.19:0/1708334445 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9ad406a920 msgr2=0x7f9ad410f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:18.211 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:18.209+0000 7f9ada400700 1 --2- 172.21.15.19:0/1708334445 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9ad406a920 0x7f9ad410f160 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f9acc009990 tx=0x7f9acc009ca0 comp rx=0 tx=0).stop 2024-03-18T15:17:18.211 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:17:18 smithi019 ceph-mon[31522]: pgmap v384: 129 pgs: 3 peering, 126 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-18T15:17:18.211 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:17:18 smithi019 ceph-mon[31522]: Health check failed: Reduced data availability: 2 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-03-18T15:17:18.212 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:18.210+0000 7f9ada400700 1 -- 172.21.15.19:0/1708334445 shutdown_connections 2024-03-18T15:17:18.212 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:18.210+0000 7f9ada400700 1 --2- 172.21.15.19:0/1708334445 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9ad406a920 0x7f9ad410f160 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:18.212 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:18.210+0000 7f9ada400700 1 --2- 172.21.15.19:0/1708334445 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9ad4108320 0x7f9ad4108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:18.212 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:18.210+0000 7f9ada400700 1 --2- 172.21.15.19:0/1708334445 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9ad4069f80 0x7f9ad406a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:18.212 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:18.210+0000 7f9ada400700 1 -- 172.21.15.19:0/1708334445 >> 172.21.15.19:0/1708334445 conn(0x7f9ad4078100 msgr2=0x7f9ad4078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:17:18.212 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:18.210+0000 7f9ada400700 1 -- 172.21.15.19:0/1708334445 shutdown_connections 2024-03-18T15:17:18.213 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:18.210+0000 7f9ada400700 1 -- 172.21.15.19:0/1708334445 wait complete. 2024-03-18T15:17:18.213 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:18.211+0000 7f9ada400700 1 Processor -- start 2024-03-18T15:17:18.213 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:18.211+0000 7f9ada400700 1 -- start start 2024-03-18T15:17:18.213 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:18.212+0000 7f9ada400700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9ad4069f80 0x7f9ad41012a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:18.213 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:18.212+0000 7f9ada400700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9ad406a920 0x7f9ad41017e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:18.213 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:18.212+0000 7f9ada400700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9ad4108320 0x7f9ad4101d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:18.213 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:18.212+0000 7f9ada400700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f9ad41125c0 con 0x7f9ad4108320 2024-03-18T15:17:18.213 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:18.212+0000 7f9ada400700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9ad4112440 con 0x7f9ad4069f80 2024-03-18T15:17:18.213 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:18.212+0000 7f9ada400700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f9ad4112740 con 0x7f9ad406a920 2024-03-18T15:17:18.213 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:18.212+0000 7f9acbe00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9ad406a920 0x7f9ad41017e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:18.213 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:18.212+0000 7f9ad3400700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9ad4069f80 0x7f9ad41012a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:18.213 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:18.212+0000 7f9ad3400700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9ad4069f80 0x7f9ad41012a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:56048/0 (socket says 172.21.15.19:56048) 2024-03-18T15:17:18.213 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:18.212+0000 7f9acbe00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9ad406a920 0x7f9ad41017e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.19:35376/0 (socket says 172.21.15.19:35376) 2024-03-18T15:17:18.213 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:18.212+0000 7f9ad3400700 1 -- 172.21.15.19:0/2094855168 learned_addr learned my addr 172.21.15.19:0/2094855168 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:17:18.214 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:18.212+0000 7f9ad3e00700 1 --2- 172.21.15.19:0/2094855168 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9ad4108320 0x7f9ad4101d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:18.215 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:18.212+0000 7f9ad3400700 1 -- 172.21.15.19:0/2094855168 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9ad406a920 msgr2=0x7f9ad41017e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:18.215 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:18.213+0000 7f9ad3400700 1 --2- 172.21.15.19:0/2094855168 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9ad406a920 0x7f9ad41017e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:18.215 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:18.213+0000 7f9ad3400700 1 -- 172.21.15.19:0/2094855168 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9ad4108320 msgr2=0x7f9ad4101d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:18.215 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:18.213+0000 7f9ad3400700 1 --2- 172.21.15.19:0/2094855168 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9ad4108320 0x7f9ad4101d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:18.215 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:18.213+0000 7f9ad3400700 1 -- 172.21.15.19:0/2094855168 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ad41a6ff0 con 0x7f9ad4069f80 2024-03-18T15:17:18.215 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:18.213+0000 7f9acbe00700 1 --2- 172.21.15.19:0/2094855168 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9ad406a920 0x7f9ad41017e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:17:18.215 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:18.213+0000 7f9ad3e00700 1 --2- 172.21.15.19:0/2094855168 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9ad4108320 0x7f9ad4101d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:17:18.215 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:18.213+0000 7f9ad3400700 1 --2- 172.21.15.19:0/2094855168 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9ad4069f80 0x7f9ad41012a0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f9ac400ef10 tx=0x7f9ac400c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:17:18.215 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:18.213+0000 7f9ad0c00700 1 -- 172.21.15.19:0/2094855168 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ac400c9c0 con 0x7f9ad4069f80 2024-03-18T15:17:18.215 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:18.213+0000 7f9ad0c00700 1 -- 172.21.15.19:0/2094855168 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ac4019070 con 0x7f9ad4069f80 2024-03-18T15:17:18.216 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:18.213+0000 7f9ad0c00700 1 -- 172.21.15.19:0/2094855168 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ac4007e60 con 0x7f9ad4069f80 2024-03-18T15:17:18.216 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:18.213+0000 7f9ada400700 1 -- 172.21.15.19:0/2094855168 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ad41a7280 con 0x7f9ad4069f80 2024-03-18T15:17:18.216 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:18.213+0000 7f9ada400700 1 -- 172.21.15.19:0/2094855168 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ad41a7810 con 0x7f9ad4069f80 2024-03-18T15:17:18.216 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:18.214+0000 7f9ada400700 1 -- 172.21.15.19:0/2094855168 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ad4068f90 con 0x7f9ad4069f80 2024-03-18T15:17:18.216 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:18.217+0000 7f9ad0c00700 1 -- 172.21.15.19:0/2094855168 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f9ac4012050 con 0x7f9ad4069f80 2024-03-18T15:17:18.218 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:18.218+0000 7f9ad0c00700 1 --2- 172.21.15.19:0/2094855168 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f9ab80a2910 0x7f9ab80a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:18.218 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:18.218+0000 7f9ad0c00700 1 -- 172.21.15.19:0/2094855168 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f9ac40ced30 con 0x7f9ad4069f80 2024-03-18T15:17:18.218 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:18.218+0000 7f9acbe00700 1 --2- 172.21.15.19:0/2094855168 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f9ab80a2910 0x7f9ab80a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:18.219 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:18.219+0000 7f9acbe00700 1 --2- 172.21.15.19:0/2094855168 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f9ab80a2910 0x7f9ab80a4dd0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f9ad40786e0 tx=0x7f9abc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:17:18.222 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:18.222+0000 7f9ad0c00700 1 -- 172.21.15.19:0/2094855168 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9ac4010040 con 0x7f9ad4069f80 2024-03-18T15:17:18.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:17:18 smithi104 ceph-mon[36070]: pgmap v384: 129 pgs: 3 peering, 126 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-18T15:17:18.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:17:18 smithi104 ceph-mon[36070]: Health check failed: Reduced data availability: 2 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-03-18T15:17:18.487 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:18.487+0000 7f9ada400700 1 -- 172.21.15.19:0/2094855168 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f9ad4102780 con 0x7f9ad4069f80 2024-03-18T15:17:18.487 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:18.487+0000 7f9ad0c00700 1 -- 172.21.15.19:0/2094855168 <== mon.2 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+197 (secure 0 0 0) 0x7f9ac409b9a0 con 0x7f9ad4069f80 2024-03-18T15:17:18.488 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-18T15:17:18.488 INFO:teuthology.orchestra.run.smithi019.stdout:{"status":"HEALTH_WARN","checks":{"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 2 pgs inactive, 3 pgs peering","count":5},"muted":false}},"mutes":[]} 2024-03-18T15:17:18.490 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:18.490+0000 7f9ada400700 1 -- 172.21.15.19:0/2094855168 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f9ab80a2910 msgr2=0x7f9ab80a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:18.490 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:18.490+0000 7f9ada400700 1 --2- 172.21.15.19:0/2094855168 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f9ab80a2910 0x7f9ab80a4dd0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f9ad40786e0 tx=0x7f9abc009260 comp rx=0 tx=0).stop 2024-03-18T15:17:18.491 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:18.490+0000 7f9ada400700 1 -- 172.21.15.19:0/2094855168 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9ad4069f80 msgr2=0x7f9ad41012a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:18.491 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:18.490+0000 7f9ada400700 1 --2- 172.21.15.19:0/2094855168 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9ad4069f80 0x7f9ad41012a0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f9ac400ef10 tx=0x7f9ac400c3a0 comp rx=0 tx=0).stop 2024-03-18T15:17:18.491 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:18.490+0000 7f9ada400700 1 -- 172.21.15.19:0/2094855168 shutdown_connections 2024-03-18T15:17:18.491 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:18.490+0000 7f9ada400700 1 --2- 172.21.15.19:0/2094855168 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f9ab80a2910 0x7f9ab80a4dd0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:18.491 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:18.490+0000 7f9ada400700 1 --2- 172.21.15.19:0/2094855168 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9ad4108320 0x7f9ad4101d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:18.491 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:18.490+0000 7f9ada400700 1 --2- 172.21.15.19:0/2094855168 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9ad4069f80 0x7f9ad41012a0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:18.491 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:18.490+0000 7f9ada400700 1 --2- 172.21.15.19:0/2094855168 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9ad406a920 0x7f9ad41017e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:18.491 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:18.490+0000 7f9ada400700 1 -- 172.21.15.19:0/2094855168 >> 172.21.15.19:0/2094855168 conn(0x7f9ad4078100 msgr2=0x7f9ad40ffde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:17:18.491 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:18.491+0000 7f9ada400700 1 -- 172.21.15.19:0/2094855168 shutdown_connections 2024-03-18T15:17:18.492 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:18.491+0000 7f9ada400700 1 -- 172.21.15.19:0/2094855168 wait complete. 2024-03-18T15:17:18.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:17:18 smithi033 ceph-mon[36648]: pgmap v384: 129 pgs: 3 peering, 126 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-18T15:17:18.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:17:18 smithi033 ceph-mon[36648]: Health check failed: Reduced data availability: 2 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-03-18T15:17:19.214 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:17:19 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/2094855168' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-18T15:17:19.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:17:19 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/2094855168' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-18T15:17:19.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:17:19 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/2094855168' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-18T15:17:20.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:17:20 smithi104 ceph-mon[36070]: pgmap v385: 129 pgs: 3 peering, 126 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 383 B/s wr, 10 op/s 2024-03-18T15:17:20.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:17:20 smithi019 ceph-mon[31522]: pgmap v385: 129 pgs: 3 peering, 126 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 383 B/s wr, 10 op/s 2024-03-18T15:17:20.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:17:20 smithi033 ceph-mon[36648]: pgmap v385: 129 pgs: 3 peering, 126 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 383 B/s wr, 10 op/s 2024-03-18T15:17:22.295 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 -- ceph health --format=json 2024-03-18T15:17:22.394 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:17:22 smithi019 ceph-mon[31522]: pgmap v386: 129 pgs: 3 peering, 126 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 383 B/s wr, 9 op/s 2024-03-18T15:17:22.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:17:22 smithi104 ceph-mon[36070]: pgmap v386: 129 pgs: 3 peering, 126 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 383 B/s wr, 9 op/s 2024-03-18T15:17:22.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:17:22 smithi033 ceph-mon[36648]: pgmap v386: 129 pgs: 3 peering, 126 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 383 B/s wr, 9 op/s 2024-03-18T15:17:22.798 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:17:23.207 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:17:23 smithi019 ceph-mon[31522]: pgmap v387: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 383 B/s wr, 11 op/s 2024-03-18T15:17:23.207 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:17:23 smithi019 ceph-mon[31522]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 3 pgs peering) 2024-03-18T15:17:23.207 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:17:23 smithi019 ceph-mon[31522]: Cluster is now healthy 2024-03-18T15:17:23.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:17:23 smithi104 ceph-mon[36070]: pgmap v387: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 383 B/s wr, 11 op/s 2024-03-18T15:17:23.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:17:23 smithi104 ceph-mon[36070]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 3 pgs peering) 2024-03-18T15:17:23.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:17:23 smithi104 ceph-mon[36070]: Cluster is now healthy 2024-03-18T15:17:23.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:17:23 smithi033 ceph-mon[36648]: pgmap v387: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 383 B/s wr, 11 op/s 2024-03-18T15:17:23.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:17:23 smithi033 ceph-mon[36648]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 3 pgs peering) 2024-03-18T15:17:23.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:17:23 smithi033 ceph-mon[36648]: Cluster is now healthy 2024-03-18T15:17:25.252 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:25.250+0000 7f5b14e00700 1 -- 172.21.15.19:0/2565997052 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5b10108320 msgr2=0x7f5b10108700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:25.252 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:25.250+0000 7f5b14e00700 1 --2- 172.21.15.19:0/2565997052 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5b10108320 0x7f5b10108700 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f5b00009990 tx=0x7f5b00009ca0 comp rx=0 tx=0).stop 2024-03-18T15:17:25.252 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:25.250+0000 7f5b14e00700 1 -- 172.21.15.19:0/2565997052 shutdown_connections 2024-03-18T15:17:25.252 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:25.250+0000 7f5b14e00700 1 --2- 172.21.15.19:0/2565997052 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5b1006a920 0x7f5b1010f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:25.252 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:25.250+0000 7f5b14e00700 1 --2- 172.21.15.19:0/2565997052 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5b10108320 0x7f5b10108700 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:25.252 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:25.250+0000 7f5b14e00700 1 --2- 172.21.15.19:0/2565997052 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b10069f80 0x7f5b1006a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:25.252 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:25.250+0000 7f5b14e00700 1 -- 172.21.15.19:0/2565997052 >> 172.21.15.19:0/2565997052 conn(0x7f5b10078100 msgr2=0x7f5b10078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:17:25.252 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:25.250+0000 7f5b14e00700 1 -- 172.21.15.19:0/2565997052 shutdown_connections 2024-03-18T15:17:25.252 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:25.250+0000 7f5b14e00700 1 -- 172.21.15.19:0/2565997052 wait complete. 2024-03-18T15:17:25.252 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:25.252+0000 7f5b14e00700 1 Processor -- start 2024-03-18T15:17:25.253 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:25.252+0000 7f5b14e00700 1 -- start start 2024-03-18T15:17:25.254 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:25.253+0000 7f5b14e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5b10069f80 0x7f5b101012a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:25.254 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:25.253+0000 7f5b14e00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b1006a920 0x7f5b101017e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:25.254 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:25.253+0000 7f5b14e00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5b10108320 0x7f5b10101d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:25.254 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:25.253+0000 7f5b14e00700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f5b101125d0 con 0x7f5b10108320 2024-03-18T15:17:25.254 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:25.253+0000 7f5b14e00700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5b10112450 con 0x7f5b10069f80 2024-03-18T15:17:25.255 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:25.253+0000 7f5b14e00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f5b10112750 con 0x7f5b1006a920 2024-03-18T15:17:25.255 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:25.253+0000 7f5b0de00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5b10069f80 0x7f5b101012a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:25.255 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:25.253+0000 7f5b0de00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5b10069f80 0x7f5b101012a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:55828/0 (socket says 172.21.15.19:55828) 2024-03-18T15:17:25.255 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:25.253+0000 7f5b0de00700 1 -- 172.21.15.19:0/2588886344 learned_addr learned my addr 172.21.15.19:0/2588886344 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:17:25.255 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:25.253+0000 7f5b05400700 1 --2- 172.21.15.19:0/2588886344 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b1006a920 0x7f5b101017e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:25.255 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:25.253+0000 7f5b0e800700 1 --2- 172.21.15.19:0/2588886344 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5b10108320 0x7f5b10101d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:25.255 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:25.253+0000 7f5b0de00700 1 -- 172.21.15.19:0/2588886344 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b1006a920 msgr2=0x7f5b101017e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:25.255 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:25.253+0000 7f5b0de00700 1 --2- 172.21.15.19:0/2588886344 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b1006a920 0x7f5b101017e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:25.255 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:25.253+0000 7f5b0de00700 1 -- 172.21.15.19:0/2588886344 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5b10108320 msgr2=0x7f5b10101d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:25.255 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:25.253+0000 7f5b0de00700 1 --2- 172.21.15.19:0/2588886344 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5b10108320 0x7f5b10101d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:25.255 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:25.253+0000 7f5b0de00700 1 -- 172.21.15.19:0/2588886344 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b101a6f70 con 0x7f5b10069f80 2024-03-18T15:17:25.255 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:25.254+0000 7f5b0de00700 1 --2- 172.21.15.19:0/2588886344 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5b10069f80 0x7f5b101012a0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f5b00005910 tx=0x7f5b000047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:17:25.255 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:25.254+0000 7f5b07400700 1 -- 172.21.15.19:0/2588886344 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b00027070 con 0x7f5b10069f80 2024-03-18T15:17:25.256 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:25.254+0000 7f5b07400700 1 -- 172.21.15.19:0/2588886344 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b0000b6c0 con 0x7f5b10069f80 2024-03-18T15:17:25.256 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:25.254+0000 7f5b14e00700 1 -- 172.21.15.19:0/2588886344 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b101a7200 con 0x7f5b10069f80 2024-03-18T15:17:25.256 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:25.254+0000 7f5b07400700 1 -- 172.21.15.19:0/2588886344 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b00017650 con 0x7f5b10069f80 2024-03-18T15:17:25.256 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:25.254+0000 7f5b14e00700 1 -- 172.21.15.19:0/2588886344 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b101a76b0 con 0x7f5b10069f80 2024-03-18T15:17:25.256 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:25.255+0000 7f5b14e00700 1 -- 172.21.15.19:0/2588886344 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b10068f90 con 0x7f5b10069f80 2024-03-18T15:17:25.257 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:25.257+0000 7f5b07400700 1 -- 172.21.15.19:0/2588886344 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f5b0002a070 con 0x7f5b10069f80 2024-03-18T15:17:25.258 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:25.258+0000 7f5b07400700 1 --2- 172.21.15.19:0/2588886344 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f5af00a2910 0x7f5af00a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:25.259 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:25.259+0000 7f5b07400700 1 -- 172.21.15.19:0/2588886344 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f5b000cfd80 con 0x7f5b10069f80 2024-03-18T15:17:25.259 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:25.259+0000 7f5b05400700 1 --2- 172.21.15.19:0/2588886344 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f5af00a2910 0x7f5af00a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:25.259 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:25.260+0000 7f5b05400700 1 --2- 172.21.15.19:0/2588886344 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f5af00a2910 0x7f5af00a4dd0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f5b10078700 tx=0x7f5af8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:17:25.262 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:25.262+0000 7f5b07400700 1 -- 172.21.15.19:0/2588886344 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5b0005e740 con 0x7f5b10069f80 2024-03-18T15:17:25.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:17:25 smithi019 ceph-mon[31522]: pgmap v388: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 329 B/s wr, 11 op/s 2024-03-18T15:17:25.544 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:25.544+0000 7f5b14e00700 1 -- 172.21.15.19:0/2588886344 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f5b1002db70 con 0x7f5b10069f80 2024-03-18T15:17:25.545 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:25.544+0000 7f5b07400700 1 -- 172.21.15.19:0/2588886344 <== mon.2 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f5b00013070 con 0x7f5b10069f80 2024-03-18T15:17:25.545 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-18T15:17:25.545 INFO:teuthology.orchestra.run.smithi019.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-18T15:17:25.547 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:25.547+0000 7f5b14e00700 1 -- 172.21.15.19:0/2588886344 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f5af00a2910 msgr2=0x7f5af00a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:25.547 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:25.547+0000 7f5b14e00700 1 --2- 172.21.15.19:0/2588886344 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f5af00a2910 0x7f5af00a4dd0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f5b10078700 tx=0x7f5af8009260 comp rx=0 tx=0).stop 2024-03-18T15:17:25.547 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:25.547+0000 7f5b14e00700 1 -- 172.21.15.19:0/2588886344 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5b10069f80 msgr2=0x7f5b101012a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:25.547 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:25.547+0000 7f5b14e00700 1 --2- 172.21.15.19:0/2588886344 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5b10069f80 0x7f5b101012a0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f5b00005910 tx=0x7f5b000047b0 comp rx=0 tx=0).stop 2024-03-18T15:17:25.548 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:25.547+0000 7f5b14e00700 1 -- 172.21.15.19:0/2588886344 shutdown_connections 2024-03-18T15:17:25.548 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:25.547+0000 7f5b14e00700 1 --2- 172.21.15.19:0/2588886344 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f5af00a2910 0x7f5af00a4dd0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:25.548 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:25.547+0000 7f5b14e00700 1 --2- 172.21.15.19:0/2588886344 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5b10108320 0x7f5b10101d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:25.548 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:25.547+0000 7f5b14e00700 1 --2- 172.21.15.19:0/2588886344 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5b10069f80 0x7f5b101012a0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:25.548 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:25.547+0000 7f5b14e00700 1 --2- 172.21.15.19:0/2588886344 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b1006a920 0x7f5b101017e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:25.548 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:25.547+0000 7f5b14e00700 1 -- 172.21.15.19:0/2588886344 >> 172.21.15.19:0/2588886344 conn(0x7f5b10078100 msgr2=0x7f5b100ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:17:25.548 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:25.548+0000 7f5b14e00700 1 -- 172.21.15.19:0/2588886344 shutdown_connections 2024-03-18T15:17:25.549 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:25.548+0000 7f5b14e00700 1 -- 172.21.15.19:0/2588886344 wait complete. 2024-03-18T15:17:25.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:17:25 smithi104 ceph-mon[36070]: pgmap v388: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 329 B/s wr, 11 op/s 2024-03-18T15:17:25.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:17:25 smithi033 ceph-mon[36648]: pgmap v388: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 329 B/s wr, 11 op/s 2024-03-18T15:17:26.605 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:17:26 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/2588886344' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-18T15:17:26.671 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-18T15:17:26.672 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T15:17:26.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:17:26 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/2588886344' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-18T15:17:26.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:17:26 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/2588886344' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-18T15:17:26.903 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:26.903+0000 7f283b200640 1 -- 172.21.15.19:0/2051283766 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2834108e10 msgr2=0x7f2834109290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:26.903 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:26.903+0000 7f283b200640 1 --2- 172.21.15.19:0/2051283766 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2834108e10 0x7f2834109290 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f2824009900 tx=0x7f282402bc40 comp rx=0 tx=0).stop 2024-03-18T15:17:26.903 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:26.903+0000 7f283b200640 1 -- 172.21.15.19:0/2051283766 shutdown_connections 2024-03-18T15:17:26.903 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:26.903+0000 7f283b200640 1 --2- 172.21.15.19:0/2051283766 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f28341097d0 0x7f2834110060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:26.903 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:26.903+0000 7f283b200640 1 --2- 172.21.15.19:0/2051283766 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2834108e10 0x7f2834109290 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:26.903 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:26.903+0000 7f283b200640 1 --2- 172.21.15.19:0/2051283766 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2834107c20 0x7f2834108040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:26.903 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:26.903+0000 7f283b200640 1 -- 172.21.15.19:0/2051283766 >> 172.21.15.19:0/2051283766 conn(0x7f2834076bb0 msgr2=0x7f2834078ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:17:26.903 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:26.903+0000 7f283b200640 1 -- 172.21.15.19:0/2051283766 shutdown_connections 2024-03-18T15:17:26.904 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:26.903+0000 7f283b200640 1 -- 172.21.15.19:0/2051283766 wait complete. 2024-03-18T15:17:26.905 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:26.904+0000 7f283b200640 1 Processor -- start 2024-03-18T15:17:26.905 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:26.904+0000 7f283b200640 1 -- start start 2024-03-18T15:17:26.905 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:26.904+0000 7f283b200640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2834107c20 0x7f283419def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:26.905 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:26.904+0000 7f283b200640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2834108e10 0x7f283419e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:26.905 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:26.904+0000 7f283b200640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f28341097d0 0x7f28341a54a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:26.905 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:26.904+0000 7f283b200640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f2834113880 con 0x7f28341097d0 2024-03-18T15:17:26.905 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:26.904+0000 7f283b200640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2834113700 con 0x7f2834108e10 2024-03-18T15:17:26.906 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:26.904+0000 7f283b200640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f2834113a00 con 0x7f2834107c20 2024-03-18T15:17:26.906 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:26.905+0000 7f2839e00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2834107c20 0x7f283419def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:26.906 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:26.905+0000 7f2839400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2834108e10 0x7f283419e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:26.906 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:26.905+0000 7f283a800640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f28341097d0 0x7f28341a54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:26.906 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:26.905+0000 7f2839e00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2834107c20 0x7f283419def0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.19:42448/0 (socket says 172.21.15.19:42448) 2024-03-18T15:17:26.906 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:26.905+0000 7f2839400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2834108e10 0x7f283419e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:55854/0 (socket says 172.21.15.19:55854) 2024-03-18T15:17:26.906 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:26.905+0000 7f2839e00640 1 -- 172.21.15.19:0/3935431717 learned_addr learned my addr 172.21.15.19:0/3935431717 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:17:26.906 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:26.905+0000 7f2839400640 1 -- 172.21.15.19:0/3935431717 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2834107c20 msgr2=0x7f283419def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:26.906 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:26.905+0000 7f2839400640 1 --2- 172.21.15.19:0/3935431717 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2834107c20 0x7f283419def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:26.906 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:26.905+0000 7f2839400640 1 -- 172.21.15.19:0/3935431717 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f28341097d0 msgr2=0x7f28341a54a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:26.906 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:26.905+0000 7f2839400640 1 --2- 172.21.15.19:0/3935431717 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f28341097d0 0x7f28341a54a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:26.906 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:26.905+0000 7f2839400640 1 -- 172.21.15.19:0/3935431717 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28341a5b10 con 0x7f2834108e10 2024-03-18T15:17:26.906 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:26.905+0000 7f2839400640 1 --2- 172.21.15.19:0/3935431717 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2834108e10 0x7f283419e430 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f2824009230 tx=0x7f282402c660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:17:26.907 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:26.906+0000 7f282ea00640 1 -- 172.21.15.19:0/3935431717 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2824041070 con 0x7f2834108e10 2024-03-18T15:17:26.907 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:26.906+0000 7f282ea00640 1 -- 172.21.15.19:0/3935431717 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2824004450 con 0x7f2834108e10 2024-03-18T15:17:26.907 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:26.906+0000 7f283b200640 1 -- 172.21.15.19:0/3935431717 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28341a5da0 con 0x7f2834108e10 2024-03-18T15:17:26.907 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:26.906+0000 7f282ea00640 1 -- 172.21.15.19:0/3935431717 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2824003e50 con 0x7f2834108e10 2024-03-18T15:17:26.907 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:26.906+0000 7f283b200640 1 -- 172.21.15.19:0/3935431717 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28341a6280 con 0x7f2834108e10 2024-03-18T15:17:26.908 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:26.907+0000 7f283b200640 1 -- 172.21.15.19:0/3935431717 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f283410ef90 con 0x7f2834108e10 2024-03-18T15:17:26.908 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:26.909+0000 7f282ea00640 1 -- 172.21.15.19:0/3935431717 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f2824044070 con 0x7f2834108e10 2024-03-18T15:17:26.910 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:26.910+0000 7f282ea00640 1 --2- 172.21.15.19:0/3935431717 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f281c0a2a20 0x7f281c0a4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:26.910 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:26.910+0000 7f282ea00640 1 -- 172.21.15.19:0/3935431717 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f28240f0860 con 0x7f2834108e10 2024-03-18T15:17:26.910 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:26.910+0000 7f2839e00640 1 --2- 172.21.15.19:0/3935431717 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f281c0a2a20 0x7f281c0a4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:26.911 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:26.911+0000 7f2839e00640 1 --2- 172.21.15.19:0/3935431717 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f281c0a2a20 0x7f281c0a4ee0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f28280058c0 tx=0x7f2828005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:17:26.919 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:26.919+0000 7f282ea00640 1 -- 172.21.15.19:0/3935431717 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f28240bce80 con 0x7f2834108e10 2024-03-18T15:17:27.132 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.132+0000 7f283b200640 1 -- 172.21.15.19:0/3935431717 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f28340624b0 con 0x7f2834108e10 2024-03-18T15:17:27.134 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.134+0000 7f282ea00640 1 -- 172.21.15.19:0/3935431717 <== mon.2 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24809 (secure 0 0 0) 0x7f28240c0f20 con 0x7f2834108e10 2024-03-18T15:17:27.135 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-18T15:17:27.135 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":93,"fsid":"d4dc2ba6-e538-11ee-95c9-87774f69a715","created":"2024-03-18T15:05:57.538551+0000","modified":"2024-03-18T15:17:16.033313+0000","last_up_change":"2024-03-18T15:14:12.009522+0000","last_in_change":"2024-03-18T15:13:52.555387+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T15:09:56.843349+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T15:15:08.941047+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T15:15:10.044671+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T15:15:13.621916+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"37b5a5cd-27ae-4e77-9b05-774b817b2ee9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6802","nonce":402333948},{"type":"v1","addr":"172.21.15.19:6803","nonce":402333948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6804","nonce":402333948},{"type":"v1","addr":"172.21.15.19:6805","nonce":402333948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6808","nonce":402333948},{"type":"v1","addr":"172.21.15.19:6809","nonce":402333948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6806","nonce":402333948},{"type":"v1","addr":"172.21.15.19:6807","nonce":402333948}]},"public_addr":"172.21.15.19:6803/402333948","cluster_addr":"172.21.15.19:6805/402333948","heartbeat_back_addr":"172.21.15.19:6809/402333948","heartbeat_front_addr":"172.21.15.19:6807/402333948","state":["exists","up"]},{"osd":1,"uuid":"3a1134e8-ede7-4dbd-9be1-eefd2cbb43bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1951880113},{"type":"v1","addr":"172.21.15.33:6801","nonce":1951880113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1951880113},{"type":"v1","addr":"172.21.15.33:6803","nonce":1951880113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1951880113},{"type":"v1","addr":"172.21.15.33:6807","nonce":1951880113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1951880113},{"type":"v1","addr":"172.21.15.33:6805","nonce":1951880113}]},"public_addr":"172.21.15.33:6801/1951880113","cluster_addr":"172.21.15.33:6803/1951880113","heartbeat_back_addr":"172.21.15.33:6807/1951880113","heartbeat_front_addr":"172.21.15.33:6805/1951880113","state":["exists","up"]},{"osd":2,"uuid":"f383a164-a3a0-49fd-a9e8-9c2178d8cfb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":1004727949},{"type":"v1","addr":"172.21.15.104:6801","nonce":1004727949}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":1004727949},{"type":"v1","addr":"172.21.15.104:6803","nonce":1004727949}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":1004727949},{"type":"v1","addr":"172.21.15.104:6807","nonce":1004727949}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":1004727949},{"type":"v1","addr":"172.21.15.104:6805","nonce":1004727949}]},"public_addr":"172.21.15.104:6801/1004727949","cluster_addr":"172.21.15.104:6803/1004727949","heartbeat_back_addr":"172.21.15.104:6807/1004727949","heartbeat_front_addr":"172.21.15.104:6805/1004727949","state":["exists","up"]},{"osd":3,"uuid":"c409e116-28bf-45dd-a791-539af22abe1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6810","nonce":3571525854},{"type":"v1","addr":"172.21.15.19:6811","nonce":3571525854}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6812","nonce":3571525854},{"type":"v1","addr":"172.21.15.19:6813","nonce":3571525854}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6816","nonce":3571525854},{"type":"v1","addr":"172.21.15.19:6817","nonce":3571525854}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6814","nonce":3571525854},{"type":"v1","addr":"172.21.15.19:6815","nonce":3571525854}]},"public_addr":"172.21.15.19:6811/3571525854","cluster_addr":"172.21.15.19:6813/3571525854","heartbeat_back_addr":"172.21.15.19:6817/3571525854","heartbeat_front_addr":"172.21.15.19:6815/3571525854","state":["exists","up"]},{"osd":4,"uuid":"696eb719-0b3e-43ae-b5f1-ccd73a3b2843","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":624901749},{"type":"v1","addr":"172.21.15.33:6809","nonce":624901749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":624901749},{"type":"v1","addr":"172.21.15.33:6811","nonce":624901749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":624901749},{"type":"v1","addr":"172.21.15.33:6815","nonce":624901749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":624901749},{"type":"v1","addr":"172.21.15.33:6813","nonce":624901749}]},"public_addr":"172.21.15.33:6809/624901749","cluster_addr":"172.21.15.33:6811/624901749","heartbeat_back_addr":"172.21.15.33:6815/624901749","heartbeat_front_addr":"172.21.15.33:6813/624901749","state":["exists","up"]},{"osd":5,"uuid":"21789779-8bb9-42c7-b9ee-437985f11e3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":1258208297},{"type":"v1","addr":"172.21.15.104:6809","nonce":1258208297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":1258208297},{"type":"v1","addr":"172.21.15.104:6811","nonce":1258208297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":1258208297},{"type":"v1","addr":"172.21.15.104:6815","nonce":1258208297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":1258208297},{"type":"v1","addr":"172.21.15.104:6813","nonce":1258208297}]},"public_addr":"172.21.15.104:6809/1258208297","cluster_addr":"172.21.15.104:6811/1258208297","heartbeat_back_addr":"172.21.15.104:6815/1258208297","heartbeat_front_addr":"172.21.15.104:6813/1258208297","state":["exists","up"]},{"osd":6,"uuid":"0e92a2ef-43cd-487c-9203-e219158aef8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6818","nonce":2782482301},{"type":"v1","addr":"172.21.15.19:6819","nonce":2782482301}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6820","nonce":2782482301},{"type":"v1","addr":"172.21.15.19:6821","nonce":2782482301}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6824","nonce":2782482301},{"type":"v1","addr":"172.21.15.19:6825","nonce":2782482301}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6822","nonce":2782482301},{"type":"v1","addr":"172.21.15.19:6823","nonce":2782482301}]},"public_addr":"172.21.15.19:6819/2782482301","cluster_addr":"172.21.15.19:6821/2782482301","heartbeat_back_addr":"172.21.15.19:6825/2782482301","heartbeat_front_addr":"172.21.15.19:6823/2782482301","state":["exists","up"]},{"osd":7,"uuid":"a16e39a1-3685-491d-b376-3e80a2084072","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":4238160988},{"type":"v1","addr":"172.21.15.33:6817","nonce":4238160988}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":4238160988},{"type":"v1","addr":"172.21.15.33:6819","nonce":4238160988}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":4238160988},{"type":"v1","addr":"172.21.15.33:6823","nonce":4238160988}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":4238160988},{"type":"v1","addr":"172.21.15.33:6821","nonce":4238160988}]},"public_addr":"172.21.15.33:6817/4238160988","cluster_addr":"172.21.15.33:6819/4238160988","heartbeat_back_addr":"172.21.15.33:6823/4238160988","heartbeat_front_addr":"172.21.15.33:6821/4238160988","state":["exists","up"]},{"osd":8,"uuid":"d5c3fa92-cd5d-4349-9c7d-d5666bacde16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":1014779813},{"type":"v1","addr":"172.21.15.104:6817","nonce":1014779813}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":1014779813},{"type":"v1","addr":"172.21.15.104:6819","nonce":1014779813}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":1014779813},{"type":"v1","addr":"172.21.15.104:6823","nonce":1014779813}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":1014779813},{"type":"v1","addr":"172.21.15.104:6821","nonce":1014779813}]},"public_addr":"172.21.15.104:6817/1014779813","cluster_addr":"172.21.15.104:6819/1014779813","heartbeat_back_addr":"172.21.15.104:6823/1014779813","heartbeat_front_addr":"172.21.15.104:6821/1014779813","state":["exists","up"]},{"osd":9,"uuid":"eed2e9b2-26d3-4c46-88ec-34b5596fe991","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6826","nonce":2641151009},{"type":"v1","addr":"172.21.15.19:6827","nonce":2641151009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6828","nonce":2641151009},{"type":"v1","addr":"172.21.15.19:6829","nonce":2641151009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6832","nonce":2641151009},{"type":"v1","addr":"172.21.15.19:6833","nonce":2641151009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6830","nonce":2641151009},{"type":"v1","addr":"172.21.15.19:6831","nonce":2641151009}]},"public_addr":"172.21.15.19:6827/2641151009","cluster_addr":"172.21.15.19:6829/2641151009","heartbeat_back_addr":"172.21.15.19:6833/2641151009","heartbeat_front_addr":"172.21.15.19:6831/2641151009","state":["exists","up"]},{"osd":10,"uuid":"f3c35ea5-fbe7-4dc4-93c5-b92d1a714db3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":1346058869},{"type":"v1","addr":"172.21.15.33:6825","nonce":1346058869}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":1346058869},{"type":"v1","addr":"172.21.15.33:6827","nonce":1346058869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":1346058869},{"type":"v1","addr":"172.21.15.33:6831","nonce":1346058869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":1346058869},{"type":"v1","addr":"172.21.15.33:6829","nonce":1346058869}]},"public_addr":"172.21.15.33:6825/1346058869","cluster_addr":"172.21.15.33:6827/1346058869","heartbeat_back_addr":"172.21.15.33:6831/1346058869","heartbeat_front_addr":"172.21.15.33:6829/1346058869","state":["exists","up"]},{"osd":11,"uuid":"c8b51571-f807-4b62-90b6-7529e09e59da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":3759498281},{"type":"v1","addr":"172.21.15.104:6825","nonce":3759498281}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":3759498281},{"type":"v1","addr":"172.21.15.104:6827","nonce":3759498281}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":3759498281},{"type":"v1","addr":"172.21.15.104:6831","nonce":3759498281}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":3759498281},{"type":"v1","addr":"172.21.15.104:6829","nonce":3759498281}]},"public_addr":"172.21.15.104:6825/3759498281","cluster_addr":"172.21.15.104:6827/3759498281","heartbeat_back_addr":"172.21.15.104:6831/3759498281","heartbeat_front_addr":"172.21.15.104:6829/3759498281","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:09:05.267795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:09:30.234068+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:09:53.870232+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:10:26.471296+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:10:53.180341+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:11:17.801361+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:11:51.364816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:12:18.306408+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:12:43.978744+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:13:18.213964+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:13:45.437544+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:14:10.148580+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":3,"to":6}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"3.37","mappings":[{"from":5,"to":8}]},{"pgid":"3.3a","mappings":[{"from":4,"to":7}]},{"pgid":"4.8","mappings":[{"from":5,"to":8}]},{"pgid":"4.f","mappings":[{"from":9,"to":10}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.19:0/3860783274":"2024-03-19T15:07:14.150983+0000","172.21.15.19:0/2737223213":"2024-03-19T15:07:14.150983+0000","172.21.15.19:0/176699025":"2024-03-19T15:06:38.688405+0000","172.21.15.19:0/2316969005":"2024-03-19T15:07:14.150983+0000","172.21.15.19:6801/2776008962":"2024-03-19T15:07:14.150983+0000","172.21.15.19:0/2793980440":"2024-03-19T15:06:38.688405+0000","172.21.15.19:6800/3264337803":"2024-03-19T15:06:38.688405+0000","172.21.15.19:6801/3264337803":"2024-03-19T15:06:38.688405+0000","172.21.15.19:0/3166705295":"2024-03-19T15:06:38.688405+0000","172.21.15.19:6800/2776008962":"2024-03-19T15:07:14.150983+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T15:17:27.138 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.137+0000 7f283b200640 1 -- 172.21.15.19:0/3935431717 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f281c0a2a20 msgr2=0x7f281c0a4ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:27.138 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.137+0000 7f283b200640 1 --2- 172.21.15.19:0/3935431717 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f281c0a2a20 0x7f281c0a4ee0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f28280058c0 tx=0x7f2828005f90 comp rx=0 tx=0).stop 2024-03-18T15:17:27.138 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.137+0000 7f283b200640 1 -- 172.21.15.19:0/3935431717 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2834108e10 msgr2=0x7f283419e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:27.138 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.137+0000 7f283b200640 1 --2- 172.21.15.19:0/3935431717 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2834108e10 0x7f283419e430 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f2824009230 tx=0x7f282402c660 comp rx=0 tx=0).stop 2024-03-18T15:17:27.138 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.137+0000 7f283b200640 1 -- 172.21.15.19:0/3935431717 shutdown_connections 2024-03-18T15:17:27.138 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.137+0000 7f283b200640 1 --2- 172.21.15.19:0/3935431717 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f28341097d0 0x7f28341a54a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:27.138 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.137+0000 7f283b200640 1 --2- 172.21.15.19:0/3935431717 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f281c0a2a20 0x7f281c0a4ee0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:27.138 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.137+0000 7f283b200640 1 --2- 172.21.15.19:0/3935431717 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2834108e10 0x7f283419e430 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:27.138 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.137+0000 7f283b200640 1 --2- 172.21.15.19:0/3935431717 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2834107c20 0x7f283419def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:27.138 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.137+0000 7f283b200640 1 -- 172.21.15.19:0/3935431717 >> 172.21.15.19:0/3935431717 conn(0x7f2834076bb0 msgr2=0x7f283410c160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:17:27.138 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.138+0000 7f283b200640 1 -- 172.21.15.19:0/3935431717 shutdown_connections 2024-03-18T15:17:27.138 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.138+0000 7f283b200640 1 -- 172.21.15.19:0/3935431717 wait complete. 2024-03-18T15:17:27.157 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T15:09:56.843349+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T15:15:08.941047+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T15:15:10.044671+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-18T15:15:13.621916+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-18T15:17:27.157 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T15:17:27.377 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.376+0000 7f8e51c00640 1 -- 172.21.15.19:0/2173313811 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8e4c103290 msgr2=0x7f8e4c109b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:27.377 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.376+0000 7f8e51c00640 1 --2- 172.21.15.19:0/2173313811 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8e4c103290 0x7f8e4c109b20 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f8e44009ca0 tx=0x7f8e4402c100 comp rx=0 tx=0).stop 2024-03-18T15:17:27.377 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.376+0000 7f8e51c00640 1 -- 172.21.15.19:0/2173313811 shutdown_connections 2024-03-18T15:17:27.377 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.376+0000 7f8e51c00640 1 --2- 172.21.15.19:0/2173313811 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8e4c103290 0x7f8e4c109b20 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:27.377 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.376+0000 7f8e51c00640 1 --2- 172.21.15.19:0/2173313811 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8e4c1028d0 0x7f8e4c102d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:27.377 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.376+0000 7f8e51c00640 1 --2- 172.21.15.19:0/2173313811 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8e4c1016e0 0x7f8e4c101b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:27.377 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.376+0000 7f8e51c00640 1 -- 172.21.15.19:0/2173313811 >> 172.21.15.19:0/2173313811 conn(0x7f8e4c0fcc50 msgr2=0x7f8e4c0ff090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:17:27.377 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.376+0000 7f8e51c00640 1 -- 172.21.15.19:0/2173313811 shutdown_connections 2024-03-18T15:17:27.377 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.377+0000 7f8e51c00640 1 -- 172.21.15.19:0/2173313811 wait complete. 2024-03-18T15:17:27.379 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.378+0000 7f8e51c00640 1 Processor -- start 2024-03-18T15:17:27.380 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.378+0000 7f8e51c00640 1 -- start start 2024-03-18T15:17:27.380 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.378+0000 7f8e51c00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8e4c1016e0 0x7f8e4c199b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:27.380 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.378+0000 7f8e51c00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8e4c1028d0 0x7f8e4c19a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:27.380 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.378+0000 7f8e51c00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8e4c103290 0x7f8e4c1a1110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:27.380 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.378+0000 7f8e51c00640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f8e4c10d360 con 0x7f8e4c103290 2024-03-18T15:17:27.380 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.378+0000 7f8e51c00640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8e4c10d1e0 con 0x7f8e4c1028d0 2024-03-18T15:17:27.380 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.378+0000 7f8e51c00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f8e4c10d4e0 con 0x7f8e4c1016e0 2024-03-18T15:17:27.380 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.379+0000 7f8e4be00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8e4c1016e0 0x7f8e4c199b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:27.380 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.379+0000 7f8e4be00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8e4c1016e0 0x7f8e4c199b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.19:42456/0 (socket says 172.21.15.19:42456) 2024-03-18T15:17:27.380 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.379+0000 7f8e4be00640 1 -- 172.21.15.19:0/204246719 learned_addr learned my addr 172.21.15.19:0/204246719 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:17:27.381 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.379+0000 7f8e43400640 1 --2- 172.21.15.19:0/204246719 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8e4c1028d0 0x7f8e4c19a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:27.381 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.379+0000 7f8e51200640 1 --2- 172.21.15.19:0/204246719 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8e4c103290 0x7f8e4c1a1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:27.381 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.379+0000 7f8e4be00640 1 -- 172.21.15.19:0/204246719 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8e4c1028d0 msgr2=0x7f8e4c19a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:27.381 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.379+0000 7f8e4be00640 1 --2- 172.21.15.19:0/204246719 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8e4c1028d0 0x7f8e4c19a0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:27.381 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.379+0000 7f8e4be00640 1 -- 172.21.15.19:0/204246719 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8e4c103290 msgr2=0x7f8e4c1a1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:27.381 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.379+0000 7f8e4be00640 1 --2- 172.21.15.19:0/204246719 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8e4c103290 0x7f8e4c1a1110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:27.381 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.379+0000 7f8e4be00640 1 -- 172.21.15.19:0/204246719 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e4c1a1780 con 0x7f8e4c1016e0 2024-03-18T15:17:27.381 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.379+0000 7f8e4be00640 1 --2- 172.21.15.19:0/204246719 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8e4c1016e0 0x7f8e4c199b60 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f8e3c00e9c0 tx=0x7f8e3c00ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:17:27.381 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.380+0000 7f8e49600640 1 -- 172.21.15.19:0/204246719 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e3c00cd80 con 0x7f8e4c1016e0 2024-03-18T15:17:27.381 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.380+0000 7f8e49600640 1 -- 172.21.15.19:0/204246719 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e3c0035f0 con 0x7f8e4c1016e0 2024-03-18T15:17:27.381 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.380+0000 7f8e49600640 1 -- 172.21.15.19:0/204246719 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e3c010690 con 0x7f8e4c1016e0 2024-03-18T15:17:27.382 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.380+0000 7f8e51c00640 1 -- 172.21.15.19:0/204246719 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e4c1a1a70 con 0x7f8e4c1016e0 2024-03-18T15:17:27.382 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.380+0000 7f8e51c00640 1 -- 172.21.15.19:0/204246719 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e4c1a1fc0 con 0x7f8e4c1016e0 2024-03-18T15:17:27.382 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.381+0000 7f8e51c00640 1 -- 172.21.15.19:0/204246719 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e4c1a2250 con 0x7f8e4c1016e0 2024-03-18T15:17:27.383 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.383+0000 7f8e49600640 1 -- 172.21.15.19:0/204246719 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f8e3c0037a0 con 0x7f8e4c1016e0 2024-03-18T15:17:27.384 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.384+0000 7f8e49600640 1 --2- 172.21.15.19:0/204246719 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f8e300a2880 0x7f8e300a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:27.385 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.385+0000 7f8e49600640 1 -- 172.21.15.19:0/204246719 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f8e3c0cf100 con 0x7f8e4c1016e0 2024-03-18T15:17:27.385 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.385+0000 7f8e43400640 1 --2- 172.21.15.19:0/204246719 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f8e300a2880 0x7f8e300a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:27.385 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.385+0000 7f8e43400640 1 --2- 172.21.15.19:0/204246719 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f8e300a2880 0x7f8e300a4d40 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f8e4c19b070 tx=0x7f8e3400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:17:27.389 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.389+0000 7f8e49600640 1 -- 172.21.15.19:0/204246719 <== mon.1 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8e3c09bdd0 con 0x7f8e4c1016e0 2024-03-18T15:17:27.592 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.592+0000 7f8e51c00640 1 -- 172.21.15.19:0/204246719 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f8e4c1a2420 con 0x7f8e4c1016e0 2024-03-18T15:17:27.592 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.593+0000 7f8e49600640 1 -- 172.21.15.19:0/204246719 <== mon.1 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7f8e3c09b6d0 con 0x7f8e4c1016e0 2024-03-18T15:17:27.593 INFO:teuthology.orchestra.run.smithi019.stdout:pg_num: 1 2024-03-18T15:17:27.595 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.595+0000 7f8e51c00640 1 -- 172.21.15.19:0/204246719 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f8e300a2880 msgr2=0x7f8e300a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:27.596 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.595+0000 7f8e51c00640 1 --2- 172.21.15.19:0/204246719 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f8e300a2880 0x7f8e300a4d40 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f8e4c19b070 tx=0x7f8e3400b410 comp rx=0 tx=0).stop 2024-03-18T15:17:27.596 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.595+0000 7f8e51c00640 1 -- 172.21.15.19:0/204246719 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8e4c1016e0 msgr2=0x7f8e4c199b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:27.596 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.595+0000 7f8e51c00640 1 --2- 172.21.15.19:0/204246719 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8e4c1016e0 0x7f8e4c199b60 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f8e3c00e9c0 tx=0x7f8e3c00ee80 comp rx=0 tx=0).stop 2024-03-18T15:17:27.596 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.595+0000 7f8e51c00640 1 -- 172.21.15.19:0/204246719 shutdown_connections 2024-03-18T15:17:27.596 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.595+0000 7f8e51c00640 1 --2- 172.21.15.19:0/204246719 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8e4c103290 0x7f8e4c1a1110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:27.596 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.595+0000 7f8e51c00640 1 --2- 172.21.15.19:0/204246719 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f8e300a2880 0x7f8e300a4d40 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:27.596 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.595+0000 7f8e51c00640 1 --2- 172.21.15.19:0/204246719 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8e4c1028d0 0x7f8e4c19a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:27.596 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.595+0000 7f8e51c00640 1 --2- 172.21.15.19:0/204246719 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8e4c1016e0 0x7f8e4c199b60 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:27.596 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.595+0000 7f8e51c00640 1 -- 172.21.15.19:0/204246719 >> 172.21.15.19:0/204246719 conn(0x7f8e4c0fcc50 msgr2=0x7f8e4c073a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:17:27.596 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.596+0000 7f8e51c00640 1 -- 172.21.15.19:0/204246719 shutdown_connections 2024-03-18T15:17:27.596 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.596+0000 7f8e51c00640 1 -- 172.21.15.19:0/204246719 wait complete. 2024-03-18T15:17:27.616 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T15:17:27.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:17:27 smithi104 ceph-mon[36070]: pgmap v389: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 297 B/s wr, 10 op/s 2024-03-18T15:17:27.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:17:27 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/3935431717' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:17:27.703 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:17:27 smithi019 ceph-mon[31522]: pgmap v389: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 297 B/s wr, 10 op/s 2024-03-18T15:17:27.703 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:17:27 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/3935431717' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:17:27.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:17:27 smithi033 ceph-mon[36648]: pgmap v389: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 297 B/s wr, 10 op/s 2024-03-18T15:17:27.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:17:27 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/3935431717' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:17:27.850 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.850+0000 7f27a5600640 1 -- 172.21.15.19:0/4281451188 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f27a01028d0 msgr2=0x7f27a0102d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:27.850 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.850+0000 7f27a5600640 1 --2- 172.21.15.19:0/4281451188 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f27a01028d0 0x7f27a0102d50 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f278c009900 tx=0x7f278c02bc40 comp rx=0 tx=0).stop 2024-03-18T15:17:27.850 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.850+0000 7f27a5600640 1 -- 172.21.15.19:0/4281451188 shutdown_connections 2024-03-18T15:17:27.850 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.850+0000 7f27a5600640 1 --2- 172.21.15.19:0/4281451188 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f27a0103290 0x7f27a0109b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:27.851 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.850+0000 7f27a5600640 1 --2- 172.21.15.19:0/4281451188 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f27a01028d0 0x7f27a0102d50 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:27.851 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.850+0000 7f27a5600640 1 --2- 172.21.15.19:0/4281451188 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f27a01016e0 0x7f27a0101b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:27.851 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.850+0000 7f27a5600640 1 -- 172.21.15.19:0/4281451188 >> 172.21.15.19:0/4281451188 conn(0x7f27a00fcc50 msgr2=0x7f27a00ff090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:17:27.851 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.850+0000 7f27a5600640 1 -- 172.21.15.19:0/4281451188 shutdown_connections 2024-03-18T15:17:27.851 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.850+0000 7f27a5600640 1 -- 172.21.15.19:0/4281451188 wait complete. 2024-03-18T15:17:27.853 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.851+0000 7f27a5600640 1 Processor -- start 2024-03-18T15:17:27.853 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.851+0000 7f27a5600640 1 -- start start 2024-03-18T15:17:27.853 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.851+0000 7f27a5600640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f27a01016e0 0x7f27a01979d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:27.853 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.851+0000 7f27a5600640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f27a01028d0 0x7f27a0197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:27.853 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.851+0000 7f27a5600640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f27a0103290 0x7f27a019ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:27.853 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.851+0000 7f27a5600640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f27a010d390 con 0x7f27a01016e0 2024-03-18T15:17:27.853 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.851+0000 7f27a5600640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f27a010d210 con 0x7f27a0103290 2024-03-18T15:17:27.853 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.851+0000 7f27a5600640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f27a010d510 con 0x7f27a01028d0 2024-03-18T15:17:27.854 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.852+0000 7f279f400640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f27a01028d0 0x7f27a0197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:27.854 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.852+0000 7f279f400640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f27a01028d0 0x7f27a0197f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.19:42476/0 (socket says 172.21.15.19:42476) 2024-03-18T15:17:27.854 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.852+0000 7f279f400640 1 -- 172.21.15.19:0/2896153346 learned_addr learned my addr 172.21.15.19:0/2896153346 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:17:27.854 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.852+0000 7f27a4c00640 1 --2- 172.21.15.19:0/2896153346 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f27a0103290 0x7f27a019ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:27.854 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.852+0000 7f279fe00640 1 --2- 172.21.15.19:0/2896153346 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f27a01016e0 0x7f27a01979d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:27.854 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.852+0000 7f279f400640 1 -- 172.21.15.19:0/2896153346 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f27a0103290 msgr2=0x7f27a019ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:27.854 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.852+0000 7f279f400640 1 --2- 172.21.15.19:0/2896153346 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f27a0103290 0x7f27a019ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:27.854 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.852+0000 7f279f400640 1 -- 172.21.15.19:0/2896153346 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f27a01016e0 msgr2=0x7f27a01979d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:27.854 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.852+0000 7f279f400640 1 --2- 172.21.15.19:0/2896153346 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f27a01016e0 0x7f27a01979d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:27.854 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.852+0000 7f279f400640 1 -- 172.21.15.19:0/2896153346 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27a019f5f0 con 0x7f27a01028d0 2024-03-18T15:17:27.854 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.852+0000 7f27a4c00640 1 --2- 172.21.15.19:0/2896153346 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f27a0103290 0x7f27a019ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:17:27.854 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.853+0000 7f279f400640 1 --2- 172.21.15.19:0/2896153346 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f27a01028d0 0x7f27a0197f10 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f278c02c8f0 tx=0x7f278c02c660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:17:27.854 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.853+0000 7f279cc00640 1 -- 172.21.15.19:0/2896153346 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f278c041070 con 0x7f27a01028d0 2024-03-18T15:17:27.854 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.853+0000 7f279cc00640 1 -- 172.21.15.19:0/2896153346 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f278c004450 con 0x7f27a01028d0 2024-03-18T15:17:27.855 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.853+0000 7f279cc00640 1 -- 172.21.15.19:0/2896153346 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f278c034440 con 0x7f27a01028d0 2024-03-18T15:17:27.855 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.853+0000 7f27a5600640 1 -- 172.21.15.19:0/2896153346 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27a019f880 con 0x7f27a01028d0 2024-03-18T15:17:27.855 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.853+0000 7f27a5600640 1 -- 172.21.15.19:0/2896153346 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27a019fdc0 con 0x7f27a01028d0 2024-03-18T15:17:27.855 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.854+0000 7f27a5600640 1 -- 172.21.15.19:0/2896153346 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27a0108a00 con 0x7f27a01028d0 2024-03-18T15:17:27.856 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.856+0000 7f279cc00640 1 -- 172.21.15.19:0/2896153346 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f278c044070 con 0x7f27a01028d0 2024-03-18T15:17:27.857 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.857+0000 7f279cc00640 1 --2- 172.21.15.19:0/2896153346 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f27840a2880 0x7f27840a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:27.857 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.857+0000 7f279cc00640 1 -- 172.21.15.19:0/2896153346 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f278c0f0cc0 con 0x7f27a01028d0 2024-03-18T15:17:27.857 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.858+0000 7f279fe00640 1 --2- 172.21.15.19:0/2896153346 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f27840a2880 0x7f27840a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:27.858 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.858+0000 7f279fe00640 1 --2- 172.21.15.19:0/2896153346 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f27840a2880 0x7f27840a4d40 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f27a0102730 tx=0x7f2790009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:17:27.864 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:27.864+0000 7f279cc00640 1 -- 172.21.15.19:0/2896153346 <== mon.1 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f278c0bd2e0 con 0x7f27a01028d0 2024-03-18T15:17:28.069 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.069+0000 7f27a5600640 1 -- 172.21.15.19:0/2896153346 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f27a002d340 con 0x7f27a01028d0 2024-03-18T15:17:28.070 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.070+0000 7f279cc00640 1 -- 172.21.15.19:0/2896153346 <== mon.1 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7f278c0c1380 con 0x7f27a01028d0 2024-03-18T15:17:28.070 INFO:teuthology.orchestra.run.smithi019.stdout:pg_num: 32 2024-03-18T15:17:28.073 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.073+0000 7f27a5600640 1 -- 172.21.15.19:0/2896153346 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f27840a2880 msgr2=0x7f27840a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:28.073 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.073+0000 7f27a5600640 1 --2- 172.21.15.19:0/2896153346 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f27840a2880 0x7f27840a4d40 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f27a0102730 tx=0x7f2790009500 comp rx=0 tx=0).stop 2024-03-18T15:17:28.073 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.073+0000 7f27a5600640 1 -- 172.21.15.19:0/2896153346 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f27a01028d0 msgr2=0x7f27a0197f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:28.073 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.073+0000 7f27a5600640 1 --2- 172.21.15.19:0/2896153346 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f27a01028d0 0x7f27a0197f10 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f278c02c8f0 tx=0x7f278c02c660 comp rx=0 tx=0).stop 2024-03-18T15:17:28.073 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.073+0000 7f27a5600640 1 -- 172.21.15.19:0/2896153346 shutdown_connections 2024-03-18T15:17:28.073 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.073+0000 7f27a5600640 1 --2- 172.21.15.19:0/2896153346 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f27840a2880 0x7f27840a4d40 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:28.073 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.073+0000 7f27a5600640 1 --2- 172.21.15.19:0/2896153346 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f27a0103290 0x7f27a019ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:28.073 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.073+0000 7f27a5600640 1 --2- 172.21.15.19:0/2896153346 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f27a01028d0 0x7f27a0197f10 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:28.074 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.073+0000 7f27a5600640 1 --2- 172.21.15.19:0/2896153346 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f27a01016e0 0x7f27a01979d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:28.074 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.073+0000 7f27a5600640 1 -- 172.21.15.19:0/2896153346 >> 172.21.15.19:0/2896153346 conn(0x7f27a00fcc50 msgr2=0x7f27a0105bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:17:28.074 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.073+0000 7f27a5600640 1 -- 172.21.15.19:0/2896153346 shutdown_connections 2024-03-18T15:17:28.074 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.073+0000 7f27a5600640 1 -- 172.21.15.19:0/2896153346 wait complete. 2024-03-18T15:17:28.094 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T15:17:28.317 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.317+0000 7f8385000640 1 -- 172.21.15.19:0/2823756375 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8380108ff0 msgr2=0x7f8380109470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:28.318 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.317+0000 7f8385000640 1 --2- 172.21.15.19:0/2823756375 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8380108ff0 0x7f8380109470 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f836c009900 tx=0x7f836c02bc40 comp rx=0 tx=0).stop 2024-03-18T15:17:28.318 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.317+0000 7f8385000640 1 -- 172.21.15.19:0/2823756375 shutdown_connections 2024-03-18T15:17:28.318 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.317+0000 7f8385000640 1 --2- 172.21.15.19:0/2823756375 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f83801099b0 0x7f8380110240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:28.318 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.317+0000 7f8385000640 1 --2- 172.21.15.19:0/2823756375 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8380108ff0 0x7f8380109470 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:28.318 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.317+0000 7f8385000640 1 --2- 172.21.15.19:0/2823756375 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8380107e00 0x7f8380108220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:28.318 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.317+0000 7f8385000640 1 -- 172.21.15.19:0/2823756375 >> 172.21.15.19:0/2823756375 conn(0x7f8380076da0 msgr2=0x7f83800791e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:17:28.318 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.317+0000 7f8385000640 1 -- 172.21.15.19:0/2823756375 shutdown_connections 2024-03-18T15:17:28.318 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.317+0000 7f8385000640 1 -- 172.21.15.19:0/2823756375 wait complete. 2024-03-18T15:17:28.320 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.319+0000 7f8385000640 1 Processor -- start 2024-03-18T15:17:28.320 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.319+0000 7f8385000640 1 -- start start 2024-03-18T15:17:28.321 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.319+0000 7f8385000640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8380107e00 0x7f838019e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:28.321 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.319+0000 7f8385000640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8380108ff0 0x7f838019e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:28.321 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.319+0000 7f8385000640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f83801099b0 0x7f83801a5600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:28.321 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.319+0000 7f8385000640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f8380113ad0 con 0x7f8380108ff0 2024-03-18T15:17:28.321 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.319+0000 7f8385000640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8380113950 con 0x7f8380107e00 2024-03-18T15:17:28.321 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.319+0000 7f8385000640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f8380113c50 con 0x7f83801099b0 2024-03-18T15:17:28.321 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.319+0000 7f837de00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8380107e00 0x7f838019e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:28.321 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.319+0000 7f837e800640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f83801099b0 0x7f83801a5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:28.321 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.319+0000 7f837e800640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f83801099b0 0x7f83801a5600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.19:42490/0 (socket says 172.21.15.19:42490) 2024-03-18T15:17:28.321 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.319+0000 7f837de00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8380107e00 0x7f838019e050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:55900/0 (socket says 172.21.15.19:55900) 2024-03-18T15:17:28.321 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.319+0000 7f837e800640 1 -- 172.21.15.19:0/2238199329 learned_addr learned my addr 172.21.15.19:0/2238199329 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:17:28.321 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.320+0000 7f837d400640 1 --2- 172.21.15.19:0/2238199329 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8380108ff0 0x7f838019e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:28.322 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.320+0000 7f837de00640 1 -- 172.21.15.19:0/2238199329 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f83801099b0 msgr2=0x7f83801a5600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:28.322 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.320+0000 7f837de00640 1 --2- 172.21.15.19:0/2238199329 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f83801099b0 0x7f83801a5600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:28.322 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.320+0000 7f837de00640 1 -- 172.21.15.19:0/2238199329 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8380108ff0 msgr2=0x7f838019e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:28.322 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.320+0000 7f837de00640 1 --2- 172.21.15.19:0/2238199329 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8380108ff0 0x7f838019e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:28.322 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.320+0000 7f837de00640 1 -- 172.21.15.19:0/2238199329 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83801a5c70 con 0x7f8380107e00 2024-03-18T15:17:28.322 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.320+0000 7f837e800640 1 --2- 172.21.15.19:0/2238199329 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f83801099b0 0x7f83801a5600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:17:28.322 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.320+0000 7f837de00640 1 --2- 172.21.15.19:0/2238199329 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8380107e00 0x7f838019e050 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f837000c940 tx=0x7f837000ce00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:17:28.322 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.320+0000 7f8376a00640 1 -- 172.21.15.19:0/2238199329 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8370007bb0 con 0x7f8380107e00 2024-03-18T15:17:28.322 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.320+0000 7f8376a00640 1 -- 172.21.15.19:0/2238199329 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8370007d50 con 0x7f8380107e00 2024-03-18T15:17:28.322 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.320+0000 7f8385000640 1 -- 172.21.15.19:0/2238199329 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83801a5f60 con 0x7f8380107e00 2024-03-18T15:17:28.322 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.321+0000 7f8376a00640 1 -- 172.21.15.19:0/2238199329 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8370018790 con 0x7f8380107e00 2024-03-18T15:17:28.322 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.321+0000 7f8385000640 1 -- 172.21.15.19:0/2238199329 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83801a63d0 con 0x7f8380107e00 2024-03-18T15:17:28.323 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.322+0000 7f8385000640 1 -- 172.21.15.19:0/2238199329 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f838010f050 con 0x7f8380107e00 2024-03-18T15:17:28.325 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.325+0000 7f8376a00640 1 -- 172.21.15.19:0/2238199329 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f837000f450 con 0x7f8380107e00 2024-03-18T15:17:28.326 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.326+0000 7f8376a00640 1 --2- 172.21.15.19:0/2238199329 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f83640a2880 0x7f83640a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:28.326 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.326+0000 7f8376a00640 1 -- 172.21.15.19:0/2238199329 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f83700cefa0 con 0x7f8380107e00 2024-03-18T15:17:28.326 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.327+0000 7f837d400640 1 --2- 172.21.15.19:0/2238199329 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f83640a2880 0x7f83640a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:28.327 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.327+0000 7f837d400640 1 --2- 172.21.15.19:0/2238199329 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f83640a2880 0x7f83640a4d40 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f836c02c250 tx=0x7f836c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:17:28.328 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.328+0000 7f8376a00640 1 -- 172.21.15.19:0/2238199329 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8370014030 con 0x7f8380107e00 2024-03-18T15:17:28.538 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.538+0000 7f8385000640 1 -- 172.21.15.19:0/2238199329 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f838002d410 con 0x7f8380107e00 2024-03-18T15:17:28.539 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.539+0000 7f8376a00640 1 -- 172.21.15.19:0/2238199329 <== mon.2 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7f837009bd20 con 0x7f8380107e00 2024-03-18T15:17:28.539 INFO:teuthology.orchestra.run.smithi019.stdout:pg_num: 64 2024-03-18T15:17:28.542 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.542+0000 7f8385000640 1 -- 172.21.15.19:0/2238199329 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f83640a2880 msgr2=0x7f83640a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:28.542 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.542+0000 7f8385000640 1 --2- 172.21.15.19:0/2238199329 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f83640a2880 0x7f83640a4d40 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f836c02c250 tx=0x7f836c005fd0 comp rx=0 tx=0).stop 2024-03-18T15:17:28.542 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.542+0000 7f8385000640 1 -- 172.21.15.19:0/2238199329 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8380107e00 msgr2=0x7f838019e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:28.543 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.542+0000 7f8385000640 1 --2- 172.21.15.19:0/2238199329 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8380107e00 0x7f838019e050 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f837000c940 tx=0x7f837000ce00 comp rx=0 tx=0).stop 2024-03-18T15:17:28.543 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.542+0000 7f8385000640 1 -- 172.21.15.19:0/2238199329 shutdown_connections 2024-03-18T15:17:28.543 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.542+0000 7f8385000640 1 --2- 172.21.15.19:0/2238199329 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f83801099b0 0x7f83801a5600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:28.543 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.542+0000 7f8385000640 1 --2- 172.21.15.19:0/2238199329 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8380108ff0 0x7f838019e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:28.543 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.542+0000 7f8385000640 1 --2- 172.21.15.19:0/2238199329 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f83640a2880 0x7f83640a4d40 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:28.543 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.542+0000 7f8385000640 1 --2- 172.21.15.19:0/2238199329 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8380107e00 0x7f838019e050 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:28.543 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.542+0000 7f8385000640 1 -- 172.21.15.19:0/2238199329 >> 172.21.15.19:0/2238199329 conn(0x7f8380076da0 msgr2=0x7f838010c220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:17:28.543 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.542+0000 7f8385000640 1 -- 172.21.15.19:0/2238199329 shutdown_connections 2024-03-18T15:17:28.543 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.543+0000 7f8385000640 1 -- 172.21.15.19:0/2238199329 wait complete. 2024-03-18T15:17:28.562 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-18T15:17:28.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:17:28 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/204246719' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T15:17:28.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:17:28 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/2896153346' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T15:17:28.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:17:28 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/204246719' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T15:17:28.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:17:28 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/2896153346' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T15:17:28.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:17:28 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/204246719' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T15:17:28.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:17:28 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/2896153346' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T15:17:28.775 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.774+0000 7f8d7fe00640 1 -- 172.21.15.19:0/1676119757 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8d80108ef0 msgr2=0x7f8d80109370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:28.775 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.774+0000 7f8d7fe00640 1 --2- 172.21.15.19:0/1676119757 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8d80108ef0 0x7f8d80109370 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f8d68009900 tx=0x7f8d6802bc50 comp rx=0 tx=0).stop 2024-03-18T15:17:28.775 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.775+0000 7f8d7fe00640 1 -- 172.21.15.19:0/1676119757 shutdown_connections 2024-03-18T15:17:28.775 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.775+0000 7f8d7fe00640 1 --2- 172.21.15.19:0/1676119757 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8d801098b0 0x7f8d80110140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:28.775 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.775+0000 7f8d7fe00640 1 --2- 172.21.15.19:0/1676119757 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8d80108ef0 0x7f8d80109370 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:28.775 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.775+0000 7f8d7fe00640 1 --2- 172.21.15.19:0/1676119757 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8d80107d00 0x7f8d80108120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:28.775 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.775+0000 7f8d7fe00640 1 -- 172.21.15.19:0/1676119757 >> 172.21.15.19:0/1676119757 conn(0x7f8d80076cb0 msgr2=0x7f8d800790d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:17:28.775 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.775+0000 7f8d7fe00640 1 -- 172.21.15.19:0/1676119757 shutdown_connections 2024-03-18T15:17:28.775 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.775+0000 7f8d7fe00640 1 -- 172.21.15.19:0/1676119757 wait complete. 2024-03-18T15:17:28.776 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.776+0000 7f8d7fe00640 1 Processor -- start 2024-03-18T15:17:28.776 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.776+0000 7f8d7fe00640 1 -- start start 2024-03-18T15:17:28.777 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.776+0000 7f8d7fe00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8d80107d00 0x7f8d8019dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:28.777 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.776+0000 7f8d7fe00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8d80108ef0 0x7f8d8019e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:28.777 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.777+0000 7f8d7fe00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8d801098b0 0x7f8d801a5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:28.777 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.777+0000 7f8d7fe00640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f8d801139d0 con 0x7f8d80108ef0 2024-03-18T15:17:28.777 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.777+0000 7f8d7fe00640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8d80113850 con 0x7f8d801098b0 2024-03-18T15:17:28.777 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.777+0000 7f8d7fe00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f8d80113b50 con 0x7f8d80107d00 2024-03-18T15:17:28.778 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.777+0000 7f8d7ea00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8d80107d00 0x7f8d8019dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:28.778 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.777+0000 7f8d7f400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8d801098b0 0x7f8d801a5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:28.778 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.777+0000 7f8d7ea00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8d80107d00 0x7f8d8019dfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.19:42520/0 (socket says 172.21.15.19:42520) 2024-03-18T15:17:28.778 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.777+0000 7f8d7ea00640 1 -- 172.21.15.19:0/3776628814 learned_addr learned my addr 172.21.15.19:0/3776628814 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:17:28.778 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.777+0000 7f8d77e00640 1 --2- 172.21.15.19:0/3776628814 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8d80108ef0 0x7f8d8019e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:28.778 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.777+0000 7f8d7ea00640 1 -- 172.21.15.19:0/3776628814 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8d801098b0 msgr2=0x7f8d801a5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:28.778 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.777+0000 7f8d7ea00640 1 --2- 172.21.15.19:0/3776628814 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8d801098b0 0x7f8d801a5560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:28.778 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.777+0000 7f8d7ea00640 1 -- 172.21.15.19:0/3776628814 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8d80108ef0 msgr2=0x7f8d8019e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:28.778 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.777+0000 7f8d7ea00640 1 --2- 172.21.15.19:0/3776628814 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8d80108ef0 0x7f8d8019e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:28.778 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.777+0000 7f8d7ea00640 1 -- 172.21.15.19:0/3776628814 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d801a5bd0 con 0x7f8d80107d00 2024-03-18T15:17:28.778 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.778+0000 7f8d7ea00640 1 --2- 172.21.15.19:0/3776628814 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8d80107d00 0x7f8d8019dfb0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f8d7000e9c0 tx=0x7f8d7000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:17:28.779 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.778+0000 7f8d77400640 1 -- 172.21.15.19:0/3776628814 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d7000cd80 con 0x7f8d80107d00 2024-03-18T15:17:28.779 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.778+0000 7f8d77400640 1 -- 172.21.15.19:0/3776628814 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d700035f0 con 0x7f8d80107d00 2024-03-18T15:17:28.779 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.778+0000 7f8d77400640 1 -- 172.21.15.19:0/3776628814 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d70010690 con 0x7f8d80107d00 2024-03-18T15:17:28.779 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.778+0000 7f8d7fe00640 1 -- 172.21.15.19:0/3776628814 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d801a5ec0 con 0x7f8d80107d00 2024-03-18T15:17:28.779 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.778+0000 7f8d7fe00640 1 -- 172.21.15.19:0/3776628814 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d801a6320 con 0x7f8d80107d00 2024-03-18T15:17:28.779 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.779+0000 7f8d7fe00640 1 -- 172.21.15.19:0/3776628814 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d8010f020 con 0x7f8d80107d00 2024-03-18T15:17:28.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.781+0000 7f8d77400640 1 -- 172.21.15.19:0/3776628814 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f8d700037a0 con 0x7f8d80107d00 2024-03-18T15:17:28.782 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.782+0000 7f8d77400640 1 --2- 172.21.15.19:0/3776628814 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f8d640a2800 0x7f8d640a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:28.782 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.783+0000 7f8d77400640 1 -- 172.21.15.19:0/3776628814 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f8d700cf2d0 con 0x7f8d80107d00 2024-03-18T15:17:28.783 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.783+0000 7f8d77e00640 1 --2- 172.21.15.19:0/3776628814 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f8d640a2800 0x7f8d640a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:28.783 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.783+0000 7f8d77e00640 1 --2- 172.21.15.19:0/3776628814 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f8d640a2800 0x7f8d640a4cc0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f8d6802c260 tx=0x7f8d68005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:17:28.791 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.791+0000 7f8d77400640 1 -- 172.21.15.19:0/3776628814 <== mon.1 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8d7009bf40 con 0x7f8d80107d00 2024-03-18T15:17:28.993 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.993+0000 7f8d7fe00640 1 -- 172.21.15.19:0/3776628814 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f8d8002d1b0 con 0x7f8d80107d00 2024-03-18T15:17:28.994 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.994+0000 7f8d77400640 1 -- 172.21.15.19:0/3776628814 <== mon.1 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7f8d8002d1b0 con 0x7f8d80107d00 2024-03-18T15:17:28.994 INFO:teuthology.orchestra.run.smithi019.stdout:pg_num: 32 2024-03-18T15:17:28.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.997+0000 7f8d7fe00640 1 -- 172.21.15.19:0/3776628814 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f8d640a2800 msgr2=0x7f8d640a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:28.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.997+0000 7f8d7fe00640 1 --2- 172.21.15.19:0/3776628814 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f8d640a2800 0x7f8d640a4cc0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f8d6802c260 tx=0x7f8d68005fd0 comp rx=0 tx=0).stop 2024-03-18T15:17:28.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.997+0000 7f8d7fe00640 1 -- 172.21.15.19:0/3776628814 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8d80107d00 msgr2=0x7f8d8019dfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:28.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.997+0000 7f8d7fe00640 1 --2- 172.21.15.19:0/3776628814 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8d80107d00 0x7f8d8019dfb0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f8d7000e9c0 tx=0x7f8d7000ee80 comp rx=0 tx=0).stop 2024-03-18T15:17:28.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.997+0000 7f8d7fe00640 1 -- 172.21.15.19:0/3776628814 shutdown_connections 2024-03-18T15:17:28.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.997+0000 7f8d7fe00640 1 --2- 172.21.15.19:0/3776628814 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f8d640a2800 0x7f8d640a4cc0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:28.998 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.997+0000 7f8d7fe00640 1 --2- 172.21.15.19:0/3776628814 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8d801098b0 0x7f8d801a5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:28.998 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.997+0000 7f8d7fe00640 1 --2- 172.21.15.19:0/3776628814 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8d80108ef0 0x7f8d8019e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:28.998 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.997+0000 7f8d7fe00640 1 --2- 172.21.15.19:0/3776628814 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8d80107d00 0x7f8d8019dfb0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:28.998 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.997+0000 7f8d7fe00640 1 -- 172.21.15.19:0/3776628814 >> 172.21.15.19:0/3776628814 conn(0x7f8d80076cb0 msgr2=0x7f8d8010c1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:17:28.998 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.997+0000 7f8d7fe00640 1 -- 172.21.15.19:0/3776628814 shutdown_connections 2024-03-18T15:17:28.998 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:28.997+0000 7f8d7fe00640 1 -- 172.21.15.19:0/3776628814 wait complete. 2024-03-18T15:17:29.018 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T15:17:29.245 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:29.245+0000 7f1721800640 1 -- 172.21.15.19:0/2025997664 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f171c104bf0 msgr2=0x7f171c105070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:29.245 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:29.245+0000 7f1721800640 1 --2- 172.21.15.19:0/2025997664 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f171c104bf0 0x7f171c105070 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f1704009900 tx=0x7f170402bc30 comp rx=0 tx=0).stop 2024-03-18T15:17:29.246 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:29.245+0000 7f1721800640 1 -- 172.21.15.19:0/2025997664 shutdown_connections 2024-03-18T15:17:29.246 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:29.245+0000 7f1721800640 1 --2- 172.21.15.19:0/2025997664 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f171c1055b0 0x7f171c10be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:29.246 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:29.245+0000 7f1721800640 1 --2- 172.21.15.19:0/2025997664 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f171c104bf0 0x7f171c105070 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:29.246 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:29.245+0000 7f1721800640 1 --2- 172.21.15.19:0/2025997664 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f171c103a00 0x7f171c103e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:29.246 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:29.245+0000 7f1721800640 1 -- 172.21.15.19:0/2025997664 >> 172.21.15.19:0/2025997664 conn(0x7f171c0fef90 msgr2=0x7f171c1013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:17:29.246 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:29.245+0000 7f1721800640 1 -- 172.21.15.19:0/2025997664 shutdown_connections 2024-03-18T15:17:29.246 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:29.246+0000 7f1721800640 1 -- 172.21.15.19:0/2025997664 wait complete. 2024-03-18T15:17:29.248 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:29.246+0000 7f1721800640 1 Processor -- start 2024-03-18T15:17:29.248 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:29.247+0000 7f1721800640 1 -- start start 2024-03-18T15:17:29.248 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:29.247+0000 7f1721800640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f171c103a00 0x7f171c199cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:29.248 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:29.247+0000 7f1721800640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f171c104bf0 0x7f171c19a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:29.248 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:29.247+0000 7f1721800640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f171c1055b0 0x7f171c1a1250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:29.249 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:29.247+0000 7f1721800640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f171c10f6d0 con 0x7f171c103a00 2024-03-18T15:17:29.249 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:29.247+0000 7f1721800640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f171c10f550 con 0x7f171c104bf0 2024-03-18T15:17:29.249 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:29.247+0000 7f1721800640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f171c10f850 con 0x7f171c1055b0 2024-03-18T15:17:29.249 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:29.247+0000 7f171aa00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f171c103a00 0x7f171c199cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:29.249 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:29.247+0000 7f171a000640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f171c104bf0 0x7f171c19a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:29.249 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:29.247+0000 7f171a000640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f171c104bf0 0x7f171c19a1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:55930/0 (socket says 172.21.15.19:55930) 2024-03-18T15:17:29.249 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:29.247+0000 7f171a000640 1 -- 172.21.15.19:0/2495148428 learned_addr learned my addr 172.21.15.19:0/2495148428 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:17:29.249 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:29.247+0000 7f171b400640 1 --2- 172.21.15.19:0/2495148428 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f171c1055b0 0x7f171c1a1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:29.249 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:29.248+0000 7f171a000640 1 -- 172.21.15.19:0/2495148428 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f171c1055b0 msgr2=0x7f171c1a1250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:29.249 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:29.248+0000 7f171a000640 1 --2- 172.21.15.19:0/2495148428 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f171c1055b0 0x7f171c1a1250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:29.249 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:29.248+0000 7f171a000640 1 -- 172.21.15.19:0/2495148428 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f171c103a00 msgr2=0x7f171c199cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:29.249 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:29.248+0000 7f171a000640 1 --2- 172.21.15.19:0/2495148428 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f171c103a00 0x7f171c199cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:29.249 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:29.248+0000 7f171a000640 1 -- 172.21.15.19:0/2495148428 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f171c1a18f0 con 0x7f171c104bf0 2024-03-18T15:17:29.249 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:29.248+0000 7f171b400640 1 --2- 172.21.15.19:0/2495148428 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f171c1055b0 0x7f171c1a1250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T15:17:29.250 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:29.248+0000 7f171aa00640 1 --2- 172.21.15.19:0/2495148428 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f171c103a00 0x7f171c199cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:17:29.250 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:29.248+0000 7f171a000640 1 --2- 172.21.15.19:0/2495148428 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f171c104bf0 0x7f171c19a1f0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f1704000c00 tx=0x7f170402c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:17:29.250 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:29.248+0000 7f1713400640 1 -- 172.21.15.19:0/2495148428 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1704043020 con 0x7f171c104bf0 2024-03-18T15:17:29.250 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:29.248+0000 7f1713400640 1 -- 172.21.15.19:0/2495148428 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f17040054e0 con 0x7f171c104bf0 2024-03-18T15:17:29.250 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:29.248+0000 7f1713400640 1 -- 172.21.15.19:0/2495148428 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1704003cd0 con 0x7f171c104bf0 2024-03-18T15:17:29.250 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:29.248+0000 7f1721800640 1 -- 172.21.15.19:0/2495148428 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f171c1a1b80 con 0x7f171c104bf0 2024-03-18T15:17:29.250 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:29.249+0000 7f1721800640 1 -- 172.21.15.19:0/2495148428 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f171c1a2080 con 0x7f171c104bf0 2024-03-18T15:17:29.250 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:29.250+0000 7f1721800640 1 -- 172.21.15.19:0/2495148428 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f171c10ad20 con 0x7f171c104bf0 2024-03-18T15:17:29.251 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:29.251+0000 7f1713400640 1 -- 172.21.15.19:0/2495148428 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f1704046020 con 0x7f171c104bf0 2024-03-18T15:17:29.252 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:29.253+0000 7f1713400640 1 --2- 172.21.15.19:0/2495148428 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f17000a2880 0x7f17000a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:29.253 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:29.253+0000 7f1713400640 1 -- 172.21.15.19:0/2495148428 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f17040f1430 con 0x7f171c104bf0 2024-03-18T15:17:29.253 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:29.253+0000 7f171aa00640 1 --2- 172.21.15.19:0/2495148428 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f17000a2880 0x7f17000a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:29.254 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:29.254+0000 7f171aa00640 1 --2- 172.21.15.19:0/2495148428 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f17000a2880 0x7f17000a4d40 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f171c19ab80 tx=0x7f1714009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:17:29.255 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:29.255+0000 7f1713400640 1 -- 172.21.15.19:0/2495148428 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f170404d020 con 0x7f171c104bf0 2024-03-18T15:17:29.501 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:29.501+0000 7f1721800640 1 -- 172.21.15.19:0/2495148428 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f171c02d250 con 0x7f171c104bf0 2024-03-18T15:17:29.502 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:29.502+0000 7f1713400640 1 -- 172.21.15.19:0/2495148428 <== mon.2 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11927 (secure 0 0 0) 0x7f17040be0a0 con 0x7f171c104bf0 2024-03-18T15:17:29.503 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-18T15:17:29.579 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6835/1135329545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6834","nonce":1135329545},{"type":"v1","addr":"172.21.15.19:6835","nonce":1135329545}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14457,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6839/3833518557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6838","nonce":3833518557},{"type":"v1","addr":"172.21.15.19:6839","nonce":3833518557}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T15:15:10.589737+0000","modified":"2024-03-18T15:15:46.979999+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24560,"mds_1":24529,"mds_2":24511,"mds_3":24506,"mds_4":24499},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14475":{"gid":14475,"name":"a","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.19:6841/3176583319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6840","nonce":3176583319},{"type":"v1","addr":"172.21.15.19:6841","nonce":3176583319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24433":{"gid":24433,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6833/482725809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":482725809},{"type":"v1","addr":"172.21.15.104:6833","nonce":482725809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24434":{"gid":24434,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/2272279159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":2272279159},{"type":"v1","addr":"172.21.15.33:6833","nonce":2272279159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24469":{"gid":24469,"name":"d","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.19:6837/1778019859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6836","nonce":1778019859},{"type":"v1","addr":"172.21.15.19:6837","nonce":1778019859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6835/4051613151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":4051613151},{"type":"v1","addr":"172.21.15.33:6835","nonce":4051613151}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":17,"addr":"172.21.15.104:6835/3788595957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3788595957},{"type":"v1","addr":"172.21.15.104:6835","nonce":3788595957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":15,"addr":"172.21.15.33:6837/160187378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":160187378},{"type":"v1","addr":"172.21.15.33:6837","nonce":160187378}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24511":{"gid":24511,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":16,"addr":"172.21.15.104:6837/260984674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":260984674},{"type":"v1","addr":"172.21.15.104:6837","nonce":260984674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24529":{"gid":24529,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.104:6839/805341436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6838","nonce":805341436},{"type":"v1","addr":"172.21.15.104:6839","nonce":805341436}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.33:6839/1903591535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":1903591535},{"type":"v1","addr":"172.21.15.33:6839","nonce":1903591535}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24560,"qdb_cluster":[24506,24499,24529,24511,24560]},"id":1}]} 2024-03-18T15:17:29.581 INFO:teuthology.orchestra.run.smithi019.stderr:dumped fsmap epoch 26 2024-03-18T15:17:29.581 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:29.505+0000 7f1721800640 1 -- 172.21.15.19:0/2495148428 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f17000a2880 msgr2=0x7f17000a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:29.581 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:29.505+0000 7f1721800640 1 --2- 172.21.15.19:0/2495148428 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f17000a2880 0x7f17000a4d40 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f171c19ab80 tx=0x7f1714009500 comp rx=0 tx=0).stop 2024-03-18T15:17:29.581 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:29.505+0000 7f1721800640 1 -- 172.21.15.19:0/2495148428 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f171c104bf0 msgr2=0x7f171c19a1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:29.581 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:29.505+0000 7f1721800640 1 --2- 172.21.15.19:0/2495148428 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f171c104bf0 0x7f171c19a1f0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f1704000c00 tx=0x7f170402c650 comp rx=0 tx=0).stop 2024-03-18T15:17:29.581 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:29.505+0000 7f1721800640 1 -- 172.21.15.19:0/2495148428 shutdown_connections 2024-03-18T15:17:29.581 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:29.505+0000 7f1721800640 1 --2- 172.21.15.19:0/2495148428 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f171c1055b0 0x7f171c1a1250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:29.581 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:29.505+0000 7f1721800640 1 --2- 172.21.15.19:0/2495148428 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f17000a2880 0x7f17000a4d40 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:29.581 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:29.505+0000 7f1721800640 1 --2- 172.21.15.19:0/2495148428 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f171c104bf0 0x7f171c19a1f0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:29.581 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:29.505+0000 7f1721800640 1 --2- 172.21.15.19:0/2495148428 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f171c103a00 0x7f171c199cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:29.581 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:29.505+0000 7f1721800640 1 -- 172.21.15.19:0/2495148428 >> 172.21.15.19:0/2495148428 conn(0x7f171c0fef90 msgr2=0x7f171c107ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:17:29.582 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:29.505+0000 7f1721800640 1 -- 172.21.15.19:0/2495148428 shutdown_connections 2024-03-18T15:17:29.582 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:29.505+0000 7f1721800640 1 -- 172.21.15.19:0/2495148428 wait complete. 2024-03-18T15:17:29.582 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T15:17:29.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:17:29 smithi104 ceph-mon[36070]: pgmap v390: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-18T15:17:29.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:17:29 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/2238199329' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T15:17:29.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:17:29 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/3776628814' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T15:17:29.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:17:29 smithi019 ceph-mon[31522]: pgmap v390: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-18T15:17:29.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:17:29 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/2238199329' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T15:17:29.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:17:29 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/3776628814' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T15:17:29.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:17:29 smithi033 ceph-mon[36648]: pgmap v390: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-18T15:17:29.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:17:29 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/2238199329' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T15:17:29.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:17:29 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/3776628814' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T15:17:29.808 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:29.807+0000 7f02c3200640 1 -- 172.21.15.19:0/3477259378 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f02c4107c20 msgr2=0x7f02c4108040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:29.808 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:29.807+0000 7f02c3200640 1 --2- 172.21.15.19:0/3477259378 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f02c4107c20 0x7f02c4108040 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f02b4009900 tx=0x7f02b402bc40 comp rx=0 tx=0).stop 2024-03-18T15:17:29.808 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:29.807+0000 7f02c3200640 1 -- 172.21.15.19:0/3477259378 shutdown_connections 2024-03-18T15:17:29.808 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:29.807+0000 7f02c3200640 1 --2- 172.21.15.19:0/3477259378 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f02c41097d0 0x7f02c4110060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:29.808 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:29.807+0000 7f02c3200640 1 --2- 172.21.15.19:0/3477259378 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f02c4108e10 0x7f02c4109290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:29.808 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:29.807+0000 7f02c3200640 1 --2- 172.21.15.19:0/3477259378 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f02c4107c20 0x7f02c4108040 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:29.808 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:29.807+0000 7f02c3200640 1 -- 172.21.15.19:0/3477259378 >> 172.21.15.19:0/3477259378 conn(0x7f02c4076bb0 msgr2=0x7f02c4078ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:17:29.808 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:29.808+0000 7f02c3200640 1 -- 172.21.15.19:0/3477259378 shutdown_connections 2024-03-18T15:17:29.808 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:29.808+0000 7f02c3200640 1 -- 172.21.15.19:0/3477259378 wait complete. 2024-03-18T15:17:29.811 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:29.808+0000 7f02c3200640 1 Processor -- start 2024-03-18T15:17:29.811 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:29.809+0000 7f02c3200640 1 -- start start 2024-03-18T15:17:29.811 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:29.809+0000 7f02c3200640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f02c4107c20 0x7f02c419df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:29.811 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:29.809+0000 7f02c3200640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f02c4108e10 0x7f02c419e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:29.811 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:29.809+0000 7f02c3200640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f02c41097d0 0x7f02c41a54b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:29.811 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:29.809+0000 7f02c3200640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f02c4113890 con 0x7f02c41097d0 2024-03-18T15:17:29.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:29.809+0000 7f02c3200640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f02c4113710 con 0x7f02c4107c20 2024-03-18T15:17:29.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:29.809+0000 7f02c3200640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f02c4113a10 con 0x7f02c4108e10 2024-03-18T15:17:29.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:29.809+0000 7f02b9400640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f02c4108e10 0x7f02c419e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:29.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:29.809+0000 7f02c1e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f02c4107c20 0x7f02c419df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:29.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:29.809+0000 7f02b9400640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f02c4108e10 0x7f02c419e440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.19:42548/0 (socket says 172.21.15.19:42548) 2024-03-18T15:17:29.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:29.809+0000 7f02b9400640 1 -- 172.21.15.19:0/816373597 learned_addr learned my addr 172.21.15.19:0/816373597 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:17:29.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:29.809+0000 7f02c1e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f02c4107c20 0x7f02c419df00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:55952/0 (socket says 172.21.15.19:55952) 2024-03-18T15:17:29.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:29.809+0000 7f02c2800640 1 --2- 172.21.15.19:0/816373597 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f02c41097d0 0x7f02c41a54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:29.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:29.810+0000 7f02b9400640 1 -- 172.21.15.19:0/816373597 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f02c4107c20 msgr2=0x7f02c419df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:29.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:29.810+0000 7f02b9400640 1 --2- 172.21.15.19:0/816373597 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f02c4107c20 0x7f02c419df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:29.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:29.810+0000 7f02b9400640 1 -- 172.21.15.19:0/816373597 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f02c41097d0 msgr2=0x7f02c41a54b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:29.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:29.810+0000 7f02b9400640 1 --2- 172.21.15.19:0/816373597 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f02c41097d0 0x7f02c41a54b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:29.813 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:29.810+0000 7f02b9400640 1 -- 172.21.15.19:0/816373597 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02c41a5b20 con 0x7f02c4108e10 2024-03-18T15:17:29.813 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:29.810+0000 7f02c1e00640 1 --2- 172.21.15.19:0/816373597 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f02c4107c20 0x7f02c419df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:17:29.813 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:29.810+0000 7f02b9400640 1 --2- 172.21.15.19:0/816373597 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f02c4108e10 0x7f02c419e440 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f02ac00e9f0 tx=0x7f02ac00edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:17:29.813 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:29.810+0000 7f02bb400640 1 -- 172.21.15.19:0/816373597 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02ac00cd80 con 0x7f02c4108e10 2024-03-18T15:17:29.813 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:29.810+0000 7f02c3200640 1 -- 172.21.15.19:0/816373597 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02c41a5e10 con 0x7f02c4108e10 2024-03-18T15:17:29.813 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:29.810+0000 7f02bb400640 1 -- 172.21.15.19:0/816373597 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02ac005500 con 0x7f02c4108e10 2024-03-18T15:17:29.813 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:29.810+0000 7f02bb400640 1 -- 172.21.15.19:0/816373597 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02ac010690 con 0x7f02c4108e10 2024-03-18T15:17:29.813 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:29.810+0000 7f02c3200640 1 -- 172.21.15.19:0/816373597 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02c41a6280 con 0x7f02c4108e10 2024-03-18T15:17:29.814 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:29.811+0000 7f02c3200640 1 -- 172.21.15.19:0/816373597 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02c410ef40 con 0x7f02c4108e10 2024-03-18T15:17:29.814 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:29.814+0000 7f02bb400640 1 -- 172.21.15.19:0/816373597 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f02ac0046c0 con 0x7f02c4108e10 2024-03-18T15:17:29.814 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:29.815+0000 7f02bb400640 1 --2- 172.21.15.19:0/816373597 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f02a80a2800 0x7f02a80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:29.816 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:29.815+0000 7f02bb400640 1 -- 172.21.15.19:0/816373597 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f02ac0cf3f0 con 0x7f02c4108e10 2024-03-18T15:17:29.816 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:29.815+0000 7f02c1e00640 1 --2- 172.21.15.19:0/816373597 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f02a80a2800 0x7f02a80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:29.816 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:29.816+0000 7f02c1e00640 1 --2- 172.21.15.19:0/816373597 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f02a80a2800 0x7f02a80a4cc0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f02b4000c00 tx=0x7f02b4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:17:29.821 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:29.821+0000 7f02bb400640 1 -- 172.21.15.19:0/816373597 <== mon.1 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f02ac053740 con 0x7f02c4108e10 2024-03-18T15:17:30.028 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.028+0000 7f02c3200640 1 -- 172.21.15.19:0/816373597 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f02c402d380 con 0x7f02c4108e10 2024-03-18T15:17:30.030 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.030+0000 7f02bb400640 1 -- 172.21.15.19:0/816373597 <== mon.1 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24809 (secure 0 0 0) 0x7f02ac09c060 con 0x7f02c4108e10 2024-03-18T15:17:30.030 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-18T15:17:30.031 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":93,"fsid":"d4dc2ba6-e538-11ee-95c9-87774f69a715","created":"2024-03-18T15:05:57.538551+0000","modified":"2024-03-18T15:17:16.033313+0000","last_up_change":"2024-03-18T15:14:12.009522+0000","last_in_change":"2024-03-18T15:13:52.555387+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T15:09:56.843349+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T15:15:08.941047+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T15:15:10.044671+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T15:15:13.621916+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"37b5a5cd-27ae-4e77-9b05-774b817b2ee9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6802","nonce":402333948},{"type":"v1","addr":"172.21.15.19:6803","nonce":402333948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6804","nonce":402333948},{"type":"v1","addr":"172.21.15.19:6805","nonce":402333948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6808","nonce":402333948},{"type":"v1","addr":"172.21.15.19:6809","nonce":402333948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6806","nonce":402333948},{"type":"v1","addr":"172.21.15.19:6807","nonce":402333948}]},"public_addr":"172.21.15.19:6803/402333948","cluster_addr":"172.21.15.19:6805/402333948","heartbeat_back_addr":"172.21.15.19:6809/402333948","heartbeat_front_addr":"172.21.15.19:6807/402333948","state":["exists","up"]},{"osd":1,"uuid":"3a1134e8-ede7-4dbd-9be1-eefd2cbb43bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1951880113},{"type":"v1","addr":"172.21.15.33:6801","nonce":1951880113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1951880113},{"type":"v1","addr":"172.21.15.33:6803","nonce":1951880113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1951880113},{"type":"v1","addr":"172.21.15.33:6807","nonce":1951880113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1951880113},{"type":"v1","addr":"172.21.15.33:6805","nonce":1951880113}]},"public_addr":"172.21.15.33:6801/1951880113","cluster_addr":"172.21.15.33:6803/1951880113","heartbeat_back_addr":"172.21.15.33:6807/1951880113","heartbeat_front_addr":"172.21.15.33:6805/1951880113","state":["exists","up"]},{"osd":2,"uuid":"f383a164-a3a0-49fd-a9e8-9c2178d8cfb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":1004727949},{"type":"v1","addr":"172.21.15.104:6801","nonce":1004727949}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":1004727949},{"type":"v1","addr":"172.21.15.104:6803","nonce":1004727949}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":1004727949},{"type":"v1","addr":"172.21.15.104:6807","nonce":1004727949}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":1004727949},{"type":"v1","addr":"172.21.15.104:6805","nonce":1004727949}]},"public_addr":"172.21.15.104:6801/1004727949","cluster_addr":"172.21.15.104:6803/1004727949","heartbeat_back_addr":"172.21.15.104:6807/1004727949","heartbeat_front_addr":"172.21.15.104:6805/1004727949","state":["exists","up"]},{"osd":3,"uuid":"c409e116-28bf-45dd-a791-539af22abe1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6810","nonce":3571525854},{"type":"v1","addr":"172.21.15.19:6811","nonce":3571525854}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6812","nonce":3571525854},{"type":"v1","addr":"172.21.15.19:6813","nonce":3571525854}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6816","nonce":3571525854},{"type":"v1","addr":"172.21.15.19:6817","nonce":3571525854}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6814","nonce":3571525854},{"type":"v1","addr":"172.21.15.19:6815","nonce":3571525854}]},"public_addr":"172.21.15.19:6811/3571525854","cluster_addr":"172.21.15.19:6813/3571525854","heartbeat_back_addr":"172.21.15.19:6817/3571525854","heartbeat_front_addr":"172.21.15.19:6815/3571525854","state":["exists","up"]},{"osd":4,"uuid":"696eb719-0b3e-43ae-b5f1-ccd73a3b2843","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":624901749},{"type":"v1","addr":"172.21.15.33:6809","nonce":624901749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":624901749},{"type":"v1","addr":"172.21.15.33:6811","nonce":624901749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":624901749},{"type":"v1","addr":"172.21.15.33:6815","nonce":624901749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":624901749},{"type":"v1","addr":"172.21.15.33:6813","nonce":624901749}]},"public_addr":"172.21.15.33:6809/624901749","cluster_addr":"172.21.15.33:6811/624901749","heartbeat_back_addr":"172.21.15.33:6815/624901749","heartbeat_front_addr":"172.21.15.33:6813/624901749","state":["exists","up"]},{"osd":5,"uuid":"21789779-8bb9-42c7-b9ee-437985f11e3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":1258208297},{"type":"v1","addr":"172.21.15.104:6809","nonce":1258208297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":1258208297},{"type":"v1","addr":"172.21.15.104:6811","nonce":1258208297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":1258208297},{"type":"v1","addr":"172.21.15.104:6815","nonce":1258208297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":1258208297},{"type":"v1","addr":"172.21.15.104:6813","nonce":1258208297}]},"public_addr":"172.21.15.104:6809/1258208297","cluster_addr":"172.21.15.104:6811/1258208297","heartbeat_back_addr":"172.21.15.104:6815/1258208297","heartbeat_front_addr":"172.21.15.104:6813/1258208297","state":["exists","up"]},{"osd":6,"uuid":"0e92a2ef-43cd-487c-9203-e219158aef8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6818","nonce":2782482301},{"type":"v1","addr":"172.21.15.19:6819","nonce":2782482301}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6820","nonce":2782482301},{"type":"v1","addr":"172.21.15.19:6821","nonce":2782482301}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6824","nonce":2782482301},{"type":"v1","addr":"172.21.15.19:6825","nonce":2782482301}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6822","nonce":2782482301},{"type":"v1","addr":"172.21.15.19:6823","nonce":2782482301}]},"public_addr":"172.21.15.19:6819/2782482301","cluster_addr":"172.21.15.19:6821/2782482301","heartbeat_back_addr":"172.21.15.19:6825/2782482301","heartbeat_front_addr":"172.21.15.19:6823/2782482301","state":["exists","up"]},{"osd":7,"uuid":"a16e39a1-3685-491d-b376-3e80a2084072","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":4238160988},{"type":"v1","addr":"172.21.15.33:6817","nonce":4238160988}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":4238160988},{"type":"v1","addr":"172.21.15.33:6819","nonce":4238160988}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":4238160988},{"type":"v1","addr":"172.21.15.33:6823","nonce":4238160988}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":4238160988},{"type":"v1","addr":"172.21.15.33:6821","nonce":4238160988}]},"public_addr":"172.21.15.33:6817/4238160988","cluster_addr":"172.21.15.33:6819/4238160988","heartbeat_back_addr":"172.21.15.33:6823/4238160988","heartbeat_front_addr":"172.21.15.33:6821/4238160988","state":["exists","up"]},{"osd":8,"uuid":"d5c3fa92-cd5d-4349-9c7d-d5666bacde16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":1014779813},{"type":"v1","addr":"172.21.15.104:6817","nonce":1014779813}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":1014779813},{"type":"v1","addr":"172.21.15.104:6819","nonce":1014779813}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":1014779813},{"type":"v1","addr":"172.21.15.104:6823","nonce":1014779813}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":1014779813},{"type":"v1","addr":"172.21.15.104:6821","nonce":1014779813}]},"public_addr":"172.21.15.104:6817/1014779813","cluster_addr":"172.21.15.104:6819/1014779813","heartbeat_back_addr":"172.21.15.104:6823/1014779813","heartbeat_front_addr":"172.21.15.104:6821/1014779813","state":["exists","up"]},{"osd":9,"uuid":"eed2e9b2-26d3-4c46-88ec-34b5596fe991","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6826","nonce":2641151009},{"type":"v1","addr":"172.21.15.19:6827","nonce":2641151009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6828","nonce":2641151009},{"type":"v1","addr":"172.21.15.19:6829","nonce":2641151009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6832","nonce":2641151009},{"type":"v1","addr":"172.21.15.19:6833","nonce":2641151009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6830","nonce":2641151009},{"type":"v1","addr":"172.21.15.19:6831","nonce":2641151009}]},"public_addr":"172.21.15.19:6827/2641151009","cluster_addr":"172.21.15.19:6829/2641151009","heartbeat_back_addr":"172.21.15.19:6833/2641151009","heartbeat_front_addr":"172.21.15.19:6831/2641151009","state":["exists","up"]},{"osd":10,"uuid":"f3c35ea5-fbe7-4dc4-93c5-b92d1a714db3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":1346058869},{"type":"v1","addr":"172.21.15.33:6825","nonce":1346058869}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":1346058869},{"type":"v1","addr":"172.21.15.33:6827","nonce":1346058869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":1346058869},{"type":"v1","addr":"172.21.15.33:6831","nonce":1346058869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":1346058869},{"type":"v1","addr":"172.21.15.33:6829","nonce":1346058869}]},"public_addr":"172.21.15.33:6825/1346058869","cluster_addr":"172.21.15.33:6827/1346058869","heartbeat_back_addr":"172.21.15.33:6831/1346058869","heartbeat_front_addr":"172.21.15.33:6829/1346058869","state":["exists","up"]},{"osd":11,"uuid":"c8b51571-f807-4b62-90b6-7529e09e59da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":3759498281},{"type":"v1","addr":"172.21.15.104:6825","nonce":3759498281}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":3759498281},{"type":"v1","addr":"172.21.15.104:6827","nonce":3759498281}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":3759498281},{"type":"v1","addr":"172.21.15.104:6831","nonce":3759498281}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":3759498281},{"type":"v1","addr":"172.21.15.104:6829","nonce":3759498281}]},"public_addr":"172.21.15.104:6825/3759498281","cluster_addr":"172.21.15.104:6827/3759498281","heartbeat_back_addr":"172.21.15.104:6831/3759498281","heartbeat_front_addr":"172.21.15.104:6829/3759498281","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:09:05.267795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:09:30.234068+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:09:53.870232+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:10:26.471296+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:10:53.180341+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:11:17.801361+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:11:51.364816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:12:18.306408+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:12:43.978744+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:13:18.213964+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:13:45.437544+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:14:10.148580+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":3,"to":6}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"3.37","mappings":[{"from":5,"to":8}]},{"pgid":"3.3a","mappings":[{"from":4,"to":7}]},{"pgid":"4.8","mappings":[{"from":5,"to":8}]},{"pgid":"4.f","mappings":[{"from":9,"to":10}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.19:0/3860783274":"2024-03-19T15:07:14.150983+0000","172.21.15.19:0/2737223213":"2024-03-19T15:07:14.150983+0000","172.21.15.19:0/176699025":"2024-03-19T15:06:38.688405+0000","172.21.15.19:0/2316969005":"2024-03-19T15:07:14.150983+0000","172.21.15.19:6801/2776008962":"2024-03-19T15:07:14.150983+0000","172.21.15.19:0/2793980440":"2024-03-19T15:06:38.688405+0000","172.21.15.19:6800/3264337803":"2024-03-19T15:06:38.688405+0000","172.21.15.19:6801/3264337803":"2024-03-19T15:06:38.688405+0000","172.21.15.19:0/3166705295":"2024-03-19T15:06:38.688405+0000","172.21.15.19:6800/2776008962":"2024-03-19T15:07:14.150983+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T15:17:30.033 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.033+0000 7f02c3200640 1 -- 172.21.15.19:0/816373597 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f02a80a2800 msgr2=0x7f02a80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:30.033 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.033+0000 7f02c3200640 1 --2- 172.21.15.19:0/816373597 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f02a80a2800 0x7f02a80a4cc0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f02b4000c00 tx=0x7f02b4005fd0 comp rx=0 tx=0).stop 2024-03-18T15:17:30.034 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.033+0000 7f02c3200640 1 -- 172.21.15.19:0/816373597 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f02c4108e10 msgr2=0x7f02c419e440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:30.034 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.033+0000 7f02c3200640 1 --2- 172.21.15.19:0/816373597 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f02c4108e10 0x7f02c419e440 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f02ac00e9f0 tx=0x7f02ac00edf0 comp rx=0 tx=0).stop 2024-03-18T15:17:30.034 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.033+0000 7f02c3200640 1 -- 172.21.15.19:0/816373597 shutdown_connections 2024-03-18T15:17:30.034 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.033+0000 7f02c3200640 1 --2- 172.21.15.19:0/816373597 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f02c41097d0 0x7f02c41a54b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:30.034 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.033+0000 7f02c3200640 1 --2- 172.21.15.19:0/816373597 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f02c4108e10 0x7f02c419e440 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:30.034 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.033+0000 7f02c3200640 1 --2- 172.21.15.19:0/816373597 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f02a80a2800 0x7f02a80a4cc0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:30.034 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.033+0000 7f02c3200640 1 --2- 172.21.15.19:0/816373597 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f02c4107c20 0x7f02c419df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:30.034 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.033+0000 7f02c3200640 1 -- 172.21.15.19:0/816373597 >> 172.21.15.19:0/816373597 conn(0x7f02c4076bb0 msgr2=0x7f02c410c110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:17:30.034 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.034+0000 7f02c3200640 1 -- 172.21.15.19:0/816373597 shutdown_connections 2024-03-18T15:17:30.034 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.034+0000 7f02c3200640 1 -- 172.21.15.19:0/816373597 wait complete. 2024-03-18T15:17:30.055 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T15:09:56.843349+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T15:15:08.941047+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T15:15:10.044671+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-18T15:15:13.621916+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-18T15:17:30.056 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T15:17:30.276 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.276+0000 7fd6d1200640 1 -- 172.21.15.19:0/1912227459 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd6cc1074e0 msgr2=0x7fd6cc109930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:30.276 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.276+0000 7fd6d1200640 1 --2- 172.21.15.19:0/1912227459 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd6cc1074e0 0x7fd6cc109930 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fd6c4009ca0 tx=0x7fd6c402c100 comp rx=0 tx=0).stop 2024-03-18T15:17:30.277 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.276+0000 7fd6d1200640 1 -- 172.21.15.19:0/1912227459 shutdown_connections 2024-03-18T15:17:30.277 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.276+0000 7fd6d1200640 1 --2- 172.21.15.19:0/1912227459 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd6cc1074e0 0x7fd6cc109930 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:30.277 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.276+0000 7fd6d1200640 1 --2- 172.21.15.19:0/1912227459 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd6cc104bb0 0x7fd6cc106fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:30.277 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.276+0000 7fd6d1200640 1 --2- 172.21.15.19:0/1912227459 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd6cc102280 0x7fd6cc104670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:30.277 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.276+0000 7fd6d1200640 1 -- 172.21.15.19:0/1912227459 >> 172.21.15.19:0/1912227459 conn(0x7fd6cc0fbc70 msgr2=0x7fd6cc0fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:17:30.277 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.276+0000 7fd6d1200640 1 -- 172.21.15.19:0/1912227459 shutdown_connections 2024-03-18T15:17:30.277 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.276+0000 7fd6d1200640 1 -- 172.21.15.19:0/1912227459 wait complete. 2024-03-18T15:17:30.279 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.277+0000 7fd6d1200640 1 Processor -- start 2024-03-18T15:17:30.279 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.277+0000 7fd6d1200640 1 -- start start 2024-03-18T15:17:30.279 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.277+0000 7fd6d1200640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd6cc102280 0x7fd6cc199c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:30.279 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.278+0000 7fd6d1200640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd6cc104bb0 0x7fd6cc19a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:30.279 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.278+0000 7fd6d1200640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd6cc1074e0 0x7fd6cc1a1230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:30.279 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.278+0000 7fd6d1200640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fd6cc10d480 con 0x7fd6cc102280 2024-03-18T15:17:30.280 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.278+0000 7fd6d1200640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd6cc10d300 con 0x7fd6cc1074e0 2024-03-18T15:17:30.280 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.278+0000 7fd6d1200640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fd6cc10d600 con 0x7fd6cc104bb0 2024-03-18T15:17:30.280 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.278+0000 7fd6c9e00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd6cc102280 0x7fd6cc199c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:30.280 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.278+0000 7fd6c9e00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd6cc102280 0x7fd6cc199c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:40388/0 (socket says 172.21.15.19:40388) 2024-03-18T15:17:30.280 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.278+0000 7fd6c9e00640 1 -- 172.21.15.19:0/3627980740 learned_addr learned my addr 172.21.15.19:0/3627980740 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:17:30.280 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.278+0000 7fd6ca800640 1 --2- 172.21.15.19:0/3627980740 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd6cc1074e0 0x7fd6cc1a1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:30.280 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.278+0000 7fd6c9400640 1 --2- 172.21.15.19:0/3627980740 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd6cc104bb0 0x7fd6cc19a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:30.280 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.278+0000 7fd6ca800640 1 -- 172.21.15.19:0/3627980740 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd6cc104bb0 msgr2=0x7fd6cc19a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:30.280 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.278+0000 7fd6ca800640 1 --2- 172.21.15.19:0/3627980740 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd6cc104bb0 0x7fd6cc19a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:30.280 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.278+0000 7fd6ca800640 1 -- 172.21.15.19:0/3627980740 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd6cc102280 msgr2=0x7fd6cc199c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:30.280 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.278+0000 7fd6ca800640 1 --2- 172.21.15.19:0/3627980740 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd6cc102280 0x7fd6cc199c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:30.280 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.278+0000 7fd6ca800640 1 -- 172.21.15.19:0/3627980740 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6cc1a18d0 con 0x7fd6cc1074e0 2024-03-18T15:17:30.280 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.278+0000 7fd6c9e00640 1 --2- 172.21.15.19:0/3627980740 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd6cc102280 0x7fd6cc199c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:17:30.280 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.279+0000 7fd6ca800640 1 --2- 172.21.15.19:0/3627980740 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd6cc1074e0 0x7fd6cc1a1230 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fd6c4009bf0 tx=0x7fd6c4035d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:17:30.281 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.279+0000 7fd6c2a00640 1 -- 172.21.15.19:0/3627980740 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c4039040 con 0x7fd6cc1074e0 2024-03-18T15:17:30.281 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.279+0000 7fd6c2a00640 1 -- 172.21.15.19:0/3627980740 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd6c4044070 con 0x7fd6cc1074e0 2024-03-18T15:17:30.281 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.279+0000 7fd6d1200640 1 -- 172.21.15.19:0/3627980740 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6cc1a1b60 con 0x7fd6cc1074e0 2024-03-18T15:17:30.281 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.279+0000 7fd6c2a00640 1 -- 172.21.15.19:0/3627980740 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c40043a0 con 0x7fd6cc1074e0 2024-03-18T15:17:30.281 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.279+0000 7fd6d1200640 1 -- 172.21.15.19:0/3627980740 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6cc1a2020 con 0x7fd6cc1074e0 2024-03-18T15:17:30.281 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.280+0000 7fd6d1200640 1 -- 172.21.15.19:0/3627980740 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6cc108fd0 con 0x7fd6cc1074e0 2024-03-18T15:17:30.282 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.282+0000 7fd6c2a00640 1 -- 172.21.15.19:0/3627980740 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fd6c4004540 con 0x7fd6cc1074e0 2024-03-18T15:17:30.283 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.283+0000 7fd6c2a00640 1 --2- 172.21.15.19:0/3627980740 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fd6ac0a2800 0x7fd6ac0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:30.283 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.283+0000 7fd6c2a00640 1 -- 172.21.15.19:0/3627980740 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fd6c40f1390 con 0x7fd6cc1074e0 2024-03-18T15:17:30.283 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.284+0000 7fd6c9e00640 1 --2- 172.21.15.19:0/3627980740 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fd6ac0a2800 0x7fd6ac0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:30.284 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.284+0000 7fd6c9e00640 1 --2- 172.21.15.19:0/3627980740 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fd6ac0a2800 0x7fd6ac0a4cc0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fd6cc19ab60 tx=0x7fd6bc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:17:30.287 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.287+0000 7fd6c2a00640 1 -- 172.21.15.19:0/3627980740 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd6c40be100 con 0x7fd6cc1074e0 2024-03-18T15:17:30.493 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.492+0000 7fd6d1200640 1 -- 172.21.15.19:0/3627980740 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd6cc02d380 con 0x7fd6cc1074e0 2024-03-18T15:17:30.493 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.493+0000 7fd6c2a00640 1 -- 172.21.15.19:0/3627980740 <== mon.2 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7fd6c40bda00 con 0x7fd6cc1074e0 2024-03-18T15:17:30.493 INFO:teuthology.orchestra.run.smithi019.stdout:pg_num: 1 2024-03-18T15:17:30.497 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.496+0000 7fd6d1200640 1 -- 172.21.15.19:0/3627980740 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fd6ac0a2800 msgr2=0x7fd6ac0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:30.497 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.496+0000 7fd6d1200640 1 --2- 172.21.15.19:0/3627980740 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fd6ac0a2800 0x7fd6ac0a4cc0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fd6cc19ab60 tx=0x7fd6bc005f90 comp rx=0 tx=0).stop 2024-03-18T15:17:30.497 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.496+0000 7fd6d1200640 1 -- 172.21.15.19:0/3627980740 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd6cc1074e0 msgr2=0x7fd6cc1a1230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:30.497 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.496+0000 7fd6d1200640 1 --2- 172.21.15.19:0/3627980740 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd6cc1074e0 0x7fd6cc1a1230 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fd6c4009bf0 tx=0x7fd6c4035d10 comp rx=0 tx=0).stop 2024-03-18T15:17:30.497 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.496+0000 7fd6d1200640 1 -- 172.21.15.19:0/3627980740 shutdown_connections 2024-03-18T15:17:30.497 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.496+0000 7fd6d1200640 1 --2- 172.21.15.19:0/3627980740 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fd6ac0a2800 0x7fd6ac0a4cc0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:30.497 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.496+0000 7fd6d1200640 1 --2- 172.21.15.19:0/3627980740 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd6cc1074e0 0x7fd6cc1a1230 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:30.497 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.496+0000 7fd6d1200640 1 --2- 172.21.15.19:0/3627980740 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd6cc104bb0 0x7fd6cc19a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:30.497 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.496+0000 7fd6d1200640 1 --2- 172.21.15.19:0/3627980740 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd6cc102280 0x7fd6cc199c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:30.497 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.496+0000 7fd6d1200640 1 -- 172.21.15.19:0/3627980740 >> 172.21.15.19:0/3627980740 conn(0x7fd6cc0fbc70 msgr2=0x7fd6cc1018f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:17:30.497 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.497+0000 7fd6d1200640 1 -- 172.21.15.19:0/3627980740 shutdown_connections 2024-03-18T15:17:30.498 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.497+0000 7fd6d1200640 1 -- 172.21.15.19:0/3627980740 wait complete. 2024-03-18T15:17:30.517 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T15:17:30.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:17:30 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/2495148428' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:17:30.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:17:30 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/816373597' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:17:30.745 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.744+0000 7f560de00640 1 -- 172.21.15.19:0/521669293 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5608103a20 msgr2=0x7f5608103e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:30.745 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.744+0000 7f560de00640 1 --2- 172.21.15.19:0/521669293 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5608103a20 0x7f5608103e40 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f5600009ca0 tx=0x7f560002c100 comp rx=0 tx=0).stop 2024-03-18T15:17:30.745 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.745+0000 7f560de00640 1 -- 172.21.15.19:0/521669293 shutdown_connections 2024-03-18T15:17:30.745 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.745+0000 7f560de00640 1 --2- 172.21.15.19:0/521669293 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f56081055d0 0x7f560810be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:30.745 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.745+0000 7f560de00640 1 --2- 172.21.15.19:0/521669293 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5608104c10 0x7f5608105090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:30.745 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.745+0000 7f560de00640 1 --2- 172.21.15.19:0/521669293 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5608103a20 0x7f5608103e40 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:30.745 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.745+0000 7f560de00640 1 -- 172.21.15.19:0/521669293 >> 172.21.15.19:0/521669293 conn(0x7f56080fef90 msgr2=0x7f56081013d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:17:30.746 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.745+0000 7f560de00640 1 -- 172.21.15.19:0/521669293 shutdown_connections 2024-03-18T15:17:30.746 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.745+0000 7f560de00640 1 -- 172.21.15.19:0/521669293 wait complete. 2024-03-18T15:17:30.748 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.746+0000 7f560de00640 1 Processor -- start 2024-03-18T15:17:30.748 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.746+0000 7f560de00640 1 -- start start 2024-03-18T15:17:30.748 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.746+0000 7f560de00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5608103a20 0x7f5608199d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:30.748 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.746+0000 7f560de00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5608104c10 0x7f560819a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:30.748 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.746+0000 7f560de00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f56081055d0 0x7f56081a12f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:30.748 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.746+0000 7f560de00640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f560810f670 con 0x7f5608104c10 2024-03-18T15:17:30.748 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.746+0000 7f560de00640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f560810f4f0 con 0x7f56081055d0 2024-03-18T15:17:30.748 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.746+0000 7f560de00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f560810f7f0 con 0x7f5608103a20 2024-03-18T15:17:30.748 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.746+0000 7f5606000640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5608104c10 0x7f560819a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:30.749 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.746+0000 7f5606a00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5608103a20 0x7f5608199d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:30.749 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.746+0000 7f5607400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f56081055d0 0x7f56081a12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:30.749 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.746+0000 7f5606000640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5608104c10 0x7f560819a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:40406/0 (socket says 172.21.15.19:40406) 2024-03-18T15:17:30.749 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.746+0000 7f5606000640 1 -- 172.21.15.19:0/845359106 learned_addr learned my addr 172.21.15.19:0/845359106 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:17:30.749 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.746+0000 7f5606a00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5608103a20 0x7f5608199d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.19:42578/0 (socket says 172.21.15.19:42578) 2024-03-18T15:17:30.749 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.747+0000 7f5606a00640 1 -- 172.21.15.19:0/845359106 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f56081055d0 msgr2=0x7f56081a12f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:30.749 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.747+0000 7f5606a00640 1 --2- 172.21.15.19:0/845359106 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f56081055d0 0x7f56081a12f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:30.749 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.747+0000 7f5606a00640 1 -- 172.21.15.19:0/845359106 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5608104c10 msgr2=0x7f560819a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:30.749 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.747+0000 7f5606a00640 1 --2- 172.21.15.19:0/845359106 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5608104c10 0x7f560819a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:30.749 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.747+0000 7f5606a00640 1 -- 172.21.15.19:0/845359106 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56081a1960 con 0x7f5608103a20 2024-03-18T15:17:30.749 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.747+0000 7f5606000640 1 --2- 172.21.15.19:0/845359106 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5608104c10 0x7f560819a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:17:30.749 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.747+0000 7f5606a00640 1 --2- 172.21.15.19:0/845359106 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5608103a20 0x7f5608199d40 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f5600037040 tx=0x7f5600035d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:17:30.750 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.748+0000 7f560ca00640 1 -- 172.21.15.19:0/845359106 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5600039040 con 0x7f5608103a20 2024-03-18T15:17:30.750 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.748+0000 7f560ca00640 1 -- 172.21.15.19:0/845359106 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5600003970 con 0x7f5608103a20 2024-03-18T15:17:30.750 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.748+0000 7f560de00640 1 -- 172.21.15.19:0/845359106 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56081a1bf0 con 0x7f5608103a20 2024-03-18T15:17:30.750 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.748+0000 7f560ca00640 1 -- 172.21.15.19:0/845359106 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56000044a0 con 0x7f5608103a20 2024-03-18T15:17:30.750 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.748+0000 7f560de00640 1 -- 172.21.15.19:0/845359106 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56081a2100 con 0x7f5608103a20 2024-03-18T15:17:30.750 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.749+0000 7f560de00640 1 -- 172.21.15.19:0/845359106 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f560810adc0 con 0x7f5608103a20 2024-03-18T15:17:30.751 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.751+0000 7f560ca00640 1 -- 172.21.15.19:0/845359106 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f5600046030 con 0x7f5608103a20 2024-03-18T15:17:30.752 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.752+0000 7f560ca00640 1 --2- 172.21.15.19:0/845359106 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f55ec0a2880 0x7f55ec0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:30.752 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.752+0000 7f560ca00640 1 -- 172.21.15.19:0/845359106 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f56000f0a40 con 0x7f5608103a20 2024-03-18T15:17:30.752 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.752+0000 7f5606000640 1 --2- 172.21.15.19:0/845359106 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f55ec0a2880 0x7f55ec0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:30.753 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.753+0000 7f5606000640 1 --2- 172.21.15.19:0/845359106 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f55ec0a2880 0x7f55ec0a4d40 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f560819b250 tx=0x7f55f4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:17:30.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:17:30 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/2495148428' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:17:30.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:17:30 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/816373597' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:17:30.756 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.756+0000 7f560ca00640 1 -- 172.21.15.19:0/845359106 <== mon.1 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f560004d060 con 0x7f5608103a20 2024-03-18T15:17:30.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:17:30 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/2495148428' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:17:30.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:17:30 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/816373597' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:17:30.959 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.959+0000 7f560de00640 1 -- 172.21.15.19:0/845359106 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f56080623e0 con 0x7f5608103a20 2024-03-18T15:17:30.960 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.960+0000 7f560ca00640 1 -- 172.21.15.19:0/845359106 <== mon.1 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7f56000bd740 con 0x7f5608103a20 2024-03-18T15:17:30.960 INFO:teuthology.orchestra.run.smithi019.stdout:pg_num: 32 2024-03-18T15:17:30.963 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.963+0000 7f560de00640 1 -- 172.21.15.19:0/845359106 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f55ec0a2880 msgr2=0x7f55ec0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:30.963 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.963+0000 7f560de00640 1 --2- 172.21.15.19:0/845359106 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f55ec0a2880 0x7f55ec0a4d40 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f560819b250 tx=0x7f55f4005f90 comp rx=0 tx=0).stop 2024-03-18T15:17:30.963 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.963+0000 7f560de00640 1 -- 172.21.15.19:0/845359106 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5608103a20 msgr2=0x7f5608199d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:30.963 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.963+0000 7f560de00640 1 --2- 172.21.15.19:0/845359106 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5608103a20 0x7f5608199d40 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f5600037040 tx=0x7f5600035d10 comp rx=0 tx=0).stop 2024-03-18T15:17:30.963 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.963+0000 7f560de00640 1 -- 172.21.15.19:0/845359106 shutdown_connections 2024-03-18T15:17:30.963 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.963+0000 7f560de00640 1 --2- 172.21.15.19:0/845359106 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f55ec0a2880 0x7f55ec0a4d40 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:30.963 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.963+0000 7f560de00640 1 --2- 172.21.15.19:0/845359106 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f56081055d0 0x7f56081a12f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:30.963 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.963+0000 7f560de00640 1 --2- 172.21.15.19:0/845359106 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5608104c10 0x7f560819a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:30.963 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.963+0000 7f560de00640 1 --2- 172.21.15.19:0/845359106 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5608103a20 0x7f5608199d40 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:30.963 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.963+0000 7f560de00640 1 -- 172.21.15.19:0/845359106 >> 172.21.15.19:0/845359106 conn(0x7f56080fef90 msgr2=0x7f5608107f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:17:30.963 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.963+0000 7f560de00640 1 -- 172.21.15.19:0/845359106 shutdown_connections 2024-03-18T15:17:30.964 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:30.963+0000 7f560de00640 1 -- 172.21.15.19:0/845359106 wait complete. 2024-03-18T15:17:30.982 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T15:17:31.216 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.216+0000 7f78ab200640 1 -- 172.21.15.19:0/1123283129 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f78ac1098b0 msgr2=0x7f78ac110140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:31.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.216+0000 7f78ab200640 1 --2- 172.21.15.19:0/1123283129 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f78ac1098b0 0x7f78ac110140 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f78a4009ca0 tx=0x7f78a402c0f0 comp rx=0 tx=0).stop 2024-03-18T15:17:31.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.216+0000 7f78ab200640 1 -- 172.21.15.19:0/1123283129 shutdown_connections 2024-03-18T15:17:31.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.216+0000 7f78ab200640 1 --2- 172.21.15.19:0/1123283129 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f78ac1098b0 0x7f78ac110140 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:31.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.216+0000 7f78ab200640 1 --2- 172.21.15.19:0/1123283129 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f78ac108ef0 0x7f78ac109370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:31.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.216+0000 7f78ab200640 1 --2- 172.21.15.19:0/1123283129 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f78ac107d00 0x7f78ac108120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:31.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.216+0000 7f78ab200640 1 -- 172.21.15.19:0/1123283129 >> 172.21.15.19:0/1123283129 conn(0x7f78ac076cb0 msgr2=0x7f78ac0790d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:17:31.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.216+0000 7f78ab200640 1 -- 172.21.15.19:0/1123283129 shutdown_connections 2024-03-18T15:17:31.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.216+0000 7f78ab200640 1 -- 172.21.15.19:0/1123283129 wait complete. 2024-03-18T15:17:31.219 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.217+0000 7f78ab200640 1 Processor -- start 2024-03-18T15:17:31.219 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.218+0000 7f78ab200640 1 -- start start 2024-03-18T15:17:31.219 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.218+0000 7f78ab200640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f78ac107d00 0x7f78ac19dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:31.219 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.218+0000 7f78ab200640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f78ac108ef0 0x7f78ac19e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:31.219 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.218+0000 7f78ab200640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f78ac1098b0 0x7f78ac1a5570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:31.220 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.218+0000 7f78ab200640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f78ac113960 con 0x7f78ac108ef0 2024-03-18T15:17:31.220 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.218+0000 7f78ab200640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f78ac1137e0 con 0x7f78ac107d00 2024-03-18T15:17:31.220 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.218+0000 7f78ab200640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f78ac113ae0 con 0x7f78ac1098b0 2024-03-18T15:17:31.220 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.218+0000 7f78a9e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f78ac107d00 0x7f78ac19dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:31.220 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.218+0000 7f78a9e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f78ac107d00 0x7f78ac19dfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:56020/0 (socket says 172.21.15.19:56020) 2024-03-18T15:17:31.220 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.218+0000 7f78a9e00640 1 -- 172.21.15.19:0/1774387280 learned_addr learned my addr 172.21.15.19:0/1774387280 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:17:31.220 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.218+0000 7f78aa800640 1 --2- 172.21.15.19:0/1774387280 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f78ac1098b0 0x7f78ac1a5570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:31.220 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.218+0000 7f78a9400640 1 --2- 172.21.15.19:0/1774387280 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f78ac108ef0 0x7f78ac19e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:31.220 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.219+0000 7f78aa800640 1 -- 172.21.15.19:0/1774387280 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f78ac107d00 msgr2=0x7f78ac19dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:31.220 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.219+0000 7f78aa800640 1 --2- 172.21.15.19:0/1774387280 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f78ac107d00 0x7f78ac19dfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:31.220 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.219+0000 7f78aa800640 1 -- 172.21.15.19:0/1774387280 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f78ac108ef0 msgr2=0x7f78ac19e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:31.220 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.219+0000 7f78aa800640 1 --2- 172.21.15.19:0/1774387280 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f78ac108ef0 0x7f78ac19e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:31.220 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.219+0000 7f78aa800640 1 -- 172.21.15.19:0/1774387280 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78ac1a5be0 con 0x7f78ac1098b0 2024-03-18T15:17:31.221 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.219+0000 7f78a9e00640 1 --2- 172.21.15.19:0/1774387280 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f78ac107d00 0x7f78ac19dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:17:31.221 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.219+0000 7f78aa800640 1 --2- 172.21.15.19:0/1774387280 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f78ac1098b0 0x7f78ac1a5570 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f78a402c700 tx=0x7f78a4032f20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:17:31.221 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.219+0000 7f78a2a00640 1 -- 172.21.15.19:0/1774387280 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78a4041020 con 0x7f78ac1098b0 2024-03-18T15:17:31.221 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.219+0000 7f78a2a00640 1 -- 172.21.15.19:0/1774387280 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f78a4034cd0 con 0x7f78ac1098b0 2024-03-18T15:17:31.221 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.219+0000 7f78ab200640 1 -- 172.21.15.19:0/1774387280 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78ac1a5e70 con 0x7f78ac1098b0 2024-03-18T15:17:31.221 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.219+0000 7f78a2a00640 1 -- 172.21.15.19:0/1774387280 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78a4039040 con 0x7f78ac1098b0 2024-03-18T15:17:31.221 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.219+0000 7f78ab200640 1 -- 172.21.15.19:0/1774387280 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78ac1a62d0 con 0x7f78ac1098b0 2024-03-18T15:17:31.221 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.220+0000 7f78ab200640 1 -- 172.21.15.19:0/1774387280 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78ac10efc0 con 0x7f78ac1098b0 2024-03-18T15:17:31.222 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.222+0000 7f78a2a00640 1 -- 172.21.15.19:0/1774387280 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f78a4044020 con 0x7f78ac1098b0 2024-03-18T15:17:31.223 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.223+0000 7f78a2a00640 1 --2- 172.21.15.19:0/1774387280 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f78900a2800 0x7f78900a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:31.223 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.224+0000 7f78a2a00640 1 -- 172.21.15.19:0/1774387280 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f78a40f0c30 con 0x7f78ac1098b0 2024-03-18T15:17:31.224 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.224+0000 7f78a9e00640 1 --2- 172.21.15.19:0/1774387280 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f78900a2800 0x7f78900a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:31.224 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.224+0000 7f78a9e00640 1 --2- 172.21.15.19:0/1774387280 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f78900a2800 0x7f78900a4cc0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f789c0058c0 tx=0x7f789c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:17:31.227 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.227+0000 7f78a2a00640 1 -- 172.21.15.19:0/1774387280 <== mon.1 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f78a404b020 con 0x7f78ac1098b0 2024-03-18T15:17:31.431 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.431+0000 7f78ab200640 1 -- 172.21.15.19:0/1774387280 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f78ac0623e0 con 0x7f78ac1098b0 2024-03-18T15:17:31.432 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.432+0000 7f78a2a00640 1 -- 172.21.15.19:0/1774387280 <== mon.1 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7f78a40bd930 con 0x7f78ac1098b0 2024-03-18T15:17:31.432 INFO:teuthology.orchestra.run.smithi019.stdout:pg_num: 64 2024-03-18T15:17:31.435 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.434+0000 7f78ab200640 1 -- 172.21.15.19:0/1774387280 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f78900a2800 msgr2=0x7f78900a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:31.435 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.434+0000 7f78ab200640 1 --2- 172.21.15.19:0/1774387280 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f78900a2800 0x7f78900a4cc0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f789c0058c0 tx=0x7f789c005f90 comp rx=0 tx=0).stop 2024-03-18T15:17:31.435 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.434+0000 7f78ab200640 1 -- 172.21.15.19:0/1774387280 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f78ac1098b0 msgr2=0x7f78ac1a5570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:31.435 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.434+0000 7f78ab200640 1 --2- 172.21.15.19:0/1774387280 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f78ac1098b0 0x7f78ac1a5570 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f78a402c700 tx=0x7f78a4032f20 comp rx=0 tx=0).stop 2024-03-18T15:17:31.435 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.435+0000 7f78ab200640 1 -- 172.21.15.19:0/1774387280 shutdown_connections 2024-03-18T15:17:31.435 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.435+0000 7f78ab200640 1 --2- 172.21.15.19:0/1774387280 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f78ac1098b0 0x7f78ac1a5570 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:31.435 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.435+0000 7f78ab200640 1 --2- 172.21.15.19:0/1774387280 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f78ac108ef0 0x7f78ac19e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:31.435 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.435+0000 7f78ab200640 1 --2- 172.21.15.19:0/1774387280 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f78900a2800 0x7f78900a4cc0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:31.435 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.435+0000 7f78ab200640 1 --2- 172.21.15.19:0/1774387280 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f78ac107d00 0x7f78ac19dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:31.435 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.435+0000 7f78ab200640 1 -- 172.21.15.19:0/1774387280 >> 172.21.15.19:0/1774387280 conn(0x7f78ac076cb0 msgr2=0x7f78ac10c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:17:31.435 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.435+0000 7f78ab200640 1 -- 172.21.15.19:0/1774387280 shutdown_connections 2024-03-18T15:17:31.436 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.435+0000 7f78ab200640 1 -- 172.21.15.19:0/1774387280 wait complete. 2024-03-18T15:17:31.456 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-18T15:17:31.682 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.682+0000 7fcb82200640 1 -- 172.21.15.19:0/2248263932 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcb7c1028d0 msgr2=0x7fcb7c102d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:31.682 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.682+0000 7fcb82200640 1 --2- 172.21.15.19:0/2248263932 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcb7c1028d0 0x7fcb7c102d50 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fcb64009900 tx=0x7fcb6402bc40 comp rx=0 tx=0).stop 2024-03-18T15:17:31.683 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.682+0000 7fcb82200640 1 -- 172.21.15.19:0/2248263932 shutdown_connections 2024-03-18T15:17:31.683 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.682+0000 7fcb82200640 1 --2- 172.21.15.19:0/2248263932 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcb7c103290 0x7fcb7c109b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:31.683 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.682+0000 7fcb82200640 1 --2- 172.21.15.19:0/2248263932 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcb7c1028d0 0x7fcb7c102d50 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:31.683 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.682+0000 7fcb82200640 1 --2- 172.21.15.19:0/2248263932 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fcb7c1016e0 0x7fcb7c101b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:31.683 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.682+0000 7fcb82200640 1 -- 172.21.15.19:0/2248263932 >> 172.21.15.19:0/2248263932 conn(0x7fcb7c0fcc50 msgr2=0x7fcb7c0ff090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:17:31.683 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.682+0000 7fcb82200640 1 -- 172.21.15.19:0/2248263932 shutdown_connections 2024-03-18T15:17:31.683 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.682+0000 7fcb82200640 1 -- 172.21.15.19:0/2248263932 wait complete. 2024-03-18T15:17:31.685 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.683+0000 7fcb82200640 1 Processor -- start 2024-03-18T15:17:31.685 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.683+0000 7fcb82200640 1 -- start start 2024-03-18T15:17:31.685 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.683+0000 7fcb82200640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcb7c1016e0 0x7fcb7c199bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:31.685 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.684+0000 7fcb82200640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcb7c1028d0 0x7fcb7c19a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:31.686 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.684+0000 7fcb82200640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fcb7c103290 0x7fcb7c1a1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:31.686 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.684+0000 7fcb82200640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fcb7c10d390 con 0x7fcb7c103290 2024-03-18T15:17:31.686 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.684+0000 7fcb82200640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fcb7c10d210 con 0x7fcb7c1016e0 2024-03-18T15:17:31.686 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.684+0000 7fcb82200640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fcb7c10d510 con 0x7fcb7c1028d0 2024-03-18T15:17:31.686 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.684+0000 7fcb80e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcb7c1016e0 0x7fcb7c199bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:31.686 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.684+0000 7fcb80e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcb7c1016e0 0x7fcb7c199bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:56032/0 (socket says 172.21.15.19:56032) 2024-03-18T15:17:31.686 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.684+0000 7fcb80e00640 1 -- 172.21.15.19:0/2544124611 learned_addr learned my addr 172.21.15.19:0/2544124611 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:17:31.686 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.684+0000 7fcb81800640 1 --2- 172.21.15.19:0/2544124611 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fcb7c103290 0x7fcb7c1a1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:31.686 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.684+0000 7fcb73e00640 1 --2- 172.21.15.19:0/2544124611 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcb7c1028d0 0x7fcb7c19a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:31.686 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.684+0000 7fcb80e00640 1 -- 172.21.15.19:0/2544124611 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcb7c1028d0 msgr2=0x7fcb7c19a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:31.686 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.684+0000 7fcb80e00640 1 --2- 172.21.15.19:0/2544124611 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcb7c1028d0 0x7fcb7c19a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:31.686 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.684+0000 7fcb80e00640 1 -- 172.21.15.19:0/2544124611 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fcb7c103290 msgr2=0x7fcb7c1a1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:31.686 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.684+0000 7fcb80e00640 1 --2- 172.21.15.19:0/2544124611 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fcb7c103290 0x7fcb7c1a1170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:31.687 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.684+0000 7fcb80e00640 1 -- 172.21.15.19:0/2544124611 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb7c1a17e0 con 0x7fcb7c1016e0 2024-03-18T15:17:31.687 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.685+0000 7fcb80e00640 1 --2- 172.21.15.19:0/2544124611 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcb7c1016e0 0x7fcb7c199bc0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fcb6c00ea80 tx=0x7fcb6c00ef40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:17:31.687 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.685+0000 7fcb7a000640 1 -- 172.21.15.19:0/2544124611 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb6c00ce40 con 0x7fcb7c1016e0 2024-03-18T15:17:31.687 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.685+0000 7fcb7a000640 1 -- 172.21.15.19:0/2544124611 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcb6c011950 con 0x7fcb7c1016e0 2024-03-18T15:17:31.687 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.685+0000 7fcb82200640 1 -- 172.21.15.19:0/2544124611 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb7c1a1ad0 con 0x7fcb7c1016e0 2024-03-18T15:17:31.687 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.685+0000 7fcb7a000640 1 -- 172.21.15.19:0/2544124611 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb6c01ab80 con 0x7fcb7c1016e0 2024-03-18T15:17:31.687 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.685+0000 7fcb82200640 1 -- 172.21.15.19:0/2544124611 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb7c1a1ff0 con 0x7fcb7c1016e0 2024-03-18T15:17:31.688 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.686+0000 7fcb82200640 1 -- 172.21.15.19:0/2544124611 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb7c0684f0 con 0x7fcb7c1016e0 2024-03-18T15:17:31.688 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.688+0000 7fcb7a000640 1 -- 172.21.15.19:0/2544124611 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fcb6c011b00 con 0x7fcb7c1016e0 2024-03-18T15:17:31.689 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.689+0000 7fcb7a000640 1 --2- 172.21.15.19:0/2544124611 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fcb600a2800 0x7fcb600a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:31.689 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.689+0000 7fcb7a000640 1 -- 172.21.15.19:0/2544124611 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fcb6c0cf460 con 0x7fcb7c1016e0 2024-03-18T15:17:31.689 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.689+0000 7fcb73e00640 1 --2- 172.21.15.19:0/2544124611 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fcb600a2800 0x7fcb600a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:31.690 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.690+0000 7fcb73e00640 1 --2- 172.21.15.19:0/2544124611 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fcb600a2800 0x7fcb600a4cc0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fcb6402c250 tx=0x7fcb6402cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:17:31.693 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.693+0000 7fcb7a000640 1 -- 172.21.15.19:0/2544124611 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fcb6c016030 con 0x7fcb7c1016e0 2024-03-18T15:17:31.717 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:17:31 smithi019 ceph-mon[31522]: pgmap v391: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T15:17:31.717 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:17:31 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/3627980740' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T15:17:31.717 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:17:31 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/845359106' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T15:17:31.717 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:17:31 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/1774387280' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T15:17:31.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:17:31 smithi033 ceph-mon[36648]: pgmap v391: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T15:17:31.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:17:31 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/3627980740' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T15:17:31.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:17:31 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/845359106' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T15:17:31.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:17:31 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/1774387280' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T15:17:31.895 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.894+0000 7fcb82200640 1 -- 172.21.15.19:0/2544124611 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fcb7c1a2320 con 0x7fcb7c1016e0 2024-03-18T15:17:31.895 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.895+0000 7fcb7a000640 1 -- 172.21.15.19:0/2544124611 <== mon.2 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7fcb6c09c0d0 con 0x7fcb7c1016e0 2024-03-18T15:17:31.896 INFO:teuthology.orchestra.run.smithi019.stdout:pg_num: 32 2024-03-18T15:17:31.899 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.898+0000 7fcb82200640 1 -- 172.21.15.19:0/2544124611 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fcb600a2800 msgr2=0x7fcb600a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:31.899 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.898+0000 7fcb82200640 1 --2- 172.21.15.19:0/2544124611 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fcb600a2800 0x7fcb600a4cc0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fcb6402c250 tx=0x7fcb6402cf90 comp rx=0 tx=0).stop 2024-03-18T15:17:31.899 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.898+0000 7fcb82200640 1 -- 172.21.15.19:0/2544124611 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcb7c1016e0 msgr2=0x7fcb7c199bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:31.899 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.898+0000 7fcb82200640 1 --2- 172.21.15.19:0/2544124611 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcb7c1016e0 0x7fcb7c199bc0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fcb6c00ea80 tx=0x7fcb6c00ef40 comp rx=0 tx=0).stop 2024-03-18T15:17:31.899 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.898+0000 7fcb82200640 1 -- 172.21.15.19:0/2544124611 shutdown_connections 2024-03-18T15:17:31.899 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.898+0000 7fcb82200640 1 --2- 172.21.15.19:0/2544124611 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fcb7c103290 0x7fcb7c1a1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:31.900 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.898+0000 7fcb82200640 1 --2- 172.21.15.19:0/2544124611 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcb7c1028d0 0x7fcb7c19a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:31.900 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.898+0000 7fcb82200640 1 --2- 172.21.15.19:0/2544124611 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fcb600a2800 0x7fcb600a4cc0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:31.900 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.899+0000 7fcb82200640 1 --2- 172.21.15.19:0/2544124611 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcb7c1016e0 0x7fcb7c199bc0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:31.900 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.899+0000 7fcb82200640 1 -- 172.21.15.19:0/2544124611 >> 172.21.15.19:0/2544124611 conn(0x7fcb7c0fcc50 msgr2=0x7fcb7c073a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:17:31.900 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.899+0000 7fcb82200640 1 -- 172.21.15.19:0/2544124611 shutdown_connections 2024-03-18T15:17:31.900 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:31.899+0000 7fcb82200640 1 -- 172.21.15.19:0/2544124611 wait complete. 2024-03-18T15:17:31.918 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T15:17:31.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:17:31 smithi104 ceph-mon[36070]: pgmap v391: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T15:17:31.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:17:31 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/3627980740' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T15:17:31.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:17:31 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/845359106' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T15:17:31.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:17:31 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/1774387280' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T15:17:32.141 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.141+0000 7fbf29400640 1 -- 172.21.15.19:0/2702846145 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf24101340 msgr2=0x7fbf241017c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:32.141 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.141+0000 7fbf29400640 1 --2- 172.21.15.19:0/2702846145 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf24101340 0x7fbf241017c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fbf1c009900 tx=0x7fbf1c02bc50 comp rx=0 tx=0).stop 2024-03-18T15:17:32.141 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.141+0000 7fbf29400640 1 -- 172.21.15.19:0/2702846145 shutdown_connections 2024-03-18T15:17:32.142 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.141+0000 7fbf29400640 1 --2- 172.21.15.19:0/2702846145 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf24101340 0x7fbf241017c0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:32.142 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.141+0000 7fbf29400640 1 --2- 172.21.15.19:0/2702846145 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbf241009a0 0x7fbf24100e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:32.142 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.141+0000 7fbf29400640 1 --2- 172.21.15.19:0/2702846145 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbf24100040 0x7fbf24100460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:32.142 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.141+0000 7fbf29400640 1 -- 172.21.15.19:0/2702846145 >> 172.21.15.19:0/2702846145 conn(0x7fbf240fbc40 msgr2=0x7fbf240fe080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:17:32.142 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.141+0000 7fbf29400640 1 -- 172.21.15.19:0/2702846145 shutdown_connections 2024-03-18T15:17:32.142 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.141+0000 7fbf29400640 1 -- 172.21.15.19:0/2702846145 wait complete. 2024-03-18T15:17:32.144 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.142+0000 7fbf29400640 1 Processor -- start 2024-03-18T15:17:32.144 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.142+0000 7fbf29400640 1 -- start start 2024-03-18T15:17:32.144 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.142+0000 7fbf29400640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf24100040 0x7fbf24199d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:32.144 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.142+0000 7fbf29400640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbf241009a0 0x7fbf2419a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:32.145 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.143+0000 7fbf29400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbf24101340 0x7fbf241a1290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:32.145 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.143+0000 7fbf29400640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fbf2410f640 con 0x7fbf241009a0 2024-03-18T15:17:32.145 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.143+0000 7fbf29400640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fbf2410f4c0 con 0x7fbf24101340 2024-03-18T15:17:32.145 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.143+0000 7fbf29400640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fbf2410f7c0 con 0x7fbf24100040 2024-03-18T15:17:32.145 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.143+0000 7fbf22800640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf24100040 0x7fbf24199d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:32.145 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.143+0000 7fbf22800640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf24100040 0x7fbf24199d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.19:58144/0 (socket says 172.21.15.19:58144) 2024-03-18T15:17:32.145 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.143+0000 7fbf22800640 1 -- 172.21.15.19:0/1509029380 learned_addr learned my addr 172.21.15.19:0/1509029380 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:17:32.145 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.143+0000 7fbf21e00640 1 --2- 172.21.15.19:0/1509029380 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbf241009a0 0x7fbf2419a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:32.145 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.143+0000 7fbf23200640 1 --2- 172.21.15.19:0/1509029380 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbf24101340 0x7fbf241a1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:32.145 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.143+0000 7fbf22800640 1 -- 172.21.15.19:0/1509029380 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbf24101340 msgr2=0x7fbf241a1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:32.145 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.143+0000 7fbf22800640 1 --2- 172.21.15.19:0/1509029380 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbf24101340 0x7fbf241a1290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:32.145 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.143+0000 7fbf22800640 1 -- 172.21.15.19:0/1509029380 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbf241009a0 msgr2=0x7fbf2419a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:32.145 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.143+0000 7fbf22800640 1 --2- 172.21.15.19:0/1509029380 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbf241009a0 0x7fbf2419a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:32.145 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.143+0000 7fbf22800640 1 -- 172.21.15.19:0/1509029380 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf241a1930 con 0x7fbf24100040 2024-03-18T15:17:32.145 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.143+0000 7fbf22800640 1 --2- 172.21.15.19:0/1509029380 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf24100040 0x7fbf24199d20 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fbf1400e9f0 tx=0x7fbf1400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:17:32.146 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.144+0000 7fbf1b400640 1 -- 172.21.15.19:0/1509029380 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf1400cd80 con 0x7fbf24100040 2024-03-18T15:17:32.146 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.144+0000 7fbf1b400640 1 -- 172.21.15.19:0/1509029380 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbf14010910 con 0x7fbf24100040 2024-03-18T15:17:32.146 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.144+0000 7fbf29400640 1 -- 172.21.15.19:0/1509029380 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf241a1c20 con 0x7fbf24100040 2024-03-18T15:17:32.146 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.144+0000 7fbf1b400640 1 -- 172.21.15.19:0/1509029380 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf14018990 con 0x7fbf24100040 2024-03-18T15:17:32.146 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.144+0000 7fbf29400640 1 -- 172.21.15.19:0/1509029380 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf241a2170 con 0x7fbf24100040 2024-03-18T15:17:32.146 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.145+0000 7fbf29400640 1 -- 172.21.15.19:0/1509029380 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf2410aee0 con 0x7fbf24100040 2024-03-18T15:17:32.147 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.147+0000 7fbf1b400640 1 -- 172.21.15.19:0/1509029380 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fbf14010ac0 con 0x7fbf24100040 2024-03-18T15:17:32.148 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.148+0000 7fbf1b400640 1 --2- 172.21.15.19:0/1509029380 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fbf080a2800 0x7fbf080a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:32.148 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.149+0000 7fbf1b400640 1 -- 172.21.15.19:0/1509029380 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fbf140cf630 con 0x7fbf24100040 2024-03-18T15:17:32.149 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.149+0000 7fbf21e00640 1 --2- 172.21.15.19:0/1509029380 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fbf080a2800 0x7fbf080a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:32.149 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.149+0000 7fbf21e00640 1 --2- 172.21.15.19:0/1509029380 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fbf080a2800 0x7fbf080a4cc0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fbf0c005fd0 tx=0x7fbf0c009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:17:32.150 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.150+0000 7fbf1b400640 1 -- 172.21.15.19:0/1509029380 <== mon.1 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fbf1409bc50 con 0x7fbf24100040 2024-03-18T15:17:32.386 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.386+0000 7fbf29400640 1 -- 172.21.15.19:0/1509029380 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbf2402d1b0 con 0x7fbf24100040 2024-03-18T15:17:32.387 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.387+0000 7fbf1b400640 1 -- 172.21.15.19:0/1509029380 <== mon.1 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11927 (secure 0 0 0) 0x7fbf1409fcf0 con 0x7fbf24100040 2024-03-18T15:17:32.387 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-18T15:17:32.388 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6835/1135329545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6834","nonce":1135329545},{"type":"v1","addr":"172.21.15.19:6835","nonce":1135329545}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14457,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6839/3833518557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6838","nonce":3833518557},{"type":"v1","addr":"172.21.15.19:6839","nonce":3833518557}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T15:15:10.589737+0000","modified":"2024-03-18T15:15:46.979999+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24560,"mds_1":24529,"mds_2":24511,"mds_3":24506,"mds_4":24499},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14475":{"gid":14475,"name":"a","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.19:6841/3176583319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6840","nonce":3176583319},{"type":"v1","addr":"172.21.15.19:6841","nonce":3176583319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24433":{"gid":24433,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6833/482725809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":482725809},{"type":"v1","addr":"172.21.15.104:6833","nonce":482725809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24434":{"gid":24434,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/2272279159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":2272279159},{"type":"v1","addr":"172.21.15.33:6833","nonce":2272279159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24469":{"gid":24469,"name":"d","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.19:6837/1778019859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6836","nonce":1778019859},{"type":"v1","addr":"172.21.15.19:6837","nonce":1778019859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6835/4051613151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":4051613151},{"type":"v1","addr":"172.21.15.33:6835","nonce":4051613151}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":17,"addr":"172.21.15.104:6835/3788595957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3788595957},{"type":"v1","addr":"172.21.15.104:6835","nonce":3788595957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":15,"addr":"172.21.15.33:6837/160187378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":160187378},{"type":"v1","addr":"172.21.15.33:6837","nonce":160187378}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24511":{"gid":24511,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":16,"addr":"172.21.15.104:6837/260984674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":260984674},{"type":"v1","addr":"172.21.15.104:6837","nonce":260984674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24529":{"gid":24529,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.104:6839/805341436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6838","nonce":805341436},{"type":"v1","addr":"172.21.15.104:6839","nonce":805341436}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.33:6839/1903591535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":1903591535},{"type":"v1","addr":"172.21.15.33:6839","nonce":1903591535}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24560,"qdb_cluster":[24506,24499,24529,24511,24560]},"id":1}]} 2024-03-18T15:17:32.388 INFO:teuthology.orchestra.run.smithi019.stderr:dumped fsmap epoch 26 2024-03-18T15:17:32.391 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.391+0000 7fbf29400640 1 -- 172.21.15.19:0/1509029380 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fbf080a2800 msgr2=0x7fbf080a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:32.391 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.391+0000 7fbf29400640 1 --2- 172.21.15.19:0/1509029380 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fbf080a2800 0x7fbf080a4cc0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fbf0c005fd0 tx=0x7fbf0c009500 comp rx=0 tx=0).stop 2024-03-18T15:17:32.391 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.391+0000 7fbf29400640 1 -- 172.21.15.19:0/1509029380 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf24100040 msgr2=0x7fbf24199d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:32.391 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.391+0000 7fbf29400640 1 --2- 172.21.15.19:0/1509029380 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf24100040 0x7fbf24199d20 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fbf1400e9f0 tx=0x7fbf1400eeb0 comp rx=0 tx=0).stop 2024-03-18T15:17:32.391 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.391+0000 7fbf29400640 1 -- 172.21.15.19:0/1509029380 shutdown_connections 2024-03-18T15:17:32.391 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.391+0000 7fbf29400640 1 --2- 172.21.15.19:0/1509029380 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fbf080a2800 0x7fbf080a4cc0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:32.392 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.391+0000 7fbf29400640 1 --2- 172.21.15.19:0/1509029380 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbf24101340 0x7fbf241a1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:32.392 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.391+0000 7fbf29400640 1 --2- 172.21.15.19:0/1509029380 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbf241009a0 0x7fbf2419a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:32.392 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.391+0000 7fbf29400640 1 --2- 172.21.15.19:0/1509029380 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf24100040 0x7fbf24199d20 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:32.392 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.391+0000 7fbf29400640 1 -- 172.21.15.19:0/1509029380 >> 172.21.15.19:0/1509029380 conn(0x7fbf240fbc40 msgr2=0x7fbf240fd320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:17:32.392 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.391+0000 7fbf29400640 1 -- 172.21.15.19:0/1509029380 shutdown_connections 2024-03-18T15:17:32.392 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.391+0000 7fbf29400640 1 -- 172.21.15.19:0/1509029380 wait complete. 2024-03-18T15:17:32.409 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T15:17:32.629 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.629+0000 7f744f400640 1 -- 172.21.15.19:0/520234465 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7448107e00 msgr2=0x7f7448108220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:32.629 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.629+0000 7f744f400640 1 --2- 172.21.15.19:0/520234465 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7448107e00 0x7f7448108220 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f743c009900 tx=0x7f743c02bc50 comp rx=0 tx=0).stop 2024-03-18T15:17:32.629 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.629+0000 7f744f400640 1 -- 172.21.15.19:0/520234465 shutdown_connections 2024-03-18T15:17:32.629 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.629+0000 7f744f400640 1 --2- 172.21.15.19:0/520234465 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f74481099b0 0x7f7448110240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:32.629 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.629+0000 7f744f400640 1 --2- 172.21.15.19:0/520234465 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7448108ff0 0x7f7448109470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:32.629 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.629+0000 7f744f400640 1 --2- 172.21.15.19:0/520234465 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7448107e00 0x7f7448108220 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:32.629 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.629+0000 7f744f400640 1 -- 172.21.15.19:0/520234465 >> 172.21.15.19:0/520234465 conn(0x7f7448076dc0 msgr2=0x7f74480791e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:17:32.630 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.629+0000 7f744f400640 1 -- 172.21.15.19:0/520234465 shutdown_connections 2024-03-18T15:17:32.630 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.629+0000 7f744f400640 1 -- 172.21.15.19:0/520234465 wait complete. 2024-03-18T15:17:32.630 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.630+0000 7f744f400640 1 Processor -- start 2024-03-18T15:17:32.630 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.630+0000 7f744f400640 1 -- start start 2024-03-18T15:17:32.630 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.631+0000 7f744f400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7448107e00 0x7f744819e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:32.632 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.631+0000 7f744f400640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7448108ff0 0x7f744819e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:32.632 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.631+0000 7f744f400640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f74481099b0 0x7f74481a56c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:32.632 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.631+0000 7f744f400640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f7448113ac0 con 0x7f74481099b0 2024-03-18T15:17:32.632 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.631+0000 7f744f400640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7448113940 con 0x7f7448107e00 2024-03-18T15:17:32.632 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.631+0000 7f744f400640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f7448113c40 con 0x7f7448108ff0 2024-03-18T15:17:32.632 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.631+0000 7f744ca00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7448107e00 0x7f744819e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:32.632 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.631+0000 7f744ca00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7448107e00 0x7f744819e110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:39192/0 (socket says 172.21.15.19:39192) 2024-03-18T15:17:32.632 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.631+0000 7f744ca00640 1 -- 172.21.15.19:0/1643381460 learned_addr learned my addr 172.21.15.19:0/1643381460 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:17:32.633 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.631+0000 7f7447e00640 1 --2- 172.21.15.19:0/1643381460 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7448108ff0 0x7f744819e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:32.633 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.631+0000 7f744d400640 1 --2- 172.21.15.19:0/1643381460 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f74481099b0 0x7f74481a56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:32.633 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.631+0000 7f744ca00640 1 -- 172.21.15.19:0/1643381460 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7448108ff0 msgr2=0x7f744819e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:32.633 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.631+0000 7f744ca00640 1 --2- 172.21.15.19:0/1643381460 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7448108ff0 0x7f744819e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:32.633 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.631+0000 7f744ca00640 1 -- 172.21.15.19:0/1643381460 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f74481099b0 msgr2=0x7f74481a56c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:32.633 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.631+0000 7f744ca00640 1 --2- 172.21.15.19:0/1643381460 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f74481099b0 0x7f74481a56c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:32.633 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.631+0000 7f744ca00640 1 -- 172.21.15.19:0/1643381460 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74481a5d30 con 0x7f7448107e00 2024-03-18T15:17:32.633 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.632+0000 7f744ca00640 1 --2- 172.21.15.19:0/1643381460 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7448107e00 0x7f744819e110 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f743c02c260 tx=0x7f743c02c670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:17:32.633 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.632+0000 7f7445600640 1 -- 172.21.15.19:0/1643381460 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f743c041070 con 0x7f7448107e00 2024-03-18T15:17:32.633 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.632+0000 7f7445600640 1 -- 172.21.15.19:0/1643381460 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f743c004450 con 0x7f7448107e00 2024-03-18T15:17:32.633 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.632+0000 7f7445600640 1 -- 172.21.15.19:0/1643381460 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f743c034440 con 0x7f7448107e00 2024-03-18T15:17:32.633 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.632+0000 7f744f400640 1 -- 172.21.15.19:0/1643381460 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74481a5fc0 con 0x7f7448107e00 2024-03-18T15:17:32.634 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.633+0000 7f744f400640 1 -- 172.21.15.19:0/1643381460 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74481a63d0 con 0x7f7448107e00 2024-03-18T15:17:32.634 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.634+0000 7f744f400640 1 -- 172.21.15.19:0/1643381460 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f744810f0c0 con 0x7f7448107e00 2024-03-18T15:17:32.635 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.635+0000 7f7445600640 1 -- 172.21.15.19:0/1643381460 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f743c044070 con 0x7f7448107e00 2024-03-18T15:17:32.637 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.637+0000 7f7445600640 1 --2- 172.21.15.19:0/1643381460 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f74300a2880 0x7f74300a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:32.637 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.637+0000 7f7445600640 1 -- 172.21.15.19:0/1643381460 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f743c0f08b0 con 0x7f7448107e00 2024-03-18T15:17:32.637 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.637+0000 7f7447e00640 1 --2- 172.21.15.19:0/1643381460 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f74300a2880 0x7f74300a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:32.638 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.638+0000 7f7447e00640 1 --2- 172.21.15.19:0/1643381460 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f74300a2880 0x7f74300a4d40 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f744819f620 tx=0x7f7434005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:17:32.646 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.646+0000 7f7445600640 1 -- 172.21.15.19:0/1643381460 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f743c0bced0 con 0x7f7448107e00 2024-03-18T15:17:32.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:17:32 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/2544124611' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T15:17:32.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:17:32 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/1509029380' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:17:32.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:17:32 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/2544124611' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T15:17:32.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:17:32 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/1509029380' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:17:32.851 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.851+0000 7f744f400640 1 -- 172.21.15.19:0/1643381460 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f74480624b0 con 0x7f7448107e00 2024-03-18T15:17:32.853 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.853+0000 7f7445600640 1 -- 172.21.15.19:0/1643381460 <== mon.2 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24809 (secure 0 0 0) 0x7f743c0c0f70 con 0x7f7448107e00 2024-03-18T15:17:32.853 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-18T15:17:32.854 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":93,"fsid":"d4dc2ba6-e538-11ee-95c9-87774f69a715","created":"2024-03-18T15:05:57.538551+0000","modified":"2024-03-18T15:17:16.033313+0000","last_up_change":"2024-03-18T15:14:12.009522+0000","last_in_change":"2024-03-18T15:13:52.555387+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T15:09:56.843349+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T15:15:08.941047+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T15:15:10.044671+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T15:15:13.621916+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"37b5a5cd-27ae-4e77-9b05-774b817b2ee9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6802","nonce":402333948},{"type":"v1","addr":"172.21.15.19:6803","nonce":402333948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6804","nonce":402333948},{"type":"v1","addr":"172.21.15.19:6805","nonce":402333948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6808","nonce":402333948},{"type":"v1","addr":"172.21.15.19:6809","nonce":402333948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6806","nonce":402333948},{"type":"v1","addr":"172.21.15.19:6807","nonce":402333948}]},"public_addr":"172.21.15.19:6803/402333948","cluster_addr":"172.21.15.19:6805/402333948","heartbeat_back_addr":"172.21.15.19:6809/402333948","heartbeat_front_addr":"172.21.15.19:6807/402333948","state":["exists","up"]},{"osd":1,"uuid":"3a1134e8-ede7-4dbd-9be1-eefd2cbb43bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1951880113},{"type":"v1","addr":"172.21.15.33:6801","nonce":1951880113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1951880113},{"type":"v1","addr":"172.21.15.33:6803","nonce":1951880113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1951880113},{"type":"v1","addr":"172.21.15.33:6807","nonce":1951880113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1951880113},{"type":"v1","addr":"172.21.15.33:6805","nonce":1951880113}]},"public_addr":"172.21.15.33:6801/1951880113","cluster_addr":"172.21.15.33:6803/1951880113","heartbeat_back_addr":"172.21.15.33:6807/1951880113","heartbeat_front_addr":"172.21.15.33:6805/1951880113","state":["exists","up"]},{"osd":2,"uuid":"f383a164-a3a0-49fd-a9e8-9c2178d8cfb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":1004727949},{"type":"v1","addr":"172.21.15.104:6801","nonce":1004727949}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":1004727949},{"type":"v1","addr":"172.21.15.104:6803","nonce":1004727949}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":1004727949},{"type":"v1","addr":"172.21.15.104:6807","nonce":1004727949}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":1004727949},{"type":"v1","addr":"172.21.15.104:6805","nonce":1004727949}]},"public_addr":"172.21.15.104:6801/1004727949","cluster_addr":"172.21.15.104:6803/1004727949","heartbeat_back_addr":"172.21.15.104:6807/1004727949","heartbeat_front_addr":"172.21.15.104:6805/1004727949","state":["exists","up"]},{"osd":3,"uuid":"c409e116-28bf-45dd-a791-539af22abe1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6810","nonce":3571525854},{"type":"v1","addr":"172.21.15.19:6811","nonce":3571525854}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6812","nonce":3571525854},{"type":"v1","addr":"172.21.15.19:6813","nonce":3571525854}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6816","nonce":3571525854},{"type":"v1","addr":"172.21.15.19:6817","nonce":3571525854}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6814","nonce":3571525854},{"type":"v1","addr":"172.21.15.19:6815","nonce":3571525854}]},"public_addr":"172.21.15.19:6811/3571525854","cluster_addr":"172.21.15.19:6813/3571525854","heartbeat_back_addr":"172.21.15.19:6817/3571525854","heartbeat_front_addr":"172.21.15.19:6815/3571525854","state":["exists","up"]},{"osd":4,"uuid":"696eb719-0b3e-43ae-b5f1-ccd73a3b2843","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":624901749},{"type":"v1","addr":"172.21.15.33:6809","nonce":624901749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":624901749},{"type":"v1","addr":"172.21.15.33:6811","nonce":624901749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":624901749},{"type":"v1","addr":"172.21.15.33:6815","nonce":624901749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":624901749},{"type":"v1","addr":"172.21.15.33:6813","nonce":624901749}]},"public_addr":"172.21.15.33:6809/624901749","cluster_addr":"172.21.15.33:6811/624901749","heartbeat_back_addr":"172.21.15.33:6815/624901749","heartbeat_front_addr":"172.21.15.33:6813/624901749","state":["exists","up"]},{"osd":5,"uuid":"21789779-8bb9-42c7-b9ee-437985f11e3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":1258208297},{"type":"v1","addr":"172.21.15.104:6809","nonce":1258208297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":1258208297},{"type":"v1","addr":"172.21.15.104:6811","nonce":1258208297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":1258208297},{"type":"v1","addr":"172.21.15.104:6815","nonce":1258208297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":1258208297},{"type":"v1","addr":"172.21.15.104:6813","nonce":1258208297}]},"public_addr":"172.21.15.104:6809/1258208297","cluster_addr":"172.21.15.104:6811/1258208297","heartbeat_back_addr":"172.21.15.104:6815/1258208297","heartbeat_front_addr":"172.21.15.104:6813/1258208297","state":["exists","up"]},{"osd":6,"uuid":"0e92a2ef-43cd-487c-9203-e219158aef8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6818","nonce":2782482301},{"type":"v1","addr":"172.21.15.19:6819","nonce":2782482301}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6820","nonce":2782482301},{"type":"v1","addr":"172.21.15.19:6821","nonce":2782482301}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6824","nonce":2782482301},{"type":"v1","addr":"172.21.15.19:6825","nonce":2782482301}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6822","nonce":2782482301},{"type":"v1","addr":"172.21.15.19:6823","nonce":2782482301}]},"public_addr":"172.21.15.19:6819/2782482301","cluster_addr":"172.21.15.19:6821/2782482301","heartbeat_back_addr":"172.21.15.19:6825/2782482301","heartbeat_front_addr":"172.21.15.19:6823/2782482301","state":["exists","up"]},{"osd":7,"uuid":"a16e39a1-3685-491d-b376-3e80a2084072","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":4238160988},{"type":"v1","addr":"172.21.15.33:6817","nonce":4238160988}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":4238160988},{"type":"v1","addr":"172.21.15.33:6819","nonce":4238160988}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":4238160988},{"type":"v1","addr":"172.21.15.33:6823","nonce":4238160988}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":4238160988},{"type":"v1","addr":"172.21.15.33:6821","nonce":4238160988}]},"public_addr":"172.21.15.33:6817/4238160988","cluster_addr":"172.21.15.33:6819/4238160988","heartbeat_back_addr":"172.21.15.33:6823/4238160988","heartbeat_front_addr":"172.21.15.33:6821/4238160988","state":["exists","up"]},{"osd":8,"uuid":"d5c3fa92-cd5d-4349-9c7d-d5666bacde16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":1014779813},{"type":"v1","addr":"172.21.15.104:6817","nonce":1014779813}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":1014779813},{"type":"v1","addr":"172.21.15.104:6819","nonce":1014779813}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":1014779813},{"type":"v1","addr":"172.21.15.104:6823","nonce":1014779813}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":1014779813},{"type":"v1","addr":"172.21.15.104:6821","nonce":1014779813}]},"public_addr":"172.21.15.104:6817/1014779813","cluster_addr":"172.21.15.104:6819/1014779813","heartbeat_back_addr":"172.21.15.104:6823/1014779813","heartbeat_front_addr":"172.21.15.104:6821/1014779813","state":["exists","up"]},{"osd":9,"uuid":"eed2e9b2-26d3-4c46-88ec-34b5596fe991","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6826","nonce":2641151009},{"type":"v1","addr":"172.21.15.19:6827","nonce":2641151009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6828","nonce":2641151009},{"type":"v1","addr":"172.21.15.19:6829","nonce":2641151009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6832","nonce":2641151009},{"type":"v1","addr":"172.21.15.19:6833","nonce":2641151009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6830","nonce":2641151009},{"type":"v1","addr":"172.21.15.19:6831","nonce":2641151009}]},"public_addr":"172.21.15.19:6827/2641151009","cluster_addr":"172.21.15.19:6829/2641151009","heartbeat_back_addr":"172.21.15.19:6833/2641151009","heartbeat_front_addr":"172.21.15.19:6831/2641151009","state":["exists","up"]},{"osd":10,"uuid":"f3c35ea5-fbe7-4dc4-93c5-b92d1a714db3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":1346058869},{"type":"v1","addr":"172.21.15.33:6825","nonce":1346058869}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":1346058869},{"type":"v1","addr":"172.21.15.33:6827","nonce":1346058869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":1346058869},{"type":"v1","addr":"172.21.15.33:6831","nonce":1346058869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":1346058869},{"type":"v1","addr":"172.21.15.33:6829","nonce":1346058869}]},"public_addr":"172.21.15.33:6825/1346058869","cluster_addr":"172.21.15.33:6827/1346058869","heartbeat_back_addr":"172.21.15.33:6831/1346058869","heartbeat_front_addr":"172.21.15.33:6829/1346058869","state":["exists","up"]},{"osd":11,"uuid":"c8b51571-f807-4b62-90b6-7529e09e59da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":3759498281},{"type":"v1","addr":"172.21.15.104:6825","nonce":3759498281}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":3759498281},{"type":"v1","addr":"172.21.15.104:6827","nonce":3759498281}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":3759498281},{"type":"v1","addr":"172.21.15.104:6831","nonce":3759498281}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":3759498281},{"type":"v1","addr":"172.21.15.104:6829","nonce":3759498281}]},"public_addr":"172.21.15.104:6825/3759498281","cluster_addr":"172.21.15.104:6827/3759498281","heartbeat_back_addr":"172.21.15.104:6831/3759498281","heartbeat_front_addr":"172.21.15.104:6829/3759498281","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:09:05.267795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:09:30.234068+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:09:53.870232+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:10:26.471296+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:10:53.180341+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:11:17.801361+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:11:51.364816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:12:18.306408+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:12:43.978744+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:13:18.213964+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:13:45.437544+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:14:10.148580+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":3,"to":6}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"3.37","mappings":[{"from":5,"to":8}]},{"pgid":"3.3a","mappings":[{"from":4,"to":7}]},{"pgid":"4.8","mappings":[{"from":5,"to":8}]},{"pgid":"4.f","mappings":[{"from":9,"to":10}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.19:0/3860783274":"2024-03-19T15:07:14.150983+0000","172.21.15.19:0/2737223213":"2024-03-19T15:07:14.150983+0000","172.21.15.19:0/176699025":"2024-03-19T15:06:38.688405+0000","172.21.15.19:0/2316969005":"2024-03-19T15:07:14.150983+0000","172.21.15.19:6801/2776008962":"2024-03-19T15:07:14.150983+0000","172.21.15.19:0/2793980440":"2024-03-19T15:06:38.688405+0000","172.21.15.19:6800/3264337803":"2024-03-19T15:06:38.688405+0000","172.21.15.19:6801/3264337803":"2024-03-19T15:06:38.688405+0000","172.21.15.19:0/3166705295":"2024-03-19T15:06:38.688405+0000","172.21.15.19:6800/2776008962":"2024-03-19T15:07:14.150983+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T15:17:32.856 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.855+0000 7f744f400640 1 -- 172.21.15.19:0/1643381460 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f74300a2880 msgr2=0x7f74300a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:32.856 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.855+0000 7f744f400640 1 --2- 172.21.15.19:0/1643381460 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f74300a2880 0x7f74300a4d40 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f744819f620 tx=0x7f7434005f90 comp rx=0 tx=0).stop 2024-03-18T15:17:32.856 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.856+0000 7f744f400640 1 -- 172.21.15.19:0/1643381460 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7448107e00 msgr2=0x7f744819e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:32.856 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.856+0000 7f744f400640 1 --2- 172.21.15.19:0/1643381460 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7448107e00 0x7f744819e110 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f743c02c260 tx=0x7f743c02c670 comp rx=0 tx=0).stop 2024-03-18T15:17:32.856 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.856+0000 7f744f400640 1 -- 172.21.15.19:0/1643381460 shutdown_connections 2024-03-18T15:17:32.856 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.856+0000 7f744f400640 1 --2- 172.21.15.19:0/1643381460 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f74481099b0 0x7f74481a56c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:32.856 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.856+0000 7f744f400640 1 --2- 172.21.15.19:0/1643381460 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7448108ff0 0x7f744819e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:32.856 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.856+0000 7f744f400640 1 --2- 172.21.15.19:0/1643381460 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f74300a2880 0x7f74300a4d40 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:32.857 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.856+0000 7f744f400640 1 --2- 172.21.15.19:0/1643381460 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7448107e00 0x7f744819e110 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:32.857 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.856+0000 7f744f400640 1 -- 172.21.15.19:0/1643381460 >> 172.21.15.19:0/1643381460 conn(0x7f7448076dc0 msgr2=0x7f744810c290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:17:32.857 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.856+0000 7f744f400640 1 -- 172.21.15.19:0/1643381460 shutdown_connections 2024-03-18T15:17:32.857 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:32.856+0000 7f744f400640 1 -- 172.21.15.19:0/1643381460 wait complete. 2024-03-18T15:17:32.877 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T15:17:32.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:17:32 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/2544124611' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T15:17:32.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:17:32 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/1509029380' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:17:33.102 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.102+0000 7f0eada00640 1 -- 172.21.15.19:0/2066699960 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0ea8077d80 msgr2=0x7f0ea8078620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:33.102 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.102+0000 7f0eada00640 1 --2- 172.21.15.19:0/2066699960 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0ea8077d80 0x7f0ea8078620 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f0ea0009ca0 tx=0x7f0ea002c0f0 comp rx=0 tx=0).stop 2024-03-18T15:17:33.102 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.102+0000 7f0eada00640 1 -- 172.21.15.19:0/2066699960 shutdown_connections 2024-03-18T15:17:33.102 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.102+0000 7f0eada00640 1 --2- 172.21.15.19:0/2066699960 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0ea8077d80 0x7f0ea8078620 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:33.102 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.102+0000 7f0eada00640 1 --2- 172.21.15.19:0/2066699960 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0ea80773e0 0x7f0ea8077840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:33.102 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.102+0000 7f0eada00640 1 --2- 172.21.15.19:0/2066699960 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0ea80761a0 0x7f0ea80765c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:33.103 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.102+0000 7f0eada00640 1 -- 172.21.15.19:0/2066699960 >> 172.21.15.19:0/2066699960 conn(0x7f0ea80fed50 msgr2=0x7f0ea8101190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:17:33.103 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.102+0000 7f0eada00640 1 -- 172.21.15.19:0/2066699960 shutdown_connections 2024-03-18T15:17:33.103 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.102+0000 7f0eada00640 1 -- 172.21.15.19:0/2066699960 wait complete. 2024-03-18T15:17:33.103 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.104+0000 7f0eada00640 1 Processor -- start 2024-03-18T15:17:33.104 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.104+0000 7f0eada00640 1 -- start start 2024-03-18T15:17:33.105 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.104+0000 7f0eada00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0ea80761a0 0x7f0ea819de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:33.105 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.104+0000 7f0eada00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0ea80773e0 0x7f0ea819e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:33.105 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.104+0000 7f0eada00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0ea8077d80 0x7f0ea81a5400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:33.105 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.104+0000 7f0eada00640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f0ea8113870 con 0x7f0ea80773e0 2024-03-18T15:17:33.105 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.104+0000 7f0eada00640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f0ea81136f0 con 0x7f0ea8077d80 2024-03-18T15:17:33.105 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.104+0000 7f0eada00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f0ea81139f0 con 0x7f0ea80761a0 2024-03-18T15:17:33.105 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.105+0000 7f0ea7e00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0ea80761a0 0x7f0ea819de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:33.105 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.105+0000 7f0ea7e00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0ea80761a0 0x7f0ea819de50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.19:58182/0 (socket says 172.21.15.19:58182) 2024-03-18T15:17:33.105 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.105+0000 7f0ea7e00640 1 -- 172.21.15.19:0/4009405727 learned_addr learned my addr 172.21.15.19:0/4009405727 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:17:33.105 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.105+0000 7f0ea7400640 1 --2- 172.21.15.19:0/4009405727 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0ea80773e0 0x7f0ea819e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:33.105 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.105+0000 7f0ead000640 1 --2- 172.21.15.19:0/4009405727 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0ea8077d80 0x7f0ea81a5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:33.105 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.105+0000 7f0ea7e00640 1 -- 172.21.15.19:0/4009405727 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0ea8077d80 msgr2=0x7f0ea81a5400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:33.106 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.105+0000 7f0ea7e00640 1 --2- 172.21.15.19:0/4009405727 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0ea8077d80 0x7f0ea81a5400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:33.106 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.105+0000 7f0ea7e00640 1 -- 172.21.15.19:0/4009405727 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0ea80773e0 msgr2=0x7f0ea819e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:33.106 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.105+0000 7f0ea7e00640 1 --2- 172.21.15.19:0/4009405727 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0ea80773e0 0x7f0ea819e390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:33.106 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.105+0000 7f0ea7e00640 1 -- 172.21.15.19:0/4009405727 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ea81a5a70 con 0x7f0ea80761a0 2024-03-18T15:17:33.106 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.105+0000 7f0ea7e00640 1 --2- 172.21.15.19:0/4009405727 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0ea80761a0 0x7f0ea819de50 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f0e9800d950 tx=0x7f0e9800de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:17:33.107 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.105+0000 7f0ea4c00640 1 -- 172.21.15.19:0/4009405727 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e98017030 con 0x7f0ea80761a0 2024-03-18T15:17:33.107 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.105+0000 7f0eada00640 1 -- 172.21.15.19:0/4009405727 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ea81a5d60 con 0x7f0ea80761a0 2024-03-18T15:17:33.107 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.105+0000 7f0ea4c00640 1 -- 172.21.15.19:0/4009405727 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0e98004020 con 0x7f0ea80761a0 2024-03-18T15:17:33.107 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.106+0000 7f0ea4c00640 1 -- 172.21.15.19:0/4009405727 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e980047a0 con 0x7f0ea80761a0 2024-03-18T15:17:33.107 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.106+0000 7f0eada00640 1 -- 172.21.15.19:0/4009405727 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ea81a61f0 con 0x7f0ea80761a0 2024-03-18T15:17:33.107 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.107+0000 7f0eada00640 1 -- 172.21.15.19:0/4009405727 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ea8110150 con 0x7f0ea80761a0 2024-03-18T15:17:33.109 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.109+0000 7f0ea4c00640 1 -- 172.21.15.19:0/4009405727 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f0e980041d0 con 0x7f0ea80761a0 2024-03-18T15:17:33.110 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.110+0000 7f0ea4c00640 1 --2- 172.21.15.19:0/4009405727 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f0e8c0a2800 0x7f0e8c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:33.110 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.110+0000 7f0ea4c00640 1 -- 172.21.15.19:0/4009405727 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f0e980ce9e0 con 0x7f0ea80761a0 2024-03-18T15:17:33.111 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.111+0000 7f0ea7400640 1 --2- 172.21.15.19:0/4009405727 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f0e8c0a2800 0x7f0e8c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:33.111 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.112+0000 7f0ea7400640 1 --2- 172.21.15.19:0/4009405727 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f0e8c0a2800 0x7f0e8c0a4cc0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f0ea819f360 tx=0x7f0e94005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:17:33.119 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.119+0000 7f0ea4c00640 1 -- 172.21.15.19:0/4009405727 <== mon.1 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0e9809b650 con 0x7f0ea80761a0 2024-03-18T15:17:33.364 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.364+0000 7f0eada00640 1 -- 172.21.15.19:0/4009405727 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0ea802d1b0 con 0x7f0ea80761a0 2024-03-18T15:17:33.365 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.365+0000 7f0ea4c00640 1 -- 172.21.15.19:0/4009405727 <== mon.1 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11927 (secure 0 0 0) 0x7f0e9809af50 con 0x7f0ea80761a0 2024-03-18T15:17:33.365 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-18T15:17:33.365 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6835/1135329545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6834","nonce":1135329545},{"type":"v1","addr":"172.21.15.19:6835","nonce":1135329545}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14457,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6839/3833518557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6838","nonce":3833518557},{"type":"v1","addr":"172.21.15.19:6839","nonce":3833518557}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T15:15:10.589737+0000","modified":"2024-03-18T15:15:46.979999+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24560,"mds_1":24529,"mds_2":24511,"mds_3":24506,"mds_4":24499},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14475":{"gid":14475,"name":"a","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.19:6841/3176583319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6840","nonce":3176583319},{"type":"v1","addr":"172.21.15.19:6841","nonce":3176583319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24433":{"gid":24433,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6833/482725809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":482725809},{"type":"v1","addr":"172.21.15.104:6833","nonce":482725809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24434":{"gid":24434,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/2272279159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":2272279159},{"type":"v1","addr":"172.21.15.33:6833","nonce":2272279159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24469":{"gid":24469,"name":"d","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.19:6837/1778019859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6836","nonce":1778019859},{"type":"v1","addr":"172.21.15.19:6837","nonce":1778019859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6835/4051613151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":4051613151},{"type":"v1","addr":"172.21.15.33:6835","nonce":4051613151}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":17,"addr":"172.21.15.104:6835/3788595957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3788595957},{"type":"v1","addr":"172.21.15.104:6835","nonce":3788595957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":15,"addr":"172.21.15.33:6837/160187378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":160187378},{"type":"v1","addr":"172.21.15.33:6837","nonce":160187378}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24511":{"gid":24511,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":16,"addr":"172.21.15.104:6837/260984674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":260984674},{"type":"v1","addr":"172.21.15.104:6837","nonce":260984674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24529":{"gid":24529,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.104:6839/805341436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6838","nonce":805341436},{"type":"v1","addr":"172.21.15.104:6839","nonce":805341436}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.33:6839/1903591535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":1903591535},{"type":"v1","addr":"172.21.15.33:6839","nonce":1903591535}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24560,"qdb_cluster":[24506,24499,24529,24511,24560]},"id":1}]} 2024-03-18T15:17:33.366 INFO:teuthology.orchestra.run.smithi019.stderr:dumped fsmap epoch 26 2024-03-18T15:17:33.368 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.368+0000 7f0eada00640 1 -- 172.21.15.19:0/4009405727 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f0e8c0a2800 msgr2=0x7f0e8c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:33.368 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.368+0000 7f0eada00640 1 --2- 172.21.15.19:0/4009405727 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f0e8c0a2800 0x7f0e8c0a4cc0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f0ea819f360 tx=0x7f0e94005f90 comp rx=0 tx=0).stop 2024-03-18T15:17:33.369 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.368+0000 7f0eada00640 1 -- 172.21.15.19:0/4009405727 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0ea80761a0 msgr2=0x7f0ea819de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:33.369 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.368+0000 7f0eada00640 1 --2- 172.21.15.19:0/4009405727 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0ea80761a0 0x7f0ea819de50 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f0e9800d950 tx=0x7f0e9800de10 comp rx=0 tx=0).stop 2024-03-18T15:17:33.369 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.368+0000 7f0eada00640 1 -- 172.21.15.19:0/4009405727 shutdown_connections 2024-03-18T15:17:33.369 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.368+0000 7f0eada00640 1 --2- 172.21.15.19:0/4009405727 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f0e8c0a2800 0x7f0e8c0a4cc0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:33.369 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.368+0000 7f0eada00640 1 --2- 172.21.15.19:0/4009405727 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0ea8077d80 0x7f0ea81a5400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:33.369 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.368+0000 7f0eada00640 1 --2- 172.21.15.19:0/4009405727 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0ea80773e0 0x7f0ea819e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:33.369 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.368+0000 7f0eada00640 1 --2- 172.21.15.19:0/4009405727 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0ea80761a0 0x7f0ea819de50 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:33.370 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.368+0000 7f0eada00640 1 -- 172.21.15.19:0/4009405727 >> 172.21.15.19:0/4009405727 conn(0x7f0ea80fed50 msgr2=0x7f0ea8100850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:17:33.370 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.368+0000 7f0eada00640 1 -- 172.21.15.19:0/4009405727 shutdown_connections 2024-03-18T15:17:33.370 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.368+0000 7f0eada00640 1 -- 172.21.15.19:0/4009405727 wait complete. 2024-03-18T15:17:33.388 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-18T15:15:10.589737+0000', 'modified': '2024-03-18T15:15:46.979999+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24560, 'mds_1': 24529, 'mds_2': 24511, 'mds_3': 24506, 'mds_4': 24499}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14475': {'gid': 14475, 'name': 'a', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.19:6841/3176583319', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.19:6840', 'nonce': 3176583319}, {'type': 'v1', 'addr': '172.21.15.19:6841', 'nonce': 3176583319}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24433': {'gid': 24433, 'name': 'l', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.104:6833/482725809', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.104:6832', 'nonce': 482725809}, {'type': 'v1', 'addr': '172.21.15.104:6833', 'nonce': 482725809}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24434': {'gid': 24434, 'name': 'k', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.33:6833/2272279159', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.33:6832', 'nonce': 2272279159}, {'type': 'v1', 'addr': '172.21.15.33:6833', 'nonce': 2272279159}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24469': {'gid': 24469, 'name': 'd', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.19:6837/1778019859', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.19:6836', 'nonce': 1778019859}, {'type': 'v1', 'addr': '172.21.15.19:6837', 'nonce': 1778019859}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24482': {'gid': 24482, 'name': 'e', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.33:6835/4051613151', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.33:6834', 'nonce': 4051613151}, {'type': 'v1', 'addr': '172.21.15.33:6835', 'nonce': 4051613151}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24499': {'gid': 24499, 'name': 'f', 'rank': 4, 'incarnation': 25, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.104:6835/3788595957', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.104:6834', 'nonce': 3788595957}, {'type': 'v1', 'addr': '172.21.15.104:6835', 'nonce': 3788595957}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24506': {'gid': 24506, 'name': 'h', 'rank': 3, 'incarnation': 23, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.33:6837/160187378', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.33:6836', 'nonce': 160187378}, {'type': 'v1', 'addr': '172.21.15.33:6837', 'nonce': 160187378}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24511': {'gid': 24511, 'name': 'c', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.104:6837/260984674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.104:6836', 'nonce': 260984674}, {'type': 'v1', 'addr': '172.21.15.104:6837', 'nonce': 260984674}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24529': {'gid': 24529, 'name': 'i', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.104:6839/805341436', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.104:6838', 'nonce': 805341436}, {'type': 'v1', 'addr': '172.21.15.104:6839', 'nonce': 805341436}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24560': {'gid': 24560, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.33:6839/1903591535', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.33:6838', 'nonce': 1903591535}, {'type': 'v1', 'addr': '172.21.15.33:6839', 'nonce': 1903591535}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24560, 'qdb_cluster': [24506, 24499, 24529, 24511, 24560]} 2024-03-18T15:17:33.389 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-03-18T15:17:33.389 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.f status 2024-03-18T15:17:33.615 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.614+0000 7f5c17200640 1 -- 172.21.15.19:0/412645162 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5c180fff30 msgr2=0x7f5c18100350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:33.615 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.614+0000 7f5c17200640 1 --2- 172.21.15.19:0/412645162 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5c180fff30 0x7f5c18100350 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f5c08009ca0 tx=0x7f5c0802c0f0 comp rx=0 tx=0).stop 2024-03-18T15:17:33.615 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.614+0000 7f5c17200640 1 -- 172.21.15.19:0/412645162 shutdown_connections 2024-03-18T15:17:33.615 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.614+0000 7f5c17200640 1 --2- 172.21.15.19:0/412645162 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5c18101230 0x7f5c181016b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:33.615 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.614+0000 7f5c17200640 1 --2- 172.21.15.19:0/412645162 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5c18100890 0x7f5c18100cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:33.615 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.615+0000 7f5c17200640 1 --2- 172.21.15.19:0/412645162 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5c180fff30 0x7f5c18100350 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:33.615 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.615+0000 7f5c17200640 1 -- 172.21.15.19:0/412645162 >> 172.21.15.19:0/412645162 conn(0x7f5c180fbb50 msgr2=0x7f5c180fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:17:33.615 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.615+0000 7f5c17200640 1 -- 172.21.15.19:0/412645162 shutdown_connections 2024-03-18T15:17:33.615 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.615+0000 7f5c17200640 1 -- 172.21.15.19:0/412645162 wait complete. 2024-03-18T15:17:33.618 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.616+0000 7f5c17200640 1 Processor -- start 2024-03-18T15:17:33.618 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.616+0000 7f5c17200640 1 -- start start 2024-03-18T15:17:33.618 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.616+0000 7f5c17200640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5c180fff30 0x7f5c18199ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:33.618 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.616+0000 7f5c17200640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5c18100890 0x7f5c1819a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:33.618 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.616+0000 7f5c17200640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5c18101230 0x7f5c181a1150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:33.618 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.616+0000 7f5c17200640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f5c1810f520 con 0x7f5c18100890 2024-03-18T15:17:33.618 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.616+0000 7f5c17200640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5c1810f3a0 con 0x7f5c18101230 2024-03-18T15:17:33.618 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.616+0000 7f5c17200640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f5c1810f6a0 con 0x7f5c180fff30 2024-03-18T15:17:33.618 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.617+0000 7f5c15e00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5c180fff30 0x7f5c18199ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:33.618 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.617+0000 7f5c15e00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5c180fff30 0x7f5c18199ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.19:58204/0 (socket says 172.21.15.19:58204) 2024-03-18T15:17:33.619 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.617+0000 7f5c15e00640 1 -- 172.21.15.19:0/62660267 learned_addr learned my addr 172.21.15.19:0/62660267 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:17:33.619 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.617+0000 7f5c0d400640 1 --2- 172.21.15.19:0/62660267 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5c18100890 0x7f5c1819a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:33.619 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.617+0000 7f5c16800640 1 --2- 172.21.15.19:0/62660267 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5c18101230 0x7f5c181a1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:33.619 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.617+0000 7f5c15e00640 1 -- 172.21.15.19:0/62660267 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5c18101230 msgr2=0x7f5c181a1150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:33.619 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.617+0000 7f5c15e00640 1 --2- 172.21.15.19:0/62660267 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5c18101230 0x7f5c181a1150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:33.619 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.617+0000 7f5c15e00640 1 -- 172.21.15.19:0/62660267 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5c18100890 msgr2=0x7f5c1819a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:33.619 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.617+0000 7f5c15e00640 1 --2- 172.21.15.19:0/62660267 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5c18100890 0x7f5c1819a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:33.619 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.617+0000 7f5c15e00640 1 -- 172.21.15.19:0/62660267 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c181a17c0 con 0x7f5c180fff30 2024-03-18T15:17:33.619 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.618+0000 7f5c15e00640 1 --2- 172.21.15.19:0/62660267 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5c180fff30 0x7f5c18199ba0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f5c080325f0 tx=0x7f5c08032ec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:17:33.619 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.618+0000 7f5c0f400640 1 -- 172.21.15.19:0/62660267 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c08041020 con 0x7f5c180fff30 2024-03-18T15:17:33.619 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.618+0000 7f5c0f400640 1 -- 172.21.15.19:0/62660267 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5c08034cd0 con 0x7f5c180fff30 2024-03-18T15:17:33.619 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.618+0000 7f5c0f400640 1 -- 172.21.15.19:0/62660267 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c08039040 con 0x7f5c180fff30 2024-03-18T15:17:33.619 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.618+0000 7f5c17200640 1 -- 172.21.15.19:0/62660267 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c181a1a50 con 0x7f5c180fff30 2024-03-18T15:17:33.619 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.618+0000 7f5c17200640 1 -- 172.21.15.19:0/62660267 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c181a1ec0 con 0x7f5c180fff30 2024-03-18T15:17:33.621 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.621+0000 7f5c1f630740 1 Processor -- start 2024-03-18T15:17:33.621 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.621+0000 7f5c0f400640 1 -- 172.21.15.19:0/62660267 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f5c08044020 con 0x7f5c180fff30 2024-03-18T15:17:33.621 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.621+0000 7f5c1f630740 1 -- start start 2024-03-18T15:17:33.622 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.622+0000 7f5c1f630740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f3d3d0b340 0x55f3d3d0b760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:33.622 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.622+0000 7f5c1f630740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f3d3d0bd30 0x55f3d3d10830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:33.623 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.622+0000 7f5c1f630740 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55f3d3d10ea0 0x55f3d3d13330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:33.623 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.622+0000 7f5c1f630740 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x55f3d3c6fc50 con 0x55f3d3d10ea0 2024-03-18T15:17:33.623 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.622+0000 7f5c1f630740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55f3d3c189d0 con 0x55f3d3d0bd30 2024-03-18T15:17:33.623 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.622+0000 7f5c1f630740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55f3d3c6bcf0 con 0x55f3d3d0b340 2024-03-18T15:17:33.623 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.622+0000 7f5c15e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f3d3d0bd30 0x55f3d3d10830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:33.623 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.622+0000 7f5c16800640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f3d3d0b340 0x55f3d3d0b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:33.623 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.622+0000 7f5c16800640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f3d3d0b340 0x55f3d3d0b760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.19:58218/0 (socket says 172.21.15.19:58218) 2024-03-18T15:17:33.623 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.622+0000 7f5c16800640 1 -- 172.21.15.19:0/763581751 learned_addr learned my addr 172.21.15.19:0/763581751 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:17:33.623 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.622+0000 7f5c0d400640 1 --2- 172.21.15.19:0/763581751 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55f3d3d10ea0 0x55f3d3d13330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:33.623 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.622+0000 7f5c15e00640 1 -- 172.21.15.19:0/763581751 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f3d3d0b340 msgr2=0x55f3d3d0b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:17:33.623 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.622+0000 7f5c15e00640 1 --2- 172.21.15.19:0/763581751 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f3d3d0b340 0x55f3d3d0b760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:33.623 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.622+0000 7f5c15e00640 1 -- 172.21.15.19:0/763581751 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55f3d3d10ea0 msgr2=0x55f3d3d13330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:17:33.623 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.622+0000 7f5c15e00640 1 --2- 172.21.15.19:0/763581751 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55f3d3d10ea0 0x55f3d3d13330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:33.623 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.622+0000 7f5c15e00640 1 -- 172.21.15.19:0/763581751 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f3d3c32f10 con 0x55f3d3d0bd30 2024-03-18T15:17:33.624 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.622+0000 7f5c16800640 1 --2- 172.21.15.19:0/763581751 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f3d3d0b340 0x55f3d3d0b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:17:33.624 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.623+0000 7f5c15e00640 1 --2- 172.21.15.19:0/763581751 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f3d3d0bd30 0x55f3d3d10830 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f5c080ee2a0 tx=0x7f5c080f4c60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4312e816103ba3d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:17:33.628 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.623+0000 7f5c07000640 1 -- 172.21.15.19:0/763581751 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c08100070 con 0x55f3d3d0bd30 2024-03-18T15:17:33.628 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.623+0000 7f5c0f400640 1 --2- 172.21.15.19:0/62660267 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f5bfc0a2880 0x7f5bfc0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:33.628 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.623+0000 7f5c07000640 1 -- 172.21.15.19:0/763581751 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5c080f9c70 con 0x55f3d3d0bd30 2024-03-18T15:17:33.628 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.623+0000 7f5c07000640 1 -- 172.21.15.19:0/763581751 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c080b94d0 con 0x55f3d3d0bd30 2024-03-18T15:17:33.628 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.623+0000 7f5c0f400640 1 -- 172.21.15.19:0/62660267 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f5c080f0c30 con 0x7f5c180fff30 2024-03-18T15:17:33.628 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.624+0000 7f5c16800640 1 --2- 172.21.15.19:0/62660267 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f5bfc0a2880 0x7f5bfc0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:33.628 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.624+0000 7f5c1f630740 1 -- 172.21.15.19:0/763581751 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f3d3d0bd30 msgr2=0x55f3d3d10830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:33.628 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.624+0000 7f5c1f630740 1 --2- 172.21.15.19:0/763581751 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f3d3d0bd30 0x55f3d3d10830 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f5c080ee2a0 tx=0x7f5c080f4c60 comp rx=0 tx=0).stop 2024-03-18T15:17:33.628 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.624+0000 7f5c1f630740 1 -- 172.21.15.19:0/763581751 shutdown_connections 2024-03-18T15:17:33.628 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.624+0000 7f5c1f630740 1 --2- 172.21.15.19:0/763581751 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55f3d3d10ea0 0x55f3d3d13330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:33.629 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.624+0000 7f5c1f630740 1 --2- 172.21.15.19:0/763581751 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f3d3d0bd30 0x55f3d3d10830 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:33.629 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.624+0000 7f5c1f630740 1 --2- 172.21.15.19:0/763581751 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f3d3d0b340 0x55f3d3d0b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:33.629 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.624+0000 7f5c1f630740 1 -- 172.21.15.19:0/763581751 >> 172.21.15.19:0/763581751 conn(0x55f3d3d068b0 msgr2=0x55f3d3d06cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:17:33.629 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.624+0000 7f5c1f630740 1 -- 172.21.15.19:0/763581751 shutdown_connections 2024-03-18T15:17:33.629 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.624+0000 7f5c1f630740 1 -- 172.21.15.19:0/763581751 wait complete. 2024-03-18T15:17:33.629 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.624+0000 7f5c16800640 1 --2- 172.21.15.19:0/62660267 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f5bfc0a2880 0x7f5bfc0a4d40 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x55f3d3d0c590 tx=0x7f5c10005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:17:33.629 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.625+0000 7f5c1f630740 1 -- start start 2024-03-18T15:17:33.629 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.625+0000 7f5c1f630740 1 Processor -- start 2024-03-18T15:17:33.629 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.626+0000 7f5c1f630740 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55f3d3d0b340 0x55f3d3d15200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:33.629 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.626+0000 7f5c1f630740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f3d3d0bd30 0x55f3d3d15740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:33.629 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.626+0000 7f5c1f630740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f3d3d10ea0 0x55f3d3d1c780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:33.629 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.626+0000 7f5c1f630740 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x55f3d3d13c10 con 0x55f3d3d0b340 2024-03-18T15:17:33.629 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.626+0000 7f5c1f630740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55f3d3c2be20 con 0x55f3d3d10ea0 2024-03-18T15:17:33.629 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.626+0000 7f5c1f630740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55f3d3d13d90 con 0x55f3d3d0bd30 2024-03-18T15:17:33.630 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.626+0000 7f5c1f630740 10 client.0 _ll_register_callbacks cb 0x55f3d3c2b7f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T15:17:33.630 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.626+0000 7f5c16800640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f3d3d10ea0 0x55f3d3d1c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:33.630 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.626+0000 7f5c0d400640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f3d3d0bd30 0x55f3d3d15740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:33.630 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.626+0000 7f5c0d400640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f3d3d0bd30 0x55f3d3d15740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.19:58232/0 (socket says 172.21.15.19:58232) 2024-03-18T15:17:33.630 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.626+0000 7f5c15e00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55f3d3d0b340 0x55f3d3d15200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:33.630 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.626+0000 7f5c16800640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f3d3d10ea0 0x55f3d3d1c780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:39268/0 (socket says 172.21.15.19:39268) 2024-03-18T15:17:33.630 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.626+0000 7f5c0d400640 1 -- 172.21.15.19:0/3023932126 learned_addr learned my addr 172.21.15.19:0/3023932126 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:17:33.630 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.626+0000 7f5c16800640 1 -- 172.21.15.19:0/3023932126 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f3d3d0bd30 msgr2=0x55f3d3d15740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:17:33.630 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.626+0000 7f5c16800640 1 --2- 172.21.15.19:0/3023932126 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f3d3d0bd30 0x55f3d3d15740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:33.630 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.626+0000 7f5c16800640 1 -- 172.21.15.19:0/3023932126 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55f3d3d0b340 msgr2=0x55f3d3d15200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:17:33.630 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.626+0000 7f5c16800640 1 --2- 172.21.15.19:0/3023932126 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55f3d3d0b340 0x55f3d3d15200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:33.630 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.626+0000 7f5c16800640 1 -- 172.21.15.19:0/3023932126 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f3d39842d0 con 0x55f3d3d10ea0 2024-03-18T15:17:33.630 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.626+0000 7f5c0d400640 1 --2- 172.21.15.19:0/3023932126 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f3d3d0bd30 0x55f3d3d15740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:17:33.631 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.627+0000 7f5c16800640 1 --2- 172.21.15.19:0/3023932126 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f3d3d10ea0 0x55f3d3d1c780 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f5c1001e9d0 tx=0x7f5c10004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=96b9931d87ef052d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:17:33.631 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.627+0000 7f5bfb400640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-18T15:17:33.631 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.627+0000 7f5bfb400640 1 -- 172.21.15.19:0/3023932126 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c10005020 con 0x55f3d3d10ea0 2024-03-18T15:17:33.631 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.627+0000 7f5bfb400640 1 -- 172.21.15.19:0/3023932126 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f5c10018030 con 0x55f3d3d10ea0 2024-03-18T15:17:33.631 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.627+0000 7f5c1f630740 1 -- 172.21.15.19:0/3023932126 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5c080f9c70 con 0x55f3d3d10ea0 2024-03-18T15:17:33.631 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.627+0000 7f5bfb400640 1 -- 172.21.15.19:0/3023932126 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f5c10034050 con 0x55f3d3d10ea0 2024-03-18T15:17:33.631 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.627+0000 7f5c1f630740 10 client.25136 fetch_fsmap learned FSMap version 26 2024-03-18T15:17:33.631 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.627+0000 7f5c1f630740 1 -- 172.21.15.19:0/3023932126 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55f3d3a4e6b0 con 0x55f3d3d10ea0 2024-03-18T15:17:33.631 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.627+0000 7f5bfb400640 1 -- 172.21.15.19:0/3023932126 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5c080f9c70 con 0x55f3d3d10ea0 2024-03-18T15:17:33.631 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.628+0000 7f5bfb400640 1 -- 172.21.15.19:0/3023932126 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 26) ==== 8362+0+0 (secure 0 0 0) 0x7f5c10029420 con 0x55f3d3d10ea0 2024-03-18T15:17:33.631 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.628+0000 7f5c1f630740 10 client.25136 fetch_fsmap finished waiting for FSMap version 26 2024-03-18T15:17:33.631 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.628+0000 7f5c1f630740 10 client.25136 resolve_mds: resolved name 'f' to daemon mds.f 2024-03-18T15:17:33.631 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.628+0000 7f5c1f630740 20 client.25136 populate_metadata read hostname 'smithi019' 2024-03-18T15:17:33.631 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.628+0000 7f5c1f630740 1 --2- 172.21.15.19:0/3023932126 >> [v2:172.21.15.104:6834/3788595957,v1:172.21.15.104:6835/3788595957] conn(0x55f3d3d22010 0x55f3d3d23480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:33.631 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.628+0000 7f5c1f630740 4 client.25136 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T15:17:33.632 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.628+0000 7f5c1f630740 1 -- 172.21.15.19:0/3023932126 --> [v2:172.21.15.104:6834/3788595957,v1:172.21.15.104:6835/3788595957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f3d3ccd2b0 con 0x55f3d3d22010 2024-03-18T15:17:33.632 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.628+0000 7f5c15e00640 1 --2- 172.21.15.19:0/3023932126 >> [v2:172.21.15.104:6834/3788595957,v1:172.21.15.104:6835/3788595957] conn(0x55f3d3d22010 0x55f3d3d23480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:33.633 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.629+0000 7f5c15e00640 1 --2- 172.21.15.19:0/3023932126 >> [v2:172.21.15.104:6834/3788595957,v1:172.21.15.104:6835/3788595957] conn(0x55f3d3d22010 0x55f3d3d23480 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=69fa97cd5e1fbedb server_cookie=f02a0fd9f248eb80 in_seq=0 out_seq=0 2024-03-18T15:17:33.633 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.629+0000 7f5bfb400640 10 client.25136 ms_handle_connect on v2:172.21.15.104:6834/3788595957 2024-03-18T15:17:33.633 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.629+0000 7f5bfb400640 1 -- 172.21.15.19:0/3023932126 <== mds.4 v2:172.21.15.104:6834/3788595957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f3d3ccd2b0 con 0x55f3d3d22010 2024-03-18T15:17:33.634 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.629+0000 7f5bfb400640 10 client.25136 handle_command_reply: tid=0 2024-03-18T15:17:33.634 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.629+0000 7f5c1f630740 1 client.25136 shutdown 2024-03-18T15:17:33.634 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.630+0000 7f5c1f630740 1 -- 172.21.15.19:0/3023932126 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f3d3d10ea0 msgr2=0x55f3d3d1c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:33.634 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.630+0000 7f5c1f630740 1 --2- 172.21.15.19:0/3023932126 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f3d3d10ea0 0x55f3d3d1c780 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f5c1001e9d0 tx=0x7f5c10004730 comp rx=0 tx=0).stop 2024-03-18T15:17:33.634 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.630+0000 7f5c1f630740 1 -- 172.21.15.19:0/3023932126 shutdown_connections 2024-03-18T15:17:33.634 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.630+0000 7f5c1f630740 1 --2- 172.21.15.19:0/3023932126 >> [v2:172.21.15.104:6834/3788595957,v1:172.21.15.104:6835/3788595957] conn(0x55f3d3d22010 0x55f3d3d23480 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:33.634 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.630+0000 7f5c1f630740 1 --2- 172.21.15.19:0/3023932126 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f3d3d10ea0 0x55f3d3d1c780 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:33.634 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.630+0000 7f5c1f630740 1 --2- 172.21.15.19:0/3023932126 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f3d3d0bd30 0x55f3d3d15740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:33.634 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.630+0000 7f5bfb400640 0 client.25136 ms_handle_reset on v2:172.21.15.104:6834/3788595957 2024-03-18T15:17:33.634 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.630+0000 7f5c1f630740 1 --2- 172.21.15.19:0/3023932126 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55f3d3d0b340 0x55f3d3d15200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:33.634 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.630+0000 7f5c1f630740 1 -- 172.21.15.19:0/3023932126 >> 172.21.15.19:0/3023932126 conn(0x55f3d3d068b0 msgr2=0x55f3d3d05a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:17:33.634 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.630+0000 7f5c1f630740 1 -- 172.21.15.19:0/3023932126 shutdown_connections 2024-03-18T15:17:33.634 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.630+0000 7f5c1f630740 1 -- 172.21.15.19:0/3023932126 wait complete. 2024-03-18T15:17:33.634 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.630+0000 7f5c1f630740 20 client.25136 trim_cache size 0 max 16384 2024-03-18T15:17:33.651 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.650+0000 7f5c1f630740 1 Processor -- start 2024-03-18T15:17:33.651 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.650+0000 7f5c1f630740 1 -- start start 2024-03-18T15:17:33.651 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.650+0000 7f5c1f630740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f3d3d0b340 0x55f3d3d01900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:33.651 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.650+0000 7f5c1f630740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f3d3d0bd30 0x55f3d3d1c740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:33.651 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.650+0000 7f5c1f630740 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55f3d3d10ea0 0x55f3d3d15200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:33.651 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.650+0000 7f5c1f630740 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x55f3d3d217c0 con 0x55f3d3d10ea0 2024-03-18T15:17:33.651 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.650+0000 7f5c1f630740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55f3d3d21e90 con 0x55f3d3d0b340 2024-03-18T15:17:33.651 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.650+0000 7f5c1f630740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55f3d3d22560 con 0x55f3d3d0bd30 2024-03-18T15:17:33.651 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.650+0000 7f5c16800640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55f3d3d10ea0 0x55f3d3d15200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:33.651 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.650+0000 7f5c0d400640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f3d3d0bd30 0x55f3d3d1c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:33.651 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.650+0000 7f5c15e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f3d3d0b340 0x55f3d3d01900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:33.651 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.650+0000 7f5c16800640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55f3d3d10ea0 0x55f3d3d15200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:50442/0 (socket says 172.21.15.19:50442) 2024-03-18T15:17:33.652 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.650+0000 7f5c16800640 1 -- 172.21.15.19:0/3638533492 learned_addr learned my addr 172.21.15.19:0/3638533492 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:17:33.652 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.650+0000 7f5c15e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f3d3d0b340 0x55f3d3d01900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:39282/0 (socket says 172.21.15.19:39282) 2024-03-18T15:17:33.652 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.650+0000 7f5c0d400640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f3d3d0bd30 0x55f3d3d1c740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.19:58238/0 (socket says 172.21.15.19:58238) 2024-03-18T15:17:33.652 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.651+0000 7f5c16800640 1 -- 172.21.15.19:0/3638533492 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f3d3d0bd30 msgr2=0x55f3d3d1c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:17:33.652 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.651+0000 7f5c16800640 1 --2- 172.21.15.19:0/3638533492 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f3d3d0bd30 0x55f3d3d1c740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:33.652 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.651+0000 7f5c16800640 1 -- 172.21.15.19:0/3638533492 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f3d3d0b340 msgr2=0x55f3d3d01900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:17:33.652 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.651+0000 7f5c16800640 1 --2- 172.21.15.19:0/3638533492 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f3d3d0b340 0x55f3d3d01900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:33.652 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.651+0000 7f5c16800640 1 -- 172.21.15.19:0/3638533492 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f3d3d142f0 con 0x55f3d3d10ea0 2024-03-18T15:17:33.652 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.651+0000 7f5c16800640 1 --2- 172.21.15.19:0/3638533492 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55f3d3d10ea0 0x55f3d3d15200 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f5c10012cb0 tx=0x7f5c100092a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f80bb64231aa874b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:17:33.656 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.651+0000 7f5c06600640 1 -- 172.21.15.19:0/3638533492 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c1003d070 con 0x55f3d3d10ea0 2024-03-18T15:17:33.656 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.651+0000 7f5c06600640 1 -- 172.21.15.19:0/3638533492 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5c10029420 con 0x55f3d3d10ea0 2024-03-18T15:17:33.656 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.651+0000 7f5c06600640 1 -- 172.21.15.19:0/3638533492 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c100047a0 con 0x55f3d3d10ea0 2024-03-18T15:17:33.657 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.652+0000 7f5c1f630740 1 -- 172.21.15.19:0/3638533492 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55f3d3d10ea0 msgr2=0x55f3d3d15200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:33.657 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.652+0000 7f5c1f630740 1 --2- 172.21.15.19:0/3638533492 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55f3d3d10ea0 0x55f3d3d15200 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f5c10012cb0 tx=0x7f5c100092a0 comp rx=0 tx=0).stop 2024-03-18T15:17:33.657 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.652+0000 7f5c1f630740 1 -- 172.21.15.19:0/3638533492 shutdown_connections 2024-03-18T15:17:33.657 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.652+0000 7f5c1f630740 1 --2- 172.21.15.19:0/3638533492 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55f3d3d10ea0 0x55f3d3d15200 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:33.657 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.652+0000 7f5c1f630740 1 --2- 172.21.15.19:0/3638533492 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f3d3d0bd30 0x55f3d3d1c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:33.657 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.652+0000 7f5c1f630740 1 --2- 172.21.15.19:0/3638533492 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f3d3d0b340 0x55f3d3d01900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:33.657 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.652+0000 7f5c1f630740 1 -- 172.21.15.19:0/3638533492 >> 172.21.15.19:0/3638533492 conn(0x55f3d3d068b0 msgr2=0x55f3d3d20df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:17:33.657 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.652+0000 7f5c1f630740 1 -- 172.21.15.19:0/3638533492 shutdown_connections 2024-03-18T15:17:33.657 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.652+0000 7f5c1f630740 1 -- 172.21.15.19:0/3638533492 wait complete. 2024-03-18T15:17:33.657 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.653+0000 7f5c1f630740 1 -- start start 2024-03-18T15:17:33.657 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.653+0000 7f5c1f630740 1 Processor -- start 2024-03-18T15:17:33.657 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.653+0000 7f5c1f630740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f3d3d0b340 0x55f3d3d1a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:33.657 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.653+0000 7f5c1f630740 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55f3d3d0bd30 0x55f3d3d83d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:33.658 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.653+0000 7f5c1f630740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f3d3d10ea0 0x55f3d3d862d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:33.658 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.653+0000 7f5c1f630740 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x55f3d3d23300 con 0x55f3d3d0bd30 2024-03-18T15:17:33.658 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.653+0000 7f5c1f630740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55f3d3d22c30 con 0x55f3d3d0b340 2024-03-18T15:17:33.658 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.653+0000 7f5c1f630740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55f3d3d239d0 con 0x55f3d3d10ea0 2024-03-18T15:17:33.658 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.653+0000 7f5c1f630740 10 client.0 _ll_register_callbacks cb 0x7f5c1001e890 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T15:17:33.658 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.653+0000 7f5c15e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f3d3d0b340 0x55f3d3d1a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:33.658 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.653+0000 7f5c15e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f3d3d0b340 0x55f3d3d1a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:39290/0 (socket says 172.21.15.19:39290) 2024-03-18T15:17:33.658 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.653+0000 7f5c15e00640 1 -- 172.21.15.19:0/797331192 learned_addr learned my addr 172.21.15.19:0/797331192 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:17:33.658 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.653+0000 7f5c16800640 1 --2- 172.21.15.19:0/797331192 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f3d3d10ea0 0x55f3d3d862d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:33.658 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.653+0000 7f5c0d400640 1 --2- 172.21.15.19:0/797331192 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55f3d3d0bd30 0x55f3d3d83d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:33.658 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.654+0000 7f5c15e00640 1 -- 172.21.15.19:0/797331192 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f3d3d10ea0 msgr2=0x55f3d3d862d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:17:33.658 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.654+0000 7f5c15e00640 1 --2- 172.21.15.19:0/797331192 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f3d3d10ea0 0x55f3d3d862d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:33.658 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.654+0000 7f5c15e00640 1 -- 172.21.15.19:0/797331192 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55f3d3d0bd30 msgr2=0x55f3d3d83d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:17:33.658 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.654+0000 7f5c15e00640 1 --2- 172.21.15.19:0/797331192 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55f3d3d0bd30 0x55f3d3d83d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:33.659 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.654+0000 7f5c15e00640 1 -- 172.21.15.19:0/797331192 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f3d3d86b00 con 0x55f3d3d0b340 2024-03-18T15:17:33.659 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.654+0000 7f5c16800640 1 --2- 172.21.15.19:0/797331192 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f3d3d10ea0 0x55f3d3d862d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:17:33.659 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.654+0000 7f5c15e00640 1 --2- 172.21.15.19:0/797331192 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f3d3d0b340 0x55f3d3d1a730 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f5c08000f80 tx=0x7f5c0802c0f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bb724701a9615a0b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:17:33.659 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.654+0000 7f5bfb400640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-18T15:17:33.659 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.654+0000 7f5bfb400640 1 -- 172.21.15.19:0/797331192 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c080bc9b0 con 0x55f3d3d0b340 2024-03-18T15:17:33.659 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.654+0000 7f5bfb400640 1 -- 172.21.15.19:0/797331192 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f5c080bb530 con 0x55f3d3d0b340 2024-03-18T15:17:33.659 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.654+0000 7f5c1f630740 1 -- 172.21.15.19:0/797331192 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5c10029420 con 0x55f3d3d0b340 2024-03-18T15:17:33.659 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.655+0000 7f5bfb400640 1 -- 172.21.15.19:0/797331192 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f5c080bcb90 con 0x55f3d3d0b340 2024-03-18T15:17:33.659 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.655+0000 7f5c1f630740 10 client.25142 fetch_fsmap learned FSMap version 26 2024-03-18T15:17:33.659 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.655+0000 7f5c1f630740 1 -- 172.21.15.19:0/797331192 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55f3d3d87e00 con 0x55f3d3d0b340 2024-03-18T15:17:33.659 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.655+0000 7f5bfb400640 1 -- 172.21.15.19:0/797331192 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5c10029420 con 0x55f3d3d0b340 2024-03-18T15:17:33.659 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.655+0000 7f5bfb400640 1 -- 172.21.15.19:0/797331192 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 26) ==== 8362+0+0 (secure 0 0 0) 0x7f5c0803c030 con 0x55f3d3d0b340 2024-03-18T15:17:33.660 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.655+0000 7f5c1f630740 10 client.25142 fetch_fsmap finished waiting for FSMap version 26 2024-03-18T15:17:33.660 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.655+0000 7f5c1f630740 10 client.25142 resolve_mds: resolved name 'f' to daemon mds.f 2024-03-18T15:17:33.660 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.655+0000 7f5c1f630740 20 client.25142 populate_metadata read hostname 'smithi019' 2024-03-18T15:17:33.660 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.655+0000 7f5c1f630740 1 --2- 172.21.15.19:0/797331192 >> [v2:172.21.15.104:6834/3788595957,v1:172.21.15.104:6835/3788595957] conn(0x55f3d3d8c5f0 0x55f3d3d8b300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:33.660 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.655+0000 7f5c1f630740 4 client.25142 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-18T15:17:33.660 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.655+0000 7f5c1f630740 1 -- 172.21.15.19:0/797331192 --> [v2:172.21.15.104:6834/3788595957,v1:172.21.15.104:6835/3788595957] -- command(tid 0: {"prefix": "status"}) -- 0x55f3d3c7d190 con 0x55f3d3d8c5f0 2024-03-18T15:17:33.660 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.655+0000 7f5c0d400640 1 --2- 172.21.15.19:0/797331192 >> [v2:172.21.15.104:6834/3788595957,v1:172.21.15.104:6835/3788595957] conn(0x55f3d3d8c5f0 0x55f3d3d8b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:33.660 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.656+0000 7f5c0d400640 1 --2- 172.21.15.19:0/797331192 >> [v2:172.21.15.104:6834/3788595957,v1:172.21.15.104:6835/3788595957] conn(0x55f3d3d8c5f0 0x55f3d3d8b300 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=75f0082d6b9366df server_cookie=ee6ff4d1b6a7bfc5 in_seq=0 out_seq=0 2024-03-18T15:17:33.660 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.656+0000 7f5bfb400640 10 client.25142 ms_handle_connect on v2:172.21.15.104:6834/3788595957 2024-03-18T15:17:33.660 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.656+0000 7f5bfb400640 1 -- 172.21.15.19:0/797331192 <== mds.4 v2:172.21.15.104:6834/3788595957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55f3d3c7d190 con 0x55f3d3d8c5f0 2024-03-18T15:17:33.660 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.656+0000 7f5bfb400640 10 client.25142 handle_command_reply: tid=0 2024-03-18T15:17:33.660 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.656+0000 7f5c1f630740 1 client.25142 shutdown 2024-03-18T15:17:33.660 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.656+0000 7f5c1f630740 1 -- 172.21.15.19:0/797331192 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f3d3d0b340 msgr2=0x55f3d3d1a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:33.660 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.656+0000 7f5c1f630740 1 --2- 172.21.15.19:0/797331192 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f3d3d0b340 0x55f3d3d1a730 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f5c08000f80 tx=0x7f5c0802c0f0 comp rx=0 tx=0).stop 2024-03-18T15:17:33.660 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.656+0000 7f5c1f630740 1 -- 172.21.15.19:0/797331192 shutdown_connections 2024-03-18T15:17:33.661 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.656+0000 7f5c1f630740 1 --2- 172.21.15.19:0/797331192 >> [v2:172.21.15.104:6834/3788595957,v1:172.21.15.104:6835/3788595957] conn(0x55f3d3d8c5f0 0x55f3d3d8b300 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:33.661 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.656+0000 7f5c1f630740 1 --2- 172.21.15.19:0/797331192 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f3d3d10ea0 0x55f3d3d862d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:33.661 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.656+0000 7f5c1f630740 1 --2- 172.21.15.19:0/797331192 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55f3d3d0bd30 0x55f3d3d83d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:33.661 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.656+0000 7f5c1f630740 1 --2- 172.21.15.19:0/797331192 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f3d3d0b340 0x55f3d3d1a730 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:33.661 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.656+0000 7f5c1f630740 1 -- 172.21.15.19:0/797331192 >> 172.21.15.19:0/797331192 conn(0x55f3d3d068b0 msgr2=0x55f3d3d01900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:17:33.661 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.656+0000 7f5bfb400640 0 client.25142 ms_handle_reset on v2:172.21.15.104:6834/3788595957 2024-03-18T15:17:33.661 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.656+0000 7f5c1f630740 1 -- 172.21.15.19:0/797331192 shutdown_connections 2024-03-18T15:17:33.661 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.657+0000 7f5c1f630740 1 -- 172.21.15.19:0/797331192 wait complete. 2024-03-18T15:17:33.661 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.657+0000 7f5c1f630740 20 client.25142 trim_cache size 0 max 16384 2024-03-18T15:17:33.661 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-03-18T15:17:33.662 INFO:teuthology.orchestra.run.smithi019.stdout: "cluster_fsid": "d4dc2ba6-e538-11ee-95c9-87774f69a715", 2024-03-18T15:17:33.662 INFO:teuthology.orchestra.run.smithi019.stdout: "whoami": 4, 2024-03-18T15:17:33.662 INFO:teuthology.orchestra.run.smithi019.stdout: "id": 24499, 2024-03-18T15:17:33.662 INFO:teuthology.orchestra.run.smithi019.stdout: "want_state": "up:active", 2024-03-18T15:17:33.662 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "up:active", 2024-03-18T15:17:33.662 INFO:teuthology.orchestra.run.smithi019.stdout: "fs_name": "cephfs", 2024-03-18T15:17:33.662 INFO:teuthology.orchestra.run.smithi019.stdout: "rank_uptime": 107.124263344, 2024-03-18T15:17:33.662 INFO:teuthology.orchestra.run.smithi019.stdout: "mdsmap_epoch": 26, 2024-03-18T15:17:33.662 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap_epoch": 93, 2024-03-18T15:17:33.662 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap_epoch_barrier": 89, 2024-03-18T15:17:33.662 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 169.30681183600001 2024-03-18T15:17:33.663 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-03-18T15:17:33.663 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.657+0000 7f5c17200640 1 -- 172.21.15.19:0/62660267 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f5bfc0a2880 msgr2=0x7f5bfc0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:33.663 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.657+0000 7f5c17200640 1 --2- 172.21.15.19:0/62660267 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f5bfc0a2880 0x7f5bfc0a4d40 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x55f3d3d0c590 tx=0x7f5c10005f90 comp rx=0 tx=0).stop 2024-03-18T15:17:33.663 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.657+0000 7f5c17200640 1 -- 172.21.15.19:0/62660267 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5c180fff30 msgr2=0x7f5c18199ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:33.663 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.657+0000 7f5c17200640 1 --2- 172.21.15.19:0/62660267 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5c180fff30 0x7f5c18199ba0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f5c080325f0 tx=0x7f5c08032ec0 comp rx=0 tx=0).stop 2024-03-18T15:17:33.663 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.657+0000 7f5c17200640 1 -- 172.21.15.19:0/62660267 shutdown_connections 2024-03-18T15:17:33.663 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.657+0000 7f5c17200640 1 --2- 172.21.15.19:0/62660267 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f5bfc0a2880 0x7f5bfc0a4d40 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:33.664 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.657+0000 7f5c17200640 1 --2- 172.21.15.19:0/62660267 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5c18101230 0x7f5c181a1150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:33.664 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.657+0000 7f5c17200640 1 --2- 172.21.15.19:0/62660267 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5c18100890 0x7f5c1819a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:33.664 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.657+0000 7f5c17200640 1 --2- 172.21.15.19:0/62660267 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5c180fff30 0x7f5c18199ba0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:33.664 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.657+0000 7f5c17200640 1 -- 172.21.15.19:0/62660267 >> 172.21.15.19:0/62660267 conn(0x7f5c180fbb50 msgr2=0x7f5c180fd290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:17:33.664 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.658+0000 7f5c17200640 1 -- 172.21.15.19:0/62660267 shutdown_connections 2024-03-18T15:17:33.664 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.658+0000 7f5c17200640 1 -- 172.21.15.19:0/62660267 wait complete. 2024-03-18T15:17:33.674 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:17:33 smithi019 ceph-mon[31522]: pgmap v392: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T15:17:33.674 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:17:33 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/1643381460' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:17:33.674 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:17:33 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/4009405727' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:17:33.678 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-03-18T15:17:33.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:17:33 smithi033 ceph-mon[36648]: pgmap v392: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T15:17:33.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:17:33 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/1643381460' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:17:33.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:17:33 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/4009405727' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:17:33.904 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.903+0000 7f0f62e00640 1 -- 172.21.15.19:0/3783216203 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f5c102180 msgr2=0x7f0f5c108a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:33.904 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.903+0000 7f0f62e00640 1 --2- 172.21.15.19:0/3783216203 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f5c102180 0x7f0f5c108a10 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f0f54009c90 tx=0x7f0f5402c0b0 comp rx=0 tx=0).stop 2024-03-18T15:17:33.904 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.903+0000 7f0f62e00640 1 -- 172.21.15.19:0/3783216203 shutdown_connections 2024-03-18T15:17:33.904 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.903+0000 7f0f62e00640 1 --2- 172.21.15.19:0/3783216203 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f5c102180 0x7f0f5c108a10 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:33.904 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.903+0000 7f0f62e00640 1 --2- 172.21.15.19:0/3783216203 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0f5c1017c0 0x7f0f5c101c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:33.904 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.903+0000 7f0f62e00640 1 --2- 172.21.15.19:0/3783216203 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0f5c1005d0 0x7f0f5c1009f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:33.904 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.903+0000 7f0f62e00640 1 -- 172.21.15.19:0/3783216203 >> 172.21.15.19:0/3783216203 conn(0x7f0f5c0fbb60 msgr2=0x7f0f5c0fdf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:17:33.904 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.903+0000 7f0f62e00640 1 -- 172.21.15.19:0/3783216203 shutdown_connections 2024-03-18T15:17:33.904 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.904+0000 7f0f62e00640 1 -- 172.21.15.19:0/3783216203 wait complete. 2024-03-18T15:17:33.906 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.905+0000 7f0f62e00640 1 Processor -- start 2024-03-18T15:17:33.906 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.905+0000 7f0f62e00640 1 -- start start 2024-03-18T15:17:33.906 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.905+0000 7f0f62e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0f5c1005d0 0x7f0f5c1979c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:33.906 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.905+0000 7f0f62e00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0f5c1017c0 0x7f0f5c197f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:33.906 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.905+0000 7f0f62e00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f5c102180 0x7f0f5c19ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:33.906 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.906+0000 7f0f62e00640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f0f5c10c250 con 0x7f0f5c1017c0 2024-03-18T15:17:33.906 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.906+0000 7f0f62e00640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f0f5c10c0d0 con 0x7f0f5c1005d0 2024-03-18T15:17:33.906 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.906+0000 7f0f62e00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f0f5c10c3d0 con 0x7f0f5c102180 2024-03-18T15:17:33.906 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.906+0000 7f0f61a00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0f5c1005d0 0x7f0f5c1979c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:33.906 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.906+0000 7f0f61a00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0f5c1005d0 0x7f0f5c1979c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:39296/0 (socket says 172.21.15.19:39296) 2024-03-18T15:17:33.906 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.906+0000 7f0f61a00640 1 -- 172.21.15.19:0/3580296332 learned_addr learned my addr 172.21.15.19:0/3580296332 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:17:33.907 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.906+0000 7f0f61000640 1 --2- 172.21.15.19:0/3580296332 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0f5c1017c0 0x7f0f5c197f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:33.907 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.906+0000 7f0f62400640 1 --2- 172.21.15.19:0/3580296332 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f5c102180 0x7f0f5c19ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:33.908 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.906+0000 7f0f61a00640 1 -- 172.21.15.19:0/3580296332 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f5c102180 msgr2=0x7f0f5c19ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:33.908 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.906+0000 7f0f61a00640 1 --2- 172.21.15.19:0/3580296332 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f5c102180 0x7f0f5c19ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:33.908 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.906+0000 7f0f61a00640 1 -- 172.21.15.19:0/3580296332 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0f5c1017c0 msgr2=0x7f0f5c197f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:33.908 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.906+0000 7f0f61a00640 1 --2- 172.21.15.19:0/3580296332 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0f5c1017c0 0x7f0f5c197f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:33.908 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.906+0000 7f0f61a00640 1 -- 172.21.15.19:0/3580296332 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f5c19f5e0 con 0x7f0f5c1005d0 2024-03-18T15:17:33.908 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.907+0000 7f0f61a00640 1 --2- 172.21.15.19:0/3580296332 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0f5c1005d0 0x7f0f5c1979c0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f0f5000e9c0 tx=0x7f0f5000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:17:33.908 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.907+0000 7f0f5a000640 1 -- 172.21.15.19:0/3580296332 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f5000cd80 con 0x7f0f5c1005d0 2024-03-18T15:17:33.909 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.907+0000 7f0f5a000640 1 -- 172.21.15.19:0/3580296332 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0f500035f0 con 0x7f0f5c1005d0 2024-03-18T15:17:33.909 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.907+0000 7f0f62e00640 1 -- 172.21.15.19:0/3580296332 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f5c19f8d0 con 0x7f0f5c1005d0 2024-03-18T15:17:33.909 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.907+0000 7f0f5a000640 1 -- 172.21.15.19:0/3580296332 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f50010690 con 0x7f0f5c1005d0 2024-03-18T15:17:33.909 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.907+0000 7f0f62e00640 1 -- 172.21.15.19:0/3580296332 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f5c19fe70 con 0x7f0f5c1005d0 2024-03-18T15:17:33.912 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.911+0000 7f0f67072740 1 Processor -- start 2024-03-18T15:17:33.912 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.912+0000 7f0f67072740 1 -- start start 2024-03-18T15:17:33.912 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.912+0000 7f0f5a000640 1 -- 172.21.15.19:0/3580296332 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f0f500037a0 con 0x7f0f5c1005d0 2024-03-18T15:17:33.913 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.912+0000 7f0f67072740 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x557ebab8b340 0x557ebab8b760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:33.913 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.912+0000 7f0f67072740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557ebab8bd30 0x557ebab90830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:33.913 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.912+0000 7f0f67072740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557ebab90ea0 0x557ebab93330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:33.913 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.912+0000 7f0f67072740 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x557ebaaefc50 con 0x557ebab8b340 2024-03-18T15:17:33.913 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.912+0000 7f0f67072740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x557ebaa989d0 con 0x557ebab8bd30 2024-03-18T15:17:33.913 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.912+0000 7f0f67072740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x557ebaaebcf0 con 0x557ebab90ea0 2024-03-18T15:17:33.913 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.912+0000 7f0f61a00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557ebab8bd30 0x557ebab90830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:33.913 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.912+0000 7f0f61000640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557ebab90ea0 0x557ebab93330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:33.913 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.912+0000 7f0f62400640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x557ebab8b340 0x557ebab8b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:33.913 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.912+0000 7f0f61a00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557ebab8bd30 0x557ebab90830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:39298/0 (socket says 172.21.15.19:39298) 2024-03-18T15:17:33.913 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.912+0000 7f0f61a00640 1 -- 172.21.15.19:0/3915266043 learned_addr learned my addr 172.21.15.19:0/3915266043 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:17:33.920 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.913+0000 7f0f61a00640 1 -- 172.21.15.19:0/3915266043 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557ebab90ea0 msgr2=0x557ebab93330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:17:33.921 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.913+0000 7f0f61a00640 1 --2- 172.21.15.19:0/3915266043 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557ebab90ea0 0x557ebab93330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:33.921 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.913+0000 7f0f61a00640 1 -- 172.21.15.19:0/3915266043 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x557ebab8b340 msgr2=0x557ebab8b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:17:33.921 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.913+0000 7f0f61a00640 1 --2- 172.21.15.19:0/3915266043 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x557ebab8b340 0x557ebab8b760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:33.921 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.913+0000 7f0f61a00640 1 -- 172.21.15.19:0/3915266043 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557ebaab2f10 con 0x557ebab8bd30 2024-03-18T15:17:33.921 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.913+0000 7f0f61000640 1 --2- 172.21.15.19:0/3915266043 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557ebab90ea0 0x557ebab93330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:17:33.921 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.913+0000 7f0f61a00640 1 --2- 172.21.15.19:0/3915266043 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557ebab8bd30 0x557ebab90830 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f0f500d71b0 tx=0x7f0f500d7670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9af1cf0586d81291 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:17:33.921 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.913+0000 7f0f4ea00640 1 -- 172.21.15.19:0/3915266043 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f500e2070 con 0x557ebab8bd30 2024-03-18T15:17:33.921 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.913+0000 7f0f4ea00640 1 -- 172.21.15.19:0/3915266043 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0f500cfa90 con 0x557ebab8bd30 2024-03-18T15:17:33.921 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.914+0000 7f0f4ea00640 1 -- 172.21.15.19:0/3915266043 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f500dd9e0 con 0x557ebab8bd30 2024-03-18T15:17:33.921 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.914+0000 7f0f5a000640 1 --2- 172.21.15.19:0/3580296332 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f0f440a2880 0x7f0f440a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:33.921 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.914+0000 7f0f5a000640 1 -- 172.21.15.19:0/3580296332 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f0f500cf1c0 con 0x7f0f5c1005d0 2024-03-18T15:17:33.921 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.914+0000 7f0f67072740 1 -- 172.21.15.19:0/3915266043 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557ebab8bd30 msgr2=0x557ebab90830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:33.922 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.914+0000 7f0f67072740 1 --2- 172.21.15.19:0/3915266043 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557ebab8bd30 0x557ebab90830 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f0f500d71b0 tx=0x7f0f500d7670 comp rx=0 tx=0).stop 2024-03-18T15:17:33.922 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.914+0000 7f0f62400640 1 --2- 172.21.15.19:0/3580296332 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f0f440a2880 0x7f0f440a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:33.922 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.914+0000 7f0f67072740 1 -- 172.21.15.19:0/3915266043 shutdown_connections 2024-03-18T15:17:33.922 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.914+0000 7f0f67072740 1 --2- 172.21.15.19:0/3915266043 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557ebab90ea0 0x557ebab93330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:33.922 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.914+0000 7f0f67072740 1 --2- 172.21.15.19:0/3915266043 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557ebab8bd30 0x557ebab90830 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:33.922 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.914+0000 7f0f67072740 1 --2- 172.21.15.19:0/3915266043 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x557ebab8b340 0x557ebab8b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:33.922 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.914+0000 7f0f67072740 1 -- 172.21.15.19:0/3915266043 >> 172.21.15.19:0/3915266043 conn(0x557ebab868b0 msgr2=0x557ebab86cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:17:33.922 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.914+0000 7f0f67072740 1 -- 172.21.15.19:0/3915266043 shutdown_connections 2024-03-18T15:17:33.922 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.915+0000 7f0f67072740 1 -- 172.21.15.19:0/3915266043 wait complete. 2024-03-18T15:17:33.922 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.915+0000 7f0f67072740 1 -- start start 2024-03-18T15:17:33.922 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.915+0000 7f0f62400640 1 --2- 172.21.15.19:0/3580296332 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f0f440a2880 0x7f0f440a4d40 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f0f540011e0 tx=0x7f0f5402e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:17:33.922 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.915+0000 7f0f67072740 1 Processor -- start 2024-03-18T15:17:33.922 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.916+0000 7f0f67072740 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x557ebab8b340 0x557ebab95470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:33.922 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.916+0000 7f0f67072740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557ebab8bd30 0x557ebab959b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:33.923 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.916+0000 7f0f67072740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557ebab90ea0 0x557ebab9c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:33.923 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.916+0000 7f0f67072740 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x557ebab93e50 con 0x557ebab8b340 2024-03-18T15:17:33.923 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.916+0000 7f0f67072740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x557ebaaabe20 con 0x557ebab8bd30 2024-03-18T15:17:33.923 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.916+0000 7f0f67072740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x557ebab93fd0 con 0x557ebab90ea0 2024-03-18T15:17:33.923 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.916+0000 7f0f67072740 10 client.0 _ll_register_callbacks cb 0x557ebaaab7f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T15:17:33.923 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.916+0000 7f0f61a00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x557ebab8b340 0x557ebab95470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:33.923 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.916+0000 7f0f61000640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557ebab8bd30 0x557ebab959b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:33.923 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.916+0000 7f0f61a00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x557ebab8b340 0x557ebab95470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:50496/0 (socket says 172.21.15.19:50496) 2024-03-18T15:17:33.923 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.916+0000 7f0f61a00640 1 -- 172.21.15.19:0/3888239624 learned_addr learned my addr 172.21.15.19:0/3888239624 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:17:33.923 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.916+0000 7f0f62400640 1 --2- 172.21.15.19:0/3888239624 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557ebab90ea0 0x557ebab9c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:33.923 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.916+0000 7f0f61000640 1 -- 172.21.15.19:0/3888239624 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557ebab90ea0 msgr2=0x557ebab9c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:17:33.923 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.916+0000 7f0f61000640 1 --2- 172.21.15.19:0/3888239624 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557ebab90ea0 0x557ebab9c9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:33.923 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.916+0000 7f0f61000640 1 -- 172.21.15.19:0/3888239624 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x557ebab8b340 msgr2=0x557ebab95470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:17:33.923 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.916+0000 7f0f61000640 1 --2- 172.21.15.19:0/3888239624 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x557ebab8b340 0x557ebab95470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:33.923 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.916+0000 7f0f61000640 1 -- 172.21.15.19:0/3888239624 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557eba8042d0 con 0x557ebab8bd30 2024-03-18T15:17:33.924 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.916+0000 7f0f61a00640 1 --2- 172.21.15.19:0/3888239624 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x557ebab8b340 0x557ebab95470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:17:33.924 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.917+0000 7f0f61000640 1 --2- 172.21.15.19:0/3888239624 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557ebab8bd30 0x557ebab959b0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f0f5c198ed0 tx=0x7f0f48012b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8b9fccb0a1f1d748 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:17:33.924 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.917+0000 7f0f43400640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-18T15:17:33.924 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.917+0000 7f0f43400640 1 -- 172.21.15.19:0/3888239624 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f48004050 con 0x557ebab8bd30 2024-03-18T15:17:33.924 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.917+0000 7f0f67072740 1 -- 172.21.15.19:0/3888239624 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0f500cfa90 con 0x557ebab8bd30 2024-03-18T15:17:33.924 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.917+0000 7f0f43400640 1 -- 172.21.15.19:0/3888239624 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f0f480041f0 con 0x557ebab8bd30 2024-03-18T15:17:33.924 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.917+0000 7f0f43400640 1 -- 172.21.15.19:0/3888239624 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f0f48025050 con 0x557ebab8bd30 2024-03-18T15:17:33.924 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.917+0000 7f0f67072740 10 client.25154 fetch_fsmap learned FSMap version 26 2024-03-18T15:17:33.924 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.917+0000 7f0f67072740 1 -- 172.21.15.19:0/3888239624 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x557eba8ce6b0 con 0x557ebab8bd30 2024-03-18T15:17:33.924 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.918+0000 7f0f43400640 1 -- 172.21.15.19:0/3888239624 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0f500cfa90 con 0x557ebab8bd30 2024-03-18T15:17:33.924 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.918+0000 7f0f43400640 1 -- 172.21.15.19:0/3888239624 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 26) ==== 8362+0+0 (secure 0 0 0) 0x7f0f48003a60 con 0x557ebab8bd30 2024-03-18T15:17:33.924 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.918+0000 7f0f67072740 10 client.25154 fetch_fsmap finished waiting for FSMap version 26 2024-03-18T15:17:33.924 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.918+0000 7f0f67072740 10 client.25154 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-18T15:17:33.924 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.918+0000 7f0f67072740 20 client.25154 populate_metadata read hostname 'smithi019' 2024-03-18T15:17:33.924 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.918+0000 7f0f67072740 1 --2- 172.21.15.19:0/3888239624 >> [v2:172.21.15.33:6836/160187378,v1:172.21.15.33:6837/160187378] conn(0x557ebaba22e0 0x557ebaba3750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:33.925 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.918+0000 7f0f67072740 4 client.25154 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T15:17:33.925 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.918+0000 7f0f67072740 1 -- 172.21.15.19:0/3888239624 --> [v2:172.21.15.33:6836/160187378,v1:172.21.15.33:6837/160187378] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557ebab4d2b0 con 0x557ebaba22e0 2024-03-18T15:17:33.925 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.918+0000 7f0f61a00640 1 --2- 172.21.15.19:0/3888239624 >> [v2:172.21.15.33:6836/160187378,v1:172.21.15.33:6837/160187378] conn(0x557ebaba22e0 0x557ebaba3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:33.925 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.919+0000 7f0f61a00640 1 --2- 172.21.15.19:0/3888239624 >> [v2:172.21.15.33:6836/160187378,v1:172.21.15.33:6837/160187378] conn(0x557ebaba22e0 0x557ebaba3750 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=af5a9a788d3f1164 server_cookie=b4311d903856434b in_seq=0 out_seq=0 2024-03-18T15:17:33.925 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.919+0000 7f0f43400640 10 client.25154 ms_handle_connect on v2:172.21.15.33:6836/160187378 2024-03-18T15:17:33.925 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.920+0000 7f0f43400640 1 -- 172.21.15.19:0/3888239624 <== mds.3 v2:172.21.15.33:6836/160187378 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557ebab4d2b0 con 0x557ebaba22e0 2024-03-18T15:17:33.925 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.920+0000 7f0f43400640 10 client.25154 handle_command_reply: tid=0 2024-03-18T15:17:33.925 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.920+0000 7f0f67072740 1 client.25154 shutdown 2024-03-18T15:17:33.925 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.920+0000 7f0f67072740 1 -- 172.21.15.19:0/3888239624 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557ebab8bd30 msgr2=0x557ebab959b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:33.925 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.920+0000 7f0f67072740 1 --2- 172.21.15.19:0/3888239624 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557ebab8bd30 0x557ebab959b0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f0f5c198ed0 tx=0x7f0f48012b60 comp rx=0 tx=0).stop 2024-03-18T15:17:33.925 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.920+0000 7f0f67072740 1 -- 172.21.15.19:0/3888239624 shutdown_connections 2024-03-18T15:17:33.925 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.920+0000 7f0f67072740 1 --2- 172.21.15.19:0/3888239624 >> [v2:172.21.15.33:6836/160187378,v1:172.21.15.33:6837/160187378] conn(0x557ebaba22e0 0x557ebaba3750 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:33.925 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.920+0000 7f0f67072740 1 --2- 172.21.15.19:0/3888239624 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557ebab90ea0 0x557ebab9c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:33.925 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.920+0000 7f0f67072740 1 --2- 172.21.15.19:0/3888239624 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557ebab8bd30 0x557ebab959b0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:33.925 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.920+0000 7f0f43400640 0 client.25154 ms_handle_reset on v2:172.21.15.33:6836/160187378 2024-03-18T15:17:33.926 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.920+0000 7f0f67072740 1 --2- 172.21.15.19:0/3888239624 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x557ebab8b340 0x557ebab95470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:33.926 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.920+0000 7f0f67072740 1 -- 172.21.15.19:0/3888239624 >> 172.21.15.19:0/3888239624 conn(0x557ebab868b0 msgr2=0x557ebab85a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:17:33.926 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.920+0000 7f0f67072740 1 -- 172.21.15.19:0/3888239624 shutdown_connections 2024-03-18T15:17:33.926 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.920+0000 7f0f67072740 1 -- 172.21.15.19:0/3888239624 wait complete. 2024-03-18T15:17:33.926 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.921+0000 7f0f67072740 20 client.25154 trim_cache size 0 max 16384 2024-03-18T15:17:33.941 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.940+0000 7f0f67072740 1 Processor -- start 2024-03-18T15:17:33.941 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.941+0000 7f0f67072740 1 -- start start 2024-03-18T15:17:33.941 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.941+0000 7f0f67072740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557ebab8b340 0x557ebab81900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:33.941 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.941+0000 7f0f67072740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557ebab8bd30 0x557ebab95080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:33.941 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.941+0000 7f0f67072740 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x557ebab90ea0 0x557ebab955c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:33.941 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.941+0000 7f0f67072740 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x557ebaba1a90 con 0x557ebab90ea0 2024-03-18T15:17:33.941 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.941+0000 7f0f67072740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x557ebaba2160 con 0x557ebab8b340 2024-03-18T15:17:33.942 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.941+0000 7f0f67072740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x557ebaba2830 con 0x557ebab8bd30 2024-03-18T15:17:33.946 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.941+0000 7f0f62400640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x557ebab90ea0 0x557ebab955c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:33.946 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.941+0000 7f0f61a00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557ebab8b340 0x557ebab81900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:33.946 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.942+0000 7f0f61a00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557ebab8b340 0x557ebab81900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:39324/0 (socket says 172.21.15.19:39324) 2024-03-18T15:17:33.946 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.942+0000 7f0f61a00640 1 -- 172.21.15.19:0/3285510434 learned_addr learned my addr 172.21.15.19:0/3285510434 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:17:33.946 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.942+0000 7f0f61000640 1 --2- 172.21.15.19:0/3285510434 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557ebab8bd30 0x557ebab95080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:33.946 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.942+0000 7f0f61a00640 1 -- 172.21.15.19:0/3285510434 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557ebab8bd30 msgr2=0x557ebab95080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:17:33.946 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.942+0000 7f0f61a00640 1 --2- 172.21.15.19:0/3285510434 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557ebab8bd30 0x557ebab95080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:33.947 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.942+0000 7f0f61a00640 1 -- 172.21.15.19:0/3285510434 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x557ebab90ea0 msgr2=0x557ebab955c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:17:33.947 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.942+0000 7f0f61a00640 1 --2- 172.21.15.19:0/3285510434 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x557ebab90ea0 0x557ebab955c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:33.947 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.942+0000 7f0f61a00640 1 -- 172.21.15.19:0/3285510434 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557ebab8c110 con 0x557ebab8b340 2024-03-18T15:17:33.947 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.942+0000 7f0f61a00640 1 --2- 172.21.15.19:0/3285510434 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557ebab8b340 0x557ebab81900 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f0f500d7b70 tx=0x7f0f500d7ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=49a6ec04b507d8f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:17:33.947 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.943+0000 7f0f4e000640 1 -- 172.21.15.19:0/3285510434 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f5009b500 con 0x557ebab8b340 2024-03-18T15:17:33.947 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.943+0000 7f0f4e000640 1 -- 172.21.15.19:0/3285510434 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0f5009be30 con 0x557ebab8b340 2024-03-18T15:17:33.947 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.943+0000 7f0f4e000640 1 -- 172.21.15.19:0/3285510434 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f5009b090 con 0x557ebab8b340 2024-03-18T15:17:33.947 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.943+0000 7f0f67072740 1 -- 172.21.15.19:0/3285510434 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557ebab8b340 msgr2=0x557ebab81900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:33.947 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.943+0000 7f0f67072740 1 --2- 172.21.15.19:0/3285510434 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557ebab8b340 0x557ebab81900 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f0f500d7b70 tx=0x7f0f500d7ed0 comp rx=0 tx=0).stop 2024-03-18T15:17:33.947 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.943+0000 7f0f67072740 1 -- 172.21.15.19:0/3285510434 shutdown_connections 2024-03-18T15:17:33.947 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.943+0000 7f0f67072740 1 --2- 172.21.15.19:0/3285510434 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x557ebab90ea0 0x557ebab955c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:33.948 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.943+0000 7f0f67072740 1 --2- 172.21.15.19:0/3285510434 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557ebab8bd30 0x557ebab95080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:33.948 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.943+0000 7f0f67072740 1 --2- 172.21.15.19:0/3285510434 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557ebab8b340 0x557ebab81900 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:33.948 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.943+0000 7f0f67072740 1 -- 172.21.15.19:0/3285510434 >> 172.21.15.19:0/3285510434 conn(0x557ebab868b0 msgr2=0x557ebaba10c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:17:33.948 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.943+0000 7f0f67072740 1 -- 172.21.15.19:0/3285510434 shutdown_connections 2024-03-18T15:17:33.948 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.943+0000 7f0f67072740 1 -- 172.21.15.19:0/3285510434 wait complete. 2024-03-18T15:17:33.948 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.944+0000 7f0f67072740 1 -- start start 2024-03-18T15:17:33.948 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.944+0000 7f0f67072740 1 Processor -- start 2024-03-18T15:17:33.948 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.944+0000 7f0f67072740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557ebab8b340 0x557ebab8ee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:33.948 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.944+0000 7f0f67072740 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x557ebab8bd30 0x557ebab8f340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:33.948 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.944+0000 7f0f67072740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557ebab90ea0 0x557ebac0a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:33.949 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.944+0000 7f0f67072740 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x557ebaba35d0 con 0x557ebab8bd30 2024-03-18T15:17:33.949 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.944+0000 7f0f67072740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x557ebaba2f00 con 0x557ebab8b340 2024-03-18T15:17:33.949 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.945+0000 7f0f67072740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x557ebaba3ca0 con 0x557ebab90ea0 2024-03-18T15:17:33.949 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.945+0000 7f0f67072740 10 client.0 _ll_register_callbacks cb 0x7f0f48009dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T15:17:33.949 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.945+0000 7f0f62400640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557ebab90ea0 0x557ebac0a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:33.949 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.945+0000 7f0f62400640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557ebab90ea0 0x557ebac0a5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.19:58288/0 (socket says 172.21.15.19:58288) 2024-03-18T15:17:33.949 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.945+0000 7f0f62400640 1 -- 172.21.15.19:0/2223272999 learned_addr learned my addr 172.21.15.19:0/2223272999 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:17:33.949 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.945+0000 7f0f61a00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557ebab8b340 0x557ebab8ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:33.949 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.945+0000 7f0f61000640 1 --2- 172.21.15.19:0/2223272999 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x557ebab8bd30 0x557ebab8f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:33.949 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.945+0000 7f0f62400640 1 -- 172.21.15.19:0/2223272999 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557ebab8b340 msgr2=0x557ebab8ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:17:33.950 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.945+0000 7f0f62400640 1 --2- 172.21.15.19:0/2223272999 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557ebab8b340 0x557ebab8ee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:33.950 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.945+0000 7f0f62400640 1 -- 172.21.15.19:0/2223272999 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x557ebab8bd30 msgr2=0x557ebab8f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:17:33.950 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.945+0000 7f0f62400640 1 --2- 172.21.15.19:0/2223272999 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x557ebab8bd30 0x557ebab8f340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:33.950 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.945+0000 7f0f62400640 1 -- 172.21.15.19:0/2223272999 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557ebac0abb0 con 0x557ebab90ea0 2024-03-18T15:17:33.950 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.946+0000 7f0f62400640 1 --2- 172.21.15.19:0/2223272999 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557ebab90ea0 0x557ebac0a5a0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f0f540324b0 tx=0x7f0f5402c0b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=232e4c7fdc8802bd server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:17:33.950 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.946+0000 7f0f43400640 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-03-18T15:17:33.950 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.946+0000 7f0f43400640 1 -- 172.21.15.19:0/2223272999 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f54039040 con 0x557ebab90ea0 2024-03-18T15:17:33.950 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.946+0000 7f0f67072740 1 -- 172.21.15.19:0/2223272999 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0f5009be30 con 0x557ebab90ea0 2024-03-18T15:17:33.950 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.946+0000 7f0f43400640 1 -- 172.21.15.19:0/2223272999 <== mon.1 v2:172.21.15.104:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f0f54005040 con 0x557ebab90ea0 2024-03-18T15:17:33.950 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.946+0000 7f0f43400640 1 -- 172.21.15.19:0/2223272999 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f0f54005240 con 0x557ebab90ea0 2024-03-18T15:17:33.950 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.946+0000 7f0f67072740 10 client.25174 fetch_fsmap learned FSMap version 26 2024-03-18T15:17:33.951 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.946+0000 7f0f67072740 1 -- 172.21.15.19:0/2223272999 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x557ebac0bf20 con 0x557ebab90ea0 2024-03-18T15:17:33.951 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.946+0000 7f0f43400640 1 -- 172.21.15.19:0/2223272999 <== mon.1 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0f5009be30 con 0x557ebab90ea0 2024-03-18T15:17:33.951 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:17:33 smithi104 ceph-mon[36070]: pgmap v392: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T15:17:33.952 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:17:33 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/1643381460' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:17:33.952 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:17:33 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/4009405727' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:17:33.954 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.946+0000 7f0f43400640 1 -- 172.21.15.19:0/2223272999 <== mon.1 v2:172.21.15.104:3300/0 5 ==== fsmap(e 26) ==== 8362+0+0 (secure 0 0 0) 0x7f0f54040b40 con 0x557ebab90ea0 2024-03-18T15:17:33.954 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.947+0000 7f0f67072740 10 client.25174 fetch_fsmap finished waiting for FSMap version 26 2024-03-18T15:17:33.954 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.947+0000 7f0f67072740 10 client.25174 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-18T15:17:33.954 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.947+0000 7f0f67072740 20 client.25174 populate_metadata read hostname 'smithi019' 2024-03-18T15:17:33.954 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.947+0000 7f0f67072740 1 --2- 172.21.15.19:0/2223272999 >> [v2:172.21.15.33:6836/160187378,v1:172.21.15.33:6837/160187378] conn(0x557ebac10ba0 0x557ebac0f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:33.954 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.947+0000 7f0f67072740 4 client.25174 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-18T15:17:33.954 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.947+0000 7f0f67072740 1 -- 172.21.15.19:0/2223272999 --> [v2:172.21.15.33:6836/160187378,v1:172.21.15.33:6837/160187378] -- command(tid 0: {"prefix": "status"}) -- 0x557ebaafd190 con 0x557ebac10ba0 2024-03-18T15:17:33.955 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.947+0000 7f0f61a00640 1 --2- 172.21.15.19:0/2223272999 >> [v2:172.21.15.33:6836/160187378,v1:172.21.15.33:6837/160187378] conn(0x557ebac10ba0 0x557ebac0f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:33.955 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.947+0000 7f0f61a00640 1 --2- 172.21.15.19:0/2223272999 >> [v2:172.21.15.33:6836/160187378,v1:172.21.15.33:6837/160187378] conn(0x557ebac10ba0 0x557ebac0f8b0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=3dad636a0dfefeda server_cookie=9ad0c09812e7de48 in_seq=0 out_seq=0 2024-03-18T15:17:33.955 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.947+0000 7f0f43400640 10 client.25174 ms_handle_connect on v2:172.21.15.33:6836/160187378 2024-03-18T15:17:33.955 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.947+0000 7f0f43400640 1 -- 172.21.15.19:0/2223272999 <== mds.3 v2:172.21.15.33:6836/160187378 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x557ebaafd190 con 0x557ebac10ba0 2024-03-18T15:17:33.955 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.947+0000 7f0f43400640 10 client.25174 handle_command_reply: tid=0 2024-03-18T15:17:33.955 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.948+0000 7f0f67072740 1 client.25174 shutdown 2024-03-18T15:17:33.955 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.948+0000 7f0f67072740 1 -- 172.21.15.19:0/2223272999 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557ebab90ea0 msgr2=0x557ebac0a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:33.955 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.948+0000 7f0f67072740 1 --2- 172.21.15.19:0/2223272999 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557ebab90ea0 0x557ebac0a5a0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f0f540324b0 tx=0x7f0f5402c0b0 comp rx=0 tx=0).stop 2024-03-18T15:17:33.955 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.948+0000 7f0f67072740 1 -- 172.21.15.19:0/2223272999 shutdown_connections 2024-03-18T15:17:33.955 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.948+0000 7f0f67072740 1 --2- 172.21.15.19:0/2223272999 >> [v2:172.21.15.33:6836/160187378,v1:172.21.15.33:6837/160187378] conn(0x557ebac10ba0 0x557ebac0f8b0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:33.955 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.948+0000 7f0f67072740 1 --2- 172.21.15.19:0/2223272999 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557ebab90ea0 0x557ebac0a5a0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:33.955 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.948+0000 7f0f67072740 1 --2- 172.21.15.19:0/2223272999 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x557ebab8bd30 0x557ebab8f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:33.955 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.948+0000 7f0f67072740 1 --2- 172.21.15.19:0/2223272999 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557ebab8b340 0x557ebab8ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:33.955 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.948+0000 7f0f67072740 1 -- 172.21.15.19:0/2223272999 >> 172.21.15.19:0/2223272999 conn(0x557ebab868b0 msgr2=0x557ebab90010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:17:33.956 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.948+0000 7f0f43400640 0 client.25174 ms_handle_reset on v2:172.21.15.33:6836/160187378 2024-03-18T15:17:33.956 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.948+0000 7f0f67072740 1 -- 172.21.15.19:0/2223272999 shutdown_connections 2024-03-18T15:17:33.956 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.948+0000 7f0f67072740 1 -- 172.21.15.19:0/2223272999 wait complete. 2024-03-18T15:17:33.956 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.948+0000 7f0f67072740 20 client.25174 trim_cache size 0 max 16384 2024-03-18T15:17:33.956 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-03-18T15:17:33.956 INFO:teuthology.orchestra.run.smithi019.stdout: "cluster_fsid": "d4dc2ba6-e538-11ee-95c9-87774f69a715", 2024-03-18T15:17:33.956 INFO:teuthology.orchestra.run.smithi019.stdout: "whoami": 3, 2024-03-18T15:17:33.956 INFO:teuthology.orchestra.run.smithi019.stdout: "id": 24506, 2024-03-18T15:17:33.956 INFO:teuthology.orchestra.run.smithi019.stdout: "want_state": "up:active", 2024-03-18T15:17:33.956 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "up:active", 2024-03-18T15:17:33.956 INFO:teuthology.orchestra.run.smithi019.stdout: "fs_name": "cephfs", 2024-03-18T15:17:33.957 INFO:teuthology.orchestra.run.smithi019.stdout: "rank_uptime": 108.459840568, 2024-03-18T15:17:33.957 INFO:teuthology.orchestra.run.smithi019.stdout: "mdsmap_epoch": 26, 2024-03-18T15:17:33.957 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap_epoch": 93, 2024-03-18T15:17:33.957 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap_epoch_barrier": 89, 2024-03-18T15:17:33.957 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 163.862041721 2024-03-18T15:17:33.957 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-03-18T15:17:33.958 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.949+0000 7f0f62e00640 1 -- 172.21.15.19:0/3580296332 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f0f440a2880 msgr2=0x7f0f440a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:33.958 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.949+0000 7f0f62e00640 1 --2- 172.21.15.19:0/3580296332 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f0f440a2880 0x7f0f440a4d40 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f0f540011e0 tx=0x7f0f5402e040 comp rx=0 tx=0).stop 2024-03-18T15:17:33.958 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.949+0000 7f0f62e00640 1 -- 172.21.15.19:0/3580296332 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0f5c1005d0 msgr2=0x7f0f5c1979c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:33.958 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.949+0000 7f0f62e00640 1 --2- 172.21.15.19:0/3580296332 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0f5c1005d0 0x7f0f5c1979c0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f0f5000e9c0 tx=0x7f0f5000ee80 comp rx=0 tx=0).stop 2024-03-18T15:17:33.958 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.949+0000 7f0f62e00640 1 -- 172.21.15.19:0/3580296332 shutdown_connections 2024-03-18T15:17:33.958 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.949+0000 7f0f62e00640 1 --2- 172.21.15.19:0/3580296332 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f5c102180 0x7f0f5c19ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:33.958 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.949+0000 7f0f62e00640 1 --2- 172.21.15.19:0/3580296332 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0f5c1017c0 0x7f0f5c197f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:33.958 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.949+0000 7f0f62e00640 1 --2- 172.21.15.19:0/3580296332 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f0f440a2880 0x7f0f440a4d40 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:33.958 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.949+0000 7f0f62e00640 1 --2- 172.21.15.19:0/3580296332 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0f5c1005d0 0x7f0f5c1979c0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:33.958 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.949+0000 7f0f62e00640 1 -- 172.21.15.19:0/3580296332 >> 172.21.15.19:0/3580296332 conn(0x7f0f5c0fbb60 msgr2=0x7f0f5c069cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:17:33.958 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.949+0000 7f0f62e00640 1 -- 172.21.15.19:0/3580296332 shutdown_connections 2024-03-18T15:17:33.958 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:33.949+0000 7f0f62e00640 1 -- 172.21.15.19:0/3580296332 wait complete. 2024-03-18T15:17:33.969 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-18T15:17:34.201 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.201+0000 7fe0c4c00640 1 -- 172.21.15.19:0/298780179 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe0c01028b0 msgr2=0x7fe0c0102d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:34.201 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.201+0000 7fe0c4c00640 1 --2- 172.21.15.19:0/298780179 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe0c01028b0 0x7fe0c0102d30 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fe0a8009900 tx=0x7fe0a802bc40 comp rx=0 tx=0).stop 2024-03-18T15:17:34.202 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.201+0000 7fe0c4c00640 1 -- 172.21.15.19:0/298780179 shutdown_connections 2024-03-18T15:17:34.202 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.201+0000 7fe0c4c00640 1 --2- 172.21.15.19:0/298780179 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe0c0103270 0x7fe0c0109b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:34.202 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.201+0000 7fe0c4c00640 1 --2- 172.21.15.19:0/298780179 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe0c01028b0 0x7fe0c0102d30 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:34.202 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.201+0000 7fe0c4c00640 1 --2- 172.21.15.19:0/298780179 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe0c01016c0 0x7fe0c0101ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:34.202 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.201+0000 7fe0c4c00640 1 -- 172.21.15.19:0/298780179 >> 172.21.15.19:0/298780179 conn(0x7fe0c00fcc50 msgr2=0x7fe0c00ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:17:34.202 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.201+0000 7fe0c4c00640 1 -- 172.21.15.19:0/298780179 shutdown_connections 2024-03-18T15:17:34.202 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.201+0000 7fe0c4c00640 1 -- 172.21.15.19:0/298780179 wait complete. 2024-03-18T15:17:34.204 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.202+0000 7fe0c4c00640 1 Processor -- start 2024-03-18T15:17:34.204 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.202+0000 7fe0c4c00640 1 -- start start 2024-03-18T15:17:34.205 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.202+0000 7fe0c4c00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe0c01016c0 0x7fe0c0199bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:34.205 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.202+0000 7fe0c4c00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe0c01028b0 0x7fe0c019a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:34.205 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.202+0000 7fe0c4c00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe0c0103270 0x7fe0c01a1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:34.205 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.202+0000 7fe0c4c00640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fe0c010d370 con 0x7fe0c0103270 2024-03-18T15:17:34.205 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.202+0000 7fe0c4c00640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe0c010d1f0 con 0x7fe0c01028b0 2024-03-18T15:17:34.205 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.202+0000 7fe0c4c00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fe0c010d4f0 con 0x7fe0c01016c0 2024-03-18T15:17:34.205 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.203+0000 7fe0bf400640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe0c01016c0 0x7fe0c0199bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:34.205 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.203+0000 7fe0b7e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe0c01028b0 0x7fe0c019a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:34.205 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.203+0000 7fe0bfe00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe0c0103270 0x7fe0c01a1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:34.205 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.203+0000 7fe0bf400640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe0c01016c0 0x7fe0c0199bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.19:58316/0 (socket says 172.21.15.19:58316) 2024-03-18T15:17:34.205 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.203+0000 7fe0bf400640 1 -- 172.21.15.19:0/1424166876 learned_addr learned my addr 172.21.15.19:0/1424166876 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:17:34.205 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.203+0000 7fe0bf400640 1 -- 172.21.15.19:0/1424166876 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe0c01028b0 msgr2=0x7fe0c019a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:34.205 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.203+0000 7fe0bf400640 1 --2- 172.21.15.19:0/1424166876 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe0c01028b0 0x7fe0c019a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:34.205 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.203+0000 7fe0bf400640 1 -- 172.21.15.19:0/1424166876 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe0c0103270 msgr2=0x7fe0c01a1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:34.206 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.203+0000 7fe0bf400640 1 --2- 172.21.15.19:0/1424166876 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe0c0103270 0x7fe0c01a1170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:34.206 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.203+0000 7fe0bf400640 1 -- 172.21.15.19:0/1424166876 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0c01a17e0 con 0x7fe0c01016c0 2024-03-18T15:17:34.206 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.203+0000 7fe0b7e00640 1 --2- 172.21.15.19:0/1424166876 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe0c01028b0 0x7fe0c019a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:17:34.206 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.203+0000 7fe0bfe00640 1 --2- 172.21.15.19:0/1424166876 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe0c0103270 0x7fe0c01a1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:17:34.206 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.204+0000 7fe0bf400640 1 --2- 172.21.15.19:0/1424166876 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe0c01016c0 0x7fe0c0199bc0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fe0b000e9c0 tx=0x7fe0b000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:17:34.206 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.204+0000 7fe0bcc00640 1 -- 172.21.15.19:0/1424166876 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0b000cd80 con 0x7fe0c01016c0 2024-03-18T15:17:34.206 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.204+0000 7fe0bcc00640 1 -- 172.21.15.19:0/1424166876 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe0b00035f0 con 0x7fe0c01016c0 2024-03-18T15:17:34.206 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.204+0000 7fe0bcc00640 1 -- 172.21.15.19:0/1424166876 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0b0010690 con 0x7fe0c01016c0 2024-03-18T15:17:34.206 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.204+0000 7fe0c4c00640 1 -- 172.21.15.19:0/1424166876 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0c01a1ad0 con 0x7fe0c01016c0 2024-03-18T15:17:34.206 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.204+0000 7fe0c4c00640 1 -- 172.21.15.19:0/1424166876 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0c01a1fa0 con 0x7fe0c01016c0 2024-03-18T15:17:34.207 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.207+0000 7fe0c8fa0740 1 Processor -- start 2024-03-18T15:17:34.207 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.207+0000 7fe0c8fa0740 1 -- start start 2024-03-18T15:17:34.207 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.208+0000 7fe0bcc00640 1 -- 172.21.15.19:0/1424166876 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fe0b00037a0 con 0x7fe0c01016c0 2024-03-18T15:17:34.210 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.208+0000 7fe0c8fa0740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d371117340 0x55d371117760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:34.210 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.208+0000 7fe0c8fa0740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d371117d30 0x55d37111c830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:34.210 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.208+0000 7fe0c8fa0740 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55d37111cea0 0x55d37111f330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:34.210 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.208+0000 7fe0c8fa0740 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x55d37107bc50 con 0x55d37111cea0 2024-03-18T15:17:34.210 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.208+0000 7fe0c8fa0740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55d3710249d0 con 0x55d371117340 2024-03-18T15:17:34.210 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.208+0000 7fe0c8fa0740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55d371077cf0 con 0x55d371117d30 2024-03-18T15:17:34.211 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.208+0000 7fe0bfe00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d371117340 0x55d371117760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:34.211 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.208+0000 7fe0bf400640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d371117d30 0x55d37111c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:34.211 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.208+0000 7fe0bfe00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d371117340 0x55d371117760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:39370/0 (socket says 172.21.15.19:39370) 2024-03-18T15:17:34.211 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.208+0000 7fe0bfe00640 1 -- 172.21.15.19:0/2762594718 learned_addr learned my addr 172.21.15.19:0/2762594718 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:17:34.211 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.208+0000 7fe0b7e00640 1 --2- 172.21.15.19:0/2762594718 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55d37111cea0 0x55d37111f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:34.211 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.208+0000 7fe0bfe00640 1 -- 172.21.15.19:0/2762594718 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d371117d30 msgr2=0x55d37111c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:17:34.211 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.208+0000 7fe0bfe00640 1 --2- 172.21.15.19:0/2762594718 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d371117d30 0x55d37111c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:34.211 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.208+0000 7fe0bfe00640 1 -- 172.21.15.19:0/2762594718 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55d37111cea0 msgr2=0x55d37111f330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:17:34.211 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.208+0000 7fe0bfe00640 1 --2- 172.21.15.19:0/2762594718 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55d37111cea0 0x55d37111f330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:34.211 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.208+0000 7fe0bfe00640 1 -- 172.21.15.19:0/2762594718 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d37103ef10 con 0x55d371117340 2024-03-18T15:17:34.211 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.209+0000 7fe0bfe00640 1 --2- 172.21.15.19:0/2762594718 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d371117340 0x55d371117760 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fe0b80117e0 tx=0x7fe0b8011ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f689e111520ad295 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:17:34.212 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.209+0000 7fe0afe00640 1 -- 172.21.15.19:0/2762594718 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0b8011310 con 0x55d371117340 2024-03-18T15:17:34.212 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.209+0000 7fe0afe00640 1 -- 172.21.15.19:0/2762594718 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe0b8003e50 con 0x55d371117340 2024-03-18T15:17:34.212 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.209+0000 7fe0afe00640 1 -- 172.21.15.19:0/2762594718 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0b8004560 con 0x55d371117340 2024-03-18T15:17:34.212 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.209+0000 7fe0bcc00640 1 --2- 172.21.15.19:0/1424166876 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fe0a40a2880 0x7fe0a40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:34.212 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.209+0000 7fe0bcc00640 1 -- 172.21.15.19:0/1424166876 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fe0b00cf160 con 0x7fe0c01016c0 2024-03-18T15:17:34.212 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.210+0000 7fe0bf400640 1 --2- 172.21.15.19:0/1424166876 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fe0a40a2880 0x7fe0a40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:34.212 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.210+0000 7fe0c8fa0740 1 -- 172.21.15.19:0/2762594718 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d371117340 msgr2=0x55d371117760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:34.212 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.210+0000 7fe0c8fa0740 1 --2- 172.21.15.19:0/2762594718 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d371117340 0x55d371117760 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fe0b80117e0 tx=0x7fe0b8011ca0 comp rx=0 tx=0).stop 2024-03-18T15:17:34.212 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.210+0000 7fe0c8fa0740 1 -- 172.21.15.19:0/2762594718 shutdown_connections 2024-03-18T15:17:34.212 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.210+0000 7fe0c8fa0740 1 --2- 172.21.15.19:0/2762594718 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55d37111cea0 0x55d37111f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:34.212 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.210+0000 7fe0c8fa0740 1 --2- 172.21.15.19:0/2762594718 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d371117d30 0x55d37111c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:34.212 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.210+0000 7fe0c8fa0740 1 --2- 172.21.15.19:0/2762594718 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d371117340 0x55d371117760 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:34.213 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.210+0000 7fe0c8fa0740 1 -- 172.21.15.19:0/2762594718 >> 172.21.15.19:0/2762594718 conn(0x55d3711128b0 msgr2=0x55d371112cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:17:34.213 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.210+0000 7fe0c8fa0740 1 -- 172.21.15.19:0/2762594718 shutdown_connections 2024-03-18T15:17:34.213 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.210+0000 7fe0bf400640 1 --2- 172.21.15.19:0/1424166876 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fe0a40a2880 0x7fe0a40a4d40 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x55d371118130 tx=0x7fe0b00d2c30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:17:34.213 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.210+0000 7fe0c8fa0740 1 -- 172.21.15.19:0/2762594718 wait complete. 2024-03-18T15:17:34.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.211+0000 7fe0c8fa0740 1 -- start start 2024-03-18T15:17:34.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.211+0000 7fe0c8fa0740 1 Processor -- start 2024-03-18T15:17:34.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.211+0000 7fe0c8fa0740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d371117340 0x55d371121490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:34.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.211+0000 7fe0c8fa0740 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55d371117d30 0x55d3711219d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:34.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.211+0000 7fe0c8fa0740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d37111cea0 0x55d371128a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:34.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.211+0000 7fe0c8fa0740 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x55d37111fe50 con 0x55d371117d30 2024-03-18T15:17:34.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.211+0000 7fe0c8fa0740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55d371037e20 con 0x55d371117340 2024-03-18T15:17:34.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.211+0000 7fe0c8fa0740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55d37111ffd0 con 0x55d37111cea0 2024-03-18T15:17:34.218 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.211+0000 7fe0c8fa0740 10 client.0 _ll_register_callbacks cb 0x55d3710377f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T15:17:34.218 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.212+0000 7fe0b7e00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55d371117d30 0x55d3711219d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:34.218 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.212+0000 7fe0bfe00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d371117340 0x55d371121490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:34.218 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.212+0000 7fe0b7e00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55d371117d30 0x55d3711219d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:50558/0 (socket says 172.21.15.19:50558) 2024-03-18T15:17:34.218 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.212+0000 7fe0b7e00640 1 -- 172.21.15.19:0/1405758109 learned_addr learned my addr 172.21.15.19:0/1405758109 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:17:34.218 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.212+0000 7fe0bfe00640 1 --2- 172.21.15.19:0/1405758109 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d37111cea0 0x55d371128a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:34.218 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.212+0000 7fe0b7e00640 1 -- 172.21.15.19:0/1405758109 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d37111cea0 msgr2=0x55d371128a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:17:34.218 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.212+0000 7fe0b7e00640 1 --2- 172.21.15.19:0/1405758109 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d37111cea0 0x55d371128a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:34.218 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.212+0000 7fe0b7e00640 1 -- 172.21.15.19:0/1405758109 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d371117340 msgr2=0x55d371121490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:17:34.218 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.212+0000 7fe0b7e00640 1 --2- 172.21.15.19:0/1405758109 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d371117340 0x55d371121490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:34.219 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.212+0000 7fe0b7e00640 1 -- 172.21.15.19:0/1405758109 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d370d902d0 con 0x55d371117d30 2024-03-18T15:17:34.219 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.212+0000 7fe0bfe00640 1 --2- 172.21.15.19:0/1405758109 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d37111cea0 0x55d371128a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T15:17:34.219 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.213+0000 7fe0b7e00640 1 --2- 172.21.15.19:0/1405758109 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55d371117d30 0x55d3711219d0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe0a802efd0 tx=0x7fe0a8009560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=900802c43bbd77f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:17:34.219 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.213+0000 7fe0acc00640 10 client.0 ms_handle_connect on v2:172.21.15.19:3300/0 2024-03-18T15:17:34.219 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.213+0000 7fe0acc00640 1 -- 172.21.15.19:0/1405758109 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0a80048c0 con 0x55d371117d30 2024-03-18T15:17:34.219 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.213+0000 7fe0acc00640 1 -- 172.21.15.19:0/1405758109 <== mon.0 v2:172.21.15.19:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fe0a8004a60 con 0x55d371117d30 2024-03-18T15:17:34.219 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.213+0000 7fe0c8fa0740 1 -- 172.21.15.19:0/1405758109 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe0b8003e50 con 0x55d371117d30 2024-03-18T15:17:34.219 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.213+0000 7fe0acc00640 1 -- 172.21.15.19:0/1405758109 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fe0a802cd40 con 0x55d371117d30 2024-03-18T15:17:34.219 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.213+0000 7fe0c8fa0740 10 client.15078 fetch_fsmap learned FSMap version 26 2024-03-18T15:17:34.219 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.213+0000 7fe0c8fa0740 1 -- 172.21.15.19:0/1405758109 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55d370e5a6b0 con 0x55d371117d30 2024-03-18T15:17:34.219 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.214+0000 7fe0acc00640 1 -- 172.21.15.19:0/1405758109 <== mon.0 v2:172.21.15.19:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe0b8003e50 con 0x55d371117d30 2024-03-18T15:17:34.219 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.214+0000 7fe0acc00640 1 -- 172.21.15.19:0/1405758109 <== mon.0 v2:172.21.15.19:3300/0 5 ==== fsmap(e 26) ==== 8362+0+0 (secure 0 0 0) 0x7fe0a8003850 con 0x55d371117d30 2024-03-18T15:17:34.220 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.214+0000 7fe0c8fa0740 10 client.15078 fetch_fsmap finished waiting for FSMap version 26 2024-03-18T15:17:34.220 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.214+0000 7fe0c8fa0740 10 client.15078 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-18T15:17:34.220 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.214+0000 7fe0c8fa0740 20 client.15078 populate_metadata read hostname 'smithi019' 2024-03-18T15:17:34.220 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.214+0000 7fe0c8fa0740 1 --2- 172.21.15.19:0/1405758109 >> [v2:172.21.15.104:6836/260984674,v1:172.21.15.104:6837/260984674] conn(0x55d37112e290 0x55d37112f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:34.220 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.214+0000 7fe0c8fa0740 4 client.15078 mds_command: new command op to 24511 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T15:17:34.220 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.214+0000 7fe0c8fa0740 1 -- 172.21.15.19:0/1405758109 --> [v2:172.21.15.104:6836/260984674,v1:172.21.15.104:6837/260984674] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d3710d92b0 con 0x55d37112e290 2024-03-18T15:17:34.220 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.214+0000 7fe0bfe00640 1 --2- 172.21.15.19:0/1405758109 >> [v2:172.21.15.104:6836/260984674,v1:172.21.15.104:6837/260984674] conn(0x55d37112e290 0x55d37112f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:34.220 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.215+0000 7fe0bfe00640 1 --2- 172.21.15.19:0/1405758109 >> [v2:172.21.15.104:6836/260984674,v1:172.21.15.104:6837/260984674] conn(0x55d37112e290 0x55d37112f700 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9929281c556e9663 server_cookie=242d07f50017543a in_seq=0 out_seq=0 2024-03-18T15:17:34.220 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.215+0000 7fe0acc00640 10 client.15078 ms_handle_connect on v2:172.21.15.104:6836/260984674 2024-03-18T15:17:34.220 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.216+0000 7fe0acc00640 1 -- 172.21.15.19:0/1405758109 <== mds.2 v2:172.21.15.104:6836/260984674 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d3710d92b0 con 0x55d37112e290 2024-03-18T15:17:34.220 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.216+0000 7fe0acc00640 10 client.15078 handle_command_reply: tid=0 2024-03-18T15:17:34.220 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.216+0000 7fe0c8fa0740 1 client.15078 shutdown 2024-03-18T15:17:34.221 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.216+0000 7fe0c8fa0740 1 -- 172.21.15.19:0/1405758109 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55d371117d30 msgr2=0x55d3711219d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:34.221 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.216+0000 7fe0c8fa0740 1 --2- 172.21.15.19:0/1405758109 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55d371117d30 0x55d3711219d0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe0a802efd0 tx=0x7fe0a8009560 comp rx=0 tx=0).stop 2024-03-18T15:17:34.221 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.216+0000 7fe0c8fa0740 1 -- 172.21.15.19:0/1405758109 shutdown_connections 2024-03-18T15:17:34.221 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.216+0000 7fe0c8fa0740 1 --2- 172.21.15.19:0/1405758109 >> [v2:172.21.15.104:6836/260984674,v1:172.21.15.104:6837/260984674] conn(0x55d37112e290 0x55d37112f700 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:34.221 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.216+0000 7fe0c8fa0740 1 --2- 172.21.15.19:0/1405758109 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d37111cea0 0x55d371128a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:34.221 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.216+0000 7fe0c8fa0740 1 --2- 172.21.15.19:0/1405758109 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55d371117d30 0x55d3711219d0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:34.221 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.216+0000 7fe0c8fa0740 1 --2- 172.21.15.19:0/1405758109 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d371117340 0x55d371121490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:34.221 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.216+0000 7fe0c8fa0740 1 -- 172.21.15.19:0/1405758109 >> 172.21.15.19:0/1405758109 conn(0x55d3711128b0 msgr2=0x55d371111a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:17:34.221 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.216+0000 7fe0acc00640 0 client.15078 ms_handle_reset on v2:172.21.15.104:6836/260984674 2024-03-18T15:17:34.221 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.216+0000 7fe0c8fa0740 1 -- 172.21.15.19:0/1405758109 shutdown_connections 2024-03-18T15:17:34.221 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.217+0000 7fe0c8fa0740 1 -- 172.21.15.19:0/1405758109 wait complete. 2024-03-18T15:17:34.221 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.217+0000 7fe0c8fa0740 20 client.15078 trim_cache size 0 max 16384 2024-03-18T15:17:34.237 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.237+0000 7fe0c8fa0740 1 Processor -- start 2024-03-18T15:17:34.237 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.237+0000 7fe0c8fa0740 1 -- start start 2024-03-18T15:17:34.237 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.237+0000 7fe0c8fa0740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d371117340 0x55d3711210a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:34.237 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.237+0000 7fe0c8fa0740 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55d371117d30 0x55d3711289d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:34.237 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.237+0000 7fe0c8fa0740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d37111cea0 0x55d3711215e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:34.237 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.237+0000 7fe0c8fa0740 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x55d37112da40 con 0x55d371117d30 2024-03-18T15:17:34.237 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.237+0000 7fe0c8fa0740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55d37112e110 con 0x55d371117340 2024-03-18T15:17:34.237 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.237+0000 7fe0c8fa0740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55d37112e7e0 con 0x55d37111cea0 2024-03-18T15:17:34.239 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.238+0000 7fe0b7e00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55d371117d30 0x55d3711289d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:34.239 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.238+0000 7fe0b7e00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55d371117d30 0x55d3711289d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:50568/0 (socket says 172.21.15.19:50568) 2024-03-18T15:17:34.239 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.238+0000 7fe0b7e00640 1 -- 172.21.15.19:0/694007489 learned_addr learned my addr 172.21.15.19:0/694007489 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:17:34.239 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.238+0000 7fe0bfe00640 1 --2- 172.21.15.19:0/694007489 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d371117340 0x55d3711210a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:34.239 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.238+0000 7fe0b7e00640 1 -- 172.21.15.19:0/694007489 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d37111cea0 msgr2=0x55d3711215e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:17:34.239 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.238+0000 7fe0bfe00640 1 --2- 172.21.15.19:0/694007489 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d37111cea0 0x55d3711215e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:34.239 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.238+0000 7fe0b7e00640 1 --2- 172.21.15.19:0/694007489 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d37111cea0 0x55d3711215e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:34.239 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.238+0000 7fe0b7e00640 1 -- 172.21.15.19:0/694007489 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d371117340 msgr2=0x55d3711210a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:17:34.240 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.238+0000 7fe0b7e00640 1 --2- 172.21.15.19:0/694007489 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d371117340 0x55d3711210a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:34.240 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.238+0000 7fe0b7e00640 1 -- 172.21.15.19:0/694007489 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d37111c0e0 con 0x55d371117d30 2024-03-18T15:17:34.240 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.238+0000 7fe0bfe00640 1 --2- 172.21.15.19:0/694007489 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d37111cea0 0x55d3711215e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T15:17:34.240 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.239+0000 7fe0b7e00640 1 --2- 172.21.15.19:0/694007489 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55d371117d30 0x55d3711289d0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fe0a8003eb0 tx=0x7fe0a802bc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dad6860008982cf6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:17:34.240 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.239+0000 7fe0af400640 1 -- 172.21.15.19:0/694007489 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0a805a020 con 0x55d371117d30 2024-03-18T15:17:34.240 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.239+0000 7fe0af400640 1 -- 172.21.15.19:0/694007489 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe0a802cd40 con 0x55d371117d30 2024-03-18T15:17:34.240 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.239+0000 7fe0af400640 1 -- 172.21.15.19:0/694007489 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0a80349c0 con 0x55d371117d30 2024-03-18T15:17:34.245 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.240+0000 7fe0c8fa0740 1 -- 172.21.15.19:0/694007489 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55d371117d30 msgr2=0x55d3711289d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:34.245 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.240+0000 7fe0c8fa0740 1 --2- 172.21.15.19:0/694007489 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55d371117d30 0x55d3711289d0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fe0a8003eb0 tx=0x7fe0a802bc40 comp rx=0 tx=0).stop 2024-03-18T15:17:34.245 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.240+0000 7fe0c8fa0740 1 -- 172.21.15.19:0/694007489 shutdown_connections 2024-03-18T15:17:34.245 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.240+0000 7fe0c8fa0740 1 --2- 172.21.15.19:0/694007489 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d37111cea0 0x55d3711215e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:34.246 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.240+0000 7fe0c8fa0740 1 --2- 172.21.15.19:0/694007489 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55d371117d30 0x55d3711289d0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:34.246 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.240+0000 7fe0c8fa0740 1 --2- 172.21.15.19:0/694007489 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d371117340 0x55d3711210a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:34.246 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.240+0000 7fe0c8fa0740 1 -- 172.21.15.19:0/694007489 >> 172.21.15.19:0/694007489 conn(0x55d3711128b0 msgr2=0x55d37110d900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:17:34.246 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.240+0000 7fe0c8fa0740 1 -- 172.21.15.19:0/694007489 shutdown_connections 2024-03-18T15:17:34.246 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.240+0000 7fe0c8fa0740 1 -- 172.21.15.19:0/694007489 wait complete. 2024-03-18T15:17:34.246 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.240+0000 7fe0c8fa0740 1 -- start start 2024-03-18T15:17:34.246 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.241+0000 7fe0c8fa0740 1 Processor -- start 2024-03-18T15:17:34.246 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.241+0000 7fe0c8fa0740 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55d371117340 0x55d3711247a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:34.246 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.241+0000 7fe0c8fa0740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d371117d30 0x55d371124ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:34.246 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.241+0000 7fe0c8fa0740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d37111cea0 0x55d371127d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:34.246 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.241+0000 7fe0c8fa0740 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x55d37112f580 con 0x55d371117340 2024-03-18T15:17:34.246 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.241+0000 7fe0c8fa0740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55d37112eeb0 con 0x55d37111cea0 2024-03-18T15:17:34.247 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.241+0000 7fe0c8fa0740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55d37112fc50 con 0x55d371117d30 2024-03-18T15:17:34.247 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.241+0000 7fe0c8fa0740 10 client.0 _ll_register_callbacks cb 0x7fe0a8004370 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T15:17:34.247 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.241+0000 7fe0b7e00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d371117d30 0x55d371124ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:34.247 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.241+0000 7fe0b7e00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d371117d30 0x55d371124ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.19:58348/0 (socket says 172.21.15.19:58348) 2024-03-18T15:17:34.247 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.241+0000 7fe0b7e00640 1 -- 172.21.15.19:0/1853238193 learned_addr learned my addr 172.21.15.19:0/1853238193 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:17:34.247 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.241+0000 7fe0bfe00640 1 --2- 172.21.15.19:0/1853238193 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55d371117340 0x55d3711247a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:34.247 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.242+0000 7fe0bfe00640 1 --2- 172.21.15.19:0/1853238193 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d37111cea0 0x55d371127d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:34.247 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.242+0000 7fe0b7e00640 1 -- 172.21.15.19:0/1853238193 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d37111cea0 msgr2=0x55d371127d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:17:34.247 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.242+0000 7fe0b7e00640 1 --2- 172.21.15.19:0/1853238193 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d37111cea0 0x55d371127d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:34.247 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.242+0000 7fe0b7e00640 1 -- 172.21.15.19:0/1853238193 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55d371117340 msgr2=0x55d3711247a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:17:34.247 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.242+0000 7fe0b7e00640 1 --2- 172.21.15.19:0/1853238193 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55d371117340 0x55d3711247a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:34.247 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.242+0000 7fe0b7e00640 1 -- 172.21.15.19:0/1853238193 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d3711282f0 con 0x55d371117d30 2024-03-18T15:17:34.248 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.242+0000 7fe0bfe00640 1 --2- 172.21.15.19:0/1853238193 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55d371117340 0x55d3711247a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T15:17:34.248 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.242+0000 7fe0b7e00640 1 --2- 172.21.15.19:0/1853238193 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d371117d30 0x55d371124ce0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fe0a8003d40 tx=0x7fe0a8003820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c5c19b4cbc7264f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:17:34.248 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.242+0000 7fe0acc00640 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-03-18T15:17:34.248 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.242+0000 7fe0acc00640 1 -- 172.21.15.19:0/1853238193 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0a805a070 con 0x55d371117d30 2024-03-18T15:17:34.248 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.242+0000 7fe0c8fa0740 1 -- 172.21.15.19:0/1853238193 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe0a802cd40 con 0x55d371117d30 2024-03-18T15:17:34.248 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.242+0000 7fe0acc00640 1 -- 172.21.15.19:0/1853238193 <== mon.1 v2:172.21.15.104:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fe0a8004ef0 con 0x55d371117d30 2024-03-18T15:17:34.248 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.243+0000 7fe0acc00640 1 -- 172.21.15.19:0/1853238193 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fe0a806d050 con 0x55d371117d30 2024-03-18T15:17:34.248 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.243+0000 7fe0c8fa0740 10 client.25189 fetch_fsmap learned FSMap version 26 2024-03-18T15:17:34.248 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.243+0000 7fe0c8fa0740 1 -- 172.21.15.19:0/1853238193 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55d3711945c0 con 0x55d371117d30 2024-03-18T15:17:34.248 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.243+0000 7fe0acc00640 1 -- 172.21.15.19:0/1853238193 <== mon.1 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe0a802cd40 con 0x55d371117d30 2024-03-18T15:17:34.248 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.243+0000 7fe0acc00640 1 -- 172.21.15.19:0/1853238193 <== mon.1 v2:172.21.15.104:3300/0 5 ==== fsmap(e 26) ==== 8362+0+0 (secure 0 0 0) 0x7fe0a8034830 con 0x55d371117d30 2024-03-18T15:17:34.248 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.243+0000 7fe0c8fa0740 10 client.25189 fetch_fsmap finished waiting for FSMap version 26 2024-03-18T15:17:34.249 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.243+0000 7fe0c8fa0740 10 client.25189 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-18T15:17:34.249 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.243+0000 7fe0c8fa0740 20 client.25189 populate_metadata read hostname 'smithi019' 2024-03-18T15:17:34.249 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.243+0000 7fe0c8fa0740 1 --2- 172.21.15.19:0/1853238193 >> [v2:172.21.15.104:6836/260984674,v1:172.21.15.104:6837/260984674] conn(0x55d371198c80 0x55d371197990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:34.249 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.243+0000 7fe0c8fa0740 4 client.25189 mds_command: new command op to 24511 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-18T15:17:34.249 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.243+0000 7fe0c8fa0740 1 -- 172.21.15.19:0/1853238193 --> [v2:172.21.15.104:6836/260984674,v1:172.21.15.104:6837/260984674] -- command(tid 0: {"prefix": "status"}) -- 0x55d371089190 con 0x55d371198c80 2024-03-18T15:17:34.249 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.243+0000 7fe0bfe00640 1 --2- 172.21.15.19:0/1853238193 >> [v2:172.21.15.104:6836/260984674,v1:172.21.15.104:6837/260984674] conn(0x55d371198c80 0x55d371197990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:34.249 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.244+0000 7fe0bfe00640 1 --2- 172.21.15.19:0/1853238193 >> [v2:172.21.15.104:6836/260984674,v1:172.21.15.104:6837/260984674] conn(0x55d371198c80 0x55d371197990 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9d7328de713618dd server_cookie=77b4b6b693bdf749 in_seq=0 out_seq=0 2024-03-18T15:17:34.249 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.244+0000 7fe0acc00640 10 client.25189 ms_handle_connect on v2:172.21.15.104:6836/260984674 2024-03-18T15:17:34.249 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.244+0000 7fe0acc00640 1 -- 172.21.15.19:0/1853238193 <== mds.2 v2:172.21.15.104:6836/260984674 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x55d371089190 con 0x55d371198c80 2024-03-18T15:17:34.249 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.244+0000 7fe0acc00640 10 client.25189 handle_command_reply: tid=0 2024-03-18T15:17:34.249 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.244+0000 7fe0c8fa0740 1 client.25189 shutdown 2024-03-18T15:17:34.249 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.244+0000 7fe0c8fa0740 1 -- 172.21.15.19:0/1853238193 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d371117d30 msgr2=0x55d371124ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:34.250 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.244+0000 7fe0c8fa0740 1 --2- 172.21.15.19:0/1853238193 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d371117d30 0x55d371124ce0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fe0a8003d40 tx=0x7fe0a8003820 comp rx=0 tx=0).stop 2024-03-18T15:17:34.250 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.244+0000 7fe0c8fa0740 1 -- 172.21.15.19:0/1853238193 shutdown_connections 2024-03-18T15:17:34.250 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.244+0000 7fe0c8fa0740 1 --2- 172.21.15.19:0/1853238193 >> [v2:172.21.15.104:6836/260984674,v1:172.21.15.104:6837/260984674] conn(0x55d371198c80 0x55d371197990 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:34.250 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.244+0000 7fe0c8fa0740 1 --2- 172.21.15.19:0/1853238193 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d37111cea0 0x55d371127d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:34.250 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.244+0000 7fe0c8fa0740 1 --2- 172.21.15.19:0/1853238193 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d371117d30 0x55d371124ce0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:34.250 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.244+0000 7fe0acc00640 0 client.25189 ms_handle_reset on v2:172.21.15.104:6836/260984674 2024-03-18T15:17:34.250 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.244+0000 7fe0c8fa0740 1 --2- 172.21.15.19:0/1853238193 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55d371117340 0x55d3711247a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:34.250 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.244+0000 7fe0c8fa0740 1 -- 172.21.15.19:0/1853238193 >> 172.21.15.19:0/1853238193 conn(0x55d3711128b0 msgr2=0x55d3711210a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:17:34.250 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.245+0000 7fe0c8fa0740 1 -- 172.21.15.19:0/1853238193 shutdown_connections 2024-03-18T15:17:34.250 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.245+0000 7fe0c8fa0740 1 -- 172.21.15.19:0/1853238193 wait complete. 2024-03-18T15:17:34.250 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.245+0000 7fe0c8fa0740 20 client.25189 trim_cache size 0 max 16384 2024-03-18T15:17:34.251 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-03-18T15:17:34.251 INFO:teuthology.orchestra.run.smithi019.stdout: "cluster_fsid": "d4dc2ba6-e538-11ee-95c9-87774f69a715", 2024-03-18T15:17:34.251 INFO:teuthology.orchestra.run.smithi019.stdout: "whoami": 2, 2024-03-18T15:17:34.251 INFO:teuthology.orchestra.run.smithi019.stdout: "id": 24511, 2024-03-18T15:17:34.251 INFO:teuthology.orchestra.run.smithi019.stdout: "want_state": "up:active", 2024-03-18T15:17:34.251 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "up:active", 2024-03-18T15:17:34.251 INFO:teuthology.orchestra.run.smithi019.stdout: "fs_name": "cephfs", 2024-03-18T15:17:34.251 INFO:teuthology.orchestra.run.smithi019.stdout: "rank_uptime": 109.771892189, 2024-03-18T15:17:34.251 INFO:teuthology.orchestra.run.smithi019.stdout: "mdsmap_epoch": 26, 2024-03-18T15:17:34.251 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap_epoch": 93, 2024-03-18T15:17:34.251 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap_epoch_barrier": 89, 2024-03-18T15:17:34.252 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 167.066026128 2024-03-18T15:17:34.252 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-03-18T15:17:34.252 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.245+0000 7fe0c4c00640 1 -- 172.21.15.19:0/1424166876 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fe0a40a2880 msgr2=0x7fe0a40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:34.253 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.245+0000 7fe0c4c00640 1 --2- 172.21.15.19:0/1424166876 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fe0a40a2880 0x7fe0a40a4d40 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x55d371118130 tx=0x7fe0b00d2c30 comp rx=0 tx=0).stop 2024-03-18T15:17:34.253 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.245+0000 7fe0c4c00640 1 -- 172.21.15.19:0/1424166876 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe0c01016c0 msgr2=0x7fe0c0199bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:34.253 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.245+0000 7fe0c4c00640 1 --2- 172.21.15.19:0/1424166876 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe0c01016c0 0x7fe0c0199bc0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fe0b000e9c0 tx=0x7fe0b000ee80 comp rx=0 tx=0).stop 2024-03-18T15:17:34.253 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.245+0000 7fe0c4c00640 1 -- 172.21.15.19:0/1424166876 shutdown_connections 2024-03-18T15:17:34.253 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.245+0000 7fe0c4c00640 1 --2- 172.21.15.19:0/1424166876 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe0c0103270 0x7fe0c01a1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:34.253 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.245+0000 7fe0c4c00640 1 --2- 172.21.15.19:0/1424166876 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fe0a40a2880 0x7fe0a40a4d40 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:34.253 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.245+0000 7fe0c4c00640 1 --2- 172.21.15.19:0/1424166876 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe0c01028b0 0x7fe0c019a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:34.253 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.245+0000 7fe0c4c00640 1 --2- 172.21.15.19:0/1424166876 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe0c01016c0 0x7fe0c0199bc0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:34.253 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.245+0000 7fe0c4c00640 1 -- 172.21.15.19:0/1424166876 >> 172.21.15.19:0/1424166876 conn(0x7fe0c00fcc50 msgr2=0x7fe0c0073a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:17:34.253 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.246+0000 7fe0c4c00640 1 -- 172.21.15.19:0/1424166876 shutdown_connections 2024-03-18T15:17:34.253 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.246+0000 7fe0c4c00640 1 -- 172.21.15.19:0/1424166876 wait complete. 2024-03-18T15:17:34.266 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-03-18T15:17:34.485 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.485+0000 7f65fd400640 1 -- 172.21.15.19:0/651756866 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f65f81098b0 msgr2=0x7f65f8110140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:34.485 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.485+0000 7f65fd400640 1 --2- 172.21.15.19:0/651756866 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f65f81098b0 0x7f65f8110140 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f65f0009ca0 tx=0x7f65f002c100 comp rx=0 tx=0).stop 2024-03-18T15:17:34.485 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.485+0000 7f65fd400640 1 -- 172.21.15.19:0/651756866 shutdown_connections 2024-03-18T15:17:34.485 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.485+0000 7f65fd400640 1 --2- 172.21.15.19:0/651756866 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f65f81098b0 0x7f65f8110140 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:34.485 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.485+0000 7f65fd400640 1 --2- 172.21.15.19:0/651756866 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f65f8108ef0 0x7f65f8109370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:34.485 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.485+0000 7f65fd400640 1 --2- 172.21.15.19:0/651756866 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f65f8107d00 0x7f65f8108120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:34.485 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.485+0000 7f65fd400640 1 -- 172.21.15.19:0/651756866 >> 172.21.15.19:0/651756866 conn(0x7f65f8076cb0 msgr2=0x7f65f80790d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:17:34.485 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.485+0000 7f65fd400640 1 -- 172.21.15.19:0/651756866 shutdown_connections 2024-03-18T15:17:34.485 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.485+0000 7f65fd400640 1 -- 172.21.15.19:0/651756866 wait complete. 2024-03-18T15:17:34.487 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.486+0000 7f65fd400640 1 Processor -- start 2024-03-18T15:17:34.487 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.486+0000 7f65fd400640 1 -- start start 2024-03-18T15:17:34.487 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.486+0000 7f65fd400640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f65f8107d00 0x7f65f819df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:34.487 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.486+0000 7f65fd400640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f65f8108ef0 0x7f65f819e480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:34.487 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.486+0000 7f65fd400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f65f81098b0 0x7f65f81a54f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:34.487 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.486+0000 7f65fd400640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f65f8113980 con 0x7f65f8107d00 2024-03-18T15:17:34.487 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.486+0000 7f65fd400640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f65f8113800 con 0x7f65f81098b0 2024-03-18T15:17:34.487 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.486+0000 7f65fd400640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f65f8113b00 con 0x7f65f8108ef0 2024-03-18T15:17:34.487 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.486+0000 7f65fca00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f65f81098b0 0x7f65f81a54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:34.487 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.486+0000 7f65f7e00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f65f8107d00 0x7f65f819df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:34.488 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.486+0000 7f65f7400640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f65f8108ef0 0x7f65f819e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:34.488 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.487+0000 7f65fca00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f65f81098b0 0x7f65f81a54f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:39424/0 (socket says 172.21.15.19:39424) 2024-03-18T15:17:34.488 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.487+0000 7f65fca00640 1 -- 172.21.15.19:0/462878193 learned_addr learned my addr 172.21.15.19:0/462878193 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:17:34.488 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.487+0000 7f65fca00640 1 -- 172.21.15.19:0/462878193 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f65f8108ef0 msgr2=0x7f65f819e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:34.488 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.487+0000 7f65fca00640 1 --2- 172.21.15.19:0/462878193 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f65f8108ef0 0x7f65f819e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:34.488 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.487+0000 7f65fca00640 1 -- 172.21.15.19:0/462878193 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f65f8107d00 msgr2=0x7f65f819df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:34.488 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.487+0000 7f65fca00640 1 --2- 172.21.15.19:0/462878193 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f65f8107d00 0x7f65f819df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:34.488 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.487+0000 7f65fca00640 1 -- 172.21.15.19:0/462878193 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65f81a5b60 con 0x7f65f81098b0 2024-03-18T15:17:34.488 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.487+0000 7f65fca00640 1 --2- 172.21.15.19:0/462878193 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f65f81098b0 0x7f65f81a54f0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f65f0037040 tx=0x7f65f0035d90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:17:34.489 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.487+0000 7f65f4c00640 1 -- 172.21.15.19:0/462878193 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65f0039040 con 0x7f65f81098b0 2024-03-18T15:17:34.489 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.487+0000 7f65f4c00640 1 -- 172.21.15.19:0/462878193 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f65f0044070 con 0x7f65f81098b0 2024-03-18T15:17:34.489 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.488+0000 7f65fd400640 1 -- 172.21.15.19:0/462878193 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65f81a5df0 con 0x7f65f81098b0 2024-03-18T15:17:34.489 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.488+0000 7f65f4c00640 1 -- 172.21.15.19:0/462878193 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65f00044d0 con 0x7f65f81098b0 2024-03-18T15:17:34.489 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.488+0000 7f65fd400640 1 -- 172.21.15.19:0/462878193 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65f81a62b0 con 0x7f65f81098b0 2024-03-18T15:17:34.490 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.491+0000 7f65f4c00640 1 -- 172.21.15.19:0/462878193 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f65f0004670 con 0x7f65f81098b0 2024-03-18T15:17:34.491 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.491+0000 7f66017af740 1 Processor -- start 2024-03-18T15:17:34.492 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.491+0000 7f66017af740 1 -- start start 2024-03-18T15:17:34.492 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.491+0000 7f66017af740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f5ec8e1340 0x55f5ec8e1760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:34.492 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.491+0000 7f66017af740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f5ec8e1d30 0x55f5ec8e6830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:34.492 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.491+0000 7f66017af740 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55f5ec8e6ea0 0x55f5ec8e9330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:34.492 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.491+0000 7f66017af740 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x55f5ec845c50 con 0x55f5ec8e6ea0 2024-03-18T15:17:34.492 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.491+0000 7f66017af740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55f5ec7ee9d0 con 0x55f5ec8e1340 2024-03-18T15:17:34.492 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.491+0000 7f66017af740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55f5ec841cf0 con 0x55f5ec8e1d30 2024-03-18T15:17:34.492 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.492+0000 7f65f7e00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f5ec8e1d30 0x55f5ec8e6830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:34.492 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.492+0000 7f65f7e00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f5ec8e1d30 0x55f5ec8e6830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.19:58382/0 (socket says 172.21.15.19:58382) 2024-03-18T15:17:34.492 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.492+0000 7f65f7e00640 1 -- 172.21.15.19:0/301394022 learned_addr learned my addr 172.21.15.19:0/301394022 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:17:34.492 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.492+0000 7f65f7400640 1 --2- 172.21.15.19:0/301394022 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f5ec8e1340 0x55f5ec8e1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:34.492 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.492+0000 7f65f7400640 1 --2- 172.21.15.19:0/301394022 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55f5ec8e6ea0 0x55f5ec8e9330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:34.494 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.492+0000 7f65f7e00640 1 -- 172.21.15.19:0/301394022 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f5ec8e1340 msgr2=0x55f5ec8e1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:17:34.494 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.492+0000 7f65f7e00640 1 --2- 172.21.15.19:0/301394022 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f5ec8e1340 0x55f5ec8e1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:34.494 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.492+0000 7f65f7e00640 1 -- 172.21.15.19:0/301394022 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55f5ec8e6ea0 msgr2=0x55f5ec8e9330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:17:34.494 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.492+0000 7f65f7e00640 1 --2- 172.21.15.19:0/301394022 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55f5ec8e6ea0 0x55f5ec8e9330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:34.494 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.492+0000 7f65f7e00640 1 -- 172.21.15.19:0/301394022 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f5ec808f10 con 0x55f5ec8e1d30 2024-03-18T15:17:34.495 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.492+0000 7f65f4c00640 1 --2- 172.21.15.19:0/462878193 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f65dc0a2880 0x7f65dc0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:34.495 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.493+0000 7f65f4c00640 1 -- 172.21.15.19:0/462878193 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f65f00f1460 con 0x7f65f81098b0 2024-03-18T15:17:34.495 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.493+0000 7f65f7e00640 1 --2- 172.21.15.19:0/301394022 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f5ec8e1d30 0x55f5ec8e6830 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f65e800bcb0 tx=0x7f65e80107a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7c026d09168128b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:17:34.495 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.493+0000 7f65f7400640 1 --2- 172.21.15.19:0/462878193 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f65dc0a2880 0x7f65dc0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:34.495 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.493+0000 7f65ed200640 1 -- 172.21.15.19:0/301394022 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65e8004020 con 0x55f5ec8e1d30 2024-03-18T15:17:34.495 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.493+0000 7f65ed200640 1 -- 172.21.15.19:0/301394022 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f65e8005500 con 0x55f5ec8e1d30 2024-03-18T15:17:34.495 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.493+0000 7f65ed200640 1 -- 172.21.15.19:0/301394022 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65e8004d80 con 0x55f5ec8e1d30 2024-03-18T15:17:34.495 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.494+0000 7f66017af740 1 -- 172.21.15.19:0/301394022 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f5ec8e1d30 msgr2=0x55f5ec8e6830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:34.495 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.494+0000 7f65f7400640 1 --2- 172.21.15.19:0/462878193 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f65dc0a2880 0x7f65dc0a4d40 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x55f5ec8e2590 tx=0x7f65e4004730 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:17:34.495 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.494+0000 7f66017af740 1 --2- 172.21.15.19:0/301394022 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f5ec8e1d30 0x55f5ec8e6830 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f65e800bcb0 tx=0x7f65e80107a0 comp rx=0 tx=0).stop 2024-03-18T15:17:34.495 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.494+0000 7f66017af740 1 -- 172.21.15.19:0/301394022 shutdown_connections 2024-03-18T15:17:34.495 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.494+0000 7f66017af740 1 --2- 172.21.15.19:0/301394022 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55f5ec8e6ea0 0x55f5ec8e9330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:34.495 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.494+0000 7f66017af740 1 --2- 172.21.15.19:0/301394022 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f5ec8e1d30 0x55f5ec8e6830 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:34.495 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.494+0000 7f66017af740 1 --2- 172.21.15.19:0/301394022 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f5ec8e1340 0x55f5ec8e1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:34.496 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.494+0000 7f66017af740 1 -- 172.21.15.19:0/301394022 >> 172.21.15.19:0/301394022 conn(0x55f5ec8dc8b0 msgr2=0x55f5ec8dccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:17:34.496 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.494+0000 7f66017af740 1 -- 172.21.15.19:0/301394022 shutdown_connections 2024-03-18T15:17:34.496 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.494+0000 7f66017af740 1 -- 172.21.15.19:0/301394022 wait complete. 2024-03-18T15:17:34.496 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.494+0000 7f66017af740 1 -- start start 2024-03-18T15:17:34.496 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.494+0000 7f66017af740 1 Processor -- start 2024-03-18T15:17:34.499 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.495+0000 7f66017af740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f5ec8e1340 0x55f5ec8eb490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:34.499 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.495+0000 7f66017af740 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55f5ec8e1d30 0x55f5ec8eb9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:34.500 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.495+0000 7f66017af740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f5ec8e6ea0 0x55f5ec8f2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:34.500 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.495+0000 7f66017af740 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x55f5ec8e9e50 con 0x55f5ec8e1d30 2024-03-18T15:17:34.500 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.495+0000 7f66017af740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55f5ec801e20 con 0x55f5ec8e1340 2024-03-18T15:17:34.500 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.495+0000 7f66017af740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55f5ec8e9fd0 con 0x55f5ec8e6ea0 2024-03-18T15:17:34.500 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.495+0000 7f66017af740 10 client.0 _ll_register_callbacks cb 0x55f5ec8017f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T15:17:34.500 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.495+0000 7f65f7e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f5ec8e1340 0x55f5ec8eb490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:34.500 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.495+0000 7f65fca00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f5ec8e6ea0 0x55f5ec8f2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:34.500 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.495+0000 7f65f7e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f5ec8e1340 0x55f5ec8eb490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:39440/0 (socket says 172.21.15.19:39440) 2024-03-18T15:17:34.500 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.495+0000 7f65fca00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f5ec8e6ea0 0x55f5ec8f2a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.19:58396/0 (socket says 172.21.15.19:58396) 2024-03-18T15:17:34.500 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.495+0000 7f65f7e00640 1 -- 172.21.15.19:0/741658414 learned_addr learned my addr 172.21.15.19:0/741658414 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:17:34.500 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.495+0000 7f65f7400640 1 --2- 172.21.15.19:0/741658414 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55f5ec8e1d30 0x55f5ec8eb9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:34.500 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.496+0000 7f65fca00640 1 -- 172.21.15.19:0/741658414 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f5ec8e1340 msgr2=0x55f5ec8eb490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:17:34.500 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.496+0000 7f65fca00640 1 --2- 172.21.15.19:0/741658414 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f5ec8e1340 0x55f5ec8eb490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:34.501 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.496+0000 7f65fca00640 1 -- 172.21.15.19:0/741658414 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55f5ec8e1d30 msgr2=0x55f5ec8eb9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:17:34.501 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.496+0000 7f65fca00640 1 --2- 172.21.15.19:0/741658414 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55f5ec8e1d30 0x55f5ec8eb9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:34.501 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.496+0000 7f65fca00640 1 -- 172.21.15.19:0/741658414 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f5ec55a2d0 con 0x55f5ec8e6ea0 2024-03-18T15:17:34.501 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.496+0000 7f65f7400640 1 --2- 172.21.15.19:0/741658414 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55f5ec8e1d30 0x55f5ec8eb9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T15:17:34.501 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.496+0000 7f65f7e00640 1 --2- 172.21.15.19:0/741658414 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f5ec8e1340 0x55f5ec8eb490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:17:34.501 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.496+0000 7f65fca00640 1 --2- 172.21.15.19:0/741658414 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f5ec8e6ea0 0x55f5ec8f2a10 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f65f00bd6d0 tx=0x7f65f00bd330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8f6a1cffc1857f01 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:17:34.501 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.496+0000 7f65e1600640 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-03-18T15:17:34.501 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.496+0000 7f65e1600640 1 -- 172.21.15.19:0/741658414 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65f00bd9d0 con 0x55f5ec8e6ea0 2024-03-18T15:17:34.501 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.496+0000 7f65e1600640 1 -- 172.21.15.19:0/741658414 <== mon.1 v2:172.21.15.104:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f65f0032890 con 0x55f5ec8e6ea0 2024-03-18T15:17:34.501 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.496+0000 7f66017af740 1 -- 172.21.15.19:0/741658414 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f65e8005500 con 0x55f5ec8e6ea0 2024-03-18T15:17:34.501 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.496+0000 7f65e1600640 1 -- 172.21.15.19:0/741658414 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f65f00c1b20 con 0x55f5ec8e6ea0 2024-03-18T15:17:34.501 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.496+0000 7f66017af740 10 client.25201 fetch_fsmap learned FSMap version 26 2024-03-18T15:17:34.501 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.496+0000 7f66017af740 1 -- 172.21.15.19:0/741658414 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55f5ec6246b0 con 0x55f5ec8e6ea0 2024-03-18T15:17:34.501 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.497+0000 7f65e1600640 1 -- 172.21.15.19:0/741658414 <== mon.1 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f65e8005500 con 0x55f5ec8e6ea0 2024-03-18T15:17:34.502 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.497+0000 7f65e1600640 1 -- 172.21.15.19:0/741658414 <== mon.1 v2:172.21.15.104:3300/0 5 ==== fsmap(e 26) ==== 8362+0+0 (secure 0 0 0) 0x7f65f003c030 con 0x55f5ec8e6ea0 2024-03-18T15:17:34.502 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.497+0000 7f66017af740 10 client.25201 fetch_fsmap finished waiting for FSMap version 26 2024-03-18T15:17:34.502 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.497+0000 7f66017af740 10 client.25201 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-18T15:17:34.502 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.497+0000 7f66017af740 20 client.25201 populate_metadata read hostname 'smithi019' 2024-03-18T15:17:34.502 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.497+0000 7f66017af740 1 --2- 172.21.15.19:0/741658414 >> [v2:172.21.15.104:6838/805341436,v1:172.21.15.104:6839/805341436] conn(0x55f5ec8f8290 0x55f5ec8f9700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:34.502 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.497+0000 7f66017af740 4 client.25201 mds_command: new command op to 24529 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T15:17:34.502 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.497+0000 7f66017af740 1 -- 172.21.15.19:0/741658414 --> [v2:172.21.15.104:6838/805341436,v1:172.21.15.104:6839/805341436] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f5ec8a32b0 con 0x55f5ec8f8290 2024-03-18T15:17:34.502 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.497+0000 7f65f7e00640 1 --2- 172.21.15.19:0/741658414 >> [v2:172.21.15.104:6838/805341436,v1:172.21.15.104:6839/805341436] conn(0x55f5ec8f8290 0x55f5ec8f9700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:34.502 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.498+0000 7f65f7e00640 1 --2- 172.21.15.19:0/741658414 >> [v2:172.21.15.104:6838/805341436,v1:172.21.15.104:6839/805341436] conn(0x55f5ec8f8290 0x55f5ec8f9700 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2c30954ca271fe3d server_cookie=e3990ffd9d71135f in_seq=0 out_seq=0 2024-03-18T15:17:34.502 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.498+0000 7f65e1600640 10 client.25201 ms_handle_connect on v2:172.21.15.104:6838/805341436 2024-03-18T15:17:34.502 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.499+0000 7f65e1600640 1 -- 172.21.15.19:0/741658414 <== mds.1 v2:172.21.15.104:6838/805341436 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f5ec8a32b0 con 0x55f5ec8f8290 2024-03-18T15:17:34.502 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.499+0000 7f65e1600640 10 client.25201 handle_command_reply: tid=0 2024-03-18T15:17:34.502 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.499+0000 7f66017af740 1 client.25201 shutdown 2024-03-18T15:17:34.502 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.499+0000 7f66017af740 1 -- 172.21.15.19:0/741658414 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f5ec8e6ea0 msgr2=0x55f5ec8f2a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:34.503 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.499+0000 7f66017af740 1 --2- 172.21.15.19:0/741658414 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f5ec8e6ea0 0x55f5ec8f2a10 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f65f00bd6d0 tx=0x7f65f00bd330 comp rx=0 tx=0).stop 2024-03-18T15:17:34.503 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.499+0000 7f66017af740 1 -- 172.21.15.19:0/741658414 shutdown_connections 2024-03-18T15:17:34.503 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.499+0000 7f66017af740 1 --2- 172.21.15.19:0/741658414 >> [v2:172.21.15.104:6838/805341436,v1:172.21.15.104:6839/805341436] conn(0x55f5ec8f8290 0x55f5ec8f9700 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:34.503 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.499+0000 7f66017af740 1 --2- 172.21.15.19:0/741658414 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f5ec8e6ea0 0x55f5ec8f2a10 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:34.503 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.499+0000 7f66017af740 1 --2- 172.21.15.19:0/741658414 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55f5ec8e1d30 0x55f5ec8eb9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:34.503 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.499+0000 7f65e1600640 0 client.25201 ms_handle_reset on v2:172.21.15.104:6838/805341436 2024-03-18T15:17:34.503 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.499+0000 7f66017af740 1 --2- 172.21.15.19:0/741658414 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f5ec8e1340 0x55f5ec8eb490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:34.503 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.499+0000 7f66017af740 1 -- 172.21.15.19:0/741658414 >> 172.21.15.19:0/741658414 conn(0x55f5ec8dc8b0 msgr2=0x55f5ec8dba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:17:34.503 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.499+0000 7f66017af740 1 -- 172.21.15.19:0/741658414 shutdown_connections 2024-03-18T15:17:34.503 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.500+0000 7f66017af740 1 -- 172.21.15.19:0/741658414 wait complete. 2024-03-18T15:17:34.503 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.500+0000 7f66017af740 20 client.25201 trim_cache size 0 max 16384 2024-03-18T15:17:34.519 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.519+0000 7f66017af740 1 Processor -- start 2024-03-18T15:17:34.519 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.520+0000 7f66017af740 1 -- start start 2024-03-18T15:17:34.520 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.520+0000 7f66017af740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f5ec8e1340 0x55f5ec8eb490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:34.520 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.520+0000 7f66017af740 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55f5ec8e1d30 0x55f5ec8eb9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:34.520 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.520+0000 7f66017af740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f5ec8e6ea0 0x55f5ec8f29d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:34.520 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.520+0000 7f66017af740 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x55f5ec8f7a40 con 0x55f5ec8e1d30 2024-03-18T15:17:34.520 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.520+0000 7f66017af740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55f5ec8f8110 con 0x55f5ec8e1340 2024-03-18T15:17:34.520 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.520+0000 7f66017af740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55f5ec8f87e0 con 0x55f5ec8e6ea0 2024-03-18T15:17:34.520 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.520+0000 7f65f7400640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55f5ec8e1d30 0x55f5ec8eb9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:34.521 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.520+0000 7f65f7400640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55f5ec8e1d30 0x55f5ec8eb9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:50630/0 (socket says 172.21.15.19:50630) 2024-03-18T15:17:34.521 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.520+0000 7f65f7400640 1 -- 172.21.15.19:0/3440937478 learned_addr learned my addr 172.21.15.19:0/3440937478 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:17:34.521 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.520+0000 7f65fca00640 1 --2- 172.21.15.19:0/3440937478 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f5ec8e6ea0 0x55f5ec8f29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:34.521 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.520+0000 7f65f7e00640 1 --2- 172.21.15.19:0/3440937478 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f5ec8e1340 0x55f5ec8eb490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:34.522 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.521+0000 7f65f7400640 1 -- 172.21.15.19:0/3440937478 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f5ec8e6ea0 msgr2=0x55f5ec8f29d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:17:34.523 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.521+0000 7f65f7400640 1 --2- 172.21.15.19:0/3440937478 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f5ec8e6ea0 0x55f5ec8f29d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:34.523 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.521+0000 7f65f7400640 1 -- 172.21.15.19:0/3440937478 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f5ec8e1340 msgr2=0x55f5ec8eb490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:17:34.523 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.521+0000 7f65f7400640 1 --2- 172.21.15.19:0/3440937478 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f5ec8e1340 0x55f5ec8eb490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:34.523 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.521+0000 7f65f7400640 1 -- 172.21.15.19:0/3440937478 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f5ec8d8340 con 0x55f5ec8e1d30 2024-03-18T15:17:34.523 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.521+0000 7f65fca00640 1 --2- 172.21.15.19:0/3440937478 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f5ec8e6ea0 0x55f5ec8f29d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:17:34.523 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.521+0000 7f65f7e00640 1 --2- 172.21.15.19:0/3440937478 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f5ec8e1340 0x55f5ec8eb490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:17:34.523 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.521+0000 7f65f7400640 1 --2- 172.21.15.19:0/3440937478 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55f5ec8e1d30 0x55f5ec8eb9d0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f65e4003f30 tx=0x7f65e4016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31d87cdb3078debb server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:17:34.523 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.521+0000 7f65e3e00640 1 -- 172.21.15.19:0/3440937478 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65e4011bb0 con 0x55f5ec8e1d30 2024-03-18T15:17:34.523 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.521+0000 7f65e3e00640 1 -- 172.21.15.19:0/3440937478 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f65e4011d50 con 0x55f5ec8e1d30 2024-03-18T15:17:34.523 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.521+0000 7f65e3e00640 1 -- 172.21.15.19:0/3440937478 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65e4020410 con 0x55f5ec8e1d30 2024-03-18T15:17:34.523 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.522+0000 7f66017af740 1 -- 172.21.15.19:0/3440937478 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55f5ec8e1d30 msgr2=0x55f5ec8eb9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:34.523 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.522+0000 7f66017af740 1 --2- 172.21.15.19:0/3440937478 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55f5ec8e1d30 0x55f5ec8eb9d0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f65e4003f30 tx=0x7f65e4016000 comp rx=0 tx=0).stop 2024-03-18T15:17:34.523 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.522+0000 7f66017af740 1 -- 172.21.15.19:0/3440937478 shutdown_connections 2024-03-18T15:17:34.523 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.522+0000 7f66017af740 1 --2- 172.21.15.19:0/3440937478 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f5ec8e6ea0 0x55f5ec8f29d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:34.524 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.522+0000 7f66017af740 1 --2- 172.21.15.19:0/3440937478 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55f5ec8e1d30 0x55f5ec8eb9d0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:34.524 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.522+0000 7f66017af740 1 --2- 172.21.15.19:0/3440937478 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f5ec8e1340 0x55f5ec8eb490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:34.524 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.522+0000 7f66017af740 1 -- 172.21.15.19:0/3440937478 >> 172.21.15.19:0/3440937478 conn(0x55f5ec8dc8b0 msgr2=0x55f5ec8d7900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:17:34.524 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.522+0000 7f66017af740 1 -- 172.21.15.19:0/3440937478 shutdown_connections 2024-03-18T15:17:34.524 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.522+0000 7f66017af740 1 -- 172.21.15.19:0/3440937478 wait complete. 2024-03-18T15:17:34.524 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.523+0000 7f66017af740 1 -- start start 2024-03-18T15:17:34.524 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.523+0000 7f66017af740 1 Processor -- start 2024-03-18T15:17:34.527 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.523+0000 7f66017af740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f5ec8e1340 0x55f5ec8edef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:34.527 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.523+0000 7f66017af740 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55f5ec8e1d30 0x55f5ec8ee430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:34.527 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.523+0000 7f66017af740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f5ec8e6ea0 0x55f5ec8f1450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:34.527 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.523+0000 7f66017af740 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x55f5ec8f9580 con 0x55f5ec8e1d30 2024-03-18T15:17:34.527 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.523+0000 7f66017af740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55f5ec8f8eb0 con 0x55f5ec8e1340 2024-03-18T15:17:34.527 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.523+0000 7f66017af740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55f5ec8f9c50 con 0x55f5ec8e6ea0 2024-03-18T15:17:34.527 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.523+0000 7f66017af740 10 client.0 _ll_register_callbacks cb 0x7f65f00b9c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T15:17:34.527 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.524+0000 7f65f7e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f5ec8e1340 0x55f5ec8edef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:34.528 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.524+0000 7f65f7400640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55f5ec8e1d30 0x55f5ec8ee430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:34.528 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.524+0000 7f65f7e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f5ec8e1340 0x55f5ec8edef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:39460/0 (socket says 172.21.15.19:39460) 2024-03-18T15:17:34.528 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.524+0000 7f65f7e00640 1 -- 172.21.15.19:0/442262260 learned_addr learned my addr 172.21.15.19:0/442262260 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:17:34.528 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.524+0000 7f65fca00640 1 --2- 172.21.15.19:0/442262260 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f5ec8e6ea0 0x55f5ec8f1450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:34.528 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.524+0000 7f65f7e00640 1 -- 172.21.15.19:0/442262260 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f5ec8e6ea0 msgr2=0x55f5ec8f1450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:17:34.528 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.524+0000 7f65f7e00640 1 --2- 172.21.15.19:0/442262260 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f5ec8e6ea0 0x55f5ec8f1450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:34.528 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.524+0000 7f65f7e00640 1 -- 172.21.15.19:0/442262260 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55f5ec8e1d30 msgr2=0x55f5ec8ee430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:17:34.528 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.524+0000 7f65f7e00640 1 --2- 172.21.15.19:0/442262260 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55f5ec8e1d30 0x55f5ec8ee430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:34.528 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.524+0000 7f65f7e00640 1 -- 172.21.15.19:0/442262260 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f5ec8f1a40 con 0x55f5ec8e1340 2024-03-18T15:17:34.528 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.524+0000 7f65f7e00640 1 --2- 172.21.15.19:0/442262260 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f5ec8e1340 0x55f5ec8edef0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f65e8010ca0 tx=0x7f65e800b5b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1bb566a455f63720 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:17:34.528 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.524+0000 7f65e1600640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-18T15:17:34.528 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.524+0000 7f65e1600640 1 -- 172.21.15.19:0/442262260 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65e8005230 con 0x55f5ec8e1340 2024-03-18T15:17:34.528 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.525+0000 7f65e1600640 1 -- 172.21.15.19:0/442262260 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f65e80053d0 con 0x55f5ec8e1340 2024-03-18T15:17:34.529 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.525+0000 7f66017af740 1 -- 172.21.15.19:0/442262260 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f65e4011d50 con 0x55f5ec8e1340 2024-03-18T15:17:34.529 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.525+0000 7f65e1600640 1 -- 172.21.15.19:0/442262260 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f65e8038050 con 0x55f5ec8e1340 2024-03-18T15:17:34.529 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.525+0000 7f66017af740 10 client.25196 fetch_fsmap learned FSMap version 26 2024-03-18T15:17:34.529 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.525+0000 7f66017af740 1 -- 172.21.15.19:0/442262260 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55f5ec8f2d70 con 0x55f5ec8e1340 2024-03-18T15:17:34.529 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.525+0000 7f65e1600640 1 -- 172.21.15.19:0/442262260 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f65e4011d50 con 0x55f5ec8e1340 2024-03-18T15:17:34.529 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.525+0000 7f65e1600640 1 -- 172.21.15.19:0/442262260 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 26) ==== 8362+0+0 (secure 0 0 0) 0x7f65e8033030 con 0x55f5ec8e1340 2024-03-18T15:17:34.529 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.525+0000 7f66017af740 10 client.25196 fetch_fsmap finished waiting for FSMap version 26 2024-03-18T15:17:34.529 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.525+0000 7f66017af740 10 client.25196 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-18T15:17:34.529 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.525+0000 7f66017af740 20 client.25196 populate_metadata read hostname 'smithi019' 2024-03-18T15:17:34.529 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.525+0000 7f66017af740 1 --2- 172.21.15.19:0/442262260 >> [v2:172.21.15.104:6838/805341436,v1:172.21.15.104:6839/805341436] conn(0x55f5ec962600 0x55f5ec961310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:34.529 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.525+0000 7f66017af740 4 client.25196 mds_command: new command op to 24529 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-18T15:17:34.529 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.525+0000 7f66017af740 1 -- 172.21.15.19:0/442262260 --> [v2:172.21.15.104:6838/805341436,v1:172.21.15.104:6839/805341436] -- command(tid 0: {"prefix": "status"}) -- 0x55f5ec8e97b0 con 0x55f5ec962600 2024-03-18T15:17:34.529 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.526+0000 7f65f7400640 1 --2- 172.21.15.19:0/442262260 >> [v2:172.21.15.104:6838/805341436,v1:172.21.15.104:6839/805341436] conn(0x55f5ec962600 0x55f5ec961310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:34.529 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.526+0000 7f65f7400640 1 --2- 172.21.15.19:0/442262260 >> [v2:172.21.15.104:6838/805341436,v1:172.21.15.104:6839/805341436] conn(0x55f5ec962600 0x55f5ec961310 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1386ea882f03b8fe server_cookie=819efab6d2422cc1 in_seq=0 out_seq=0 2024-03-18T15:17:34.529 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.526+0000 7f65e1600640 10 client.25196 ms_handle_connect on v2:172.21.15.104:6838/805341436 2024-03-18T15:17:34.530 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.526+0000 7f65e1600640 1 -- 172.21.15.19:0/442262260 <== mds.1 v2:172.21.15.104:6838/805341436 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55f5ec8e97b0 con 0x55f5ec962600 2024-03-18T15:17:34.530 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.526+0000 7f65e1600640 10 client.25196 handle_command_reply: tid=0 2024-03-18T15:17:34.530 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.526+0000 7f66017af740 1 client.25196 shutdown 2024-03-18T15:17:34.530 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.527+0000 7f66017af740 1 -- 172.21.15.19:0/442262260 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f5ec8e1340 msgr2=0x55f5ec8edef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:34.530 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.527+0000 7f66017af740 1 --2- 172.21.15.19:0/442262260 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f5ec8e1340 0x55f5ec8edef0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f65e8010ca0 tx=0x7f65e800b5b0 comp rx=0 tx=0).stop 2024-03-18T15:17:34.530 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.527+0000 7f66017af740 1 -- 172.21.15.19:0/442262260 shutdown_connections 2024-03-18T15:17:34.530 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.527+0000 7f66017af740 1 --2- 172.21.15.19:0/442262260 >> [v2:172.21.15.104:6838/805341436,v1:172.21.15.104:6839/805341436] conn(0x55f5ec962600 0x55f5ec961310 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:34.530 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.527+0000 7f66017af740 1 --2- 172.21.15.19:0/442262260 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f5ec8e6ea0 0x55f5ec8f1450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:34.530 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.527+0000 7f66017af740 1 --2- 172.21.15.19:0/442262260 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55f5ec8e1d30 0x55f5ec8ee430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:34.530 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.527+0000 7f66017af740 1 --2- 172.21.15.19:0/442262260 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f5ec8e1340 0x55f5ec8edef0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:34.530 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.527+0000 7f66017af740 1 -- 172.21.15.19:0/442262260 >> 172.21.15.19:0/442262260 conn(0x55f5ec8dc8b0 msgr2=0x55f5ec8eb490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:17:34.530 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.527+0000 7f65e1600640 0 client.25196 ms_handle_reset on v2:172.21.15.104:6838/805341436 2024-03-18T15:17:34.530 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.527+0000 7f66017af740 1 -- 172.21.15.19:0/442262260 shutdown_connections 2024-03-18T15:17:34.531 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.527+0000 7f66017af740 1 -- 172.21.15.19:0/442262260 wait complete. 2024-03-18T15:17:34.531 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.527+0000 7f66017af740 20 client.25196 trim_cache size 0 max 16384 2024-03-18T15:17:34.531 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-03-18T15:17:34.531 INFO:teuthology.orchestra.run.smithi019.stdout: "cluster_fsid": "d4dc2ba6-e538-11ee-95c9-87774f69a715", 2024-03-18T15:17:34.531 INFO:teuthology.orchestra.run.smithi019.stdout: "whoami": 1, 2024-03-18T15:17:34.531 INFO:teuthology.orchestra.run.smithi019.stdout: "id": 24529, 2024-03-18T15:17:34.531 INFO:teuthology.orchestra.run.smithi019.stdout: "want_state": "up:active", 2024-03-18T15:17:34.531 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "up:active", 2024-03-18T15:17:34.531 INFO:teuthology.orchestra.run.smithi019.stdout: "fs_name": "cephfs", 2024-03-18T15:17:34.531 INFO:teuthology.orchestra.run.smithi019.stdout: "rank_uptime": 111.068010853, 2024-03-18T15:17:34.531 INFO:teuthology.orchestra.run.smithi019.stdout: "mdsmap_epoch": 26, 2024-03-18T15:17:34.532 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap_epoch": 93, 2024-03-18T15:17:34.532 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap_epoch_barrier": 89, 2024-03-18T15:17:34.532 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 161.58640116699999 2024-03-18T15:17:34.532 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-03-18T15:17:34.532 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.528+0000 7f65fd400640 1 -- 172.21.15.19:0/462878193 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f65dc0a2880 msgr2=0x7f65dc0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:34.533 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.528+0000 7f65fd400640 1 --2- 172.21.15.19:0/462878193 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f65dc0a2880 0x7f65dc0a4d40 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x55f5ec8e2590 tx=0x7f65e4004730 comp rx=0 tx=0).stop 2024-03-18T15:17:34.533 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.528+0000 7f65fd400640 1 -- 172.21.15.19:0/462878193 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f65f81098b0 msgr2=0x7f65f81a54f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:34.533 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.528+0000 7f65fd400640 1 --2- 172.21.15.19:0/462878193 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f65f81098b0 0x7f65f81a54f0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f65f0037040 tx=0x7f65f0035d90 comp rx=0 tx=0).stop 2024-03-18T15:17:34.533 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.528+0000 7f65fd400640 1 -- 172.21.15.19:0/462878193 shutdown_connections 2024-03-18T15:17:34.533 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.528+0000 7f65fd400640 1 --2- 172.21.15.19:0/462878193 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f65dc0a2880 0x7f65dc0a4d40 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:34.533 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.528+0000 7f65fd400640 1 --2- 172.21.15.19:0/462878193 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f65f81098b0 0x7f65f81a54f0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:34.533 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.528+0000 7f65fd400640 1 --2- 172.21.15.19:0/462878193 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f65f8108ef0 0x7f65f819e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:34.533 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.528+0000 7f65fd400640 1 --2- 172.21.15.19:0/462878193 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f65f8107d00 0x7f65f819df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:34.533 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.528+0000 7f65fd400640 1 -- 172.21.15.19:0/462878193 >> 172.21.15.19:0/462878193 conn(0x7f65f8076cb0 msgr2=0x7f65f810c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:17:34.534 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.528+0000 7f65fd400640 1 -- 172.21.15.19:0/462878193 shutdown_connections 2024-03-18T15:17:34.534 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.528+0000 7f65fd400640 1 -- 172.21.15.19:0/462878193 wait complete. 2024-03-18T15:17:34.547 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-18T15:17:34.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.777+0000 7f9667c00640 1 -- 172.21.15.19:0/868561635 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9660104cd0 msgr2=0x7f96601070c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:34.782 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.777+0000 7f9667c00640 1 --2- 172.21.15.19:0/868561635 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9660104cd0 0x7f96601070c0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f964c009900 tx=0x7f964c02bc30 comp rx=0 tx=0).stop 2024-03-18T15:17:34.782 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.777+0000 7f9667c00640 1 -- 172.21.15.19:0/868561635 shutdown_connections 2024-03-18T15:17:34.782 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.777+0000 7f9667c00640 1 --2- 172.21.15.19:0/868561635 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9660107600 0x7f96601099f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:34.782 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.777+0000 7f9667c00640 1 --2- 172.21.15.19:0/868561635 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9660104cd0 0x7f96601070c0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:34.782 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.777+0000 7f9667c00640 1 --2- 172.21.15.19:0/868561635 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f96601023a0 0x7f9660104790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:34.783 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.777+0000 7f9667c00640 1 -- 172.21.15.19:0/868561635 >> 172.21.15.19:0/868561635 conn(0x7f96600fbd10 msgr2=0x7f96600fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:17:34.783 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.777+0000 7f9667c00640 1 -- 172.21.15.19:0/868561635 shutdown_connections 2024-03-18T15:17:34.783 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.777+0000 7f9667c00640 1 -- 172.21.15.19:0/868561635 wait complete. 2024-03-18T15:17:34.783 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.778+0000 7f9667c00640 1 Processor -- start 2024-03-18T15:17:34.783 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.779+0000 7f9667c00640 1 -- start start 2024-03-18T15:17:34.783 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.779+0000 7f9667c00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f96601023a0 0x7f9660199d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:34.784 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.779+0000 7f9667c00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9660104cd0 0x7f966019a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:34.784 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.779+0000 7f9667c00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9660107600 0x7f96601a1320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:34.784 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.779+0000 7f9667c00640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f966010d480 con 0x7f9660107600 2024-03-18T15:17:34.784 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.779+0000 7f9667c00640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f966010d300 con 0x7f96601023a0 2024-03-18T15:17:34.784 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.779+0000 7f9667c00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f966010d600 con 0x7f9660104cd0 2024-03-18T15:17:34.784 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.779+0000 7f9665200640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f96601023a0 0x7f9660199d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:34.784 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.779+0000 7f9665200640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f96601023a0 0x7f9660199d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:39468/0 (socket says 172.21.15.19:39468) 2024-03-18T15:17:34.784 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.779+0000 7f9665200640 1 -- 172.21.15.19:0/3164262390 learned_addr learned my addr 172.21.15.19:0/3164262390 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:17:34.784 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.779+0000 7f965fe00640 1 --2- 172.21.15.19:0/3164262390 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9660104cd0 0x7f966019a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:34.785 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.779+0000 7f9665c00640 1 --2- 172.21.15.19:0/3164262390 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9660107600 0x7f96601a1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:34.785 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.780+0000 7f9665200640 1 -- 172.21.15.19:0/3164262390 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9660104cd0 msgr2=0x7f966019a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:34.785 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.780+0000 7f9665200640 1 --2- 172.21.15.19:0/3164262390 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9660104cd0 0x7f966019a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:34.785 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.780+0000 7f9665200640 1 -- 172.21.15.19:0/3164262390 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9660107600 msgr2=0x7f96601a1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:34.785 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.780+0000 7f9665200640 1 --2- 172.21.15.19:0/3164262390 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9660107600 0x7f96601a1320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:34.786 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.780+0000 7f9665200640 1 -- 172.21.15.19:0/3164262390 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96601a1990 con 0x7f96601023a0 2024-03-18T15:17:34.786 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.780+0000 7f9665200640 1 --2- 172.21.15.19:0/3164262390 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f96601023a0 0x7f9660199d70 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f965000eab0 tx=0x7f965000ef70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:17:34.786 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.780+0000 7f965d600640 1 -- 172.21.15.19:0/3164262390 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f965000ce40 con 0x7f96601023a0 2024-03-18T15:17:34.786 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.780+0000 7f965d600640 1 -- 172.21.15.19:0/3164262390 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9650011950 con 0x7f96601023a0 2024-03-18T15:17:34.786 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.780+0000 7f965d600640 1 -- 172.21.15.19:0/3164262390 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9650010ae0 con 0x7f96601023a0 2024-03-18T15:17:34.786 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.780+0000 7f9667c00640 1 -- 172.21.15.19:0/3164262390 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96601a1c80 con 0x7f96601023a0 2024-03-18T15:17:34.786 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.781+0000 7f9667c00640 1 -- 172.21.15.19:0/3164262390 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96601a2150 con 0x7f96601023a0 2024-03-18T15:17:34.793 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.784+0000 7f965d600640 1 -- 172.21.15.19:0/3164262390 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f9650011b00 con 0x7f96601023a0 2024-03-18T15:17:34.793 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.784+0000 7f966a9b7740 1 Processor -- start 2024-03-18T15:17:34.793 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.784+0000 7f966a9b7740 1 -- start start 2024-03-18T15:17:34.793 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.784+0000 7f966a9b7740 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55b359458160 0x55b359458580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:34.794 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.784+0000 7f966a9b7740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b359458b50 0x55b35945d650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:34.794 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.784+0000 7f966a9b7740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b35945dcc0 0x55b359460150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:34.794 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.784+0000 7f966a9b7740 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x55b3593bcc50 con 0x55b359458160 2024-03-18T15:17:34.794 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.784+0000 7f966a9b7740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55b3593659d0 con 0x55b35945dcc0 2024-03-18T15:17:34.794 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.784+0000 7f966a9b7740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55b3593b8cf0 con 0x55b359458b50 2024-03-18T15:17:34.794 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.785+0000 7f9665200640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b359458b50 0x55b35945d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:34.794 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.785+0000 7f965fe00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b35945dcc0 0x55b359460150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:34.794 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.785+0000 7f9665200640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b359458b50 0x55b35945d650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.19:58434/0 (socket says 172.21.15.19:58434) 2024-03-18T15:17:34.795 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.785+0000 7f9665200640 1 -- 172.21.15.19:0/3332790321 learned_addr learned my addr 172.21.15.19:0/3332790321 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:17:34.795 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.785+0000 7f9665c00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55b359458160 0x55b359458580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:34.795 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.785+0000 7f9665200640 1 -- 172.21.15.19:0/3332790321 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b35945dcc0 msgr2=0x55b359460150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:17:34.795 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.785+0000 7f9665200640 1 --2- 172.21.15.19:0/3332790321 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b35945dcc0 0x55b359460150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:34.795 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.785+0000 7f9665200640 1 -- 172.21.15.19:0/3332790321 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55b359458160 msgr2=0x55b359458580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:17:34.795 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.785+0000 7f9665200640 1 --2- 172.21.15.19:0/3332790321 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55b359458160 0x55b359458580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:34.795 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.785+0000 7f9665200640 1 -- 172.21.15.19:0/3332790321 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b35937ff10 con 0x55b359458b50 2024-03-18T15:17:34.795 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.785+0000 7f9665c00640 1 --2- 172.21.15.19:0/3332790321 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55b359458160 0x55b359458580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T15:17:34.795 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.785+0000 7f9665200640 1 --2- 172.21.15.19:0/3332790321 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b359458b50 0x55b35945d650 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f96500d7580 tx=0x7f96500d7670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=612d576a4ad82e90 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:17:34.795 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.786+0000 7f9656000640 1 -- 172.21.15.19:0/3332790321 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96500d1520 con 0x55b359458b50 2024-03-18T15:17:34.796 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.786+0000 7f9656000640 1 -- 172.21.15.19:0/3332790321 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f96500cfcf0 con 0x55b359458b50 2024-03-18T15:17:34.796 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.786+0000 7f9656000640 1 -- 172.21.15.19:0/3332790321 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96500decc0 con 0x55b359458b50 2024-03-18T15:17:34.796 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.786+0000 7f965d600640 1 --2- 172.21.15.19:0/3164262390 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f96440a2880 0x7f96440a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:34.796 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.786+0000 7f965d600640 1 -- 172.21.15.19:0/3164262390 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f96500cf420 con 0x7f96601023a0 2024-03-18T15:17:34.796 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.786+0000 7f9665c00640 1 --2- 172.21.15.19:0/3164262390 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f96440a2880 0x7f96440a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:34.796 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.786+0000 7f966a9b7740 1 -- 172.21.15.19:0/3332790321 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b359458b50 msgr2=0x55b35945d650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:34.796 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.786+0000 7f966a9b7740 1 --2- 172.21.15.19:0/3332790321 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b359458b50 0x55b35945d650 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f96500d7580 tx=0x7f96500d7670 comp rx=0 tx=0).stop 2024-03-18T15:17:34.796 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.787+0000 7f966a9b7740 1 -- 172.21.15.19:0/3332790321 shutdown_connections 2024-03-18T15:17:34.796 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.787+0000 7f966a9b7740 1 --2- 172.21.15.19:0/3332790321 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b35945dcc0 0x55b359460150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:34.796 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.787+0000 7f966a9b7740 1 --2- 172.21.15.19:0/3332790321 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b359458b50 0x55b35945d650 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:34.797 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.787+0000 7f966a9b7740 1 --2- 172.21.15.19:0/3332790321 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55b359458160 0x55b359458580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:34.797 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.787+0000 7f966a9b7740 1 -- 172.21.15.19:0/3332790321 >> 172.21.15.19:0/3332790321 conn(0x55b3594536d0 msgr2=0x55b359453ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:17:34.797 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.787+0000 7f966a9b7740 1 -- 172.21.15.19:0/3332790321 shutdown_connections 2024-03-18T15:17:34.797 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.787+0000 7f966a9b7740 1 -- 172.21.15.19:0/3332790321 wait complete. 2024-03-18T15:17:34.797 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.787+0000 7f9665c00640 1 --2- 172.21.15.19:0/3164262390 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f96440a2880 0x7f96440a4d40 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x55b3594593b0 tx=0x7f9658005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:17:34.797 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.787+0000 7f966a9b7740 1 -- start start 2024-03-18T15:17:34.797 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.787+0000 7f966a9b7740 1 Processor -- start 2024-03-18T15:17:34.797 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.788+0000 7f966a9b7740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b359458160 0x55b3594622b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:34.797 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.788+0000 7f966a9b7740 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55b359458b50 0x55b3594627f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:34.797 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.788+0000 7f966a9b7740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b35945dcc0 0x55b359469830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:34.797 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.788+0000 7f966a9b7740 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x55b359460c70 con 0x55b359458b50 2024-03-18T15:17:34.798 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.788+0000 7f966a9b7740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55b359378e20 con 0x55b359458160 2024-03-18T15:17:34.798 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.788+0000 7f966a9b7740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55b359460df0 con 0x55b35945dcc0 2024-03-18T15:17:34.798 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.788+0000 7f966a9b7740 10 client.0 _ll_register_callbacks cb 0x55b3593787f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T15:17:34.798 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.788+0000 7f9665c00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b35945dcc0 0x55b359469830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:34.798 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.788+0000 7f9665200640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b359458160 0x55b3594622b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:34.798 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.788+0000 7f9665200640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b359458160 0x55b3594622b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:39484/0 (socket says 172.21.15.19:39484) 2024-03-18T15:17:34.798 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.788+0000 7f9665c00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b35945dcc0 0x55b359469830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.19:58444/0 (socket says 172.21.15.19:58444) 2024-03-18T15:17:34.798 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.788+0000 7f9665200640 1 -- 172.21.15.19:0/4286451827 learned_addr learned my addr 172.21.15.19:0/4286451827 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:17:34.798 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.788+0000 7f965fe00640 1 --2- 172.21.15.19:0/4286451827 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55b359458b50 0x55b3594627f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:34.798 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.789+0000 7f9665200640 1 -- 172.21.15.19:0/4286451827 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b35945dcc0 msgr2=0x55b359469830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:17:34.798 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.789+0000 7f9665200640 1 --2- 172.21.15.19:0/4286451827 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b35945dcc0 0x55b359469830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:34.799 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.789+0000 7f9665200640 1 -- 172.21.15.19:0/4286451827 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55b359458b50 msgr2=0x55b3594627f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:17:34.799 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.789+0000 7f9665200640 1 --2- 172.21.15.19:0/4286451827 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55b359458b50 0x55b3594627f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:34.799 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.789+0000 7f9665200640 1 -- 172.21.15.19:0/4286451827 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b3590d12d0 con 0x55b359458160 2024-03-18T15:17:34.799 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.789+0000 7f9665200640 1 --2- 172.21.15.19:0/4286451827 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b359458160 0x55b3594622b0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f9650097df0 tx=0x7f96500cfea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b0d11cb41e5147 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:17:34.799 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.789+0000 7f964aa00640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-18T15:17:34.799 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.789+0000 7f964aa00640 1 -- 172.21.15.19:0/4286451827 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9650097c30 con 0x55b359458160 2024-03-18T15:17:34.799 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.789+0000 7f964aa00640 1 -- 172.21.15.19:0/4286451827 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f96500ccc00 con 0x55b359458160 2024-03-18T15:17:34.799 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.789+0000 7f966a9b7740 1 -- 172.21.15.19:0/4286451827 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f96500cfcf0 con 0x55b359458160 2024-03-18T15:17:34.799 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.789+0000 7f964aa00640 1 -- 172.21.15.19:0/4286451827 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f96500d7ea0 con 0x55b359458160 2024-03-18T15:17:34.799 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.789+0000 7f966a9b7740 10 client.25208 fetch_fsmap learned FSMap version 26 2024-03-18T15:17:34.799 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.789+0000 7f966a9b7740 1 -- 172.21.15.19:0/4286451827 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55b35919b6b0 con 0x55b359458160 2024-03-18T15:17:34.799 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.790+0000 7f964aa00640 1 -- 172.21.15.19:0/4286451827 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f96500cfcf0 con 0x55b359458160 2024-03-18T15:17:34.800 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.790+0000 7f964aa00640 1 -- 172.21.15.19:0/4286451827 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 26) ==== 8362+0+0 (secure 0 0 0) 0x7f965000f040 con 0x55b359458160 2024-03-18T15:17:34.800 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.790+0000 7f966a9b7740 10 client.25208 fetch_fsmap finished waiting for FSMap version 26 2024-03-18T15:17:34.800 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.790+0000 7f966a9b7740 10 client.25208 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-18T15:17:34.800 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.790+0000 7f966a9b7740 20 client.25208 populate_metadata read hostname 'smithi019' 2024-03-18T15:17:34.800 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.790+0000 7f966a9b7740 1 --2- 172.21.15.19:0/4286451827 >> [v2:172.21.15.33:6838/1903591535,v1:172.21.15.33:6839/1903591535] conn(0x55b35946dd20 0x55b35946fe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:34.800 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.790+0000 7f966a9b7740 4 client.25208 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T15:17:34.800 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.790+0000 7f966a9b7740 1 -- 172.21.15.19:0/4286451827 --> [v2:172.21.15.33:6838/1903591535,v1:172.21.15.33:6839/1903591535] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b35941a2b0 con 0x55b35946dd20 2024-03-18T15:17:34.800 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.790+0000 7f965fe00640 1 --2- 172.21.15.19:0/4286451827 >> [v2:172.21.15.33:6838/1903591535,v1:172.21.15.33:6839/1903591535] conn(0x55b35946dd20 0x55b35946fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:34.800 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.791+0000 7f965fe00640 1 --2- 172.21.15.19:0/4286451827 >> [v2:172.21.15.33:6838/1903591535,v1:172.21.15.33:6839/1903591535] conn(0x55b35946dd20 0x55b35946fe20 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b0dbac7388c1b67d server_cookie=b2c80f490da98aad in_seq=0 out_seq=0 2024-03-18T15:17:34.800 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.791+0000 7f964aa00640 10 client.25208 ms_handle_connect on v2:172.21.15.33:6838/1903591535 2024-03-18T15:17:34.801 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.791+0000 7f964aa00640 1 -- 172.21.15.19:0/4286451827 <== mds.0 v2:172.21.15.33:6838/1903591535 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b35941a2b0 con 0x55b35946dd20 2024-03-18T15:17:34.801 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.792+0000 7f964aa00640 10 client.25208 handle_command_reply: tid=0 2024-03-18T15:17:34.801 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.792+0000 7f966a9b7740 1 client.25208 shutdown 2024-03-18T15:17:34.801 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.792+0000 7f966a9b7740 1 -- 172.21.15.19:0/4286451827 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b359458160 msgr2=0x55b3594622b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:34.801 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.792+0000 7f966a9b7740 1 --2- 172.21.15.19:0/4286451827 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b359458160 0x55b3594622b0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f9650097df0 tx=0x7f96500cfea0 comp rx=0 tx=0).stop 2024-03-18T15:17:34.801 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.792+0000 7f966a9b7740 1 -- 172.21.15.19:0/4286451827 shutdown_connections 2024-03-18T15:17:34.801 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.792+0000 7f966a9b7740 1 --2- 172.21.15.19:0/4286451827 >> [v2:172.21.15.33:6838/1903591535,v1:172.21.15.33:6839/1903591535] conn(0x55b35946dd20 0x55b35946fe20 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:34.801 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.792+0000 7f966a9b7740 1 --2- 172.21.15.19:0/4286451827 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b35945dcc0 0x55b359469830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:34.801 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.792+0000 7f966a9b7740 1 --2- 172.21.15.19:0/4286451827 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55b359458b50 0x55b3594627f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:34.802 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.792+0000 7f966a9b7740 1 --2- 172.21.15.19:0/4286451827 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b359458160 0x55b3594622b0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:34.802 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.792+0000 7f966a9b7740 1 -- 172.21.15.19:0/4286451827 >> 172.21.15.19:0/4286451827 conn(0x55b3594536d0 msgr2=0x55b359452a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:17:34.802 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.792+0000 7f964aa00640 0 client.25208 ms_handle_reset on v2:172.21.15.33:6838/1903591535 2024-03-18T15:17:34.802 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.792+0000 7f966a9b7740 1 -- 172.21.15.19:0/4286451827 shutdown_connections 2024-03-18T15:17:34.802 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.792+0000 7f966a9b7740 1 -- 172.21.15.19:0/4286451827 wait complete. 2024-03-18T15:17:34.802 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.792+0000 7f966a9b7740 20 client.25208 trim_cache size 0 max 16384 2024-03-18T15:17:34.814 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.813+0000 7f966a9b7740 1 Processor -- start 2024-03-18T15:17:34.814 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.813+0000 7f966a9b7740 1 -- start start 2024-03-18T15:17:34.814 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.813+0000 7f966a9b7740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b359458160 0x55b359461ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:34.814 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.813+0000 7f966a9b7740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b359458b50 0x55b359462400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:34.814 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.813+0000 7f966a9b7740 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55b35945dcc0 0x55b3594595c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:34.814 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.813+0000 7f966a9b7740 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x55b35946e160 con 0x55b35945dcc0 2024-03-18T15:17:34.814 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.813+0000 7f966a9b7740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55b35946e830 con 0x55b359458b50 2024-03-18T15:17:34.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.813+0000 7f966a9b7740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55b35946ef00 con 0x55b359458160 2024-03-18T15:17:34.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.814+0000 7f9665200640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b359458160 0x55b359461ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:34.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.814+0000 7f965fe00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b359458b50 0x55b359462400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:34.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.814+0000 7f9665200640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b359458160 0x55b359461ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.19:58448/0 (socket says 172.21.15.19:58448) 2024-03-18T15:17:34.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.814+0000 7f9665200640 1 -- 172.21.15.19:0/1030242384 learned_addr learned my addr 172.21.15.19:0/1030242384 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:17:34.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.814+0000 7f9665c00640 1 --2- 172.21.15.19:0/1030242384 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55b35945dcc0 0x55b3594595c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:34.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.814+0000 7f965fe00640 1 -- 172.21.15.19:0/1030242384 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b359458160 msgr2=0x55b359461ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:17:34.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.814+0000 7f965fe00640 1 --2- 172.21.15.19:0/1030242384 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b359458160 0x55b359461ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:34.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.814+0000 7f965fe00640 1 -- 172.21.15.19:0/1030242384 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55b35945dcc0 msgr2=0x55b3594595c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:17:34.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.814+0000 7f965fe00640 1 --2- 172.21.15.19:0/1030242384 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55b35945dcc0 0x55b3594595c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:34.816 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.814+0000 7f965fe00640 1 -- 172.21.15.19:0/1030242384 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b359458f30 con 0x55b359458b50 2024-03-18T15:17:34.816 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.814+0000 7f9665200640 1 --2- 172.21.15.19:0/1030242384 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b359458160 0x55b359461ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:17:34.821 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.814+0000 7f965fe00640 1 --2- 172.21.15.19:0/1030242384 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b359458b50 0x55b359462400 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f964c02c170 tx=0x7f964c005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=16fae0c28588d8f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:17:34.821 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.815+0000 7f9655600640 1 -- 172.21.15.19:0/1030242384 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f964c02cb20 con 0x55b359458b50 2024-03-18T15:17:34.821 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.815+0000 7f9655600640 1 -- 172.21.15.19:0/1030242384 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f964c02ccc0 con 0x55b359458b50 2024-03-18T15:17:34.821 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.815+0000 7f9655600640 1 -- 172.21.15.19:0/1030242384 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f964c005040 con 0x55b359458b50 2024-03-18T15:17:34.821 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.815+0000 7f966a9b7740 1 -- 172.21.15.19:0/1030242384 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b359458b50 msgr2=0x55b359462400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:34.821 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.815+0000 7f966a9b7740 1 --2- 172.21.15.19:0/1030242384 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b359458b50 0x55b359462400 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f964c02c170 tx=0x7f964c005fd0 comp rx=0 tx=0).stop 2024-03-18T15:17:34.821 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.815+0000 7f966a9b7740 1 -- 172.21.15.19:0/1030242384 shutdown_connections 2024-03-18T15:17:34.822 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.815+0000 7f966a9b7740 1 --2- 172.21.15.19:0/1030242384 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55b35945dcc0 0x55b3594595c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:34.822 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.815+0000 7f966a9b7740 1 --2- 172.21.15.19:0/1030242384 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b359458b50 0x55b359462400 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:34.822 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.815+0000 7f966a9b7740 1 --2- 172.21.15.19:0/1030242384 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b359458160 0x55b359461ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:34.822 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.815+0000 7f966a9b7740 1 -- 172.21.15.19:0/1030242384 >> 172.21.15.19:0/1030242384 conn(0x55b3594536d0 msgr2=0x55b359453ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:17:34.822 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.815+0000 7f966a9b7740 1 -- 172.21.15.19:0/1030242384 shutdown_connections 2024-03-18T15:17:34.822 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.815+0000 7f966a9b7740 1 -- 172.21.15.19:0/1030242384 wait complete. 2024-03-18T15:17:34.822 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.816+0000 7f966a9b7740 1 -- start start 2024-03-18T15:17:34.822 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.816+0000 7f966a9b7740 1 Processor -- start 2024-03-18T15:17:34.822 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.816+0000 7f966a9b7740 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55b359458160 0x55b359467a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:34.822 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.816+0000 7f966a9b7740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b359458b50 0x55b359467f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:34.822 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.816+0000 7f966a9b7740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b35945dcc0 0x55b3594684f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:34.822 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.816+0000 7f966a9b7740 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x55b35946fca0 con 0x55b359458160 2024-03-18T15:17:34.822 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.816+0000 7f966a9b7740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55b35946f5d0 con 0x55b35945dcc0 2024-03-18T15:17:34.823 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.816+0000 7f966a9b7740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55b359470370 con 0x55b359458b50 2024-03-18T15:17:34.823 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.816+0000 7f966a9b7740 10 client.0 _ll_register_callbacks cb 0x7f96500d7400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T15:17:34.823 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.817+0000 7f965fe00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b359458b50 0x55b359467f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:34.823 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.817+0000 7f9665c00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b35945dcc0 0x55b3594684f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:34.823 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.817+0000 7f965fe00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b359458b50 0x55b359467f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.19:58454/0 (socket says 172.21.15.19:58454) 2024-03-18T15:17:34.823 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.817+0000 7f965fe00640 1 -- 172.21.15.19:0/4273682522 learned_addr learned my addr 172.21.15.19:0/4273682522 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:17:34.823 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.817+0000 7f9665200640 1 --2- 172.21.15.19:0/4273682522 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55b359458160 0x55b359467a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:34.823 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.817+0000 7f965fe00640 1 -- 172.21.15.19:0/4273682522 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b35945dcc0 msgr2=0x55b3594684f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:17:34.823 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.817+0000 7f965fe00640 1 --2- 172.21.15.19:0/4273682522 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b35945dcc0 0x55b3594684f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:34.823 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.817+0000 7f965fe00640 1 -- 172.21.15.19:0/4273682522 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55b359458160 msgr2=0x55b359467a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:17:34.823 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.817+0000 7f965fe00640 1 --2- 172.21.15.19:0/4273682522 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55b359458160 0x55b359467a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:34.823 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.817+0000 7f965fe00640 1 -- 172.21.15.19:0/4273682522 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b359468d20 con 0x55b359458b50 2024-03-18T15:17:34.823 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.817+0000 7f965fe00640 1 --2- 172.21.15.19:0/4273682522 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b359458b50 0x55b359467f70 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f964c0096c0 tx=0x7f964c02cb20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b4ffc286b27c16e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:17:34.823 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.817+0000 7f964aa00640 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-03-18T15:17:34.824 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.818+0000 7f964aa00640 1 -- 172.21.15.19:0/4273682522 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f964c055060 con 0x55b359458b50 2024-03-18T15:17:34.824 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.818+0000 7f964aa00640 1 -- 172.21.15.19:0/4273682522 <== mon.1 v2:172.21.15.104:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f964c046b10 con 0x55b359458b50 2024-03-18T15:17:34.824 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.818+0000 7f966a9b7740 1 -- 172.21.15.19:0/4273682522 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f964c02ccc0 con 0x55b359458b50 2024-03-18T15:17:34.824 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.818+0000 7f964aa00640 1 -- 172.21.15.19:0/4273682522 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f964c046d30 con 0x55b359458b50 2024-03-18T15:17:34.824 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.818+0000 7f966a9b7740 10 client.25225 fetch_fsmap learned FSMap version 26 2024-03-18T15:17:34.824 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.818+0000 7f966a9b7740 1 -- 172.21.15.19:0/4273682522 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55b3594d4ce0 con 0x55b359458b50 2024-03-18T15:17:34.824 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.818+0000 7f964aa00640 1 -- 172.21.15.19:0/4273682522 <== mon.1 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f964c02ccc0 con 0x55b359458b50 2024-03-18T15:17:34.824 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.818+0000 7f964aa00640 1 -- 172.21.15.19:0/4273682522 <== mon.1 v2:172.21.15.104:3300/0 5 ==== fsmap(e 26) ==== 8362+0+0 (secure 0 0 0) 0x7f964c0046c0 con 0x55b359458b50 2024-03-18T15:17:34.824 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.818+0000 7f966a9b7740 10 client.25225 fetch_fsmap finished waiting for FSMap version 26 2024-03-18T15:17:34.824 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.818+0000 7f966a9b7740 10 client.25225 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-18T15:17:34.824 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.818+0000 7f966a9b7740 20 client.25225 populate_metadata read hostname 'smithi019' 2024-03-18T15:17:34.824 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.818+0000 7f966a9b7740 1 --2- 172.21.15.19:0/4273682522 >> [v2:172.21.15.33:6838/1903591535,v1:172.21.15.33:6839/1903591535] conn(0x55b3594d94a0 0x55b3594d81b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:34.824 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.818+0000 7f966a9b7740 4 client.25225 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-18T15:17:34.824 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.819+0000 7f966a9b7740 1 -- 172.21.15.19:0/4273682522 --> [v2:172.21.15.33:6838/1903591535,v1:172.21.15.33:6839/1903591535] -- command(tid 0: {"prefix": "status"}) -- 0x55b3593ca190 con 0x55b3594d94a0 2024-03-18T15:17:34.825 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.819+0000 7f9665200640 1 --2- 172.21.15.19:0/4273682522 >> [v2:172.21.15.33:6838/1903591535,v1:172.21.15.33:6839/1903591535] conn(0x55b3594d94a0 0x55b3594d81b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:34.825 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.819+0000 7f9665200640 1 --2- 172.21.15.19:0/4273682522 >> [v2:172.21.15.33:6838/1903591535,v1:172.21.15.33:6839/1903591535] conn(0x55b3594d94a0 0x55b3594d81b0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fb6b7692aac15386 server_cookie=4cfe4b2ee1fc1e29 in_seq=0 out_seq=0 2024-03-18T15:17:34.825 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.819+0000 7f964aa00640 10 client.25225 ms_handle_connect on v2:172.21.15.33:6838/1903591535 2024-03-18T15:17:34.825 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.819+0000 7f964aa00640 1 -- 172.21.15.19:0/4273682522 <== mds.0 v2:172.21.15.33:6838/1903591535 1 ==== command_reply(tid 0: 0 ) ==== 8+0+326 (crc 0 0 0) 0x55b3593ca190 con 0x55b3594d94a0 2024-03-18T15:17:34.825 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.819+0000 7f964aa00640 10 client.25225 handle_command_reply: tid=0 2024-03-18T15:17:34.825 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.819+0000 7f966a9b7740 1 client.25225 shutdown 2024-03-18T15:17:34.825 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.820+0000 7f966a9b7740 1 -- 172.21.15.19:0/4273682522 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b359458b50 msgr2=0x55b359467f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:34.825 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.820+0000 7f966a9b7740 1 --2- 172.21.15.19:0/4273682522 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b359458b50 0x55b359467f70 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f964c0096c0 tx=0x7f964c02cb20 comp rx=0 tx=0).stop 2024-03-18T15:17:34.825 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.820+0000 7f966a9b7740 1 -- 172.21.15.19:0/4273682522 shutdown_connections 2024-03-18T15:17:34.825 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.820+0000 7f966a9b7740 1 --2- 172.21.15.19:0/4273682522 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b35945dcc0 0x55b3594684f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:34.825 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.820+0000 7f966a9b7740 1 --2- 172.21.15.19:0/4273682522 >> [v2:172.21.15.33:6838/1903591535,v1:172.21.15.33:6839/1903591535] conn(0x55b3594d94a0 0x55b3594d81b0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:34.825 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.820+0000 7f966a9b7740 1 --2- 172.21.15.19:0/4273682522 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b359458b50 0x55b359467f70 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:34.825 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.820+0000 7f966a9b7740 1 --2- 172.21.15.19:0/4273682522 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55b359458160 0x55b359467a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:34.825 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.820+0000 7f964aa00640 0 client.25225 ms_handle_reset on v2:172.21.15.33:6838/1903591535 2024-03-18T15:17:34.825 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.820+0000 7f966a9b7740 1 -- 172.21.15.19:0/4273682522 >> 172.21.15.19:0/4273682522 conn(0x55b3594536d0 msgr2=0x55b359461ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:17:34.826 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.820+0000 7f966a9b7740 1 -- 172.21.15.19:0/4273682522 shutdown_connections 2024-03-18T15:17:34.826 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.820+0000 7f966a9b7740 1 -- 172.21.15.19:0/4273682522 wait complete. 2024-03-18T15:17:34.826 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.820+0000 7f966a9b7740 20 client.25225 trim_cache size 0 max 16384 2024-03-18T15:17:34.826 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-03-18T15:17:34.826 INFO:teuthology.orchestra.run.smithi019.stdout: "cluster_fsid": "d4dc2ba6-e538-11ee-95c9-87774f69a715", 2024-03-18T15:17:34.826 INFO:teuthology.orchestra.run.smithi019.stdout: "whoami": 0, 2024-03-18T15:17:34.826 INFO:teuthology.orchestra.run.smithi019.stdout: "id": 24560, 2024-03-18T15:17:34.826 INFO:teuthology.orchestra.run.smithi019.stdout: "want_state": "up:active", 2024-03-18T15:17:34.826 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "up:active", 2024-03-18T15:17:34.827 INFO:teuthology.orchestra.run.smithi019.stdout: "fs_name": "cephfs", 2024-03-18T15:17:34.827 INFO:teuthology.orchestra.run.smithi019.stdout: "rank_uptime": 143.70277322800001, 2024-03-18T15:17:34.827 INFO:teuthology.orchestra.run.smithi019.stdout: "mdsmap_epoch": 26, 2024-03-18T15:17:34.827 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap_epoch": 93, 2024-03-18T15:17:34.827 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap_epoch_barrier": 89, 2024-03-18T15:17:34.827 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 150.52951754 2024-03-18T15:17:34.827 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-03-18T15:17:34.828 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.821+0000 7f9667c00640 1 -- 172.21.15.19:0/3164262390 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f96440a2880 msgr2=0x7f96440a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:34.828 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.821+0000 7f9667c00640 1 --2- 172.21.15.19:0/3164262390 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f96440a2880 0x7f96440a4d40 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x55b3594593b0 tx=0x7f9658005f90 comp rx=0 tx=0).stop 2024-03-18T15:17:34.828 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.821+0000 7f9667c00640 1 -- 172.21.15.19:0/3164262390 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f96601023a0 msgr2=0x7f9660199d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:34.828 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.821+0000 7f9667c00640 1 --2- 172.21.15.19:0/3164262390 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f96601023a0 0x7f9660199d70 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f965000eab0 tx=0x7f965000ef70 comp rx=0 tx=0).stop 2024-03-18T15:17:34.828 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.821+0000 7f9667c00640 1 -- 172.21.15.19:0/3164262390 shutdown_connections 2024-03-18T15:17:34.828 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.821+0000 7f9667c00640 1 --2- 172.21.15.19:0/3164262390 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9660107600 0x7f96601a1320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:34.828 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.821+0000 7f9667c00640 1 --2- 172.21.15.19:0/3164262390 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9660104cd0 0x7f966019a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:34.828 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.821+0000 7f9667c00640 1 --2- 172.21.15.19:0/3164262390 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f96440a2880 0x7f96440a4d40 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:34.828 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.821+0000 7f9667c00640 1 --2- 172.21.15.19:0/3164262390 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f96601023a0 0x7f9660199d70 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:34.828 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.821+0000 7f9667c00640 1 -- 172.21.15.19:0/3164262390 >> 172.21.15.19:0/3164262390 conn(0x7f96600fbd10 msgr2=0x7f9660101b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:17:34.828 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.821+0000 7f9667c00640 1 -- 172.21.15.19:0/3164262390 shutdown_connections 2024-03-18T15:17:34.829 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:34.821+0000 7f9667c00640 1 -- 172.21.15.19:0/3164262390 wait complete. 2024-03-18T15:17:34.839 INFO:tasks.cephadm:Setup complete, yielding 2024-03-18T15:17:34.840 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-18T15:17:34.855 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi019.front.sepia.ceph.com 2024-03-18T15:17:34.855 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 -- bash -c 'ceph orch status' 2024-03-18T15:17:35.346 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:17:35.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:17:35 smithi019 ceph-mon[31522]: pgmap v393: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T15:17:35.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:17:35 smithi033 ceph-mon[36648]: pgmap v393: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T15:17:35.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:17:35 smithi104 ceph-mon[36070]: pgmap v393: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T15:17:37.562 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:37.561+0000 7f69d6800700 1 -- 172.21.15.19:0/278498464 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f69d0103ef0 msgr2=0x7f69d010a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:37.563 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:37.561+0000 7f69d6800700 1 --2- 172.21.15.19:0/278498464 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f69d0103ef0 0x7f69d010a7a0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f69c800b0a0 tx=0x7f69c800b3b0 comp rx=0 tx=0).stop 2024-03-18T15:17:37.563 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:37.561+0000 7f69d6800700 1 -- 172.21.15.19:0/278498464 shutdown_connections 2024-03-18T15:17:37.563 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:37.561+0000 7f69d6800700 1 --2- 172.21.15.19:0/278498464 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f69d0103ef0 0x7f69d010a7a0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:37.563 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:37.561+0000 7f69d6800700 1 --2- 172.21.15.19:0/278498464 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f69d0102350 0x7f69d0102770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:37.563 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:37.561+0000 7f69d6800700 1 --2- 172.21.15.19:0/278498464 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f69d0103550 0x7f69d01039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:37.563 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:37.561+0000 7f69d6800700 1 -- 172.21.15.19:0/278498464 >> 172.21.15.19:0/278498464 conn(0x7f69d00fd8f0 msgr2=0x7f69d00ffd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:17:37.563 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:37.562+0000 7f69d6800700 1 -- 172.21.15.19:0/278498464 shutdown_connections 2024-03-18T15:17:37.563 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:37.562+0000 7f69d6800700 1 -- 172.21.15.19:0/278498464 wait complete. 2024-03-18T15:17:37.563 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:37.563+0000 7f69d6800700 1 Processor -- start 2024-03-18T15:17:37.563 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:37.563+0000 7f69d6800700 1 -- start start 2024-03-18T15:17:37.565 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:37.563+0000 7f69d6800700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f69d0102350 0x7f69d019a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:37.565 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:37.563+0000 7f69d6800700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f69d0103550 0x7f69d019ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:37.565 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:37.563+0000 7f69d6800700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f69d0103ef0 0x7f69d01a1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:37.565 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:37.563+0000 7f69d6800700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f69d010df90 con 0x7f69d0102350 2024-03-18T15:17:37.565 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:37.563+0000 7f69d6800700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f69d010de10 con 0x7f69d0103550 2024-03-18T15:17:37.565 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:37.563+0000 7f69d6800700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f69d010e110 con 0x7f69d0103ef0 2024-03-18T15:17:37.565 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:37.563+0000 7f69cea00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f69d0103550 0x7f69d019ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:37.565 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:37.563+0000 7f69cfe00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f69d0103ef0 0x7f69d01a1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:37.565 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:37.564+0000 7f69cea00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f69d0103550 0x7f69d019ac60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:39502/0 (socket says 172.21.15.19:39502) 2024-03-18T15:17:37.565 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:37.564+0000 7f69cea00700 1 -- 172.21.15.19:0/1811435223 learned_addr learned my addr 172.21.15.19:0/1811435223 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:17:37.565 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:37.564+0000 7f69cf400700 1 --2- 172.21.15.19:0/1811435223 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f69d0102350 0x7f69d019a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:37.566 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:37.564+0000 7f69cfe00700 1 -- 172.21.15.19:0/1811435223 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f69d0103550 msgr2=0x7f69d019ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:37.566 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:37.564+0000 7f69cfe00700 1 --2- 172.21.15.19:0/1811435223 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f69d0103550 0x7f69d019ac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:37.566 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:37.564+0000 7f69cfe00700 1 -- 172.21.15.19:0/1811435223 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f69d0102350 msgr2=0x7f69d019a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:37.566 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:37.564+0000 7f69cfe00700 1 --2- 172.21.15.19:0/1811435223 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f69d0102350 0x7f69d019a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:37.566 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:37.564+0000 7f69cfe00700 1 -- 172.21.15.19:0/1811435223 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69d01a2380 con 0x7f69d0103ef0 2024-03-18T15:17:37.566 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:37.564+0000 7f69cea00700 1 --2- 172.21.15.19:0/1811435223 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f69d0103550 0x7f69d019ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:17:37.566 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:37.564+0000 7f69cfe00700 1 --2- 172.21.15.19:0/1811435223 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f69d0103ef0 0x7f69d01a1ce0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f69c8007fd0 tx=0x7f69c8004f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:17:37.566 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:37.565+0000 7f69c7e00700 1 -- 172.21.15.19:0/1811435223 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69c8021070 con 0x7f69d0103ef0 2024-03-18T15:17:37.566 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:37.565+0000 7f69c7e00700 1 -- 172.21.15.19:0/1811435223 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f69c8007b80 con 0x7f69d0103ef0 2024-03-18T15:17:37.567 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:37.565+0000 7f69d6800700 1 -- 172.21.15.19:0/1811435223 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69d01a2610 con 0x7f69d0103ef0 2024-03-18T15:17:37.567 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:37.565+0000 7f69c7e00700 1 -- 172.21.15.19:0/1811435223 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69c80052a0 con 0x7f69d0103ef0 2024-03-18T15:17:37.567 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:37.565+0000 7f69d6800700 1 -- 172.21.15.19:0/1811435223 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69d01a2ac0 con 0x7f69d0103ef0 2024-03-18T15:17:37.567 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:37.566+0000 7f69d6800700 1 -- 172.21.15.19:0/1811435223 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69d0068f90 con 0x7f69d0103ef0 2024-03-18T15:17:37.567 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:37.568+0000 7f69c7e00700 1 -- 172.21.15.19:0/1811435223 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f69c8024070 con 0x7f69d0103ef0 2024-03-18T15:17:37.569 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:37.569+0000 7f69c7e00700 1 --2- 172.21.15.19:0/1811435223 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f69b40a2910 0x7f69b40a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:37.569 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:37.569+0000 7f69c7e00700 1 -- 172.21.15.19:0/1811435223 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f69c80d0160 con 0x7f69d0103ef0 2024-03-18T15:17:37.569 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:37.569+0000 7f69cf400700 1 --2- 172.21.15.19:0/1811435223 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f69b40a2910 0x7f69b40a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:37.570 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:37.570+0000 7f69cf400700 1 --2- 172.21.15.19:0/1811435223 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f69b40a2910 0x7f69b40a4dd0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f69d01033b0 tx=0x7f69c0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:17:37.577 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:37.577+0000 7f69c7e00700 1 -- 172.21.15.19:0/1811435223 <== mon.1 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f69c8007d30 con 0x7f69d0103ef0 2024-03-18T15:17:37.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:17:37 smithi019 ceph-mon[31522]: pgmap v394: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T15:17:37.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:17:37 smithi033 ceph-mon[36648]: pgmap v394: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T15:17:37.829 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:37.829+0000 7f69d6800700 1 -- 172.21.15.19:0/1811435223 --> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7f69d01a2d20 con 0x7f69b40a2910 2024-03-18T15:17:37.831 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:37.831+0000 7f69c7e00700 1 -- 172.21.15.19:0/1811435223 <== mgr.14152 v2:172.21.15.19:6800/2017714236 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f69d01a2d20 con 0x7f69b40a2910 2024-03-18T15:17:37.831 INFO:teuthology.orchestra.run.smithi019.stdout:Backend: cephadm 2024-03-18T15:17:37.832 INFO:teuthology.orchestra.run.smithi019.stdout:Available: Yes 2024-03-18T15:17:37.832 INFO:teuthology.orchestra.run.smithi019.stdout:Paused: No 2024-03-18T15:17:37.834 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:37.834+0000 7f69d6800700 1 -- 172.21.15.19:0/1811435223 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f69b40a2910 msgr2=0x7f69b40a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:37.835 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:37.834+0000 7f69d6800700 1 --2- 172.21.15.19:0/1811435223 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f69b40a2910 0x7f69b40a4dd0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f69d01033b0 tx=0x7f69c0009260 comp rx=0 tx=0).stop 2024-03-18T15:17:37.835 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:37.834+0000 7f69d6800700 1 -- 172.21.15.19:0/1811435223 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f69d0103ef0 msgr2=0x7f69d01a1ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:37.835 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:37.834+0000 7f69d6800700 1 --2- 172.21.15.19:0/1811435223 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f69d0103ef0 0x7f69d01a1ce0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f69c8007fd0 tx=0x7f69c8004f40 comp rx=0 tx=0).stop 2024-03-18T15:17:37.835 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:37.834+0000 7f69d6800700 1 -- 172.21.15.19:0/1811435223 shutdown_connections 2024-03-18T15:17:37.835 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:37.834+0000 7f69d6800700 1 --2- 172.21.15.19:0/1811435223 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f69b40a2910 0x7f69b40a4dd0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:37.835 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:37.834+0000 7f69d6800700 1 --2- 172.21.15.19:0/1811435223 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f69d0103ef0 0x7f69d01a1ce0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:37.835 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:37.834+0000 7f69d6800700 1 --2- 172.21.15.19:0/1811435223 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f69d0102350 0x7f69d019a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:37.835 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:37.834+0000 7f69d6800700 1 --2- 172.21.15.19:0/1811435223 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f69d0103550 0x7f69d019ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:37.835 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:37.834+0000 7f69d6800700 1 -- 172.21.15.19:0/1811435223 >> 172.21.15.19:0/1811435223 conn(0x7f69d00fd8f0 msgr2=0x7f69d0074500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:17:37.835 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:37.835+0000 7f69d6800700 1 -- 172.21.15.19:0/1811435223 shutdown_connections 2024-03-18T15:17:37.835 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:37.835+0000 7f69d6800700 1 -- 172.21.15.19:0/1811435223 wait complete. 2024-03-18T15:17:37.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:17:37 smithi104 ceph-mon[36070]: pgmap v394: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T15:17:38.607 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 -- bash -c 'ceph orch ps' 2024-03-18T15:17:38.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:17:38 smithi033 ceph-mon[36648]: from='client.25231 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:17:38.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:17:38 smithi104 ceph-mon[36070]: from='client.25231 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:17:38.964 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:17:38 smithi019 ceph-mon[31522]: from='client.25231 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:17:39.088 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:17:39.495 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:17:39 smithi019 ceph-mon[31522]: pgmap v395: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T15:17:39.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:17:39 smithi033 ceph-mon[36648]: pgmap v395: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T15:17:39.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:17:39 smithi104 ceph-mon[36070]: pgmap v395: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T15:17:41.281 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:41.279+0000 7fa71ea00700 1 -- 172.21.15.19:0/1978383915 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa718103ea0 msgr2=0x7fa71810a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:41.281 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:41.279+0000 7fa71ea00700 1 --2- 172.21.15.19:0/1978383915 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa718103ea0 0x7fa71810a750 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fa71000b0d0 tx=0x7fa71000b3e0 comp rx=0 tx=0).stop 2024-03-18T15:17:41.281 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:41.279+0000 7fa71ea00700 1 -- 172.21.15.19:0/1978383915 shutdown_connections 2024-03-18T15:17:41.281 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:41.279+0000 7fa71ea00700 1 --2- 172.21.15.19:0/1978383915 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa718103ea0 0x7fa71810a750 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:41.281 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:41.279+0000 7fa71ea00700 1 --2- 172.21.15.19:0/1978383915 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa718102300 0x7fa718102720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:41.281 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:41.279+0000 7fa71ea00700 1 --2- 172.21.15.19:0/1978383915 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa718103500 0x7fa718103960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:41.281 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:41.279+0000 7fa71ea00700 1 -- 172.21.15.19:0/1978383915 >> 172.21.15.19:0/1978383915 conn(0x7fa7180fd8e0 msgr2=0x7fa7180ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:17:41.281 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:41.279+0000 7fa71ea00700 1 -- 172.21.15.19:0/1978383915 shutdown_connections 2024-03-18T15:17:41.281 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:41.279+0000 7fa71ea00700 1 -- 172.21.15.19:0/1978383915 wait complete. 2024-03-18T15:17:41.283 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:41.280+0000 7fa71ea00700 1 Processor -- start 2024-03-18T15:17:41.283 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:41.281+0000 7fa71ea00700 1 -- start start 2024-03-18T15:17:41.283 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:41.281+0000 7fa71ea00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa718102300 0x7fa7181984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:41.283 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:41.281+0000 7fa71ea00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa718103500 0x7fa718198a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:41.283 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:41.281+0000 7fa71ea00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa718103ea0 0x7fa71819fa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:41.283 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:41.281+0000 7fa71ea00700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fa71810df20 con 0x7fa718102300 2024-03-18T15:17:41.283 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:41.281+0000 7fa71ea00700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa71810dda0 con 0x7fa718103ea0 2024-03-18T15:17:41.283 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:41.281+0000 7fa71ea00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fa71810e0a0 con 0x7fa718103500 2024-03-18T15:17:41.283 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:41.281+0000 7fa717400700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa718103500 0x7fa718198a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:41.284 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:41.281+0000 7fa71ca00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa718103ea0 0x7fa71819fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:41.284 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:41.281+0000 7fa717400700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa718103500 0x7fa718198a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.19:58496/0 (socket says 172.21.15.19:58496) 2024-03-18T15:17:41.284 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:41.281+0000 7fa717400700 1 -- 172.21.15.19:0/2544695967 learned_addr learned my addr 172.21.15.19:0/2544695967 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:17:41.284 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:41.282+0000 7fa717e00700 1 --2- 172.21.15.19:0/2544695967 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa718102300 0x7fa7181984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:41.284 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:41.282+0000 7fa717400700 1 -- 172.21.15.19:0/2544695967 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa718103ea0 msgr2=0x7fa71819fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:41.284 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:41.282+0000 7fa717400700 1 --2- 172.21.15.19:0/2544695967 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa718103ea0 0x7fa71819fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:41.284 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:41.282+0000 7fa717400700 1 -- 172.21.15.19:0/2544695967 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa718102300 msgr2=0x7fa7181984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:41.284 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:41.282+0000 7fa717400700 1 --2- 172.21.15.19:0/2544695967 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa718102300 0x7fa7181984d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:41.284 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:41.282+0000 7fa717400700 1 -- 172.21.15.19:0/2544695967 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7181a0130 con 0x7fa718103500 2024-03-18T15:17:41.284 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:41.282+0000 7fa717e00700 1 --2- 172.21.15.19:0/2544695967 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa718102300 0x7fa7181984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T15:17:41.284 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:41.282+0000 7fa71ca00700 1 --2- 172.21.15.19:0/2544695967 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa718103ea0 0x7fa71819fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:17:41.285 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:41.282+0000 7fa717400700 1 --2- 172.21.15.19:0/2544695967 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa718103500 0x7fa718198a10 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fa70400e9d0 tx=0x7fa70400ece0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:17:41.285 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:41.283+0000 7fa714c00700 1 -- 172.21.15.19:0/2544695967 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa70400cbd0 con 0x7fa718103500 2024-03-18T15:17:41.285 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:41.283+0000 7fa714c00700 1 -- 172.21.15.19:0/2544695967 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa70400cd70 con 0x7fa718103500 2024-03-18T15:17:41.285 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:41.283+0000 7fa71ea00700 1 -- 172.21.15.19:0/2544695967 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7181a0420 con 0x7fa718103500 2024-03-18T15:17:41.285 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:41.283+0000 7fa714c00700 1 -- 172.21.15.19:0/2544695967 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa704018660 con 0x7fa718103500 2024-03-18T15:17:41.285 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:41.283+0000 7fa71ea00700 1 -- 172.21.15.19:0/2544695967 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7181a0960 con 0x7fa718103500 2024-03-18T15:17:41.286 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:41.284+0000 7fa71ea00700 1 -- 172.21.15.19:0/2544695967 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa718068f90 con 0x7fa718103500 2024-03-18T15:17:41.287 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:41.286+0000 7fa714c00700 1 -- 172.21.15.19:0/2544695967 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fa704047a60 con 0x7fa718103500 2024-03-18T15:17:41.288 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:41.287+0000 7fa714c00700 1 --2- 172.21.15.19:0/2544695967 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fa6fc0a2890 0x7fa6fc0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:41.288 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:41.287+0000 7fa714c00700 1 -- 172.21.15.19:0/2544695967 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fa7040cf080 con 0x7fa718103500 2024-03-18T15:17:41.288 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:41.288+0000 7fa717e00700 1 --2- 172.21.15.19:0/2544695967 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fa6fc0a2890 0x7fa6fc0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:41.289 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:41.288+0000 7fa717e00700 1 --2- 172.21.15.19:0/2544695967 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fa6fc0a2890 0x7fa6fc0a4d50 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fa718103360 tx=0x7fa70800b670 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:17:41.295 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:41.295+0000 7fa714c00700 1 -- 172.21.15.19:0/2544695967 <== mon.1 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa70409bcf0 con 0x7fa718103500 2024-03-18T15:17:41.535 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:41.534+0000 7fa71ea00700 1 -- 172.21.15.19:0/2544695967 --> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7fa718062710 con 0x7fa6fc0a2890 2024-03-18T15:17:41.556 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:41.555+0000 7fa714c00700 1 -- 172.21.15.19:0/2544695967 <== mgr.14152 v2:172.21.15.19:6800/2017714236 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4123 (secure 0 0 0) 0x7fa718062710 con 0x7fa6fc0a2890 2024-03-18T15:17:41.557 INFO:teuthology.orchestra.run.smithi019.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-18T15:17:41.557 INFO:teuthology.orchestra.run.smithi019.stdout:mds.a smithi019 running (2m) 33s ago 2m 15.2M - 19.0.0-2162-g3d78f5ba be86dc8cb22b a8b0d96b5a0e 2024-03-18T15:17:41.557 INFO:teuthology.orchestra.run.smithi019.stdout:mds.b smithi033 running (2m) 88s ago 2m 25.6M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 79906bb04e44 2024-03-18T15:17:41.557 INFO:teuthology.orchestra.run.smithi019.stdout:mds.c smithi104 running (2m) 88s ago 2m 17.0M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 57fbc4a22e68 2024-03-18T15:17:41.557 INFO:teuthology.orchestra.run.smithi019.stdout:mds.d smithi019 running (3m) 33s ago 3m 15.3M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 4dbf9e96e798 2024-03-18T15:17:41.557 INFO:teuthology.orchestra.run.smithi019.stdout:mds.e smithi033 running (3m) 88s ago 3m 18.2M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 59af355be2ae 2024-03-18T15:17:41.557 INFO:teuthology.orchestra.run.smithi019.stdout:mds.f smithi104 running (2m) 88s ago 2m 16.8M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 19039ff9850c 2024-03-18T15:17:41.558 INFO:teuthology.orchestra.run.smithi019.stdout:mds.g smithi019 running (2m) 33s ago 2m 12.4M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 939a29143c15 2024-03-18T15:17:41.558 INFO:teuthology.orchestra.run.smithi019.stdout:mds.h smithi033 running (2m) 88s ago 2m 16.2M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 10fcda2e0d01 2024-03-18T15:17:41.558 INFO:teuthology.orchestra.run.smithi019.stdout:mds.i smithi104 running (2m) 88s ago 2m 17.3M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 128cfe848767 2024-03-18T15:17:41.558 INFO:teuthology.orchestra.run.smithi019.stdout:mds.j smithi019 running (3m) 33s ago 3m 12.5M - 19.0.0-2162-g3d78f5ba be86dc8cb22b a7249e0bbf37 2024-03-18T15:17:41.558 INFO:teuthology.orchestra.run.smithi019.stdout:mds.k smithi033 running (3m) 88s ago 3m 14.6M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 9e9cbccb6206 2024-03-18T15:17:41.558 INFO:teuthology.orchestra.run.smithi019.stdout:mds.l smithi104 running (3m) 88s ago 3m 17.3M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 4c7a166e007e 2024-03-18T15:17:41.558 INFO:teuthology.orchestra.run.smithi019.stdout:mgr.x smithi019 *:9283,8765 running (11m) 33s ago 11m 514M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 8574ee959649 2024-03-18T15:17:41.558 INFO:teuthology.orchestra.run.smithi019.stdout:mgr.y smithi033 *:8443,8765 running (9m) 88s ago 9m 452M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 1946eb032c4c 2024-03-18T15:17:41.558 INFO:teuthology.orchestra.run.smithi019.stdout:mgr.z smithi104 *:8443,8765 running (9m) 88s ago 9m 452M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 363a73115173 2024-03-18T15:17:41.558 INFO:teuthology.orchestra.run.smithi019.stdout:mon.a smithi019 running (11m) 33s ago 11m 48.5M 2048M 19.0.0-2162-g3d78f5ba be86dc8cb22b 21fe4b08ac45 2024-03-18T15:17:41.558 INFO:teuthology.orchestra.run.smithi019.stdout:mon.b smithi033 running (9m) 88s ago 9m 41.0M 2048M 19.0.0-2162-g3d78f5ba be86dc8cb22b 5d7d00c66554 2024-03-18T15:17:41.558 INFO:teuthology.orchestra.run.smithi019.stdout:mon.c smithi104 running (9m) 88s ago 9m 41.4M 2048M 19.0.0-2162-g3d78f5ba be86dc8cb22b b6b4027b6382 2024-03-18T15:17:41.558 INFO:teuthology.orchestra.run.smithi019.stdout:osd.0 smithi019 running (8m) 33s ago 8m 44.9M 4273M 19.0.0-2162-g3d78f5ba be86dc8cb22b 8fd8596f3203 2024-03-18T15:17:41.558 INFO:teuthology.orchestra.run.smithi019.stdout:osd.1 smithi033 running (8m) 88s ago 8m 60.5M 4271M 19.0.0-2162-g3d78f5ba be86dc8cb22b aaaeef516cad 2024-03-18T15:17:41.559 INFO:teuthology.orchestra.run.smithi019.stdout:osd.2 smithi104 running (7m) 88s ago 7m 46.8M 4270M 19.0.0-2162-g3d78f5ba be86dc8cb22b 76e6f4e2e8c6 2024-03-18T15:17:41.559 INFO:teuthology.orchestra.run.smithi019.stdout:osd.3 smithi019 running (7m) 33s ago 7m 43.0M 4273M 19.0.0-2162-g3d78f5ba be86dc8cb22b e7106c105729 2024-03-18T15:17:41.559 INFO:teuthology.orchestra.run.smithi019.stdout:osd.4 smithi033 running (6m) 88s ago 6m 52.9M 4271M 19.0.0-2162-g3d78f5ba be86dc8cb22b e70bbf90c568 2024-03-18T15:17:41.559 INFO:teuthology.orchestra.run.smithi019.stdout:osd.5 smithi104 running (6m) 88s ago 6m 47.0M 4270M 19.0.0-2162-g3d78f5ba be86dc8cb22b 2af319d597e4 2024-03-18T15:17:41.559 INFO:teuthology.orchestra.run.smithi019.stdout:osd.6 smithi019 running (5m) 33s ago 5m 53.3M 4273M 19.0.0-2162-g3d78f5ba be86dc8cb22b 812afcb2a2c5 2024-03-18T15:17:41.559 INFO:teuthology.orchestra.run.smithi019.stdout:osd.7 smithi033 running (5m) 88s ago 5m 44.1M 4271M 19.0.0-2162-g3d78f5ba be86dc8cb22b 32f063c62870 2024-03-18T15:17:41.559 INFO:teuthology.orchestra.run.smithi019.stdout:osd.8 smithi104 running (5m) 88s ago 5m 53.3M 4270M 19.0.0-2162-g3d78f5ba be86dc8cb22b b039a05751e2 2024-03-18T15:17:41.559 INFO:teuthology.orchestra.run.smithi019.stdout:osd.9 smithi019 running (4m) 33s ago 4m 53.3M 4273M 19.0.0-2162-g3d78f5ba be86dc8cb22b a3f72f1982ef 2024-03-18T15:17:41.559 INFO:teuthology.orchestra.run.smithi019.stdout:osd.10 smithi033 running (4m) 88s ago 4m 43.6M 4271M 19.0.0-2162-g3d78f5ba be86dc8cb22b b597756955a7 2024-03-18T15:17:41.559 INFO:teuthology.orchestra.run.smithi019.stdout:osd.11 smithi104 running (3m) 88s ago 3m 44.2M 4270M 19.0.0-2162-g3d78f5ba be86dc8cb22b 8eaa5539e49e 2024-03-18T15:17:41.560 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:41.558+0000 7fa71ea00700 1 -- 172.21.15.19:0/2544695967 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fa6fc0a2890 msgr2=0x7fa6fc0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:41.560 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:41.558+0000 7fa71ea00700 1 --2- 172.21.15.19:0/2544695967 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fa6fc0a2890 0x7fa6fc0a4d50 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fa718103360 tx=0x7fa70800b670 comp rx=0 tx=0).stop 2024-03-18T15:17:41.560 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:41.558+0000 7fa71ea00700 1 -- 172.21.15.19:0/2544695967 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa718103500 msgr2=0x7fa718198a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:41.560 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:41.558+0000 7fa71ea00700 1 --2- 172.21.15.19:0/2544695967 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa718103500 0x7fa718198a10 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fa70400e9d0 tx=0x7fa70400ece0 comp rx=0 tx=0).stop 2024-03-18T15:17:41.560 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:41.558+0000 7fa71ea00700 1 -- 172.21.15.19:0/2544695967 shutdown_connections 2024-03-18T15:17:41.560 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:41.558+0000 7fa71ea00700 1 --2- 172.21.15.19:0/2544695967 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fa6fc0a2890 0x7fa6fc0a4d50 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:41.560 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:41.558+0000 7fa71ea00700 1 --2- 172.21.15.19:0/2544695967 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa718103ea0 0x7fa71819fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:41.560 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:41.558+0000 7fa71ea00700 1 --2- 172.21.15.19:0/2544695967 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa718102300 0x7fa7181984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:41.560 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:41.558+0000 7fa71ea00700 1 --2- 172.21.15.19:0/2544695967 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa718103500 0x7fa718198a10 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:41.560 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:41.558+0000 7fa71ea00700 1 -- 172.21.15.19:0/2544695967 >> 172.21.15.19:0/2544695967 conn(0x7fa7180fd8e0 msgr2=0x7fa718106800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:17:41.560 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:41.558+0000 7fa71ea00700 1 -- 172.21.15.19:0/2544695967 shutdown_connections 2024-03-18T15:17:41.561 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:41.558+0000 7fa71ea00700 1 -- 172.21.15.19:0/2544695967 wait complete. 2024-03-18T15:17:41.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:17:41 smithi033 ceph-mon[36648]: pgmap v396: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T15:17:41.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:17:41 smithi104 ceph-mon[36070]: pgmap v396: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T15:17:41.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:17:41 smithi019 ceph-mon[31522]: pgmap v396: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T15:17:42.350 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 -- bash -c 'ceph orch ls' 2024-03-18T15:17:42.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:17:42 smithi019 ceph-mon[31522]: from='client.25237 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:17:42.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:17:42 smithi033 ceph-mon[36648]: from='client.25237 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:17:42.821 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:17:42.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:17:42 smithi104 ceph-mon[36070]: from='client.25237 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:17:43.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:17:43 smithi019 ceph-mon[31522]: pgmap v397: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T15:17:43.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:17:43 smithi033 ceph-mon[36648]: pgmap v397: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T15:17:43.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:17:43 smithi104 ceph-mon[36070]: pgmap v397: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T15:17:44.946 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:44.944+0000 7f640ec00700 1 -- 172.21.15.19:0/863283324 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6408102300 msgr2=0x7f6408102720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:44.947 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:44.944+0000 7f640ec00700 1 --2- 172.21.15.19:0/863283324 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6408102300 0x7f6408102720 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f63f8009990 tx=0x7f63f8009ca0 comp rx=0 tx=0).stop 2024-03-18T15:17:44.947 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:44.945+0000 7f640ec00700 1 -- 172.21.15.19:0/863283324 shutdown_connections 2024-03-18T15:17:44.947 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:44.945+0000 7f640ec00700 1 --2- 172.21.15.19:0/863283324 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6408102300 0x7f6408102720 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:44.947 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:44.945+0000 7f640ec00700 1 --2- 172.21.15.19:0/863283324 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6408103ea0 0x7f640810a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:44.947 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:44.945+0000 7f640ec00700 1 --2- 172.21.15.19:0/863283324 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6408103500 0x7f6408103960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:44.947 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:44.945+0000 7f640ec00700 1 -- 172.21.15.19:0/863283324 >> 172.21.15.19:0/863283324 conn(0x7f64080fd8e0 msgr2=0x7f64080ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:17:44.947 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:44.945+0000 7f640ec00700 1 -- 172.21.15.19:0/863283324 shutdown_connections 2024-03-18T15:17:44.947 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:44.945+0000 7f640ec00700 1 -- 172.21.15.19:0/863283324 wait complete. 2024-03-18T15:17:44.949 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:44.946+0000 7f640ec00700 1 Processor -- start 2024-03-18T15:17:44.949 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:44.946+0000 7f640ec00700 1 -- start start 2024-03-18T15:17:44.949 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:44.947+0000 7f640ec00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6408102300 0x7f6408198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:44.949 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:44.947+0000 7f640ec00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6408103500 0x7f6408198ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:44.949 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:44.947+0000 7f640ec00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6408103ea0 0x7f640819fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:44.949 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:44.947+0000 7f640ec00700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f640810df20 con 0x7f6408102300 2024-03-18T15:17:44.949 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:44.947+0000 7f640ec00700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f640810dda0 con 0x7f6408103500 2024-03-18T15:17:44.949 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:44.947+0000 7f640ec00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f640810e0a0 con 0x7f6408103ea0 2024-03-18T15:17:44.949 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:44.947+0000 7f63ffe00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6408103500 0x7f6408198ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:44.949 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:44.947+0000 7f6407e00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6408102300 0x7f6408198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:44.949 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:44.947+0000 7f63ffe00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6408103500 0x7f6408198ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:46940/0 (socket says 172.21.15.19:46940) 2024-03-18T15:17:44.949 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:44.947+0000 7f63ffe00700 1 -- 172.21.15.19:0/1215347180 learned_addr learned my addr 172.21.15.19:0/1215347180 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:17:44.950 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:44.947+0000 7f640cc00700 1 --2- 172.21.15.19:0/1215347180 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6408103ea0 0x7f640819fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:44.950 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:44.948+0000 7f63ffe00700 1 -- 172.21.15.19:0/1215347180 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6408103ea0 msgr2=0x7f640819fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:44.950 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:44.948+0000 7f63ffe00700 1 --2- 172.21.15.19:0/1215347180 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6408103ea0 0x7f640819fb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:44.950 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:44.948+0000 7f63ffe00700 1 -- 172.21.15.19:0/1215347180 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6408102300 msgr2=0x7f6408198590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:44.950 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:44.948+0000 7f63ffe00700 1 --2- 172.21.15.19:0/1215347180 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6408102300 0x7f6408198590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:44.950 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:44.948+0000 7f63ffe00700 1 -- 172.21.15.19:0/1215347180 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64081a01f0 con 0x7f6408103500 2024-03-18T15:17:44.950 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:44.948+0000 7f63ffe00700 1 --2- 172.21.15.19:0/1215347180 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6408103500 0x7f6408198ad0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f63f000e9a0 tx=0x7f63f000ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:17:44.951 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:44.948+0000 7f6405600700 1 -- 172.21.15.19:0/1215347180 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63f0020070 con 0x7f6408103500 2024-03-18T15:17:44.951 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:44.948+0000 7f6405600700 1 -- 172.21.15.19:0/1215347180 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f63f0013d00 con 0x7f6408103500 2024-03-18T15:17:44.951 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:44.948+0000 7f640ec00700 1 -- 172.21.15.19:0/1215347180 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6408074560 con 0x7f6408103500 2024-03-18T15:17:44.951 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:44.948+0000 7f6405600700 1 -- 172.21.15.19:0/1215347180 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63f001b900 con 0x7f6408103500 2024-03-18T15:17:44.951 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:44.949+0000 7f640ec00700 1 -- 172.21.15.19:0/1215347180 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6408074aa0 con 0x7f6408103500 2024-03-18T15:17:44.951 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:44.950+0000 7f640ec00700 1 -- 172.21.15.19:0/1215347180 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6408068f90 con 0x7f6408103500 2024-03-18T15:17:44.952 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:44.951+0000 7f6405600700 1 -- 172.21.15.19:0/1215347180 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f63f0023070 con 0x7f6408103500 2024-03-18T15:17:44.954 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:44.953+0000 7f6405600700 1 --2- 172.21.15.19:0/1215347180 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f63ec0a2890 0x7f63ec0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:44.954 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:44.953+0000 7f6405600700 1 -- 172.21.15.19:0/1215347180 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f63f00d2da0 con 0x7f6408103500 2024-03-18T15:17:44.954 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:44.953+0000 7f6407e00700 1 --2- 172.21.15.19:0/1215347180 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f63ec0a2890 0x7f63ec0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:44.954 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:44.954+0000 7f6407e00700 1 --2- 172.21.15.19:0/1215347180 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f63ec0a2890 0x7f63ec0a4d50 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f63f8000c00 tx=0x7f63f8005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:17:44.963 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:44.962+0000 7f6405600700 1 -- 172.21.15.19:0/1215347180 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f63f009fa10 con 0x7f6408103500 2024-03-18T15:17:45.207 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:45.206+0000 7f640ec00700 1 -- 172.21.15.19:0/1215347180 --> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7f6408062710 con 0x7f63ec0a2890 2024-03-18T15:17:45.226 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:45.225+0000 7f6405600700 1 -- 172.21.15.19:0/1215347180 <== mgr.14152 v2:172.21.15.19:6800/2017714236 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7f6408062710 con 0x7f63ec0a2890 2024-03-18T15:17:45.226 INFO:teuthology.orchestra.run.smithi019.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-18T15:17:45.226 INFO:teuthology.orchestra.run.smithi019.stdout:mds.all 12/12 92s ago 3m smithi019=a;smithi019=d;smithi019=g;smithi019=j;smithi033=b;smithi033=e;smithi033=h;smithi033=k;smithi104=c;smithi104=f;smithi104=i;smithi104=l;count:12 2024-03-18T15:17:45.226 INFO:teuthology.orchestra.run.smithi019.stdout:mgr 3/3 92s ago 9m smithi019=x;smithi033=y;smithi104=z;count:3 2024-03-18T15:17:45.226 INFO:teuthology.orchestra.run.smithi019.stdout:mon 3/3 92s ago 9m smithi019:172.21.15.19=a;smithi033:172.21.15.33=b;smithi104:172.21.15.104=c;count:3 2024-03-18T15:17:45.226 INFO:teuthology.orchestra.run.smithi019.stdout:osd 12 92s ago - 2024-03-18T15:17:45.229 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:45.228+0000 7f640ec00700 1 -- 172.21.15.19:0/1215347180 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f63ec0a2890 msgr2=0x7f63ec0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:45.229 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:45.228+0000 7f640ec00700 1 --2- 172.21.15.19:0/1215347180 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f63ec0a2890 0x7f63ec0a4d50 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f63f8000c00 tx=0x7f63f8005fb0 comp rx=0 tx=0).stop 2024-03-18T15:17:45.229 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:45.228+0000 7f640ec00700 1 -- 172.21.15.19:0/1215347180 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6408103500 msgr2=0x7f6408198ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:45.229 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:45.228+0000 7f640ec00700 1 --2- 172.21.15.19:0/1215347180 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6408103500 0x7f6408198ad0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f63f000e9a0 tx=0x7f63f000ecb0 comp rx=0 tx=0).stop 2024-03-18T15:17:45.230 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:45.228+0000 7f640ec00700 1 -- 172.21.15.19:0/1215347180 shutdown_connections 2024-03-18T15:17:45.230 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:45.228+0000 7f640ec00700 1 --2- 172.21.15.19:0/1215347180 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f63ec0a2890 0x7f63ec0a4d50 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:45.230 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:45.228+0000 7f640ec00700 1 --2- 172.21.15.19:0/1215347180 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6408103ea0 0x7f640819fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:45.230 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:45.228+0000 7f640ec00700 1 --2- 172.21.15.19:0/1215347180 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6408102300 0x7f6408198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:45.230 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:45.228+0000 7f640ec00700 1 --2- 172.21.15.19:0/1215347180 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6408103500 0x7f6408198ad0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:45.230 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:45.228+0000 7f640ec00700 1 -- 172.21.15.19:0/1215347180 >> 172.21.15.19:0/1215347180 conn(0x7f64080fd8e0 msgr2=0x7f6408106920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:17:45.230 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:45.229+0000 7f640ec00700 1 -- 172.21.15.19:0/1215347180 shutdown_connections 2024-03-18T15:17:45.230 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:45.229+0000 7f640ec00700 1 -- 172.21.15.19:0/1215347180 wait complete. 2024-03-18T15:17:45.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:17:45 smithi033 ceph-mon[36648]: pgmap v398: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T15:17:45.924 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:17:45 smithi019 ceph-mon[31522]: pgmap v398: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T15:17:45.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:17:45 smithi104 ceph-mon[36070]: pgmap v398: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T15:17:46.177 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 -- bash -c 'ceph orch host ls' 2024-03-18T15:17:46.644 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:17:46.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:17:46 smithi019 ceph-mon[31522]: from='client.25232 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:17:46.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:17:46 smithi033 ceph-mon[36648]: from='client.25232 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:17:46.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:17:46 smithi104 ceph-mon[36070]: from='client.25232 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:17:47.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:17:47 smithi033 ceph-mon[36648]: pgmap v399: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T15:17:47.827 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:17:47 smithi019 ceph-mon[31522]: pgmap v399: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T15:17:47.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:17:47 smithi104 ceph-mon[36070]: pgmap v399: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T15:17:48.972 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:48.970+0000 7f45b9e00700 1 -- 172.21.15.19:0/3692872166 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f45b4103500 msgr2=0x7f45b4103960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:48.972 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:48.970+0000 7f45b9e00700 1 --2- 172.21.15.19:0/3692872166 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f45b4103500 0x7f45b4103960 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f45a00099c0 tx=0x7f45a0009cd0 comp rx=0 tx=0).stop 2024-03-18T15:17:48.972 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:48.970+0000 7f45b9e00700 1 -- 172.21.15.19:0/3692872166 shutdown_connections 2024-03-18T15:17:48.972 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:48.970+0000 7f45b9e00700 1 --2- 172.21.15.19:0/3692872166 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f45b4103ea0 0x7f45b410a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:48.973 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:48.970+0000 7f45b9e00700 1 --2- 172.21.15.19:0/3692872166 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f45b4102300 0x7f45b4102720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:48.973 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:48.970+0000 7f45b9e00700 1 --2- 172.21.15.19:0/3692872166 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f45b4103500 0x7f45b4103960 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:48.973 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:48.970+0000 7f45b9e00700 1 -- 172.21.15.19:0/3692872166 >> 172.21.15.19:0/3692872166 conn(0x7f45b40fd8e0 msgr2=0x7f45b40ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:17:48.973 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:48.970+0000 7f45b9e00700 1 -- 172.21.15.19:0/3692872166 shutdown_connections 2024-03-18T15:17:48.973 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:48.971+0000 7f45b9e00700 1 -- 172.21.15.19:0/3692872166 wait complete. 2024-03-18T15:17:48.974 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:48.972+0000 7f45b9e00700 1 Processor -- start 2024-03-18T15:17:48.974 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:48.972+0000 7f45b9e00700 1 -- start start 2024-03-18T15:17:48.974 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:48.973+0000 7f45b9e00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f45b4102300 0x7f45b419a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:48.974 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:48.973+0000 7f45b9e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f45b4103500 0x7f45b419ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:48.974 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:48.973+0000 7f45b9e00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f45b4103ea0 0x7f45b41a1d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:48.974 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:48.973+0000 7f45b9e00700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f45b410df20 con 0x7f45b4103ea0 2024-03-18T15:17:48.974 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:48.973+0000 7f45b9e00700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f45b410dda0 con 0x7f45b4103500 2024-03-18T15:17:48.974 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:48.973+0000 7f45b9e00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f45b410e0a0 con 0x7f45b4102300 2024-03-18T15:17:48.974 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:48.973+0000 7f45b2000700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f45b4103500 0x7f45b419ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:48.974 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:48.973+0000 7f45b2a00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f45b4102300 0x7f45b419a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:48.974 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:48.973+0000 7f45b2000700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f45b4103500 0x7f45b419ac80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:46960/0 (socket says 172.21.15.19:46960) 2024-03-18T15:17:48.975 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:48.973+0000 7f45b2000700 1 -- 172.21.15.19:0/3802458365 learned_addr learned my addr 172.21.15.19:0/3802458365 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:17:48.975 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:48.973+0000 7f45b3400700 1 --2- 172.21.15.19:0/3802458365 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f45b4103ea0 0x7f45b41a1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:48.975 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:48.973+0000 7f45b2000700 1 -- 172.21.15.19:0/3802458365 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f45b4102300 msgr2=0x7f45b419a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:48.975 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:48.973+0000 7f45b2000700 1 --2- 172.21.15.19:0/3802458365 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f45b4102300 0x7f45b419a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:48.976 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:48.973+0000 7f45b2000700 1 -- 172.21.15.19:0/3802458365 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f45b4103ea0 msgr2=0x7f45b41a1d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:48.976 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:48.973+0000 7f45b2000700 1 --2- 172.21.15.19:0/3802458365 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f45b4103ea0 0x7f45b41a1d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:48.976 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:48.973+0000 7f45b2000700 1 -- 172.21.15.19:0/3802458365 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45b41a23a0 con 0x7f45b4103500 2024-03-18T15:17:48.976 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:48.973+0000 7f45b3400700 1 --2- 172.21.15.19:0/3802458365 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f45b4103ea0 0x7f45b41a1d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T15:17:48.976 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:48.974+0000 7f45b2000700 1 --2- 172.21.15.19:0/3802458365 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f45b4103500 0x7f45b419ac80 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f45a0003790 tx=0x7f45a000bee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:17:48.976 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:48.974+0000 7f45b8a00700 1 -- 172.21.15.19:0/3802458365 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45a0009e60 con 0x7f45b4103500 2024-03-18T15:17:48.976 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:48.974+0000 7f45b8a00700 1 -- 172.21.15.19:0/3802458365 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f45a000f460 con 0x7f45b4103500 2024-03-18T15:17:48.976 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:48.974+0000 7f45b9e00700 1 -- 172.21.15.19:0/3802458365 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45b41a2630 con 0x7f45b4103500 2024-03-18T15:17:48.976 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:48.974+0000 7f45b8a00700 1 -- 172.21.15.19:0/3802458365 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45a0017610 con 0x7f45b4103500 2024-03-18T15:17:48.976 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:48.974+0000 7f45b9e00700 1 -- 172.21.15.19:0/3802458365 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45b41a2ae0 con 0x7f45b4103500 2024-03-18T15:17:48.976 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:48.975+0000 7f45b9e00700 1 -- 172.21.15.19:0/3802458365 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45b4068f90 con 0x7f45b4103500 2024-03-18T15:17:48.978 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:48.977+0000 7f45b8a00700 1 -- 172.21.15.19:0/3802458365 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f45a000f610 con 0x7f45b4103500 2024-03-18T15:17:48.979 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:48.978+0000 7f45b8a00700 1 --2- 172.21.15.19:0/3802458365 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f45980a2910 0x7f45980a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:48.979 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:48.979+0000 7f45b8a00700 1 -- 172.21.15.19:0/3802458365 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f45a00cfbf0 con 0x7f45b4103500 2024-03-18T15:17:48.980 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:48.979+0000 7f45b2a00700 1 --2- 172.21.15.19:0/3802458365 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f45980a2910 0x7f45980a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:48.980 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:48.979+0000 7f45b2a00700 1 --2- 172.21.15.19:0/3802458365 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f45980a2910 0x7f45980a4dd0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f45b4103360 tx=0x7f45a4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:17:48.984 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:48.983+0000 7f45b8a00700 1 -- 172.21.15.19:0/3802458365 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f45a0013070 con 0x7f45b4103500 2024-03-18T15:17:49.224 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:49.223+0000 7f45b9e00700 1 -- 172.21.15.19:0/3802458365 --> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7f45b4075c20 con 0x7f45980a2910 2024-03-18T15:17:49.227 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:49.226+0000 7f45b8a00700 1 -- 172.21.15.19:0/3802458365 <== mgr.14152 v2:172.21.15.19:6800/2017714236 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7f45b4075c20 con 0x7f45980a2910 2024-03-18T15:17:49.227 INFO:teuthology.orchestra.run.smithi019.stdout:HOST ADDR LABELS STATUS 2024-03-18T15:17:49.227 INFO:teuthology.orchestra.run.smithi019.stdout:smithi019 172.21.15.19 2024-03-18T15:17:49.227 INFO:teuthology.orchestra.run.smithi019.stdout:smithi033 172.21.15.33 2024-03-18T15:17:49.228 INFO:teuthology.orchestra.run.smithi019.stdout:smithi104 172.21.15.104 2024-03-18T15:17:49.228 INFO:teuthology.orchestra.run.smithi019.stdout:3 hosts in cluster 2024-03-18T15:17:49.231 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:49.229+0000 7f45b9e00700 1 -- 172.21.15.19:0/3802458365 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f45980a2910 msgr2=0x7f45980a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:49.231 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:49.229+0000 7f45b9e00700 1 --2- 172.21.15.19:0/3802458365 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f45980a2910 0x7f45980a4dd0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f45b4103360 tx=0x7f45a4009260 comp rx=0 tx=0).stop 2024-03-18T15:17:49.231 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:49.229+0000 7f45b9e00700 1 -- 172.21.15.19:0/3802458365 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f45b4103500 msgr2=0x7f45b419ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:49.231 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:49.229+0000 7f45b9e00700 1 --2- 172.21.15.19:0/3802458365 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f45b4103500 0x7f45b419ac80 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f45a0003790 tx=0x7f45a000bee0 comp rx=0 tx=0).stop 2024-03-18T15:17:49.231 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:49.230+0000 7f45b9e00700 1 -- 172.21.15.19:0/3802458365 shutdown_connections 2024-03-18T15:17:49.231 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:49.230+0000 7f45b9e00700 1 --2- 172.21.15.19:0/3802458365 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f45980a2910 0x7f45980a4dd0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:49.231 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:49.230+0000 7f45b9e00700 1 --2- 172.21.15.19:0/3802458365 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f45b4102300 0x7f45b419a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:49.231 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:49.230+0000 7f45b9e00700 1 --2- 172.21.15.19:0/3802458365 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f45b4103ea0 0x7f45b41a1d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:49.231 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:49.230+0000 7f45b9e00700 1 --2- 172.21.15.19:0/3802458365 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f45b4103500 0x7f45b419ac80 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:49.231 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:49.230+0000 7f45b9e00700 1 -- 172.21.15.19:0/3802458365 >> 172.21.15.19:0/3802458365 conn(0x7f45b40fd8e0 msgr2=0x7f45b4074500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:17:49.231 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:49.230+0000 7f45b9e00700 1 -- 172.21.15.19:0/3802458365 shutdown_connections 2024-03-18T15:17:49.232 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:49.230+0000 7f45b9e00700 1 -- 172.21.15.19:0/3802458365 wait complete. 2024-03-18T15:17:49.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:17:49 smithi104 ceph-mon[36070]: pgmap v400: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T15:17:49.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:17:49 smithi019 ceph-mon[31522]: pgmap v400: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T15:17:50.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:17:49 smithi033 ceph-mon[36648]: pgmap v400: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T15:17:50.194 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 -- bash -c 'ceph orch device ls' 2024-03-18T15:17:50.684 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:17:50.840 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:17:50 smithi019 ceph-mon[31522]: from='client.25238 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:17:50.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:17:50 smithi104 ceph-mon[36070]: from='client.25238 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:17:51.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:17:50 smithi033 ceph-mon[36648]: from='client.25238 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:17:51.887 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:17:51 smithi019 ceph-mon[31522]: pgmap v401: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T15:17:51.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:17:51 smithi104 ceph-mon[36070]: pgmap v401: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T15:17:52.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:17:51 smithi033 ceph-mon[36648]: pgmap v401: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T15:17:53.014 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:53.012+0000 7f7d43e00700 1 -- 172.21.15.19:0/659492999 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7d3c074d30 msgr2=0x7f7d3c0751b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:53.015 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:53.012+0000 7f7d43e00700 1 --2- 172.21.15.19:0/659492999 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7d3c074d30 0x7f7d3c0751b0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f7d280098f0 tx=0x7f7d28009c00 comp rx=0 tx=0).stop 2024-03-18T15:17:53.015 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:53.012+0000 7f7d43e00700 1 -- 172.21.15.19:0/659492999 shutdown_connections 2024-03-18T15:17:53.015 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:53.012+0000 7f7d43e00700 1 --2- 172.21.15.19:0/659492999 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d3c10e8e0 0x7f7d3c110cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:53.015 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:53.012+0000 7f7d43e00700 1 --2- 172.21.15.19:0/659492999 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7d3c076300 0x7f7d3c074760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:53.015 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:53.012+0000 7f7d43e00700 1 --2- 172.21.15.19:0/659492999 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7d3c074d30 0x7f7d3c0751b0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:53.015 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:53.012+0000 7f7d43e00700 1 -- 172.21.15.19:0/659492999 >> 172.21.15.19:0/659492999 conn(0x7f7d3c0fd8e0 msgr2=0x7f7d3c0ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:17:53.015 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:53.013+0000 7f7d43e00700 1 -- 172.21.15.19:0/659492999 shutdown_connections 2024-03-18T15:17:53.015 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:53.013+0000 7f7d43e00700 1 -- 172.21.15.19:0/659492999 wait complete. 2024-03-18T15:17:53.016 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:53.014+0000 7f7d43e00700 1 Processor -- start 2024-03-18T15:17:53.017 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:53.015+0000 7f7d43e00700 1 -- start start 2024-03-18T15:17:53.017 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:53.015+0000 7f7d43e00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7d3c074d30 0x7f7d3c19ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:53.017 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:53.015+0000 7f7d43e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7d3c076300 0x7f7d3c19f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:53.017 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:53.015+0000 7f7d43e00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d3c10e8e0 0x7f7d3c1a61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:53.017 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:53.015+0000 7f7d43e00700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f7d3c114530 con 0x7f7d3c074d30 2024-03-18T15:17:53.017 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:53.015+0000 7f7d43e00700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7d3c1143b0 con 0x7f7d3c076300 2024-03-18T15:17:53.017 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:53.015+0000 7f7d43e00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f7d3c1146b0 con 0x7f7d3c10e8e0 2024-03-18T15:17:53.017 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:53.015+0000 7f7d40a00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7d3c076300 0x7f7d3c19f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:53.017 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:53.015+0000 7f7d41400700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7d3c074d30 0x7f7d3c19ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:53.017 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:53.015+0000 7f7d40a00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7d3c076300 0x7f7d3c19f180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:34134/0 (socket says 172.21.15.19:34134) 2024-03-18T15:17:53.017 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:53.015+0000 7f7d40a00700 1 -- 172.21.15.19:0/3093505646 learned_addr learned my addr 172.21.15.19:0/3093505646 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:17:53.017 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:53.016+0000 7f7d41e00700 1 --2- 172.21.15.19:0/3093505646 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d3c10e8e0 0x7f7d3c1a61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:53.018 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:53.016+0000 7f7d40a00700 1 -- 172.21.15.19:0/3093505646 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d3c10e8e0 msgr2=0x7f7d3c1a61c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:53.018 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:53.016+0000 7f7d40a00700 1 --2- 172.21.15.19:0/3093505646 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d3c10e8e0 0x7f7d3c1a61c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:53.018 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:53.016+0000 7f7d40a00700 1 -- 172.21.15.19:0/3093505646 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7d3c074d30 msgr2=0x7f7d3c19ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:53.018 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:53.016+0000 7f7d40a00700 1 --2- 172.21.15.19:0/3093505646 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7d3c074d30 0x7f7d3c19ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:53.018 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:53.016+0000 7f7d40a00700 1 -- 172.21.15.19:0/3093505646 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d3c1a6890 con 0x7f7d3c076300 2024-03-18T15:17:53.019 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:53.016+0000 7f7d40a00700 1 --2- 172.21.15.19:0/3093505646 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7d3c076300 0x7f7d3c19f180 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f7d2800fca0 tx=0x7f7d28005250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:17:53.019 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:53.016+0000 7f7d3a000700 1 -- 172.21.15.19:0/3093505646 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d2800f950 con 0x7f7d3c076300 2024-03-18T15:17:53.019 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:53.017+0000 7f7d3a000700 1 -- 172.21.15.19:0/3093505646 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7d2800ba40 con 0x7f7d3c076300 2024-03-18T15:17:53.019 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:53.017+0000 7f7d3a000700 1 -- 172.21.15.19:0/3093505646 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d28003c80 con 0x7f7d3c076300 2024-03-18T15:17:53.019 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:53.017+0000 7f7d43e00700 1 -- 172.21.15.19:0/3093505646 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d3c1a6b20 con 0x7f7d3c076300 2024-03-18T15:17:53.019 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:53.017+0000 7f7d43e00700 1 -- 172.21.15.19:0/3093505646 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d3c1a6f20 con 0x7f7d3c076300 2024-03-18T15:17:53.019 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:53.018+0000 7f7d43e00700 1 -- 172.21.15.19:0/3093505646 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d3c02db70 con 0x7f7d3c076300 2024-03-18T15:17:53.021 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:53.020+0000 7f7d3a000700 1 -- 172.21.15.19:0/3093505646 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f7d2800bbf0 con 0x7f7d3c076300 2024-03-18T15:17:53.022 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:53.021+0000 7f7d3a000700 1 --2- 172.21.15.19:0/3093505646 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f7d240a2ab0 0x7f7d240a4f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:53.022 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:53.021+0000 7f7d3a000700 1 -- 172.21.15.19:0/3093505646 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f7d280cfec0 con 0x7f7d3c076300 2024-03-18T15:17:53.022 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:53.021+0000 7f7d41400700 1 --2- 172.21.15.19:0/3093505646 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f7d240a2ab0 0x7f7d240a4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:53.023 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:53.022+0000 7f7d41400700 1 --2- 172.21.15.19:0/3093505646 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f7d240a2ab0 0x7f7d240a4f70 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f7d3c1a0120 tx=0x7f7d30009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:17:53.026 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:53.025+0000 7f7d3a000700 1 -- 172.21.15.19:0/3093505646 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7d28013030 con 0x7f7d3c076300 2024-03-18T15:17:53.260 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:53.259+0000 7f7d43e00700 1 -- 172.21.15.19:0/3093505646 --> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7f7d3c062710 con 0x7f7d240a2ab0 2024-03-18T15:17:53.264 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:53.263+0000 7f7d3a000700 1 -- 172.21.15.19:0/3093505646 <== mgr.14152 v2:172.21.15.19:6800/2017714236 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7f7d3c062710 con 0x7f7d240a2ab0 2024-03-18T15:17:53.264 INFO:teuthology.orchestra.run.smithi019.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-03-18T15:17:53.264 INFO:teuthology.orchestra.run.smithi019.stdout:smithi019 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT5330002G400BGN 372G No 4m ago Has a FileSystem, LVM detected 2024-03-18T15:17:53.264 INFO:teuthology.orchestra.run.smithi019.stdout:smithi033 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT5330008Q400BGN 372G No 4m ago Has a FileSystem, LVM detected 2024-03-18T15:17:53.264 INFO:teuthology.orchestra.run.smithi019.stdout:smithi104 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400NW400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-03-18T15:17:53.267 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:53.265+0000 7f7d43e00700 1 -- 172.21.15.19:0/3093505646 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f7d240a2ab0 msgr2=0x7f7d240a4f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:53.267 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:53.265+0000 7f7d43e00700 1 --2- 172.21.15.19:0/3093505646 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f7d240a2ab0 0x7f7d240a4f70 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f7d3c1a0120 tx=0x7f7d30009260 comp rx=0 tx=0).stop 2024-03-18T15:17:53.267 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:53.265+0000 7f7d43e00700 1 -- 172.21.15.19:0/3093505646 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7d3c076300 msgr2=0x7f7d3c19f180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:53.267 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:53.265+0000 7f7d43e00700 1 --2- 172.21.15.19:0/3093505646 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7d3c076300 0x7f7d3c19f180 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f7d2800fca0 tx=0x7f7d28005250 comp rx=0 tx=0).stop 2024-03-18T15:17:53.267 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:53.265+0000 7f7d43e00700 1 -- 172.21.15.19:0/3093505646 shutdown_connections 2024-03-18T15:17:53.267 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:53.265+0000 7f7d43e00700 1 --2- 172.21.15.19:0/3093505646 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f7d240a2ab0 0x7f7d240a4f70 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:53.267 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:53.265+0000 7f7d43e00700 1 --2- 172.21.15.19:0/3093505646 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d3c10e8e0 0x7f7d3c1a61c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:53.267 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:53.265+0000 7f7d43e00700 1 --2- 172.21.15.19:0/3093505646 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7d3c074d30 0x7f7d3c19ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:53.267 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:53.265+0000 7f7d43e00700 1 --2- 172.21.15.19:0/3093505646 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7d3c076300 0x7f7d3c19f180 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:53.267 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:53.265+0000 7f7d43e00700 1 -- 172.21.15.19:0/3093505646 >> 172.21.15.19:0/3093505646 conn(0x7f7d3c0fd8e0 msgr2=0x7f7d3c101c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:17:53.267 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:53.266+0000 7f7d43e00700 1 -- 172.21.15.19:0/3093505646 shutdown_connections 2024-03-18T15:17:53.267 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:53.266+0000 7f7d43e00700 1 -- 172.21.15.19:0/3093505646 wait complete. 2024-03-18T15:17:53.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:17:53 smithi104 ceph-mon[36070]: pgmap v402: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T15:17:53.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:17:53 smithi019 ceph-mon[31522]: pgmap v402: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T15:17:54.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:17:53 smithi033 ceph-mon[36648]: pgmap v402: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T15:17:54.253 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-18T15:17:54.262 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi019.front.sepia.ceph.com 2024-03-18T15:17:54.262 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 -- bash -c 'ceph fs dump' 2024-03-18T15:17:54.735 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:17:54.881 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:17:54 smithi019 ceph-mon[31522]: from='client.25244 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:17:54.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:17:54 smithi104 ceph-mon[36070]: from='client.25244 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:17:55.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:17:54 smithi033 ceph-mon[36648]: from='client.25244 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:17:55.794 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:17:55 smithi019 ceph-mon[31522]: pgmap v403: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T15:17:55.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:17:55 smithi104 ceph-mon[36070]: pgmap v403: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T15:17:56.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:17:55 smithi033 ceph-mon[36648]: pgmap v403: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T15:17:57.051 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:57.050+0000 7f312e000700 1 -- 172.21.15.19:0/3596452389 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3128102d60 msgr2=0x7f3128105150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:57.051 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:57.050+0000 7f312e000700 1 --2- 172.21.15.19:0/3596452389 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3128102d60 0x7f3128105150 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f311800b0a0 tx=0x7f311800b3b0 comp rx=0 tx=0).stop 2024-03-18T15:17:57.051 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:57.050+0000 7f312e000700 1 -- 172.21.15.19:0/3596452389 shutdown_connections 2024-03-18T15:17:57.051 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:57.050+0000 7f312e000700 1 --2- 172.21.15.19:0/3596452389 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3128102d60 0x7f3128105150 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:57.051 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:57.050+0000 7f312e000700 1 --2- 172.21.15.19:0/3596452389 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3128107fc0 0x7f312810a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:57.052 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:57.050+0000 7f312e000700 1 --2- 172.21.15.19:0/3596452389 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3128105690 0x7f3128107a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:57.052 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:57.050+0000 7f312e000700 1 -- 172.21.15.19:0/3596452389 >> 172.21.15.19:0/3596452389 conn(0x7f31280fc770 msgr2=0x7f31280febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:17:57.052 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:57.050+0000 7f312e000700 1 -- 172.21.15.19:0/3596452389 shutdown_connections 2024-03-18T15:17:57.052 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:57.050+0000 7f312e000700 1 -- 172.21.15.19:0/3596452389 wait complete. 2024-03-18T15:17:57.054 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:57.051+0000 7f312e000700 1 Processor -- start 2024-03-18T15:17:57.054 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:57.051+0000 7f312e000700 1 -- start start 2024-03-18T15:17:57.054 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:57.051+0000 7f312e000700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3128102d60 0x7f312819a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:57.054 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:57.052+0000 7f312e000700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3128105690 0x7f312819ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:57.054 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:57.052+0000 7f312e000700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3128107fc0 0x7f31281a1cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:57.054 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:57.052+0000 7f312e000700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f312810def0 con 0x7f3128105690 2024-03-18T15:17:57.054 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:57.052+0000 7f312e000700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f312810dd70 con 0x7f3128107fc0 2024-03-18T15:17:57.055 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:57.052+0000 7f312e000700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f312810e070 con 0x7f3128102d60 2024-03-18T15:17:57.055 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:57.052+0000 7f3127400700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3128102d60 0x7f312819a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:57.055 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:57.052+0000 7f3127400700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3128102d60 0x7f312819a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.19:56520/0 (socket says 172.21.15.19:56520) 2024-03-18T15:17:57.055 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:57.052+0000 7f3127400700 1 -- 172.21.15.19:0/2323959795 learned_addr learned my addr 172.21.15.19:0/2323959795 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:17:57.055 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:57.052+0000 7f311ea00700 1 --2- 172.21.15.19:0/2323959795 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3128105690 0x7f312819ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:57.055 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:57.052+0000 7f3127e00700 1 --2- 172.21.15.19:0/2323959795 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3128107fc0 0x7f31281a1cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:57.055 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:57.052+0000 7f3127400700 1 -- 172.21.15.19:0/2323959795 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3128107fc0 msgr2=0x7f31281a1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:57.055 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:57.052+0000 7f3127400700 1 --2- 172.21.15.19:0/2323959795 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3128107fc0 0x7f31281a1cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:57.055 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:57.052+0000 7f3127400700 1 -- 172.21.15.19:0/2323959795 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3128105690 msgr2=0x7f312819ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:57.055 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:57.052+0000 7f3127400700 1 --2- 172.21.15.19:0/2323959795 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3128105690 0x7f312819ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:57.055 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:57.052+0000 7f3127400700 1 -- 172.21.15.19:0/2323959795 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31281a2390 con 0x7f3128102d60 2024-03-18T15:17:57.055 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:57.052+0000 7f311ea00700 1 --2- 172.21.15.19:0/2323959795 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3128105690 0x7f312819ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T15:17:57.055 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:57.053+0000 7f3127400700 1 --2- 172.21.15.19:0/2323959795 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3128102d60 0x7f312819a730 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f3118004d00 tx=0x7f3118004de0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:17:57.055 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:57.053+0000 7f3124c00700 1 -- 172.21.15.19:0/2323959795 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3118021070 con 0x7f3128102d60 2024-03-18T15:17:57.056 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:57.053+0000 7f3124c00700 1 -- 172.21.15.19:0/2323959795 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3118007c80 con 0x7f3128102d60 2024-03-18T15:17:57.056 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:57.053+0000 7f312e000700 1 -- 172.21.15.19:0/2323959795 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31281a2620 con 0x7f3128102d60 2024-03-18T15:17:57.056 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:57.053+0000 7f3124c00700 1 -- 172.21.15.19:0/2323959795 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3118005560 con 0x7f3128102d60 2024-03-18T15:17:57.056 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:57.053+0000 7f312e000700 1 -- 172.21.15.19:0/2323959795 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31281a2ad0 con 0x7f3128102d60 2024-03-18T15:17:57.056 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:57.054+0000 7f312e000700 1 -- 172.21.15.19:0/2323959795 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3128068f90 con 0x7f3128102d60 2024-03-18T15:17:57.057 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:57.056+0000 7f3124c00700 1 -- 172.21.15.19:0/2323959795 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f3118024070 con 0x7f3128102d60 2024-03-18T15:17:57.058 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:57.057+0000 7f3124c00700 1 --2- 172.21.15.19:0/2323959795 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f310c0a2890 0x7f310c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:17:57.059 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:57.058+0000 7f3124c00700 1 -- 172.21.15.19:0/2323959795 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f311800f020 con 0x7f3128102d60 2024-03-18T15:17:57.059 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:57.058+0000 7f311ea00700 1 --2- 172.21.15.19:0/2323959795 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f310c0a2890 0x7f310c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:17:57.059 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:57.059+0000 7f311ea00700 1 --2- 172.21.15.19:0/2323959795 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f310c0a2890 0x7f310c0a4d50 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f3110009cc0 tx=0x7f3110009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:17:57.065 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:57.064+0000 7f3124c00700 1 -- 172.21.15.19:0/2323959795 <== mon.1 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f311802b020 con 0x7f3128102d60 2024-03-18T15:17:57.344 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:57.343+0000 7f312e000700 1 -- 172.21.15.19:0/2323959795 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7f3128062ee0 con 0x7f3128102d60 2024-03-18T15:17:57.345 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:57.344+0000 7f3124c00700 1 -- 172.21.15.19:0/2323959795 <== mon.1 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 26 v26) ==== 76+0+3088 (secure 0 0 0) 0x7f311809d860 con 0x7f3128102d60 2024-03-18T15:17:57.345 INFO:teuthology.orchestra.run.smithi019.stdout:e26 2024-03-18T15:17:57.346 INFO:teuthology.orchestra.run.smithi019.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-18T15:17:57.346 INFO:teuthology.orchestra.run.smithi019.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-18T15:17:57.346 INFO:teuthology.orchestra.run.smithi019.stdout:legacy client fscid: 1 2024-03-18T15:17:57.346 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-18T15:17:57.346 INFO:teuthology.orchestra.run.smithi019.stdout:Filesystem 'cephfs' (1) 2024-03-18T15:17:57.346 INFO:teuthology.orchestra.run.smithi019.stdout:fs_name cephfs 2024-03-18T15:17:57.346 INFO:teuthology.orchestra.run.smithi019.stdout:epoch 26 2024-03-18T15:17:57.346 INFO:teuthology.orchestra.run.smithi019.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-18T15:17:57.346 INFO:teuthology.orchestra.run.smithi019.stdout:created 2024-03-18T15:15:10.589737+0000 2024-03-18T15:17:57.346 INFO:teuthology.orchestra.run.smithi019.stdout:modified 2024-03-18T15:15:46.979999+0000 2024-03-18T15:17:57.346 INFO:teuthology.orchestra.run.smithi019.stdout:tableserver 0 2024-03-18T15:17:57.346 INFO:teuthology.orchestra.run.smithi019.stdout:root 0 2024-03-18T15:17:57.346 INFO:teuthology.orchestra.run.smithi019.stdout:session_timeout 300 2024-03-18T15:17:57.347 INFO:teuthology.orchestra.run.smithi019.stdout:session_autoclose 300 2024-03-18T15:17:57.347 INFO:teuthology.orchestra.run.smithi019.stdout:max_file_size 1099511627776 2024-03-18T15:17:57.347 INFO:teuthology.orchestra.run.smithi019.stdout:max_xattr_size 65536 2024-03-18T15:17:57.347 INFO:teuthology.orchestra.run.smithi019.stdout:required_client_features {} 2024-03-18T15:17:57.347 INFO:teuthology.orchestra.run.smithi019.stdout:last_failure 0 2024-03-18T15:17:57.347 INFO:teuthology.orchestra.run.smithi019.stdout:last_failure_osd_epoch 0 2024-03-18T15:17:57.347 INFO:teuthology.orchestra.run.smithi019.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-18T15:17:57.347 INFO:teuthology.orchestra.run.smithi019.stdout:max_mds 5 2024-03-18T15:17:57.347 INFO:teuthology.orchestra.run.smithi019.stdout:in 0,1,2,3,4 2024-03-18T15:17:57.347 INFO:teuthology.orchestra.run.smithi019.stdout:up {0=24560,1=24529,2=24511,3=24506,4=24499} 2024-03-18T15:17:57.347 INFO:teuthology.orchestra.run.smithi019.stdout:failed 2024-03-18T15:17:57.347 INFO:teuthology.orchestra.run.smithi019.stdout:damaged 2024-03-18T15:17:57.347 INFO:teuthology.orchestra.run.smithi019.stdout:stopped 2024-03-18T15:17:57.347 INFO:teuthology.orchestra.run.smithi019.stdout:data_pools [3,4] 2024-03-18T15:17:57.347 INFO:teuthology.orchestra.run.smithi019.stdout:metadata_pool 2 2024-03-18T15:17:57.348 INFO:teuthology.orchestra.run.smithi019.stdout:inline_data disabled 2024-03-18T15:17:57.348 INFO:teuthology.orchestra.run.smithi019.stdout:balancer 2024-03-18T15:17:57.348 INFO:teuthology.orchestra.run.smithi019.stdout:bal_rank_mask -1 2024-03-18T15:17:57.348 INFO:teuthology.orchestra.run.smithi019.stdout:standby_count_wanted 0 2024-03-18T15:17:57.348 INFO:teuthology.orchestra.run.smithi019.stdout:qdb_cluster leader: 24560 members: 24506,24499,24529,24511,24560 2024-03-18T15:17:57.348 INFO:teuthology.orchestra.run.smithi019.stdout:[mds.b{0:24560} state up:active seq 3 addr [v2:172.21.15.33:6838/1903591535,v1:172.21.15.33:6839/1903591535] compat {c=[1],r=[1],i=[1fff]}] 2024-03-18T15:17:57.348 INFO:teuthology.orchestra.run.smithi019.stdout:[mds.e{0:24482} state up:standby-replay seq 1 addr [v2:172.21.15.33:6834/4051613151,v1:172.21.15.33:6835/4051613151] compat {c=[1],r=[1],i=[1fff]}] 2024-03-18T15:17:57.348 INFO:teuthology.orchestra.run.smithi019.stdout:[mds.i{1:24529} state up:active seq 14 addr [v2:172.21.15.104:6838/805341436,v1:172.21.15.104:6839/805341436] compat {c=[1],r=[1],i=[1fff]}] 2024-03-18T15:17:57.348 INFO:teuthology.orchestra.run.smithi019.stdout:[mds.d{1:24469} state up:standby-replay seq 1 addr [v2:172.21.15.19:6836/1778019859,v1:172.21.15.19:6837/1778019859] compat {c=[1],r=[1],i=[1fff]}] 2024-03-18T15:17:57.348 INFO:teuthology.orchestra.run.smithi019.stdout:[mds.c{2:24511} state up:active seq 16 addr [v2:172.21.15.104:6836/260984674,v1:172.21.15.104:6837/260984674] compat {c=[1],r=[1],i=[1fff]}] 2024-03-18T15:17:57.348 INFO:teuthology.orchestra.run.smithi019.stdout:[mds.k{2:24434} state up:standby-replay seq 1 addr [v2:172.21.15.33:6832/2272279159,v1:172.21.15.33:6833/2272279159] compat {c=[1],r=[1],i=[1fff]}] 2024-03-18T15:17:57.348 INFO:teuthology.orchestra.run.smithi019.stdout:[mds.h{3:24506} state up:active seq 15 addr [v2:172.21.15.33:6836/160187378,v1:172.21.15.33:6837/160187378] compat {c=[1],r=[1],i=[1fff]}] 2024-03-18T15:17:57.348 INFO:teuthology.orchestra.run.smithi019.stdout:[mds.l{3:24433} state up:standby-replay seq 1 addr [v2:172.21.15.104:6832/482725809,v1:172.21.15.104:6833/482725809] compat {c=[1],r=[1],i=[1fff]}] 2024-03-18T15:17:57.348 INFO:teuthology.orchestra.run.smithi019.stdout:[mds.f{4:24499} state up:active seq 17 addr [v2:172.21.15.104:6834/3788595957,v1:172.21.15.104:6835/3788595957] compat {c=[1],r=[1],i=[1fff]}] 2024-03-18T15:17:57.349 INFO:teuthology.orchestra.run.smithi019.stdout:[mds.a{4:14475} state up:standby-replay seq 1 addr [v2:172.21.15.19:6840/3176583319,v1:172.21.15.19:6841/3176583319] compat {c=[1],r=[1],i=[1fff]}] 2024-03-18T15:17:57.349 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-18T15:17:57.349 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-18T15:17:57.349 INFO:teuthology.orchestra.run.smithi019.stdout:Standby daemons: 2024-03-18T15:17:57.349 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-18T15:17:57.349 INFO:teuthology.orchestra.run.smithi019.stdout:[mds.j{-1:14391} state up:standby seq 1 addr [v2:172.21.15.19:6834/1135329545,v1:172.21.15.19:6835/1135329545] compat {c=[1],r=[1],i=[1fff]}] 2024-03-18T15:17:57.349 INFO:teuthology.orchestra.run.smithi019.stdout:[mds.g{-1:14457} state up:standby seq 1 addr [v2:172.21.15.19:6838/3833518557,v1:172.21.15.19:6839/3833518557] compat {c=[1],r=[1],i=[1fff]}] 2024-03-18T15:17:57.350 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:57.347+0000 7f312e000700 1 -- 172.21.15.19:0/2323959795 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f310c0a2890 msgr2=0x7f310c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:57.350 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:57.347+0000 7f312e000700 1 --2- 172.21.15.19:0/2323959795 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f310c0a2890 0x7f310c0a4d50 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f3110009cc0 tx=0x7f3110009260 comp rx=0 tx=0).stop 2024-03-18T15:17:57.350 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:57.347+0000 7f312e000700 1 -- 172.21.15.19:0/2323959795 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3128102d60 msgr2=0x7f312819a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:17:57.350 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:57.347+0000 7f312e000700 1 --2- 172.21.15.19:0/2323959795 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3128102d60 0x7f312819a730 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f3118004d00 tx=0x7f3118004de0 comp rx=0 tx=0).stop 2024-03-18T15:17:57.350 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:57.347+0000 7f312e000700 1 -- 172.21.15.19:0/2323959795 shutdown_connections 2024-03-18T15:17:57.350 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:57.347+0000 7f312e000700 1 --2- 172.21.15.19:0/2323959795 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f310c0a2890 0x7f310c0a4d50 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:57.350 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:57.347+0000 7f312e000700 1 --2- 172.21.15.19:0/2323959795 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3128102d60 0x7f312819a730 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:57.350 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:57.347+0000 7f312e000700 1 --2- 172.21.15.19:0/2323959795 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3128107fc0 0x7f31281a1cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:57.350 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:57.347+0000 7f312e000700 1 --2- 172.21.15.19:0/2323959795 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3128105690 0x7f312819ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:17:57.350 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:57.347+0000 7f312e000700 1 -- 172.21.15.19:0/2323959795 >> 172.21.15.19:0/2323959795 conn(0x7f31280fc770 msgr2=0x7f3128102550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:17:57.350 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:57.347+0000 7f312e000700 1 -- 172.21.15.19:0/2323959795 shutdown_connections 2024-03-18T15:17:57.350 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:17:57.347+0000 7f312e000700 1 -- 172.21.15.19:0/2323959795 wait complete. 2024-03-18T15:17:57.351 INFO:teuthology.orchestra.run.smithi019.stderr:dumped fsmap epoch 26 2024-03-18T15:17:57.713 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:17:57 smithi019 ceph-mon[31522]: pgmap v404: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T15:17:57.713 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:17:57 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/2323959795' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-18T15:17:57.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:17:57 smithi104 ceph-mon[36070]: pgmap v404: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T15:17:57.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:17:57 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/2323959795' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-18T15:17:58.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:17:57 smithi033 ceph-mon[36648]: pgmap v404: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T15:17:58.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:17:57 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/2323959795' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-18T15:17:58.124 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 -- bash -c 'ceph osd dump' 2024-03-18T15:17:58.580 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:17:59.782 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:17:59 smithi019 ceph-mon[31522]: pgmap v405: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T15:17:59.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:17:59 smithi104 ceph-mon[36070]: pgmap v405: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T15:18:00.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:17:59 smithi033 ceph-mon[36648]: pgmap v405: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T15:18:00.981 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:00.979+0000 7f3d47e00700 1 -- 172.21.15.19:0/8617566 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3d48103de0 msgr2=0x7f3d4810a690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:00.981 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:00.979+0000 7f3d47e00700 1 --2- 172.21.15.19:0/8617566 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3d48103de0 0x7f3d4810a690 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f3d4000b0d0 tx=0x7f3d4000b3e0 comp rx=0 tx=0).stop 2024-03-18T15:18:00.981 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:00.979+0000 7f3d47e00700 1 -- 172.21.15.19:0/8617566 shutdown_connections 2024-03-18T15:18:00.981 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:00.979+0000 7f3d47e00700 1 --2- 172.21.15.19:0/8617566 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3d48103de0 0x7f3d4810a690 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:00.982 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:00.979+0000 7f3d47e00700 1 --2- 172.21.15.19:0/8617566 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3d48102240 0x7f3d48102660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:00.982 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:00.979+0000 7f3d47e00700 1 --2- 172.21.15.19:0/8617566 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3d48103440 0x7f3d481038a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:00.982 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:00.979+0000 7f3d47e00700 1 -- 172.21.15.19:0/8617566 >> 172.21.15.19:0/8617566 conn(0x7f3d480fd7e0 msgr2=0x7f3d480ffc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:18:00.982 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:00.979+0000 7f3d47e00700 1 -- 172.21.15.19:0/8617566 shutdown_connections 2024-03-18T15:18:00.982 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:00.979+0000 7f3d47e00700 1 -- 172.21.15.19:0/8617566 wait complete. 2024-03-18T15:18:00.982 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:00.980+0000 7f3d47e00700 1 Processor -- start 2024-03-18T15:18:00.982 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:00.980+0000 7f3d47e00700 1 -- start start 2024-03-18T15:18:00.983 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:00.980+0000 7f3d47e00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3d48102240 0x7f3d4819a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:00.984 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:00.981+0000 7f3d47e00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3d48103440 0x7f3d4819ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:00.984 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:00.981+0000 7f3d47e00700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3d48103de0 0x7f3d481a1b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:00.984 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:00.981+0000 7f3d47e00700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f3d4810de50 con 0x7f3d48102240 2024-03-18T15:18:00.984 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:00.981+0000 7f3d47e00700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f3d4810dcd0 con 0x7f3d48103de0 2024-03-18T15:18:00.984 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:00.981+0000 7f3d47e00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f3d4810dfd0 con 0x7f3d48103440 2024-03-18T15:18:00.984 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:00.981+0000 7f3d46a00700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3d48102240 0x7f3d4819a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:00.984 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:00.981+0000 7f3d46000700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3d48103440 0x7f3d4819ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:00.984 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:00.981+0000 7f3d47400700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3d48103de0 0x7f3d481a1b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:00.984 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:00.981+0000 7f3d47400700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3d48103de0 0x7f3d481a1b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:34172/0 (socket says 172.21.15.19:34172) 2024-03-18T15:18:00.984 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:00.981+0000 7f3d46000700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3d48103440 0x7f3d4819ab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.19:56532/0 (socket says 172.21.15.19:56532) 2024-03-18T15:18:00.984 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:00.981+0000 7f3d47400700 1 -- 172.21.15.19:0/430216203 learned_addr learned my addr 172.21.15.19:0/430216203 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:18:00.985 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:00.981+0000 7f3d47400700 1 -- 172.21.15.19:0/430216203 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3d48103440 msgr2=0x7f3d4819ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:00.985 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:00.981+0000 7f3d47400700 1 --2- 172.21.15.19:0/430216203 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3d48103440 0x7f3d4819ab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:00.985 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:00.981+0000 7f3d47400700 1 -- 172.21.15.19:0/430216203 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3d48102240 msgr2=0x7f3d4819a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:00.985 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:00.981+0000 7f3d47400700 1 --2- 172.21.15.19:0/430216203 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3d48102240 0x7f3d4819a5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:00.985 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:00.981+0000 7f3d47400700 1 -- 172.21.15.19:0/430216203 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d481a2220 con 0x7f3d48103de0 2024-03-18T15:18:00.985 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:00.982+0000 7f3d47400700 1 --2- 172.21.15.19:0/430216203 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3d48103de0 0x7f3d481a1b80 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f3d4000b770 tx=0x7f3d400079a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:18:00.985 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:00.982+0000 7f3d3f400700 1 -- 172.21.15.19:0/430216203 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d40003c00 con 0x7f3d48103de0 2024-03-18T15:18:00.985 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:00.982+0000 7f3d3f400700 1 -- 172.21.15.19:0/430216203 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3d40003da0 con 0x7f3d48103de0 2024-03-18T15:18:00.985 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:00.982+0000 7f3d3f400700 1 -- 172.21.15.19:0/430216203 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d4001b880 con 0x7f3d48103de0 2024-03-18T15:18:00.985 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:00.982+0000 7f3d47e00700 1 -- 172.21.15.19:0/430216203 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d481a24b0 con 0x7f3d48103de0 2024-03-18T15:18:00.985 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:00.982+0000 7f3d47e00700 1 -- 172.21.15.19:0/430216203 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d481a2960 con 0x7f3d48103de0 2024-03-18T15:18:00.986 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:00.983+0000 7f3d47e00700 1 -- 172.21.15.19:0/430216203 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d48068f90 con 0x7f3d48103de0 2024-03-18T15:18:00.986 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:00.985+0000 7f3d3f400700 1 -- 172.21.15.19:0/430216203 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f3d40026450 con 0x7f3d48103de0 2024-03-18T15:18:00.987 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:00.986+0000 7f3d3f400700 1 --2- 172.21.15.19:0/430216203 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f3d2c0a2910 0x7f3d2c0a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:00.987 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:00.986+0000 7f3d3f400700 1 -- 172.21.15.19:0/430216203 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f3d4000f020 con 0x7f3d48103de0 2024-03-18T15:18:00.987 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:00.986+0000 7f3d46a00700 1 --2- 172.21.15.19:0/430216203 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f3d2c0a2910 0x7f3d2c0a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:00.988 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:00.987+0000 7f3d46a00700 1 --2- 172.21.15.19:0/430216203 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f3d2c0a2910 0x7f3d2c0a4dd0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f3d481032a0 tx=0x7f3d38009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:18:00.991 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:00.991+0000 7f3d3f400700 1 -- 172.21.15.19:0/430216203 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3d4009ccd0 con 0x7f3d48103de0 2024-03-18T15:18:01.207 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:01.206+0000 7f3d47e00700 1 -- 172.21.15.19:0/430216203 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7f3d481a3240 con 0x7f3d48103de0 2024-03-18T15:18:01.208 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:01.207+0000 7f3d3f400700 1 -- 172.21.15.19:0/430216203 <== mon.2 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v93) ==== 56+0+5696 (secure 0 0 0) 0x7f3d4009c5d0 con 0x7f3d48103de0 2024-03-18T15:18:01.209 INFO:teuthology.orchestra.run.smithi019.stdout:epoch 93 2024-03-18T15:18:01.209 INFO:teuthology.orchestra.run.smithi019.stdout:fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 2024-03-18T15:18:01.209 INFO:teuthology.orchestra.run.smithi019.stdout:created 2024-03-18T15:05:57.538551+0000 2024-03-18T15:18:01.209 INFO:teuthology.orchestra.run.smithi019.stdout:modified 2024-03-18T15:17:16.033313+0000 2024-03-18T15:18:01.209 INFO:teuthology.orchestra.run.smithi019.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-03-18T15:18:01.210 INFO:teuthology.orchestra.run.smithi019.stdout:crush_version 27 2024-03-18T15:18:01.210 INFO:teuthology.orchestra.run.smithi019.stdout:full_ratio 0.95 2024-03-18T15:18:01.210 INFO:teuthology.orchestra.run.smithi019.stdout:backfillfull_ratio 0.9 2024-03-18T15:18:01.210 INFO:teuthology.orchestra.run.smithi019.stdout:nearfull_ratio 0.85 2024-03-18T15:18:01.210 INFO:teuthology.orchestra.run.smithi019.stdout:require_min_compat_client luminous 2024-03-18T15:18:01.210 INFO:teuthology.orchestra.run.smithi019.stdout:min_compat_client luminous 2024-03-18T15:18:01.210 INFO:teuthology.orchestra.run.smithi019.stdout:require_osd_release squid 2024-03-18T15:18:01.210 INFO:teuthology.orchestra.run.smithi019.stdout:stretch_mode_enabled false 2024-03-18T15:18:01.210 INFO:teuthology.orchestra.run.smithi019.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-03-18T15:18:01.210 INFO:teuthology.orchestra.run.smithi019.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 84 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-03-18T15:18:01.211 INFO:teuthology.orchestra.run.smithi019.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 85 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.05 application cephfs read_balance_score 1.50 2024-03-18T15:18:01.211 INFO:teuthology.orchestra.run.smithi019.stdout:pool 4 'cephfs_data_ec' erasure profile cephfs_data_ec size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 89 flags hashpspool,ec_overwrites stripe_width 8192 pg_num_min 64 target_size_ratio 0.9 application cephfs 2024-03-18T15:18:01.211 INFO:teuthology.orchestra.run.smithi019.stdout:max_osd 12 2024-03-18T15:18:01.211 INFO:teuthology.orchestra.run.smithi019.stdout:osd.0 up in weight 1 up_from 8 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.19:6802/402333948,v1:172.21.15.19:6803/402333948] [v2:172.21.15.19:6804/402333948,v1:172.21.15.19:6805/402333948] exists,up 37b5a5cd-27ae-4e77-9b05-774b817b2ee9 2024-03-18T15:18:01.211 INFO:teuthology.orchestra.run.smithi019.stdout:osd.1 up in weight 1 up_from 13 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.33:6800/1951880113,v1:172.21.15.33:6801/1951880113] [v2:172.21.15.33:6802/1951880113,v1:172.21.15.33:6803/1951880113] exists,up 3a1134e8-ede7-4dbd-9be1-eefd2cbb43bb 2024-03-18T15:18:01.211 INFO:teuthology.orchestra.run.smithi019.stdout:osd.2 up in weight 1 up_from 18 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.104:6800/1004727949,v1:172.21.15.104:6801/1004727949] [v2:172.21.15.104:6802/1004727949,v1:172.21.15.104:6803/1004727949] exists,up f383a164-a3a0-49fd-a9e8-9c2178d8cfb7 2024-03-18T15:18:01.211 INFO:teuthology.orchestra.run.smithi019.stdout:osd.3 up in weight 1 up_from 25 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.19:6810/3571525854,v1:172.21.15.19:6811/3571525854] [v2:172.21.15.19:6812/3571525854,v1:172.21.15.19:6813/3571525854] exists,up c409e116-28bf-45dd-a791-539af22abe1e 2024-03-18T15:18:01.211 INFO:teuthology.orchestra.run.smithi019.stdout:osd.4 up in weight 1 up_from 33 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.33:6808/624901749,v1:172.21.15.33:6809/624901749] [v2:172.21.15.33:6810/624901749,v1:172.21.15.33:6811/624901749] exists,up 696eb719-0b3e-43ae-b5f1-ccd73a3b2843 2024-03-18T15:18:01.211 INFO:teuthology.orchestra.run.smithi019.stdout:osd.5 up in weight 1 up_from 38 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.104:6808/1258208297,v1:172.21.15.104:6809/1258208297] [v2:172.21.15.104:6810/1258208297,v1:172.21.15.104:6811/1258208297] exists,up 21789779-8bb9-42c7-b9ee-437985f11e3d 2024-03-18T15:18:01.211 INFO:teuthology.orchestra.run.smithi019.stdout:osd.6 up in weight 1 up_from 46 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.19:6818/2782482301,v1:172.21.15.19:6819/2782482301] [v2:172.21.15.19:6820/2782482301,v1:172.21.15.19:6821/2782482301] exists,up 0e92a2ef-43cd-487c-9203-e219158aef8a 2024-03-18T15:18:01.212 INFO:teuthology.orchestra.run.smithi019.stdout:osd.7 up in weight 1 up_from 51 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.33:6816/4238160988,v1:172.21.15.33:6817/4238160988] [v2:172.21.15.33:6818/4238160988,v1:172.21.15.33:6819/4238160988] exists,up a16e39a1-3685-491d-b376-3e80a2084072 2024-03-18T15:18:01.212 INFO:teuthology.orchestra.run.smithi019.stdout:osd.8 up in weight 1 up_from 59 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.104:6816/1014779813,v1:172.21.15.104:6817/1014779813] [v2:172.21.15.104:6818/1014779813,v1:172.21.15.104:6819/1014779813] exists,up d5c3fa92-cd5d-4349-9c7d-d5666bacde16 2024-03-18T15:18:01.212 INFO:teuthology.orchestra.run.smithi019.stdout:osd.9 up in weight 1 up_from 67 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.19:6826/2641151009,v1:172.21.15.19:6827/2641151009] [v2:172.21.15.19:6828/2641151009,v1:172.21.15.19:6829/2641151009] exists,up eed2e9b2-26d3-4c46-88ec-34b5596fe991 2024-03-18T15:18:01.212 INFO:teuthology.orchestra.run.smithi019.stdout:osd.10 up in weight 1 up_from 72 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.33:6824/1346058869,v1:172.21.15.33:6825/1346058869] [v2:172.21.15.33:6826/1346058869,v1:172.21.15.33:6827/1346058869] exists,up f3c35ea5-fbe7-4dc4-93c5-b92d1a714db3 2024-03-18T15:18:01.212 INFO:teuthology.orchestra.run.smithi019.stdout:osd.11 up in weight 1 up_from 80 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.104:6824/3759498281,v1:172.21.15.104:6825/3759498281] [v2:172.21.15.104:6826/3759498281,v1:172.21.15.104:6827/3759498281] exists,up c8b51571-f807-4b62-90b6-7529e09e59da 2024-03-18T15:18:01.212 INFO:teuthology.orchestra.run.smithi019.stdout:pg_upmap_items 2.4 [2,11] 2024-03-18T15:18:01.212 INFO:teuthology.orchestra.run.smithi019.stdout:pg_upmap_items 2.1b [2,4] 2024-03-18T15:18:01.212 INFO:teuthology.orchestra.run.smithi019.stdout:pg_upmap_items 2.1d [8,11] 2024-03-18T15:18:01.212 INFO:teuthology.orchestra.run.smithi019.stdout:pg_upmap_items 3.10 [5,9] 2024-03-18T15:18:01.212 INFO:teuthology.orchestra.run.smithi019.stdout:pg_upmap_items 3.14 [4,6] 2024-03-18T15:18:01.212 INFO:teuthology.orchestra.run.smithi019.stdout:pg_upmap_items 3.1b [3,6] 2024-03-18T15:18:01.212 INFO:teuthology.orchestra.run.smithi019.stdout:pg_upmap_items 3.24 [4,9] 2024-03-18T15:18:01.212 INFO:teuthology.orchestra.run.smithi019.stdout:pg_upmap_items 3.27 [3,9] 2024-03-18T15:18:01.213 INFO:teuthology.orchestra.run.smithi019.stdout:pg_upmap_items 3.37 [5,8] 2024-03-18T15:18:01.213 INFO:teuthology.orchestra.run.smithi019.stdout:pg_upmap_items 3.3a [4,7] 2024-03-18T15:18:01.213 INFO:teuthology.orchestra.run.smithi019.stdout:pg_upmap_items 4.8 [5,8] 2024-03-18T15:18:01.213 INFO:teuthology.orchestra.run.smithi019.stdout:pg_upmap_items 4.f [9,10] 2024-03-18T15:18:01.213 INFO:teuthology.orchestra.run.smithi019.stdout:pg_upmap_items 4.10 [9,8] 2024-03-18T15:18:01.213 INFO:teuthology.orchestra.run.smithi019.stdout:pg_upmap_items 4.13 [9,8] 2024-03-18T15:18:01.213 INFO:teuthology.orchestra.run.smithi019.stdout:blocklist 172.21.15.19:0/3860783274 expires 2024-03-19T15:07:14.150983+0000 2024-03-18T15:18:01.213 INFO:teuthology.orchestra.run.smithi019.stdout:blocklist 172.21.15.19:0/2737223213 expires 2024-03-19T15:07:14.150983+0000 2024-03-18T15:18:01.213 INFO:teuthology.orchestra.run.smithi019.stdout:blocklist 172.21.15.19:0/176699025 expires 2024-03-19T15:06:38.688405+0000 2024-03-18T15:18:01.213 INFO:teuthology.orchestra.run.smithi019.stdout:blocklist 172.21.15.19:0/2316969005 expires 2024-03-19T15:07:14.150983+0000 2024-03-18T15:18:01.214 INFO:teuthology.orchestra.run.smithi019.stdout:blocklist 172.21.15.19:6801/2776008962 expires 2024-03-19T15:07:14.150983+0000 2024-03-18T15:18:01.214 INFO:teuthology.orchestra.run.smithi019.stdout:blocklist 172.21.15.19:0/2793980440 expires 2024-03-19T15:06:38.688405+0000 2024-03-18T15:18:01.214 INFO:teuthology.orchestra.run.smithi019.stdout:blocklist 172.21.15.19:6800/3264337803 expires 2024-03-19T15:06:38.688405+0000 2024-03-18T15:18:01.214 INFO:teuthology.orchestra.run.smithi019.stdout:blocklist 172.21.15.19:6801/3264337803 expires 2024-03-19T15:06:38.688405+0000 2024-03-18T15:18:01.214 INFO:teuthology.orchestra.run.smithi019.stdout:blocklist 172.21.15.19:0/3166705295 expires 2024-03-19T15:06:38.688405+0000 2024-03-18T15:18:01.214 INFO:teuthology.orchestra.run.smithi019.stdout:blocklist 172.21.15.19:6800/2776008962 expires 2024-03-19T15:07:14.150983+0000 2024-03-18T15:18:01.215 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:01.210+0000 7f3d47e00700 1 -- 172.21.15.19:0/430216203 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f3d2c0a2910 msgr2=0x7f3d2c0a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:01.215 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:01.210+0000 7f3d47e00700 1 --2- 172.21.15.19:0/430216203 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f3d2c0a2910 0x7f3d2c0a4dd0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f3d481032a0 tx=0x7f3d38009260 comp rx=0 tx=0).stop 2024-03-18T15:18:01.215 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:01.210+0000 7f3d47e00700 1 -- 172.21.15.19:0/430216203 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3d48103de0 msgr2=0x7f3d481a1b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:01.215 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:01.210+0000 7f3d47e00700 1 --2- 172.21.15.19:0/430216203 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3d48103de0 0x7f3d481a1b80 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f3d4000b770 tx=0x7f3d400079a0 comp rx=0 tx=0).stop 2024-03-18T15:18:01.215 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:01.210+0000 7f3d47e00700 1 -- 172.21.15.19:0/430216203 shutdown_connections 2024-03-18T15:18:01.215 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:01.210+0000 7f3d47e00700 1 --2- 172.21.15.19:0/430216203 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f3d2c0a2910 0x7f3d2c0a4dd0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:01.215 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:01.210+0000 7f3d47e00700 1 --2- 172.21.15.19:0/430216203 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3d48103de0 0x7f3d481a1b80 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:01.215 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:01.210+0000 7f3d47e00700 1 --2- 172.21.15.19:0/430216203 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3d48102240 0x7f3d4819a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:01.215 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:01.210+0000 7f3d47e00700 1 --2- 172.21.15.19:0/430216203 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3d48103440 0x7f3d4819ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:01.215 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:01.210+0000 7f3d47e00700 1 -- 172.21.15.19:0/430216203 >> 172.21.15.19:0/430216203 conn(0x7f3d480fd7e0 msgr2=0x7f3d48074500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:18:01.216 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:01.210+0000 7f3d47e00700 1 -- 172.21.15.19:0/430216203 shutdown_connections 2024-03-18T15:18:01.216 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:01.211+0000 7f3d47e00700 1 -- 172.21.15.19:0/430216203 wait complete. 2024-03-18T15:18:01.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:01 smithi104 ceph-mon[36070]: pgmap v406: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T15:18:01.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:01 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/430216203' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-18T15:18:01.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:01 smithi019 ceph-mon[31522]: pgmap v406: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T15:18:01.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:01 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/430216203' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-18T15:18:02.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:01 smithi033 ceph-mon[36648]: pgmap v406: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T15:18:02.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:01 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/430216203' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-18T15:18:02.158 INFO:teuthology.run_tasks:Running task fs.ready... 2024-03-18T15:18:02.169 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T15:18:02.406 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:02.404+0000 7f2745400640 1 -- 172.21.15.19:0/1302086775 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f27400773e0 msgr2=0x7f2740077840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:02.406 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:02.404+0000 7f2745400640 1 --2- 172.21.15.19:0/1302086775 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f27400773e0 0x7f2740077840 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f2728009900 tx=0x7f272802bc30 comp rx=0 tx=0).stop 2024-03-18T15:18:02.406 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:02.405+0000 7f2745400640 1 -- 172.21.15.19:0/1302086775 shutdown_connections 2024-03-18T15:18:02.406 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:02.405+0000 7f2745400640 1 --2- 172.21.15.19:0/1302086775 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2740077d80 0x7f2740078620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:02.406 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:02.405+0000 7f2745400640 1 --2- 172.21.15.19:0/1302086775 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f27400773e0 0x7f2740077840 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:02.406 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:02.405+0000 7f2745400640 1 --2- 172.21.15.19:0/1302086775 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f27400761a0 0x7f27400765c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:02.406 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:02.405+0000 7f2745400640 1 -- 172.21.15.19:0/1302086775 >> 172.21.15.19:0/1302086775 conn(0x7f27400fed70 msgr2=0x7f27401011b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:18:02.407 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:02.405+0000 7f2745400640 1 -- 172.21.15.19:0/1302086775 shutdown_connections 2024-03-18T15:18:02.407 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:02.405+0000 7f2745400640 1 -- 172.21.15.19:0/1302086775 wait complete. 2024-03-18T15:18:02.408 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:02.406+0000 7f2745400640 1 Processor -- start 2024-03-18T15:18:02.409 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:02.406+0000 7f2745400640 1 -- start start 2024-03-18T15:18:02.409 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:02.406+0000 7f2745400640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f27400761a0 0x7f274019ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:02.409 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:02.406+0000 7f2745400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f27400773e0 0x7f274019e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:02.409 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:02.406+0000 7f2745400640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2740077d80 0x7f27401a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:02.409 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:02.406+0000 7f2745400640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f2740113890 con 0x7f2740077d80 2024-03-18T15:18:02.409 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:02.406+0000 7f2745400640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2740113710 con 0x7f27400773e0 2024-03-18T15:18:02.409 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:02.406+0000 7f2745400640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f2740113a10 con 0x7f27400761a0 2024-03-18T15:18:02.409 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:02.406+0000 7f273fe00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f27400761a0 0x7f274019ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:02.409 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:02.406+0000 7f273fe00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f27400761a0 0x7f274019ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.19:39078/0 (socket says 172.21.15.19:39078) 2024-03-18T15:18:02.409 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:02.406+0000 7f273fe00640 1 -- 172.21.15.19:0/750156489 learned_addr learned my addr 172.21.15.19:0/750156489 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:18:02.409 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:02.406+0000 7f2737400640 1 --2- 172.21.15.19:0/750156489 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f27400773e0 0x7f274019e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:02.409 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:02.407+0000 7f2744a00640 1 --2- 172.21.15.19:0/750156489 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2740077d80 0x7f27401a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:02.409 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:02.407+0000 7f273fe00640 1 -- 172.21.15.19:0/750156489 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f27400773e0 msgr2=0x7f274019e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:02.410 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:02.407+0000 7f273fe00640 1 --2- 172.21.15.19:0/750156489 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f27400773e0 0x7f274019e410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:02.410 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:02.407+0000 7f273fe00640 1 -- 172.21.15.19:0/750156489 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2740077d80 msgr2=0x7f27401a5480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:02.410 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:02.407+0000 7f273fe00640 1 --2- 172.21.15.19:0/750156489 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2740077d80 0x7f27401a5480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:02.410 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:02.407+0000 7f273fe00640 1 -- 172.21.15.19:0/750156489 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27401a5af0 con 0x7f27400761a0 2024-03-18T15:18:02.410 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:02.407+0000 7f273fe00640 1 --2- 172.21.15.19:0/750156489 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f27400761a0 0x7f274019ded0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f273000e9f0 tx=0x7f273000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:18:02.410 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:02.407+0000 7f273d600640 1 -- 172.21.15.19:0/750156489 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f273000cd80 con 0x7f27400761a0 2024-03-18T15:18:02.410 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:02.407+0000 7f273d600640 1 -- 172.21.15.19:0/750156489 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2730010910 con 0x7f27400761a0 2024-03-18T15:18:02.410 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:02.407+0000 7f2745400640 1 -- 172.21.15.19:0/750156489 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27401a5de0 con 0x7f27400761a0 2024-03-18T15:18:02.410 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:02.407+0000 7f273d600640 1 -- 172.21.15.19:0/750156489 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2730018990 con 0x7f27400761a0 2024-03-18T15:18:02.410 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:02.408+0000 7f2745400640 1 -- 172.21.15.19:0/750156489 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27401a6240 con 0x7f27400761a0 2024-03-18T15:18:02.411 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:02.409+0000 7f2745400640 1 -- 172.21.15.19:0/750156489 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2740110170 con 0x7f27400761a0 2024-03-18T15:18:02.411 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:02.411+0000 7f273d600640 1 -- 172.21.15.19:0/750156489 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f2730010ac0 con 0x7f27400761a0 2024-03-18T15:18:02.413 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:02.412+0000 7f273d600640 1 --2- 172.21.15.19:0/750156489 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f27200a2800 0x7f27200a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:02.413 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:02.412+0000 7f273d600640 1 -- 172.21.15.19:0/750156489 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f27300cf400 con 0x7f27400761a0 2024-03-18T15:18:02.413 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:02.412+0000 7f2737400640 1 --2- 172.21.15.19:0/750156489 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f27200a2800 0x7f27200a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:02.414 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:02.413+0000 7f2737400640 1 --2- 172.21.15.19:0/750156489 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f27200a2800 0x7f27200a4cc0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f272802c240 tx=0x7f272802df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:18:02.416 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:02.416+0000 7f273d600640 1 -- 172.21.15.19:0/750156489 <== mon.1 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f273009c070 con 0x7f27400761a0 2024-03-18T15:18:02.620 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:02.619+0000 7f2745400640 1 -- 172.21.15.19:0/750156489 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f274002d350 con 0x7f27400761a0 2024-03-18T15:18:02.623 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:02.622+0000 7f273d600640 1 -- 172.21.15.19:0/750156489 <== mon.1 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24809 (secure 0 0 0) 0x7f273009b970 con 0x7f27400761a0 2024-03-18T15:18:02.623 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-18T15:18:02.624 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":93,"fsid":"d4dc2ba6-e538-11ee-95c9-87774f69a715","created":"2024-03-18T15:05:57.538551+0000","modified":"2024-03-18T15:17:16.033313+0000","last_up_change":"2024-03-18T15:14:12.009522+0000","last_in_change":"2024-03-18T15:13:52.555387+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T15:09:56.843349+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T15:15:08.941047+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T15:15:10.044671+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T15:15:13.621916+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"37b5a5cd-27ae-4e77-9b05-774b817b2ee9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6802","nonce":402333948},{"type":"v1","addr":"172.21.15.19:6803","nonce":402333948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6804","nonce":402333948},{"type":"v1","addr":"172.21.15.19:6805","nonce":402333948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6808","nonce":402333948},{"type":"v1","addr":"172.21.15.19:6809","nonce":402333948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6806","nonce":402333948},{"type":"v1","addr":"172.21.15.19:6807","nonce":402333948}]},"public_addr":"172.21.15.19:6803/402333948","cluster_addr":"172.21.15.19:6805/402333948","heartbeat_back_addr":"172.21.15.19:6809/402333948","heartbeat_front_addr":"172.21.15.19:6807/402333948","state":["exists","up"]},{"osd":1,"uuid":"3a1134e8-ede7-4dbd-9be1-eefd2cbb43bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1951880113},{"type":"v1","addr":"172.21.15.33:6801","nonce":1951880113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1951880113},{"type":"v1","addr":"172.21.15.33:6803","nonce":1951880113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1951880113},{"type":"v1","addr":"172.21.15.33:6807","nonce":1951880113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1951880113},{"type":"v1","addr":"172.21.15.33:6805","nonce":1951880113}]},"public_addr":"172.21.15.33:6801/1951880113","cluster_addr":"172.21.15.33:6803/1951880113","heartbeat_back_addr":"172.21.15.33:6807/1951880113","heartbeat_front_addr":"172.21.15.33:6805/1951880113","state":["exists","up"]},{"osd":2,"uuid":"f383a164-a3a0-49fd-a9e8-9c2178d8cfb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":1004727949},{"type":"v1","addr":"172.21.15.104:6801","nonce":1004727949}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":1004727949},{"type":"v1","addr":"172.21.15.104:6803","nonce":1004727949}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":1004727949},{"type":"v1","addr":"172.21.15.104:6807","nonce":1004727949}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":1004727949},{"type":"v1","addr":"172.21.15.104:6805","nonce":1004727949}]},"public_addr":"172.21.15.104:6801/1004727949","cluster_addr":"172.21.15.104:6803/1004727949","heartbeat_back_addr":"172.21.15.104:6807/1004727949","heartbeat_front_addr":"172.21.15.104:6805/1004727949","state":["exists","up"]},{"osd":3,"uuid":"c409e116-28bf-45dd-a791-539af22abe1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6810","nonce":3571525854},{"type":"v1","addr":"172.21.15.19:6811","nonce":3571525854}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6812","nonce":3571525854},{"type":"v1","addr":"172.21.15.19:6813","nonce":3571525854}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6816","nonce":3571525854},{"type":"v1","addr":"172.21.15.19:6817","nonce":3571525854}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6814","nonce":3571525854},{"type":"v1","addr":"172.21.15.19:6815","nonce":3571525854}]},"public_addr":"172.21.15.19:6811/3571525854","cluster_addr":"172.21.15.19:6813/3571525854","heartbeat_back_addr":"172.21.15.19:6817/3571525854","heartbeat_front_addr":"172.21.15.19:6815/3571525854","state":["exists","up"]},{"osd":4,"uuid":"696eb719-0b3e-43ae-b5f1-ccd73a3b2843","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":624901749},{"type":"v1","addr":"172.21.15.33:6809","nonce":624901749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":624901749},{"type":"v1","addr":"172.21.15.33:6811","nonce":624901749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":624901749},{"type":"v1","addr":"172.21.15.33:6815","nonce":624901749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":624901749},{"type":"v1","addr":"172.21.15.33:6813","nonce":624901749}]},"public_addr":"172.21.15.33:6809/624901749","cluster_addr":"172.21.15.33:6811/624901749","heartbeat_back_addr":"172.21.15.33:6815/624901749","heartbeat_front_addr":"172.21.15.33:6813/624901749","state":["exists","up"]},{"osd":5,"uuid":"21789779-8bb9-42c7-b9ee-437985f11e3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":1258208297},{"type":"v1","addr":"172.21.15.104:6809","nonce":1258208297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":1258208297},{"type":"v1","addr":"172.21.15.104:6811","nonce":1258208297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":1258208297},{"type":"v1","addr":"172.21.15.104:6815","nonce":1258208297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":1258208297},{"type":"v1","addr":"172.21.15.104:6813","nonce":1258208297}]},"public_addr":"172.21.15.104:6809/1258208297","cluster_addr":"172.21.15.104:6811/1258208297","heartbeat_back_addr":"172.21.15.104:6815/1258208297","heartbeat_front_addr":"172.21.15.104:6813/1258208297","state":["exists","up"]},{"osd":6,"uuid":"0e92a2ef-43cd-487c-9203-e219158aef8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6818","nonce":2782482301},{"type":"v1","addr":"172.21.15.19:6819","nonce":2782482301}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6820","nonce":2782482301},{"type":"v1","addr":"172.21.15.19:6821","nonce":2782482301}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6824","nonce":2782482301},{"type":"v1","addr":"172.21.15.19:6825","nonce":2782482301}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6822","nonce":2782482301},{"type":"v1","addr":"172.21.15.19:6823","nonce":2782482301}]},"public_addr":"172.21.15.19:6819/2782482301","cluster_addr":"172.21.15.19:6821/2782482301","heartbeat_back_addr":"172.21.15.19:6825/2782482301","heartbeat_front_addr":"172.21.15.19:6823/2782482301","state":["exists","up"]},{"osd":7,"uuid":"a16e39a1-3685-491d-b376-3e80a2084072","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":4238160988},{"type":"v1","addr":"172.21.15.33:6817","nonce":4238160988}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":4238160988},{"type":"v1","addr":"172.21.15.33:6819","nonce":4238160988}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":4238160988},{"type":"v1","addr":"172.21.15.33:6823","nonce":4238160988}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":4238160988},{"type":"v1","addr":"172.21.15.33:6821","nonce":4238160988}]},"public_addr":"172.21.15.33:6817/4238160988","cluster_addr":"172.21.15.33:6819/4238160988","heartbeat_back_addr":"172.21.15.33:6823/4238160988","heartbeat_front_addr":"172.21.15.33:6821/4238160988","state":["exists","up"]},{"osd":8,"uuid":"d5c3fa92-cd5d-4349-9c7d-d5666bacde16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":1014779813},{"type":"v1","addr":"172.21.15.104:6817","nonce":1014779813}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":1014779813},{"type":"v1","addr":"172.21.15.104:6819","nonce":1014779813}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":1014779813},{"type":"v1","addr":"172.21.15.104:6823","nonce":1014779813}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":1014779813},{"type":"v1","addr":"172.21.15.104:6821","nonce":1014779813}]},"public_addr":"172.21.15.104:6817/1014779813","cluster_addr":"172.21.15.104:6819/1014779813","heartbeat_back_addr":"172.21.15.104:6823/1014779813","heartbeat_front_addr":"172.21.15.104:6821/1014779813","state":["exists","up"]},{"osd":9,"uuid":"eed2e9b2-26d3-4c46-88ec-34b5596fe991","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6826","nonce":2641151009},{"type":"v1","addr":"172.21.15.19:6827","nonce":2641151009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6828","nonce":2641151009},{"type":"v1","addr":"172.21.15.19:6829","nonce":2641151009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6832","nonce":2641151009},{"type":"v1","addr":"172.21.15.19:6833","nonce":2641151009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6830","nonce":2641151009},{"type":"v1","addr":"172.21.15.19:6831","nonce":2641151009}]},"public_addr":"172.21.15.19:6827/2641151009","cluster_addr":"172.21.15.19:6829/2641151009","heartbeat_back_addr":"172.21.15.19:6833/2641151009","heartbeat_front_addr":"172.21.15.19:6831/2641151009","state":["exists","up"]},{"osd":10,"uuid":"f3c35ea5-fbe7-4dc4-93c5-b92d1a714db3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":1346058869},{"type":"v1","addr":"172.21.15.33:6825","nonce":1346058869}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":1346058869},{"type":"v1","addr":"172.21.15.33:6827","nonce":1346058869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":1346058869},{"type":"v1","addr":"172.21.15.33:6831","nonce":1346058869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":1346058869},{"type":"v1","addr":"172.21.15.33:6829","nonce":1346058869}]},"public_addr":"172.21.15.33:6825/1346058869","cluster_addr":"172.21.15.33:6827/1346058869","heartbeat_back_addr":"172.21.15.33:6831/1346058869","heartbeat_front_addr":"172.21.15.33:6829/1346058869","state":["exists","up"]},{"osd":11,"uuid":"c8b51571-f807-4b62-90b6-7529e09e59da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":3759498281},{"type":"v1","addr":"172.21.15.104:6825","nonce":3759498281}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":3759498281},{"type":"v1","addr":"172.21.15.104:6827","nonce":3759498281}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":3759498281},{"type":"v1","addr":"172.21.15.104:6831","nonce":3759498281}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":3759498281},{"type":"v1","addr":"172.21.15.104:6829","nonce":3759498281}]},"public_addr":"172.21.15.104:6825/3759498281","cluster_addr":"172.21.15.104:6827/3759498281","heartbeat_back_addr":"172.21.15.104:6831/3759498281","heartbeat_front_addr":"172.21.15.104:6829/3759498281","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:09:05.267795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:09:30.234068+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:09:53.870232+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:10:26.471296+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:10:53.180341+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:11:17.801361+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:11:51.364816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:12:18.306408+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:12:43.978744+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:13:18.213964+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:13:45.437544+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:14:10.148580+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":3,"to":6}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"3.37","mappings":[{"from":5,"to":8}]},{"pgid":"3.3a","mappings":[{"from":4,"to":7}]},{"pgid":"4.8","mappings":[{"from":5,"to":8}]},{"pgid":"4.f","mappings":[{"from":9,"to":10}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.19:0/3860783274":"2024-03-19T15:07:14.150983+0000","172.21.15.19:0/2737223213":"2024-03-19T15:07:14.150983+0000","172.21.15.19:0/176699025":"2024-03-19T15:06:38.688405+0000","172.21.15.19:0/2316969005":"2024-03-19T15:07:14.150983+0000","172.21.15.19:6801/2776008962":"2024-03-19T15:07:14.150983+0000","172.21.15.19:0/2793980440":"2024-03-19T15:06:38.688405+0000","172.21.15.19:6800/3264337803":"2024-03-19T15:06:38.688405+0000","172.21.15.19:6801/3264337803":"2024-03-19T15:06:38.688405+0000","172.21.15.19:0/3166705295":"2024-03-19T15:06:38.688405+0000","172.21.15.19:6800/2776008962":"2024-03-19T15:07:14.150983+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T15:18:02.626 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:02.625+0000 7f2745400640 1 -- 172.21.15.19:0/750156489 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f27200a2800 msgr2=0x7f27200a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:02.626 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:02.625+0000 7f2745400640 1 --2- 172.21.15.19:0/750156489 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f27200a2800 0x7f27200a4cc0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f272802c240 tx=0x7f272802df90 comp rx=0 tx=0).stop 2024-03-18T15:18:02.626 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:02.625+0000 7f2745400640 1 -- 172.21.15.19:0/750156489 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f27400761a0 msgr2=0x7f274019ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:02.626 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:02.625+0000 7f2745400640 1 --2- 172.21.15.19:0/750156489 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f27400761a0 0x7f274019ded0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f273000e9f0 tx=0x7f273000eeb0 comp rx=0 tx=0).stop 2024-03-18T15:18:02.626 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:02.625+0000 7f2745400640 1 -- 172.21.15.19:0/750156489 shutdown_connections 2024-03-18T15:18:02.626 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:02.625+0000 7f2745400640 1 --2- 172.21.15.19:0/750156489 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2740077d80 0x7f27401a5480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:02.626 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:02.625+0000 7f2745400640 1 --2- 172.21.15.19:0/750156489 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f27200a2800 0x7f27200a4cc0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:02.626 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:02.625+0000 7f2745400640 1 --2- 172.21.15.19:0/750156489 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f27400773e0 0x7f274019e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:02.626 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:02.625+0000 7f2745400640 1 --2- 172.21.15.19:0/750156489 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f27400761a0 0x7f274019ded0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:02.627 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:02.625+0000 7f2745400640 1 -- 172.21.15.19:0/750156489 >> 172.21.15.19:0/750156489 conn(0x7f27400fed70 msgr2=0x7f2740100870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:18:02.627 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:02.625+0000 7f2745400640 1 -- 172.21.15.19:0/750156489 shutdown_connections 2024-03-18T15:18:02.627 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:02.625+0000 7f2745400640 1 -- 172.21.15.19:0/750156489 wait complete. 2024-03-18T15:18:02.644 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T15:09:56.843349+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T15:15:08.941047+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T15:15:10.044671+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-18T15:15:13.621916+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-18T15:18:02.645 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T15:18:02.857 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:02.856+0000 7fc86f000640 1 -- 172.21.15.19:0/462818153 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc868075820 msgr2=0x7fc868073c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:02.857 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:02.856+0000 7fc86f000640 1 --2- 172.21.15.19:0/462818153 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc868075820 0x7fc868073c70 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fc858009930 tx=0x7fc85802bc30 comp rx=0 tx=0).stop 2024-03-18T15:18:02.857 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:02.856+0000 7fc86f000640 1 -- 172.21.15.19:0/462818153 shutdown_connections 2024-03-18T15:18:02.857 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:02.856+0000 7fc86f000640 1 --2- 172.21.15.19:0/462818153 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc86810ddc0 0x7fc8681101b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:02.857 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:02.856+0000 7fc86f000640 1 --2- 172.21.15.19:0/462818153 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc868074240 0x7fc8680746c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:02.858 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:02.856+0000 7fc86f000640 1 --2- 172.21.15.19:0/462818153 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc868075820 0x7fc868073c70 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:02.858 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:02.856+0000 7fc86f000640 1 -- 172.21.15.19:0/462818153 >> 172.21.15.19:0/462818153 conn(0x7fc8680fcd80 msgr2=0x7fc8680ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:18:02.858 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:02.856+0000 7fc86f000640 1 -- 172.21.15.19:0/462818153 shutdown_connections 2024-03-18T15:18:02.858 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:02.856+0000 7fc86f000640 1 -- 172.21.15.19:0/462818153 wait complete. 2024-03-18T15:18:02.860 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:02.857+0000 7fc86f000640 1 Processor -- start 2024-03-18T15:18:02.860 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:02.857+0000 7fc86f000640 1 -- start start 2024-03-18T15:18:02.860 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:02.858+0000 7fc86f000640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc868074240 0x7fc86819e150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:02.860 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:02.858+0000 7fc86f000640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc868075820 0x7fc86819e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:02.860 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:02.858+0000 7fc86f000640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc86810ddc0 0x7fc8681a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:02.860 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:02.858+0000 7fc86f000640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fc868113a60 con 0x7fc868075820 2024-03-18T15:18:02.860 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:02.858+0000 7fc86f000640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc8681138e0 con 0x7fc868074240 2024-03-18T15:18:02.860 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:02.858+0000 7fc86f000640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fc868113be0 con 0x7fc86810ddc0 2024-03-18T15:18:02.861 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:02.858+0000 7fc867e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc868074240 0x7fc86819e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:02.861 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:02.858+0000 7fc867e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc868074240 0x7fc86819e150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:46032/0 (socket says 172.21.15.19:46032) 2024-03-18T15:18:02.861 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:02.858+0000 7fc867e00640 1 -- 172.21.15.19:0/175327555 learned_addr learned my addr 172.21.15.19:0/175327555 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:18:02.861 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:02.858+0000 7fc86d000640 1 --2- 172.21.15.19:0/175327555 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc86810ddc0 0x7fc8681a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:02.861 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:02.858+0000 7fc85fe00640 1 --2- 172.21.15.19:0/175327555 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc868075820 0x7fc86819e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:02.861 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:02.858+0000 7fc867e00640 1 -- 172.21.15.19:0/175327555 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc86810ddc0 msgr2=0x7fc8681a5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:02.861 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:02.858+0000 7fc867e00640 1 --2- 172.21.15.19:0/175327555 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc86810ddc0 0x7fc8681a5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:02.861 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:02.858+0000 7fc867e00640 1 -- 172.21.15.19:0/175327555 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc868075820 msgr2=0x7fc86819e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:02.861 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:02.858+0000 7fc867e00640 1 --2- 172.21.15.19:0/175327555 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc868075820 0x7fc86819e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:02.861 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:02.858+0000 7fc867e00640 1 -- 172.21.15.19:0/175327555 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8681a5d70 con 0x7fc868074240 2024-03-18T15:18:02.861 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:02.858+0000 7fc86d000640 1 --2- 172.21.15.19:0/175327555 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc86810ddc0 0x7fc8681a5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:18:02.861 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:02.859+0000 7fc867e00640 1 --2- 172.21.15.19:0/175327555 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc868074240 0x7fc86819e150 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fc858000c00 tx=0x7fc85802c640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:18:02.861 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:02.859+0000 7fc865600640 1 -- 172.21.15.19:0/175327555 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc858041070 con 0x7fc868074240 2024-03-18T15:18:02.861 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:02.859+0000 7fc865600640 1 -- 172.21.15.19:0/175327555 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc858004450 con 0x7fc868074240 2024-03-18T15:18:02.862 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:02.859+0000 7fc865600640 1 -- 172.21.15.19:0/175327555 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc858034440 con 0x7fc868074240 2024-03-18T15:18:02.862 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:02.859+0000 7fc86f000640 1 -- 172.21.15.19:0/175327555 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8681a6000 con 0x7fc868074240 2024-03-18T15:18:02.862 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:02.859+0000 7fc86f000640 1 -- 172.21.15.19:0/175327555 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8681a64a0 con 0x7fc868074240 2024-03-18T15:18:02.862 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:02.860+0000 7fc86f000640 1 -- 172.21.15.19:0/175327555 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc868103f90 con 0x7fc868074240 2024-03-18T15:18:02.863 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:02.862+0000 7fc865600640 1 -- 172.21.15.19:0/175327555 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fc858044070 con 0x7fc868074240 2024-03-18T15:18:02.864 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:02.863+0000 7fc865600640 1 --2- 172.21.15.19:0/175327555 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fc84c0a2880 0x7fc84c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:02.865 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:02.864+0000 7fc865600640 1 -- 172.21.15.19:0/175327555 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fc8580f08a0 con 0x7fc868074240 2024-03-18T15:18:02.865 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:02.864+0000 7fc85fe00640 1 --2- 172.21.15.19:0/175327555 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fc84c0a2880 0x7fc84c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:02.865 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:02.865+0000 7fc85fe00640 1 --2- 172.21.15.19:0/175327555 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fc84c0a2880 0x7fc84c0a4d40 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fc8500095f0 tx=0x7fc850005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:18:02.868 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:02.867+0000 7fc865600640 1 -- 172.21.15.19:0/175327555 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc8580bcf50 con 0x7fc868074240 2024-03-18T15:18:03.071 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.070+0000 7fc86f000640 1 -- 172.21.15.19:0/175327555 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fc86802d410 con 0x7fc868074240 2024-03-18T15:18:03.072 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.071+0000 7fc865600640 1 -- 172.21.15.19:0/175327555 <== mon.2 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7fc8580c0ff0 con 0x7fc868074240 2024-03-18T15:18:03.072 INFO:teuthology.orchestra.run.smithi019.stdout:pg_num: 1 2024-03-18T15:18:03.075 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.074+0000 7fc86f000640 1 -- 172.21.15.19:0/175327555 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fc84c0a2880 msgr2=0x7fc84c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:03.075 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.074+0000 7fc86f000640 1 --2- 172.21.15.19:0/175327555 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fc84c0a2880 0x7fc84c0a4d40 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fc8500095f0 tx=0x7fc850005f90 comp rx=0 tx=0).stop 2024-03-18T15:18:03.075 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.074+0000 7fc86f000640 1 -- 172.21.15.19:0/175327555 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc868074240 msgr2=0x7fc86819e150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:03.075 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.074+0000 7fc86f000640 1 --2- 172.21.15.19:0/175327555 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc868074240 0x7fc86819e150 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fc858000c00 tx=0x7fc85802c640 comp rx=0 tx=0).stop 2024-03-18T15:18:03.075 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.074+0000 7fc86f000640 1 -- 172.21.15.19:0/175327555 shutdown_connections 2024-03-18T15:18:03.075 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.074+0000 7fc86f000640 1 --2- 172.21.15.19:0/175327555 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc86810ddc0 0x7fc8681a5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:03.075 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.074+0000 7fc86f000640 1 --2- 172.21.15.19:0/175327555 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc868075820 0x7fc86819e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:03.075 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.074+0000 7fc86f000640 1 --2- 172.21.15.19:0/175327555 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fc84c0a2880 0x7fc84c0a4d40 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:03.075 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.074+0000 7fc86f000640 1 --2- 172.21.15.19:0/175327555 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc868074240 0x7fc86819e150 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:03.076 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.074+0000 7fc86f000640 1 -- 172.21.15.19:0/175327555 >> 172.21.15.19:0/175327555 conn(0x7fc8680fcd80 msgr2=0x7fc868101160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:18:03.076 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.074+0000 7fc86f000640 1 -- 172.21.15.19:0/175327555 shutdown_connections 2024-03-18T15:18:03.076 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.074+0000 7fc86f000640 1 -- 172.21.15.19:0/175327555 wait complete. 2024-03-18T15:18:03.094 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T15:18:03.322 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.321+0000 7f8e83200640 1 -- 172.21.15.19:0/3378050102 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8e841016c0 msgr2=0x7f8e84101ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:03.322 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.321+0000 7f8e83200640 1 --2- 172.21.15.19:0/3378050102 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8e841016c0 0x7f8e84101ae0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f8e74009900 tx=0x7f8e7402bc30 comp rx=0 tx=0).stop 2024-03-18T15:18:03.322 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.321+0000 7f8e83200640 1 -- 172.21.15.19:0/3378050102 shutdown_connections 2024-03-18T15:18:03.322 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.321+0000 7f8e83200640 1 --2- 172.21.15.19:0/3378050102 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8e84103270 0x7f8e84109b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:03.323 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.321+0000 7f8e83200640 1 --2- 172.21.15.19:0/3378050102 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8e841028b0 0x7f8e84102d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:03.323 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.321+0000 7f8e83200640 1 --2- 172.21.15.19:0/3378050102 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8e841016c0 0x7f8e84101ae0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:03.323 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.321+0000 7f8e83200640 1 -- 172.21.15.19:0/3378050102 >> 172.21.15.19:0/3378050102 conn(0x7f8e840fcc50 msgr2=0x7f8e840ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:18:03.323 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.321+0000 7f8e83200640 1 -- 172.21.15.19:0/3378050102 shutdown_connections 2024-03-18T15:18:03.323 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.321+0000 7f8e83200640 1 -- 172.21.15.19:0/3378050102 wait complete. 2024-03-18T15:18:03.325 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.322+0000 7f8e83200640 1 Processor -- start 2024-03-18T15:18:03.325 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.322+0000 7f8e83200640 1 -- start start 2024-03-18T15:18:03.325 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.322+0000 7f8e83200640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8e841016c0 0x7f8e84199c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:03.325 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.322+0000 7f8e83200640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8e841028b0 0x7f8e8419a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:03.325 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.323+0000 7f8e83200640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8e84103270 0x7f8e841a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:03.325 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.323+0000 7f8e83200640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f8e8410d340 con 0x7f8e841016c0 2024-03-18T15:18:03.325 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.323+0000 7f8e83200640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8e8410d1c0 con 0x7f8e841028b0 2024-03-18T15:18:03.325 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.323+0000 7f8e83200640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f8e8410d4c0 con 0x7f8e84103270 2024-03-18T15:18:03.325 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.323+0000 7f8e79400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8e841028b0 0x7f8e8419a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:03.325 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.323+0000 7f8e79400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8e841028b0 0x7f8e8419a160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:46046/0 (socket says 172.21.15.19:46046) 2024-03-18T15:18:03.325 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.323+0000 7f8e79400640 1 -- 172.21.15.19:0/3912708431 learned_addr learned my addr 172.21.15.19:0/3912708431 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:18:03.325 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.323+0000 7f8e81e00640 1 --2- 172.21.15.19:0/3912708431 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8e841016c0 0x7f8e84199c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:03.326 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.323+0000 7f8e82800640 1 --2- 172.21.15.19:0/3912708431 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8e84103270 0x7f8e841a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:03.326 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.323+0000 7f8e79400640 1 -- 172.21.15.19:0/3912708431 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8e84103270 msgr2=0x7f8e841a1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:03.326 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.323+0000 7f8e79400640 1 --2- 172.21.15.19:0/3912708431 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8e84103270 0x7f8e841a1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:03.326 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.323+0000 7f8e79400640 1 -- 172.21.15.19:0/3912708431 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8e841016c0 msgr2=0x7f8e84199c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:03.326 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.323+0000 7f8e79400640 1 --2- 172.21.15.19:0/3912708431 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8e841016c0 0x7f8e84199c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:03.326 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.323+0000 7f8e79400640 1 -- 172.21.15.19:0/3912708431 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e841a1830 con 0x7f8e841028b0 2024-03-18T15:18:03.326 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.324+0000 7f8e79400640 1 --2- 172.21.15.19:0/3912708431 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8e841028b0 0x7f8e8419a160 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f8e6c00e9f0 tx=0x7f8e6c00edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:18:03.326 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.324+0000 7f8e7b400640 1 -- 172.21.15.19:0/3912708431 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e6c00cd80 con 0x7f8e841028b0 2024-03-18T15:18:03.326 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.324+0000 7f8e7b400640 1 -- 172.21.15.19:0/3912708431 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e6c005500 con 0x7f8e841028b0 2024-03-18T15:18:03.326 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.324+0000 7f8e7b400640 1 -- 172.21.15.19:0/3912708431 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e6c010690 con 0x7f8e841028b0 2024-03-18T15:18:03.326 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.324+0000 7f8e83200640 1 -- 172.21.15.19:0/3912708431 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e841a1b20 con 0x7f8e841028b0 2024-03-18T15:18:03.326 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.324+0000 7f8e83200640 1 -- 172.21.15.19:0/3912708431 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e841a1ff0 con 0x7f8e841028b0 2024-03-18T15:18:03.327 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.325+0000 7f8e83200640 1 -- 172.21.15.19:0/3912708431 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e84068940 con 0x7f8e841028b0 2024-03-18T15:18:03.328 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.327+0000 7f8e7b400640 1 -- 172.21.15.19:0/3912708431 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f8e6c0046c0 con 0x7f8e841028b0 2024-03-18T15:18:03.329 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.328+0000 7f8e7b400640 1 --2- 172.21.15.19:0/3912708431 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f8e680a2800 0x7f8e680a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:03.329 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.329+0000 7f8e7b400640 1 -- 172.21.15.19:0/3912708431 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f8e6c0cf4f0 con 0x7f8e841028b0 2024-03-18T15:18:03.330 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.329+0000 7f8e81e00640 1 --2- 172.21.15.19:0/3912708431 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f8e680a2800 0x7f8e680a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:03.330 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.329+0000 7f8e81e00640 1 --2- 172.21.15.19:0/3912708431 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f8e680a2800 0x7f8e680a4cc0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f8e74000c00 tx=0x7f8e74005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:18:03.331 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.331+0000 7f8e7b400640 1 -- 172.21.15.19:0/3912708431 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8e6c053740 con 0x7f8e841028b0 2024-03-18T15:18:03.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:03 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/750156489' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:18:03.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:03 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/750156489' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:18:03.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:03 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/750156489' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:18:03.540 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.539+0000 7f8e83200640 1 -- 172.21.15.19:0/3912708431 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f8e841a2410 con 0x7f8e841028b0 2024-03-18T15:18:03.541 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.540+0000 7f8e7b400640 1 -- 172.21.15.19:0/3912708431 <== mon.2 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7f8e6c09c160 con 0x7f8e841028b0 2024-03-18T15:18:03.541 INFO:teuthology.orchestra.run.smithi019.stdout:pg_num: 32 2024-03-18T15:18:03.544 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.543+0000 7f8e83200640 1 -- 172.21.15.19:0/3912708431 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f8e680a2800 msgr2=0x7f8e680a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:03.544 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.543+0000 7f8e83200640 1 --2- 172.21.15.19:0/3912708431 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f8e680a2800 0x7f8e680a4cc0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f8e74000c00 tx=0x7f8e74005fd0 comp rx=0 tx=0).stop 2024-03-18T15:18:03.544 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.543+0000 7f8e83200640 1 -- 172.21.15.19:0/3912708431 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8e841028b0 msgr2=0x7f8e8419a160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:03.544 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.543+0000 7f8e83200640 1 --2- 172.21.15.19:0/3912708431 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8e841028b0 0x7f8e8419a160 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f8e6c00e9f0 tx=0x7f8e6c00edf0 comp rx=0 tx=0).stop 2024-03-18T15:18:03.544 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.543+0000 7f8e83200640 1 -- 172.21.15.19:0/3912708431 shutdown_connections 2024-03-18T15:18:03.544 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.543+0000 7f8e83200640 1 --2- 172.21.15.19:0/3912708431 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8e84103270 0x7f8e841a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:03.544 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.543+0000 7f8e83200640 1 --2- 172.21.15.19:0/3912708431 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f8e680a2800 0x7f8e680a4cc0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:03.544 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.543+0000 7f8e83200640 1 --2- 172.21.15.19:0/3912708431 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8e841028b0 0x7f8e8419a160 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:03.544 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.543+0000 7f8e83200640 1 --2- 172.21.15.19:0/3912708431 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8e841016c0 0x7f8e84199c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:03.544 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.543+0000 7f8e83200640 1 -- 172.21.15.19:0/3912708431 >> 172.21.15.19:0/3912708431 conn(0x7f8e840fcc50 msgr2=0x7f8e84073a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:18:03.545 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.543+0000 7f8e83200640 1 -- 172.21.15.19:0/3912708431 shutdown_connections 2024-03-18T15:18:03.545 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.543+0000 7f8e83200640 1 -- 172.21.15.19:0/3912708431 wait complete. 2024-03-18T15:18:03.562 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T15:18:03.786 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.785+0000 7f2551e00640 1 -- 172.21.15.19:0/4049661530 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f254c107e00 msgr2=0x7f254c108220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:03.786 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.785+0000 7f2551e00640 1 --2- 172.21.15.19:0/4049661530 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f254c107e00 0x7f254c108220 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f253c009ca0 tx=0x7f253c02c100 comp rx=0 tx=0).stop 2024-03-18T15:18:03.787 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.785+0000 7f2551e00640 1 -- 172.21.15.19:0/4049661530 shutdown_connections 2024-03-18T15:18:03.787 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.785+0000 7f2551e00640 1 --2- 172.21.15.19:0/4049661530 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f254c1099b0 0x7f254c110240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:03.787 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.785+0000 7f2551e00640 1 --2- 172.21.15.19:0/4049661530 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f254c108ff0 0x7f254c109470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:03.787 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.785+0000 7f2551e00640 1 --2- 172.21.15.19:0/4049661530 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f254c107e00 0x7f254c108220 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:03.787 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.785+0000 7f2551e00640 1 -- 172.21.15.19:0/4049661530 >> 172.21.15.19:0/4049661530 conn(0x7f254c076da0 msgr2=0x7f254c0791e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:18:03.787 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.785+0000 7f2551e00640 1 -- 172.21.15.19:0/4049661530 shutdown_connections 2024-03-18T15:18:03.787 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.785+0000 7f2551e00640 1 -- 172.21.15.19:0/4049661530 wait complete. 2024-03-18T15:18:03.789 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.786+0000 7f2551e00640 1 Processor -- start 2024-03-18T15:18:03.789 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.787+0000 7f2551e00640 1 -- start start 2024-03-18T15:18:03.789 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.787+0000 7f2551e00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f254c107e00 0x7f254c19e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:03.789 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.787+0000 7f2551e00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f254c108ff0 0x7f254c19e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:03.789 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.787+0000 7f2551e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f254c1099b0 0x7f254c1a5640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:03.789 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.787+0000 7f2551e00640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f254c113a50 con 0x7f254c108ff0 2024-03-18T15:18:03.790 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.787+0000 7f2551e00640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f254c1138d0 con 0x7f254c1099b0 2024-03-18T15:18:03.790 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.787+0000 7f2551e00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f254c113bd0 con 0x7f254c107e00 2024-03-18T15:18:03.790 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.787+0000 7f254aa00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f254c107e00 0x7f254c19e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:03.790 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.787+0000 7f254aa00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f254c107e00 0x7f254c19e090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.19:39122/0 (socket says 172.21.15.19:39122) 2024-03-18T15:18:03.790 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.787+0000 7f254aa00640 1 -- 172.21.15.19:0/3913577174 learned_addr learned my addr 172.21.15.19:0/3913577174 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:18:03.790 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.787+0000 7f254b400640 1 --2- 172.21.15.19:0/3913577174 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f254c1099b0 0x7f254c1a5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:03.790 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.787+0000 7f254a000640 1 --2- 172.21.15.19:0/3913577174 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f254c108ff0 0x7f254c19e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:03.790 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.788+0000 7f254aa00640 1 -- 172.21.15.19:0/3913577174 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f254c1099b0 msgr2=0x7f254c1a5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:03.790 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.788+0000 7f254aa00640 1 --2- 172.21.15.19:0/3913577174 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f254c1099b0 0x7f254c1a5640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:03.790 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.788+0000 7f254aa00640 1 -- 172.21.15.19:0/3913577174 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f254c108ff0 msgr2=0x7f254c19e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:03.790 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.788+0000 7f254aa00640 1 --2- 172.21.15.19:0/3913577174 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f254c108ff0 0x7f254c19e5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:03.790 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.788+0000 7f254aa00640 1 -- 172.21.15.19:0/3913577174 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f254c1a5cb0 con 0x7f254c107e00 2024-03-18T15:18:03.790 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.788+0000 7f254b400640 1 --2- 172.21.15.19:0/3913577174 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f254c1099b0 0x7f254c1a5640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:18:03.790 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.788+0000 7f254aa00640 1 --2- 172.21.15.19:0/3913577174 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f254c107e00 0x7f254c19e090 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f253c037040 tx=0x7f253c035d90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:18:03.791 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.788+0000 7f2550a00640 1 -- 172.21.15.19:0/3913577174 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f253c039040 con 0x7f254c107e00 2024-03-18T15:18:03.791 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.788+0000 7f2550a00640 1 -- 172.21.15.19:0/3913577174 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f253c003970 con 0x7f254c107e00 2024-03-18T15:18:03.791 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.788+0000 7f2550a00640 1 -- 172.21.15.19:0/3913577174 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f253c0044b0 con 0x7f254c107e00 2024-03-18T15:18:03.791 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.788+0000 7f2551e00640 1 -- 172.21.15.19:0/3913577174 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f254c1a5f40 con 0x7f254c107e00 2024-03-18T15:18:03.791 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.789+0000 7f2551e00640 1 -- 172.21.15.19:0/3913577174 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f254c1a6350 con 0x7f254c107e00 2024-03-18T15:18:03.791 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.789+0000 7f2551e00640 1 -- 172.21.15.19:0/3913577174 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f254c10f120 con 0x7f254c107e00 2024-03-18T15:18:03.792 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.791+0000 7f2550a00640 1 -- 172.21.15.19:0/3913577174 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f253c046030 con 0x7f254c107e00 2024-03-18T15:18:03.794 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.793+0000 7f2550a00640 1 --2- 172.21.15.19:0/3913577174 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f25300a2800 0x7f25300a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:03.794 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.793+0000 7f2550a00640 1 -- 172.21.15.19:0/3913577174 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f253c0f0c80 con 0x7f254c107e00 2024-03-18T15:18:03.794 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.793+0000 7f254a000640 1 --2- 172.21.15.19:0/3913577174 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f25300a2800 0x7f25300a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:03.794 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.794+0000 7f254a000640 1 --2- 172.21.15.19:0/3913577174 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f25300a2800 0x7f25300a4cc0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f254c19f5a0 tx=0x7f253800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:18:03.803 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:03.802+0000 7f2550a00640 1 -- 172.21.15.19:0/3913577174 <== mon.1 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f253c04d060 con 0x7f254c107e00 2024-03-18T15:18:04.009 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.008+0000 7f2551e00640 1 -- 172.21.15.19:0/3913577174 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f254c062530 con 0x7f254c107e00 2024-03-18T15:18:04.011 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.010+0000 7f2550a00640 1 -- 172.21.15.19:0/3913577174 <== mon.1 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7f253c0bd980 con 0x7f254c107e00 2024-03-18T15:18:04.011 INFO:teuthology.orchestra.run.smithi019.stdout:pg_num: 64 2024-03-18T15:18:04.013 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.012+0000 7f2551e00640 1 -- 172.21.15.19:0/3913577174 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f25300a2800 msgr2=0x7f25300a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:04.013 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.012+0000 7f2551e00640 1 --2- 172.21.15.19:0/3913577174 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f25300a2800 0x7f25300a4cc0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f254c19f5a0 tx=0x7f253800b390 comp rx=0 tx=0).stop 2024-03-18T15:18:04.013 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.012+0000 7f2551e00640 1 -- 172.21.15.19:0/3913577174 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f254c107e00 msgr2=0x7f254c19e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:04.013 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.012+0000 7f2551e00640 1 --2- 172.21.15.19:0/3913577174 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f254c107e00 0x7f254c19e090 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f253c037040 tx=0x7f253c035d90 comp rx=0 tx=0).stop 2024-03-18T15:18:04.014 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.013+0000 7f2551e00640 1 -- 172.21.15.19:0/3913577174 shutdown_connections 2024-03-18T15:18:04.014 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.013+0000 7f2551e00640 1 --2- 172.21.15.19:0/3913577174 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f25300a2800 0x7f25300a4cc0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:04.014 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.013+0000 7f2551e00640 1 --2- 172.21.15.19:0/3913577174 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f254c1099b0 0x7f254c1a5640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:04.014 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.013+0000 7f2551e00640 1 --2- 172.21.15.19:0/3913577174 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f254c108ff0 0x7f254c19e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:04.014 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.013+0000 7f2551e00640 1 --2- 172.21.15.19:0/3913577174 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f254c107e00 0x7f254c19e090 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:04.014 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.013+0000 7f2551e00640 1 -- 172.21.15.19:0/3913577174 >> 172.21.15.19:0/3913577174 conn(0x7f254c076da0 msgr2=0x7f254c10c2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:18:04.014 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.013+0000 7f2551e00640 1 -- 172.21.15.19:0/3913577174 shutdown_connections 2024-03-18T15:18:04.015 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.013+0000 7f2551e00640 1 -- 172.21.15.19:0/3913577174 wait complete. 2024-03-18T15:18:04.033 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-18T15:18:04.258 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.257+0000 7fa343200640 1 -- 172.21.15.19:0/1781556960 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa344105490 msgr2=0x7fa34410bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:04.258 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.257+0000 7fa343200640 1 --2- 172.21.15.19:0/1781556960 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa344105490 0x7fa34410bd20 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fa33c009ca0 tx=0x7fa33c02c0f0 comp rx=0 tx=0).stop 2024-03-18T15:18:04.258 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.257+0000 7fa343200640 1 -- 172.21.15.19:0/1781556960 shutdown_connections 2024-03-18T15:18:04.259 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.257+0000 7fa343200640 1 --2- 172.21.15.19:0/1781556960 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa344105490 0x7fa34410bd20 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:04.259 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.257+0000 7fa343200640 1 --2- 172.21.15.19:0/1781556960 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa344104ad0 0x7fa344104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:04.259 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.257+0000 7fa343200640 1 --2- 172.21.15.19:0/1781556960 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa3441038e0 0x7fa344103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:04.259 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.257+0000 7fa343200640 1 -- 172.21.15.19:0/1781556960 >> 172.21.15.19:0/1781556960 conn(0x7fa3440fee50 msgr2=0x7fa344101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:18:04.259 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.257+0000 7fa343200640 1 -- 172.21.15.19:0/1781556960 shutdown_connections 2024-03-18T15:18:04.259 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.257+0000 7fa343200640 1 -- 172.21.15.19:0/1781556960 wait complete. 2024-03-18T15:18:04.260 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.258+0000 7fa343200640 1 Processor -- start 2024-03-18T15:18:04.260 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.259+0000 7fa343200640 1 -- start start 2024-03-18T15:18:04.260 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.259+0000 7fa343200640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa3441038e0 0x7fa344199b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:04.261 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.259+0000 7fa343200640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa344104ad0 0x7fa34419a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:04.261 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.259+0000 7fa343200640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa344105490 0x7fa3441a10d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:04.261 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.259+0000 7fa343200640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fa34410f560 con 0x7fa344104ad0 2024-03-18T15:18:04.261 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.259+0000 7fa343200640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa34410f3e0 con 0x7fa344105490 2024-03-18T15:18:04.261 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.259+0000 7fa343200640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fa34410f6e0 con 0x7fa3441038e0 2024-03-18T15:18:04.261 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.259+0000 7fa342800640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa344105490 0x7fa3441a10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:04.261 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.259+0000 7fa342800640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa344105490 0x7fa3441a10d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:46076/0 (socket says 172.21.15.19:46076) 2024-03-18T15:18:04.261 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.259+0000 7fa341e00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa3441038e0 0x7fa344199b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:04.261 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.259+0000 7fa342800640 1 -- 172.21.15.19:0/935280876 learned_addr learned my addr 172.21.15.19:0/935280876 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:18:04.261 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.259+0000 7fa341400640 1 --2- 172.21.15.19:0/935280876 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa344104ad0 0x7fa34419a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:04.262 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.260+0000 7fa342800640 1 -- 172.21.15.19:0/935280876 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa3441038e0 msgr2=0x7fa344199b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:04.262 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.260+0000 7fa342800640 1 --2- 172.21.15.19:0/935280876 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa3441038e0 0x7fa344199b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:04.262 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.260+0000 7fa342800640 1 -- 172.21.15.19:0/935280876 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa344104ad0 msgr2=0x7fa34419a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:04.262 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.260+0000 7fa342800640 1 --2- 172.21.15.19:0/935280876 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa344104ad0 0x7fa34419a060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:04.262 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.260+0000 7fa342800640 1 -- 172.21.15.19:0/935280876 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3441a1740 con 0x7fa344105490 2024-03-18T15:18:04.263 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.260+0000 7fa342800640 1 --2- 172.21.15.19:0/935280876 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa344105490 0x7fa3441a10d0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fa33c02c630 tx=0x7fa33c032ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:18:04.263 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.260+0000 7fa33aa00640 1 -- 172.21.15.19:0/935280876 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa33c041020 con 0x7fa344105490 2024-03-18T15:18:04.263 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.260+0000 7fa33aa00640 1 -- 172.21.15.19:0/935280876 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa33c034cd0 con 0x7fa344105490 2024-03-18T15:18:04.263 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.260+0000 7fa33aa00640 1 -- 172.21.15.19:0/935280876 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa33c039040 con 0x7fa344105490 2024-03-18T15:18:04.263 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.260+0000 7fa343200640 1 -- 172.21.15.19:0/935280876 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3441a19d0 con 0x7fa344105490 2024-03-18T15:18:04.263 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.260+0000 7fa343200640 1 -- 172.21.15.19:0/935280876 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3441a1e70 con 0x7fa344105490 2024-03-18T15:18:04.263 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.262+0000 7fa343200640 1 -- 172.21.15.19:0/935280876 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa34410aba0 con 0x7fa344105490 2024-03-18T15:18:04.264 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.263+0000 7fa33aa00640 1 -- 172.21.15.19:0/935280876 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fa33c044020 con 0x7fa344105490 2024-03-18T15:18:04.265 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.264+0000 7fa33aa00640 1 --2- 172.21.15.19:0/935280876 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fa3280a2880 0x7fa3280a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:04.265 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.265+0000 7fa33aa00640 1 -- 172.21.15.19:0/935280876 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fa33c0f0cd0 con 0x7fa344105490 2024-03-18T15:18:04.266 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.265+0000 7fa341e00640 1 --2- 172.21.15.19:0/935280876 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fa3280a2880 0x7fa3280a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:04.266 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.266+0000 7fa341e00640 1 --2- 172.21.15.19:0/935280876 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fa3280a2880 0x7fa3280a4d40 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fa344104930 tx=0x7fa334005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:18:04.270 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.269+0000 7fa33aa00640 1 -- 172.21.15.19:0/935280876 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa33c04b020 con 0x7fa344105490 2024-03-18T15:18:04.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:04 smithi104 ceph-mon[36070]: pgmap v407: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T15:18:04.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:04 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/175327555' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T15:18:04.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:04 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/3912708431' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T15:18:04.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:04 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/3913577174' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T15:18:04.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:04 smithi019 ceph-mon[31522]: pgmap v407: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T15:18:04.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:04 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/175327555' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T15:18:04.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:04 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/3912708431' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T15:18:04.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:04 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/3913577174' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T15:18:04.473 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.473+0000 7fa343200640 1 -- 172.21.15.19:0/935280876 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fa34402d1b0 con 0x7fa344105490 2024-03-18T15:18:04.474 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.473+0000 7fa33aa00640 1 -- 172.21.15.19:0/935280876 <== mon.2 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7fa33c0bd9d0 con 0x7fa344105490 2024-03-18T15:18:04.474 INFO:teuthology.orchestra.run.smithi019.stdout:pg_num: 32 2024-03-18T15:18:04.477 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.476+0000 7fa343200640 1 -- 172.21.15.19:0/935280876 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fa3280a2880 msgr2=0x7fa3280a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:04.477 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.476+0000 7fa343200640 1 --2- 172.21.15.19:0/935280876 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fa3280a2880 0x7fa3280a4d40 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fa344104930 tx=0x7fa334005f90 comp rx=0 tx=0).stop 2024-03-18T15:18:04.477 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.476+0000 7fa343200640 1 -- 172.21.15.19:0/935280876 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa344105490 msgr2=0x7fa3441a10d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:04.477 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.476+0000 7fa343200640 1 --2- 172.21.15.19:0/935280876 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa344105490 0x7fa3441a10d0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fa33c02c630 tx=0x7fa33c032ea0 comp rx=0 tx=0).stop 2024-03-18T15:18:04.478 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.477+0000 7fa343200640 1 -- 172.21.15.19:0/935280876 shutdown_connections 2024-03-18T15:18:04.478 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.477+0000 7fa343200640 1 --2- 172.21.15.19:0/935280876 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fa3280a2880 0x7fa3280a4d40 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:04.478 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.477+0000 7fa343200640 1 --2- 172.21.15.19:0/935280876 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa344105490 0x7fa3441a10d0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:04.478 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.477+0000 7fa343200640 1 --2- 172.21.15.19:0/935280876 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa344104ad0 0x7fa34419a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:04.478 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.477+0000 7fa343200640 1 --2- 172.21.15.19:0/935280876 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa3441038e0 0x7fa344199b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:04.478 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.477+0000 7fa343200640 1 -- 172.21.15.19:0/935280876 >> 172.21.15.19:0/935280876 conn(0x7fa3440fee50 msgr2=0x7fa344107d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:18:04.479 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.477+0000 7fa343200640 1 -- 172.21.15.19:0/935280876 shutdown_connections 2024-03-18T15:18:04.479 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.477+0000 7fa343200640 1 -- 172.21.15.19:0/935280876 wait complete. 2024-03-18T15:18:04.497 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T15:18:04.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:04 smithi033 ceph-mon[36648]: pgmap v407: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T15:18:04.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:04 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/175327555' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T15:18:04.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:04 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/3912708431' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T15:18:04.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:04 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/3913577174' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T15:18:04.718 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.716+0000 7f35ca000640 1 -- 172.21.15.19:0/1282159738 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f35c4107e00 msgr2=0x7f35c4108220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:04.718 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.716+0000 7f35ca000640 1 --2- 172.21.15.19:0/1282159738 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f35c4107e00 0x7f35c4108220 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f35b4009900 tx=0x7f35b402bc30 comp rx=0 tx=0).stop 2024-03-18T15:18:04.718 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.717+0000 7f35ca000640 1 -- 172.21.15.19:0/1282159738 shutdown_connections 2024-03-18T15:18:04.718 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.717+0000 7f35ca000640 1 --2- 172.21.15.19:0/1282159738 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f35c41099b0 0x7f35c4110240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:04.718 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.717+0000 7f35ca000640 1 --2- 172.21.15.19:0/1282159738 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f35c4108ff0 0x7f35c4109470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:04.718 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.717+0000 7f35ca000640 1 --2- 172.21.15.19:0/1282159738 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f35c4107e00 0x7f35c4108220 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:04.718 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.717+0000 7f35ca000640 1 -- 172.21.15.19:0/1282159738 >> 172.21.15.19:0/1282159738 conn(0x7f35c4076dc0 msgr2=0x7f35c40791e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:18:04.718 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.717+0000 7f35ca000640 1 -- 172.21.15.19:0/1282159738 shutdown_connections 2024-03-18T15:18:04.718 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.717+0000 7f35ca000640 1 -- 172.21.15.19:0/1282159738 wait complete. 2024-03-18T15:18:04.721 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.718+0000 7f35ca000640 1 Processor -- start 2024-03-18T15:18:04.721 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.718+0000 7f35ca000640 1 -- start start 2024-03-18T15:18:04.721 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.718+0000 7f35ca000640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f35c4107e00 0x7f35c419e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:04.721 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.718+0000 7f35ca000640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f35c4108ff0 0x7f35c419e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:04.721 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.718+0000 7f35ca000640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f35c41099b0 0x7f35c41a56b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:04.721 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.718+0000 7f35ca000640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f35c4113a50 con 0x7f35c4108ff0 2024-03-18T15:18:04.721 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.718+0000 7f35ca000640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f35c41138d0 con 0x7f35c41099b0 2024-03-18T15:18:04.721 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.718+0000 7f35ca000640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f35c4113bd0 con 0x7f35c4107e00 2024-03-18T15:18:04.721 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.718+0000 7f35c3400640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f35c4107e00 0x7f35c419e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:04.721 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.718+0000 7f35c2a00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f35c4108ff0 0x7f35c419e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:04.722 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.718+0000 7f35c3400640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f35c4107e00 0x7f35c419e100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.19:39146/0 (socket says 172.21.15.19:39146) 2024-03-18T15:18:04.722 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.718+0000 7f35c3e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f35c41099b0 0x7f35c41a56b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:04.722 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.718+0000 7f35c3400640 1 -- 172.21.15.19:0/914352894 learned_addr learned my addr 172.21.15.19:0/914352894 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:18:04.722 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.719+0000 7f35c3400640 1 -- 172.21.15.19:0/914352894 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f35c41099b0 msgr2=0x7f35c41a56b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:04.722 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.719+0000 7f35c3400640 1 --2- 172.21.15.19:0/914352894 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f35c41099b0 0x7f35c41a56b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:04.722 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.719+0000 7f35c3400640 1 -- 172.21.15.19:0/914352894 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f35c4108ff0 msgr2=0x7f35c419e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:04.722 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.719+0000 7f35c3400640 1 --2- 172.21.15.19:0/914352894 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f35c4108ff0 0x7f35c419e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:04.722 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.719+0000 7f35c3400640 1 -- 172.21.15.19:0/914352894 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35c41a5d20 con 0x7f35c4107e00 2024-03-18T15:18:04.722 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.719+0000 7f35c3e00640 1 --2- 172.21.15.19:0/914352894 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f35c41099b0 0x7f35c41a56b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:18:04.722 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.719+0000 7f35c3400640 1 --2- 172.21.15.19:0/914352894 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f35c4107e00 0x7f35c419e100 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f35b4000c00 tx=0x7f35b402c640 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:18:04.722 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.719+0000 7f35bbe00640 1 -- 172.21.15.19:0/914352894 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35b4041070 con 0x7f35c4107e00 2024-03-18T15:18:04.722 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.720+0000 7f35ca000640 1 -- 172.21.15.19:0/914352894 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35c41a5fb0 con 0x7f35c4107e00 2024-03-18T15:18:04.723 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.720+0000 7f35bbe00640 1 -- 172.21.15.19:0/914352894 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f35b4004450 con 0x7f35c4107e00 2024-03-18T15:18:04.723 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.720+0000 7f35bbe00640 1 -- 172.21.15.19:0/914352894 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35b4003df0 con 0x7f35c4107e00 2024-03-18T15:18:04.723 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.720+0000 7f35ca000640 1 -- 172.21.15.19:0/914352894 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35c41a6410 con 0x7f35c4107e00 2024-03-18T15:18:04.723 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.721+0000 7f35ca000640 1 -- 172.21.15.19:0/914352894 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35c410f120 con 0x7f35c4107e00 2024-03-18T15:18:04.723 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.723+0000 7f35bbe00640 1 -- 172.21.15.19:0/914352894 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f35b4044070 con 0x7f35c4107e00 2024-03-18T15:18:04.725 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.724+0000 7f35bbe00640 1 --2- 172.21.15.19:0/914352894 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f35a80a2880 0x7f35a80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:04.725 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.724+0000 7f35bbe00640 1 -- 172.21.15.19:0/914352894 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f35b40f0ac0 con 0x7f35c4107e00 2024-03-18T15:18:04.725 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.724+0000 7f35c2a00640 1 --2- 172.21.15.19:0/914352894 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f35a80a2880 0x7f35a80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:04.726 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.725+0000 7f35c2a00640 1 --2- 172.21.15.19:0/914352894 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f35a80a2880 0x7f35a80a4d40 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f35c419f610 tx=0x7f35b0009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:18:04.728 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.727+0000 7f35bbe00640 1 -- 172.21.15.19:0/914352894 <== mon.1 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f35b40bd7c0 con 0x7f35c4107e00 2024-03-18T15:18:04.976 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.975+0000 7f35ca000640 1 -- 172.21.15.19:0/914352894 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f35c402d1b0 con 0x7f35c4107e00 2024-03-18T15:18:04.977 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.977+0000 7f35bbe00640 1 -- 172.21.15.19:0/914352894 <== mon.1 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11927 (secure 0 0 0) 0x7f35b40bd0c0 con 0x7f35c4107e00 2024-03-18T15:18:04.978 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-18T15:18:04.978 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6835/1135329545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6834","nonce":1135329545},{"type":"v1","addr":"172.21.15.19:6835","nonce":1135329545}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14457,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6839/3833518557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6838","nonce":3833518557},{"type":"v1","addr":"172.21.15.19:6839","nonce":3833518557}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T15:15:10.589737+0000","modified":"2024-03-18T15:15:46.979999+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24560,"mds_1":24529,"mds_2":24511,"mds_3":24506,"mds_4":24499},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14475":{"gid":14475,"name":"a","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.19:6841/3176583319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6840","nonce":3176583319},{"type":"v1","addr":"172.21.15.19:6841","nonce":3176583319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24433":{"gid":24433,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6833/482725809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":482725809},{"type":"v1","addr":"172.21.15.104:6833","nonce":482725809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24434":{"gid":24434,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/2272279159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":2272279159},{"type":"v1","addr":"172.21.15.33:6833","nonce":2272279159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24469":{"gid":24469,"name":"d","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.19:6837/1778019859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6836","nonce":1778019859},{"type":"v1","addr":"172.21.15.19:6837","nonce":1778019859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6835/4051613151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":4051613151},{"type":"v1","addr":"172.21.15.33:6835","nonce":4051613151}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":17,"addr":"172.21.15.104:6835/3788595957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3788595957},{"type":"v1","addr":"172.21.15.104:6835","nonce":3788595957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":15,"addr":"172.21.15.33:6837/160187378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":160187378},{"type":"v1","addr":"172.21.15.33:6837","nonce":160187378}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24511":{"gid":24511,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":16,"addr":"172.21.15.104:6837/260984674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":260984674},{"type":"v1","addr":"172.21.15.104:6837","nonce":260984674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24529":{"gid":24529,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.104:6839/805341436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6838","nonce":805341436},{"type":"v1","addr":"172.21.15.104:6839","nonce":805341436}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.33:6839/1903591535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":1903591535},{"type":"v1","addr":"172.21.15.33:6839","nonce":1903591535}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24560,"qdb_cluster":[24506,24499,24529,24511,24560]},"id":1}]} 2024-03-18T15:18:04.978 INFO:teuthology.orchestra.run.smithi019.stderr:dumped fsmap epoch 26 2024-03-18T15:18:04.981 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.979+0000 7f35ca000640 1 -- 172.21.15.19:0/914352894 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f35a80a2880 msgr2=0x7f35a80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:04.981 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.979+0000 7f35ca000640 1 --2- 172.21.15.19:0/914352894 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f35a80a2880 0x7f35a80a4d40 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f35c419f610 tx=0x7f35b0009500 comp rx=0 tx=0).stop 2024-03-18T15:18:04.981 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.979+0000 7f35ca000640 1 -- 172.21.15.19:0/914352894 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f35c4107e00 msgr2=0x7f35c419e100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:04.981 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.979+0000 7f35ca000640 1 --2- 172.21.15.19:0/914352894 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f35c4107e00 0x7f35c419e100 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f35b4000c00 tx=0x7f35b402c640 comp rx=0 tx=0).stop 2024-03-18T15:18:04.981 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.980+0000 7f35ca000640 1 -- 172.21.15.19:0/914352894 shutdown_connections 2024-03-18T15:18:04.981 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.980+0000 7f35ca000640 1 --2- 172.21.15.19:0/914352894 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f35a80a2880 0x7f35a80a4d40 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:04.981 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.980+0000 7f35ca000640 1 --2- 172.21.15.19:0/914352894 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f35c41099b0 0x7f35c41a56b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:04.981 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.980+0000 7f35ca000640 1 --2- 172.21.15.19:0/914352894 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f35c4108ff0 0x7f35c419e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:04.981 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.980+0000 7f35ca000640 1 --2- 172.21.15.19:0/914352894 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f35c4107e00 0x7f35c419e100 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:04.981 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.980+0000 7f35ca000640 1 -- 172.21.15.19:0/914352894 >> 172.21.15.19:0/914352894 conn(0x7f35c4076dc0 msgr2=0x7f35c410c2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:18:04.982 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.980+0000 7f35ca000640 1 -- 172.21.15.19:0/914352894 shutdown_connections 2024-03-18T15:18:04.982 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:04.980+0000 7f35ca000640 1 -- 172.21.15.19:0/914352894 wait complete. 2024-03-18T15:18:05.001 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T15:18:05.227 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.226+0000 7f4fff800640 1 -- 172.21.15.19:0/2198493305 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4ff81055b0 msgr2=0x7f4ff810be40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:05.227 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.226+0000 7f4fff800640 1 --2- 172.21.15.19:0/2198493305 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4ff81055b0 0x7f4ff810be40 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f4ff0009ca0 tx=0x7f4ff002c100 comp rx=0 tx=0).stop 2024-03-18T15:18:05.227 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.226+0000 7f4fff800640 1 -- 172.21.15.19:0/2198493305 shutdown_connections 2024-03-18T15:18:05.227 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.226+0000 7f4fff800640 1 --2- 172.21.15.19:0/2198493305 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4ff81055b0 0x7f4ff810be40 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:05.228 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.226+0000 7f4fff800640 1 --2- 172.21.15.19:0/2198493305 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4ff8104bf0 0x7f4ff8105070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:05.228 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.226+0000 7f4fff800640 1 --2- 172.21.15.19:0/2198493305 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4ff8103a00 0x7f4ff8103e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:05.228 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.226+0000 7f4fff800640 1 -- 172.21.15.19:0/2198493305 >> 172.21.15.19:0/2198493305 conn(0x7f4ff80fef90 msgr2=0x7f4ff81013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:18:05.228 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.226+0000 7f4fff800640 1 -- 172.21.15.19:0/2198493305 shutdown_connections 2024-03-18T15:18:05.228 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.226+0000 7f4fff800640 1 -- 172.21.15.19:0/2198493305 wait complete. 2024-03-18T15:18:05.230 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.227+0000 7f4fff800640 1 Processor -- start 2024-03-18T15:18:05.230 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.227+0000 7f4fff800640 1 -- start start 2024-03-18T15:18:05.230 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.227+0000 7f4fff800640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4ff8103a00 0x7f4ff8199c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:05.230 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.227+0000 7f4fff800640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4ff8104bf0 0x7f4ff819a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:05.230 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.227+0000 7f4fff800640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4ff81055b0 0x7f4ff81a11f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:05.230 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.227+0000 7f4fff800640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f4ff810f680 con 0x7f4ff81055b0 2024-03-18T15:18:05.230 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.227+0000 7f4fff800640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4ff810f500 con 0x7f4ff8103a00 2024-03-18T15:18:05.230 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.227+0000 7f4fff800640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f4ff810f800 con 0x7f4ff8104bf0 2024-03-18T15:18:05.230 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.228+0000 7f4ffce00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4ff8103a00 0x7f4ff8199c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:05.230 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.228+0000 7f4ff7e00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4ff8104bf0 0x7f4ff819a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:05.230 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.228+0000 7f4ffce00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4ff8103a00 0x7f4ff8199c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:46108/0 (socket says 172.21.15.19:46108) 2024-03-18T15:18:05.230 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.228+0000 7f4ffce00640 1 -- 172.21.15.19:0/3820182721 learned_addr learned my addr 172.21.15.19:0/3820182721 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:18:05.231 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.228+0000 7f4ff7e00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4ff8104bf0 0x7f4ff819a1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.19:39162/0 (socket says 172.21.15.19:39162) 2024-03-18T15:18:05.231 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.228+0000 7f4ffd800640 1 --2- 172.21.15.19:0/3820182721 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4ff81055b0 0x7f4ff81a11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:05.231 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.228+0000 7f4ffce00640 1 -- 172.21.15.19:0/3820182721 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4ff8104bf0 msgr2=0x7f4ff819a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:05.231 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.228+0000 7f4ffce00640 1 --2- 172.21.15.19:0/3820182721 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4ff8104bf0 0x7f4ff819a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:05.231 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.228+0000 7f4ffce00640 1 -- 172.21.15.19:0/3820182721 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4ff81055b0 msgr2=0x7f4ff81a11f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:05.231 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.228+0000 7f4ffce00640 1 --2- 172.21.15.19:0/3820182721 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4ff81055b0 0x7f4ff81a11f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:05.231 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.228+0000 7f4ffce00640 1 -- 172.21.15.19:0/3820182721 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ff81a1890 con 0x7f4ff8103a00 2024-03-18T15:18:05.231 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.228+0000 7f4ffce00640 1 --2- 172.21.15.19:0/3820182721 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4ff8103a00 0x7f4ff8199c80 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f4fec00e9c0 tx=0x7f4fec00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:18:05.231 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.229+0000 7f4ff5600640 1 -- 172.21.15.19:0/3820182721 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fec00cd80 con 0x7f4ff8103a00 2024-03-18T15:18:05.231 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.229+0000 7f4ff5600640 1 -- 172.21.15.19:0/3820182721 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4fec0035f0 con 0x7f4ff8103a00 2024-03-18T15:18:05.231 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.229+0000 7f4fff800640 1 -- 172.21.15.19:0/3820182721 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ff81a1b80 con 0x7f4ff8103a00 2024-03-18T15:18:05.231 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.229+0000 7f4ff5600640 1 -- 172.21.15.19:0/3820182721 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fec010690 con 0x7f4ff8103a00 2024-03-18T15:18:05.231 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.229+0000 7f4fff800640 1 -- 172.21.15.19:0/3820182721 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ff81a20d0 con 0x7f4ff8103a00 2024-03-18T15:18:05.232 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.230+0000 7f4fff800640 1 -- 172.21.15.19:0/3820182721 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ff810ad20 con 0x7f4ff8103a00 2024-03-18T15:18:05.232 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.232+0000 7f4ff5600640 1 -- 172.21.15.19:0/3820182721 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f4fec0037a0 con 0x7f4ff8103a00 2024-03-18T15:18:05.234 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.233+0000 7f4ff5600640 1 --2- 172.21.15.19:0/3820182721 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f4fe00a2880 0x7f4fe00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:05.234 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.233+0000 7f4ff5600640 1 -- 172.21.15.19:0/3820182721 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f4fec0cf1f0 con 0x7f4ff8103a00 2024-03-18T15:18:05.234 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.233+0000 7f4ff7e00640 1 --2- 172.21.15.19:0/3820182721 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f4fe00a2880 0x7f4fe00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:05.235 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.234+0000 7f4ff7e00640 1 --2- 172.21.15.19:0/3820182721 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f4fe00a2880 0x7f4fe00a4d40 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f4fe4009bd0 tx=0x7f4fe4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:18:05.238 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.237+0000 7f4ff5600640 1 -- 172.21.15.19:0/3820182721 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4fec09be60 con 0x7f4ff8103a00 2024-03-18T15:18:05.407 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:05 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/935280876' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T15:18:05.407 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:05 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/914352894' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:18:05.443 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.442+0000 7f4fff800640 1 -- 172.21.15.19:0/3820182721 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4ff8062480 con 0x7f4ff8103a00 2024-03-18T15:18:05.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:05 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/935280876' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T15:18:05.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:05 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/914352894' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:18:05.446 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.445+0000 7f4ff5600640 1 -- 172.21.15.19:0/3820182721 <== mon.2 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24809 (secure 0 0 0) 0x7f4fec0860e0 con 0x7f4ff8103a00 2024-03-18T15:18:05.446 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-18T15:18:05.447 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":93,"fsid":"d4dc2ba6-e538-11ee-95c9-87774f69a715","created":"2024-03-18T15:05:57.538551+0000","modified":"2024-03-18T15:17:16.033313+0000","last_up_change":"2024-03-18T15:14:12.009522+0000","last_in_change":"2024-03-18T15:13:52.555387+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T15:09:56.843349+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T15:15:08.941047+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T15:15:10.044671+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T15:15:13.621916+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"37b5a5cd-27ae-4e77-9b05-774b817b2ee9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6802","nonce":402333948},{"type":"v1","addr":"172.21.15.19:6803","nonce":402333948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6804","nonce":402333948},{"type":"v1","addr":"172.21.15.19:6805","nonce":402333948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6808","nonce":402333948},{"type":"v1","addr":"172.21.15.19:6809","nonce":402333948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6806","nonce":402333948},{"type":"v1","addr":"172.21.15.19:6807","nonce":402333948}]},"public_addr":"172.21.15.19:6803/402333948","cluster_addr":"172.21.15.19:6805/402333948","heartbeat_back_addr":"172.21.15.19:6809/402333948","heartbeat_front_addr":"172.21.15.19:6807/402333948","state":["exists","up"]},{"osd":1,"uuid":"3a1134e8-ede7-4dbd-9be1-eefd2cbb43bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1951880113},{"type":"v1","addr":"172.21.15.33:6801","nonce":1951880113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1951880113},{"type":"v1","addr":"172.21.15.33:6803","nonce":1951880113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1951880113},{"type":"v1","addr":"172.21.15.33:6807","nonce":1951880113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1951880113},{"type":"v1","addr":"172.21.15.33:6805","nonce":1951880113}]},"public_addr":"172.21.15.33:6801/1951880113","cluster_addr":"172.21.15.33:6803/1951880113","heartbeat_back_addr":"172.21.15.33:6807/1951880113","heartbeat_front_addr":"172.21.15.33:6805/1951880113","state":["exists","up"]},{"osd":2,"uuid":"f383a164-a3a0-49fd-a9e8-9c2178d8cfb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":1004727949},{"type":"v1","addr":"172.21.15.104:6801","nonce":1004727949}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":1004727949},{"type":"v1","addr":"172.21.15.104:6803","nonce":1004727949}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":1004727949},{"type":"v1","addr":"172.21.15.104:6807","nonce":1004727949}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":1004727949},{"type":"v1","addr":"172.21.15.104:6805","nonce":1004727949}]},"public_addr":"172.21.15.104:6801/1004727949","cluster_addr":"172.21.15.104:6803/1004727949","heartbeat_back_addr":"172.21.15.104:6807/1004727949","heartbeat_front_addr":"172.21.15.104:6805/1004727949","state":["exists","up"]},{"osd":3,"uuid":"c409e116-28bf-45dd-a791-539af22abe1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6810","nonce":3571525854},{"type":"v1","addr":"172.21.15.19:6811","nonce":3571525854}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6812","nonce":3571525854},{"type":"v1","addr":"172.21.15.19:6813","nonce":3571525854}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6816","nonce":3571525854},{"type":"v1","addr":"172.21.15.19:6817","nonce":3571525854}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6814","nonce":3571525854},{"type":"v1","addr":"172.21.15.19:6815","nonce":3571525854}]},"public_addr":"172.21.15.19:6811/3571525854","cluster_addr":"172.21.15.19:6813/3571525854","heartbeat_back_addr":"172.21.15.19:6817/3571525854","heartbeat_front_addr":"172.21.15.19:6815/3571525854","state":["exists","up"]},{"osd":4,"uuid":"696eb719-0b3e-43ae-b5f1-ccd73a3b2843","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":624901749},{"type":"v1","addr":"172.21.15.33:6809","nonce":624901749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":624901749},{"type":"v1","addr":"172.21.15.33:6811","nonce":624901749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":624901749},{"type":"v1","addr":"172.21.15.33:6815","nonce":624901749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":624901749},{"type":"v1","addr":"172.21.15.33:6813","nonce":624901749}]},"public_addr":"172.21.15.33:6809/624901749","cluster_addr":"172.21.15.33:6811/624901749","heartbeat_back_addr":"172.21.15.33:6815/624901749","heartbeat_front_addr":"172.21.15.33:6813/624901749","state":["exists","up"]},{"osd":5,"uuid":"21789779-8bb9-42c7-b9ee-437985f11e3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":1258208297},{"type":"v1","addr":"172.21.15.104:6809","nonce":1258208297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":1258208297},{"type":"v1","addr":"172.21.15.104:6811","nonce":1258208297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":1258208297},{"type":"v1","addr":"172.21.15.104:6815","nonce":1258208297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":1258208297},{"type":"v1","addr":"172.21.15.104:6813","nonce":1258208297}]},"public_addr":"172.21.15.104:6809/1258208297","cluster_addr":"172.21.15.104:6811/1258208297","heartbeat_back_addr":"172.21.15.104:6815/1258208297","heartbeat_front_addr":"172.21.15.104:6813/1258208297","state":["exists","up"]},{"osd":6,"uuid":"0e92a2ef-43cd-487c-9203-e219158aef8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6818","nonce":2782482301},{"type":"v1","addr":"172.21.15.19:6819","nonce":2782482301}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6820","nonce":2782482301},{"type":"v1","addr":"172.21.15.19:6821","nonce":2782482301}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6824","nonce":2782482301},{"type":"v1","addr":"172.21.15.19:6825","nonce":2782482301}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6822","nonce":2782482301},{"type":"v1","addr":"172.21.15.19:6823","nonce":2782482301}]},"public_addr":"172.21.15.19:6819/2782482301","cluster_addr":"172.21.15.19:6821/2782482301","heartbeat_back_addr":"172.21.15.19:6825/2782482301","heartbeat_front_addr":"172.21.15.19:6823/2782482301","state":["exists","up"]},{"osd":7,"uuid":"a16e39a1-3685-491d-b376-3e80a2084072","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":4238160988},{"type":"v1","addr":"172.21.15.33:6817","nonce":4238160988}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":4238160988},{"type":"v1","addr":"172.21.15.33:6819","nonce":4238160988}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":4238160988},{"type":"v1","addr":"172.21.15.33:6823","nonce":4238160988}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":4238160988},{"type":"v1","addr":"172.21.15.33:6821","nonce":4238160988}]},"public_addr":"172.21.15.33:6817/4238160988","cluster_addr":"172.21.15.33:6819/4238160988","heartbeat_back_addr":"172.21.15.33:6823/4238160988","heartbeat_front_addr":"172.21.15.33:6821/4238160988","state":["exists","up"]},{"osd":8,"uuid":"d5c3fa92-cd5d-4349-9c7d-d5666bacde16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":1014779813},{"type":"v1","addr":"172.21.15.104:6817","nonce":1014779813}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":1014779813},{"type":"v1","addr":"172.21.15.104:6819","nonce":1014779813}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":1014779813},{"type":"v1","addr":"172.21.15.104:6823","nonce":1014779813}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":1014779813},{"type":"v1","addr":"172.21.15.104:6821","nonce":1014779813}]},"public_addr":"172.21.15.104:6817/1014779813","cluster_addr":"172.21.15.104:6819/1014779813","heartbeat_back_addr":"172.21.15.104:6823/1014779813","heartbeat_front_addr":"172.21.15.104:6821/1014779813","state":["exists","up"]},{"osd":9,"uuid":"eed2e9b2-26d3-4c46-88ec-34b5596fe991","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6826","nonce":2641151009},{"type":"v1","addr":"172.21.15.19:6827","nonce":2641151009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6828","nonce":2641151009},{"type":"v1","addr":"172.21.15.19:6829","nonce":2641151009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6832","nonce":2641151009},{"type":"v1","addr":"172.21.15.19:6833","nonce":2641151009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6830","nonce":2641151009},{"type":"v1","addr":"172.21.15.19:6831","nonce":2641151009}]},"public_addr":"172.21.15.19:6827/2641151009","cluster_addr":"172.21.15.19:6829/2641151009","heartbeat_back_addr":"172.21.15.19:6833/2641151009","heartbeat_front_addr":"172.21.15.19:6831/2641151009","state":["exists","up"]},{"osd":10,"uuid":"f3c35ea5-fbe7-4dc4-93c5-b92d1a714db3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":1346058869},{"type":"v1","addr":"172.21.15.33:6825","nonce":1346058869}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":1346058869},{"type":"v1","addr":"172.21.15.33:6827","nonce":1346058869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":1346058869},{"type":"v1","addr":"172.21.15.33:6831","nonce":1346058869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":1346058869},{"type":"v1","addr":"172.21.15.33:6829","nonce":1346058869}]},"public_addr":"172.21.15.33:6825/1346058869","cluster_addr":"172.21.15.33:6827/1346058869","heartbeat_back_addr":"172.21.15.33:6831/1346058869","heartbeat_front_addr":"172.21.15.33:6829/1346058869","state":["exists","up"]},{"osd":11,"uuid":"c8b51571-f807-4b62-90b6-7529e09e59da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":3759498281},{"type":"v1","addr":"172.21.15.104:6825","nonce":3759498281}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":3759498281},{"type":"v1","addr":"172.21.15.104:6827","nonce":3759498281}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":3759498281},{"type":"v1","addr":"172.21.15.104:6831","nonce":3759498281}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":3759498281},{"type":"v1","addr":"172.21.15.104:6829","nonce":3759498281}]},"public_addr":"172.21.15.104:6825/3759498281","cluster_addr":"172.21.15.104:6827/3759498281","heartbeat_back_addr":"172.21.15.104:6831/3759498281","heartbeat_front_addr":"172.21.15.104:6829/3759498281","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:09:05.267795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:09:30.234068+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:09:53.870232+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:10:26.471296+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:10:53.180341+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:11:17.801361+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:11:51.364816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:12:18.306408+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:12:43.978744+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:13:18.213964+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:13:45.437544+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:14:10.148580+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":3,"to":6}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"3.37","mappings":[{"from":5,"to":8}]},{"pgid":"3.3a","mappings":[{"from":4,"to":7}]},{"pgid":"4.8","mappings":[{"from":5,"to":8}]},{"pgid":"4.f","mappings":[{"from":9,"to":10}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.19:0/3860783274":"2024-03-19T15:07:14.150983+0000","172.21.15.19:0/2737223213":"2024-03-19T15:07:14.150983+0000","172.21.15.19:0/176699025":"2024-03-19T15:06:38.688405+0000","172.21.15.19:0/2316969005":"2024-03-19T15:07:14.150983+0000","172.21.15.19:6801/2776008962":"2024-03-19T15:07:14.150983+0000","172.21.15.19:0/2793980440":"2024-03-19T15:06:38.688405+0000","172.21.15.19:6800/3264337803":"2024-03-19T15:06:38.688405+0000","172.21.15.19:6801/3264337803":"2024-03-19T15:06:38.688405+0000","172.21.15.19:0/3166705295":"2024-03-19T15:06:38.688405+0000","172.21.15.19:6800/2776008962":"2024-03-19T15:07:14.150983+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T15:18:05.449 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.448+0000 7f4fff800640 1 -- 172.21.15.19:0/3820182721 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f4fe00a2880 msgr2=0x7f4fe00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:05.449 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.448+0000 7f4fff800640 1 --2- 172.21.15.19:0/3820182721 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f4fe00a2880 0x7f4fe00a4d40 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f4fe4009bd0 tx=0x7f4fe4005f90 comp rx=0 tx=0).stop 2024-03-18T15:18:05.449 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.448+0000 7f4fff800640 1 -- 172.21.15.19:0/3820182721 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4ff8103a00 msgr2=0x7f4ff8199c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:05.449 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.448+0000 7f4fff800640 1 --2- 172.21.15.19:0/3820182721 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4ff8103a00 0x7f4ff8199c80 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f4fec00e9c0 tx=0x7f4fec00ee80 comp rx=0 tx=0).stop 2024-03-18T15:18:05.449 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.448+0000 7f4fff800640 1 -- 172.21.15.19:0/3820182721 shutdown_connections 2024-03-18T15:18:05.449 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.448+0000 7f4fff800640 1 --2- 172.21.15.19:0/3820182721 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4ff81055b0 0x7f4ff81a11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:05.449 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.448+0000 7f4fff800640 1 --2- 172.21.15.19:0/3820182721 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4ff8104bf0 0x7f4ff819a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:05.450 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.448+0000 7f4fff800640 1 --2- 172.21.15.19:0/3820182721 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f4fe00a2880 0x7f4fe00a4d40 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:05.450 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.448+0000 7f4fff800640 1 --2- 172.21.15.19:0/3820182721 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4ff8103a00 0x7f4ff8199c80 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:05.450 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.448+0000 7f4fff800640 1 -- 172.21.15.19:0/3820182721 >> 172.21.15.19:0/3820182721 conn(0x7f4ff80fef90 msgr2=0x7f4ff8107ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:18:05.450 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.448+0000 7f4fff800640 1 -- 172.21.15.19:0/3820182721 shutdown_connections 2024-03-18T15:18:05.450 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.448+0000 7f4fff800640 1 -- 172.21.15.19:0/3820182721 wait complete. 2024-03-18T15:18:05.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:05 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/935280876' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T15:18:05.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:05 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/914352894' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:18:05.471 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T15:09:56.843349+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T15:15:08.941047+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T15:15:10.044671+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-18T15:15:13.621916+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-18T15:18:05.471 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T15:18:05.694 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.692+0000 7f1b51000640 1 -- 172.21.15.19:0/2392772449 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1b4c1028d0 msgr2=0x7f1b4c102d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:05.694 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.692+0000 7f1b51000640 1 --2- 172.21.15.19:0/2392772449 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1b4c1028d0 0x7f1b4c102d50 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f1b38009900 tx=0x7f1b3802bc40 comp rx=0 tx=0).stop 2024-03-18T15:18:05.694 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.693+0000 7f1b51000640 1 -- 172.21.15.19:0/2392772449 shutdown_connections 2024-03-18T15:18:05.694 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.693+0000 7f1b51000640 1 --2- 172.21.15.19:0/2392772449 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1b4c103290 0x7f1b4c109b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:05.694 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.693+0000 7f1b51000640 1 --2- 172.21.15.19:0/2392772449 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1b4c1028d0 0x7f1b4c102d50 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:05.694 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.693+0000 7f1b51000640 1 --2- 172.21.15.19:0/2392772449 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1b4c1016e0 0x7f1b4c101b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:05.695 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.693+0000 7f1b51000640 1 -- 172.21.15.19:0/2392772449 >> 172.21.15.19:0/2392772449 conn(0x7f1b4c0fcc50 msgr2=0x7f1b4c0ff090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:18:05.695 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.693+0000 7f1b51000640 1 -- 172.21.15.19:0/2392772449 shutdown_connections 2024-03-18T15:18:05.695 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.693+0000 7f1b51000640 1 -- 172.21.15.19:0/2392772449 wait complete. 2024-03-18T15:18:05.697 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.694+0000 7f1b51000640 1 Processor -- start 2024-03-18T15:18:05.697 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.694+0000 7f1b51000640 1 -- start start 2024-03-18T15:18:05.697 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.695+0000 7f1b51000640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1b4c1016e0 0x7f1b4c199c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:05.697 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.695+0000 7f1b51000640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1b4c1028d0 0x7f1b4c19a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:05.697 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.695+0000 7f1b51000640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1b4c103290 0x7f1b4c1a11c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:05.697 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.695+0000 7f1b51000640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f1b4c10d390 con 0x7f1b4c103290 2024-03-18T15:18:05.697 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.695+0000 7f1b51000640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1b4c10d210 con 0x7f1b4c1028d0 2024-03-18T15:18:05.697 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.695+0000 7f1b51000640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f1b4c10d510 con 0x7f1b4c1016e0 2024-03-18T15:18:05.697 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.695+0000 7f1b4b400640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1b4c1016e0 0x7f1b4c199c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:05.698 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.695+0000 7f1b4aa00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1b4c1028d0 0x7f1b4c19a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:05.698 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.695+0000 7f1b4b400640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1b4c1016e0 0x7f1b4c199c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.19:39188/0 (socket says 172.21.15.19:39188) 2024-03-18T15:18:05.698 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.695+0000 7f1b4b400640 1 -- 172.21.15.19:0/598313908 learned_addr learned my addr 172.21.15.19:0/598313908 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:18:05.698 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.695+0000 7f1b4be00640 1 --2- 172.21.15.19:0/598313908 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1b4c103290 0x7f1b4c1a11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:05.698 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.695+0000 7f1b4b400640 1 -- 172.21.15.19:0/598313908 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1b4c1028d0 msgr2=0x7f1b4c19a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:05.698 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.695+0000 7f1b4b400640 1 --2- 172.21.15.19:0/598313908 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1b4c1028d0 0x7f1b4c19a190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:05.698 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.695+0000 7f1b4b400640 1 -- 172.21.15.19:0/598313908 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1b4c103290 msgr2=0x7f1b4c1a11c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:05.698 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.695+0000 7f1b4b400640 1 --2- 172.21.15.19:0/598313908 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1b4c103290 0x7f1b4c1a11c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:05.698 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.695+0000 7f1b4b400640 1 -- 172.21.15.19:0/598313908 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b4c1a1860 con 0x7f1b4c1016e0 2024-03-18T15:18:05.698 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.695+0000 7f1b4aa00640 1 --2- 172.21.15.19:0/598313908 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1b4c1028d0 0x7f1b4c19a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:18:05.698 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.696+0000 7f1b4b400640 1 --2- 172.21.15.19:0/598313908 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1b4c1016e0 0x7f1b4c199c50 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f1b3c00e9c0 tx=0x7f1b3c00ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:18:05.698 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.696+0000 7f1b43e00640 1 -- 172.21.15.19:0/598313908 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b3c00cd80 con 0x7f1b4c1016e0 2024-03-18T15:18:05.698 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.696+0000 7f1b43e00640 1 -- 172.21.15.19:0/598313908 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1b3c0035f0 con 0x7f1b4c1016e0 2024-03-18T15:18:05.699 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.696+0000 7f1b43e00640 1 -- 172.21.15.19:0/598313908 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b3c010690 con 0x7f1b4c1016e0 2024-03-18T15:18:05.699 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.696+0000 7f1b51000640 1 -- 172.21.15.19:0/598313908 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b4c1a1b50 con 0x7f1b4c1016e0 2024-03-18T15:18:05.699 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.696+0000 7f1b51000640 1 -- 172.21.15.19:0/598313908 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b4c1a20f0 con 0x7f1b4c1016e0 2024-03-18T15:18:05.699 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.697+0000 7f1b51000640 1 -- 172.21.15.19:0/598313908 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b4c1a22c0 con 0x7f1b4c1016e0 2024-03-18T15:18:05.700 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.700+0000 7f1b43e00640 1 -- 172.21.15.19:0/598313908 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f1b3c0037a0 con 0x7f1b4c1016e0 2024-03-18T15:18:05.702 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.701+0000 7f1b43e00640 1 --2- 172.21.15.19:0/598313908 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f1b300a2880 0x7f1b300a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:05.702 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.701+0000 7f1b43e00640 1 -- 172.21.15.19:0/598313908 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f1b3c0cf250 con 0x7f1b4c1016e0 2024-03-18T15:18:05.702 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.701+0000 7f1b4aa00640 1 --2- 172.21.15.19:0/598313908 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f1b300a2880 0x7f1b300a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:05.703 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.702+0000 7f1b4aa00640 1 --2- 172.21.15.19:0/598313908 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f1b300a2880 0x7f1b300a4d40 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f1b3802c250 tx=0x7f1b3802cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:18:05.709 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.708+0000 7f1b43e00640 1 -- 172.21.15.19:0/598313908 <== mon.1 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1b3c09bec0 con 0x7f1b4c1016e0 2024-03-18T15:18:05.915 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.914+0000 7f1b51000640 1 -- 172.21.15.19:0/598313908 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f1b4c1a2490 con 0x7f1b4c1016e0 2024-03-18T15:18:05.916 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.915+0000 7f1b43e00640 1 -- 172.21.15.19:0/598313908 <== mon.1 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7f1b3c09b7c0 con 0x7f1b4c1016e0 2024-03-18T15:18:05.916 INFO:teuthology.orchestra.run.smithi019.stdout:pg_num: 1 2024-03-18T15:18:05.919 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.918+0000 7f1b51000640 1 -- 172.21.15.19:0/598313908 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f1b300a2880 msgr2=0x7f1b300a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:05.919 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.918+0000 7f1b51000640 1 --2- 172.21.15.19:0/598313908 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f1b300a2880 0x7f1b300a4d40 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f1b3802c250 tx=0x7f1b3802cf90 comp rx=0 tx=0).stop 2024-03-18T15:18:05.919 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.918+0000 7f1b51000640 1 -- 172.21.15.19:0/598313908 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1b4c1016e0 msgr2=0x7f1b4c199c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:05.919 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.918+0000 7f1b51000640 1 --2- 172.21.15.19:0/598313908 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1b4c1016e0 0x7f1b4c199c50 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f1b3c00e9c0 tx=0x7f1b3c00ee80 comp rx=0 tx=0).stop 2024-03-18T15:18:05.919 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.918+0000 7f1b51000640 1 -- 172.21.15.19:0/598313908 shutdown_connections 2024-03-18T15:18:05.919 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.918+0000 7f1b51000640 1 --2- 172.21.15.19:0/598313908 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1b4c103290 0x7f1b4c1a11c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:05.919 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.918+0000 7f1b51000640 1 --2- 172.21.15.19:0/598313908 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f1b300a2880 0x7f1b300a4d40 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:05.919 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.918+0000 7f1b51000640 1 --2- 172.21.15.19:0/598313908 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1b4c1028d0 0x7f1b4c19a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:05.919 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.918+0000 7f1b51000640 1 --2- 172.21.15.19:0/598313908 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1b4c1016e0 0x7f1b4c199c50 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:05.920 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.918+0000 7f1b51000640 1 -- 172.21.15.19:0/598313908 >> 172.21.15.19:0/598313908 conn(0x7f1b4c0fcc50 msgr2=0x7f1b4c073a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:18:05.920 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.918+0000 7f1b51000640 1 -- 172.21.15.19:0/598313908 shutdown_connections 2024-03-18T15:18:05.920 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:05.918+0000 7f1b51000640 1 -- 172.21.15.19:0/598313908 wait complete. 2024-03-18T15:18:05.940 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T15:18:06.170 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.169+0000 7fc259400640 1 -- 172.21.15.19:0/1857252890 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc254102140 msgr2=0x7fc254104530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:06.170 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.169+0000 7fc259400640 1 --2- 172.21.15.19:0/1857252890 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc254102140 0x7fc254104530 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fc24c009860 tx=0x7fc24c02bb70 comp rx=0 tx=0).stop 2024-03-18T15:18:06.170 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.169+0000 7fc259400640 1 -- 172.21.15.19:0/1857252890 shutdown_connections 2024-03-18T15:18:06.170 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.169+0000 7fc259400640 1 --2- 172.21.15.19:0/1857252890 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc2541073a0 0x7fc2541097f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:06.170 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.169+0000 7fc259400640 1 --2- 172.21.15.19:0/1857252890 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc254104a70 0x7fc254106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:06.170 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.169+0000 7fc259400640 1 --2- 172.21.15.19:0/1857252890 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc254102140 0x7fc254104530 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:06.170 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.169+0000 7fc259400640 1 -- 172.21.15.19:0/1857252890 >> 172.21.15.19:0/1857252890 conn(0x7fc2540fbb50 msgr2=0x7fc2540fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:18:06.171 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.169+0000 7fc259400640 1 -- 172.21.15.19:0/1857252890 shutdown_connections 2024-03-18T15:18:06.171 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.169+0000 7fc259400640 1 -- 172.21.15.19:0/1857252890 wait complete. 2024-03-18T15:18:06.172 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.170+0000 7fc259400640 1 Processor -- start 2024-03-18T15:18:06.172 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.170+0000 7fc259400640 1 -- start start 2024-03-18T15:18:06.172 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.170+0000 7fc259400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc254102140 0x7fc254101ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:06.172 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.170+0000 7fc259400640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc254104a70 0x7fc2541000f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:06.172 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.171+0000 7fc259400640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc2541073a0 0x7fc254100630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:06.172 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.171+0000 7fc259400640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fc25410d370 con 0x7fc254104a70 2024-03-18T15:18:06.172 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.171+0000 7fc259400640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc25410d1f0 con 0x7fc254102140 2024-03-18T15:18:06.172 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.171+0000 7fc259400640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fc25410d4f0 con 0x7fc2541073a0 2024-03-18T15:18:06.172 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.171+0000 7fc253e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc254102140 0x7fc254101ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:06.172 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.171+0000 7fc253e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc254102140 0x7fc254101ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:46128/0 (socket says 172.21.15.19:46128) 2024-03-18T15:18:06.172 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.171+0000 7fc253e00640 1 -- 172.21.15.19:0/2221460305 learned_addr learned my addr 172.21.15.19:0/2221460305 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:18:06.173 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.171+0000 7fc253400640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc254104a70 0x7fc2541000f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:06.173 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.171+0000 7fc258a00640 1 --2- 172.21.15.19:0/2221460305 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc2541073a0 0x7fc254100630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:06.174 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.171+0000 7fc253e00640 1 -- 172.21.15.19:0/2221460305 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc2541073a0 msgr2=0x7fc254100630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:06.174 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.171+0000 7fc253e00640 1 --2- 172.21.15.19:0/2221460305 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc2541073a0 0x7fc254100630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:06.174 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.171+0000 7fc253e00640 1 -- 172.21.15.19:0/2221460305 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc254104a70 msgr2=0x7fc2541000f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:06.174 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.171+0000 7fc253e00640 1 --2- 172.21.15.19:0/2221460305 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc254104a70 0x7fc2541000f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:06.174 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.171+0000 7fc253e00640 1 -- 172.21.15.19:0/2221460305 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc254100ec0 con 0x7fc254102140 2024-03-18T15:18:06.174 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.171+0000 7fc258a00640 1 --2- 172.21.15.19:0/2221460305 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc2541073a0 0x7fc254100630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:18:06.174 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.172+0000 7fc253e00640 1 --2- 172.21.15.19:0/2221460305 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc254102140 0x7fc254101ab0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fc24c0349e0 tx=0x7fc24c032450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:18:06.174 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.172+0000 7fc250c00640 1 -- 172.21.15.19:0/2221460305 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc24c041030 con 0x7fc254102140 2024-03-18T15:18:06.174 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.172+0000 7fc250c00640 1 -- 172.21.15.19:0/2221460305 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc24c02c510 con 0x7fc254102140 2024-03-18T15:18:06.174 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.172+0000 7fc250c00640 1 -- 172.21.15.19:0/2221460305 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc24c039040 con 0x7fc254102140 2024-03-18T15:18:06.175 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.172+0000 7fc259400640 1 -- 172.21.15.19:0/2221460305 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2541a1aa0 con 0x7fc254102140 2024-03-18T15:18:06.175 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.172+0000 7fc259400640 1 -- 172.21.15.19:0/2221460305 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2541a2040 con 0x7fc254102140 2024-03-18T15:18:06.175 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.173+0000 7fc259400640 1 -- 172.21.15.19:0/2221460305 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2540fd640 con 0x7fc254102140 2024-03-18T15:18:06.176 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.176+0000 7fc250c00640 1 -- 172.21.15.19:0/2221460305 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fc24c044070 con 0x7fc254102140 2024-03-18T15:18:06.178 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.177+0000 7fc250c00640 1 --2- 172.21.15.19:0/2221460305 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fc2380a2880 0x7fc2380a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:06.178 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.177+0000 7fc250c00640 1 -- 172.21.15.19:0/2221460305 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fc24c0f0330 con 0x7fc254102140 2024-03-18T15:18:06.178 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.177+0000 7fc253400640 1 --2- 172.21.15.19:0/2221460305 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fc2380a2880 0x7fc2380a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:06.179 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.178+0000 7fc253400640 1 --2- 172.21.15.19:0/2221460305 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fc2380a2880 0x7fc2380a4d40 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fc254101820 tx=0x7fc23c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:18:06.185 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.184+0000 7fc250c00640 1 -- 172.21.15.19:0/2221460305 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc24c0bd020 con 0x7fc254102140 2024-03-18T15:18:06.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:06 smithi019 ceph-mon[31522]: pgmap v408: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T15:18:06.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:06 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/3820182721' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:18:06.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:06 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/598313908' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T15:18:06.390 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.389+0000 7fc259400640 1 -- 172.21.15.19:0/2221460305 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fc25402d400 con 0x7fc254102140 2024-03-18T15:18:06.391 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.389+0000 7fc250c00640 1 -- 172.21.15.19:0/2221460305 <== mon.2 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7fc24c0bc920 con 0x7fc254102140 2024-03-18T15:18:06.391 INFO:teuthology.orchestra.run.smithi019.stdout:pg_num: 32 2024-03-18T15:18:06.393 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.392+0000 7fc259400640 1 -- 172.21.15.19:0/2221460305 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fc2380a2880 msgr2=0x7fc2380a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:06.393 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.392+0000 7fc259400640 1 --2- 172.21.15.19:0/2221460305 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fc2380a2880 0x7fc2380a4d40 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fc254101820 tx=0x7fc23c009260 comp rx=0 tx=0).stop 2024-03-18T15:18:06.393 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.392+0000 7fc259400640 1 -- 172.21.15.19:0/2221460305 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc254102140 msgr2=0x7fc254101ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:06.394 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.392+0000 7fc259400640 1 --2- 172.21.15.19:0/2221460305 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc254102140 0x7fc254101ab0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fc24c0349e0 tx=0x7fc24c032450 comp rx=0 tx=0).stop 2024-03-18T15:18:06.394 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.392+0000 7fc259400640 1 -- 172.21.15.19:0/2221460305 shutdown_connections 2024-03-18T15:18:06.394 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.392+0000 7fc259400640 1 --2- 172.21.15.19:0/2221460305 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc2541073a0 0x7fc254100630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:06.394 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.392+0000 7fc259400640 1 --2- 172.21.15.19:0/2221460305 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc254104a70 0x7fc2541000f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:06.394 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.392+0000 7fc259400640 1 --2- 172.21.15.19:0/2221460305 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fc2380a2880 0x7fc2380a4d40 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:06.394 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.392+0000 7fc259400640 1 --2- 172.21.15.19:0/2221460305 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc254102140 0x7fc254101ab0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:06.394 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.392+0000 7fc259400640 1 -- 172.21.15.19:0/2221460305 >> 172.21.15.19:0/2221460305 conn(0x7fc2540fbb50 msgr2=0x7fc254102ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:18:06.394 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.392+0000 7fc259400640 1 -- 172.21.15.19:0/2221460305 shutdown_connections 2024-03-18T15:18:06.394 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.392+0000 7fc259400640 1 -- 172.21.15.19:0/2221460305 wait complete. 2024-03-18T15:18:06.415 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T15:18:06.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:06 smithi104 ceph-mon[36070]: pgmap v408: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T15:18:06.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:06 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/3820182721' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:18:06.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:06 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/598313908' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T15:18:06.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:06 smithi033 ceph-mon[36648]: pgmap v408: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T15:18:06.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:06 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/3820182721' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:18:06.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:06 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/598313908' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T15:18:06.643 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.642+0000 7f45cfe00640 1 -- 172.21.15.19:0/308421896 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f45d01028b0 msgr2=0x7f45d0102d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:06.643 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.642+0000 7f45cfe00640 1 --2- 172.21.15.19:0/308421896 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f45d01028b0 0x7f45d0102d30 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f45b8009900 tx=0x7f45b802bc50 comp rx=0 tx=0).stop 2024-03-18T15:18:06.643 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.642+0000 7f45cfe00640 1 -- 172.21.15.19:0/308421896 shutdown_connections 2024-03-18T15:18:06.643 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.642+0000 7f45cfe00640 1 --2- 172.21.15.19:0/308421896 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f45d0103270 0x7f45d0109b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:06.644 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.642+0000 7f45cfe00640 1 --2- 172.21.15.19:0/308421896 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f45d01028b0 0x7f45d0102d30 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:06.644 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.642+0000 7f45cfe00640 1 --2- 172.21.15.19:0/308421896 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f45d01016c0 0x7f45d0101ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:06.644 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.642+0000 7f45cfe00640 1 -- 172.21.15.19:0/308421896 >> 172.21.15.19:0/308421896 conn(0x7f45d00fcc50 msgr2=0x7f45d00ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:18:06.644 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.642+0000 7f45cfe00640 1 -- 172.21.15.19:0/308421896 shutdown_connections 2024-03-18T15:18:06.644 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.642+0000 7f45cfe00640 1 -- 172.21.15.19:0/308421896 wait complete. 2024-03-18T15:18:06.644 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.643+0000 7f45cfe00640 1 Processor -- start 2024-03-18T15:18:06.644 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.644+0000 7f45cfe00640 1 -- start start 2024-03-18T15:18:06.646 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.644+0000 7f45cfe00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f45d01016c0 0x7f45d0075590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:06.646 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.644+0000 7f45cfe00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f45d01028b0 0x7f45d0073bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:06.646 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.644+0000 7f45cfe00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f45d0103270 0x7f45d0074110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:06.647 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.644+0000 7f45cfe00640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f45d010d390 con 0x7f45d01016c0 2024-03-18T15:18:06.647 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.644+0000 7f45cfe00640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f45d010d210 con 0x7f45d0103270 2024-03-18T15:18:06.647 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.644+0000 7f45cfe00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f45d010d510 con 0x7f45d01028b0 2024-03-18T15:18:06.647 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.644+0000 7f45cf400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f45d0103270 0x7f45d0074110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:06.647 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.644+0000 7f45cf400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f45d0103270 0x7f45d0074110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:46138/0 (socket says 172.21.15.19:46138) 2024-03-18T15:18:06.647 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.644+0000 7f45cf400640 1 -- 172.21.15.19:0/2836947064 learned_addr learned my addr 172.21.15.19:0/2836947064 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:18:06.647 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.644+0000 7f45c6000640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f45d01028b0 0x7f45d0073bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:06.647 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.645+0000 7f45cea00640 1 --2- 172.21.15.19:0/2836947064 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f45d01016c0 0x7f45d0075590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:06.647 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.645+0000 7f45cf400640 1 -- 172.21.15.19:0/2836947064 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f45d01028b0 msgr2=0x7f45d0073bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:06.647 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.645+0000 7f45cf400640 1 --2- 172.21.15.19:0/2836947064 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f45d01028b0 0x7f45d0073bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:06.647 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.645+0000 7f45cf400640 1 -- 172.21.15.19:0/2836947064 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f45d01016c0 msgr2=0x7f45d0075590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:06.647 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.645+0000 7f45cf400640 1 --2- 172.21.15.19:0/2836947064 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f45d01016c0 0x7f45d0075590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:06.647 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.645+0000 7f45cf400640 1 -- 172.21.15.19:0/2836947064 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45d00749a0 con 0x7f45d0103270 2024-03-18T15:18:06.647 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.645+0000 7f45cea00640 1 --2- 172.21.15.19:0/2836947064 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f45d01016c0 0x7f45d0075590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T15:18:06.648 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.645+0000 7f45cf400640 1 --2- 172.21.15.19:0/2836947064 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f45d0103270 0x7f45d0074110 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f45c800e970 tx=0x7f45c800ed70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:18:06.648 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.645+0000 7f45d4a00640 1 -- 172.21.15.19:0/2836947064 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45c800ccc0 con 0x7f45d0103270 2024-03-18T15:18:06.648 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.645+0000 7f45d4a00640 1 -- 172.21.15.19:0/2836947064 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f45c800ce60 con 0x7f45d0103270 2024-03-18T15:18:06.648 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.645+0000 7f45d4a00640 1 -- 172.21.15.19:0/2836947064 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45c8010490 con 0x7f45d0103270 2024-03-18T15:18:06.648 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.645+0000 7f45cfe00640 1 -- 172.21.15.19:0/2836947064 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45d01a1ac0 con 0x7f45d0103270 2024-03-18T15:18:06.648 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.646+0000 7f45cfe00640 1 -- 172.21.15.19:0/2836947064 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45d01a2030 con 0x7f45d0103270 2024-03-18T15:18:06.648 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.647+0000 7f45cfe00640 1 -- 172.21.15.19:0/2836947064 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45d0108980 con 0x7f45d0103270 2024-03-18T15:18:06.649 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.648+0000 7f45d4a00640 1 -- 172.21.15.19:0/2836947064 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f45c8005020 con 0x7f45d0103270 2024-03-18T15:18:06.651 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.650+0000 7f45d4a00640 1 --2- 172.21.15.19:0/2836947064 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f45b40a2880 0x7f45b40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:06.651 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.650+0000 7f45d4a00640 1 -- 172.21.15.19:0/2836947064 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f45c80cf2b0 con 0x7f45d0103270 2024-03-18T15:18:06.651 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.650+0000 7f45cea00640 1 --2- 172.21.15.19:0/2836947064 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f45b40a2880 0x7f45b40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:06.651 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.651+0000 7f45cea00640 1 --2- 172.21.15.19:0/2836947064 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f45b40a2880 0x7f45b40a4d40 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f45d0102710 tx=0x7f45c0009540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:18:06.655 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.654+0000 7f45d4a00640 1 -- 172.21.15.19:0/2836947064 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f45c809bf80 con 0x7f45d0103270 2024-03-18T15:18:06.870 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.869+0000 7f45cfe00640 1 -- 172.21.15.19:0/2836947064 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f45d00623e0 con 0x7f45d0103270 2024-03-18T15:18:06.870 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.870+0000 7f45d4a00640 1 -- 172.21.15.19:0/2836947064 <== mon.2 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7f45c809b880 con 0x7f45d0103270 2024-03-18T15:18:06.871 INFO:teuthology.orchestra.run.smithi019.stdout:pg_num: 64 2024-03-18T15:18:06.873 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.872+0000 7f45cfe00640 1 -- 172.21.15.19:0/2836947064 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f45b40a2880 msgr2=0x7f45b40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:06.873 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.872+0000 7f45cfe00640 1 --2- 172.21.15.19:0/2836947064 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f45b40a2880 0x7f45b40a4d40 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f45d0102710 tx=0x7f45c0009540 comp rx=0 tx=0).stop 2024-03-18T15:18:06.874 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.872+0000 7f45cfe00640 1 -- 172.21.15.19:0/2836947064 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f45d0103270 msgr2=0x7f45d0074110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:06.874 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.872+0000 7f45cfe00640 1 --2- 172.21.15.19:0/2836947064 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f45d0103270 0x7f45d0074110 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f45c800e970 tx=0x7f45c800ed70 comp rx=0 tx=0).stop 2024-03-18T15:18:06.874 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.872+0000 7f45cfe00640 1 -- 172.21.15.19:0/2836947064 shutdown_connections 2024-03-18T15:18:06.874 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.872+0000 7f45cfe00640 1 --2- 172.21.15.19:0/2836947064 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f45b40a2880 0x7f45b40a4d40 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:06.874 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.872+0000 7f45cfe00640 1 --2- 172.21.15.19:0/2836947064 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f45d0103270 0x7f45d0074110 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:06.874 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.872+0000 7f45cfe00640 1 --2- 172.21.15.19:0/2836947064 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f45d01028b0 0x7f45d0073bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:06.874 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.872+0000 7f45cfe00640 1 --2- 172.21.15.19:0/2836947064 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f45d01016c0 0x7f45d0075590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:06.874 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.872+0000 7f45cfe00640 1 -- 172.21.15.19:0/2836947064 >> 172.21.15.19:0/2836947064 conn(0x7f45d00fcc50 msgr2=0x7f45d0105b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:18:06.874 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.873+0000 7f45cfe00640 1 -- 172.21.15.19:0/2836947064 shutdown_connections 2024-03-18T15:18:06.874 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:06.873+0000 7f45cfe00640 1 -- 172.21.15.19:0/2836947064 wait complete. 2024-03-18T15:18:06.893 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-18T15:18:07.121 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.120+0000 7ffb34c00640 1 -- 172.21.15.19:0/2177411667 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffb301016e0 msgr2=0x7ffb30101b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:07.121 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.120+0000 7ffb34c00640 1 --2- 172.21.15.19:0/2177411667 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffb301016e0 0x7ffb30101b00 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7ffb18009900 tx=0x7ffb1802bc30 comp rx=0 tx=0).stop 2024-03-18T15:18:07.121 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.120+0000 7ffb34c00640 1 -- 172.21.15.19:0/2177411667 shutdown_connections 2024-03-18T15:18:07.122 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.120+0000 7ffb34c00640 1 --2- 172.21.15.19:0/2177411667 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffb30103290 0x7ffb30109b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:07.122 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.120+0000 7ffb34c00640 1 --2- 172.21.15.19:0/2177411667 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ffb301028d0 0x7ffb30102d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:07.122 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.120+0000 7ffb34c00640 1 --2- 172.21.15.19:0/2177411667 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffb301016e0 0x7ffb30101b00 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:07.122 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.120+0000 7ffb34c00640 1 -- 172.21.15.19:0/2177411667 >> 172.21.15.19:0/2177411667 conn(0x7ffb300fcc50 msgr2=0x7ffb300ff090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:18:07.122 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.120+0000 7ffb34c00640 1 -- 172.21.15.19:0/2177411667 shutdown_connections 2024-03-18T15:18:07.122 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.120+0000 7ffb34c00640 1 -- 172.21.15.19:0/2177411667 wait complete. 2024-03-18T15:18:07.124 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.121+0000 7ffb34c00640 1 Processor -- start 2024-03-18T15:18:07.124 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.121+0000 7ffb34c00640 1 -- start start 2024-03-18T15:18:07.124 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.121+0000 7ffb34c00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffb301016e0 0x7ffb30075590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:07.124 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.121+0000 7ffb34c00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ffb301028d0 0x7ffb30073bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:07.124 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.121+0000 7ffb34c00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffb30103290 0x7ffb30074110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:07.124 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.121+0000 7ffb34c00640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7ffb3010d380 con 0x7ffb301028d0 2024-03-18T15:18:07.124 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.121+0000 7ffb34c00640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ffb3010d200 con 0x7ffb301016e0 2024-03-18T15:18:07.124 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.121+0000 7ffb34c00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7ffb3010d500 con 0x7ffb30103290 2024-03-18T15:18:07.125 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.122+0000 7ffb2f400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffb301016e0 0x7ffb30075590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:07.125 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.122+0000 7ffb2ea00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ffb301028d0 0x7ffb30073bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:07.125 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.122+0000 7ffb2fe00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffb30103290 0x7ffb30074110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:07.125 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.122+0000 7ffb2f400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffb301016e0 0x7ffb30075590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:46152/0 (socket says 172.21.15.19:46152) 2024-03-18T15:18:07.125 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.122+0000 7ffb2f400640 1 -- 172.21.15.19:0/730171778 learned_addr learned my addr 172.21.15.19:0/730171778 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:18:07.125 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.122+0000 7ffb2f400640 1 -- 172.21.15.19:0/730171778 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffb30103290 msgr2=0x7ffb30074110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:07.125 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.122+0000 7ffb2f400640 1 --2- 172.21.15.19:0/730171778 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffb30103290 0x7ffb30074110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:07.125 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.122+0000 7ffb2f400640 1 -- 172.21.15.19:0/730171778 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ffb301028d0 msgr2=0x7ffb30073bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:07.125 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.122+0000 7ffb2f400640 1 --2- 172.21.15.19:0/730171778 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ffb301028d0 0x7ffb30073bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:07.125 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.122+0000 7ffb2f400640 1 -- 172.21.15.19:0/730171778 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb300749a0 con 0x7ffb301016e0 2024-03-18T15:18:07.125 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.122+0000 7ffb2f400640 1 --2- 172.21.15.19:0/730171778 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffb301016e0 0x7ffb30075590 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7ffb18000c00 tx=0x7ffb1802c6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:18:07.125 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.123+0000 7ffb27e00640 1 -- 172.21.15.19:0/730171778 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb18043020 con 0x7ffb301016e0 2024-03-18T15:18:07.125 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.123+0000 7ffb27e00640 1 -- 172.21.15.19:0/730171778 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffb180054e0 con 0x7ffb301016e0 2024-03-18T15:18:07.126 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.123+0000 7ffb27e00640 1 -- 172.21.15.19:0/730171778 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb18003ce0 con 0x7ffb301016e0 2024-03-18T15:18:07.126 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.123+0000 7ffb34c00640 1 -- 172.21.15.19:0/730171778 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb301a1b40 con 0x7ffb301016e0 2024-03-18T15:18:07.126 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.123+0000 7ffb34c00640 1 -- 172.21.15.19:0/730171778 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb301a2010 con 0x7ffb301016e0 2024-03-18T15:18:07.126 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.124+0000 7ffb34c00640 1 -- 172.21.15.19:0/730171778 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb30108a50 con 0x7ffb301016e0 2024-03-18T15:18:07.126 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.125+0000 7ffb27e00640 1 -- 172.21.15.19:0/730171778 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7ffb18046020 con 0x7ffb301016e0 2024-03-18T15:18:07.128 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.127+0000 7ffb27e00640 1 --2- 172.21.15.19:0/730171778 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7ffb140a2880 0x7ffb140a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:07.128 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.127+0000 7ffb27e00640 1 -- 172.21.15.19:0/730171778 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7ffb180f0f00 con 0x7ffb301016e0 2024-03-18T15:18:07.128 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.127+0000 7ffb2ea00640 1 --2- 172.21.15.19:0/730171778 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7ffb140a2880 0x7ffb140a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:07.128 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.128+0000 7ffb2ea00640 1 --2- 172.21.15.19:0/730171778 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7ffb140a2880 0x7ffb140a4d40 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7ffb30075300 tx=0x7ffb20009760 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:18:07.133 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.132+0000 7ffb27e00640 1 -- 172.21.15.19:0/730171778 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ffb1804d020 con 0x7ffb301016e0 2024-03-18T15:18:07.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:07 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/2221460305' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T15:18:07.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:07 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/2836947064' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T15:18:07.340 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.339+0000 7ffb34c00640 1 -- 172.21.15.19:0/730171778 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7ffb3002d1b0 con 0x7ffb301016e0 2024-03-18T15:18:07.340 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.339+0000 7ffb27e00640 1 -- 172.21.15.19:0/730171778 <== mon.2 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7ffb180bdb70 con 0x7ffb301016e0 2024-03-18T15:18:07.341 INFO:teuthology.orchestra.run.smithi019.stdout:pg_num: 32 2024-03-18T15:18:07.343 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.342+0000 7ffb34c00640 1 -- 172.21.15.19:0/730171778 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7ffb140a2880 msgr2=0x7ffb140a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:07.343 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.342+0000 7ffb34c00640 1 --2- 172.21.15.19:0/730171778 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7ffb140a2880 0x7ffb140a4d40 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7ffb30075300 tx=0x7ffb20009760 comp rx=0 tx=0).stop 2024-03-18T15:18:07.343 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.342+0000 7ffb34c00640 1 -- 172.21.15.19:0/730171778 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffb301016e0 msgr2=0x7ffb30075590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:07.343 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.342+0000 7ffb34c00640 1 --2- 172.21.15.19:0/730171778 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffb301016e0 0x7ffb30075590 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7ffb18000c00 tx=0x7ffb1802c6c0 comp rx=0 tx=0).stop 2024-03-18T15:18:07.343 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.342+0000 7ffb34c00640 1 -- 172.21.15.19:0/730171778 shutdown_connections 2024-03-18T15:18:07.343 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.342+0000 7ffb34c00640 1 --2- 172.21.15.19:0/730171778 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffb30103290 0x7ffb30074110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:07.343 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.342+0000 7ffb34c00640 1 --2- 172.21.15.19:0/730171778 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ffb301028d0 0x7ffb30073bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:07.343 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.342+0000 7ffb34c00640 1 --2- 172.21.15.19:0/730171778 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7ffb140a2880 0x7ffb140a4d40 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:07.344 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.342+0000 7ffb34c00640 1 --2- 172.21.15.19:0/730171778 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffb301016e0 0x7ffb30075590 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:07.344 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.342+0000 7ffb34c00640 1 -- 172.21.15.19:0/730171778 >> 172.21.15.19:0/730171778 conn(0x7ffb300fcc50 msgr2=0x7ffb30105c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:18:07.344 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.342+0000 7ffb34c00640 1 -- 172.21.15.19:0/730171778 shutdown_connections 2024-03-18T15:18:07.344 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.343+0000 7ffb34c00640 1 -- 172.21.15.19:0/730171778 wait complete. 2024-03-18T15:18:07.364 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T15:18:07.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:07 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/2221460305' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T15:18:07.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:07 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/2836947064' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T15:18:07.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:07 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/2221460305' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T15:18:07.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:07 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/2836947064' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T15:18:07.586 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.584+0000 7fd9c1400640 1 -- 172.21.15.19:0/3236175740 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd9bc1055b0 msgr2=0x7fd9bc10be40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:07.586 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.585+0000 7fd9c1400640 1 --2- 172.21.15.19:0/3236175740 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd9bc1055b0 0x7fd9bc10be40 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fd9b4009ca0 tx=0x7fd9b402c0f0 comp rx=0 tx=0).stop 2024-03-18T15:18:07.586 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.585+0000 7fd9c1400640 1 -- 172.21.15.19:0/3236175740 shutdown_connections 2024-03-18T15:18:07.586 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.585+0000 7fd9c1400640 1 --2- 172.21.15.19:0/3236175740 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd9bc1055b0 0x7fd9bc10be40 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:07.586 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.585+0000 7fd9c1400640 1 --2- 172.21.15.19:0/3236175740 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd9bc104bf0 0x7fd9bc105070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:07.586 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.585+0000 7fd9c1400640 1 --2- 172.21.15.19:0/3236175740 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd9bc103a00 0x7fd9bc103e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:07.586 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.585+0000 7fd9c1400640 1 -- 172.21.15.19:0/3236175740 >> 172.21.15.19:0/3236175740 conn(0x7fd9bc0fef90 msgr2=0x7fd9bc1013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:18:07.587 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.585+0000 7fd9c1400640 1 -- 172.21.15.19:0/3236175740 shutdown_connections 2024-03-18T15:18:07.587 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.585+0000 7fd9c1400640 1 -- 172.21.15.19:0/3236175740 wait complete. 2024-03-18T15:18:07.588 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.586+0000 7fd9c1400640 1 Processor -- start 2024-03-18T15:18:07.589 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.586+0000 7fd9c1400640 1 -- start start 2024-03-18T15:18:07.589 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.586+0000 7fd9c1400640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd9bc103a00 0x7fd9bc199c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:07.589 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.586+0000 7fd9c1400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd9bc104bf0 0x7fd9bc19a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:07.589 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.586+0000 7fd9c1400640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd9bc1055b0 0x7fd9bc1a1210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:07.589 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.586+0000 7fd9c1400640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fd9bc10f660 con 0x7fd9bc103a00 2024-03-18T15:18:07.589 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.586+0000 7fd9c1400640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd9bc10f4e0 con 0x7fd9bc104bf0 2024-03-18T15:18:07.589 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.586+0000 7fd9c1400640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fd9bc10f7e0 con 0x7fd9bc1055b0 2024-03-18T15:18:07.589 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.587+0000 7fd9ba800640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd9bc103a00 0x7fd9bc199c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:07.589 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.587+0000 7fd9b9e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd9bc104bf0 0x7fd9bc19a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:07.589 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.587+0000 7fd9b9e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd9bc104bf0 0x7fd9bc19a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:46170/0 (socket says 172.21.15.19:46170) 2024-03-18T15:18:07.589 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.587+0000 7fd9bb200640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd9bc1055b0 0x7fd9bc1a1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:07.589 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.587+0000 7fd9b9e00640 1 -- 172.21.15.19:0/2837558383 learned_addr learned my addr 172.21.15.19:0/2837558383 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:18:07.589 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.587+0000 7fd9bb200640 1 -- 172.21.15.19:0/2837558383 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd9bc104bf0 msgr2=0x7fd9bc19a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:07.590 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.587+0000 7fd9bb200640 1 --2- 172.21.15.19:0/2837558383 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd9bc104bf0 0x7fd9bc19a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:07.590 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.587+0000 7fd9bb200640 1 -- 172.21.15.19:0/2837558383 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd9bc103a00 msgr2=0x7fd9bc199c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:07.590 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.587+0000 7fd9bb200640 1 --2- 172.21.15.19:0/2837558383 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd9bc103a00 0x7fd9bc199c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:07.590 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.587+0000 7fd9bb200640 1 -- 172.21.15.19:0/2837558383 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9bc1a18b0 con 0x7fd9bc1055b0 2024-03-18T15:18:07.590 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.587+0000 7fd9b9e00640 1 --2- 172.21.15.19:0/2837558383 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd9bc104bf0 0x7fd9bc19a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:18:07.590 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.587+0000 7fd9ba800640 1 --2- 172.21.15.19:0/2837558383 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd9bc103a00 0x7fd9bc199c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:18:07.590 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.587+0000 7fd9bb200640 1 --2- 172.21.15.19:0/2837558383 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd9bc1055b0 0x7fd9bc1a1210 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fd9b4037040 tx=0x7fd9b4032ea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:18:07.590 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.588+0000 7fd9b3400640 1 -- 172.21.15.19:0/2837558383 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9b4041020 con 0x7fd9bc1055b0 2024-03-18T15:18:07.590 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.588+0000 7fd9b3400640 1 -- 172.21.15.19:0/2837558383 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd9b4034cd0 con 0x7fd9bc1055b0 2024-03-18T15:18:07.590 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.588+0000 7fd9b3400640 1 -- 172.21.15.19:0/2837558383 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9b4039040 con 0x7fd9bc1055b0 2024-03-18T15:18:07.590 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.588+0000 7fd9c1400640 1 -- 172.21.15.19:0/2837558383 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9bc1a1b40 con 0x7fd9bc1055b0 2024-03-18T15:18:07.590 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.588+0000 7fd9c1400640 1 -- 172.21.15.19:0/2837558383 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9bc1a2050 con 0x7fd9bc1055b0 2024-03-18T15:18:07.591 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.589+0000 7fd9c1400640 1 -- 172.21.15.19:0/2837558383 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9bc10ad20 con 0x7fd9bc1055b0 2024-03-18T15:18:07.591 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.591+0000 7fd9b3400640 1 -- 172.21.15.19:0/2837558383 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fd9b4044020 con 0x7fd9bc1055b0 2024-03-18T15:18:07.593 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.592+0000 7fd9b3400640 1 --2- 172.21.15.19:0/2837558383 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fd9a00a2880 0x7fd9a00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:07.593 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.592+0000 7fd9b3400640 1 -- 172.21.15.19:0/2837558383 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fd9b40f0b90 con 0x7fd9bc1055b0 2024-03-18T15:18:07.593 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.592+0000 7fd9ba800640 1 --2- 172.21.15.19:0/2837558383 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fd9a00a2880 0x7fd9a00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:07.594 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.593+0000 7fd9ba800640 1 --2- 172.21.15.19:0/2837558383 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fd9a00a2880 0x7fd9a00a4d40 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fd9bc19ab40 tx=0x7fd9ac005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:18:07.602 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.601+0000 7fd9b3400640 1 -- 172.21.15.19:0/2837558383 <== mon.1 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd9b404b020 con 0x7fd9bc1055b0 2024-03-18T15:18:07.855 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.854+0000 7fd9c1400640 1 -- 172.21.15.19:0/2837558383 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd9bc062480 con 0x7fd9bc1055b0 2024-03-18T15:18:07.856 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.855+0000 7fd9b3400640 1 -- 172.21.15.19:0/2837558383 <== mon.1 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11927 (secure 0 0 0) 0x7fd9b40bd890 con 0x7fd9bc1055b0 2024-03-18T15:18:07.856 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-18T15:18:07.856 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6835/1135329545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6834","nonce":1135329545},{"type":"v1","addr":"172.21.15.19:6835","nonce":1135329545}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14457,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6839/3833518557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6838","nonce":3833518557},{"type":"v1","addr":"172.21.15.19:6839","nonce":3833518557}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T15:15:10.589737+0000","modified":"2024-03-18T15:15:46.979999+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24560,"mds_1":24529,"mds_2":24511,"mds_3":24506,"mds_4":24499},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14475":{"gid":14475,"name":"a","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.19:6841/3176583319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6840","nonce":3176583319},{"type":"v1","addr":"172.21.15.19:6841","nonce":3176583319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24433":{"gid":24433,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6833/482725809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":482725809},{"type":"v1","addr":"172.21.15.104:6833","nonce":482725809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24434":{"gid":24434,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/2272279159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":2272279159},{"type":"v1","addr":"172.21.15.33:6833","nonce":2272279159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24469":{"gid":24469,"name":"d","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.19:6837/1778019859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6836","nonce":1778019859},{"type":"v1","addr":"172.21.15.19:6837","nonce":1778019859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6835/4051613151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":4051613151},{"type":"v1","addr":"172.21.15.33:6835","nonce":4051613151}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":17,"addr":"172.21.15.104:6835/3788595957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3788595957},{"type":"v1","addr":"172.21.15.104:6835","nonce":3788595957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":15,"addr":"172.21.15.33:6837/160187378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":160187378},{"type":"v1","addr":"172.21.15.33:6837","nonce":160187378}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24511":{"gid":24511,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":16,"addr":"172.21.15.104:6837/260984674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":260984674},{"type":"v1","addr":"172.21.15.104:6837","nonce":260984674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24529":{"gid":24529,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.104:6839/805341436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6838","nonce":805341436},{"type":"v1","addr":"172.21.15.104:6839","nonce":805341436}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.33:6839/1903591535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":1903591535},{"type":"v1","addr":"172.21.15.33:6839","nonce":1903591535}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24560,"qdb_cluster":[24506,24499,24529,24511,24560]},"id":1}]} 2024-03-18T15:18:07.856 INFO:teuthology.orchestra.run.smithi019.stderr:dumped fsmap epoch 26 2024-03-18T15:18:07.859 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.858+0000 7fd9c1400640 1 -- 172.21.15.19:0/2837558383 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fd9a00a2880 msgr2=0x7fd9a00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:07.859 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.858+0000 7fd9c1400640 1 --2- 172.21.15.19:0/2837558383 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fd9a00a2880 0x7fd9a00a4d40 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fd9bc19ab40 tx=0x7fd9ac005f90 comp rx=0 tx=0).stop 2024-03-18T15:18:07.859 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.858+0000 7fd9c1400640 1 -- 172.21.15.19:0/2837558383 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd9bc1055b0 msgr2=0x7fd9bc1a1210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:07.859 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.858+0000 7fd9c1400640 1 --2- 172.21.15.19:0/2837558383 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd9bc1055b0 0x7fd9bc1a1210 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fd9b4037040 tx=0x7fd9b4032ea0 comp rx=0 tx=0).stop 2024-03-18T15:18:07.859 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.858+0000 7fd9c1400640 1 -- 172.21.15.19:0/2837558383 shutdown_connections 2024-03-18T15:18:07.859 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.858+0000 7fd9c1400640 1 --2- 172.21.15.19:0/2837558383 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd9bc1055b0 0x7fd9bc1a1210 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:07.859 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.858+0000 7fd9c1400640 1 --2- 172.21.15.19:0/2837558383 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fd9a00a2880 0x7fd9a00a4d40 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:07.859 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.858+0000 7fd9c1400640 1 --2- 172.21.15.19:0/2837558383 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd9bc104bf0 0x7fd9bc19a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:07.860 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.858+0000 7fd9c1400640 1 --2- 172.21.15.19:0/2837558383 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd9bc103a00 0x7fd9bc199c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:07.860 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.858+0000 7fd9c1400640 1 -- 172.21.15.19:0/2837558383 >> 172.21.15.19:0/2837558383 conn(0x7fd9bc0fef90 msgr2=0x7fd9bc107ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:18:07.860 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.858+0000 7fd9c1400640 1 -- 172.21.15.19:0/2837558383 shutdown_connections 2024-03-18T15:18:07.860 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:07.858+0000 7fd9c1400640 1 -- 172.21.15.19:0/2837558383 wait complete. 2024-03-18T15:18:07.879 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T15:18:08.113 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.112+0000 7fb3a8e00640 1 -- 172.21.15.19:0/1997363343 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb3a4108e10 msgr2=0x7fb3a4109290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:08.113 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.112+0000 7fb3a8e00640 1 --2- 172.21.15.19:0/1997363343 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb3a4108e10 0x7fb3a4109290 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fb390009900 tx=0x7fb39002bc40 comp rx=0 tx=0).stop 2024-03-18T15:18:08.114 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.112+0000 7fb3a8e00640 1 -- 172.21.15.19:0/1997363343 shutdown_connections 2024-03-18T15:18:08.114 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.112+0000 7fb3a8e00640 1 --2- 172.21.15.19:0/1997363343 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb3a41097d0 0x7fb3a4110060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:08.114 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.112+0000 7fb3a8e00640 1 --2- 172.21.15.19:0/1997363343 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb3a4108e10 0x7fb3a4109290 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:08.114 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.112+0000 7fb3a8e00640 1 --2- 172.21.15.19:0/1997363343 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb3a4107c20 0x7fb3a4108040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:08.114 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.112+0000 7fb3a8e00640 1 -- 172.21.15.19:0/1997363343 >> 172.21.15.19:0/1997363343 conn(0x7fb3a4076bb0 msgr2=0x7fb3a4078ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:18:08.114 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.112+0000 7fb3a8e00640 1 -- 172.21.15.19:0/1997363343 shutdown_connections 2024-03-18T15:18:08.114 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.112+0000 7fb3a8e00640 1 -- 172.21.15.19:0/1997363343 wait complete. 2024-03-18T15:18:08.116 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.113+0000 7fb3a8e00640 1 Processor -- start 2024-03-18T15:18:08.116 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.113+0000 7fb3a8e00640 1 -- start start 2024-03-18T15:18:08.116 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.114+0000 7fb3a8e00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb3a4107c20 0x7fb3a419df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:08.117 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.114+0000 7fb3a8e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb3a4108e10 0x7fb3a419e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:08.117 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.114+0000 7fb3a8e00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb3a41097d0 0x7fb3a41a54b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:08.117 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.114+0000 7fb3a8e00640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fb3a41138f0 con 0x7fb3a4107c20 2024-03-18T15:18:08.117 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.114+0000 7fb3a8e00640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb3a4113770 con 0x7fb3a4108e10 2024-03-18T15:18:08.117 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.114+0000 7fb3a8e00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fb3a4113a70 con 0x7fb3a41097d0 2024-03-18T15:18:08.117 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.114+0000 7fb3a2a00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb3a4108e10 0x7fb3a419e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:08.117 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.114+0000 7fb3a3e00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb3a41097d0 0x7fb3a41a54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:08.117 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.114+0000 7fb3a2a00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb3a4108e10 0x7fb3a419e440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:46202/0 (socket says 172.21.15.19:46202) 2024-03-18T15:18:08.117 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.114+0000 7fb3a3400640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb3a4107c20 0x7fb3a419df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:08.117 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.114+0000 7fb3a3e00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb3a41097d0 0x7fb3a41a54b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.19:39290/0 (socket says 172.21.15.19:39290) 2024-03-18T15:18:08.117 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.114+0000 7fb3a2a00640 1 -- 172.21.15.19:0/1609730460 learned_addr learned my addr 172.21.15.19:0/1609730460 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:18:08.117 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.115+0000 7fb3a3e00640 1 -- 172.21.15.19:0/1609730460 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb3a4108e10 msgr2=0x7fb3a419e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:08.117 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.115+0000 7fb3a3e00640 1 --2- 172.21.15.19:0/1609730460 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb3a4108e10 0x7fb3a419e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:08.117 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.115+0000 7fb3a3e00640 1 -- 172.21.15.19:0/1609730460 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb3a4107c20 msgr2=0x7fb3a419df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:08.118 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.115+0000 7fb3a3e00640 1 --2- 172.21.15.19:0/1609730460 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb3a4107c20 0x7fb3a419df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:08.118 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.115+0000 7fb3a3e00640 1 -- 172.21.15.19:0/1609730460 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3a41a5b20 con 0x7fb3a41097d0 2024-03-18T15:18:08.118 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.115+0000 7fb3a3e00640 1 --2- 172.21.15.19:0/1609730460 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb3a41097d0 0x7fb3a41a54b0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fb39c009fd0 tx=0x7fb39c00ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:18:08.118 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.115+0000 7fb39be00640 1 -- 172.21.15.19:0/1609730460 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb39c00ce20 con 0x7fb3a41097d0 2024-03-18T15:18:08.118 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.115+0000 7fb39be00640 1 -- 172.21.15.19:0/1609730460 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb39c005020 con 0x7fb3a41097d0 2024-03-18T15:18:08.118 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.115+0000 7fb39be00640 1 -- 172.21.15.19:0/1609730460 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb39c010720 con 0x7fb3a41097d0 2024-03-18T15:18:08.118 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.115+0000 7fb3a8e00640 1 -- 172.21.15.19:0/1609730460 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3a41a5e10 con 0x7fb3a41097d0 2024-03-18T15:18:08.118 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.116+0000 7fb3a8e00640 1 -- 172.21.15.19:0/1609730460 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3a41a6220 con 0x7fb3a41097d0 2024-03-18T15:18:08.119 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.117+0000 7fb3a8e00640 1 -- 172.21.15.19:0/1609730460 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3a410eee0 con 0x7fb3a41097d0 2024-03-18T15:18:08.119 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.118+0000 7fb39be00640 1 -- 172.21.15.19:0/1609730460 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fb39c005540 con 0x7fb3a41097d0 2024-03-18T15:18:08.121 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.120+0000 7fb39be00640 1 --2- 172.21.15.19:0/1609730460 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fb3880a2800 0x7fb3880a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:08.121 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.120+0000 7fb39be00640 1 -- 172.21.15.19:0/1609730460 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fb39c0cf570 con 0x7fb3a41097d0 2024-03-18T15:18:08.121 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.120+0000 7fb3a3400640 1 --2- 172.21.15.19:0/1609730460 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fb3880a2800 0x7fb3880a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:08.121 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.121+0000 7fb3a3400640 1 --2- 172.21.15.19:0/1609730460 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fb3880a2800 0x7fb3880a4cc0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fb3a4108c70 tx=0x7fb394005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:18:08.124 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.124+0000 7fb39be00640 1 -- 172.21.15.19:0/1609730460 <== mon.1 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb39c09c1e0 con 0x7fb3a41097d0 2024-03-18T15:18:08.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:08 smithi019 ceph-mon[31522]: pgmap v409: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T15:18:08.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:08 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/730171778' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T15:18:08.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:08 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/2837558383' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:18:08.331 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.330+0000 7fb3a8e00640 1 -- 172.21.15.19:0/1609730460 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb3a40624e0 con 0x7fb3a41097d0 2024-03-18T15:18:08.333 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.332+0000 7fb39be00640 1 -- 172.21.15.19:0/1609730460 <== mon.1 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24809 (secure 0 0 0) 0x7fb39c09bae0 con 0x7fb3a41097d0 2024-03-18T15:18:08.334 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-18T15:18:08.334 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":93,"fsid":"d4dc2ba6-e538-11ee-95c9-87774f69a715","created":"2024-03-18T15:05:57.538551+0000","modified":"2024-03-18T15:17:16.033313+0000","last_up_change":"2024-03-18T15:14:12.009522+0000","last_in_change":"2024-03-18T15:13:52.555387+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T15:09:56.843349+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T15:15:08.941047+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T15:15:10.044671+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T15:15:13.621916+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"37b5a5cd-27ae-4e77-9b05-774b817b2ee9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6802","nonce":402333948},{"type":"v1","addr":"172.21.15.19:6803","nonce":402333948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6804","nonce":402333948},{"type":"v1","addr":"172.21.15.19:6805","nonce":402333948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6808","nonce":402333948},{"type":"v1","addr":"172.21.15.19:6809","nonce":402333948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6806","nonce":402333948},{"type":"v1","addr":"172.21.15.19:6807","nonce":402333948}]},"public_addr":"172.21.15.19:6803/402333948","cluster_addr":"172.21.15.19:6805/402333948","heartbeat_back_addr":"172.21.15.19:6809/402333948","heartbeat_front_addr":"172.21.15.19:6807/402333948","state":["exists","up"]},{"osd":1,"uuid":"3a1134e8-ede7-4dbd-9be1-eefd2cbb43bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1951880113},{"type":"v1","addr":"172.21.15.33:6801","nonce":1951880113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1951880113},{"type":"v1","addr":"172.21.15.33:6803","nonce":1951880113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1951880113},{"type":"v1","addr":"172.21.15.33:6807","nonce":1951880113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1951880113},{"type":"v1","addr":"172.21.15.33:6805","nonce":1951880113}]},"public_addr":"172.21.15.33:6801/1951880113","cluster_addr":"172.21.15.33:6803/1951880113","heartbeat_back_addr":"172.21.15.33:6807/1951880113","heartbeat_front_addr":"172.21.15.33:6805/1951880113","state":["exists","up"]},{"osd":2,"uuid":"f383a164-a3a0-49fd-a9e8-9c2178d8cfb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":1004727949},{"type":"v1","addr":"172.21.15.104:6801","nonce":1004727949}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":1004727949},{"type":"v1","addr":"172.21.15.104:6803","nonce":1004727949}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":1004727949},{"type":"v1","addr":"172.21.15.104:6807","nonce":1004727949}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":1004727949},{"type":"v1","addr":"172.21.15.104:6805","nonce":1004727949}]},"public_addr":"172.21.15.104:6801/1004727949","cluster_addr":"172.21.15.104:6803/1004727949","heartbeat_back_addr":"172.21.15.104:6807/1004727949","heartbeat_front_addr":"172.21.15.104:6805/1004727949","state":["exists","up"]},{"osd":3,"uuid":"c409e116-28bf-45dd-a791-539af22abe1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6810","nonce":3571525854},{"type":"v1","addr":"172.21.15.19:6811","nonce":3571525854}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6812","nonce":3571525854},{"type":"v1","addr":"172.21.15.19:6813","nonce":3571525854}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6816","nonce":3571525854},{"type":"v1","addr":"172.21.15.19:6817","nonce":3571525854}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6814","nonce":3571525854},{"type":"v1","addr":"172.21.15.19:6815","nonce":3571525854}]},"public_addr":"172.21.15.19:6811/3571525854","cluster_addr":"172.21.15.19:6813/3571525854","heartbeat_back_addr":"172.21.15.19:6817/3571525854","heartbeat_front_addr":"172.21.15.19:6815/3571525854","state":["exists","up"]},{"osd":4,"uuid":"696eb719-0b3e-43ae-b5f1-ccd73a3b2843","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":624901749},{"type":"v1","addr":"172.21.15.33:6809","nonce":624901749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":624901749},{"type":"v1","addr":"172.21.15.33:6811","nonce":624901749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":624901749},{"type":"v1","addr":"172.21.15.33:6815","nonce":624901749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":624901749},{"type":"v1","addr":"172.21.15.33:6813","nonce":624901749}]},"public_addr":"172.21.15.33:6809/624901749","cluster_addr":"172.21.15.33:6811/624901749","heartbeat_back_addr":"172.21.15.33:6815/624901749","heartbeat_front_addr":"172.21.15.33:6813/624901749","state":["exists","up"]},{"osd":5,"uuid":"21789779-8bb9-42c7-b9ee-437985f11e3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":1258208297},{"type":"v1","addr":"172.21.15.104:6809","nonce":1258208297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":1258208297},{"type":"v1","addr":"172.21.15.104:6811","nonce":1258208297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":1258208297},{"type":"v1","addr":"172.21.15.104:6815","nonce":1258208297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":1258208297},{"type":"v1","addr":"172.21.15.104:6813","nonce":1258208297}]},"public_addr":"172.21.15.104:6809/1258208297","cluster_addr":"172.21.15.104:6811/1258208297","heartbeat_back_addr":"172.21.15.104:6815/1258208297","heartbeat_front_addr":"172.21.15.104:6813/1258208297","state":["exists","up"]},{"osd":6,"uuid":"0e92a2ef-43cd-487c-9203-e219158aef8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6818","nonce":2782482301},{"type":"v1","addr":"172.21.15.19:6819","nonce":2782482301}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6820","nonce":2782482301},{"type":"v1","addr":"172.21.15.19:6821","nonce":2782482301}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6824","nonce":2782482301},{"type":"v1","addr":"172.21.15.19:6825","nonce":2782482301}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6822","nonce":2782482301},{"type":"v1","addr":"172.21.15.19:6823","nonce":2782482301}]},"public_addr":"172.21.15.19:6819/2782482301","cluster_addr":"172.21.15.19:6821/2782482301","heartbeat_back_addr":"172.21.15.19:6825/2782482301","heartbeat_front_addr":"172.21.15.19:6823/2782482301","state":["exists","up"]},{"osd":7,"uuid":"a16e39a1-3685-491d-b376-3e80a2084072","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":4238160988},{"type":"v1","addr":"172.21.15.33:6817","nonce":4238160988}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":4238160988},{"type":"v1","addr":"172.21.15.33:6819","nonce":4238160988}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":4238160988},{"type":"v1","addr":"172.21.15.33:6823","nonce":4238160988}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":4238160988},{"type":"v1","addr":"172.21.15.33:6821","nonce":4238160988}]},"public_addr":"172.21.15.33:6817/4238160988","cluster_addr":"172.21.15.33:6819/4238160988","heartbeat_back_addr":"172.21.15.33:6823/4238160988","heartbeat_front_addr":"172.21.15.33:6821/4238160988","state":["exists","up"]},{"osd":8,"uuid":"d5c3fa92-cd5d-4349-9c7d-d5666bacde16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":1014779813},{"type":"v1","addr":"172.21.15.104:6817","nonce":1014779813}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":1014779813},{"type":"v1","addr":"172.21.15.104:6819","nonce":1014779813}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":1014779813},{"type":"v1","addr":"172.21.15.104:6823","nonce":1014779813}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":1014779813},{"type":"v1","addr":"172.21.15.104:6821","nonce":1014779813}]},"public_addr":"172.21.15.104:6817/1014779813","cluster_addr":"172.21.15.104:6819/1014779813","heartbeat_back_addr":"172.21.15.104:6823/1014779813","heartbeat_front_addr":"172.21.15.104:6821/1014779813","state":["exists","up"]},{"osd":9,"uuid":"eed2e9b2-26d3-4c46-88ec-34b5596fe991","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6826","nonce":2641151009},{"type":"v1","addr":"172.21.15.19:6827","nonce":2641151009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6828","nonce":2641151009},{"type":"v1","addr":"172.21.15.19:6829","nonce":2641151009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6832","nonce":2641151009},{"type":"v1","addr":"172.21.15.19:6833","nonce":2641151009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6830","nonce":2641151009},{"type":"v1","addr":"172.21.15.19:6831","nonce":2641151009}]},"public_addr":"172.21.15.19:6827/2641151009","cluster_addr":"172.21.15.19:6829/2641151009","heartbeat_back_addr":"172.21.15.19:6833/2641151009","heartbeat_front_addr":"172.21.15.19:6831/2641151009","state":["exists","up"]},{"osd":10,"uuid":"f3c35ea5-fbe7-4dc4-93c5-b92d1a714db3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":1346058869},{"type":"v1","addr":"172.21.15.33:6825","nonce":1346058869}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":1346058869},{"type":"v1","addr":"172.21.15.33:6827","nonce":1346058869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":1346058869},{"type":"v1","addr":"172.21.15.33:6831","nonce":1346058869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":1346058869},{"type":"v1","addr":"172.21.15.33:6829","nonce":1346058869}]},"public_addr":"172.21.15.33:6825/1346058869","cluster_addr":"172.21.15.33:6827/1346058869","heartbeat_back_addr":"172.21.15.33:6831/1346058869","heartbeat_front_addr":"172.21.15.33:6829/1346058869","state":["exists","up"]},{"osd":11,"uuid":"c8b51571-f807-4b62-90b6-7529e09e59da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":3759498281},{"type":"v1","addr":"172.21.15.104:6825","nonce":3759498281}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":3759498281},{"type":"v1","addr":"172.21.15.104:6827","nonce":3759498281}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":3759498281},{"type":"v1","addr":"172.21.15.104:6831","nonce":3759498281}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":3759498281},{"type":"v1","addr":"172.21.15.104:6829","nonce":3759498281}]},"public_addr":"172.21.15.104:6825/3759498281","cluster_addr":"172.21.15.104:6827/3759498281","heartbeat_back_addr":"172.21.15.104:6831/3759498281","heartbeat_front_addr":"172.21.15.104:6829/3759498281","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:09:05.267795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:09:30.234068+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:09:53.870232+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:10:26.471296+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:10:53.180341+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:11:17.801361+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:11:51.364816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:12:18.306408+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:12:43.978744+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:13:18.213964+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:13:45.437544+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:14:10.148580+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":3,"to":6}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"3.37","mappings":[{"from":5,"to":8}]},{"pgid":"3.3a","mappings":[{"from":4,"to":7}]},{"pgid":"4.8","mappings":[{"from":5,"to":8}]},{"pgid":"4.f","mappings":[{"from":9,"to":10}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.19:0/3860783274":"2024-03-19T15:07:14.150983+0000","172.21.15.19:0/2737223213":"2024-03-19T15:07:14.150983+0000","172.21.15.19:0/176699025":"2024-03-19T15:06:38.688405+0000","172.21.15.19:0/2316969005":"2024-03-19T15:07:14.150983+0000","172.21.15.19:6801/2776008962":"2024-03-19T15:07:14.150983+0000","172.21.15.19:0/2793980440":"2024-03-19T15:06:38.688405+0000","172.21.15.19:6800/3264337803":"2024-03-19T15:06:38.688405+0000","172.21.15.19:6801/3264337803":"2024-03-19T15:06:38.688405+0000","172.21.15.19:0/3166705295":"2024-03-19T15:06:38.688405+0000","172.21.15.19:6800/2776008962":"2024-03-19T15:07:14.150983+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T15:18:08.336 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.335+0000 7fb3a8e00640 1 -- 172.21.15.19:0/1609730460 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fb3880a2800 msgr2=0x7fb3880a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:08.336 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.335+0000 7fb3a8e00640 1 --2- 172.21.15.19:0/1609730460 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fb3880a2800 0x7fb3880a4cc0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fb3a4108c70 tx=0x7fb394005f90 comp rx=0 tx=0).stop 2024-03-18T15:18:08.336 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.335+0000 7fb3a8e00640 1 -- 172.21.15.19:0/1609730460 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb3a41097d0 msgr2=0x7fb3a41a54b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:08.336 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.335+0000 7fb3a8e00640 1 --2- 172.21.15.19:0/1609730460 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb3a41097d0 0x7fb3a41a54b0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fb39c009fd0 tx=0x7fb39c00ef10 comp rx=0 tx=0).stop 2024-03-18T15:18:08.337 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.335+0000 7fb3a8e00640 1 -- 172.21.15.19:0/1609730460 shutdown_connections 2024-03-18T15:18:08.337 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.335+0000 7fb3a8e00640 1 --2- 172.21.15.19:0/1609730460 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb3a41097d0 0x7fb3a41a54b0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:08.337 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.335+0000 7fb3a8e00640 1 --2- 172.21.15.19:0/1609730460 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fb3880a2800 0x7fb3880a4cc0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:08.337 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.335+0000 7fb3a8e00640 1 --2- 172.21.15.19:0/1609730460 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb3a4108e10 0x7fb3a419e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:08.337 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.335+0000 7fb3a8e00640 1 --2- 172.21.15.19:0/1609730460 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb3a4107c20 0x7fb3a419df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:08.337 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.335+0000 7fb3a8e00640 1 -- 172.21.15.19:0/1609730460 >> 172.21.15.19:0/1609730460 conn(0x7fb3a4076bb0 msgr2=0x7fb3a410c0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:18:08.337 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.336+0000 7fb3a8e00640 1 -- 172.21.15.19:0/1609730460 shutdown_connections 2024-03-18T15:18:08.337 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.336+0000 7fb3a8e00640 1 -- 172.21.15.19:0/1609730460 wait complete. 2024-03-18T15:18:08.358 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-18T15:15:10.589737+0000', 'modified': '2024-03-18T15:15:46.979999+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24560, 'mds_1': 24529, 'mds_2': 24511, 'mds_3': 24506, 'mds_4': 24499}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14475': {'gid': 14475, 'name': 'a', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.19:6841/3176583319', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.19:6840', 'nonce': 3176583319}, {'type': 'v1', 'addr': '172.21.15.19:6841', 'nonce': 3176583319}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24433': {'gid': 24433, 'name': 'l', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.104:6833/482725809', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.104:6832', 'nonce': 482725809}, {'type': 'v1', 'addr': '172.21.15.104:6833', 'nonce': 482725809}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24434': {'gid': 24434, 'name': 'k', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.33:6833/2272279159', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.33:6832', 'nonce': 2272279159}, {'type': 'v1', 'addr': '172.21.15.33:6833', 'nonce': 2272279159}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24469': {'gid': 24469, 'name': 'd', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.19:6837/1778019859', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.19:6836', 'nonce': 1778019859}, {'type': 'v1', 'addr': '172.21.15.19:6837', 'nonce': 1778019859}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24482': {'gid': 24482, 'name': 'e', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.33:6835/4051613151', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.33:6834', 'nonce': 4051613151}, {'type': 'v1', 'addr': '172.21.15.33:6835', 'nonce': 4051613151}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24499': {'gid': 24499, 'name': 'f', 'rank': 4, 'incarnation': 25, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.104:6835/3788595957', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.104:6834', 'nonce': 3788595957}, {'type': 'v1', 'addr': '172.21.15.104:6835', 'nonce': 3788595957}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24506': {'gid': 24506, 'name': 'h', 'rank': 3, 'incarnation': 23, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.33:6837/160187378', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.33:6836', 'nonce': 160187378}, {'type': 'v1', 'addr': '172.21.15.33:6837', 'nonce': 160187378}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24511': {'gid': 24511, 'name': 'c', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.104:6837/260984674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.104:6836', 'nonce': 260984674}, {'type': 'v1', 'addr': '172.21.15.104:6837', 'nonce': 260984674}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24529': {'gid': 24529, 'name': 'i', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.104:6839/805341436', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.104:6838', 'nonce': 805341436}, {'type': 'v1', 'addr': '172.21.15.104:6839', 'nonce': 805341436}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24560': {'gid': 24560, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.33:6839/1903591535', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.33:6838', 'nonce': 1903591535}, {'type': 'v1', 'addr': '172.21.15.33:6839', 'nonce': 1903591535}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24560, 'qdb_cluster': [24506, 24499, 24529, 24511, 24560]} 2024-03-18T15:18:08.358 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-03-18T15:18:08.358 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.f status 2024-03-18T15:18:08.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:08 smithi104 ceph-mon[36070]: pgmap v409: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T15:18:08.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:08 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/730171778' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T15:18:08.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:08 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/2837558383' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:18:08.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:08 smithi033 ceph-mon[36648]: pgmap v409: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T15:18:08.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:08 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/730171778' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T15:18:08.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:08 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/2837558383' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:18:08.576 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.575+0000 7fae2c800640 1 -- 172.21.15.19:0/3274886811 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae24104bf0 msgr2=0x7fae24105070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:08.577 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.575+0000 7fae2c800640 1 --2- 172.21.15.19:0/3274886811 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae24104bf0 0x7fae24105070 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fae14009900 tx=0x7fae1402bc50 comp rx=0 tx=0).stop 2024-03-18T15:18:08.577 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.575+0000 7fae2c800640 1 -- 172.21.15.19:0/3274886811 shutdown_connections 2024-03-18T15:18:08.577 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.575+0000 7fae2c800640 1 --2- 172.21.15.19:0/3274886811 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fae241055b0 0x7fae2410be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:08.577 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.575+0000 7fae2c800640 1 --2- 172.21.15.19:0/3274886811 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae24104bf0 0x7fae24105070 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:08.577 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.575+0000 7fae2c800640 1 --2- 172.21.15.19:0/3274886811 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fae24103a00 0x7fae24103e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:08.577 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.575+0000 7fae2c800640 1 -- 172.21.15.19:0/3274886811 >> 172.21.15.19:0/3274886811 conn(0x7fae240fef90 msgr2=0x7fae241013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:18:08.577 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.575+0000 7fae2c800640 1 -- 172.21.15.19:0/3274886811 shutdown_connections 2024-03-18T15:18:08.577 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.575+0000 7fae2c800640 1 -- 172.21.15.19:0/3274886811 wait complete. 2024-03-18T15:18:08.580 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.577+0000 7fae2c800640 1 Processor -- start 2024-03-18T15:18:08.580 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.577+0000 7fae2c800640 1 -- start start 2024-03-18T15:18:08.580 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.577+0000 7fae2c800640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fae24103a00 0x7fae24199d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:08.580 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.577+0000 7fae2c800640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae24104bf0 0x7fae2419a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:08.580 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.577+0000 7fae2c800640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fae241055b0 0x7fae241a1280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:08.580 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.577+0000 7fae2c800640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fae2410f6b0 con 0x7fae241055b0 2024-03-18T15:18:08.580 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.577+0000 7fae2c800640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fae2410f530 con 0x7fae24104bf0 2024-03-18T15:18:08.580 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.577+0000 7fae2c800640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fae2410f830 con 0x7fae24103a00 2024-03-18T15:18:08.581 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.577+0000 7fae29e00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fae24103a00 0x7fae24199d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:08.581 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.577+0000 7fae29e00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fae24103a00 0x7fae24199d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.19:39308/0 (socket says 172.21.15.19:39308) 2024-03-18T15:18:08.581 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.577+0000 7fae29e00640 1 -- 172.21.15.19:0/2051917613 learned_addr learned my addr 172.21.15.19:0/2051917613 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:18:08.581 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.577+0000 7fae29400640 1 --2- 172.21.15.19:0/2051917613 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae24104bf0 0x7fae2419a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:08.581 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.578+0000 7fae2a800640 1 --2- 172.21.15.19:0/2051917613 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fae241055b0 0x7fae241a1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:08.581 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.578+0000 7fae29e00640 1 -- 172.21.15.19:0/2051917613 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae24104bf0 msgr2=0x7fae2419a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:08.581 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.578+0000 7fae29e00640 1 --2- 172.21.15.19:0/2051917613 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae24104bf0 0x7fae2419a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:08.581 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.578+0000 7fae29e00640 1 -- 172.21.15.19:0/2051917613 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fae241055b0 msgr2=0x7fae241a1280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:08.581 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.578+0000 7fae29e00640 1 --2- 172.21.15.19:0/2051917613 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fae241055b0 0x7fae241a1280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:08.581 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.578+0000 7fae29e00640 1 -- 172.21.15.19:0/2051917613 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae241a1920 con 0x7fae24103a00 2024-03-18T15:18:08.582 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.578+0000 7fae2a800640 1 --2- 172.21.15.19:0/2051917613 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fae241055b0 0x7fae241a1280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T15:18:08.582 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.578+0000 7fae29e00640 1 --2- 172.21.15.19:0/2051917613 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fae24103a00 0x7fae24199d10 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fae1800e9c0 tx=0x7fae1800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:18:08.582 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.578+0000 7fae1ea00640 1 -- 172.21.15.19:0/2051917613 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae1800cd80 con 0x7fae24103a00 2024-03-18T15:18:08.582 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.578+0000 7fae1ea00640 1 -- 172.21.15.19:0/2051917613 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae180035f0 con 0x7fae24103a00 2024-03-18T15:18:08.582 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.578+0000 7fae1ea00640 1 -- 172.21.15.19:0/2051917613 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae18010690 con 0x7fae24103a00 2024-03-18T15:18:08.582 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.578+0000 7fae2c800640 1 -- 172.21.15.19:0/2051917613 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae241a1c10 con 0x7fae24103a00 2024-03-18T15:18:08.582 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.579+0000 7fae2c800640 1 -- 172.21.15.19:0/2051917613 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae241a2110 con 0x7fae24103a00 2024-03-18T15:18:08.586 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.582+0000 7fae2f5e5740 1 Processor -- start 2024-03-18T15:18:08.586 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.582+0000 7fae1ea00640 1 -- 172.21.15.19:0/2051917613 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fae180037a0 con 0x7fae24103a00 2024-03-18T15:18:08.586 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.582+0000 7fae2f5e5740 1 -- start start 2024-03-18T15:18:08.586 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.582+0000 7fae2f5e5740 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x562b8dc41160 0x562b8dc41580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:08.586 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.582+0000 7fae2f5e5740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562b8dc41b50 0x562b8dc46650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:08.586 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.582+0000 7fae2f5e5740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562b8dc46cc0 0x562b8dc49150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:08.587 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.582+0000 7fae2f5e5740 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x562b8dba5c50 con 0x562b8dc41160 2024-03-18T15:18:08.587 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.582+0000 7fae2f5e5740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x562b8db4e9d0 con 0x562b8dc46cc0 2024-03-18T15:18:08.587 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.582+0000 7fae2f5e5740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x562b8dba1cf0 con 0x562b8dc41b50 2024-03-18T15:18:08.587 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.583+0000 7fae29e00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562b8dc41b50 0x562b8dc46650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:08.587 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.583+0000 7fae29e00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562b8dc41b50 0x562b8dc46650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.19:39324/0 (socket says 172.21.15.19:39324) 2024-03-18T15:18:08.587 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.583+0000 7fae29400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562b8dc46cc0 0x562b8dc49150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:08.587 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.583+0000 7fae29e00640 1 -- 172.21.15.19:0/1252991381 learned_addr learned my addr 172.21.15.19:0/1252991381 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:18:08.587 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.583+0000 7fae2a800640 1 --2- 172.21.15.19:0/1252991381 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x562b8dc41160 0x562b8dc41580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:08.587 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.583+0000 7fae29400640 1 -- 172.21.15.19:0/1252991381 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562b8dc41b50 msgr2=0x562b8dc46650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:18:08.587 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.583+0000 7fae29400640 1 --2- 172.21.15.19:0/1252991381 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562b8dc41b50 0x562b8dc46650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:08.588 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.583+0000 7fae29400640 1 -- 172.21.15.19:0/1252991381 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x562b8dc41160 msgr2=0x562b8dc41580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:18:08.588 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.583+0000 7fae29400640 1 --2- 172.21.15.19:0/1252991381 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x562b8dc41160 0x562b8dc41580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:08.588 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.583+0000 7fae29400640 1 -- 172.21.15.19:0/1252991381 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562b8db68f10 con 0x562b8dc46cc0 2024-03-18T15:18:08.588 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.583+0000 7fae1ea00640 1 --2- 172.21.15.19:0/2051917613 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fae0c0a2880 0x7fae0c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:08.588 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.583+0000 7fae29400640 1 --2- 172.21.15.19:0/1252991381 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562b8dc46cc0 0x562b8dc49150 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fae14036040 tx=0x7fae14009560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=489e7635944cdd0d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:18:08.588 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.584+0000 7fae1ea00640 1 -- 172.21.15.19:0/2051917613 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fae180cf2e0 con 0x7fae24103a00 2024-03-18T15:18:08.588 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.584+0000 7fae2a800640 1 --2- 172.21.15.19:0/2051917613 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fae0c0a2880 0x7fae0c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:08.588 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.584+0000 7fae13000640 1 -- 172.21.15.19:0/1252991381 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae14042070 con 0x562b8dc46cc0 2024-03-18T15:18:08.588 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.584+0000 7fae13000640 1 -- 172.21.15.19:0/1252991381 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae1402cba0 con 0x562b8dc46cc0 2024-03-18T15:18:08.588 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.584+0000 7fae13000640 1 -- 172.21.15.19:0/1252991381 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae14005130 con 0x562b8dc46cc0 2024-03-18T15:18:08.588 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.584+0000 7fae2f5e5740 1 -- 172.21.15.19:0/1252991381 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562b8dc46cc0 msgr2=0x562b8dc49150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:08.589 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.584+0000 7fae2f5e5740 1 --2- 172.21.15.19:0/1252991381 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562b8dc46cc0 0x562b8dc49150 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fae14036040 tx=0x7fae14009560 comp rx=0 tx=0).stop 2024-03-18T15:18:08.589 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.584+0000 7fae2a800640 1 --2- 172.21.15.19:0/2051917613 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fae0c0a2880 0x7fae0c0a4d40 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x562b8dc423b0 tx=0x7fae20009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:18:08.589 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.585+0000 7fae2f5e5740 1 -- 172.21.15.19:0/1252991381 shutdown_connections 2024-03-18T15:18:08.589 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.585+0000 7fae2f5e5740 1 --2- 172.21.15.19:0/1252991381 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562b8dc46cc0 0x562b8dc49150 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:08.589 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.585+0000 7fae2f5e5740 1 --2- 172.21.15.19:0/1252991381 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562b8dc41b50 0x562b8dc46650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:08.589 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.585+0000 7fae2f5e5740 1 --2- 172.21.15.19:0/1252991381 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x562b8dc41160 0x562b8dc41580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:08.589 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.585+0000 7fae2f5e5740 1 -- 172.21.15.19:0/1252991381 >> 172.21.15.19:0/1252991381 conn(0x562b8dc3c6d0 msgr2=0x562b8dc3cad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:18:08.589 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.585+0000 7fae2f5e5740 1 -- 172.21.15.19:0/1252991381 shutdown_connections 2024-03-18T15:18:08.589 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.585+0000 7fae2f5e5740 1 -- 172.21.15.19:0/1252991381 wait complete. 2024-03-18T15:18:08.589 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.585+0000 7fae2f5e5740 1 -- start start 2024-03-18T15:18:08.589 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.585+0000 7fae2f5e5740 1 Processor -- start 2024-03-18T15:18:08.595 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.586+0000 7fae2f5e5740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562b8dc41160 0x562b8dc4b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:08.595 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.586+0000 7fae2f5e5740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562b8dc41b50 0x562b8dc4b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:08.595 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.586+0000 7fae2f5e5740 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x562b8dc46cc0 0x562b8dc527c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:08.595 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.586+0000 7fae2f5e5740 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x562b8dc49c10 con 0x562b8dc46cc0 2024-03-18T15:18:08.595 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.586+0000 7fae2f5e5740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x562b8db61e20 con 0x562b8dc41160 2024-03-18T15:18:08.595 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.586+0000 7fae2f5e5740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x562b8dc49d90 con 0x562b8dc41b50 2024-03-18T15:18:08.595 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.586+0000 7fae2f5e5740 10 client.0 _ll_register_callbacks cb 0x562b8db617f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T15:18:08.596 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.586+0000 7fae29e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562b8dc41160 0x562b8dc4b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:08.596 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.586+0000 7fae29400640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562b8dc41b50 0x562b8dc4b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:08.596 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.586+0000 7fae29e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562b8dc41160 0x562b8dc4b2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:46242/0 (socket says 172.21.15.19:46242) 2024-03-18T15:18:08.596 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.586+0000 7fae29e00640 1 -- 172.21.15.19:0/3515203082 learned_addr learned my addr 172.21.15.19:0/3515203082 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:18:08.596 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.586+0000 7fae2a800640 1 --2- 172.21.15.19:0/3515203082 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x562b8dc46cc0 0x562b8dc527c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:08.596 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.586+0000 7fae29e00640 1 -- 172.21.15.19:0/3515203082 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562b8dc41b50 msgr2=0x562b8dc4b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:18:08.596 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.586+0000 7fae29e00640 1 --2- 172.21.15.19:0/3515203082 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562b8dc41b50 0x562b8dc4b7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:08.596 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.586+0000 7fae29e00640 1 -- 172.21.15.19:0/3515203082 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x562b8dc46cc0 msgr2=0x562b8dc527c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:18:08.597 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.586+0000 7fae29e00640 1 --2- 172.21.15.19:0/3515203082 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x562b8dc46cc0 0x562b8dc527c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:08.597 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.586+0000 7fae29e00640 1 -- 172.21.15.19:0/3515203082 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562b8d8ba2d0 con 0x562b8dc41160 2024-03-18T15:18:08.597 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.587+0000 7fae29e00640 1 --2- 172.21.15.19:0/3515203082 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562b8dc41160 0x562b8dc4b2b0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fae180a3ac0 tx=0x7fae18005b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1760286df67c536d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:18:08.597 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.587+0000 7fae07400640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-18T15:18:08.597 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.587+0000 7fae07400640 1 -- 172.21.15.19:0/3515203082 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae1809b390 con 0x562b8dc41160 2024-03-18T15:18:08.597 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.587+0000 7fae07400640 1 -- 172.21.15.19:0/3515203082 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fae1809aca0 con 0x562b8dc41160 2024-03-18T15:18:08.597 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.587+0000 7fae2f5e5740 1 -- 172.21.15.19:0/3515203082 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fae1402cba0 con 0x562b8dc41160 2024-03-18T15:18:08.597 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.587+0000 7fae07400640 1 -- 172.21.15.19:0/3515203082 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fae18099990 con 0x562b8dc41160 2024-03-18T15:18:08.597 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.587+0000 7fae2f5e5740 10 client.25346 fetch_fsmap learned FSMap version 26 2024-03-18T15:18:08.598 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.587+0000 7fae2f5e5740 1 -- 172.21.15.19:0/3515203082 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x562b8d9846b0 con 0x562b8dc41160 2024-03-18T15:18:08.598 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.587+0000 7fae07400640 1 -- 172.21.15.19:0/3515203082 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae1402cba0 con 0x562b8dc41160 2024-03-18T15:18:08.598 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.588+0000 7fae07400640 1 -- 172.21.15.19:0/3515203082 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 26) ==== 8362+0+0 (secure 0 0 0) 0x7fae1809cb80 con 0x562b8dc41160 2024-03-18T15:18:08.598 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.588+0000 7fae2f5e5740 10 client.25346 fetch_fsmap finished waiting for FSMap version 26 2024-03-18T15:18:08.598 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.588+0000 7fae2f5e5740 10 client.25346 resolve_mds: resolved name 'f' to daemon mds.f 2024-03-18T15:18:08.598 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.588+0000 7fae2f5e5740 20 client.25346 populate_metadata read hostname 'smithi019' 2024-03-18T15:18:08.598 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.588+0000 7fae2f5e5740 1 --2- 172.21.15.19:0/3515203082 >> [v2:172.21.15.104:6834/3788595957,v1:172.21.15.104:6835/3788595957] conn(0x562b8dc58090 0x562b8dc59500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:08.598 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.588+0000 7fae2f5e5740 4 client.25346 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T15:18:08.598 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.588+0000 7fae2f5e5740 1 -- 172.21.15.19:0/3515203082 --> [v2:172.21.15.104:6834/3788595957,v1:172.21.15.104:6835/3788595957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562b8dc032b0 con 0x562b8dc58090 2024-03-18T15:18:08.599 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.588+0000 7fae29400640 1 --2- 172.21.15.19:0/3515203082 >> [v2:172.21.15.104:6834/3788595957,v1:172.21.15.104:6835/3788595957] conn(0x562b8dc58090 0x562b8dc59500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:08.599 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.588+0000 7fae29400640 1 --2- 172.21.15.19:0/3515203082 >> [v2:172.21.15.104:6834/3788595957,v1:172.21.15.104:6835/3788595957] conn(0x562b8dc58090 0x562b8dc59500 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=d3ab2bbe46e08a2d server_cookie=6f421be8fd6606b8 in_seq=0 out_seq=0 2024-03-18T15:18:08.599 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.588+0000 7fae07400640 10 client.25346 ms_handle_connect on v2:172.21.15.104:6834/3788595957 2024-03-18T15:18:08.599 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.589+0000 7fae07400640 1 -- 172.21.15.19:0/3515203082 <== mds.4 v2:172.21.15.104:6834/3788595957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x562b8dc032b0 con 0x562b8dc58090 2024-03-18T15:18:08.599 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.589+0000 7fae07400640 10 client.25346 handle_command_reply: tid=0 2024-03-18T15:18:08.599 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.590+0000 7fae2f5e5740 1 client.25346 shutdown 2024-03-18T15:18:08.599 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.590+0000 7fae2f5e5740 1 -- 172.21.15.19:0/3515203082 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562b8dc41160 msgr2=0x562b8dc4b2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:08.599 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.590+0000 7fae2f5e5740 1 --2- 172.21.15.19:0/3515203082 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562b8dc41160 0x562b8dc4b2b0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fae180a3ac0 tx=0x7fae18005b20 comp rx=0 tx=0).stop 2024-03-18T15:18:08.600 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.590+0000 7fae2f5e5740 1 -- 172.21.15.19:0/3515203082 shutdown_connections 2024-03-18T15:18:08.600 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.590+0000 7fae2f5e5740 1 --2- 172.21.15.19:0/3515203082 >> [v2:172.21.15.104:6834/3788595957,v1:172.21.15.104:6835/3788595957] conn(0x562b8dc58090 0x562b8dc59500 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:08.600 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.590+0000 7fae2f5e5740 1 --2- 172.21.15.19:0/3515203082 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x562b8dc46cc0 0x562b8dc527c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:08.600 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.590+0000 7fae07400640 0 client.25346 ms_handle_reset on v2:172.21.15.104:6834/3788595957 2024-03-18T15:18:08.600 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.590+0000 7fae2f5e5740 1 --2- 172.21.15.19:0/3515203082 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562b8dc41b50 0x562b8dc4b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:08.600 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.590+0000 7fae2f5e5740 1 --2- 172.21.15.19:0/3515203082 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562b8dc41160 0x562b8dc4b2b0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:08.600 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.590+0000 7fae2f5e5740 1 -- 172.21.15.19:0/3515203082 >> 172.21.15.19:0/3515203082 conn(0x562b8dc3c6d0 msgr2=0x562b8dc3ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:18:08.600 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.590+0000 7fae2f5e5740 1 -- 172.21.15.19:0/3515203082 shutdown_connections 2024-03-18T15:18:08.601 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.590+0000 7fae2f5e5740 1 -- 172.21.15.19:0/3515203082 wait complete. 2024-03-18T15:18:08.601 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.590+0000 7fae2f5e5740 20 client.25346 trim_cache size 0 max 16384 2024-03-18T15:18:08.612 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.610+0000 7fae2f5e5740 1 Processor -- start 2024-03-18T15:18:08.612 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.610+0000 7fae2f5e5740 1 -- start start 2024-03-18T15:18:08.612 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.610+0000 7fae2f5e5740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562b8dc41160 0x562b8dc4aec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:08.613 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.610+0000 7fae2f5e5740 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x562b8dc41b50 0x562b8dc52780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:08.613 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.610+0000 7fae2f5e5740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562b8dc46cc0 0x562b8dc4b400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:08.613 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.611+0000 7fae2f5e5740 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x562b8dc57840 con 0x562b8dc41b50 2024-03-18T15:18:08.613 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.611+0000 7fae2f5e5740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x562b8dc57f10 con 0x562b8dc41160 2024-03-18T15:18:08.613 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.611+0000 7fae2f5e5740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x562b8dc585e0 con 0x562b8dc46cc0 2024-03-18T15:18:08.613 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.611+0000 7fae2a800640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562b8dc46cc0 0x562b8dc4b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:08.613 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.611+0000 7fae29400640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x562b8dc41b50 0x562b8dc52780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:08.613 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.611+0000 7fae2a800640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562b8dc46cc0 0x562b8dc4b400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.19:39342/0 (socket says 172.21.15.19:39342) 2024-03-18T15:18:08.613 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.611+0000 7fae2a800640 1 -- 172.21.15.19:0/1765128207 learned_addr learned my addr 172.21.15.19:0/1765128207 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:18:08.613 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.611+0000 7fae29e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562b8dc41160 0x562b8dc4aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:08.613 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.611+0000 7fae2a800640 1 -- 172.21.15.19:0/1765128207 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562b8dc41160 msgr2=0x562b8dc4aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:18:08.613 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.611+0000 7fae2a800640 1 --2- 172.21.15.19:0/1765128207 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562b8dc41160 0x562b8dc4aec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:08.614 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.611+0000 7fae2a800640 1 -- 172.21.15.19:0/1765128207 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x562b8dc41b50 msgr2=0x562b8dc52780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:18:08.614 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.611+0000 7fae2a800640 1 --2- 172.21.15.19:0/1765128207 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x562b8dc41b50 0x562b8dc52780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:08.614 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.611+0000 7fae2a800640 1 -- 172.21.15.19:0/1765128207 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562b8dc12750 con 0x562b8dc46cc0 2024-03-18T15:18:08.614 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.611+0000 7fae29e00640 1 --2- 172.21.15.19:0/1765128207 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562b8dc41160 0x562b8dc4aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:18:08.614 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.612+0000 7fae2a800640 1 --2- 172.21.15.19:0/1765128207 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562b8dc46cc0 0x562b8dc4b400 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fae20010950 tx=0x7fae20016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c4eeef83b1a4e101 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:18:08.619 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.612+0000 7fae12600640 1 -- 172.21.15.19:0/1765128207 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae20004b20 con 0x562b8dc46cc0 2024-03-18T15:18:08.620 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.612+0000 7fae12600640 1 -- 172.21.15.19:0/1765128207 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae2001fe10 con 0x562b8dc46cc0 2024-03-18T15:18:08.620 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.612+0000 7fae12600640 1 -- 172.21.15.19:0/1765128207 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae20005670 con 0x562b8dc46cc0 2024-03-18T15:18:08.620 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.612+0000 7fae2f5e5740 1 -- 172.21.15.19:0/1765128207 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562b8dc46cc0 msgr2=0x562b8dc4b400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:08.620 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.612+0000 7fae2f5e5740 1 --2- 172.21.15.19:0/1765128207 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562b8dc46cc0 0x562b8dc4b400 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fae20010950 tx=0x7fae20016000 comp rx=0 tx=0).stop 2024-03-18T15:18:08.620 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.613+0000 7fae2f5e5740 1 -- 172.21.15.19:0/1765128207 shutdown_connections 2024-03-18T15:18:08.620 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.613+0000 7fae2f5e5740 1 --2- 172.21.15.19:0/1765128207 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562b8dc46cc0 0x562b8dc4b400 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:08.620 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.613+0000 7fae2f5e5740 1 --2- 172.21.15.19:0/1765128207 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x562b8dc41b50 0x562b8dc52780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:08.620 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.613+0000 7fae2f5e5740 1 --2- 172.21.15.19:0/1765128207 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562b8dc41160 0x562b8dc4aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:08.620 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.613+0000 7fae2f5e5740 1 -- 172.21.15.19:0/1765128207 >> 172.21.15.19:0/1765128207 conn(0x562b8dc3c6d0 msgr2=0x562b8dc3cab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:18:08.620 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.613+0000 7fae2f5e5740 1 -- 172.21.15.19:0/1765128207 shutdown_connections 2024-03-18T15:18:08.621 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.613+0000 7fae2f5e5740 1 -- 172.21.15.19:0/1765128207 wait complete. 2024-03-18T15:18:08.621 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.613+0000 7fae2f5e5740 1 -- start start 2024-03-18T15:18:08.621 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.613+0000 7fae2f5e5740 1 Processor -- start 2024-03-18T15:18:08.621 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.614+0000 7fae2f5e5740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562b8dc41160 0x562b8dc50770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:08.621 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.614+0000 7fae2f5e5740 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x562b8dc41b50 0x562b8dcb9de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:08.621 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.614+0000 7fae2f5e5740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562b8dc46cc0 0x562b8dcbc330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:08.621 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.614+0000 7fae2f5e5740 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x562b8dc59380 con 0x562b8dc41b50 2024-03-18T15:18:08.621 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.614+0000 7fae2f5e5740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x562b8dc58cb0 con 0x562b8dc46cc0 2024-03-18T15:18:08.621 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.614+0000 7fae2f5e5740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x562b8dc59a50 con 0x562b8dc41160 2024-03-18T15:18:08.622 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.614+0000 7fae2f5e5740 10 client.0 _ll_register_callbacks cb 0x7fae180046c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T15:18:08.622 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.614+0000 7fae29e00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562b8dc41160 0x562b8dc50770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:08.622 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.614+0000 7fae29e00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562b8dc41160 0x562b8dc50770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.19:39348/0 (socket says 172.21.15.19:39348) 2024-03-18T15:18:08.622 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.614+0000 7fae29e00640 1 -- 172.21.15.19:0/642012440 learned_addr learned my addr 172.21.15.19:0/642012440 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:18:08.622 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.614+0000 7fae29400640 1 --2- 172.21.15.19:0/642012440 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x562b8dc41b50 0x562b8dcb9de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:08.622 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.614+0000 7fae2a800640 1 --2- 172.21.15.19:0/642012440 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562b8dc46cc0 0x562b8dcbc330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:08.622 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.614+0000 7fae29e00640 1 -- 172.21.15.19:0/642012440 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562b8dc46cc0 msgr2=0x562b8dcbc330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:18:08.622 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.614+0000 7fae29e00640 1 --2- 172.21.15.19:0/642012440 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562b8dc46cc0 0x562b8dcbc330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:08.622 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.614+0000 7fae29e00640 1 -- 172.21.15.19:0/642012440 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x562b8dc41b50 msgr2=0x562b8dcb9de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:18:08.622 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.614+0000 7fae29e00640 1 --2- 172.21.15.19:0/642012440 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x562b8dc41b50 0x562b8dcb9de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:08.623 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.614+0000 7fae29e00640 1 -- 172.21.15.19:0/642012440 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562b8dcbca40 con 0x562b8dc41160 2024-03-18T15:18:08.623 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.615+0000 7fae29e00640 1 --2- 172.21.15.19:0/642012440 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562b8dc41160 0x562b8dc50770 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fae18098140 tx=0x7fae180cc970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a20f8d764de4f988 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:18:08.623 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.615+0000 7fae07400640 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-03-18T15:18:08.623 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.615+0000 7fae07400640 1 -- 172.21.15.19:0/642012440 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae1809b1b0 con 0x562b8dc41160 2024-03-18T15:18:08.623 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.615+0000 7fae07400640 1 -- 172.21.15.19:0/642012440 <== mon.1 v2:172.21.15.104:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fae18097510 con 0x562b8dc41160 2024-03-18T15:18:08.623 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.615+0000 7fae2f5e5740 1 -- 172.21.15.19:0/642012440 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fae2001fe10 con 0x562b8dc41160 2024-03-18T15:18:08.623 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.615+0000 7fae07400640 1 -- 172.21.15.19:0/642012440 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fae18097a60 con 0x562b8dc41160 2024-03-18T15:18:08.624 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.615+0000 7fae2f5e5740 10 client.25363 fetch_fsmap learned FSMap version 26 2024-03-18T15:18:08.624 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.615+0000 7fae2f5e5740 1 -- 172.21.15.19:0/642012440 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x562b8dcbddb0 con 0x562b8dc41160 2024-03-18T15:18:08.624 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.615+0000 7fae07400640 1 -- 172.21.15.19:0/642012440 <== mon.1 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae2001fe10 con 0x562b8dc41160 2024-03-18T15:18:08.624 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.615+0000 7fae07400640 1 -- 172.21.15.19:0/642012440 <== mon.1 v2:172.21.15.104:3300/0 5 ==== fsmap(e 26) ==== 8362+0+0 (secure 0 0 0) 0x7fae18097710 con 0x562b8dc41160 2024-03-18T15:18:08.624 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.615+0000 7fae2f5e5740 10 client.25363 fetch_fsmap finished waiting for FSMap version 26 2024-03-18T15:18:08.624 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.616+0000 7fae2f5e5740 10 client.25363 resolve_mds: resolved name 'f' to daemon mds.f 2024-03-18T15:18:08.624 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.616+0000 7fae2f5e5740 20 client.25363 populate_metadata read hostname 'smithi019' 2024-03-18T15:18:08.624 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.616+0000 7fae2f5e5740 1 --2- 172.21.15.19:0/642012440 >> [v2:172.21.15.104:6834/3788595957,v1:172.21.15.104:6835/3788595957] conn(0x562b8dcc29d0 0x562b8dcbee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:08.624 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.616+0000 7fae2f5e5740 4 client.25363 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-18T15:18:08.624 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.616+0000 7fae2f5e5740 1 -- 172.21.15.19:0/642012440 --> [v2:172.21.15.104:6834/3788595957,v1:172.21.15.104:6835/3788595957] -- command(tid 0: {"prefix": "status"}) -- 0x562b8dbb3190 con 0x562b8dcc29d0 2024-03-18T15:18:08.624 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.616+0000 7fae29400640 1 --2- 172.21.15.19:0/642012440 >> [v2:172.21.15.104:6834/3788595957,v1:172.21.15.104:6835/3788595957] conn(0x562b8dcc29d0 0x562b8dcbee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:08.624 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.616+0000 7fae29400640 1 --2- 172.21.15.19:0/642012440 >> [v2:172.21.15.104:6834/3788595957,v1:172.21.15.104:6835/3788595957] conn(0x562b8dcc29d0 0x562b8dcbee10 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=f17f4971f1649c8 server_cookie=5335a04f7e8a616e in_seq=0 out_seq=0 2024-03-18T15:18:08.624 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.616+0000 7fae07400640 10 client.25363 ms_handle_connect on v2:172.21.15.104:6834/3788595957 2024-03-18T15:18:08.625 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.616+0000 7fae07400640 1 -- 172.21.15.19:0/642012440 <== mds.4 v2:172.21.15.104:6834/3788595957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+321 (crc 0 0 0) 0x562b8dbb3190 con 0x562b8dcc29d0 2024-03-18T15:18:08.625 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.616+0000 7fae07400640 10 client.25363 handle_command_reply: tid=0 2024-03-18T15:18:08.625 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.616+0000 7fae2f5e5740 1 client.25363 shutdown 2024-03-18T15:18:08.625 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.617+0000 7fae2f5e5740 1 -- 172.21.15.19:0/642012440 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562b8dc41160 msgr2=0x562b8dc50770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:08.625 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.617+0000 7fae2f5e5740 1 --2- 172.21.15.19:0/642012440 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562b8dc41160 0x562b8dc50770 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fae18098140 tx=0x7fae180cc970 comp rx=0 tx=0).stop 2024-03-18T15:18:08.625 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.617+0000 7fae2f5e5740 1 -- 172.21.15.19:0/642012440 shutdown_connections 2024-03-18T15:18:08.625 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.617+0000 7fae2f5e5740 1 --2- 172.21.15.19:0/642012440 >> [v2:172.21.15.104:6834/3788595957,v1:172.21.15.104:6835/3788595957] conn(0x562b8dcc29d0 0x562b8dcbee10 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:08.625 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.617+0000 7fae2f5e5740 1 --2- 172.21.15.19:0/642012440 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562b8dc46cc0 0x562b8dcbc330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:08.626 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.617+0000 7fae2f5e5740 1 --2- 172.21.15.19:0/642012440 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x562b8dc41b50 0x562b8dcb9de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:08.626 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.617+0000 7fae07400640 0 client.25363 ms_handle_reset on v2:172.21.15.104:6834/3788595957 2024-03-18T15:18:08.626 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.617+0000 7fae2f5e5740 1 --2- 172.21.15.19:0/642012440 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562b8dc41160 0x562b8dc50770 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:08.626 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.617+0000 7fae2f5e5740 1 -- 172.21.15.19:0/642012440 >> 172.21.15.19:0/642012440 conn(0x562b8dc3c6d0 msgr2=0x562b8dc4aec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:18:08.626 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.617+0000 7fae2f5e5740 1 -- 172.21.15.19:0/642012440 shutdown_connections 2024-03-18T15:18:08.626 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.617+0000 7fae2f5e5740 1 -- 172.21.15.19:0/642012440 wait complete. 2024-03-18T15:18:08.626 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.617+0000 7fae2f5e5740 20 client.25363 trim_cache size 0 max 16384 2024-03-18T15:18:08.626 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-03-18T15:18:08.627 INFO:teuthology.orchestra.run.smithi019.stdout: "cluster_fsid": "d4dc2ba6-e538-11ee-95c9-87774f69a715", 2024-03-18T15:18:08.627 INFO:teuthology.orchestra.run.smithi019.stdout: "whoami": 4, 2024-03-18T15:18:08.627 INFO:teuthology.orchestra.run.smithi019.stdout: "id": 24499, 2024-03-18T15:18:08.627 INFO:teuthology.orchestra.run.smithi019.stdout: "want_state": "up:active", 2024-03-18T15:18:08.627 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "up:active", 2024-03-18T15:18:08.627 INFO:teuthology.orchestra.run.smithi019.stdout: "fs_name": "cephfs", 2024-03-18T15:18:08.627 INFO:teuthology.orchestra.run.smithi019.stdout: "rank_uptime": 142.085664038, 2024-03-18T15:18:08.627 INFO:teuthology.orchestra.run.smithi019.stdout: "mdsmap_epoch": 26, 2024-03-18T15:18:08.627 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap_epoch": 93, 2024-03-18T15:18:08.627 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap_epoch_barrier": 89, 2024-03-18T15:18:08.627 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 204.26821196 2024-03-18T15:18:08.628 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-03-18T15:18:08.628 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.618+0000 7fae2c800640 1 -- 172.21.15.19:0/2051917613 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fae0c0a2880 msgr2=0x7fae0c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:08.629 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.618+0000 7fae2c800640 1 --2- 172.21.15.19:0/2051917613 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fae0c0a2880 0x7fae0c0a4d40 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x562b8dc423b0 tx=0x7fae20009500 comp rx=0 tx=0).stop 2024-03-18T15:18:08.629 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.618+0000 7fae2c800640 1 -- 172.21.15.19:0/2051917613 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fae24103a00 msgr2=0x7fae24199d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:08.629 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.618+0000 7fae2c800640 1 --2- 172.21.15.19:0/2051917613 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fae24103a00 0x7fae24199d10 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fae1800e9c0 tx=0x7fae1800ee80 comp rx=0 tx=0).stop 2024-03-18T15:18:08.629 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.618+0000 7fae2c800640 1 -- 172.21.15.19:0/2051917613 shutdown_connections 2024-03-18T15:18:08.629 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.618+0000 7fae2c800640 1 --2- 172.21.15.19:0/2051917613 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fae241055b0 0x7fae241a1280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:08.629 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.618+0000 7fae2c800640 1 --2- 172.21.15.19:0/2051917613 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fae0c0a2880 0x7fae0c0a4d40 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:08.629 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.618+0000 7fae2c800640 1 --2- 172.21.15.19:0/2051917613 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae24104bf0 0x7fae2419a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:08.629 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.618+0000 7fae2c800640 1 --2- 172.21.15.19:0/2051917613 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fae24103a00 0x7fae24199d10 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:08.629 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.618+0000 7fae2c800640 1 -- 172.21.15.19:0/2051917613 >> 172.21.15.19:0/2051917613 conn(0x7fae240fef90 msgr2=0x7fae24107ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:18:08.629 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.618+0000 7fae2c800640 1 -- 172.21.15.19:0/2051917613 shutdown_connections 2024-03-18T15:18:08.629 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.618+0000 7fae2c800640 1 -- 172.21.15.19:0/2051917613 wait complete. 2024-03-18T15:18:08.636 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-03-18T15:18:08.864 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.863+0000 7f8883e00640 1 -- 172.21.15.19:0/240775205 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f88841054b0 msgr2=0x7f888410bd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:08.865 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.863+0000 7f8883e00640 1 --2- 172.21.15.19:0/240775205 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f88841054b0 0x7f888410bd40 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f887c009ca0 tx=0x7f887c02c100 comp rx=0 tx=0).stop 2024-03-18T15:18:08.865 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.863+0000 7f8883e00640 1 -- 172.21.15.19:0/240775205 shutdown_connections 2024-03-18T15:18:08.865 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.863+0000 7f8883e00640 1 --2- 172.21.15.19:0/240775205 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f88841054b0 0x7f888410bd40 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:08.865 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.863+0000 7f8883e00640 1 --2- 172.21.15.19:0/240775205 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8884104af0 0x7f8884104f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:08.865 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.863+0000 7f8883e00640 1 --2- 172.21.15.19:0/240775205 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8884103900 0x7f8884103d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:08.865 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.863+0000 7f8883e00640 1 -- 172.21.15.19:0/240775205 >> 172.21.15.19:0/240775205 conn(0x7f88840fee90 msgr2=0x7f88841012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:18:08.865 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.863+0000 7f8883e00640 1 -- 172.21.15.19:0/240775205 shutdown_connections 2024-03-18T15:18:08.865 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.863+0000 7f8883e00640 1 -- 172.21.15.19:0/240775205 wait complete. 2024-03-18T15:18:08.868 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.864+0000 7f8883e00640 1 Processor -- start 2024-03-18T15:18:08.868 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.865+0000 7f8883e00640 1 -- start start 2024-03-18T15:18:08.868 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.865+0000 7f8883e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8884103900 0x7f8884199b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:08.868 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.865+0000 7f8883e00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8884104af0 0x7f888419a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:08.868 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.865+0000 7f8883e00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f88841054b0 0x7f88841a1130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:08.868 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.865+0000 7f8883e00640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f888410f560 con 0x7f88841054b0 2024-03-18T15:18:08.868 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.865+0000 7f8883e00640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f888410f3e0 con 0x7f8884103900 2024-03-18T15:18:08.868 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.865+0000 7f8883e00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f888410f6e0 con 0x7f8884104af0 2024-03-18T15:18:08.868 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.865+0000 7f8882a00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8884103900 0x7f8884199b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:08.868 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.865+0000 7f8882a00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8884103900 0x7f8884199b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:46282/0 (socket says 172.21.15.19:46282) 2024-03-18T15:18:08.868 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.865+0000 7f8882a00640 1 -- 172.21.15.19:0/1277741730 learned_addr learned my addr 172.21.15.19:0/1277741730 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:18:08.869 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.865+0000 7f8882000640 1 --2- 172.21.15.19:0/1277741730 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8884104af0 0x7f888419a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:08.869 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.866+0000 7f8883400640 1 --2- 172.21.15.19:0/1277741730 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f88841054b0 0x7f88841a1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:08.869 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.866+0000 7f8882a00640 1 -- 172.21.15.19:0/1277741730 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8884104af0 msgr2=0x7f888419a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:08.869 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.866+0000 7f8882a00640 1 --2- 172.21.15.19:0/1277741730 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8884104af0 0x7f888419a0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:08.869 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.866+0000 7f8882a00640 1 -- 172.21.15.19:0/1277741730 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f88841054b0 msgr2=0x7f88841a1130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:08.869 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.866+0000 7f8882a00640 1 --2- 172.21.15.19:0/1277741730 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f88841054b0 0x7f88841a1130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:08.869 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.866+0000 7f8882a00640 1 -- 172.21.15.19:0/1277741730 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88841a17a0 con 0x7f8884103900 2024-03-18T15:18:08.869 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.866+0000 7f8883400640 1 --2- 172.21.15.19:0/1277741730 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f88841054b0 0x7f88841a1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T15:18:08.869 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.866+0000 7f8882a00640 1 --2- 172.21.15.19:0/1277741730 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8884103900 0x7f8884199b80 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f887400e9c0 tx=0x7f887400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:18:08.869 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.866+0000 7f887b400640 1 -- 172.21.15.19:0/1277741730 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f887400cde0 con 0x7f8884103900 2024-03-18T15:18:08.869 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.866+0000 7f887b400640 1 -- 172.21.15.19:0/1277741730 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8874010910 con 0x7f8884103900 2024-03-18T15:18:08.870 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.866+0000 7f8883e00640 1 -- 172.21.15.19:0/1277741730 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88841a1a90 con 0x7f8884103900 2024-03-18T15:18:08.870 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.866+0000 7f887b400640 1 -- 172.21.15.19:0/1277741730 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8874018a00 con 0x7f8884103900 2024-03-18T15:18:08.870 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.867+0000 7f8883e00640 1 -- 172.21.15.19:0/1277741730 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88841a2030 con 0x7f8884103900 2024-03-18T15:18:08.870 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.869+0000 7f887b400640 1 -- 172.21.15.19:0/1277741730 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f887401f020 con 0x7f8884103900 2024-03-18T15:18:08.870 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.870+0000 7f888c85c740 1 Processor -- start 2024-03-18T15:18:08.871 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.870+0000 7f888c85c740 1 -- start start 2024-03-18T15:18:08.872 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.870+0000 7f888c85c740 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x556f15d0b340 0x556f15d0b760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:08.872 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.870+0000 7f888c85c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556f15d0bd30 0x556f15d10830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:08.872 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.870+0000 7f888c85c740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556f15d10ea0 0x556f15d13330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:08.872 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.870+0000 7f888c85c740 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x556f15c6fc50 con 0x556f15d0b340 2024-03-18T15:18:08.872 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.870+0000 7f888c85c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x556f15c189d0 con 0x556f15d0bd30 2024-03-18T15:18:08.872 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.870+0000 7f888c85c740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x556f15c6bcf0 con 0x556f15d10ea0 2024-03-18T15:18:08.872 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.870+0000 7f8882000640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556f15d10ea0 0x556f15d13330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:08.872 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.870+0000 7f8883400640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x556f15d0b340 0x556f15d0b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:08.872 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.870+0000 7f8882a00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556f15d0bd30 0x556f15d10830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:08.872 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.870+0000 7f8882000640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556f15d10ea0 0x556f15d13330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.19:39364/0 (socket says 172.21.15.19:39364) 2024-03-18T15:18:08.872 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.870+0000 7f8882000640 1 -- 172.21.15.19:0/2062444746 learned_addr learned my addr 172.21.15.19:0/2062444746 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:18:08.873 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.871+0000 7f8882000640 1 -- 172.21.15.19:0/2062444746 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556f15d0bd30 msgr2=0x556f15d10830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:18:08.873 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.871+0000 7f8882000640 1 --2- 172.21.15.19:0/2062444746 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556f15d0bd30 0x556f15d10830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:08.873 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.871+0000 7f8882000640 1 -- 172.21.15.19:0/2062444746 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x556f15d0b340 msgr2=0x556f15d0b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:18:08.873 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.871+0000 7f8882000640 1 --2- 172.21.15.19:0/2062444746 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x556f15d0b340 0x556f15d0b760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:08.873 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.871+0000 7f8882000640 1 -- 172.21.15.19:0/2062444746 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556f15c32f10 con 0x556f15d10ea0 2024-03-18T15:18:08.873 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.871+0000 7f8882000640 1 --2- 172.21.15.19:0/2062444746 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556f15d10ea0 0x556f15d13330 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f8870009c90 tx=0x7f8870011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=97fe916333fedc1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:18:08.877 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.871+0000 7f886fe00640 1 -- 172.21.15.19:0/2062444746 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f887001e070 con 0x556f15d10ea0 2024-03-18T15:18:08.877 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.871+0000 7f887b400640 1 --2- 172.21.15.19:0/1277741730 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f88680a2880 0x7f88680a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:08.878 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.871+0000 7f886fe00640 1 -- 172.21.15.19:0/2062444746 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8870019b10 con 0x556f15d10ea0 2024-03-18T15:18:08.878 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.871+0000 7f886fe00640 1 -- 172.21.15.19:0/2062444746 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f887000eac0 con 0x556f15d10ea0 2024-03-18T15:18:08.878 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.872+0000 7f887b400640 1 -- 172.21.15.19:0/1277741730 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f88740cf0a0 con 0x7f8884103900 2024-03-18T15:18:08.878 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.872+0000 7f8883400640 1 --2- 172.21.15.19:0/1277741730 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f88680a2880 0x7f88680a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:08.878 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.872+0000 7f888c85c740 1 -- 172.21.15.19:0/2062444746 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556f15d10ea0 msgr2=0x556f15d13330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:08.878 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.872+0000 7f888c85c740 1 --2- 172.21.15.19:0/2062444746 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556f15d10ea0 0x556f15d13330 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f8870009c90 tx=0x7f8870011810 comp rx=0 tx=0).stop 2024-03-18T15:18:08.878 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.872+0000 7f888c85c740 1 -- 172.21.15.19:0/2062444746 shutdown_connections 2024-03-18T15:18:08.878 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.872+0000 7f888c85c740 1 --2- 172.21.15.19:0/2062444746 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556f15d10ea0 0x556f15d13330 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:08.878 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.872+0000 7f888c85c740 1 --2- 172.21.15.19:0/2062444746 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556f15d0bd30 0x556f15d10830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:08.878 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.872+0000 7f888c85c740 1 --2- 172.21.15.19:0/2062444746 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x556f15d0b340 0x556f15d0b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:08.878 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.872+0000 7f8883400640 1 --2- 172.21.15.19:0/1277741730 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f88680a2880 0x7f88680a4d40 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f887c0324c0 tx=0x7f887c005f50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:18:08.879 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.872+0000 7f888c85c740 1 -- 172.21.15.19:0/2062444746 >> 172.21.15.19:0/2062444746 conn(0x556f15d068b0 msgr2=0x556f15d06cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:18:08.879 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.872+0000 7f888c85c740 1 -- 172.21.15.19:0/2062444746 shutdown_connections 2024-03-18T15:18:08.879 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.872+0000 7f888c85c740 1 -- 172.21.15.19:0/2062444746 wait complete. 2024-03-18T15:18:08.879 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.873+0000 7f888c85c740 1 -- start start 2024-03-18T15:18:08.879 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.873+0000 7f888c85c740 1 Processor -- start 2024-03-18T15:18:08.879 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.874+0000 7f888c85c740 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x556f15d0b340 0x556f15d15490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:08.879 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.874+0000 7f888c85c740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556f15d0bd30 0x556f15d159d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:08.879 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.874+0000 7f888c85c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556f15d10ea0 0x556f15d1ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:08.879 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.874+0000 7f888c85c740 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x556f15d13e50 con 0x556f15d0b340 2024-03-18T15:18:08.879 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.874+0000 7f888c85c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x556f15c2be20 con 0x556f15d10ea0 2024-03-18T15:18:08.879 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.874+0000 7f888c85c740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x556f15d13fd0 con 0x556f15d0bd30 2024-03-18T15:18:08.879 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.874+0000 7f888c85c740 10 client.0 _ll_register_callbacks cb 0x556f15c2b7f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T15:18:08.879 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.874+0000 7f8882000640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556f15d0bd30 0x556f15d159d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:08.880 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.874+0000 7f8882000640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556f15d0bd30 0x556f15d159d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.19:39372/0 (socket says 172.21.15.19:39372) 2024-03-18T15:18:08.880 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.874+0000 7f8883400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556f15d10ea0 0x556f15d1ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:08.880 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.874+0000 7f8882000640 1 -- 172.21.15.19:0/1396196125 learned_addr learned my addr 172.21.15.19:0/1396196125 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:18:08.880 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.874+0000 7f8882a00640 1 --2- 172.21.15.19:0/1396196125 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x556f15d0b340 0x556f15d15490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:08.880 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.874+0000 7f8882000640 1 -- 172.21.15.19:0/1396196125 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556f15d10ea0 msgr2=0x556f15d1ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:18:08.880 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.874+0000 7f8882000640 1 --2- 172.21.15.19:0/1396196125 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556f15d10ea0 0x556f15d1ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:08.880 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.874+0000 7f8882000640 1 -- 172.21.15.19:0/1396196125 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x556f15d0b340 msgr2=0x556f15d15490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:18:08.880 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.874+0000 7f8882000640 1 --2- 172.21.15.19:0/1396196125 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x556f15d0b340 0x556f15d15490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:08.880 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.874+0000 7f8882000640 1 -- 172.21.15.19:0/1396196125 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556f159842d0 con 0x556f15d0bd30 2024-03-18T15:18:08.880 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.874+0000 7f8883400640 1 --2- 172.21.15.19:0/1396196125 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556f15d10ea0 0x556f15d1ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:18:08.880 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.875+0000 7f8882000640 1 --2- 172.21.15.19:0/1396196125 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556f15d0bd30 0x556f15d159d0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f8870019a20 tx=0x7f8870019a50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8706bfc7fc191ddc server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:18:08.881 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.875+0000 7f886cc00640 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-03-18T15:18:08.881 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.875+0000 7f886cc00640 1 -- 172.21.15.19:0/1396196125 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f887001e070 con 0x556f15d0bd30 2024-03-18T15:18:08.881 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.875+0000 7f886cc00640 1 -- 172.21.15.19:0/1396196125 <== mon.1 v2:172.21.15.104:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f8870004230 con 0x556f15d0bd30 2024-03-18T15:18:08.881 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.875+0000 7f888c85c740 1 -- 172.21.15.19:0/1396196125 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8870019b10 con 0x556f15d0bd30 2024-03-18T15:18:08.881 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.875+0000 7f886cc00640 1 -- 172.21.15.19:0/1396196125 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f8870033050 con 0x556f15d0bd30 2024-03-18T15:18:08.881 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.875+0000 7f888c85c740 10 client.25375 fetch_fsmap learned FSMap version 26 2024-03-18T15:18:08.881 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.875+0000 7f888c85c740 1 -- 172.21.15.19:0/1396196125 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x556f15a4e6b0 con 0x556f15d0bd30 2024-03-18T15:18:08.881 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.875+0000 7f886cc00640 1 -- 172.21.15.19:0/1396196125 <== mon.1 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8870019b10 con 0x556f15d0bd30 2024-03-18T15:18:08.881 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.876+0000 7f886cc00640 1 -- 172.21.15.19:0/1396196125 <== mon.1 v2:172.21.15.104:3300/0 5 ==== fsmap(e 26) ==== 8362+0+0 (secure 0 0 0) 0x7f887000e8e0 con 0x556f15d0bd30 2024-03-18T15:18:08.881 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.876+0000 7f888c85c740 10 client.25375 fetch_fsmap finished waiting for FSMap version 26 2024-03-18T15:18:08.881 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.876+0000 7f888c85c740 10 client.25375 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-18T15:18:08.882 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.876+0000 7f888c85c740 20 client.25375 populate_metadata read hostname 'smithi019' 2024-03-18T15:18:08.882 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.876+0000 7f888c85c740 1 --2- 172.21.15.19:0/1396196125 >> [v2:172.21.15.33:6836/160187378,v1:172.21.15.33:6837/160187378] conn(0x556f15d21f40 0x556f15d233b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:08.882 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.876+0000 7f888c85c740 4 client.25375 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T15:18:08.882 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.876+0000 7f888c85c740 1 -- 172.21.15.19:0/1396196125 --> [v2:172.21.15.33:6836/160187378,v1:172.21.15.33:6837/160187378] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556f15ccd2b0 con 0x556f15d21f40 2024-03-18T15:18:08.882 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.876+0000 7f8882a00640 1 --2- 172.21.15.19:0/1396196125 >> [v2:172.21.15.33:6836/160187378,v1:172.21.15.33:6837/160187378] conn(0x556f15d21f40 0x556f15d233b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:08.882 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.877+0000 7f8882a00640 1 --2- 172.21.15.19:0/1396196125 >> [v2:172.21.15.33:6836/160187378,v1:172.21.15.33:6837/160187378] conn(0x556f15d21f40 0x556f15d233b0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=aadd959bc79e8de7 server_cookie=707ef5c3fc78124b in_seq=0 out_seq=0 2024-03-18T15:18:08.882 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.877+0000 7f886cc00640 10 client.25375 ms_handle_connect on v2:172.21.15.33:6836/160187378 2024-03-18T15:18:08.883 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.878+0000 7f886cc00640 1 -- 172.21.15.19:0/1396196125 <== mds.3 v2:172.21.15.33:6836/160187378 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556f15ccd2b0 con 0x556f15d21f40 2024-03-18T15:18:08.884 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.878+0000 7f886cc00640 10 client.25375 handle_command_reply: tid=0 2024-03-18T15:18:08.884 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.878+0000 7f888c85c740 1 client.25375 shutdown 2024-03-18T15:18:08.884 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.878+0000 7f888c85c740 1 -- 172.21.15.19:0/1396196125 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556f15d0bd30 msgr2=0x556f15d159d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:08.884 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.878+0000 7f888c85c740 1 --2- 172.21.15.19:0/1396196125 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556f15d0bd30 0x556f15d159d0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f8870019a20 tx=0x7f8870019a50 comp rx=0 tx=0).stop 2024-03-18T15:18:08.884 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.878+0000 7f888c85c740 1 -- 172.21.15.19:0/1396196125 shutdown_connections 2024-03-18T15:18:08.884 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.878+0000 7f888c85c740 1 --2- 172.21.15.19:0/1396196125 >> [v2:172.21.15.33:6836/160187378,v1:172.21.15.33:6837/160187378] conn(0x556f15d21f40 0x556f15d233b0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:08.884 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.878+0000 7f888c85c740 1 --2- 172.21.15.19:0/1396196125 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556f15d10ea0 0x556f15d1ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:08.885 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.878+0000 7f888c85c740 1 --2- 172.21.15.19:0/1396196125 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556f15d0bd30 0x556f15d159d0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:08.885 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.878+0000 7f888c85c740 1 --2- 172.21.15.19:0/1396196125 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x556f15d0b340 0x556f15d15490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:08.885 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.878+0000 7f888c85c740 1 -- 172.21.15.19:0/1396196125 >> 172.21.15.19:0/1396196125 conn(0x556f15d068b0 msgr2=0x556f15d05a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:18:08.885 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.878+0000 7f886cc00640 0 client.25375 ms_handle_reset on v2:172.21.15.33:6836/160187378 2024-03-18T15:18:08.885 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.878+0000 7f888c85c740 1 -- 172.21.15.19:0/1396196125 shutdown_connections 2024-03-18T15:18:08.885 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.878+0000 7f888c85c740 1 -- 172.21.15.19:0/1396196125 wait complete. 2024-03-18T15:18:08.886 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.878+0000 7f888c85c740 20 client.25375 trim_cache size 0 max 16384 2024-03-18T15:18:08.903 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.900+0000 7f888c85c740 1 Processor -- start 2024-03-18T15:18:08.903 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.900+0000 7f888c85c740 1 -- start start 2024-03-18T15:18:08.903 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.901+0000 7f888c85c740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556f15d068b0 0x556f15d058f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:08.904 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.901+0000 7f888c85c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556f15d0b340 0x556f15d1c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:08.904 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.901+0000 7f888c85c740 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x556f15d0bd30 0x556f15d0c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:08.904 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.901+0000 7f888c85c740 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x556f15d216f0 con 0x556f15d0bd30 2024-03-18T15:18:08.904 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.901+0000 7f888c85c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x556f15d21dc0 con 0x556f15d0b340 2024-03-18T15:18:08.904 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.901+0000 7f888c85c740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x556f15d22490 con 0x556f15d068b0 2024-03-18T15:18:08.904 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.901+0000 7f8882000640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556f15d0b340 0x556f15d1c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:08.904 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.901+0000 7f8882a00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556f15d068b0 0x556f15d058f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:08.905 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.901+0000 7f8883400640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x556f15d0bd30 0x556f15d0c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:08.905 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.901+0000 7f8882a00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556f15d068b0 0x556f15d058f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.19:39384/0 (socket says 172.21.15.19:39384) 2024-03-18T15:18:08.905 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.901+0000 7f8882a00640 1 -- 172.21.15.19:0/787850003 learned_addr learned my addr 172.21.15.19:0/787850003 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:18:08.905 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.901+0000 7f8882000640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556f15d0b340 0x556f15d1c9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:46306/0 (socket says 172.21.15.19:46306) 2024-03-18T15:18:08.905 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.901+0000 7f8882000640 1 -- 172.21.15.19:0/787850003 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556f15d068b0 msgr2=0x556f15d058f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:18:08.905 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.901+0000 7f8882000640 1 --2- 172.21.15.19:0/787850003 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556f15d068b0 0x556f15d058f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:08.905 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.901+0000 7f8882000640 1 -- 172.21.15.19:0/787850003 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x556f15d0bd30 msgr2=0x556f15d0c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:18:08.906 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.901+0000 7f8882000640 1 --2- 172.21.15.19:0/787850003 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x556f15d0bd30 0x556f15d0c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:08.906 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.901+0000 7f8882000640 1 -- 172.21.15.19:0/787850003 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556f15d07620 con 0x556f15d0b340 2024-03-18T15:18:08.906 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.902+0000 7f8882000640 1 --2- 172.21.15.19:0/787850003 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556f15d0b340 0x556f15d1c9d0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f887000ed00 tx=0x7f8870009f00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=daeed70581156926 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:18:08.906 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.902+0000 7f886f400640 1 -- 172.21.15.19:0/787850003 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f887001e070 con 0x556f15d0b340 2024-03-18T15:18:08.906 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.902+0000 7f886f400640 1 -- 172.21.15.19:0/787850003 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f887001f040 con 0x556f15d0b340 2024-03-18T15:18:08.906 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.902+0000 7f886f400640 1 -- 172.21.15.19:0/787850003 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8870005580 con 0x556f15d0b340 2024-03-18T15:18:08.914 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.903+0000 7f888c85c740 1 -- 172.21.15.19:0/787850003 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556f15d0b340 msgr2=0x556f15d1c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:08.914 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.903+0000 7f888c85c740 1 --2- 172.21.15.19:0/787850003 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556f15d0b340 0x556f15d1c9d0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f887000ed00 tx=0x7f8870009f00 comp rx=0 tx=0).stop 2024-03-18T15:18:08.914 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.903+0000 7f888c85c740 1 -- 172.21.15.19:0/787850003 shutdown_connections 2024-03-18T15:18:08.914 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.903+0000 7f888c85c740 1 --2- 172.21.15.19:0/787850003 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x556f15d0bd30 0x556f15d0c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:08.914 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.903+0000 7f888c85c740 1 --2- 172.21.15.19:0/787850003 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556f15d0b340 0x556f15d1c9d0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:08.914 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.903+0000 7f888c85c740 1 --2- 172.21.15.19:0/787850003 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556f15d068b0 0x556f15d058f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:08.914 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.903+0000 7f888c85c740 1 -- 172.21.15.19:0/787850003 >> 172.21.15.19:0/787850003 conn(0x556f15d07000 msgr2=0x556f15d051e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:18:08.915 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.903+0000 7f888c85c740 1 -- 172.21.15.19:0/787850003 shutdown_connections 2024-03-18T15:18:08.915 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.903+0000 7f888c85c740 1 -- 172.21.15.19:0/787850003 wait complete. 2024-03-18T15:18:08.915 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.903+0000 7f888c85c740 1 -- start start 2024-03-18T15:18:08.915 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.904+0000 7f888c85c740 1 Processor -- start 2024-03-18T15:18:08.915 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.904+0000 7f888c85c740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556f15d068b0 0x556f15d18090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:08.915 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.904+0000 7f888c85c740 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x556f15d0b340 0x556f15d185d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:08.916 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.904+0000 7f888c85c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556f15d0bd30 0x556f15d1b580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:08.916 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.904+0000 7f888c85c740 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x556f15d23230 con 0x556f15d0b340 2024-03-18T15:18:08.916 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.904+0000 7f888c85c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x556f15d22b60 con 0x556f15d0bd30 2024-03-18T15:18:08.916 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.904+0000 7f888c85c740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x556f15d23900 con 0x556f15d068b0 2024-03-18T15:18:08.916 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.904+0000 7f888c85c740 10 client.0 _ll_register_callbacks cb 0x7f88740d29b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T15:18:08.916 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.904+0000 7f8882a00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556f15d068b0 0x556f15d18090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:08.916 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.904+0000 7f8883400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556f15d0bd30 0x556f15d1b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:08.916 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.904+0000 7f8882000640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x556f15d0b340 0x556f15d185d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:08.917 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.904+0000 7f8882a00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556f15d068b0 0x556f15d18090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.19:39386/0 (socket says 172.21.15.19:39386) 2024-03-18T15:18:08.917 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.904+0000 7f8882a00640 1 -- 172.21.15.19:0/99888821 learned_addr learned my addr 172.21.15.19:0/99888821 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:18:08.917 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.905+0000 7f8883400640 1 -- 172.21.15.19:0/99888821 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556f15d068b0 msgr2=0x556f15d18090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:18:08.917 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.905+0000 7f8883400640 1 --2- 172.21.15.19:0/99888821 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556f15d068b0 0x556f15d18090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:08.917 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.905+0000 7f8883400640 1 -- 172.21.15.19:0/99888821 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x556f15d0b340 msgr2=0x556f15d185d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:18:08.917 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.905+0000 7f8883400640 1 --2- 172.21.15.19:0/99888821 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x556f15d0b340 0x556f15d185d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:08.917 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.905+0000 7f8883400640 1 -- 172.21.15.19:0/99888821 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556f15d165c0 con 0x556f15d0bd30 2024-03-18T15:18:08.918 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.905+0000 7f8883400640 1 --2- 172.21.15.19:0/99888821 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556f15d0bd30 0x556f15d1b580 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f887c02c640 tx=0x7f887c02c100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=69cd5863a9982c08 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:18:08.918 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.905+0000 7f886cc00640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-18T15:18:08.918 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.905+0000 7f886cc00640 1 -- 172.21.15.19:0/99888821 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f887c052030 con 0x556f15d0bd30 2024-03-18T15:18:08.918 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.905+0000 7f888c85c740 1 -- 172.21.15.19:0/99888821 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f887001f040 con 0x556f15d0bd30 2024-03-18T15:18:08.918 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.905+0000 7f886cc00640 1 -- 172.21.15.19:0/99888821 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f887c0043d0 con 0x556f15d0bd30 2024-03-18T15:18:08.918 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.906+0000 7f886cc00640 1 -- 172.21.15.19:0/99888821 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f887c039040 con 0x556f15d0bd30 2024-03-18T15:18:08.918 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.906+0000 7f888c85c740 10 client.25370 fetch_fsmap learned FSMap version 26 2024-03-18T15:18:08.919 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.906+0000 7f888c85c740 1 -- 172.21.15.19:0/99888821 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x556f15d1cd60 con 0x556f15d0bd30 2024-03-18T15:18:08.919 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.906+0000 7f886cc00640 1 -- 172.21.15.19:0/99888821 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f887001f040 con 0x556f15d0bd30 2024-03-18T15:18:08.919 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.906+0000 7f886cc00640 1 -- 172.21.15.19:0/99888821 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 26) ==== 8362+0+0 (secure 0 0 0) 0x7f887c04e420 con 0x556f15d0bd30 2024-03-18T15:18:08.919 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.906+0000 7f888c85c740 10 client.25370 fetch_fsmap finished waiting for FSMap version 26 2024-03-18T15:18:08.919 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.906+0000 7f888c85c740 10 client.25370 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-18T15:18:08.919 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.906+0000 7f888c85c740 20 client.25370 populate_metadata read hostname 'smithi019' 2024-03-18T15:18:08.919 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.906+0000 7f888c85c740 1 --2- 172.21.15.19:0/99888821 >> [v2:172.21.15.33:6836/160187378,v1:172.21.15.33:6837/160187378] conn(0x556f15d10ea0 0x556f15d8b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:08.920 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.906+0000 7f888c85c740 4 client.25370 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-18T15:18:08.920 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.906+0000 7f888c85c740 1 -- 172.21.15.19:0/99888821 --> [v2:172.21.15.33:6836/160187378,v1:172.21.15.33:6837/160187378] -- command(tid 0: {"prefix": "status"}) -- 0x556f15c7d190 con 0x556f15d10ea0 2024-03-18T15:18:08.920 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.907+0000 7f8882a00640 1 --2- 172.21.15.19:0/99888821 >> [v2:172.21.15.33:6836/160187378,v1:172.21.15.33:6837/160187378] conn(0x556f15d10ea0 0x556f15d8b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:08.920 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.907+0000 7f8882a00640 1 --2- 172.21.15.19:0/99888821 >> [v2:172.21.15.33:6836/160187378,v1:172.21.15.33:6837/160187378] conn(0x556f15d10ea0 0x556f15d8b340 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=f5ee71801b051d66 server_cookie=5ea5d901b9fe2843 in_seq=0 out_seq=0 2024-03-18T15:18:08.920 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.907+0000 7f886cc00640 10 client.25370 ms_handle_connect on v2:172.21.15.33:6836/160187378 2024-03-18T15:18:08.920 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.907+0000 7f886cc00640 1 -- 172.21.15.19:0/99888821 <== mds.3 v2:172.21.15.33:6836/160187378 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x556f15c7d190 con 0x556f15d10ea0 2024-03-18T15:18:08.921 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.907+0000 7f886cc00640 10 client.25370 handle_command_reply: tid=0 2024-03-18T15:18:08.921 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.907+0000 7f888c85c740 1 client.25370 shutdown 2024-03-18T15:18:08.921 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.907+0000 7f888c85c740 1 -- 172.21.15.19:0/99888821 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556f15d0bd30 msgr2=0x556f15d1b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:08.921 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.907+0000 7f888c85c740 1 --2- 172.21.15.19:0/99888821 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556f15d0bd30 0x556f15d1b580 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f887c02c640 tx=0x7f887c02c100 comp rx=0 tx=0).stop 2024-03-18T15:18:08.921 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.908+0000 7f888c85c740 1 -- 172.21.15.19:0/99888821 shutdown_connections 2024-03-18T15:18:08.921 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.908+0000 7f888c85c740 1 --2- 172.21.15.19:0/99888821 >> [v2:172.21.15.33:6836/160187378,v1:172.21.15.33:6837/160187378] conn(0x556f15d10ea0 0x556f15d8b340 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:08.921 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.908+0000 7f888c85c740 1 --2- 172.21.15.19:0/99888821 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556f15d0bd30 0x556f15d1b580 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:08.921 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.908+0000 7f888c85c740 1 --2- 172.21.15.19:0/99888821 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x556f15d0b340 0x556f15d185d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:08.921 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.908+0000 7f888c85c740 1 --2- 172.21.15.19:0/99888821 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556f15d068b0 0x556f15d18090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:08.921 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.908+0000 7f888c85c740 1 -- 172.21.15.19:0/99888821 >> 172.21.15.19:0/99888821 conn(0x556f15d07000 msgr2=0x556f15d05a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:18:08.921 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.908+0000 7f886cc00640 0 client.25370 ms_handle_reset on v2:172.21.15.33:6836/160187378 2024-03-18T15:18:08.921 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.908+0000 7f888c85c740 1 -- 172.21.15.19:0/99888821 shutdown_connections 2024-03-18T15:18:08.922 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.908+0000 7f888c85c740 1 -- 172.21.15.19:0/99888821 wait complete. 2024-03-18T15:18:08.922 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.908+0000 7f888c85c740 20 client.25370 trim_cache size 0 max 16384 2024-03-18T15:18:08.922 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-03-18T15:18:08.922 INFO:teuthology.orchestra.run.smithi019.stdout: "cluster_fsid": "d4dc2ba6-e538-11ee-95c9-87774f69a715", 2024-03-18T15:18:08.922 INFO:teuthology.orchestra.run.smithi019.stdout: "whoami": 3, 2024-03-18T15:18:08.923 INFO:teuthology.orchestra.run.smithi019.stdout: "id": 24506, 2024-03-18T15:18:08.923 INFO:teuthology.orchestra.run.smithi019.stdout: "want_state": "up:active", 2024-03-18T15:18:08.923 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "up:active", 2024-03-18T15:18:08.923 INFO:teuthology.orchestra.run.smithi019.stdout: "fs_name": "cephfs", 2024-03-18T15:18:08.923 INFO:teuthology.orchestra.run.smithi019.stdout: "rank_uptime": 143.420641152, 2024-03-18T15:18:08.923 INFO:teuthology.orchestra.run.smithi019.stdout: "mdsmap_epoch": 26, 2024-03-18T15:18:08.923 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap_epoch": 93, 2024-03-18T15:18:08.923 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap_epoch_barrier": 89, 2024-03-18T15:18:08.923 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 198.82284293399999 2024-03-18T15:18:08.923 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-03-18T15:18:08.924 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.909+0000 7f8883e00640 1 -- 172.21.15.19:0/1277741730 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f88680a2880 msgr2=0x7f88680a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:08.924 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.909+0000 7f8883e00640 1 --2- 172.21.15.19:0/1277741730 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f88680a2880 0x7f88680a4d40 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f887c0324c0 tx=0x7f887c005f50 comp rx=0 tx=0).stop 2024-03-18T15:18:08.924 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.909+0000 7f8883e00640 1 -- 172.21.15.19:0/1277741730 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8884103900 msgr2=0x7f8884199b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:08.924 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.909+0000 7f8883e00640 1 --2- 172.21.15.19:0/1277741730 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8884103900 0x7f8884199b80 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f887400e9c0 tx=0x7f887400ee80 comp rx=0 tx=0).stop 2024-03-18T15:18:08.924 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.909+0000 7f8883e00640 1 -- 172.21.15.19:0/1277741730 shutdown_connections 2024-03-18T15:18:08.924 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.909+0000 7f8883e00640 1 --2- 172.21.15.19:0/1277741730 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f88841054b0 0x7f88841a1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:08.924 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.909+0000 7f8883e00640 1 --2- 172.21.15.19:0/1277741730 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8884104af0 0x7f888419a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:08.924 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.909+0000 7f8883e00640 1 --2- 172.21.15.19:0/1277741730 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f88680a2880 0x7f88680a4d40 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:08.924 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.909+0000 7f8883e00640 1 --2- 172.21.15.19:0/1277741730 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8884103900 0x7f8884199b80 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:08.925 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.909+0000 7f8883e00640 1 -- 172.21.15.19:0/1277741730 >> 172.21.15.19:0/1277741730 conn(0x7f88840fee90 msgr2=0x7f8884107df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:18:08.925 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.909+0000 7f8883e00640 1 -- 172.21.15.19:0/1277741730 shutdown_connections 2024-03-18T15:18:08.925 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:08.909+0000 7f8883e00640 1 -- 172.21.15.19:0/1277741730 wait complete. 2024-03-18T15:18:08.927 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-18T15:18:09.146 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.144+0000 7f8c23200640 1 -- 172.21.15.19:0/1717832810 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8c24104af0 msgr2=0x7f8c24104f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:09.146 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.144+0000 7f8c23200640 1 --2- 172.21.15.19:0/1717832810 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8c24104af0 0x7f8c24104f70 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f8c14009900 tx=0x7f8c1402bc30 comp rx=0 tx=0).stop 2024-03-18T15:18:09.146 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.145+0000 7f8c23200640 1 -- 172.21.15.19:0/1717832810 shutdown_connections 2024-03-18T15:18:09.146 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.145+0000 7f8c23200640 1 --2- 172.21.15.19:0/1717832810 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8c241054b0 0x7f8c2410bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.146 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.145+0000 7f8c23200640 1 --2- 172.21.15.19:0/1717832810 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8c24104af0 0x7f8c24104f70 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.147 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.145+0000 7f8c23200640 1 --2- 172.21.15.19:0/1717832810 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8c24103900 0x7f8c24103d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.147 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.145+0000 7f8c23200640 1 -- 172.21.15.19:0/1717832810 >> 172.21.15.19:0/1717832810 conn(0x7f8c240fee90 msgr2=0x7f8c241012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:18:09.147 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.145+0000 7f8c23200640 1 -- 172.21.15.19:0/1717832810 shutdown_connections 2024-03-18T15:18:09.147 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.145+0000 7f8c23200640 1 -- 172.21.15.19:0/1717832810 wait complete. 2024-03-18T15:18:09.149 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.146+0000 7f8c23200640 1 Processor -- start 2024-03-18T15:18:09.149 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.146+0000 7f8c23200640 1 -- start start 2024-03-18T15:18:09.149 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.146+0000 7f8c23200640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8c24103900 0x7f8c24199bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:09.149 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.146+0000 7f8c23200640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8c24104af0 0x7f8c2419a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:09.149 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.146+0000 7f8c23200640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8c241054b0 0x7f8c241a1180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:09.149 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.146+0000 7f8c23200640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f8c2410f5b0 con 0x7f8c24103900 2024-03-18T15:18:09.149 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.146+0000 7f8c23200640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8c2410f430 con 0x7f8c24104af0 2024-03-18T15:18:09.150 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.146+0000 7f8c23200640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f8c2410f730 con 0x7f8c241054b0 2024-03-18T15:18:09.150 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.147+0000 7f8c21e00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8c24103900 0x7f8c24199bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:09.150 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.147+0000 7f8c21400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8c24104af0 0x7f8c2419a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:09.150 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.147+0000 7f8c21e00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8c24103900 0x7f8c24199bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:45814/0 (socket says 172.21.15.19:45814) 2024-03-18T15:18:09.150 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.147+0000 7f8c21e00640 1 -- 172.21.15.19:0/641613969 learned_addr learned my addr 172.21.15.19:0/641613969 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:18:09.150 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.147+0000 7f8c22800640 1 --2- 172.21.15.19:0/641613969 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8c241054b0 0x7f8c241a1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:09.150 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.147+0000 7f8c21e00640 1 -- 172.21.15.19:0/641613969 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8c241054b0 msgr2=0x7f8c241a1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:09.150 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.147+0000 7f8c21e00640 1 --2- 172.21.15.19:0/641613969 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8c241054b0 0x7f8c241a1180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.150 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.147+0000 7f8c21e00640 1 -- 172.21.15.19:0/641613969 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8c24104af0 msgr2=0x7f8c2419a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:09.150 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.147+0000 7f8c21e00640 1 --2- 172.21.15.19:0/641613969 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8c24104af0 0x7f8c2419a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.150 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.147+0000 7f8c21e00640 1 -- 172.21.15.19:0/641613969 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c241a17f0 con 0x7f8c24103900 2024-03-18T15:18:09.150 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.147+0000 7f8c21400640 1 --2- 172.21.15.19:0/641613969 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8c24104af0 0x7f8c2419a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:18:09.150 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.147+0000 7f8c21e00640 1 --2- 172.21.15.19:0/641613969 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8c24103900 0x7f8c24199bd0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f8c0c00e9c0 tx=0x7f8c0c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:18:09.150 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.148+0000 7f8c1aa00640 1 -- 172.21.15.19:0/641613969 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c0c00cd80 con 0x7f8c24103900 2024-03-18T15:18:09.151 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.148+0000 7f8c1aa00640 1 -- 172.21.15.19:0/641613969 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8c0c0035f0 con 0x7f8c24103900 2024-03-18T15:18:09.151 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.148+0000 7f8c23200640 1 -- 172.21.15.19:0/641613969 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c241a1ae0 con 0x7f8c24103900 2024-03-18T15:18:09.151 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.148+0000 7f8c1aa00640 1 -- 172.21.15.19:0/641613969 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c0c010690 con 0x7f8c24103900 2024-03-18T15:18:09.151 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.148+0000 7f8c23200640 1 -- 172.21.15.19:0/641613969 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c241a2000 con 0x7f8c24103900 2024-03-18T15:18:09.153 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.151+0000 7f8c2bdad740 1 Processor -- start 2024-03-18T15:18:09.153 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.151+0000 7f8c2bdad740 1 -- start start 2024-03-18T15:18:09.153 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.151+0000 7f8c1aa00640 1 -- 172.21.15.19:0/641613969 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f8c0c0037a0 con 0x7f8c24103900 2024-03-18T15:18:09.153 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.151+0000 7f8c2bdad740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561f60075340 0x561f60075760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:09.153 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.151+0000 7f8c2bdad740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561f60075d30 0x561f6007a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:09.153 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.151+0000 7f8c2bdad740 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x561f6007aea0 0x561f6007d330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:09.153 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.151+0000 7f8c2bdad740 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x561f5ffd9c50 con 0x561f6007aea0 2024-03-18T15:18:09.153 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.151+0000 7f8c2bdad740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x561f5ff829d0 con 0x561f60075d30 2024-03-18T15:18:09.153 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.151+0000 7f8c2bdad740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x561f5ffd5cf0 con 0x561f60075340 2024-03-18T15:18:09.154 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.152+0000 7f8c21e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561f60075d30 0x561f6007a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:09.154 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.152+0000 7f8c22800640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561f60075340 0x561f60075760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:09.154 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.152+0000 7f8c22800640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561f60075340 0x561f60075760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.19:39402/0 (socket says 172.21.15.19:39402) 2024-03-18T15:18:09.154 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.152+0000 7f8c21e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561f60075d30 0x561f6007a830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:46350/0 (socket says 172.21.15.19:46350) 2024-03-18T15:18:09.154 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.152+0000 7f8c22800640 1 -- 172.21.15.19:0/981233788 learned_addr learned my addr 172.21.15.19:0/981233788 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:18:09.154 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.152+0000 7f8c21400640 1 --2- 172.21.15.19:0/981233788 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x561f6007aea0 0x561f6007d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:09.154 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.152+0000 7f8c21e00640 1 -- 172.21.15.19:0/981233788 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561f60075340 msgr2=0x561f60075760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:18:09.154 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.152+0000 7f8c21e00640 1 --2- 172.21.15.19:0/981233788 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561f60075340 0x561f60075760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.154 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.152+0000 7f8c21e00640 1 -- 172.21.15.19:0/981233788 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x561f6007aea0 msgr2=0x561f6007d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:18:09.154 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.152+0000 7f8c21e00640 1 --2- 172.21.15.19:0/981233788 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x561f6007aea0 0x561f6007d330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.154 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.152+0000 7f8c21e00640 1 -- 172.21.15.19:0/981233788 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f5ff9cf10 con 0x561f60075d30 2024-03-18T15:18:09.154 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.152+0000 7f8c22800640 1 --2- 172.21.15.19:0/981233788 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561f60075340 0x561f60075760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:18:09.160 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.153+0000 7f8c21e00640 1 --2- 172.21.15.19:0/981233788 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561f60075d30 0x561f6007a830 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f8c0c0cc890 tx=0x7f8c0c0d7670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ca84892d5c854fe0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:18:09.160 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.153+0000 7f8c1aa00640 1 --2- 172.21.15.19:0/641613969 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f8c080a2800 0x7f8c080a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:09.161 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.153+0000 7f8c13400640 1 -- 172.21.15.19:0/981233788 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c0c0e2070 con 0x561f60075d30 2024-03-18T15:18:09.161 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.153+0000 7f8c13400640 1 -- 172.21.15.19:0/981233788 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8c0c0cfaf0 con 0x561f60075d30 2024-03-18T15:18:09.161 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.153+0000 7f8c13400640 1 -- 172.21.15.19:0/981233788 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c0c097860 con 0x561f60075d30 2024-03-18T15:18:09.161 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.153+0000 7f8c1aa00640 1 -- 172.21.15.19:0/641613969 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f8c0c0cf220 con 0x7f8c24103900 2024-03-18T15:18:09.161 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.153+0000 7f8c22800640 1 --2- 172.21.15.19:0/641613969 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f8c080a2800 0x7f8c080a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:09.161 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.154+0000 7f8c2bdad740 1 -- 172.21.15.19:0/981233788 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561f60075d30 msgr2=0x561f6007a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:09.161 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.154+0000 7f8c2bdad740 1 --2- 172.21.15.19:0/981233788 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561f60075d30 0x561f6007a830 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f8c0c0cc890 tx=0x7f8c0c0d7670 comp rx=0 tx=0).stop 2024-03-18T15:18:09.161 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.154+0000 7f8c2bdad740 1 -- 172.21.15.19:0/981233788 shutdown_connections 2024-03-18T15:18:09.161 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.154+0000 7f8c2bdad740 1 --2- 172.21.15.19:0/981233788 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x561f6007aea0 0x561f6007d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.161 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.154+0000 7f8c2bdad740 1 --2- 172.21.15.19:0/981233788 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561f60075d30 0x561f6007a830 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.162 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.154+0000 7f8c2bdad740 1 --2- 172.21.15.19:0/981233788 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561f60075340 0x561f60075760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.162 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.154+0000 7f8c2bdad740 1 -- 172.21.15.19:0/981233788 >> 172.21.15.19:0/981233788 conn(0x561f600708b0 msgr2=0x561f60070cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:18:09.162 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.154+0000 7f8c2bdad740 1 -- 172.21.15.19:0/981233788 shutdown_connections 2024-03-18T15:18:09.162 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.154+0000 7f8c22800640 1 --2- 172.21.15.19:0/641613969 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f8c080a2800 0x7f8c080a4cc0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x561f60076590 tx=0x7f8c1c009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:18:09.162 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.154+0000 7f8c2bdad740 1 -- 172.21.15.19:0/981233788 wait complete. 2024-03-18T15:18:09.162 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.154+0000 7f8c2bdad740 1 -- start start 2024-03-18T15:18:09.162 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.155+0000 7f8c2bdad740 1 Processor -- start 2024-03-18T15:18:09.162 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.155+0000 7f8c2bdad740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561f60075340 0x561f6007f490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:09.162 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.155+0000 7f8c2bdad740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561f60075d30 0x561f6007f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:09.162 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.155+0000 7f8c2bdad740 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x561f6007aea0 0x561f60086a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:09.162 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.155+0000 7f8c2bdad740 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x561f6007de50 con 0x561f6007aea0 2024-03-18T15:18:09.162 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.155+0000 7f8c2bdad740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x561f5ff95e20 con 0x561f60075d30 2024-03-18T15:18:09.163 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.155+0000 7f8c2bdad740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x561f6007dfd0 con 0x561f60075340 2024-03-18T15:18:09.163 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.155+0000 7f8c2bdad740 10 client.0 _ll_register_callbacks cb 0x561f5ff957f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T15:18:09.163 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.155+0000 7f8c21e00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561f60075340 0x561f6007f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:09.163 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.155+0000 7f8c21e00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561f60075340 0x561f6007f490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.19:39416/0 (socket says 172.21.15.19:39416) 2024-03-18T15:18:09.163 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.155+0000 7f8c21e00640 1 -- 172.21.15.19:0/566027230 learned_addr learned my addr 172.21.15.19:0/566027230 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:18:09.163 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.155+0000 7f8c21400640 1 --2- 172.21.15.19:0/566027230 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561f60075d30 0x561f6007f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:09.163 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.156+0000 7f8c22800640 1 --2- 172.21.15.19:0/566027230 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x561f6007aea0 0x561f60086a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:09.163 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.156+0000 7f8c21e00640 1 -- 172.21.15.19:0/566027230 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561f60075d30 msgr2=0x561f6007f9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:18:09.164 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.156+0000 7f8c21e00640 1 --2- 172.21.15.19:0/566027230 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561f60075d30 0x561f6007f9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.164 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.156+0000 7f8c21e00640 1 -- 172.21.15.19:0/566027230 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x561f6007aea0 msgr2=0x561f60086a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:18:09.164 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.156+0000 7f8c21e00640 1 --2- 172.21.15.19:0/566027230 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x561f6007aea0 0x561f60086a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.164 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.156+0000 7f8c21e00640 1 -- 172.21.15.19:0/566027230 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561f5fcee2d0 con 0x561f60075340 2024-03-18T15:18:09.164 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.156+0000 7f8c21400640 1 --2- 172.21.15.19:0/566027230 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561f60075d30 0x561f6007f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T15:18:09.164 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.156+0000 7f8c21e00640 1 --2- 172.21.15.19:0/566027230 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561f60075340 0x561f6007f490 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f8c0c0a3a00 tx=0x7f8c0c0097b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fde706cc1d467331 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:18:09.164 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.156+0000 7f8c07e00640 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-03-18T15:18:09.164 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.156+0000 7f8c07e00640 1 -- 172.21.15.19:0/566027230 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c0c09af10 con 0x561f60075340 2024-03-18T15:18:09.164 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.156+0000 7f8c07e00640 1 -- 172.21.15.19:0/566027230 <== mon.1 v2:172.21.15.104:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f8c0c0cca00 con 0x561f60075340 2024-03-18T15:18:09.164 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.156+0000 7f8c2bdad740 1 -- 172.21.15.19:0/566027230 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8c0c0cfaf0 con 0x561f60075340 2024-03-18T15:18:09.165 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.156+0000 7f8c07e00640 1 -- 172.21.15.19:0/566027230 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f8c0c09b0f0 con 0x561f60075340 2024-03-18T15:18:09.165 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.156+0000 7f8c2bdad740 10 client.25393 fetch_fsmap learned FSMap version 26 2024-03-18T15:18:09.165 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.156+0000 7f8c2bdad740 1 -- 172.21.15.19:0/566027230 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x561f5fdb86b0 con 0x561f60075340 2024-03-18T15:18:09.165 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.157+0000 7f8c07e00640 1 -- 172.21.15.19:0/566027230 <== mon.1 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8c0c0cfaf0 con 0x561f60075340 2024-03-18T15:18:09.165 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.157+0000 7f8c07e00640 1 -- 172.21.15.19:0/566027230 <== mon.1 v2:172.21.15.104:3300/0 5 ==== fsmap(e 26) ==== 8362+0+0 (secure 0 0 0) 0x7f8c0c09abe0 con 0x561f60075340 2024-03-18T15:18:09.165 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.157+0000 7f8c2bdad740 10 client.25393 fetch_fsmap finished waiting for FSMap version 26 2024-03-18T15:18:09.165 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.157+0000 7f8c2bdad740 10 client.25393 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-18T15:18:09.165 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.157+0000 7f8c2bdad740 20 client.25393 populate_metadata read hostname 'smithi019' 2024-03-18T15:18:09.165 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.157+0000 7f8c2bdad740 1 --2- 172.21.15.19:0/566027230 >> [v2:172.21.15.104:6836/260984674,v1:172.21.15.104:6837/260984674] conn(0x561f6008af00 0x561f6008d060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:09.165 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.157+0000 7f8c2bdad740 4 client.25393 mds_command: new command op to 24511 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T15:18:09.165 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.157+0000 7f8c2bdad740 1 -- 172.21.15.19:0/566027230 --> [v2:172.21.15.104:6836/260984674,v1:172.21.15.104:6837/260984674] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561f600372b0 con 0x561f6008af00 2024-03-18T15:18:09.165 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.157+0000 7f8c21400640 1 --2- 172.21.15.19:0/566027230 >> [v2:172.21.15.104:6836/260984674,v1:172.21.15.104:6837/260984674] conn(0x561f6008af00 0x561f6008d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:09.165 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.158+0000 7f8c21400640 1 --2- 172.21.15.19:0/566027230 >> [v2:172.21.15.104:6836/260984674,v1:172.21.15.104:6837/260984674] conn(0x561f6008af00 0x561f6008d060 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e033d759563282d6 server_cookie=3ba57ddded3fbca7 in_seq=0 out_seq=0 2024-03-18T15:18:09.166 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.158+0000 7f8c07e00640 10 client.25393 ms_handle_connect on v2:172.21.15.104:6836/260984674 2024-03-18T15:18:09.166 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.159+0000 7f8c07e00640 1 -- 172.21.15.19:0/566027230 <== mds.2 v2:172.21.15.104:6836/260984674 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561f600372b0 con 0x561f6008af00 2024-03-18T15:18:09.166 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.159+0000 7f8c07e00640 10 client.25393 handle_command_reply: tid=0 2024-03-18T15:18:09.166 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.159+0000 7f8c2bdad740 1 client.25393 shutdown 2024-03-18T15:18:09.166 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.159+0000 7f8c2bdad740 1 -- 172.21.15.19:0/566027230 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561f60075340 msgr2=0x561f6007f490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:09.166 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.159+0000 7f8c2bdad740 1 --2- 172.21.15.19:0/566027230 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561f60075340 0x561f6007f490 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f8c0c0a3a00 tx=0x7f8c0c0097b0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.166 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.159+0000 7f8c2bdad740 1 -- 172.21.15.19:0/566027230 shutdown_connections 2024-03-18T15:18:09.166 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.159+0000 7f8c2bdad740 1 --2- 172.21.15.19:0/566027230 >> [v2:172.21.15.104:6836/260984674,v1:172.21.15.104:6837/260984674] conn(0x561f6008af00 0x561f6008d060 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.166 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.159+0000 7f8c2bdad740 1 --2- 172.21.15.19:0/566027230 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x561f6007aea0 0x561f60086a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.166 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.159+0000 7f8c07e00640 0 client.25393 ms_handle_reset on v2:172.21.15.104:6836/260984674 2024-03-18T15:18:09.166 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.159+0000 7f8c2bdad740 1 --2- 172.21.15.19:0/566027230 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561f60075d30 0x561f6007f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.166 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.159+0000 7f8c2bdad740 1 --2- 172.21.15.19:0/566027230 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561f60075340 0x561f6007f490 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.166 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.159+0000 7f8c2bdad740 1 -- 172.21.15.19:0/566027230 >> 172.21.15.19:0/566027230 conn(0x561f600708b0 msgr2=0x561f6006fa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:18:09.166 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.159+0000 7f8c2bdad740 1 -- 172.21.15.19:0/566027230 shutdown_connections 2024-03-18T15:18:09.167 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.160+0000 7f8c2bdad740 1 -- 172.21.15.19:0/566027230 wait complete. 2024-03-18T15:18:09.167 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.160+0000 7f8c2bdad740 20 client.25393 trim_cache size 0 max 16384 2024-03-18T15:18:09.181 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.180+0000 7f8c2bdad740 1 Processor -- start 2024-03-18T15:18:09.182 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.180+0000 7f8c2bdad740 1 -- start start 2024-03-18T15:18:09.182 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.180+0000 7f8c2bdad740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561f60075340 0x561f6007e760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:09.182 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.180+0000 7f8c2bdad740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561f60075d30 0x561f600869d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:09.182 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.180+0000 7f8c2bdad740 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x561f6007aea0 0x561f60070c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:09.182 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.180+0000 7f8c2bdad740 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x561f6008b3a0 con 0x561f6007aea0 2024-03-18T15:18:09.182 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.180+0000 7f8c2bdad740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x561f6008ba70 con 0x561f60075340 2024-03-18T15:18:09.182 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.180+0000 7f8c2bdad740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x561f6008c140 con 0x561f60075d30 2024-03-18T15:18:09.182 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.180+0000 7f8c21e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561f60075340 0x561f6007e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:09.182 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.180+0000 7f8c21400640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561f60075d30 0x561f600869d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:09.182 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.180+0000 7f8c21e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561f60075340 0x561f6007e760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:46376/0 (socket says 172.21.15.19:46376) 2024-03-18T15:18:09.183 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.180+0000 7f8c21e00640 1 -- 172.21.15.19:0/2493727477 learned_addr learned my addr 172.21.15.19:0/2493727477 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:18:09.183 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.181+0000 7f8c22800640 1 --2- 172.21.15.19:0/2493727477 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x561f6007aea0 0x561f60070c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:09.183 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.181+0000 7f8c21e00640 1 -- 172.21.15.19:0/2493727477 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561f60075d30 msgr2=0x561f600869d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:18:09.183 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.181+0000 7f8c21e00640 1 --2- 172.21.15.19:0/2493727477 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561f60075d30 0x561f600869d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.183 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.181+0000 7f8c21e00640 1 -- 172.21.15.19:0/2493727477 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x561f6007aea0 msgr2=0x561f60070c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:18:09.183 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.181+0000 7f8c21e00640 1 --2- 172.21.15.19:0/2493727477 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x561f6007aea0 0x561f60070c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.183 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.181+0000 7f8c21e00640 1 -- 172.21.15.19:0/2493727477 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f6007a0e0 con 0x561f60075340 2024-03-18T15:18:09.183 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.181+0000 7f8c21400640 1 --2- 172.21.15.19:0/2493727477 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561f60075d30 0x561f600869d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:18:09.186 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.181+0000 7f8c21e00640 1 --2- 172.21.15.19:0/2493727477 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561f60075340 0x561f6007e760 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f8c0c09b490 tx=0x7f8c0c00c7d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f0d04a55b243065d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:18:09.186 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.181+0000 7f8c12a00640 1 -- 172.21.15.19:0/2493727477 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c0c09b2d0 con 0x561f60075340 2024-03-18T15:18:09.187 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.181+0000 7f8c12a00640 1 -- 172.21.15.19:0/2493727477 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8c0c0d7e70 con 0x561f60075340 2024-03-18T15:18:09.187 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.181+0000 7f8c12a00640 1 -- 172.21.15.19:0/2493727477 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c0c09b560 con 0x561f60075340 2024-03-18T15:18:09.187 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.182+0000 7f8c2bdad740 1 -- 172.21.15.19:0/2493727477 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561f60075340 msgr2=0x561f6007e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:09.187 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.182+0000 7f8c2bdad740 1 --2- 172.21.15.19:0/2493727477 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561f60075340 0x561f6007e760 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f8c0c09b490 tx=0x7f8c0c00c7d0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.187 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.182+0000 7f8c2bdad740 1 -- 172.21.15.19:0/2493727477 shutdown_connections 2024-03-18T15:18:09.187 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.182+0000 7f8c2bdad740 1 --2- 172.21.15.19:0/2493727477 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x561f6007aea0 0x561f60070c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.187 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.182+0000 7f8c2bdad740 1 --2- 172.21.15.19:0/2493727477 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561f60075d30 0x561f600869d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.187 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.182+0000 7f8c2bdad740 1 --2- 172.21.15.19:0/2493727477 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561f60075340 0x561f6007e760 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.187 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.182+0000 7f8c2bdad740 1 -- 172.21.15.19:0/2493727477 >> 172.21.15.19:0/2493727477 conn(0x561f600708b0 msgr2=0x561f6007f0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:18:09.187 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.182+0000 7f8c2bdad740 1 -- 172.21.15.19:0/2493727477 shutdown_connections 2024-03-18T15:18:09.187 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.182+0000 7f8c2bdad740 1 -- 172.21.15.19:0/2493727477 wait complete. 2024-03-18T15:18:09.187 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.183+0000 7f8c2bdad740 1 -- start start 2024-03-18T15:18:09.187 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.183+0000 7f8c2bdad740 1 Processor -- start 2024-03-18T15:18:09.188 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.183+0000 7f8c2bdad740 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x561f60075340 0x561f600826d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:09.188 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.183+0000 7f8c2bdad740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561f60075d30 0x561f60082c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:09.188 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.183+0000 7f8c2bdad740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561f6007aea0 0x561f60085bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:09.188 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.183+0000 7f8c2bdad740 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x561f6008cee0 con 0x561f60075340 2024-03-18T15:18:09.188 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.183+0000 7f8c2bdad740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x561f6008c810 con 0x561f6007aea0 2024-03-18T15:18:09.188 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.183+0000 7f8c2bdad740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x561f6008d5b0 con 0x561f60075d30 2024-03-18T15:18:09.188 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.183+0000 7f8c2bdad740 10 client.0 _ll_register_callbacks cb 0x7f8c1c009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T15:18:09.188 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.183+0000 7f8c21400640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561f60075d30 0x561f60082c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:09.188 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.183+0000 7f8c22800640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561f6007aea0 0x561f60085bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:09.188 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.183+0000 7f8c21e00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x561f60075340 0x561f600826d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:09.188 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.183+0000 7f8c21400640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561f60075d30 0x561f60082c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.19:39438/0 (socket says 172.21.15.19:39438) 2024-03-18T15:18:09.188 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.183+0000 7f8c21400640 1 -- 172.21.15.19:0/3083690007 learned_addr learned my addr 172.21.15.19:0/3083690007 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:18:09.188 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.184+0000 7f8c21400640 1 -- 172.21.15.19:0/3083690007 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561f6007aea0 msgr2=0x561f60085bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:18:09.188 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.184+0000 7f8c21400640 1 --2- 172.21.15.19:0/3083690007 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561f6007aea0 0x561f60085bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.189 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.184+0000 7f8c21400640 1 -- 172.21.15.19:0/3083690007 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x561f60075340 msgr2=0x561f600826d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:18:09.189 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.184+0000 7f8c21400640 1 --2- 172.21.15.19:0/3083690007 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x561f60075340 0x561f600826d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.189 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.184+0000 7f8c21400640 1 -- 172.21.15.19:0/3083690007 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561f600861b0 con 0x561f60075d30 2024-03-18T15:18:09.189 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.184+0000 7f8c21400640 1 --2- 172.21.15.19:0/3083690007 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561f60075d30 0x561f60082c10 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f8c1402c7b0 tx=0x7f8c1404cd60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=95f3d2ca9d5af364 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:18:09.189 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.184+0000 7f8c07e00640 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-03-18T15:18:09.189 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.184+0000 7f8c07e00640 1 -- 172.21.15.19:0/3083690007 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c14053070 con 0x561f60075d30 2024-03-18T15:18:09.189 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.184+0000 7f8c2bdad740 1 -- 172.21.15.19:0/3083690007 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8c0c0d7e70 con 0x561f60075d30 2024-03-18T15:18:09.189 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.184+0000 7f8c07e00640 1 -- 172.21.15.19:0/3083690007 <== mon.1 v2:172.21.15.104:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f8c140040c0 con 0x561f60075d30 2024-03-18T15:18:09.189 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.185+0000 7f8c07e00640 1 -- 172.21.15.19:0/3083690007 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f8c1405e050 con 0x561f60075d30 2024-03-18T15:18:09.189 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.185+0000 7f8c2bdad740 10 client.25399 fetch_fsmap learned FSMap version 26 2024-03-18T15:18:09.189 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.185+0000 7f8c2bdad740 1 -- 172.21.15.19:0/3083690007 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x561f600f1f50 con 0x561f60075d30 2024-03-18T15:18:09.189 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.185+0000 7f8c07e00640 1 -- 172.21.15.19:0/3083690007 <== mon.1 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8c0c0d7e70 con 0x561f60075d30 2024-03-18T15:18:09.189 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.185+0000 7f8c07e00640 1 -- 172.21.15.19:0/3083690007 <== mon.1 v2:172.21.15.104:3300/0 5 ==== fsmap(e 26) ==== 8362+0+0 (secure 0 0 0) 0x7f8c140051d0 con 0x561f60075d30 2024-03-18T15:18:09.189 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.185+0000 7f8c2bdad740 10 client.25399 fetch_fsmap finished waiting for FSMap version 26 2024-03-18T15:18:09.190 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.185+0000 7f8c2bdad740 10 client.25399 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-18T15:18:09.190 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.185+0000 7f8c2bdad740 20 client.25399 populate_metadata read hostname 'smithi019' 2024-03-18T15:18:09.190 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.185+0000 7f8c2bdad740 1 --2- 172.21.15.19:0/3083690007 >> [v2:172.21.15.104:6836/260984674,v1:172.21.15.104:6837/260984674] conn(0x561f600f6b60 0x561f600f5870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:09.190 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.185+0000 7f8c2bdad740 4 client.25399 mds_command: new command op to 24511 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-18T15:18:09.190 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.185+0000 7f8c2bdad740 1 -- 172.21.15.19:0/3083690007 --> [v2:172.21.15.104:6836/260984674,v1:172.21.15.104:6837/260984674] -- command(tid 0: {"prefix": "status"}) -- 0x561f5ffe7190 con 0x561f600f6b60 2024-03-18T15:18:09.190 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.185+0000 7f8c21e00640 1 --2- 172.21.15.19:0/3083690007 >> [v2:172.21.15.104:6836/260984674,v1:172.21.15.104:6837/260984674] conn(0x561f600f6b60 0x561f600f5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:09.190 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.186+0000 7f8c21e00640 1 --2- 172.21.15.19:0/3083690007 >> [v2:172.21.15.104:6836/260984674,v1:172.21.15.104:6837/260984674] conn(0x561f600f6b60 0x561f600f5870 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=68e80ab79090b365 server_cookie=47543b91b8d5fc52 in_seq=0 out_seq=0 2024-03-18T15:18:09.190 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.186+0000 7f8c07e00640 10 client.25399 ms_handle_connect on v2:172.21.15.104:6836/260984674 2024-03-18T15:18:09.191 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.186+0000 7f8c07e00640 1 -- 172.21.15.19:0/3083690007 <== mds.2 v2:172.21.15.104:6836/260984674 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x561f5ffe7190 con 0x561f600f6b60 2024-03-18T15:18:09.191 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.186+0000 7f8c07e00640 10 client.25399 handle_command_reply: tid=0 2024-03-18T15:18:09.191 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.186+0000 7f8c2bdad740 1 client.25399 shutdown 2024-03-18T15:18:09.192 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.186+0000 7f8c2bdad740 1 -- 172.21.15.19:0/3083690007 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561f60075d30 msgr2=0x561f60082c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:09.192 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.186+0000 7f8c2bdad740 1 --2- 172.21.15.19:0/3083690007 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561f60075d30 0x561f60082c10 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f8c1402c7b0 tx=0x7f8c1404cd60 comp rx=0 tx=0).stop 2024-03-18T15:18:09.192 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.186+0000 7f8c2bdad740 1 -- 172.21.15.19:0/3083690007 shutdown_connections 2024-03-18T15:18:09.192 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.186+0000 7f8c2bdad740 1 --2- 172.21.15.19:0/3083690007 >> [v2:172.21.15.104:6836/260984674,v1:172.21.15.104:6837/260984674] conn(0x561f600f6b60 0x561f600f5870 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.192 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.186+0000 7f8c2bdad740 1 --2- 172.21.15.19:0/3083690007 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561f6007aea0 0x561f60085bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.192 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.186+0000 7f8c2bdad740 1 --2- 172.21.15.19:0/3083690007 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561f60075d30 0x561f60082c10 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.192 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.186+0000 7f8c2bdad740 1 --2- 172.21.15.19:0/3083690007 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x561f60075340 0x561f600826d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.192 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.186+0000 7f8c2bdad740 1 -- 172.21.15.19:0/3083690007 >> 172.21.15.19:0/3083690007 conn(0x561f600708b0 msgr2=0x561f6006f1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:18:09.192 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.186+0000 7f8c07e00640 0 client.25399 ms_handle_reset on v2:172.21.15.104:6836/260984674 2024-03-18T15:18:09.192 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.187+0000 7f8c2bdad740 1 -- 172.21.15.19:0/3083690007 shutdown_connections 2024-03-18T15:18:09.192 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.187+0000 7f8c2bdad740 1 -- 172.21.15.19:0/3083690007 wait complete. 2024-03-18T15:18:09.192 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.187+0000 7f8c2bdad740 20 client.25399 trim_cache size 0 max 16384 2024-03-18T15:18:09.193 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-03-18T15:18:09.193 INFO:teuthology.orchestra.run.smithi019.stdout: "cluster_fsid": "d4dc2ba6-e538-11ee-95c9-87774f69a715", 2024-03-18T15:18:09.193 INFO:teuthology.orchestra.run.smithi019.stdout: "whoami": 2, 2024-03-18T15:18:09.193 INFO:teuthology.orchestra.run.smithi019.stdout: "id": 24511, 2024-03-18T15:18:09.193 INFO:teuthology.orchestra.run.smithi019.stdout: "want_state": "up:active", 2024-03-18T15:18:09.193 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "up:active", 2024-03-18T15:18:09.193 INFO:teuthology.orchestra.run.smithi019.stdout: "fs_name": "cephfs", 2024-03-18T15:18:09.193 INFO:teuthology.orchestra.run.smithi019.stdout: "rank_uptime": 144.71489536799999, 2024-03-18T15:18:09.193 INFO:teuthology.orchestra.run.smithi019.stdout: "mdsmap_epoch": 26, 2024-03-18T15:18:09.193 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap_epoch": 93, 2024-03-18T15:18:09.194 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap_epoch_barrier": 89, 2024-03-18T15:18:09.194 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 202.009032561 2024-03-18T15:18:09.194 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-03-18T15:18:09.195 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.187+0000 7f8c23200640 1 -- 172.21.15.19:0/641613969 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f8c080a2800 msgr2=0x7f8c080a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:09.195 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.187+0000 7f8c23200640 1 --2- 172.21.15.19:0/641613969 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f8c080a2800 0x7f8c080a4cc0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x561f60076590 tx=0x7f8c1c009500 comp rx=0 tx=0).stop 2024-03-18T15:18:09.195 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.187+0000 7f8c23200640 1 -- 172.21.15.19:0/641613969 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8c24103900 msgr2=0x7f8c24199bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:09.195 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.187+0000 7f8c23200640 1 --2- 172.21.15.19:0/641613969 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8c24103900 0x7f8c24199bd0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f8c0c00e9c0 tx=0x7f8c0c00ee80 comp rx=0 tx=0).stop 2024-03-18T15:18:09.195 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.187+0000 7f8c23200640 1 -- 172.21.15.19:0/641613969 shutdown_connections 2024-03-18T15:18:09.195 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.187+0000 7f8c23200640 1 --2- 172.21.15.19:0/641613969 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8c241054b0 0x7f8c241a1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.195 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.187+0000 7f8c23200640 1 --2- 172.21.15.19:0/641613969 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f8c080a2800 0x7f8c080a4cc0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.195 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.187+0000 7f8c23200640 1 --2- 172.21.15.19:0/641613969 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8c24104af0 0x7f8c2419a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.195 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.187+0000 7f8c23200640 1 --2- 172.21.15.19:0/641613969 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8c24103900 0x7f8c24199bd0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.195 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.187+0000 7f8c23200640 1 -- 172.21.15.19:0/641613969 >> 172.21.15.19:0/641613969 conn(0x7f8c240fee90 msgr2=0x7f8c24107d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:18:09.195 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.188+0000 7f8c23200640 1 -- 172.21.15.19:0/641613969 shutdown_connections 2024-03-18T15:18:09.196 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.188+0000 7f8c23200640 1 -- 172.21.15.19:0/641613969 wait complete. 2024-03-18T15:18:09.203 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:09 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/1609730460' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:18:09.205 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-03-18T15:18:09.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.423+0000 7fd177e00640 1 -- 172.21.15.19:0/939418574 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd178102140 msgr2=0x7fd178104530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:09.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.423+0000 7fd177e00640 1 --2- 172.21.15.19:0/939418574 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd178102140 0x7fd178104530 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fd168009ca0 tx=0x7fd16802c100 comp rx=0 tx=0).stop 2024-03-18T15:18:09.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.423+0000 7fd177e00640 1 -- 172.21.15.19:0/939418574 shutdown_connections 2024-03-18T15:18:09.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.423+0000 7fd177e00640 1 --2- 172.21.15.19:0/939418574 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd1781073a0 0x7fd1781097f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.423+0000 7fd177e00640 1 --2- 172.21.15.19:0/939418574 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd178104a70 0x7fd178106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.423+0000 7fd177e00640 1 --2- 172.21.15.19:0/939418574 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd178102140 0x7fd178104530 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.423+0000 7fd177e00640 1 -- 172.21.15.19:0/939418574 >> 172.21.15.19:0/939418574 conn(0x7fd1780fbb50 msgr2=0x7fd1780fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:18:09.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.424+0000 7fd177e00640 1 -- 172.21.15.19:0/939418574 shutdown_connections 2024-03-18T15:18:09.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.424+0000 7fd177e00640 1 -- 172.21.15.19:0/939418574 wait complete. 2024-03-18T15:18:09.426 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.425+0000 7fd177e00640 1 Processor -- start 2024-03-18T15:18:09.426 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.425+0000 7fd177e00640 1 -- start start 2024-03-18T15:18:09.428 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.426+0000 7fd177e00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd178102140 0x7fd178199c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:09.428 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.426+0000 7fd177e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd178104a70 0x7fd17819a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:09.428 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.426+0000 7fd177e00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd1781073a0 0x7fd1781a11b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:09.428 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.426+0000 7fd177e00640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fd17810d350 con 0x7fd1781073a0 2024-03-18T15:18:09.428 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.426+0000 7fd177e00640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd17810d1d0 con 0x7fd178104a70 2024-03-18T15:18:09.428 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.426+0000 7fd177e00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fd17810d4d0 con 0x7fd178102140 2024-03-18T15:18:09.428 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.426+0000 7fd176a00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd178102140 0x7fd178199c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:09.429 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.426+0000 7fd176000640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd178104a70 0x7fd17819a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:09.429 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.426+0000 7fd176a00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd178102140 0x7fd178199c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.19:39460/0 (socket says 172.21.15.19:39460) 2024-03-18T15:18:09.429 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.426+0000 7fd176a00640 1 -- 172.21.15.19:0/4282149858 learned_addr learned my addr 172.21.15.19:0/4282149858 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:18:09.429 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.426+0000 7fd177400640 1 --2- 172.21.15.19:0/4282149858 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd1781073a0 0x7fd1781a11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:09.429 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.426+0000 7fd176a00640 1 -- 172.21.15.19:0/4282149858 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd178104a70 msgr2=0x7fd17819a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:09.429 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.426+0000 7fd176a00640 1 --2- 172.21.15.19:0/4282149858 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd178104a70 0x7fd17819a180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.429 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.426+0000 7fd176a00640 1 -- 172.21.15.19:0/4282149858 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd1781073a0 msgr2=0x7fd1781a11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:09.429 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.426+0000 7fd176a00640 1 --2- 172.21.15.19:0/4282149858 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd1781073a0 0x7fd1781a11b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.429 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.426+0000 7fd176a00640 1 -- 172.21.15.19:0/4282149858 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1781a1850 con 0x7fd178102140 2024-03-18T15:18:09.429 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.427+0000 7fd176a00640 1 --2- 172.21.15.19:0/4282149858 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd178102140 0x7fd178199c40 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fd1680325f0 tx=0x7fd168035d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:18:09.429 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.427+0000 7fd17ca00640 1 -- 172.21.15.19:0/4282149858 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd168039040 con 0x7fd178102140 2024-03-18T15:18:09.430 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.427+0000 7fd17ca00640 1 -- 172.21.15.19:0/4282149858 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd168044070 con 0x7fd178102140 2024-03-18T15:18:09.430 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.427+0000 7fd17ca00640 1 -- 172.21.15.19:0/4282149858 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd168004540 con 0x7fd178102140 2024-03-18T15:18:09.430 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.427+0000 7fd177e00640 1 -- 172.21.15.19:0/4282149858 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1781a1ae0 con 0x7fd178102140 2024-03-18T15:18:09.430 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.427+0000 7fd177e00640 1 -- 172.21.15.19:0/4282149858 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1781a1fa0 con 0x7fd178102140 2024-03-18T15:18:09.433 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.430+0000 7fd17ca00640 1 -- 172.21.15.19:0/4282149858 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fd168032890 con 0x7fd178102140 2024-03-18T15:18:09.433 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.431+0000 7fd180c18740 1 Processor -- start 2024-03-18T15:18:09.433 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.431+0000 7fd180c18740 1 -- start start 2024-03-18T15:18:09.434 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.431+0000 7fd180c18740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55931bd4a340 0x55931bd4a760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:09.434 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.431+0000 7fd180c18740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55931bd4ad30 0x55931bd4f830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:09.434 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.431+0000 7fd180c18740 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55931bd4fea0 0x55931bd52330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:09.434 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.431+0000 7fd180c18740 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x55931bcaec50 con 0x55931bd4fea0 2024-03-18T15:18:09.434 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.431+0000 7fd180c18740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55931bc579d0 con 0x55931bd4a340 2024-03-18T15:18:09.434 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.431+0000 7fd180c18740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55931bcaacf0 con 0x55931bd4ad30 2024-03-18T15:18:09.434 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.431+0000 7fd177400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55931bd4a340 0x55931bd4a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:09.434 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.431+0000 7fd177400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55931bd4a340 0x55931bd4a760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:46398/0 (socket says 172.21.15.19:46398) 2024-03-18T15:18:09.434 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.431+0000 7fd177400640 1 -- 172.21.15.19:0/2222215736 learned_addr learned my addr 172.21.15.19:0/2222215736 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:18:09.434 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.431+0000 7fd176a00640 1 --2- 172.21.15.19:0/2222215736 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55931bd4ad30 0x55931bd4f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:09.434 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.431+0000 7fd176000640 1 --2- 172.21.15.19:0/2222215736 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55931bd4fea0 0x55931bd52330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:09.434 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.432+0000 7fd177400640 1 -- 172.21.15.19:0/2222215736 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55931bd4ad30 msgr2=0x55931bd4f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:18:09.434 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.432+0000 7fd177400640 1 --2- 172.21.15.19:0/2222215736 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55931bd4ad30 0x55931bd4f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.435 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.432+0000 7fd177400640 1 -- 172.21.15.19:0/2222215736 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55931bd4fea0 msgr2=0x55931bd52330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:18:09.435 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.432+0000 7fd177400640 1 --2- 172.21.15.19:0/2222215736 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55931bd4fea0 0x55931bd52330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.435 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.432+0000 7fd177400640 1 -- 172.21.15.19:0/2222215736 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55931bc71f10 con 0x55931bd4a340 2024-03-18T15:18:09.435 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.432+0000 7fd176000640 1 --2- 172.21.15.19:0/2222215736 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55931bd4fea0 0x55931bd52330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T15:18:09.435 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.432+0000 7fd177400640 1 --2- 172.21.15.19:0/2222215736 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55931bd4a340 0x55931bd4a760 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fd170009b90 tx=0x7fd1700117b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3ad1e895eab2bcfe server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:18:09.435 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.432+0000 7fd17ca00640 1 --2- 172.21.15.19:0/4282149858 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fd15c0a2800 0x7fd15c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:09.435 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.432+0000 7fd167e00640 1 -- 172.21.15.19:0/2222215736 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd170012450 con 0x55931bd4a340 2024-03-18T15:18:09.435 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.432+0000 7fd167e00640 1 -- 172.21.15.19:0/2222215736 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd170012ad0 con 0x55931bd4a340 2024-03-18T15:18:09.435 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.432+0000 7fd167e00640 1 -- 172.21.15.19:0/2222215736 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd17000fab0 con 0x55931bd4a340 2024-03-18T15:18:09.435 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.432+0000 7fd17ca00640 1 -- 172.21.15.19:0/4282149858 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fd1680f15c0 con 0x7fd178102140 2024-03-18T15:18:09.435 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.433+0000 7fd176a00640 1 --2- 172.21.15.19:0/4282149858 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fd15c0a2800 0x7fd15c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:09.440 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.433+0000 7fd180c18740 1 -- 172.21.15.19:0/2222215736 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55931bd4a340 msgr2=0x55931bd4a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:09.440 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.433+0000 7fd180c18740 1 --2- 172.21.15.19:0/2222215736 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55931bd4a340 0x55931bd4a760 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fd170009b90 tx=0x7fd1700117b0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.440 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.433+0000 7fd180c18740 1 -- 172.21.15.19:0/2222215736 shutdown_connections 2024-03-18T15:18:09.440 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.433+0000 7fd180c18740 1 --2- 172.21.15.19:0/2222215736 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55931bd4fea0 0x55931bd52330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.440 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.433+0000 7fd180c18740 1 --2- 172.21.15.19:0/2222215736 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55931bd4ad30 0x55931bd4f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.440 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.433+0000 7fd180c18740 1 --2- 172.21.15.19:0/2222215736 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55931bd4a340 0x55931bd4a760 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.440 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.433+0000 7fd180c18740 1 -- 172.21.15.19:0/2222215736 >> 172.21.15.19:0/2222215736 conn(0x55931bd458b0 msgr2=0x55931bd45cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:18:09.440 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.433+0000 7fd180c18740 1 -- 172.21.15.19:0/2222215736 shutdown_connections 2024-03-18T15:18:09.441 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.433+0000 7fd176a00640 1 --2- 172.21.15.19:0/4282149858 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fd15c0a2800 0x7fd15c0a4cc0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fd1680bd830 tx=0x7fd1680f7060 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:18:09.441 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.433+0000 7fd180c18740 1 -- 172.21.15.19:0/2222215736 wait complete. 2024-03-18T15:18:09.441 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.434+0000 7fd180c18740 1 -- start start 2024-03-18T15:18:09.441 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.434+0000 7fd180c18740 1 Processor -- start 2024-03-18T15:18:09.441 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.434+0000 7fd180c18740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55931bd4a340 0x55931bd544f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:09.441 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.434+0000 7fd180c18740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55931bd4ad30 0x55931bd54a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:09.441 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.434+0000 7fd180c18740 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55931bd4fea0 0x55931bd5ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:09.441 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.434+0000 7fd180c18740 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x55931bd52e50 con 0x55931bd4fea0 2024-03-18T15:18:09.441 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.434+0000 7fd180c18740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55931bc6ae20 con 0x55931bd4a340 2024-03-18T15:18:09.441 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.435+0000 7fd180c18740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55931bd52fd0 con 0x55931bd4ad30 2024-03-18T15:18:09.441 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.435+0000 7fd180c18740 10 client.0 _ll_register_callbacks cb 0x55931bc6a7f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T15:18:09.441 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.435+0000 7fd176000640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55931bd4ad30 0x55931bd54a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:09.441 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.435+0000 7fd176000640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55931bd4ad30 0x55931bd54a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.19:39482/0 (socket says 172.21.15.19:39482) 2024-03-18T15:18:09.441 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.435+0000 7fd176000640 1 -- 172.21.15.19:0/1976408514 learned_addr learned my addr 172.21.15.19:0/1976408514 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:18:09.442 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.435+0000 7fd177400640 1 --2- 172.21.15.19:0/1976408514 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55931bd4a340 0x55931bd544f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:09.442 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.435+0000 7fd177400640 1 --2- 172.21.15.19:0/1976408514 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55931bd4fea0 0x55931bd5ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:09.442 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.435+0000 7fd176000640 1 -- 172.21.15.19:0/1976408514 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55931bd4a340 msgr2=0x55931bd544f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:18:09.442 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.435+0000 7fd176000640 1 --2- 172.21.15.19:0/1976408514 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55931bd4a340 0x55931bd544f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.442 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.435+0000 7fd176000640 1 -- 172.21.15.19:0/1976408514 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55931bd4fea0 msgr2=0x55931bd5ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:18:09.442 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.435+0000 7fd176000640 1 --2- 172.21.15.19:0/1976408514 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55931bd4fea0 0x55931bd5ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.442 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.435+0000 7fd176000640 1 -- 172.21.15.19:0/1976408514 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55931b9c32d0 con 0x55931bd4ad30 2024-03-18T15:18:09.442 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.435+0000 7fd177400640 1 --2- 172.21.15.19:0/1976408514 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55931bd4fea0 0x55931bd5ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T15:18:09.442 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.436+0000 7fd176000640 1 --2- 172.21.15.19:0/1976408514 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55931bd4ad30 0x55931bd54a30 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fd16000bf40 tx=0x7fd160011870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=72be78b0a2d2f276 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:18:09.442 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.436+0000 7fd164c00640 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-03-18T15:18:09.442 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.436+0000 7fd164c00640 1 -- 172.21.15.19:0/1976408514 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd160004220 con 0x55931bd4ad30 2024-03-18T15:18:09.442 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.436+0000 7fd180c18740 1 -- 172.21.15.19:0/1976408514 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd170012ad0 con 0x55931bd4ad30 2024-03-18T15:18:09.443 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.436+0000 7fd164c00640 1 -- 172.21.15.19:0/1976408514 <== mon.1 v2:172.21.15.104:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fd1600043c0 con 0x55931bd4ad30 2024-03-18T15:18:09.443 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.436+0000 7fd164c00640 1 -- 172.21.15.19:0/1976408514 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fd160025050 con 0x55931bd4ad30 2024-03-18T15:18:09.443 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.436+0000 7fd180c18740 10 client.25411 fetch_fsmap learned FSMap version 26 2024-03-18T15:18:09.443 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.436+0000 7fd180c18740 1 -- 172.21.15.19:0/1976408514 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55931ba8d6b0 con 0x55931bd4ad30 2024-03-18T15:18:09.443 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.436+0000 7fd164c00640 1 -- 172.21.15.19:0/1976408514 <== mon.1 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd170012ad0 con 0x55931bd4ad30 2024-03-18T15:18:09.443 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.437+0000 7fd164c00640 1 -- 172.21.15.19:0/1976408514 <== mon.1 v2:172.21.15.104:3300/0 5 ==== fsmap(e 26) ==== 8362+0+0 (secure 0 0 0) 0x7fd160003a30 con 0x55931bd4ad30 2024-03-18T15:18:09.443 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.437+0000 7fd180c18740 10 client.25411 fetch_fsmap finished waiting for FSMap version 26 2024-03-18T15:18:09.443 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.437+0000 7fd180c18740 10 client.25411 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-18T15:18:09.443 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.437+0000 7fd180c18740 20 client.25411 populate_metadata read hostname 'smithi019' 2024-03-18T15:18:09.443 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.437+0000 7fd180c18740 1 --2- 172.21.15.19:0/1976408514 >> [v2:172.21.15.104:6838/805341436,v1:172.21.15.104:6839/805341436] conn(0x55931bd612d0 0x55931bd62740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:09.444 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.437+0000 7fd180c18740 4 client.25411 mds_command: new command op to 24529 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T15:18:09.444 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.437+0000 7fd180c18740 1 -- 172.21.15.19:0/1976408514 --> [v2:172.21.15.104:6838/805341436,v1:172.21.15.104:6839/805341436] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55931bd0c2b0 con 0x55931bd612d0 2024-03-18T15:18:09.444 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.437+0000 7fd177400640 1 --2- 172.21.15.19:0/1976408514 >> [v2:172.21.15.104:6838/805341436,v1:172.21.15.104:6839/805341436] conn(0x55931bd612d0 0x55931bd62740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:09.444 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.438+0000 7fd177400640 1 --2- 172.21.15.19:0/1976408514 >> [v2:172.21.15.104:6838/805341436,v1:172.21.15.104:6839/805341436] conn(0x55931bd612d0 0x55931bd62740 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f8795d31edc95776 server_cookie=4d1ebc07851b865f in_seq=0 out_seq=0 2024-03-18T15:18:09.444 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.438+0000 7fd164c00640 10 client.25411 ms_handle_connect on v2:172.21.15.104:6838/805341436 2024-03-18T15:18:09.444 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.438+0000 7fd164c00640 1 -- 172.21.15.19:0/1976408514 <== mds.1 v2:172.21.15.104:6838/805341436 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55931bd0c2b0 con 0x55931bd612d0 2024-03-18T15:18:09.444 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.438+0000 7fd164c00640 10 client.25411 handle_command_reply: tid=0 2024-03-18T15:18:09.444 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.438+0000 7fd180c18740 1 client.25411 shutdown 2024-03-18T15:18:09.444 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.439+0000 7fd180c18740 1 -- 172.21.15.19:0/1976408514 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55931bd4ad30 msgr2=0x55931bd54a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:09.444 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.439+0000 7fd180c18740 1 --2- 172.21.15.19:0/1976408514 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55931bd4ad30 0x55931bd54a30 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fd16000bf40 tx=0x7fd160011870 comp rx=0 tx=0).stop 2024-03-18T15:18:09.444 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.439+0000 7fd180c18740 1 -- 172.21.15.19:0/1976408514 shutdown_connections 2024-03-18T15:18:09.444 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.439+0000 7fd180c18740 1 --2- 172.21.15.19:0/1976408514 >> [v2:172.21.15.104:6838/805341436,v1:172.21.15.104:6839/805341436] conn(0x55931bd612d0 0x55931bd62740 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.445 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.439+0000 7fd180c18740 1 --2- 172.21.15.19:0/1976408514 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55931bd4fea0 0x55931bd5ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.445 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.439+0000 7fd180c18740 1 --2- 172.21.15.19:0/1976408514 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55931bd4ad30 0x55931bd54a30 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.445 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.439+0000 7fd180c18740 1 --2- 172.21.15.19:0/1976408514 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55931bd4a340 0x55931bd544f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.445 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.439+0000 7fd164c00640 0 client.25411 ms_handle_reset on v2:172.21.15.104:6838/805341436 2024-03-18T15:18:09.445 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.439+0000 7fd180c18740 1 -- 172.21.15.19:0/1976408514 >> 172.21.15.19:0/1976408514 conn(0x55931bd458b0 msgr2=0x55931bd44a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:18:09.445 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.439+0000 7fd180c18740 1 -- 172.21.15.19:0/1976408514 shutdown_connections 2024-03-18T15:18:09.445 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.439+0000 7fd180c18740 1 -- 172.21.15.19:0/1976408514 wait complete. 2024-03-18T15:18:09.445 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.439+0000 7fd180c18740 20 client.25411 trim_cache size 0 max 16384 2024-03-18T15:18:09.446 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:09 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/1609730460' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:18:09.461 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.459+0000 7fd180c18740 1 Processor -- start 2024-03-18T15:18:09.461 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.459+0000 7fd180c18740 1 -- start start 2024-03-18T15:18:09.461 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.459+0000 7fd180c18740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55931bd4a340 0x55931bd4f510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:09.461 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.459+0000 7fd180c18740 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55931bd4ad30 0x55931bd5ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:09.461 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.459+0000 7fd180c18740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55931bd4fea0 0x55931bd45c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:09.461 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.459+0000 7fd180c18740 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x55931bd60a80 con 0x55931bd4ad30 2024-03-18T15:18:09.461 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.459+0000 7fd180c18740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55931bd61150 con 0x55931bd4a340 2024-03-18T15:18:09.461 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.459+0000 7fd180c18740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55931bd61820 con 0x55931bd4fea0 2024-03-18T15:18:09.461 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.459+0000 7fd176000640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55931bd4ad30 0x55931bd5ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:09.461 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.460+0000 7fd176000640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55931bd4ad30 0x55931bd5ba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:45876/0 (socket says 172.21.15.19:45876) 2024-03-18T15:18:09.462 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.460+0000 7fd176000640 1 -- 172.21.15.19:0/1595925827 learned_addr learned my addr 172.21.15.19:0/1595925827 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:18:09.462 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.460+0000 7fd177400640 1 --2- 172.21.15.19:0/1595925827 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55931bd4fea0 0x55931bd45c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:09.462 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.460+0000 7fd177400640 1 --2- 172.21.15.19:0/1595925827 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55931bd4a340 0x55931bd4f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:09.462 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.460+0000 7fd176000640 1 -- 172.21.15.19:0/1595925827 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55931bd4fea0 msgr2=0x55931bd45c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:18:09.462 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.460+0000 7fd176000640 1 --2- 172.21.15.19:0/1595925827 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55931bd4fea0 0x55931bd45c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.462 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.460+0000 7fd176000640 1 -- 172.21.15.19:0/1595925827 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55931bd4a340 msgr2=0x55931bd4f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:18:09.462 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.460+0000 7fd176000640 1 --2- 172.21.15.19:0/1595925827 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55931bd4a340 0x55931bd4f510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.462 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.460+0000 7fd176000640 1 -- 172.21.15.19:0/1595925827 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55931bd53910 con 0x55931bd4ad30 2024-03-18T15:18:09.462 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.460+0000 7fd177400640 1 --2- 172.21.15.19:0/1595925827 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55931bd4a340 0x55931bd4f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T15:18:09.468 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.460+0000 7fd176000640 1 --2- 172.21.15.19:0/1595925827 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55931bd4ad30 0x55931bd5ba30 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fd160003dd0 tx=0x7fd16000b710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58e88949b179ef30 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:18:09.468 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.461+0000 7fd167400640 1 -- 172.21.15.19:0/1595925827 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd160004000 con 0x55931bd4ad30 2024-03-18T15:18:09.468 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.461+0000 7fd167400640 1 -- 172.21.15.19:0/1595925827 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd16002d030 con 0x55931bd4ad30 2024-03-18T15:18:09.468 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.461+0000 7fd167400640 1 -- 172.21.15.19:0/1595925827 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd160005440 con 0x55931bd4ad30 2024-03-18T15:18:09.468 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.461+0000 7fd180c18740 1 -- 172.21.15.19:0/1595925827 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55931bd4ad30 msgr2=0x55931bd5ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:09.468 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.461+0000 7fd180c18740 1 --2- 172.21.15.19:0/1595925827 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55931bd4ad30 0x55931bd5ba30 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fd160003dd0 tx=0x7fd16000b710 comp rx=0 tx=0).stop 2024-03-18T15:18:09.468 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.461+0000 7fd180c18740 1 -- 172.21.15.19:0/1595925827 shutdown_connections 2024-03-18T15:18:09.468 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.461+0000 7fd180c18740 1 --2- 172.21.15.19:0/1595925827 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55931bd4fea0 0x55931bd45c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.469 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.461+0000 7fd180c18740 1 --2- 172.21.15.19:0/1595925827 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55931bd4ad30 0x55931bd5ba30 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.469 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.461+0000 7fd180c18740 1 --2- 172.21.15.19:0/1595925827 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55931bd4a340 0x55931bd4f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.469 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.461+0000 7fd180c18740 1 -- 172.21.15.19:0/1595925827 >> 172.21.15.19:0/1595925827 conn(0x55931bd458b0 msgr2=0x55931bd44a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:18:09.469 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.462+0000 7fd180c18740 1 -- 172.21.15.19:0/1595925827 shutdown_connections 2024-03-18T15:18:09.469 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.462+0000 7fd180c18740 1 -- 172.21.15.19:0/1595925827 wait complete. 2024-03-18T15:18:09.469 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.462+0000 7fd180c18740 1 -- start start 2024-03-18T15:18:09.469 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.462+0000 7fd180c18740 1 Processor -- start 2024-03-18T15:18:09.469 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.462+0000 7fd180c18740 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55931bd4a340 0x55931bd573d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:09.469 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.462+0000 7fd180c18740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55931bd4ad30 0x55931bd57910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:09.469 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.463+0000 7fd180c18740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55931bd4fea0 0x55931bd5a930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:09.469 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.463+0000 7fd180c18740 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x55931bd625c0 con 0x55931bd4a340 2024-03-18T15:18:09.470 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.463+0000 7fd180c18740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55931bd61ef0 con 0x55931bd4fea0 2024-03-18T15:18:09.470 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.463+0000 7fd180c18740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55931bd62c90 con 0x55931bd4ad30 2024-03-18T15:18:09.470 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.463+0000 7fd180c18740 10 client.0 _ll_register_callbacks cb 0x7fd16000ba10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T15:18:09.470 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.463+0000 7fd176000640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55931bd4ad30 0x55931bd57910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:09.470 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.463+0000 7fd177400640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55931bd4a340 0x55931bd573d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:09.470 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.463+0000 7fd176000640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55931bd4ad30 0x55931bd57910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.19:39508/0 (socket says 172.21.15.19:39508) 2024-03-18T15:18:09.470 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.463+0000 7fd176000640 1 -- 172.21.15.19:0/507130073 learned_addr learned my addr 172.21.15.19:0/507130073 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:18:09.470 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.463+0000 7fd177400640 1 --2- 172.21.15.19:0/507130073 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55931bd4fea0 0x55931bd5a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:09.470 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.463+0000 7fd176000640 1 -- 172.21.15.19:0/507130073 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55931bd4fea0 msgr2=0x55931bd5a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:18:09.470 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.463+0000 7fd176000640 1 --2- 172.21.15.19:0/507130073 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55931bd4fea0 0x55931bd5a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.470 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.463+0000 7fd176000640 1 -- 172.21.15.19:0/507130073 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55931bd4a340 msgr2=0x55931bd573d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:18:09.470 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.463+0000 7fd176000640 1 --2- 172.21.15.19:0/507130073 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55931bd4a340 0x55931bd573d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.470 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.463+0000 7fd176000640 1 -- 172.21.15.19:0/507130073 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55931bd5af20 con 0x55931bd4ad30 2024-03-18T15:18:09.471 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.463+0000 7fd177400640 1 --2- 172.21.15.19:0/507130073 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55931bd4a340 0x55931bd573d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:18:09.471 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.464+0000 7fd176000640 1 --2- 172.21.15.19:0/507130073 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55931bd4ad30 0x55931bd57910 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fd16000bf40 tx=0x7fd1600041e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7be0986a3cec5599 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:18:09.471 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.464+0000 7fd164c00640 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-03-18T15:18:09.471 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.464+0000 7fd164c00640 1 -- 172.21.15.19:0/507130073 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd160036070 con 0x55931bd4ad30 2024-03-18T15:18:09.471 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.464+0000 7fd180c18740 1 -- 172.21.15.19:0/507130073 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd16002d030 con 0x55931bd4ad30 2024-03-18T15:18:09.471 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.464+0000 7fd164c00640 1 -- 172.21.15.19:0/507130073 <== mon.1 v2:172.21.15.104:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fd160005350 con 0x55931bd4ad30 2024-03-18T15:18:09.471 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.464+0000 7fd164c00640 1 -- 172.21.15.19:0/507130073 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fd160005590 con 0x55931bd4ad30 2024-03-18T15:18:09.471 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.464+0000 7fd180c18740 10 client.25417 fetch_fsmap learned FSMap version 26 2024-03-18T15:18:09.471 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.464+0000 7fd180c18740 1 -- 172.21.15.19:0/507130073 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55931bdc71e0 con 0x55931bd4ad30 2024-03-18T15:18:09.471 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.464+0000 7fd164c00640 1 -- 172.21.15.19:0/507130073 <== mon.1 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd16002d030 con 0x55931bd4ad30 2024-03-18T15:18:09.471 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.465+0000 7fd164c00640 1 -- 172.21.15.19:0/507130073 <== mon.1 v2:172.21.15.104:3300/0 5 ==== fsmap(e 26) ==== 8362+0+0 (secure 0 0 0) 0x7fd160012ad0 con 0x55931bd4ad30 2024-03-18T15:18:09.471 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.465+0000 7fd180c18740 10 client.25417 fetch_fsmap finished waiting for FSMap version 26 2024-03-18T15:18:09.471 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.465+0000 7fd180c18740 10 client.25417 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-18T15:18:09.471 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.465+0000 7fd180c18740 20 client.25417 populate_metadata read hostname 'smithi019' 2024-03-18T15:18:09.472 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.465+0000 7fd180c18740 1 --2- 172.21.15.19:0/507130073 >> [v2:172.21.15.104:6838/805341436,v1:172.21.15.104:6839/805341436] conn(0x55931bdcbaa0 0x55931bdca7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:09.472 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.465+0000 7fd180c18740 4 client.25417 mds_command: new command op to 24529 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-18T15:18:09.472 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.465+0000 7fd180c18740 1 -- 172.21.15.19:0/507130073 --> [v2:172.21.15.104:6838/805341436,v1:172.21.15.104:6839/805341436] -- command(tid 0: {"prefix": "status"}) -- 0x55931bcbc190 con 0x55931bdcbaa0 2024-03-18T15:18:09.472 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.465+0000 7fd177400640 1 --2- 172.21.15.19:0/507130073 >> [v2:172.21.15.104:6838/805341436,v1:172.21.15.104:6839/805341436] conn(0x55931bdcbaa0 0x55931bdca7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:09.472 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.466+0000 7fd177400640 1 --2- 172.21.15.19:0/507130073 >> [v2:172.21.15.104:6838/805341436,v1:172.21.15.104:6839/805341436] conn(0x55931bdcbaa0 0x55931bdca7b0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=58f79fb46e4ee988 server_cookie=ba73fd244013a67f in_seq=0 out_seq=0 2024-03-18T15:18:09.472 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.466+0000 7fd164c00640 10 client.25417 ms_handle_connect on v2:172.21.15.104:6838/805341436 2024-03-18T15:18:09.472 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.466+0000 7fd164c00640 1 -- 172.21.15.19:0/507130073 <== mds.1 v2:172.21.15.104:6838/805341436 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x55931bcbc190 con 0x55931bdcbaa0 2024-03-18T15:18:09.472 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.466+0000 7fd164c00640 10 client.25417 handle_command_reply: tid=0 2024-03-18T15:18:09.472 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.466+0000 7fd180c18740 1 client.25417 shutdown 2024-03-18T15:18:09.472 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.466+0000 7fd180c18740 1 -- 172.21.15.19:0/507130073 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55931bd4ad30 msgr2=0x55931bd57910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:09.472 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.466+0000 7fd180c18740 1 --2- 172.21.15.19:0/507130073 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55931bd4ad30 0x55931bd57910 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fd16000bf40 tx=0x7fd1600041e0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.472 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.466+0000 7fd180c18740 1 -- 172.21.15.19:0/507130073 shutdown_connections 2024-03-18T15:18:09.472 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.466+0000 7fd180c18740 1 --2- 172.21.15.19:0/507130073 >> [v2:172.21.15.104:6838/805341436,v1:172.21.15.104:6839/805341436] conn(0x55931bdcbaa0 0x55931bdca7b0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.473 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.466+0000 7fd180c18740 1 --2- 172.21.15.19:0/507130073 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55931bd4fea0 0x55931bd5a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.473 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.466+0000 7fd180c18740 1 --2- 172.21.15.19:0/507130073 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55931bd4ad30 0x55931bd57910 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.473 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.466+0000 7fd164c00640 0 client.25417 ms_handle_reset on v2:172.21.15.104:6838/805341436 2024-03-18T15:18:09.473 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.466+0000 7fd180c18740 1 --2- 172.21.15.19:0/507130073 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55931bd4a340 0x55931bd573d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.473 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.466+0000 7fd180c18740 1 -- 172.21.15.19:0/507130073 >> 172.21.15.19:0/507130073 conn(0x55931bd458b0 msgr2=0x55931bd44a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:18:09.473 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.466+0000 7fd180c18740 1 -- 172.21.15.19:0/507130073 shutdown_connections 2024-03-18T15:18:09.473 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.466+0000 7fd180c18740 1 -- 172.21.15.19:0/507130073 wait complete. 2024-03-18T15:18:09.473 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.466+0000 7fd180c18740 20 client.25417 trim_cache size 0 max 16384 2024-03-18T15:18:09.473 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-03-18T15:18:09.474 INFO:teuthology.orchestra.run.smithi019.stdout: "cluster_fsid": "d4dc2ba6-e538-11ee-95c9-87774f69a715", 2024-03-18T15:18:09.474 INFO:teuthology.orchestra.run.smithi019.stdout: "whoami": 1, 2024-03-18T15:18:09.474 INFO:teuthology.orchestra.run.smithi019.stdout: "id": 24529, 2024-03-18T15:18:09.474 INFO:teuthology.orchestra.run.smithi019.stdout: "want_state": "up:active", 2024-03-18T15:18:09.474 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "up:active", 2024-03-18T15:18:09.474 INFO:teuthology.orchestra.run.smithi019.stdout: "fs_name": "cephfs", 2024-03-18T15:18:09.474 INFO:teuthology.orchestra.run.smithi019.stdout: "rank_uptime": 146.008405646, 2024-03-18T15:18:09.474 INFO:teuthology.orchestra.run.smithi019.stdout: "mdsmap_epoch": 26, 2024-03-18T15:18:09.474 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap_epoch": 93, 2024-03-18T15:18:09.474 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap_epoch_barrier": 89, 2024-03-18T15:18:09.474 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 196.526793048 2024-03-18T15:18:09.474 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-03-18T15:18:09.475 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.467+0000 7fd177e00640 1 -- 172.21.15.19:0/4282149858 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fd15c0a2800 msgr2=0x7fd15c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:09.475 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.467+0000 7fd177e00640 1 --2- 172.21.15.19:0/4282149858 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fd15c0a2800 0x7fd15c0a4cc0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fd1680bd830 tx=0x7fd1680f7060 comp rx=0 tx=0).stop 2024-03-18T15:18:09.475 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.467+0000 7fd177e00640 1 -- 172.21.15.19:0/4282149858 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd178102140 msgr2=0x7fd178199c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:09.475 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.467+0000 7fd177e00640 1 --2- 172.21.15.19:0/4282149858 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd178102140 0x7fd178199c40 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fd1680325f0 tx=0x7fd168035d10 comp rx=0 tx=0).stop 2024-03-18T15:18:09.475 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.467+0000 7fd177e00640 1 -- 172.21.15.19:0/4282149858 shutdown_connections 2024-03-18T15:18:09.475 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.467+0000 7fd177e00640 1 --2- 172.21.15.19:0/4282149858 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd1781073a0 0x7fd1781a11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.476 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.467+0000 7fd177e00640 1 --2- 172.21.15.19:0/4282149858 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fd15c0a2800 0x7fd15c0a4cc0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.476 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.467+0000 7fd177e00640 1 --2- 172.21.15.19:0/4282149858 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd178104a70 0x7fd17819a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.476 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.467+0000 7fd177e00640 1 --2- 172.21.15.19:0/4282149858 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd178102140 0x7fd178199c40 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.476 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.467+0000 7fd177e00640 1 -- 172.21.15.19:0/4282149858 >> 172.21.15.19:0/4282149858 conn(0x7fd1780fbb50 msgr2=0x7fd178101930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:18:09.476 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.467+0000 7fd177e00640 1 -- 172.21.15.19:0/4282149858 shutdown_connections 2024-03-18T15:18:09.476 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.467+0000 7fd177e00640 1 -- 172.21.15.19:0/4282149858 wait complete. 2024-03-18T15:18:09.489 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-18T15:18:09.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:09 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/1609730460' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:18:09.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.715+0000 7f6ccec00640 1 -- 172.21.15.19:0/1697148987 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6cc81055d0 msgr2=0x7f6cc810be60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:09.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.715+0000 7f6ccec00640 1 --2- 172.21.15.19:0/1697148987 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6cc81055d0 0x7f6cc810be60 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f6cc0009860 tx=0x7f6cc002bb70 comp rx=0 tx=0).stop 2024-03-18T15:18:09.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.715+0000 7f6ccec00640 1 -- 172.21.15.19:0/1697148987 shutdown_connections 2024-03-18T15:18:09.717 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.715+0000 7f6ccec00640 1 --2- 172.21.15.19:0/1697148987 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6cc81055d0 0x7f6cc810be60 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.717 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.715+0000 7f6ccec00640 1 --2- 172.21.15.19:0/1697148987 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6cc8104c10 0x7f6cc8105090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.717 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.715+0000 7f6ccec00640 1 --2- 172.21.15.19:0/1697148987 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6cc8103a20 0x7f6cc8103e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.717 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.715+0000 7f6ccec00640 1 -- 172.21.15.19:0/1697148987 >> 172.21.15.19:0/1697148987 conn(0x7f6cc80fef90 msgr2=0x7f6cc81013d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:18:09.717 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.715+0000 7f6ccec00640 1 -- 172.21.15.19:0/1697148987 shutdown_connections 2024-03-18T15:18:09.717 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.715+0000 7f6ccec00640 1 -- 172.21.15.19:0/1697148987 wait complete. 2024-03-18T15:18:09.719 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.716+0000 7f6ccec00640 1 Processor -- start 2024-03-18T15:18:09.719 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.716+0000 7f6ccec00640 1 -- start start 2024-03-18T15:18:09.719 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.716+0000 7f6ccec00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6cc8103a20 0x7f6cc8199cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:09.719 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.716+0000 7f6ccec00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6cc8104c10 0x7f6cc819a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:09.719 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.716+0000 7f6ccec00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6cc81055d0 0x7f6cc81a1250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:09.719 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.716+0000 7f6ccec00640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f6cc810f6a0 con 0x7f6cc8103a20 2024-03-18T15:18:09.719 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.716+0000 7f6ccec00640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6cc810f520 con 0x7f6cc8104c10 2024-03-18T15:18:09.720 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.716+0000 7f6ccec00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f6cc810f820 con 0x7f6cc81055d0 2024-03-18T15:18:09.720 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.717+0000 7f6cccc00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6cc81055d0 0x7f6cc81a1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:09.720 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.717+0000 7f6cc7400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6cc8104c10 0x7f6cc819a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:09.720 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.717+0000 7f6cc7400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6cc8104c10 0x7f6cc819a1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:46442/0 (socket says 172.21.15.19:46442) 2024-03-18T15:18:09.720 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.717+0000 7f6cc7400640 1 -- 172.21.15.19:0/3912951805 learned_addr learned my addr 172.21.15.19:0/3912951805 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:18:09.720 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.717+0000 7f6cc7e00640 1 --2- 172.21.15.19:0/3912951805 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6cc8103a20 0x7f6cc8199cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:09.720 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.717+0000 7f6cccc00640 1 -- 172.21.15.19:0/3912951805 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6cc8104c10 msgr2=0x7f6cc819a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:09.720 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.717+0000 7f6cccc00640 1 --2- 172.21.15.19:0/3912951805 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6cc8104c10 0x7f6cc819a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.720 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.717+0000 7f6cccc00640 1 -- 172.21.15.19:0/3912951805 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6cc8103a20 msgr2=0x7f6cc8199cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:09.720 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.717+0000 7f6cccc00640 1 --2- 172.21.15.19:0/3912951805 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6cc8103a20 0x7f6cc8199cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.720 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.717+0000 7f6cccc00640 1 -- 172.21.15.19:0/3912951805 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cc81a18f0 con 0x7f6cc81055d0 2024-03-18T15:18:09.720 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.717+0000 7f6cc7400640 1 --2- 172.21.15.19:0/3912951805 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6cc8104c10 0x7f6cc819a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:18:09.720 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.717+0000 7f6cccc00640 1 --2- 172.21.15.19:0/3912951805 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6cc81055d0 0x7f6cc81a1250 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f6cc0009780 tx=0x7f6cc0004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:18:09.721 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.718+0000 7f6cc4c00640 1 -- 172.21.15.19:0/3912951805 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cc0041020 con 0x7f6cc81055d0 2024-03-18T15:18:09.721 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.718+0000 7f6cc4c00640 1 -- 172.21.15.19:0/3912951805 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6cc00347b0 con 0x7f6cc81055d0 2024-03-18T15:18:09.721 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.718+0000 7f6cc4c00640 1 -- 172.21.15.19:0/3912951805 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cc0039040 con 0x7f6cc81055d0 2024-03-18T15:18:09.721 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.718+0000 7f6ccec00640 1 -- 172.21.15.19:0/3912951805 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6cc81a1b80 con 0x7f6cc81055d0 2024-03-18T15:18:09.721 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.718+0000 7f6ccec00640 1 -- 172.21.15.19:0/3912951805 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6cc81a2040 con 0x7f6cc81055d0 2024-03-18T15:18:09.722 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.721+0000 7f6cc4c00640 1 -- 172.21.15.19:0/3912951805 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f6cc0044020 con 0x7f6cc81055d0 2024-03-18T15:18:09.722 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.721+0000 7f6cd1a08740 1 Processor -- start 2024-03-18T15:18:09.722 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.721+0000 7f6cd1a08740 1 -- start start 2024-03-18T15:18:09.724 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.721+0000 7f6cd1a08740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b6121e1160 0x55b6121e1580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:09.724 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.721+0000 7f6cd1a08740 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55b6121e1b50 0x55b6121e6650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:09.724 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.721+0000 7f6cd1a08740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b6121e6cc0 0x55b6121e9150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:09.724 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.721+0000 7f6cd1a08740 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x55b612145c50 con 0x55b6121e1b50 2024-03-18T15:18:09.724 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.721+0000 7f6cd1a08740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55b6120ee9d0 con 0x55b6121e6cc0 2024-03-18T15:18:09.724 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.721+0000 7f6cd1a08740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55b612141cf0 con 0x55b6121e1160 2024-03-18T15:18:09.724 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.722+0000 7f6cc7400640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b6121e1160 0x55b6121e1580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:09.724 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.722+0000 7f6cc7e00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55b6121e1b50 0x55b6121e6650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:09.724 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.722+0000 7f6cc7400640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b6121e1160 0x55b6121e1580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.19:39536/0 (socket says 172.21.15.19:39536) 2024-03-18T15:18:09.724 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.722+0000 7f6cc7400640 1 -- 172.21.15.19:0/2869595680 learned_addr learned my addr 172.21.15.19:0/2869595680 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:18:09.725 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.722+0000 7f6cc7400640 1 --2- 172.21.15.19:0/2869595680 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b6121e6cc0 0x55b6121e9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:09.725 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.722+0000 7f6cc7e00640 1 -- 172.21.15.19:0/2869595680 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b6121e1160 msgr2=0x55b6121e1580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:18:09.725 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.722+0000 7f6cc7e00640 1 --2- 172.21.15.19:0/2869595680 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b6121e1160 0x55b6121e1580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.725 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.722+0000 7f6cc4c00640 1 --2- 172.21.15.19:0/3912951805 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f6cac0a2800 0x7f6cac0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:09.725 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.722+0000 7f6cc7e00640 1 -- 172.21.15.19:0/2869595680 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b6121e6cc0 msgr2=0x55b6121e9150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:18:09.725 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.722+0000 7f6cc7e00640 1 --2- 172.21.15.19:0/2869595680 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b6121e6cc0 0x55b6121e9150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.725 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.722+0000 7f6cc7e00640 1 -- 172.21.15.19:0/2869595680 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b612108f10 con 0x55b6121e1b50 2024-03-18T15:18:09.725 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.722+0000 7f6cc7400640 1 --2- 172.21.15.19:0/2869595680 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b6121e6cc0 0x55b6121e9150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T15:18:09.725 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.722+0000 7f6cc4c00640 1 -- 172.21.15.19:0/3912951805 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f6cc00f0840 con 0x7f6cc81055d0 2024-03-18T15:18:09.725 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.723+0000 7f6cccc00640 1 --2- 172.21.15.19:0/3912951805 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f6cac0a2800 0x7f6cac0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:09.725 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.723+0000 7f6cc7e00640 1 --2- 172.21.15.19:0/2869595680 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55b6121e1b50 0x55b6121e6650 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f6cc8104a70 tx=0x7f6cb80095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25cdcecd3e515855 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:18:09.725 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.723+0000 7f6cbd200640 1 -- 172.21.15.19:0/2869595680 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cb8003970 con 0x55b6121e1b50 2024-03-18T15:18:09.725 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.723+0000 7f6cbd200640 1 -- 172.21.15.19:0/2869595680 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6cb800d7d0 con 0x55b6121e1b50 2024-03-18T15:18:09.726 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.723+0000 7f6cbd200640 1 -- 172.21.15.19:0/2869595680 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cb80044a0 con 0x55b6121e1b50 2024-03-18T15:18:09.729 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.724+0000 7f6cd1a08740 1 -- 172.21.15.19:0/2869595680 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55b6121e1b50 msgr2=0x55b6121e6650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:09.729 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.724+0000 7f6cd1a08740 1 --2- 172.21.15.19:0/2869595680 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55b6121e1b50 0x55b6121e6650 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f6cc8104a70 tx=0x7f6cb80095c0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.729 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.724+0000 7f6cccc00640 1 --2- 172.21.15.19:0/3912951805 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f6cac0a2800 0x7f6cac0a4cc0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f6cc00b9210 tx=0x7f6cc0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:18:09.730 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.724+0000 7f6cd1a08740 1 -- 172.21.15.19:0/2869595680 shutdown_connections 2024-03-18T15:18:09.730 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.724+0000 7f6cd1a08740 1 --2- 172.21.15.19:0/2869595680 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b6121e6cc0 0x55b6121e9150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.730 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.724+0000 7f6cd1a08740 1 --2- 172.21.15.19:0/2869595680 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55b6121e1b50 0x55b6121e6650 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.730 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.724+0000 7f6cd1a08740 1 --2- 172.21.15.19:0/2869595680 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b6121e1160 0x55b6121e1580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.730 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.724+0000 7f6cd1a08740 1 -- 172.21.15.19:0/2869595680 >> 172.21.15.19:0/2869595680 conn(0x55b6121dc6d0 msgr2=0x55b6121dcad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:18:09.730 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.724+0000 7f6cd1a08740 1 -- 172.21.15.19:0/2869595680 shutdown_connections 2024-03-18T15:18:09.730 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.724+0000 7f6cd1a08740 1 -- 172.21.15.19:0/2869595680 wait complete. 2024-03-18T15:18:09.730 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.724+0000 7f6cd1a08740 1 -- start start 2024-03-18T15:18:09.730 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.725+0000 7f6cd1a08740 1 Processor -- start 2024-03-18T15:18:09.730 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.725+0000 7f6cd1a08740 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55b6121e1160 0x55b6121eb340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:09.730 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.725+0000 7f6cd1a08740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b6121e1b50 0x55b6121eb880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:09.731 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.725+0000 7f6cd1a08740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b6121e6cc0 0x55b6121f2850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:09.731 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.725+0000 7f6cd1a08740 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x55b6121e9c70 con 0x55b6121e1160 2024-03-18T15:18:09.731 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.725+0000 7f6cd1a08740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55b612101e20 con 0x55b6121e1b50 2024-03-18T15:18:09.731 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.725+0000 7f6cd1a08740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55b6121e9df0 con 0x55b6121e6cc0 2024-03-18T15:18:09.731 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.725+0000 7f6cd1a08740 10 client.0 _ll_register_callbacks cb 0x55b6121017f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T15:18:09.731 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.725+0000 7f6cc7400640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55b6121e1160 0x55b6121eb340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:09.731 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.725+0000 7f6cc7e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b6121e1b50 0x55b6121eb880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:09.731 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.726+0000 7f6cc7400640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55b6121e1160 0x55b6121eb340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:45936/0 (socket says 172.21.15.19:45936) 2024-03-18T15:18:09.731 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.726+0000 7f6cc7400640 1 -- 172.21.15.19:0/1717605197 learned_addr learned my addr 172.21.15.19:0/1717605197 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:18:09.731 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.726+0000 7f6cc7400640 1 --2- 172.21.15.19:0/1717605197 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b6121e6cc0 0x55b6121f2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:09.731 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.726+0000 7f6cc7e00640 1 -- 172.21.15.19:0/1717605197 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b6121e6cc0 msgr2=0x55b6121f2850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:18:09.731 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.726+0000 7f6cc7e00640 1 --2- 172.21.15.19:0/1717605197 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b6121e6cc0 0x55b6121f2850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.731 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.726+0000 7f6cc7e00640 1 -- 172.21.15.19:0/1717605197 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55b6121e1160 msgr2=0x55b6121eb340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:18:09.731 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.726+0000 7f6cc7e00640 1 --2- 172.21.15.19:0/1717605197 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55b6121e1160 0x55b6121eb340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.732 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.726+0000 7f6cc7e00640 1 -- 172.21.15.19:0/1717605197 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b611e5a2d0 con 0x55b6121e1b50 2024-03-18T15:18:09.732 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.726+0000 7f6cc7e00640 1 --2- 172.21.15.19:0/1717605197 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b6121e1b50 0x55b6121eb880 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x55b6121dd640 tx=0x7f6cb8009de0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ca685a776878ad27 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:18:09.732 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.726+0000 7f6cb5600640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-18T15:18:09.732 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.727+0000 7f6cb5600640 1 -- 172.21.15.19:0/1717605197 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cb8027070 con 0x55b6121e1b50 2024-03-18T15:18:09.732 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.727+0000 7f6cb5600640 1 -- 172.21.15.19:0/1717605197 <== mon.2 v2:172.21.15.33:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f6cb8003970 con 0x55b6121e1b50 2024-03-18T15:18:09.732 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.727+0000 7f6cd1a08740 1 -- 172.21.15.19:0/1717605197 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6cb800d7d0 con 0x55b6121e1b50 2024-03-18T15:18:09.732 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.727+0000 7f6cb5600640 1 -- 172.21.15.19:0/1717605197 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f6cb8033050 con 0x55b6121e1b50 2024-03-18T15:18:09.732 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.727+0000 7f6cd1a08740 10 client.25415 fetch_fsmap learned FSMap version 26 2024-03-18T15:18:09.732 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.727+0000 7f6cd1a08740 1 -- 172.21.15.19:0/1717605197 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55b611f246b0 con 0x55b6121e1b50 2024-03-18T15:18:09.732 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.727+0000 7f6cb5600640 1 -- 172.21.15.19:0/1717605197 <== mon.2 v2:172.21.15.33:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6cb800d7d0 con 0x55b6121e1b50 2024-03-18T15:18:09.732 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.727+0000 7f6cb5600640 1 -- 172.21.15.19:0/1717605197 <== mon.2 v2:172.21.15.33:3300/0 5 ==== fsmap(e 26) ==== 8362+0+0 (secure 0 0 0) 0x7f6cb800b040 con 0x55b6121e1b50 2024-03-18T15:18:09.732 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.727+0000 7f6cd1a08740 10 client.25415 fetch_fsmap finished waiting for FSMap version 26 2024-03-18T15:18:09.733 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.728+0000 7f6cd1a08740 10 client.25415 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-18T15:18:09.733 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.728+0000 7f6cd1a08740 20 client.25415 populate_metadata read hostname 'smithi019' 2024-03-18T15:18:09.733 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.728+0000 7f6cd1a08740 1 --2- 172.21.15.19:0/1717605197 >> [v2:172.21.15.33:6838/1903591535,v1:172.21.15.33:6839/1903591535] conn(0x55b6121f7cc0 0x55b6121f9130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:09.733 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.728+0000 7f6cd1a08740 4 client.25415 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T15:18:09.733 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.728+0000 7f6cd1a08740 1 -- 172.21.15.19:0/1717605197 --> [v2:172.21.15.33:6838/1903591535,v1:172.21.15.33:6839/1903591535] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b6121a32b0 con 0x55b6121f7cc0 2024-03-18T15:18:09.733 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.728+0000 7f6cc7400640 1 --2- 172.21.15.19:0/1717605197 >> [v2:172.21.15.33:6838/1903591535,v1:172.21.15.33:6839/1903591535] conn(0x55b6121f7cc0 0x55b6121f9130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:09.733 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.728+0000 7f6cc7400640 1 --2- 172.21.15.19:0/1717605197 >> [v2:172.21.15.33:6838/1903591535,v1:172.21.15.33:6839/1903591535] conn(0x55b6121f7cc0 0x55b6121f9130 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e555cdc0b31ab885 server_cookie=7846dc72874c1437 in_seq=0 out_seq=0 2024-03-18T15:18:09.733 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.728+0000 7f6cb5600640 10 client.25415 ms_handle_connect on v2:172.21.15.33:6838/1903591535 2024-03-18T15:18:09.734 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.729+0000 7f6cb5600640 1 -- 172.21.15.19:0/1717605197 <== mds.0 v2:172.21.15.33:6838/1903591535 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b6121a32b0 con 0x55b6121f7cc0 2024-03-18T15:18:09.734 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.729+0000 7f6cb5600640 10 client.25415 handle_command_reply: tid=0 2024-03-18T15:18:09.735 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.729+0000 7f6cd1a08740 1 client.25415 shutdown 2024-03-18T15:18:09.735 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.729+0000 7f6cd1a08740 1 -- 172.21.15.19:0/1717605197 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b6121e1b50 msgr2=0x55b6121eb880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:09.735 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.729+0000 7f6cd1a08740 1 --2- 172.21.15.19:0/1717605197 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b6121e1b50 0x55b6121eb880 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x55b6121dd640 tx=0x7f6cb8009de0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.735 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.729+0000 7f6cd1a08740 1 -- 172.21.15.19:0/1717605197 shutdown_connections 2024-03-18T15:18:09.735 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.729+0000 7f6cd1a08740 1 --2- 172.21.15.19:0/1717605197 >> [v2:172.21.15.33:6838/1903591535,v1:172.21.15.33:6839/1903591535] conn(0x55b6121f7cc0 0x55b6121f9130 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.735 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.729+0000 7f6cd1a08740 1 --2- 172.21.15.19:0/1717605197 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b6121e6cc0 0x55b6121f2850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.735 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.729+0000 7f6cd1a08740 1 --2- 172.21.15.19:0/1717605197 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b6121e1b50 0x55b6121eb880 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.735 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.729+0000 7f6cd1a08740 1 --2- 172.21.15.19:0/1717605197 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55b6121e1160 0x55b6121eb340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.735 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.729+0000 7f6cd1a08740 1 -- 172.21.15.19:0/1717605197 >> 172.21.15.19:0/1717605197 conn(0x55b6121dc6d0 msgr2=0x55b6121dba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:18:09.735 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.729+0000 7f6cb5600640 0 client.25415 ms_handle_reset on v2:172.21.15.33:6838/1903591535 2024-03-18T15:18:09.735 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.730+0000 7f6cd1a08740 1 -- 172.21.15.19:0/1717605197 shutdown_connections 2024-03-18T15:18:09.735 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.730+0000 7f6cd1a08740 1 -- 172.21.15.19:0/1717605197 wait complete. 2024-03-18T15:18:09.736 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.730+0000 7f6cd1a08740 20 client.25415 trim_cache size 0 max 16384 2024-03-18T15:18:09.751 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.749+0000 7f6cd1a08740 1 Processor -- start 2024-03-18T15:18:09.751 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.750+0000 7f6cd1a08740 1 -- start start 2024-03-18T15:18:09.751 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.750+0000 7f6cd1a08740 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55b6121e1160 0x55b6121eb660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:09.752 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.750+0000 7f6cd1a08740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b6121e1b50 0x55b6121f2810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:09.752 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.750+0000 7f6cd1a08740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b6121e6cc0 0x55b6121db1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:09.752 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.750+0000 7f6cd1a08740 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x55b6121f7470 con 0x55b6121e1160 2024-03-18T15:18:09.752 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.750+0000 7f6cd1a08740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55b6121f7b40 con 0x55b6121e6cc0 2024-03-18T15:18:09.752 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.750+0000 7f6cd1a08740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55b6121f8210 con 0x55b6121e1b50 2024-03-18T15:18:09.752 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.750+0000 7f6cc7400640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55b6121e1160 0x55b6121eb660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:09.752 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.750+0000 7f6cc7e00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b6121e1b50 0x55b6121f2810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:09.752 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.750+0000 7f6cc7e00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b6121e1b50 0x55b6121f2810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.19:39554/0 (socket says 172.21.15.19:39554) 2024-03-18T15:18:09.752 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.750+0000 7f6cc7e00640 1 -- 172.21.15.19:0/2134193442 learned_addr learned my addr 172.21.15.19:0/2134193442 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:18:09.752 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.750+0000 7f6cc7400640 1 --2- 172.21.15.19:0/2134193442 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b6121e6cc0 0x55b6121db1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:09.753 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.750+0000 7f6cc7e00640 1 -- 172.21.15.19:0/2134193442 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b6121e6cc0 msgr2=0x55b6121db1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:18:09.753 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.750+0000 7f6cc7e00640 1 --2- 172.21.15.19:0/2134193442 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b6121e6cc0 0x55b6121db1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.753 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.750+0000 7f6cc7e00640 1 -- 172.21.15.19:0/2134193442 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55b6121e1160 msgr2=0x55b6121eb660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:18:09.753 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.750+0000 7f6cc7e00640 1 --2- 172.21.15.19:0/2134193442 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55b6121e1160 0x55b6121eb660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.753 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.751+0000 7f6cc7e00640 1 -- 172.21.15.19:0/2134193442 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b6121b2750 con 0x55b6121e1b50 2024-03-18T15:18:09.753 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.751+0000 7f6cc7e00640 1 --2- 172.21.15.19:0/2134193442 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b6121e1b50 0x55b6121f2810 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f6cb8009b80 tx=0x7f6cb8023440 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8c7ef34e13359a34 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:18:09.758 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.751+0000 7f6cb7e00640 1 -- 172.21.15.19:0/2134193442 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cb8027070 con 0x55b6121e1b50 2024-03-18T15:18:09.758 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.751+0000 7f6cb7e00640 1 -- 172.21.15.19:0/2134193442 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6cb8003e50 con 0x55b6121e1b50 2024-03-18T15:18:09.758 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.751+0000 7f6cb7e00640 1 -- 172.21.15.19:0/2134193442 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cb8023e20 con 0x55b6121e1b50 2024-03-18T15:18:09.758 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.752+0000 7f6cd1a08740 1 -- 172.21.15.19:0/2134193442 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b6121e1b50 msgr2=0x55b6121f2810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:09.758 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.752+0000 7f6cd1a08740 1 --2- 172.21.15.19:0/2134193442 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b6121e1b50 0x55b6121f2810 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f6cb8009b80 tx=0x7f6cb8023440 comp rx=0 tx=0).stop 2024-03-18T15:18:09.758 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.752+0000 7f6cd1a08740 1 -- 172.21.15.19:0/2134193442 shutdown_connections 2024-03-18T15:18:09.758 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.752+0000 7f6cd1a08740 1 --2- 172.21.15.19:0/2134193442 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b6121e6cc0 0x55b6121db1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.759 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.752+0000 7f6cd1a08740 1 --2- 172.21.15.19:0/2134193442 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b6121e1b50 0x55b6121f2810 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.759 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.752+0000 7f6cd1a08740 1 --2- 172.21.15.19:0/2134193442 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55b6121e1160 0x55b6121eb660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.759 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.752+0000 7f6cd1a08740 1 -- 172.21.15.19:0/2134193442 >> 172.21.15.19:0/2134193442 conn(0x55b6121dc6d0 msgr2=0x55b6121eaf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:18:09.759 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.752+0000 7f6cd1a08740 1 -- 172.21.15.19:0/2134193442 shutdown_connections 2024-03-18T15:18:09.759 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.752+0000 7f6cd1a08740 1 -- 172.21.15.19:0/2134193442 wait complete. 2024-03-18T15:18:09.759 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.752+0000 7f6cd1a08740 1 -- start start 2024-03-18T15:18:09.759 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.752+0000 7f6cd1a08740 1 Processor -- start 2024-03-18T15:18:09.759 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.753+0000 7f6cd1a08740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b6121e1160 0x55b6121ee9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:09.759 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.753+0000 7f6cd1a08740 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55b6121e1b50 0x55b6121eeee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:09.759 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.753+0000 7f6cd1a08740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b6121e6cc0 0x55b6121f17b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:09.760 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.753+0000 7f6cd1a08740 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x55b6121f8fb0 con 0x55b6121e1b50 2024-03-18T15:18:09.760 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.753+0000 7f6cd1a08740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55b6121f88e0 con 0x55b6121e6cc0 2024-03-18T15:18:09.760 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.753+0000 7f6cd1a08740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x55b6121f9680 con 0x55b6121e1160 2024-03-18T15:18:09.760 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.753+0000 7f6cd1a08740 10 client.0 _ll_register_callbacks cb 0x7f6cb8009cc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T15:18:09.760 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.753+0000 7f6cc7400640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b6121e1160 0x55b6121ee9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:09.760 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.753+0000 7f6cc7400640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b6121e1160 0x55b6121ee9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.19:39556/0 (socket says 172.21.15.19:39556) 2024-03-18T15:18:09.760 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.753+0000 7f6cc7400640 1 -- 172.21.15.19:0/3941287537 learned_addr learned my addr 172.21.15.19:0/3941287537 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:18:09.760 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.753+0000 7f6cc7e00640 1 --2- 172.21.15.19:0/3941287537 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55b6121e1b50 0x55b6121eeee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:09.760 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.753+0000 7f6cc7400640 1 --2- 172.21.15.19:0/3941287537 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b6121e6cc0 0x55b6121f17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:09.760 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.753+0000 7f6cc7400640 1 -- 172.21.15.19:0/3941287537 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b6121e6cc0 msgr2=0x55b6121f17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:18:09.760 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.753+0000 7f6cc7400640 1 --2- 172.21.15.19:0/3941287537 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b6121e6cc0 0x55b6121f17b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.760 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.753+0000 7f6cc7400640 1 -- 172.21.15.19:0/3941287537 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55b6121e1b50 msgr2=0x55b6121eeee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:18:09.761 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.753+0000 7f6cc7400640 1 --2- 172.21.15.19:0/3941287537 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55b6121e1b50 0x55b6121eeee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.761 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.753+0000 7f6cc7400640 1 -- 172.21.15.19:0/3941287537 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b6121f1d40 con 0x55b6121e1160 2024-03-18T15:18:09.761 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.754+0000 7f6cc7400640 1 --2- 172.21.15.19:0/3941287537 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b6121e1160 0x55b6121ee9a0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x55b6121f9870 tx=0x7f6cb000b4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=30be900f9ace4847 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:18:09.761 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.754+0000 7f6cb5600640 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-03-18T15:18:09.761 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.754+0000 7f6cb5600640 1 -- 172.21.15.19:0/3941287537 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cb001b040 con 0x55b6121e1160 2024-03-18T15:18:09.761 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.754+0000 7f6cd1a08740 1 -- 172.21.15.19:0/3941287537 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6cb8003e50 con 0x55b6121e1160 2024-03-18T15:18:09.761 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.754+0000 7f6cb5600640 1 -- 172.21.15.19:0/3941287537 <== mon.1 v2:172.21.15.104:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f6cb0005220 con 0x55b6121e1160 2024-03-18T15:18:09.761 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.754+0000 7f6cb5600640 1 -- 172.21.15.19:0/3941287537 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f6cb0005420 con 0x55b6121e1160 2024-03-18T15:18:09.761 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.754+0000 7f6cd1a08740 10 client.25435 fetch_fsmap learned FSMap version 26 2024-03-18T15:18:09.761 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.754+0000 7f6cd1a08740 1 -- 172.21.15.19:0/3941287537 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55b61225de40 con 0x55b6121e1160 2024-03-18T15:18:09.761 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.754+0000 7f6cb5600640 1 -- 172.21.15.19:0/3941287537 <== mon.1 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6cb8003e50 con 0x55b6121e1160 2024-03-18T15:18:09.761 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.755+0000 7f6cb5600640 1 -- 172.21.15.19:0/3941287537 <== mon.1 v2:172.21.15.104:3300/0 5 ==== fsmap(e 26) ==== 8362+0+0 (secure 0 0 0) 0x7f6cb00250c0 con 0x55b6121e1160 2024-03-18T15:18:09.761 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.755+0000 7f6cd1a08740 10 client.25435 fetch_fsmap finished waiting for FSMap version 26 2024-03-18T15:18:09.762 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.755+0000 7f6cd1a08740 10 client.25435 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-18T15:18:09.762 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.755+0000 7f6cd1a08740 20 client.25435 populate_metadata read hostname 'smithi019' 2024-03-18T15:18:09.762 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.755+0000 7f6cd1a08740 1 --2- 172.21.15.19:0/3941287537 >> [v2:172.21.15.33:6838/1903591535,v1:172.21.15.33:6839/1903591535] conn(0x55b612262a10 0x55b61225ee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:09.762 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.755+0000 7f6cd1a08740 4 client.25435 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-18T15:18:09.762 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.755+0000 7f6cd1a08740 1 -- 172.21.15.19:0/3941287537 --> [v2:172.21.15.33:6838/1903591535,v1:172.21.15.33:6839/1903591535] -- command(tid 0: {"prefix": "status"}) -- 0x55b612153190 con 0x55b612262a10 2024-03-18T15:18:09.762 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.755+0000 7f6cc7e00640 1 --2- 172.21.15.19:0/3941287537 >> [v2:172.21.15.33:6838/1903591535,v1:172.21.15.33:6839/1903591535] conn(0x55b612262a10 0x55b61225ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:09.762 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.755+0000 7f6cc7e00640 1 --2- 172.21.15.19:0/3941287537 >> [v2:172.21.15.33:6838/1903591535,v1:172.21.15.33:6839/1903591535] conn(0x55b612262a10 0x55b61225ee50 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9b0dc9f86058f5ec server_cookie=e2b43838c0e0e482 in_seq=0 out_seq=0 2024-03-18T15:18:09.762 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.755+0000 7f6cb5600640 10 client.25435 ms_handle_connect on v2:172.21.15.33:6838/1903591535 2024-03-18T15:18:09.762 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.756+0000 7f6cb5600640 1 -- 172.21.15.19:0/3941287537 <== mds.0 v2:172.21.15.33:6838/1903591535 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x55b612153190 con 0x55b612262a10 2024-03-18T15:18:09.762 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.756+0000 7f6cb5600640 10 client.25435 handle_command_reply: tid=0 2024-03-18T15:18:09.762 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.756+0000 7f6cd1a08740 1 client.25435 shutdown 2024-03-18T15:18:09.762 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.756+0000 7f6cd1a08740 1 -- 172.21.15.19:0/3941287537 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b6121e1160 msgr2=0x55b6121ee9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:09.762 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.756+0000 7f6cd1a08740 1 --2- 172.21.15.19:0/3941287537 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b6121e1160 0x55b6121ee9a0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x55b6121f9870 tx=0x7f6cb000b4a0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.762 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.756+0000 7f6cd1a08740 1 -- 172.21.15.19:0/3941287537 shutdown_connections 2024-03-18T15:18:09.763 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.756+0000 7f6cd1a08740 1 --2- 172.21.15.19:0/3941287537 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b6121e6cc0 0x55b6121f17b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.763 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.756+0000 7f6cd1a08740 1 --2- 172.21.15.19:0/3941287537 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55b6121e1b50 0x55b6121eeee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.763 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.756+0000 7f6cd1a08740 1 --2- 172.21.15.19:0/3941287537 >> [v2:172.21.15.33:6838/1903591535,v1:172.21.15.33:6839/1903591535] conn(0x55b612262a10 0x55b61225ee50 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.763 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.756+0000 7f6cd1a08740 1 --2- 172.21.15.19:0/3941287537 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55b6121e1160 0x55b6121ee9a0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.763 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.756+0000 7f6cb5600640 0 client.25435 ms_handle_reset on v2:172.21.15.33:6838/1903591535 2024-03-18T15:18:09.763 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.756+0000 7f6cd1a08740 1 -- 172.21.15.19:0/3941287537 >> 172.21.15.19:0/3941287537 conn(0x55b6121dc6d0 msgr2=0x55b6121df930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:18:09.763 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.756+0000 7f6cd1a08740 1 -- 172.21.15.19:0/3941287537 shutdown_connections 2024-03-18T15:18:09.763 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.756+0000 7f6cd1a08740 1 -- 172.21.15.19:0/3941287537 wait complete. 2024-03-18T15:18:09.763 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.756+0000 7f6cd1a08740 20 client.25435 trim_cache size 0 max 16384 2024-03-18T15:18:09.763 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-03-18T15:18:09.764 INFO:teuthology.orchestra.run.smithi019.stdout: "cluster_fsid": "d4dc2ba6-e538-11ee-95c9-87774f69a715", 2024-03-18T15:18:09.764 INFO:teuthology.orchestra.run.smithi019.stdout: "whoami": 0, 2024-03-18T15:18:09.764 INFO:teuthology.orchestra.run.smithi019.stdout: "id": 24560, 2024-03-18T15:18:09.764 INFO:teuthology.orchestra.run.smithi019.stdout: "want_state": "up:active", 2024-03-18T15:18:09.764 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "up:active", 2024-03-18T15:18:09.764 INFO:teuthology.orchestra.run.smithi019.stdout: "fs_name": "cephfs", 2024-03-18T15:18:09.764 INFO:teuthology.orchestra.run.smithi019.stdout: "rank_uptime": 178.640058181, 2024-03-18T15:18:09.764 INFO:teuthology.orchestra.run.smithi019.stdout: "mdsmap_epoch": 26, 2024-03-18T15:18:09.764 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap_epoch": 93, 2024-03-18T15:18:09.764 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap_epoch_barrier": 89, 2024-03-18T15:18:09.764 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 185.466801699 2024-03-18T15:18:09.764 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-03-18T15:18:09.765 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.757+0000 7f6ccec00640 1 -- 172.21.15.19:0/3912951805 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f6cac0a2800 msgr2=0x7f6cac0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:09.765 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.757+0000 7f6ccec00640 1 --2- 172.21.15.19:0/3912951805 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f6cac0a2800 0x7f6cac0a4cc0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f6cc00b9210 tx=0x7f6cc0005f90 comp rx=0 tx=0).stop 2024-03-18T15:18:09.765 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.757+0000 7f6ccec00640 1 -- 172.21.15.19:0/3912951805 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6cc81055d0 msgr2=0x7f6cc81a1250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:09.765 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.757+0000 7f6ccec00640 1 --2- 172.21.15.19:0/3912951805 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6cc81055d0 0x7f6cc81a1250 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f6cc0009780 tx=0x7f6cc0004270 comp rx=0 tx=0).stop 2024-03-18T15:18:09.765 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.757+0000 7f6ccec00640 1 -- 172.21.15.19:0/3912951805 shutdown_connections 2024-03-18T15:18:09.765 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.757+0000 7f6ccec00640 1 --2- 172.21.15.19:0/3912951805 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6cc81055d0 0x7f6cc81a1250 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.765 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.757+0000 7f6ccec00640 1 --2- 172.21.15.19:0/3912951805 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f6cac0a2800 0x7f6cac0a4cc0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.766 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.757+0000 7f6ccec00640 1 --2- 172.21.15.19:0/3912951805 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6cc8104c10 0x7f6cc819a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.766 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.757+0000 7f6ccec00640 1 --2- 172.21.15.19:0/3912951805 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6cc8103a20 0x7f6cc8199cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:09.766 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.757+0000 7f6ccec00640 1 -- 172.21.15.19:0/3912951805 >> 172.21.15.19:0/3912951805 conn(0x7f6cc80fef90 msgr2=0x7f6cc8107f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:18:09.766 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.757+0000 7f6ccec00640 1 -- 172.21.15.19:0/3912951805 shutdown_connections 2024-03-18T15:18:09.766 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:09.757+0000 7f6ccec00640 1 -- 172.21.15.19:0/3912951805 wait complete. 2024-03-18T15:18:09.777 INFO:teuthology.run_tasks:Running task kclient... 2024-03-18T15:18:09.791 INFO:tasks.kclient:Mounting kernel clients... 2024-03-18T15:18:09.791 INFO:tasks.kclient:config is {'client.0': {}} 2024-03-18T15:18:09.791 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T15:18:09.792 DEBUG:teuthology.orchestra.run.smithi019:> ip netns list 2024-03-18T15:18:09.844 INFO:teuthology.orchestra.run.smithi019.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-03-18T15:18:09.845 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T15:18:09.845 DEBUG:teuthology.orchestra.run.smithi019:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-18T15:18:09.922 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T15:18:09.922 DEBUG:teuthology.orchestra.run.smithi019:> sudo ip link delete ceph-brx 2024-03-18T15:18:10.042 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=crc', 'wsync'], 'syntax': 'v2', 'mount_subvol_num': 1} 2024-03-18T15:18:10.042 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-18T15:18:10.042 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-18T15:18:10.042 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 -- ceph fs ls 2024-03-18T15:18:10.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:10 smithi104 ceph-mon[36070]: pgmap v410: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T15:18:10.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:10 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:18:10.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:10 smithi019 ceph-mon[31522]: pgmap v410: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T15:18:10.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:10 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:18:10.498 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:18:10.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:10 smithi033 ceph-mon[36648]: pgmap v410: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T15:18:10.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:10 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:18:11.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:11 smithi104 ceph-mon[36070]: pgmap v411: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T15:18:11.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:11 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:18:11.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:11 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:18:11.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:11 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:18:11.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:11 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:18:11.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:11 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:18:11.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:11 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:18:11.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:11 smithi019 ceph-mon[31522]: pgmap v411: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T15:18:11.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:11 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:18:11.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:11 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:18:11.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:11 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:18:11.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:11 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:18:11.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:11 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:18:11.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:11 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:18:11.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:11 smithi033 ceph-mon[36648]: pgmap v411: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T15:18:11.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:11 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:18:11.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:11 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:18:11.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:11 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:18:11.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:11 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:18:11.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:11 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:18:11.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:11 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:18:12.663 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:12.661+0000 7fc45d800700 1 -- 172.21.15.19:0/1420255695 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc458104500 msgr2=0x7fc458104980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:12.663 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:12.661+0000 7fc45d800700 1 --2- 172.21.15.19:0/1420255695 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc458104500 0x7fc458104980 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fc44000b0f0 tx=0x7fc44000b400 comp rx=0 tx=0).stop 2024-03-18T15:18:12.663 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:12.662+0000 7fc45d800700 1 -- 172.21.15.19:0/1420255695 shutdown_connections 2024-03-18T15:18:12.663 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:12.662+0000 7fc45d800700 1 --2- 172.21.15.19:0/1420255695 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc458104ec0 0x7fc458111380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:12.663 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:12.662+0000 7fc45d800700 1 --2- 172.21.15.19:0/1420255695 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc45810a5b0 0x7fc45810a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:12.663 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:12.662+0000 7fc45d800700 1 --2- 172.21.15.19:0/1420255695 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc458104500 0x7fc458104980 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:12.664 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:12.662+0000 7fc45d800700 1 -- 172.21.15.19:0/1420255695 >> 172.21.15.19:0/1420255695 conn(0x7fc4580fffb0 msgr2=0x7fc4581023d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:18:12.664 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:12.662+0000 7fc45d800700 1 -- 172.21.15.19:0/1420255695 shutdown_connections 2024-03-18T15:18:12.664 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:12.662+0000 7fc45d800700 1 -- 172.21.15.19:0/1420255695 wait complete. 2024-03-18T15:18:12.664 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:12.663+0000 7fc45d800700 1 Processor -- start 2024-03-18T15:18:12.664 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:12.663+0000 7fc45d800700 1 -- start start 2024-03-18T15:18:12.664 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:12.664+0000 7fc45d800700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc458104500 0x7fc4581a0870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:12.666 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:12.664+0000 7fc45d800700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc458104ec0 0x7fc4581a0db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:12.666 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:12.664+0000 7fc45d800700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc45810a5b0 0x7fc45819aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:12.666 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:12.664+0000 7fc45d800700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fc458114840 con 0x7fc45810a5b0 2024-03-18T15:18:12.666 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:12.664+0000 7fc45d800700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc4581146c0 con 0x7fc458104ec0 2024-03-18T15:18:12.666 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:12.664+0000 7fc45d800700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fc4581149c0 con 0x7fc458104500 2024-03-18T15:18:12.666 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:12.664+0000 7fc44e000700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc458104ec0 0x7fc4581a0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:12.666 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:12.664+0000 7fc44e000700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc458104ec0 0x7fc4581a0db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:47678/0 (socket says 172.21.15.19:47678) 2024-03-18T15:18:12.666 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:12.664+0000 7fc456a00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc458104500 0x7fc4581a0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:12.666 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:12.664+0000 7fc44e000700 1 -- 172.21.15.19:0/3191534172 learned_addr learned my addr 172.21.15.19:0/3191534172 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:18:12.666 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:12.664+0000 7fc457400700 1 --2- 172.21.15.19:0/3191534172 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc45810a5b0 0x7fc45819aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:12.667 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:12.665+0000 7fc456a00700 1 -- 172.21.15.19:0/3191534172 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc458104ec0 msgr2=0x7fc4581a0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:12.667 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:12.665+0000 7fc456a00700 1 --2- 172.21.15.19:0/3191534172 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc458104ec0 0x7fc4581a0db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:12.667 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:12.665+0000 7fc456a00700 1 -- 172.21.15.19:0/3191534172 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc45810a5b0 msgr2=0x7fc45819aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:12.667 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:12.665+0000 7fc456a00700 1 --2- 172.21.15.19:0/3191534172 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc45810a5b0 0x7fc45819aa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:12.667 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:12.665+0000 7fc456a00700 1 -- 172.21.15.19:0/3191534172 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc45819b1e0 con 0x7fc458104500 2024-03-18T15:18:12.667 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:12.665+0000 7fc44e000700 1 --2- 172.21.15.19:0/3191534172 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc458104ec0 0x7fc4581a0db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:18:12.667 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:12.665+0000 7fc456a00700 1 --2- 172.21.15.19:0/3191534172 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc458104500 0x7fc4581a0870 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fc44800e9f0 tx=0x7fc44800ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:18:12.668 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:12.665+0000 7fc44fe00700 1 -- 172.21.15.19:0/3191534172 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc44800c9c0 con 0x7fc458104500 2024-03-18T15:18:12.668 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:12.665+0000 7fc44fe00700 1 -- 172.21.15.19:0/3191534172 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc44800cb60 con 0x7fc458104500 2024-03-18T15:18:12.668 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:12.665+0000 7fc44fe00700 1 -- 172.21.15.19:0/3191534172 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc448018680 con 0x7fc458104500 2024-03-18T15:18:12.668 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:12.665+0000 7fc45d800700 1 -- 172.21.15.19:0/3191534172 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc45819b4d0 con 0x7fc458104500 2024-03-18T15:18:12.668 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:12.666+0000 7fc45d800700 1 -- 172.21.15.19:0/3191534172 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4581a7740 con 0x7fc458104500 2024-03-18T15:18:12.668 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:12.667+0000 7fc45d800700 1 -- 172.21.15.19:0/3191534172 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc458068f90 con 0x7fc458104500 2024-03-18T15:18:12.669 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:12.669+0000 7fc44fe00700 1 -- 172.21.15.19:0/3191534172 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fc448047a60 con 0x7fc458104500 2024-03-18T15:18:12.671 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:12.670+0000 7fc44fe00700 1 --2- 172.21.15.19:0/3191534172 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fc43c0a2910 0x7fc43c0a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:12.671 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:12.670+0000 7fc44fe00700 1 -- 172.21.15.19:0/3191534172 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fc4480cf110 con 0x7fc458104500 2024-03-18T15:18:12.671 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:12.670+0000 7fc44e000700 1 --2- 172.21.15.19:0/3191534172 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fc43c0a2910 0x7fc43c0a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:12.672 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:12.671+0000 7fc44e000700 1 --2- 172.21.15.19:0/3191534172 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fc43c0a2910 0x7fc43c0a4dd0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fc440009e60 tx=0x7fc44000bdb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:18:12.674 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:12.674+0000 7fc44fe00700 1 -- 172.21.15.19:0/3191534172 <== mon.1 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc44809bd80 con 0x7fc458104500 2024-03-18T15:18:12.957 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:12.956+0000 7fc45d800700 1 -- 172.21.15.19:0/3191534172 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fc45802db70 con 0x7fc458104500 2024-03-18T15:18:12.958 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:12.957+0000 7fc44fe00700 1 -- 172.21.15.19:0/3191534172 <== mon.1 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v26) ==== 53+0+88 (secure 0 0 0) 0x7fc44809b680 con 0x7fc458104500 2024-03-18T15:18:12.958 INFO:teuthology.orchestra.run.smithi019.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-18T15:18:12.961 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:12.960+0000 7fc45d800700 1 -- 172.21.15.19:0/3191534172 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fc43c0a2910 msgr2=0x7fc43c0a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:12.961 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:12.960+0000 7fc45d800700 1 --2- 172.21.15.19:0/3191534172 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fc43c0a2910 0x7fc43c0a4dd0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fc440009e60 tx=0x7fc44000bdb0 comp rx=0 tx=0).stop 2024-03-18T15:18:12.961 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:12.960+0000 7fc45d800700 1 -- 172.21.15.19:0/3191534172 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc458104500 msgr2=0x7fc4581a0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:12.961 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:12.960+0000 7fc45d800700 1 --2- 172.21.15.19:0/3191534172 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc458104500 0x7fc4581a0870 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fc44800e9f0 tx=0x7fc44800ed00 comp rx=0 tx=0).stop 2024-03-18T15:18:12.961 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:12.960+0000 7fc45d800700 1 -- 172.21.15.19:0/3191534172 shutdown_connections 2024-03-18T15:18:12.962 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:12.960+0000 7fc45d800700 1 --2- 172.21.15.19:0/3191534172 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fc43c0a2910 0x7fc43c0a4dd0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:12.962 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:12.960+0000 7fc45d800700 1 --2- 172.21.15.19:0/3191534172 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc458104500 0x7fc4581a0870 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:12.962 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:12.960+0000 7fc45d800700 1 --2- 172.21.15.19:0/3191534172 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc45810a5b0 0x7fc45819aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:12.962 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:12.960+0000 7fc45d800700 1 --2- 172.21.15.19:0/3191534172 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc458104ec0 0x7fc4581a0db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:12.962 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:12.960+0000 7fc45d800700 1 -- 172.21.15.19:0/3191534172 >> 172.21.15.19:0/3191534172 conn(0x7fc4580fffb0 msgr2=0x7fc458101690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:18:12.962 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:12.960+0000 7fc45d800700 1 -- 172.21.15.19:0/3191534172 shutdown_connections 2024-03-18T15:18:12.962 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:12.960+0000 7fc45d800700 1 -- 172.21.15.19:0/3191534172 wait complete. 2024-03-18T15:18:13.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:13 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:18:13.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:13 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:18:13.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:13 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:18:13.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:13 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:18:13.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:13 smithi104 ceph-mon[36070]: pgmap v412: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T15:18:13.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:13 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/3191534172' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-18T15:18:13.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:13 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:18:13.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:13 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:18:13.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:13 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:18:13.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:13 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:18:13.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:13 smithi019 ceph-mon[31522]: pgmap v412: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T15:18:13.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:13 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/3191534172' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-18T15:18:13.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:13 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:18:13.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:13 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:18:13.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:13 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:18:13.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:13 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:18:13.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:13 smithi033 ceph-mon[36648]: pgmap v412: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T15:18:13.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:13 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/3191534172' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-18T15:18:14.409 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-18T15:18:14.409 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-18T15:18:14.409 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi019.front.sepia.ceph.com 2024-03-18T15:18:14.409 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-18T15:18:14.409 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-18T15:18:14.410 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-18T15:18:14.410 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-18T15:18:14.410 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-18T15:18:14.410 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-18T15:18:14.410 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T15:18:14.410 DEBUG:teuthology.orchestra.run.smithi019:> ip addr 2024-03-18T15:18:14.458 INFO:teuthology.orchestra.run.smithi019.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-18T15:18:14.458 INFO:teuthology.orchestra.run.smithi019.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-18T15:18:14.458 INFO:teuthology.orchestra.run.smithi019.stdout: inet 127.0.0.1/8 scope host lo 2024-03-18T15:18:14.459 INFO:teuthology.orchestra.run.smithi019.stdout: valid_lft forever preferred_lft forever 2024-03-18T15:18:14.459 INFO:teuthology.orchestra.run.smithi019.stdout: inet6 ::1/128 scope host 2024-03-18T15:18:14.459 INFO:teuthology.orchestra.run.smithi019.stdout: valid_lft forever preferred_lft forever 2024-03-18T15:18:14.459 INFO:teuthology.orchestra.run.smithi019.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T15:18:14.459 INFO:teuthology.orchestra.run.smithi019.stdout: link/ether 0c:c4:7a:6b:fd:12 brd ff:ff:ff:ff:ff:ff 2024-03-18T15:18:14.459 INFO:teuthology.orchestra.run.smithi019.stdout: altname enp7s0f0 2024-03-18T15:18:14.459 INFO:teuthology.orchestra.run.smithi019.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T15:18:14.459 INFO:teuthology.orchestra.run.smithi019.stdout: link/ether 0c:c4:7a:6b:fd:13 brd ff:ff:ff:ff:ff:ff 2024-03-18T15:18:14.459 INFO:teuthology.orchestra.run.smithi019.stdout: altname enp7s0f1 2024-03-18T15:18:14.459 INFO:teuthology.orchestra.run.smithi019.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-18T15:18:14.459 INFO:teuthology.orchestra.run.smithi019.stdout: link/ether 0c:c4:7a:bd:16:1e brd ff:ff:ff:ff:ff:ff 2024-03-18T15:18:14.459 INFO:teuthology.orchestra.run.smithi019.stdout: altname enp1s0f0 2024-03-18T15:18:14.459 INFO:teuthology.orchestra.run.smithi019.stdout: inet 172.21.15.19/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-18T15:18:14.459 INFO:teuthology.orchestra.run.smithi019.stdout: valid_lft 41834sec preferred_lft 41834sec 2024-03-18T15:18:14.459 INFO:teuthology.orchestra.run.smithi019.stdout: inet6 fe80::ec4:7aff:febd:161e/64 scope link noprefixroute 2024-03-18T15:18:14.460 INFO:teuthology.orchestra.run.smithi019.stdout: valid_lft forever preferred_lft forever 2024-03-18T15:18:14.460 INFO:teuthology.orchestra.run.smithi019.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T15:18:14.460 INFO:teuthology.orchestra.run.smithi019.stdout: link/ether 0c:c4:7a:bd:16:1f brd ff:ff:ff:ff:ff:ff 2024-03-18T15:18:14.460 INFO:teuthology.orchestra.run.smithi019.stdout: altname enp1s0f1 2024-03-18T15:18:14.461 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-18T15:18:14.461 DEBUG:teuthology.orchestra.run.smithi019:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-18T15:18:14.461 DEBUG:teuthology.orchestra.run.smithi019:> set -e 2024-03-18T15:18:14.461 DEBUG:teuthology.orchestra.run.smithi019:> sudo ip link add name ceph-brx type bridge 2024-03-18T15:18:14.461 DEBUG:teuthology.orchestra.run.smithi019:> sudo ip addr flush dev ceph-brx 2024-03-18T15:18:14.461 DEBUG:teuthology.orchestra.run.smithi019:> sudo ip link set ceph-brx up 2024-03-18T15:18:14.461 DEBUG:teuthology.orchestra.run.smithi019:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-18T15:18:14.461 DEBUG:teuthology.orchestra.run.smithi019:> ') 2024-03-18T15:18:14.552 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:14 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T15:18:14.705 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:14 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T15:18:14.716 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T15:18:14.717 DEBUG:teuthology.orchestra.run.smithi019:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-18T15:18:14.790 INFO:teuthology.orchestra.run.smithi019.stdout:1 2024-03-18T15:18:14.795 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T15:18:14.795 DEBUG:teuthology.orchestra.run.smithi019:> route 2024-03-18T15:18:14.837 INFO:teuthology.orchestra.run.smithi019.stdout:Kernel IP routing table 2024-03-18T15:18:14.838 INFO:teuthology.orchestra.run.smithi019.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-18T15:18:14.838 INFO:teuthology.orchestra.run.smithi019.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-18T15:18:14.838 INFO:teuthology.orchestra.run.smithi019.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-18T15:18:14.838 INFO:teuthology.orchestra.run.smithi019.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-18T15:18:14.838 DEBUG:teuthology.orchestra.run.smithi019:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-18T15:18:14.838 DEBUG:teuthology.orchestra.run.smithi019:> set -e 2024-03-18T15:18:14.838 DEBUG:teuthology.orchestra.run.smithi019:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-18T15:18:14.838 DEBUG:teuthology.orchestra.run.smithi019:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-18T15:18:14.839 DEBUG:teuthology.orchestra.run.smithi019:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-18T15:18:14.839 DEBUG:teuthology.orchestra.run.smithi019:> ') 2024-03-18T15:18:14.973 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:14 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T15:18:15.093 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:15 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T15:18:15.103 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T15:18:15.103 DEBUG:teuthology.orchestra.run.smithi019:> ip netns list 2024-03-18T15:18:15.144 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T15:18:15.145 DEBUG:teuthology.orchestra.run.smithi019:> ip netns list-id 2024-03-18T15:18:15.234 DEBUG:teuthology.orchestra.run.smithi019:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-18T15:18:15.234 DEBUG:teuthology.orchestra.run.smithi019:> set -e 2024-03-18T15:18:15.234 DEBUG:teuthology.orchestra.run.smithi019:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-18T15:18:15.235 DEBUG:teuthology.orchestra.run.smithi019:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-18T15:18:15.235 DEBUG:teuthology.orchestra.run.smithi019:> ') 2024-03-18T15:18:15.327 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:15 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T15:18:15.418 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:15 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T15:18:15.428 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-18T15:18:15.428 DEBUG:teuthology.orchestra.run.smithi019:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-18T15:18:15.428 DEBUG:teuthology.orchestra.run.smithi019:> set -e 2024-03-18T15:18:15.428 DEBUG:teuthology.orchestra.run.smithi019:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-18T15:18:15.428 DEBUG:teuthology.orchestra.run.smithi019:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-18T15:18:15.428 DEBUG:teuthology.orchestra.run.smithi019:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-18T15:18:15.428 DEBUG:teuthology.orchestra.run.smithi019:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-18T15:18:15.428 DEBUG:teuthology.orchestra.run.smithi019:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-18T15:18:15.429 DEBUG:teuthology.orchestra.run.smithi019:> ') 2024-03-18T15:18:15.534 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:15 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T15:18:15.611 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:15 smithi019 ceph-mon[31522]: pgmap v413: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T15:18:15.611 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:15 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:18:15.611 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:15 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:18:15.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:15 smithi104 ceph-mon[36070]: pgmap v413: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T15:18:15.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:15 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:18:15.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:15 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:18:15.737 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:15 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T15:18:15.746 DEBUG:teuthology.orchestra.run.smithi019:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-18T15:18:15.746 DEBUG:teuthology.orchestra.run.smithi019:> set -e 2024-03-18T15:18:15.746 DEBUG:teuthology.orchestra.run.smithi019:> sudo ip link set brx.0 up 2024-03-18T15:18:15.746 DEBUG:teuthology.orchestra.run.smithi019:> sudo ip link set dev brx.0 master ceph-brx 2024-03-18T15:18:15.746 DEBUG:teuthology.orchestra.run.smithi019:> ') 2024-03-18T15:18:15.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:15 smithi033 ceph-mon[36648]: pgmap v413: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T15:18:15.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:15 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:18:15.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:15 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:18:15.848 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:15 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T15:18:15.954 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:15 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T15:18:15.963 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-18T15:18:15.963 DEBUG:teuthology.orchestra.run.smithi019:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-18T15:18:16.008 INFO:teuthology.orchestra.run.smithi019.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-18T15:18:16.008 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-18T15:18:16.008 DEBUG:teuthology.orchestra.run.smithi019:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-18T15:18:16.089 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-03-18T15:18:16.089 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-03-18T15:18:16.090 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/volumes/qa/sv_1/6e5aa47e-98f1-4468-a9ee-8887701ea7d4 2024-03-18T15:18:16.090 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T15:18:16.090 DEBUG:teuthology.orchestra.run.smithi019:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/volumes/qa/sv_1/6e5aa47e-98f1-4468-a9ee-8887701ea7d4 /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=crc,wsync,nofallback 2024-03-18T15:18:16.253 INFO:teuthology.orchestra.run.smithi019.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=crc,wsync,nofallback 2024-03-18T15:18:16.253 INFO:teuthology.orchestra.run.smithi019.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=crc,wsync". 2024-03-18T15:18:16.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:16 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:18:16.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:16 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:18:16.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:16 smithi019 ceph-mon[31522]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:18:16.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:16 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:18:16.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:16 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:18:16.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:16 smithi104 ceph-mon[36070]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:18:16.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:16 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:18:16.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:16 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:18:16.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:16 smithi033 ceph-mon[36648]: from='mgr.14152 172.21.15.19:0/2077330457' entity='mgr.x' 2024-03-18T15:18:16.880 INFO:teuthology.orchestra.run.smithi019.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=crc,wsync,nofallback 2024-03-18T15:18:16.880 INFO:teuthology.orchestra.run.smithi019.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=crc,wsync". 2024-03-18T15:18:16.880 INFO:teuthology.orchestra.run.smithi019.stdout:mount.ceph: resolved to: "172.21.15.19:3300,172.21.15.33:3300,172.21.15.104:3300" 2024-03-18T15:18:16.880 INFO:teuthology.orchestra.run.smithi019.stdout:mount.ceph: trying mount with new device syntax: 0@d4dc2ba6-e538-11ee-95c9-87774f69a715.cephfs=/volumes/qa/sv_1/6e5aa47e-98f1-4468-a9ee-8887701ea7d4 2024-03-18T15:18:16.880 INFO:teuthology.orchestra.run.smithi019.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=crc,wsync,name=0,key=0,mon_addr=172.21.15.19:3300/172.21.15.33:3300/172.21.15.104:3300" will pass to kernel 2024-03-18T15:18:16.882 INFO:teuthology.orchestra.run.smithi019.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-18T15:18:16.882 INFO:teuthology.orchestra.run.smithi019.stdout: You just mounted a file system that supports labels which does not 2024-03-18T15:18:16.882 INFO:teuthology.orchestra.run.smithi019.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-18T15:18:16.882 INFO:teuthology.orchestra.run.smithi019.stdout: applications will generate AVC messages and not be allowed access to 2024-03-18T15:18:16.882 INFO:teuthology.orchestra.run.smithi019.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-18T15:18:16.886 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-18T15:18:16.886 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T15:18:16.886 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-18T15:18:16.969 DEBUG:teuthology.orchestra.run.smithi019:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-18T15:18:17.069 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:17 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-18T15:18:17.074 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:17 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T15:18:17.083 INFO:teuthology.orchestra.run.smithi019.stdout:client15327 2024-03-18T15:18:17.083 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-03-18T15:18:17.083 DEBUG:teuthology.orchestra.run.smithi019:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T15:18:17.133 DEBUG:teuthology.orchestra.run.smithi019:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-18T15:18:17.287 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:17 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-18T15:18:17.291 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:17 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T15:18:17.300 INFO:teuthology.orchestra.run.smithi019.stdout:client15327 2024-03-18T15:18:17.301 DEBUG:teuthology.orchestra.run.smithi019:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-18T15:18:17.406 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:17 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-18T15:18:17.412 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:17 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T15:18:17.421 INFO:teuthology.orchestra.run.smithi019.stdout:client15327 2024-03-18T15:18:17.421 DEBUG:teuthology.orchestra.run.smithi019:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715.client15327/status') 2024-03-18T15:18:17.519 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:17 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T15:18:17.562 INFO:teuthology.orchestra.run.smithi019.stdout:instance: client.15327 (3)192.168.0.1:0/409133069 2024-03-18T15:18:17.562 INFO:teuthology.orchestra.run.smithi019.stdout:blocklisted: false 2024-03-18T15:18:17.562 INFO:teuthology.orchestra.run.smithi019.stderr:0+1 records in 2024-03-18T15:18:17.562 INFO:teuthology.orchestra.run.smithi019.stderr:0+1 records out 2024-03-18T15:18:17.562 INFO:teuthology.orchestra.run.smithi019.stderr:69 bytes copied, 6.7052e-05 s, 1.0 MB/s 2024-03-18T15:18:17.565 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:17 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T15:18:17.576 INFO:teuthology.run_tasks:Running task exec... 2024-03-18T15:18:17.586 INFO:teuthology.task.exec:Executing custom commands... 2024-03-18T15:18:17.586 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi019.front.sepia.ceph.com 2024-03-18T15:18:17.586 DEBUG:teuthology.orchestra.run.smithi019:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate false' 2024-03-18T15:18:17.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:17 smithi019 ceph-mon[31522]: pgmap v414: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T15:18:17.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:17 smithi033 ceph-mon[36648]: pgmap v414: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T15:18:17.802 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:17.801+0000 7f6e1c200640 1 -- 172.21.15.19:0/1416537413 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6e141000c0 msgr2=0x7f6e141004e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:17.802 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:17.801+0000 7f6e1c200640 1 --2- 172.21.15.19:0/1416537413 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6e141000c0 0x7f6e141004e0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f6e08009900 tx=0x7f6e0802bc30 comp rx=0 tx=0).stop 2024-03-18T15:18:17.802 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:17.801+0000 7f6e1c200640 1 -- 172.21.15.19:0/1416537413 shutdown_connections 2024-03-18T15:18:17.802 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:17.801+0000 7f6e1c200640 1 --2- 172.21.15.19:0/1416537413 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e141013c0 0x7f6e14101840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:17.803 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:17.801+0000 7f6e1c200640 1 --2- 172.21.15.19:0/1416537413 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6e14100a20 0x7f6e14100e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:17.803 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:17.801+0000 7f6e1c200640 1 --2- 172.21.15.19:0/1416537413 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6e141000c0 0x7f6e141004e0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:17.803 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:17.801+0000 7f6e1c200640 1 -- 172.21.15.19:0/1416537413 >> 172.21.15.19:0/1416537413 conn(0x7f6e140fbcc0 msgr2=0x7f6e140fe100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:18:17.803 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:17.801+0000 7f6e1c200640 1 -- 172.21.15.19:0/1416537413 shutdown_connections 2024-03-18T15:18:17.803 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:17.801+0000 7f6e1c200640 1 -- 172.21.15.19:0/1416537413 wait complete. 2024-03-18T15:18:17.805 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:17.802+0000 7f6e1c200640 1 Processor -- start 2024-03-18T15:18:17.805 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:17.802+0000 7f6e1c200640 1 -- start start 2024-03-18T15:18:17.805 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:17.802+0000 7f6e1c200640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e141000c0 0x7f6e14199c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:17.805 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:17.803+0000 7f6e1c200640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6e14100a20 0x7f6e1419a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:17.805 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:17.803+0000 7f6e1c200640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6e141013c0 0x7f6e141a11f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:17.805 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:17.803+0000 7f6e1c200640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f6e1410f6e0 con 0x7f6e141013c0 2024-03-18T15:18:17.806 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:17.803+0000 7f6e1c200640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6e1410f560 con 0x7f6e14100a20 2024-03-18T15:18:17.806 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:17.803+0000 7f6e1c200640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f6e1410f860 con 0x7f6e141000c0 2024-03-18T15:18:17.806 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:17.803+0000 7f6e19800640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e141000c0 0x7f6e14199c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:17.806 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:17.803+0000 7f6e18e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6e14100a20 0x7f6e1419a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:17.806 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:17.803+0000 7f6e19800640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e141000c0 0x7f6e14199c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.19:54684/0 (socket says 172.21.15.19:54684) 2024-03-18T15:18:17.806 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:17.803+0000 7f6e19800640 1 -- 172.21.15.19:0/1833858265 learned_addr learned my addr 172.21.15.19:0/1833858265 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:18:17.806 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:17.803+0000 7f6e1a200640 1 --2- 172.21.15.19:0/1833858265 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6e141013c0 0x7f6e141a11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:17.806 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:17.803+0000 7f6e19800640 1 -- 172.21.15.19:0/1833858265 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6e14100a20 msgr2=0x7f6e1419a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:17.806 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:17.803+0000 7f6e19800640 1 --2- 172.21.15.19:0/1833858265 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6e14100a20 0x7f6e1419a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:17.806 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:17.803+0000 7f6e19800640 1 -- 172.21.15.19:0/1833858265 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6e141013c0 msgr2=0x7f6e141a11f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:17.806 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:17.803+0000 7f6e19800640 1 --2- 172.21.15.19:0/1833858265 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6e141013c0 0x7f6e141a11f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:17.806 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:17.803+0000 7f6e19800640 1 -- 172.21.15.19:0/1833858265 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e141a1890 con 0x7f6e141000c0 2024-03-18T15:18:17.807 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:17.804+0000 7f6e19800640 1 --2- 172.21.15.19:0/1833858265 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e141000c0 0x7f6e14199c80 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f6e08005ab0 tx=0x7f6e0802c6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:18:17.807 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:17.804+0000 7f6e12000640 1 -- 172.21.15.19:0/1833858265 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e08043020 con 0x7f6e141000c0 2024-03-18T15:18:17.807 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:17.804+0000 7f6e12000640 1 -- 172.21.15.19:0/1833858265 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e080054e0 con 0x7f6e141000c0 2024-03-18T15:18:17.807 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:17.804+0000 7f6e1c200640 1 -- 172.21.15.19:0/1833858265 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e141a1b20 con 0x7f6e141000c0 2024-03-18T15:18:17.807 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:17.804+0000 7f6e12000640 1 -- 172.21.15.19:0/1833858265 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e08003ce0 con 0x7f6e141000c0 2024-03-18T15:18:17.807 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:17.804+0000 7f6e1c200640 1 -- 172.21.15.19:0/1833858265 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e141a2070 con 0x7f6e141000c0 2024-03-18T15:18:17.807 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:17.805+0000 7f6e1c200640 1 -- 172.21.15.19:0/1833858265 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e1404f540 con 0x7f6e141000c0 2024-03-18T15:18:17.808 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:17.807+0000 7f6e12000640 1 -- 172.21.15.19:0/1833858265 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f6e08046020 con 0x7f6e141000c0 2024-03-18T15:18:17.809 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:17.808+0000 7f6e12000640 1 --2- 172.21.15.19:0/1833858265 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f6dfc0a28a0 0x7f6dfc0a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:17.809 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:17.808+0000 7f6e12000640 1 -- 172.21.15.19:0/1833858265 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f6e080f10c0 con 0x7f6e141000c0 2024-03-18T15:18:17.809 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:17.809+0000 7f6e18e00640 1 --2- 172.21.15.19:0/1833858265 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f6dfc0a28a0 0x7f6dfc0a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:17.810 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:17.809+0000 7f6e18e00640 1 --2- 172.21.15.19:0/1833858265 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f6dfc0a28a0 0x7f6dfc0a4d60 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f6e0000bbe0 tx=0x7f6e0000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:18:17.813 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:17.812+0000 7f6e12000640 1 -- 172.21.15.19:0/1833858265 <== mon.1 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6e0804d020 con 0x7f6e141000c0 2024-03-18T15:18:17.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:17 smithi104 ceph-mon[36070]: pgmap v414: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T15:18:18.062 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:18.061+0000 7f6e1c200640 1 -- 172.21.15.19:0/1833858265 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"} v 0) -- 0x7f6e140623e0 con 0x7f6e141000c0 2024-03-18T15:18:18.461 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:18.460+0000 7f6e12000640 1 -- 172.21.15.19:0/1833858265 <== mon.1 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]=0 v27) ==== 118+0+0 (secure 0 0 0) 0x7f6e080bdd30 con 0x7f6e141000c0 2024-03-18T15:18:18.465 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:18.464+0000 7f6e1c200640 1 -- 172.21.15.19:0/1833858265 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f6dfc0a28a0 msgr2=0x7f6dfc0a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:18.465 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:18.464+0000 7f6e1c200640 1 --2- 172.21.15.19:0/1833858265 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f6dfc0a28a0 0x7f6dfc0a4d60 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f6e0000bbe0 tx=0x7f6e0000b390 comp rx=0 tx=0).stop 2024-03-18T15:18:18.465 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:18.464+0000 7f6e1c200640 1 -- 172.21.15.19:0/1833858265 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e141000c0 msgr2=0x7f6e14199c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:18.465 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:18.464+0000 7f6e1c200640 1 --2- 172.21.15.19:0/1833858265 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e141000c0 0x7f6e14199c80 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f6e08005ab0 tx=0x7f6e0802c6c0 comp rx=0 tx=0).stop 2024-03-18T15:18:18.466 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:18.464+0000 7f6e1c200640 1 -- 172.21.15.19:0/1833858265 shutdown_connections 2024-03-18T15:18:18.466 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:18.464+0000 7f6e1c200640 1 --2- 172.21.15.19:0/1833858265 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6e141013c0 0x7f6e141a11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:18.466 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:18.464+0000 7f6e1c200640 1 --2- 172.21.15.19:0/1833858265 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f6dfc0a28a0 0x7f6dfc0a4d60 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:18.466 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:18.464+0000 7f6e1c200640 1 --2- 172.21.15.19:0/1833858265 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6e14100a20 0x7f6e1419a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:18.466 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:18.464+0000 7f6e1c200640 1 --2- 172.21.15.19:0/1833858265 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e141000c0 0x7f6e14199c80 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:18.466 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:18.464+0000 7f6e1c200640 1 -- 172.21.15.19:0/1833858265 >> 172.21.15.19:0/1833858265 conn(0x7f6e140fbcc0 msgr2=0x7f6e140fd7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:18:18.466 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:18.465+0000 7f6e1c200640 1 -- 172.21.15.19:0/1833858265 shutdown_connections 2024-03-18T15:18:18.466 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:18.465+0000 7f6e1c200640 1 -- 172.21.15.19:0/1833858265 wait complete. 2024-03-18T15:18:18.486 DEBUG:teuthology.orchestra.run.smithi019:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs qa random 0.10' 2024-03-18T15:18:18.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:18 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/1833858265' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-18T15:18:18.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:18 smithi019 ceph-mon[31522]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-18T15:18:18.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:18.714+0000 7f266fe00640 1 -- 172.21.15.19:0/1690247517 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f26700761a0 msgr2=0x7f26700765c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:18.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:18.714+0000 7f266fe00640 1 --2- 172.21.15.19:0/1690247517 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f26700761a0 0x7f26700765c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f2660009900 tx=0x7f266002bc40 comp rx=0 tx=0).stop 2024-03-18T15:18:18.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:18.714+0000 7f266fe00640 1 -- 172.21.15.19:0/1690247517 shutdown_connections 2024-03-18T15:18:18.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:18.714+0000 7f266fe00640 1 --2- 172.21.15.19:0/1690247517 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2670077d80 0x7f2670078620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:18.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:18.714+0000 7f266fe00640 1 --2- 172.21.15.19:0/1690247517 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f26700773e0 0x7f2670077840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:18.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:18.714+0000 7f266fe00640 1 --2- 172.21.15.19:0/1690247517 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f26700761a0 0x7f26700765c0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:18.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:18.714+0000 7f266fe00640 1 -- 172.21.15.19:0/1690247517 >> 172.21.15.19:0/1690247517 conn(0x7f26700fedf0 msgr2=0x7f2670101210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:18:18.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:18.714+0000 7f266fe00640 1 -- 172.21.15.19:0/1690247517 shutdown_connections 2024-03-18T15:18:18.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:18.714+0000 7f266fe00640 1 -- 172.21.15.19:0/1690247517 wait complete. 2024-03-18T15:18:18.718 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:18.715+0000 7f266fe00640 1 Processor -- start 2024-03-18T15:18:18.718 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:18.716+0000 7f266fe00640 1 -- start start 2024-03-18T15:18:18.718 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:18.716+0000 7f266fe00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f26700761a0 0x7f267019df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:18.719 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:18.716+0000 7f266fe00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f26700773e0 0x7f267019e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:18.719 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:18.716+0000 7f266fe00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2670077d80 0x7f26701a54c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:18.719 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:18.716+0000 7f266fe00640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f26701138f0 con 0x7f26700761a0 2024-03-18T15:18:18.719 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:18.716+0000 7f266fe00640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2670113770 con 0x7f2670077d80 2024-03-18T15:18:18.719 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:18.716+0000 7f266fe00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f2670113a70 con 0x7f26700773e0 2024-03-18T15:18:18.719 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:18.716+0000 7f2667e00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f26700773e0 0x7f267019e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:18.719 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:18.716+0000 7f266f400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2670077d80 0x7f26701a54c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:18.719 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:18.716+0000 7f2667e00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f26700773e0 0x7f267019e450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.19:54704/0 (socket says 172.21.15.19:54704) 2024-03-18T15:18:18.719 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:18.716+0000 7f2667e00640 1 -- 172.21.15.19:0/2396777897 learned_addr learned my addr 172.21.15.19:0/2396777897 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:18:18.719 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:18.716+0000 7f266ea00640 1 --2- 172.21.15.19:0/2396777897 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f26700761a0 0x7f267019df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:18.719 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:18.717+0000 7f2667e00640 1 -- 172.21.15.19:0/2396777897 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2670077d80 msgr2=0x7f26701a54c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:18.719 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:18.717+0000 7f2667e00640 1 --2- 172.21.15.19:0/2396777897 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2670077d80 0x7f26701a54c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:18.719 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:18.717+0000 7f2667e00640 1 -- 172.21.15.19:0/2396777897 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f26700761a0 msgr2=0x7f267019df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:18.719 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:18.717+0000 7f2667e00640 1 --2- 172.21.15.19:0/2396777897 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f26700761a0 0x7f267019df10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:18.720 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:18.717+0000 7f2667e00640 1 -- 172.21.15.19:0/2396777897 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26701a5b30 con 0x7f26700773e0 2024-03-18T15:18:18.720 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:18.717+0000 7f266ea00640 1 --2- 172.21.15.19:0/2396777897 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f26700761a0 0x7f267019df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T15:18:18.720 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:18.717+0000 7f2667e00640 1 --2- 172.21.15.19:0/2396777897 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f26700773e0 0x7f267019e450 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f265800d950 tx=0x7f265800de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:18:18.720 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:18.717+0000 7f2667400640 1 -- 172.21.15.19:0/2396777897 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2658017070 con 0x7f26700773e0 2024-03-18T15:18:18.720 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:18.717+0000 7f2667400640 1 -- 172.21.15.19:0/2396777897 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2658009c30 con 0x7f26700773e0 2024-03-18T15:18:18.720 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:18.717+0000 7f2667400640 1 -- 172.21.15.19:0/2396777897 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f265800f6b0 con 0x7f26700773e0 2024-03-18T15:18:18.720 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:18.717+0000 7f266fe00640 1 -- 172.21.15.19:0/2396777897 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26701a5e20 con 0x7f26700773e0 2024-03-18T15:18:18.720 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:18.718+0000 7f266fe00640 1 -- 172.21.15.19:0/2396777897 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26701a62e0 con 0x7f26700773e0 2024-03-18T15:18:18.720 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:18.719+0000 7f266fe00640 1 -- 172.21.15.19:0/2396777897 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26701101d0 con 0x7f26700773e0 2024-03-18T15:18:18.721 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:18.721+0000 7f2667400640 1 -- 172.21.15.19:0/2396777897 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f26580109d0 con 0x7f26700773e0 2024-03-18T15:18:18.723 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:18.722+0000 7f2667400640 1 --2- 172.21.15.19:0/2396777897 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f26540a27f0 0x7f26540a4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:18.723 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:18.722+0000 7f2667400640 1 -- 172.21.15.19:0/2396777897 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f26580ceac0 con 0x7f26700773e0 2024-03-18T15:18:18.723 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:18.723+0000 7f266ea00640 1 --2- 172.21.15.19:0/2396777897 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f26540a27f0 0x7f26540a4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:18.724 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:18.723+0000 7f266ea00640 1 --2- 172.21.15.19:0/2396777897 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f26540a27f0 0x7f26540a4cb0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f266002c250 tx=0x7f2660005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:18:18.727 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:18.726+0000 7f2667400640 1 -- 172.21.15.19:0/2396777897 <== mon.1 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2658009de0 con 0x7f26700773e0 2024-03-18T15:18:18.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:18 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/1833858265' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-18T15:18:18.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:18 smithi033 ceph-mon[36648]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-18T15:18:18.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:18 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/1833858265' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-18T15:18:18.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:18 smithi104 ceph-mon[36070]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-18T15:18:18.958 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:18.957+0000 7f266fe00640 1 -- 172.21.15.19:0/2396777897 --> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) -- 0x7f2670061d80 con 0x7f26540a27f0 2024-03-18T15:18:18.984 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:18.983+0000 7f2667400640 1 -- 172.21.15.19:0/2396777897 <== mgr.14152 v2:172.21.15.19:6800/2017714236 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+3 (secure 0 0 0) 0x7f2670061d80 con 0x7f26540a27f0 2024-03-18T15:18:18.984 INFO:teuthology.orchestra.run.smithi019.stdout:{} 2024-03-18T15:18:18.987 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:18.986+0000 7f266fe00640 1 -- 172.21.15.19:0/2396777897 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f26540a27f0 msgr2=0x7f26540a4cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:18.987 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:18.986+0000 7f266fe00640 1 --2- 172.21.15.19:0/2396777897 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f26540a27f0 0x7f26540a4cb0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f266002c250 tx=0x7f2660005fd0 comp rx=0 tx=0).stop 2024-03-18T15:18:18.987 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:18.986+0000 7f266fe00640 1 -- 172.21.15.19:0/2396777897 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f26700773e0 msgr2=0x7f267019e450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:18.987 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:18.986+0000 7f266fe00640 1 --2- 172.21.15.19:0/2396777897 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f26700773e0 0x7f267019e450 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f265800d950 tx=0x7f265800de10 comp rx=0 tx=0).stop 2024-03-18T15:18:18.987 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:18.986+0000 7f266fe00640 1 -- 172.21.15.19:0/2396777897 shutdown_connections 2024-03-18T15:18:18.988 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:18.986+0000 7f266fe00640 1 --2- 172.21.15.19:0/2396777897 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f26540a27f0 0x7f26540a4cb0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:18.988 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:18.986+0000 7f266fe00640 1 --2- 172.21.15.19:0/2396777897 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2670077d80 0x7f26701a54c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:18.988 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:18.986+0000 7f266fe00640 1 --2- 172.21.15.19:0/2396777897 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f26700773e0 0x7f267019e450 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:18.988 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:18.986+0000 7f266fe00640 1 --2- 172.21.15.19:0/2396777897 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f26700761a0 0x7f267019df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:18.988 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:18.986+0000 7f266fe00640 1 -- 172.21.15.19:0/2396777897 >> 172.21.15.19:0/2396777897 conn(0x7f26700fedf0 msgr2=0x7f26701008f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:18:18.988 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:18.986+0000 7f266fe00640 1 -- 172.21.15.19:0/2396777897 shutdown_connections 2024-03-18T15:18:18.988 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:18.986+0000 7f266fe00640 1 -- 172.21.15.19:0/2396777897 wait complete. 2024-03-18T15:18:19.007 INFO:teuthology.run_tasks:Running task check-counter... 2024-03-18T15:18:19.017 INFO:teuthology.run_tasks:Running task exec... 2024-03-18T15:18:19.055 INFO:teuthology.task.exec:Executing custom commands... 2024-03-18T15:18:19.056 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi019.front.sepia.ceph.com 2024-03-18T15:18:19.056 DEBUG:teuthology.orchestra.run.smithi019:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2024-03-18T15:18:19.275 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:19.273+0000 7f2c73e00640 1 -- 172.21.15.19:0/175559338 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2c6c108f90 msgr2=0x7f2c6c109410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:19.275 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:19.273+0000 7f2c73e00640 1 --2- 172.21.15.19:0/175559338 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2c6c108f90 0x7f2c6c109410 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f2c58009900 tx=0x7f2c5802bc40 comp rx=0 tx=0).stop 2024-03-18T15:18:19.275 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:19.273+0000 7f2c73e00640 1 -- 172.21.15.19:0/175559338 shutdown_connections 2024-03-18T15:18:19.275 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:19.273+0000 7f2c73e00640 1 --2- 172.21.15.19:0/175559338 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2c6c109950 0x7f2c6c110270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:19.275 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:19.273+0000 7f2c73e00640 1 --2- 172.21.15.19:0/175559338 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2c6c108f90 0x7f2c6c109410 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:19.275 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:19.273+0000 7f2c73e00640 1 --2- 172.21.15.19:0/175559338 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2c6c107e40 0x7f2c6c108260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:19.275 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:19.273+0000 7f2c73e00640 1 -- 172.21.15.19:0/175559338 >> 172.21.15.19:0/175559338 conn(0x7f2c6c076df0 msgr2=0x7f2c6c079210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:18:19.275 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:19.273+0000 7f2c73e00640 1 -- 172.21.15.19:0/175559338 shutdown_connections 2024-03-18T15:18:19.275 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:19.273+0000 7f2c73e00640 1 -- 172.21.15.19:0/175559338 wait complete. 2024-03-18T15:18:19.278 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:19.274+0000 7f2c73e00640 1 Processor -- start 2024-03-18T15:18:19.278 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:19.275+0000 7f2c73e00640 1 -- start start 2024-03-18T15:18:19.278 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:19.275+0000 7f2c73e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2c6c107e40 0x7f2c6c19e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:19.278 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:19.275+0000 7f2c73e00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2c6c108f90 0x7f2c6c19e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:19.278 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:19.275+0000 7f2c73e00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2c6c109950 0x7f2c6c1a5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:19.278 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:19.275+0000 7f2c73e00640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f2c6c113b10 con 0x7f2c6c109950 2024-03-18T15:18:19.278 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:19.275+0000 7f2c73e00640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2c6c113990 con 0x7f2c6c107e40 2024-03-18T15:18:19.278 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:19.275+0000 7f2c73e00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f2c6c113c90 con 0x7f2c6c108f90 2024-03-18T15:18:19.278 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:19.275+0000 7f2c71400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2c6c107e40 0x7f2c6c19e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:19.278 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:19.275+0000 7f2c70a00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2c6c108f90 0x7f2c6c19e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:19.278 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:19.275+0000 7f2c71400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2c6c107e40 0x7f2c6c19e0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:47740/0 (socket says 172.21.15.19:47740) 2024-03-18T15:18:19.279 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:19.275+0000 7f2c71400640 1 -- 172.21.15.19:0/387614119 learned_addr learned my addr 172.21.15.19:0/387614119 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:18:19.279 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:19.275+0000 7f2c71e00640 1 --2- 172.21.15.19:0/387614119 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2c6c109950 0x7f2c6c1a5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:19.279 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:19.276+0000 7f2c70a00640 1 -- 172.21.15.19:0/387614119 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2c6c107e40 msgr2=0x7f2c6c19e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:19.279 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:19.276+0000 7f2c70a00640 1 --2- 172.21.15.19:0/387614119 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2c6c107e40 0x7f2c6c19e0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:19.279 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:19.276+0000 7f2c70a00640 1 -- 172.21.15.19:0/387614119 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2c6c109950 msgr2=0x7f2c6c1a5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:19.279 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:19.276+0000 7f2c70a00640 1 --2- 172.21.15.19:0/387614119 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2c6c109950 0x7f2c6c1a5670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:19.279 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:19.276+0000 7f2c70a00640 1 -- 172.21.15.19:0/387614119 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c6c1a5ce0 con 0x7f2c6c108f90 2024-03-18T15:18:19.279 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:19.276+0000 7f2c71400640 1 --2- 172.21.15.19:0/387614119 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2c6c107e40 0x7f2c6c19e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:18:19.279 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:19.276+0000 7f2c70a00640 1 --2- 172.21.15.19:0/387614119 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2c6c108f90 0x7f2c6c19e600 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f2c58009230 tx=0x7f2c58005d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:18:19.279 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:19.276+0000 7f2c6a000640 1 -- 172.21.15.19:0/387614119 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c58041070 con 0x7f2c6c108f90 2024-03-18T15:18:19.279 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:19.276+0000 7f2c6a000640 1 -- 172.21.15.19:0/387614119 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2c580044d0 con 0x7f2c6c108f90 2024-03-18T15:18:19.279 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:19.276+0000 7f2c73e00640 1 -- 172.21.15.19:0/387614119 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c6c1a5f70 con 0x7f2c6c108f90 2024-03-18T15:18:19.279 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:19.276+0000 7f2c6a000640 1 -- 172.21.15.19:0/387614119 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c58034440 con 0x7f2c6c108f90 2024-03-18T15:18:19.279 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:19.276+0000 7f2c73e00640 1 -- 172.21.15.19:0/387614119 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c6c1a6370 con 0x7f2c6c108f90 2024-03-18T15:18:19.280 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:19.277+0000 7f2c73e00640 1 -- 172.21.15.19:0/387614119 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c6c10f150 con 0x7f2c6c108f90 2024-03-18T15:18:19.280 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:19.279+0000 7f2c6a000640 1 -- 172.21.15.19:0/387614119 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f2c58044070 con 0x7f2c6c108f90 2024-03-18T15:18:19.282 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:19.281+0000 7f2c6a000640 1 --2- 172.21.15.19:0/387614119 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f2c540a28a0 0x7f2c540a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:19.282 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:19.281+0000 7f2c6a000640 1 -- 172.21.15.19:0/387614119 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f2c580f08f0 con 0x7f2c6c108f90 2024-03-18T15:18:19.282 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:19.281+0000 7f2c71400640 1 --2- 172.21.15.19:0/387614119 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f2c540a28a0 0x7f2c540a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:19.282 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:19.282+0000 7f2c71400640 1 --2- 172.21.15.19:0/387614119 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f2c540a28a0 0x7f2c540a4d60 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f2c6000e840 tx=0x7f2c60009800 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:18:19.290 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:19.289+0000 7f2c6a000640 1 -- 172.21.15.19:0/387614119 <== mon.1 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2c580bd560 con 0x7f2c6c108f90 2024-03-18T15:18:19.460 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:18:18 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:18:18.966+0000 7f4486a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T15:18:19.460 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:18:18 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:18:18.966+0000 7f4486a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T15:18:19.460 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:18:18 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:18:18.966+0000 7f4486a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T15:18:19.460 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:18:18 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:18:18.966+0000 7f4486a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T15:18:19.460 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:18:18 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:18:18.966+0000 7f4486a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T15:18:19.549 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:19.548+0000 7f2c73e00640 1 -- 172.21.15.19:0/387614119 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7f2c6c062410 con 0x7f2c6c108f90 2024-03-18T15:18:19.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:19 smithi019 ceph-mon[31522]: pgmap v415: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-18T15:18:19.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:19 smithi019 ceph-mon[31522]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-03-18T15:18:19.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:19 smithi019 ceph-mon[31522]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T15:18:19.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:19 smithi019 ceph-mon[31522]: from='client.25453 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:18:19.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:19 smithi033 ceph-mon[36648]: pgmap v415: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-18T15:18:19.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:19 smithi033 ceph-mon[36648]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-03-18T15:18:19.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:19 smithi033 ceph-mon[36648]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T15:18:19.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:19 smithi033 ceph-mon[36648]: from='client.25453 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:18:19.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:19 smithi104 ceph-mon[36070]: pgmap v415: 129 pgs: 129 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-18T15:18:19.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:19 smithi104 ceph-mon[36070]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-03-18T15:18:19.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:19 smithi104 ceph-mon[36070]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T15:18:19.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:19 smithi104 ceph-mon[36070]: from='client.25453 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:18:20.486 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:20.484+0000 7f2c6a000640 1 -- 172.21.15.19:0/387614119 <== mon.1 v2:172.21.15.104:3300/0 7 ==== mgrmap(e 19) ==== 137307+0+0 (secure 0 0 0) 0x7f2c580b94d0 con 0x7f2c6c108f90 2024-03-18T15:18:20.487 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:20.486+0000 7f2c6a000640 1 -- 172.21.15.19:0/387614119 <== mon.1 v2:172.21.15.104:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v19) ==== 92+0+0 (secure 0 0 0) 0x7f2c58099960 con 0x7f2c6c108f90 2024-03-18T15:18:20.493 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:20.491+0000 7f2c73e00640 1 -- 172.21.15.19:0/387614119 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f2c540a28a0 msgr2=0x7f2c540a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:20.493 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:20.491+0000 7f2c73e00640 1 --2- 172.21.15.19:0/387614119 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f2c540a28a0 0x7f2c540a4d60 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f2c6000e840 tx=0x7f2c60009800 comp rx=0 tx=0).stop 2024-03-18T15:18:20.493 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:20.491+0000 7f2c73e00640 1 -- 172.21.15.19:0/387614119 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2c6c108f90 msgr2=0x7f2c6c19e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:20.493 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:20.491+0000 7f2c73e00640 1 --2- 172.21.15.19:0/387614119 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2c6c108f90 0x7f2c6c19e600 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f2c58009230 tx=0x7f2c58005d50 comp rx=0 tx=0).stop 2024-03-18T15:18:20.493 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:20.492+0000 7f2c73e00640 1 -- 172.21.15.19:0/387614119 shutdown_connections 2024-03-18T15:18:20.493 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:20.492+0000 7f2c73e00640 1 --2- 172.21.15.19:0/387614119 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2c6c109950 0x7f2c6c1a5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:20.493 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:20.492+0000 7f2c73e00640 1 --2- 172.21.15.19:0/387614119 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2c6c108f90 0x7f2c6c19e600 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:20.493 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:20.492+0000 7f2c73e00640 1 --2- 172.21.15.19:0/387614119 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f2c540a28a0 0x7f2c540a4d60 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:20.493 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:20.492+0000 7f2c73e00640 1 --2- 172.21.15.19:0/387614119 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2c6c107e40 0x7f2c6c19e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:20.493 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:20.492+0000 7f2c73e00640 1 -- 172.21.15.19:0/387614119 >> 172.21.15.19:0/387614119 conn(0x7f2c6c076df0 msgr2=0x7f2c6c10c320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:18:20.493 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:20.492+0000 7f2c73e00640 1 -- 172.21.15.19:0/387614119 shutdown_connections 2024-03-18T15:18:20.494 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:20.492+0000 7f2c73e00640 1 -- 172.21.15.19:0/387614119 wait complete. 2024-03-18T15:18:20.513 DEBUG:teuthology.orchestra.run.smithi019:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/allow_m_granularity true' 2024-03-18T15:18:20.711 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:20.710+0000 7f6e4e600640 1 -- 172.21.15.19:0/1494616609 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6e48102a00 msgr2=0x7f6e48102e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:20.711 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:20.710+0000 7f6e4e600640 1 --2- 172.21.15.19:0/1494616609 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6e48102a00 0x7f6e48102e80 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f6e34009900 tx=0x7f6e3402bc30 comp rx=0 tx=0).stop 2024-03-18T15:18:20.711 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:20.710+0000 7f6e4e600640 1 -- 172.21.15.19:0/1494616609 shutdown_connections 2024-03-18T15:18:20.711 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:20.710+0000 7f6e4e600640 1 --2- 172.21.15.19:0/1494616609 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6e481033c0 0x7f6e48109c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:20.712 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:20.710+0000 7f6e4e600640 1 --2- 172.21.15.19:0/1494616609 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6e48102a00 0x7f6e48102e80 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:20.712 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:20.710+0000 7f6e4e600640 1 --2- 172.21.15.19:0/1494616609 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e48101810 0x7f6e48101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:20.712 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:20.710+0000 7f6e4e600640 1 -- 172.21.15.19:0/1494616609 >> 172.21.15.19:0/1494616609 conn(0x7f6e480fcda0 msgr2=0x7f6e480ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:18:20.712 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:20.710+0000 7f6e4e600640 1 -- 172.21.15.19:0/1494616609 shutdown_connections 2024-03-18T15:18:20.712 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:20.710+0000 7f6e4e600640 1 -- 172.21.15.19:0/1494616609 wait complete. 2024-03-18T15:18:20.713 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:20.711+0000 7f6e4e600640 1 Processor -- start 2024-03-18T15:18:20.713 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:20.711+0000 7f6e4e600640 1 -- start start 2024-03-18T15:18:20.713 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:20.712+0000 7f6e4e600640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6e48101810 0x7f6e48075590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:20.713 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:20.712+0000 7f6e4e600640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e48102a00 0x7f6e48073bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:20.713 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:20.712+0000 7f6e4e600640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6e481033c0 0x7f6e48074110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:20.714 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:20.712+0000 7f6e4e600640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f6e4810d490 con 0x7f6e48101810 2024-03-18T15:18:20.714 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:20.712+0000 7f6e4e600640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6e4810d310 con 0x7f6e481033c0 2024-03-18T15:18:20.714 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:20.712+0000 7f6e4e600640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f6e4810d610 con 0x7f6e48102a00 2024-03-18T15:18:20.714 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:20.712+0000 7f6e46a00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e48102a00 0x7f6e48073bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:20.714 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:20.712+0000 7f6e47e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6e481033c0 0x7f6e48074110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:20.714 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:20.712+0000 7f6e46a00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e48102a00 0x7f6e48073bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.19:54778/0 (socket says 172.21.15.19:54778) 2024-03-18T15:18:20.714 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:20.712+0000 7f6e46a00640 1 -- 172.21.15.19:0/2436700385 learned_addr learned my addr 172.21.15.19:0/2436700385 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:18:20.714 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:20.712+0000 7f6e47400640 1 --2- 172.21.15.19:0/2436700385 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6e48101810 0x7f6e48075590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:20.714 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:20.712+0000 7f6e47e00640 1 -- 172.21.15.19:0/2436700385 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e48102a00 msgr2=0x7f6e48073bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:20.714 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:20.712+0000 7f6e47e00640 1 --2- 172.21.15.19:0/2436700385 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e48102a00 0x7f6e48073bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:20.714 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:20.712+0000 7f6e47e00640 1 -- 172.21.15.19:0/2436700385 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6e48101810 msgr2=0x7f6e48075590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:20.714 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:20.712+0000 7f6e47e00640 1 --2- 172.21.15.19:0/2436700385 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6e48101810 0x7f6e48075590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:20.714 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:20.712+0000 7f6e47e00640 1 -- 172.21.15.19:0/2436700385 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e480749a0 con 0x7f6e481033c0 2024-03-18T15:18:20.715 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:20.712+0000 7f6e46a00640 1 --2- 172.21.15.19:0/2436700385 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e48102a00 0x7f6e48073bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:18:20.715 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:20.712+0000 7f6e47400640 1 --2- 172.21.15.19:0/2436700385 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6e48101810 0x7f6e48075590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T15:18:20.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:20.713+0000 7f6e47e00640 1 --2- 172.21.15.19:0/2436700385 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6e481033c0 0x7f6e48074110 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f6e4000ec50 tx=0x7f6e4000c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:18:20.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:20.713+0000 7f6e3fe00640 1 -- 172.21.15.19:0/2436700385 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e40003b70 con 0x7f6e481033c0 2024-03-18T15:18:20.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:20.713+0000 7f6e3fe00640 1 -- 172.21.15.19:0/2436700385 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e40003d10 con 0x7f6e481033c0 2024-03-18T15:18:20.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:20.713+0000 7f6e3fe00640 1 -- 172.21.15.19:0/2436700385 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e40010890 con 0x7f6e481033c0 2024-03-18T15:18:20.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:20.713+0000 7f6e4e600640 1 -- 172.21.15.19:0/2436700385 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e481a1b90 con 0x7f6e481033c0 2024-03-18T15:18:20.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:20.713+0000 7f6e4e600640 1 -- 172.21.15.19:0/2436700385 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e481a2060 con 0x7f6e481033c0 2024-03-18T15:18:20.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:20.715+0000 7f6e4e600640 1 -- 172.21.15.19:0/2436700385 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e4804f540 con 0x7f6e481033c0 2024-03-18T15:18:20.717 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:20.716+0000 7f6e3fe00640 1 -- 172.21.15.19:0/2436700385 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 19) ==== 137307+0+0 (secure 0 0 0) 0x7f6e40003fc0 con 0x7f6e481033c0 2024-03-18T15:18:20.718 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:20.718+0000 7f6e3fe00640 1 --2- 172.21.15.19:0/2436700385 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f6e2c0a28f0 0x7f6e2c0a4db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:20.719 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:20.718+0000 7f6e47400640 1 -- 172.21.15.19:0/2436700385 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f6e2c0a28f0 msgr2=0x7f6e2c0a4db0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.19:6800/2017714236 2024-03-18T15:18:20.719 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:20.718+0000 7f6e3fe00640 1 -- 172.21.15.19:0/2436700385 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f6e400cf590 con 0x7f6e481033c0 2024-03-18T15:18:20.719 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:20.718+0000 7f6e47400640 1 --2- 172.21.15.19:0/2436700385 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f6e2c0a28f0 0x7f6e2c0a4db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T15:18:20.722 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:20.722+0000 7f6e3fe00640 1 -- 172.21.15.19:0/2436700385 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6e4009c200 con 0x7f6e481033c0 2024-03-18T15:18:20.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:20 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/387614119' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-18T15:18:20.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:20 smithi033 ceph-mon[36648]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-18T15:18:20.757 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Mar 18 15:18:20 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-y[38034]: ignoring --setuser ceph since I am not root 2024-03-18T15:18:20.757 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Mar 18 15:18:20 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-y[38034]: ignoring --setgroup ceph since I am not root 2024-03-18T15:18:20.757 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Mar 18 15:18:20 smithi033 ceph-mgr[38038]: -- 172.21.15.33:0/2028532363 <== mon.2 v2:172.21.15.33:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x556a50d345a0 con 0x556a50d08800 2024-03-18T15:18:20.920 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:20.919+0000 7f6e47400640 1 -- 172.21.15.19:0/2436700385 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f6e2c0a28f0 msgr2=0x7f6e2c0a4db0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.19:6800/2017714236 2024-03-18T15:18:20.920 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:20.919+0000 7f6e47400640 1 --2- 172.21.15.19:0/2436700385 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f6e2c0a28f0 0x7f6e2c0a4db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T15:18:20.930 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:20.929+0000 7f6e4e600640 1 -- 172.21.15.19:0/2436700385 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7f6e48062410 con 0x7f6e481033c0 2024-03-18T15:18:20.938 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:20.937+0000 7f6e3fe00640 1 -- 172.21.15.19:0/2436700385 <== mon.2 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v28)=0 v28) ==== 138+0+0 (secure 0 0 0) 0x7f6e4009bb00 con 0x7f6e481033c0 2024-03-18T15:18:20.941 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:20.940+0000 7f6e4e600640 1 -- 172.21.15.19:0/2436700385 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f6e2c0a28f0 msgr2=0x7f6e2c0a4db0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T15:18:20.941 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:20.940+0000 7f6e4e600640 1 --2- 172.21.15.19:0/2436700385 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f6e2c0a28f0 0x7f6e2c0a4db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:20.942 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:20.940+0000 7f6e4e600640 1 -- 172.21.15.19:0/2436700385 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6e481033c0 msgr2=0x7f6e48074110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:20.942 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:20.940+0000 7f6e4e600640 1 --2- 172.21.15.19:0/2436700385 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6e481033c0 0x7f6e48074110 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f6e4000ec50 tx=0x7f6e4000c6a0 comp rx=0 tx=0).stop 2024-03-18T15:18:20.942 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:20.941+0000 7f6e4e600640 1 -- 172.21.15.19:0/2436700385 shutdown_connections 2024-03-18T15:18:20.942 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:20.941+0000 7f6e4e600640 1 --2- 172.21.15.19:0/2436700385 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f6e2c0a28f0 0x7f6e2c0a4db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:20.942 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:20.941+0000 7f6e4e600640 1 --2- 172.21.15.19:0/2436700385 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6e481033c0 0x7f6e48074110 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:20.942 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:20.941+0000 7f6e4e600640 1 --2- 172.21.15.19:0/2436700385 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e48102a00 0x7f6e48073bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:20.942 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:20.941+0000 7f6e4e600640 1 --2- 172.21.15.19:0/2436700385 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6e48101810 0x7f6e48075590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:20.942 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:20.941+0000 7f6e4e600640 1 -- 172.21.15.19:0/2436700385 >> 172.21.15.19:0/2436700385 conn(0x7f6e480fcda0 msgr2=0x7f6e48105d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:18:20.942 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:20.941+0000 7f6e4e600640 1 -- 172.21.15.19:0/2436700385 shutdown_connections 2024-03-18T15:18:20.943 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:20.941+0000 7f6e4e600640 1 -- 172.21.15.19:0/2436700385 wait complete. 2024-03-18T15:18:20.944 INFO:journalctl@ceph.mgr.z.smithi104.stdout:Mar 18 15:18:20 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-z[37936]: ignoring --setuser ceph since I am not root 2024-03-18T15:18:20.944 INFO:journalctl@ceph.mgr.z.smithi104.stdout:Mar 18 15:18:20 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-z[37936]: ignoring --setgroup ceph since I am not root 2024-03-18T15:18:20.944 INFO:journalctl@ceph.mgr.z.smithi104.stdout:Mar 18 15:18:20 smithi104 ceph-mgr[37940]: -- 172.21.15.104:0/2535066772 <== mon.1 v2:172.21.15.104:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55abb57c25a0 con 0x55abb5796800 2024-03-18T15:18:20.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:20 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/387614119' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-18T15:18:20.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:20 smithi104 ceph-mon[36070]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-18T15:18:20.956 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:18:20 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: ignoring --setuser ceph since I am not root 2024-03-18T15:18:20.956 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:18:20 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: ignoring --setgroup ceph since I am not root 2024-03-18T15:18:20.957 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:20 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/387614119' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-18T15:18:20.957 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:20 smithi019 ceph-mon[31522]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-18T15:18:20.959 DEBUG:teuthology.orchestra.run.smithi019:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/dump_on_update true' 2024-03-18T15:18:21.151 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.149+0000 7f0dfb200640 1 -- 172.21.15.19:0/4289833120 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0dfc101650 msgr2=0x7f0dfc101a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:21.151 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.149+0000 7f0dfb200640 1 --2- 172.21.15.19:0/4289833120 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0dfc101650 0x7f0dfc101a70 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f0dec009ca0 tx=0x7f0dec02c100 comp rx=0 tx=0).stop 2024-03-18T15:18:21.151 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.150+0000 7f0dfb200640 1 -- 172.21.15.19:0/4289833120 shutdown_connections 2024-03-18T15:18:21.151 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.150+0000 7f0dfb200640 1 --2- 172.21.15.19:0/4289833120 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0dfc103200 0x7f0dfc109a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:21.151 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.150+0000 7f0dfb200640 1 --2- 172.21.15.19:0/4289833120 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0dfc102840 0x7f0dfc102cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:21.151 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.150+0000 7f0dfb200640 1 --2- 172.21.15.19:0/4289833120 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0dfc101650 0x7f0dfc101a70 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:21.151 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.150+0000 7f0dfb200640 1 -- 172.21.15.19:0/4289833120 >> 172.21.15.19:0/4289833120 conn(0x7f0dfc0fcbc0 msgr2=0x7f0dfc0ff000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:18:21.151 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.150+0000 7f0dfb200640 1 -- 172.21.15.19:0/4289833120 shutdown_connections 2024-03-18T15:18:21.151 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.150+0000 7f0dfb200640 1 -- 172.21.15.19:0/4289833120 wait complete. 2024-03-18T15:18:21.153 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.151+0000 7f0dfb200640 1 Processor -- start 2024-03-18T15:18:21.153 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.151+0000 7f0dfb200640 1 -- start start 2024-03-18T15:18:21.153 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.151+0000 7f0dfb200640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0dfc101650 0x7f0dfc075590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:21.153 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.151+0000 7f0dfb200640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0dfc102840 0x7f0dfc073bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:21.153 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.151+0000 7f0dfb200640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0dfc103200 0x7f0dfc074110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:21.153 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.151+0000 7f0dfb200640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f0dfc10d2d0 con 0x7f0dfc103200 2024-03-18T15:18:21.153 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.151+0000 7f0dfb200640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f0dfc10d150 con 0x7f0dfc102840 2024-03-18T15:18:21.153 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.151+0000 7f0dfb200640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f0dfc10d450 con 0x7f0dfc101650 2024-03-18T15:18:21.153 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.152+0000 7f0df9e00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0dfc101650 0x7f0dfc075590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:21.153 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.152+0000 7f0df9e00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0dfc101650 0x7f0dfc075590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.19:54798/0 (socket says 172.21.15.19:54798) 2024-03-18T15:18:21.154 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.152+0000 7f0df9e00640 1 -- 172.21.15.19:0/3197315591 learned_addr learned my addr 172.21.15.19:0/3197315591 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:18:21.154 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.152+0000 7f0df1400640 1 --2- 172.21.15.19:0/3197315591 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0dfc102840 0x7f0dfc073bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:21.154 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.152+0000 7f0dfa800640 1 --2- 172.21.15.19:0/3197315591 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0dfc103200 0x7f0dfc074110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:21.154 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.152+0000 7f0df9e00640 1 -- 172.21.15.19:0/3197315591 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0dfc102840 msgr2=0x7f0dfc073bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:21.154 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.152+0000 7f0df9e00640 1 --2- 172.21.15.19:0/3197315591 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0dfc102840 0x7f0dfc073bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:21.154 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.152+0000 7f0df9e00640 1 -- 172.21.15.19:0/3197315591 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0dfc103200 msgr2=0x7f0dfc074110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:21.154 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.152+0000 7f0df9e00640 1 --2- 172.21.15.19:0/3197315591 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0dfc103200 0x7f0dfc074110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:21.154 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.152+0000 7f0df9e00640 1 -- 172.21.15.19:0/3197315591 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0dfc0749a0 con 0x7f0dfc101650 2024-03-18T15:18:21.154 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.152+0000 7f0dfa800640 1 --2- 172.21.15.19:0/3197315591 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0dfc103200 0x7f0dfc074110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T15:18:21.154 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.152+0000 7f0df9e00640 1 --2- 172.21.15.19:0/3197315591 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0dfc101650 0x7f0dfc075590 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f0dec000f80 tx=0x7f0dec035d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:18:21.155 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.152+0000 7f0df3400640 1 -- 172.21.15.19:0/3197315591 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dec039040 con 0x7f0dfc101650 2024-03-18T15:18:21.155 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.153+0000 7f0df3400640 1 -- 172.21.15.19:0/3197315591 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0dec044070 con 0x7f0dfc101650 2024-03-18T15:18:21.155 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.153+0000 7f0dfb200640 1 -- 172.21.15.19:0/3197315591 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0dfc1a1a20 con 0x7f0dfc101650 2024-03-18T15:18:21.155 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.153+0000 7f0df3400640 1 -- 172.21.15.19:0/3197315591 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dec004430 con 0x7f0dfc101650 2024-03-18T15:18:21.155 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.153+0000 7f0dfb200640 1 -- 172.21.15.19:0/3197315591 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0dfc1a1f40 con 0x7f0dfc101650 2024-03-18T15:18:21.155 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.154+0000 7f0dfb200640 1 -- 172.21.15.19:0/3197315591 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0dfc108a10 con 0x7f0dfc101650 2024-03-18T15:18:21.158 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.157+0000 7f0df3400640 1 -- 172.21.15.19:0/3197315591 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 19) ==== 137307+0+0 (secure 0 0 0) 0x7f0dec0045d0 con 0x7f0dfc101650 2024-03-18T15:18:21.160 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.158+0000 7f0df3400640 1 --2- 172.21.15.19:0/3197315591 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f0de00a2840 0x7f0de00a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:21.160 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.159+0000 7f0df1400640 1 -- 172.21.15.19:0/3197315591 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f0de00a2840 msgr2=0x7f0de00a4d00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.19:6800/2017714236 2024-03-18T15:18:21.160 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.159+0000 7f0df3400640 1 -- 172.21.15.19:0/3197315591 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f0dec0f1450 con 0x7f0dfc101650 2024-03-18T15:18:21.160 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.159+0000 7f0df1400640 1 --2- 172.21.15.19:0/3197315591 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f0de00a2840 0x7f0de00a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T15:18:21.166 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.165+0000 7f0df3400640 1 -- 172.21.15.19:0/3197315591 <== mon.1 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0dec0be0c0 con 0x7f0dfc101650 2024-03-18T15:18:21.361 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.360+0000 7f0df1400640 1 -- 172.21.15.19:0/3197315591 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f0de00a2840 msgr2=0x7f0de00a4d00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.19:6800/2017714236 2024-03-18T15:18:21.361 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.360+0000 7f0df1400640 1 --2- 172.21.15.19:0/3197315591 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f0de00a2840 0x7f0de00a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T15:18:21.369 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.369+0000 7f0dfb200640 1 -- 172.21.15.19:0/3197315591 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7f0dfc062480 con 0x7f0dfc101650 2024-03-18T15:18:21.377 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.376+0000 7f0df3400640 1 -- 172.21.15.19:0/3197315591 <== mon.1 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v29)=0 v29) ==== 133+0+0 (secure 0 0 0) 0x7f0dec0bd9c0 con 0x7f0dfc101650 2024-03-18T15:18:21.380 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.379+0000 7f0dfb200640 1 -- 172.21.15.19:0/3197315591 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f0de00a2840 msgr2=0x7f0de00a4d00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T15:18:21.380 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.379+0000 7f0dfb200640 1 --2- 172.21.15.19:0/3197315591 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f0de00a2840 0x7f0de00a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:21.380 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.379+0000 7f0dfb200640 1 -- 172.21.15.19:0/3197315591 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0dfc101650 msgr2=0x7f0dfc075590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:21.381 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.379+0000 7f0dfb200640 1 --2- 172.21.15.19:0/3197315591 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0dfc101650 0x7f0dfc075590 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f0dec000f80 tx=0x7f0dec035d10 comp rx=0 tx=0).stop 2024-03-18T15:18:21.381 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.379+0000 7f0dfb200640 1 -- 172.21.15.19:0/3197315591 shutdown_connections 2024-03-18T15:18:21.381 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.379+0000 7f0dfb200640 1 --2- 172.21.15.19:0/3197315591 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0dfc103200 0x7f0dfc074110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:21.381 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.379+0000 7f0dfb200640 1 --2- 172.21.15.19:0/3197315591 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7f0de00a2840 0x7f0de00a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:21.381 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.379+0000 7f0dfb200640 1 --2- 172.21.15.19:0/3197315591 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0dfc102840 0x7f0dfc073bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:21.381 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.379+0000 7f0dfb200640 1 --2- 172.21.15.19:0/3197315591 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0dfc101650 0x7f0dfc075590 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:21.381 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.379+0000 7f0dfb200640 1 -- 172.21.15.19:0/3197315591 >> 172.21.15.19:0/3197315591 conn(0x7f0dfc0fcbc0 msgr2=0x7f0dfc105be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:18:21.381 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.379+0000 7f0dfb200640 1 -- 172.21.15.19:0/3197315591 shutdown_connections 2024-03-18T15:18:21.381 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.379+0000 7f0dfb200640 1 -- 172.21.15.19:0/3197315591 wait complete. 2024-03-18T15:18:21.401 DEBUG:teuthology.orchestra.run.smithi019:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m' 2024-03-18T15:18:21.444 INFO:journalctl@ceph.mgr.z.smithi104.stdout:Mar 18 15:18:21 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-z[37936]: 2024-03-18T15:18:21.104+0000 7fa02aa11200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T15:18:21.444 INFO:journalctl@ceph.mgr.z.smithi104.stdout:Mar 18 15:18:21 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-z[37936]: 2024-03-18T15:18:21.247+0000 7fa02aa11200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T15:18:21.496 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Mar 18 15:18:21 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-y[38034]: 2024-03-18T15:18:21.075+0000 7f9af8653200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T15:18:21.496 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Mar 18 15:18:21 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-y[38034]: 2024-03-18T15:18:21.172+0000 7f9af8653200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T15:18:21.598 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.597+0000 7fd343800640 1 -- 172.21.15.19:0/145597889 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd33c107560 msgr2=0x7fd33c109950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:21.598 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.597+0000 7fd343800640 1 --2- 172.21.15.19:0/145597889 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd33c107560 0x7fd33c109950 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fd334009900 tx=0x7fd33402bc40 comp rx=0 tx=0).stop 2024-03-18T15:18:21.598 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.597+0000 7fd343800640 1 -- 172.21.15.19:0/145597889 shutdown_connections 2024-03-18T15:18:21.598 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.597+0000 7fd343800640 1 --2- 172.21.15.19:0/145597889 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd33c107560 0x7fd33c109950 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:21.598 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.597+0000 7fd343800640 1 --2- 172.21.15.19:0/145597889 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd33c104c30 0x7fd33c107020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:21.598 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.597+0000 7fd343800640 1 --2- 172.21.15.19:0/145597889 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd33c102300 0x7fd33c1046f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:21.598 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.597+0000 7fd343800640 1 -- 172.21.15.19:0/145597889 >> 172.21.15.19:0/145597889 conn(0x7fd33c0fbc90 msgr2=0x7fd33c0fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:18:21.599 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.597+0000 7fd343800640 1 -- 172.21.15.19:0/145597889 shutdown_connections 2024-03-18T15:18:21.599 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.597+0000 7fd343800640 1 -- 172.21.15.19:0/145597889 wait complete. 2024-03-18T15:18:21.600 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.598+0000 7fd343800640 1 Processor -- start 2024-03-18T15:18:21.601 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.598+0000 7fd343800640 1 -- start start 2024-03-18T15:18:21.601 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.598+0000 7fd343800640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd33c102300 0x7fd33c199c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:21.601 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.598+0000 7fd343800640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd33c104c30 0x7fd33c19a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:21.601 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.598+0000 7fd343800640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd33c107560 0x7fd33c1a1210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:21.601 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.598+0000 7fd343800640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fd33c10d4c0 con 0x7fd33c104c30 2024-03-18T15:18:21.601 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.598+0000 7fd343800640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd33c10d340 con 0x7fd33c107560 2024-03-18T15:18:21.601 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.598+0000 7fd343800640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fd33c10d640 con 0x7fd33c102300 2024-03-18T15:18:21.601 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.598+0000 7fd340e00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd33c102300 0x7fd33c199c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:21.601 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.598+0000 7fd340e00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd33c102300 0x7fd33c199c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.19:54812/0 (socket says 172.21.15.19:54812) 2024-03-18T15:18:21.601 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.598+0000 7fd340e00640 1 -- 172.21.15.19:0/108952750 learned_addr learned my addr 172.21.15.19:0/108952750 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:18:21.601 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.598+0000 7fd341800640 1 --2- 172.21.15.19:0/108952750 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd33c107560 0x7fd33c1a1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:21.602 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.599+0000 7fd33be00640 1 --2- 172.21.15.19:0/108952750 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd33c104c30 0x7fd33c19a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:21.602 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.599+0000 7fd340e00640 1 -- 172.21.15.19:0/108952750 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd33c107560 msgr2=0x7fd33c1a1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:21.602 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.599+0000 7fd340e00640 1 --2- 172.21.15.19:0/108952750 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd33c107560 0x7fd33c1a1210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:21.602 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.599+0000 7fd340e00640 1 -- 172.21.15.19:0/108952750 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd33c104c30 msgr2=0x7fd33c19a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:21.602 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.599+0000 7fd340e00640 1 --2- 172.21.15.19:0/108952750 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd33c104c30 0x7fd33c19a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:21.602 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.599+0000 7fd340e00640 1 -- 172.21.15.19:0/108952750 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd33c1a18b0 con 0x7fd33c102300 2024-03-18T15:18:21.602 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.599+0000 7fd33be00640 1 --2- 172.21.15.19:0/108952750 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd33c104c30 0x7fd33c19a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T15:18:21.602 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.599+0000 7fd340e00640 1 --2- 172.21.15.19:0/108952750 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd33c102300 0x7fd33c199c70 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fd32c00eb90 tx=0x7fd32c00c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:18:21.602 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.599+0000 7fd339600640 1 -- 172.21.15.19:0/108952750 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd32c004d20 con 0x7fd33c102300 2024-03-18T15:18:21.602 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.599+0000 7fd343800640 1 -- 172.21.15.19:0/108952750 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd33c1a1ba0 con 0x7fd33c102300 2024-03-18T15:18:21.602 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.599+0000 7fd339600640 1 -- 172.21.15.19:0/108952750 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd32c00ee90 con 0x7fd33c102300 2024-03-18T15:18:21.602 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.599+0000 7fd339600640 1 -- 172.21.15.19:0/108952750 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd32c010830 con 0x7fd33c102300 2024-03-18T15:18:21.602 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.600+0000 7fd343800640 1 -- 172.21.15.19:0/108952750 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd33c1a20a0 con 0x7fd33c102300 2024-03-18T15:18:21.603 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.601+0000 7fd343800640 1 -- 172.21.15.19:0/108952750 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd33c109050 con 0x7fd33c102300 2024-03-18T15:18:21.603 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.603+0000 7fd339600640 1 -- 172.21.15.19:0/108952750 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 19) ==== 137307+0+0 (secure 0 0 0) 0x7fd32c005020 con 0x7fd33c102300 2024-03-18T15:18:21.605 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.604+0000 7fd339600640 1 --2- 172.21.15.19:0/108952750 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fd3200a28f0 0x7fd3200a4db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:21.605 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.604+0000 7fd33be00640 1 -- 172.21.15.19:0/108952750 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fd3200a28f0 msgr2=0x7fd3200a4db0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.19:6800/2017714236 2024-03-18T15:18:21.605 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.604+0000 7fd339600640 1 -- 172.21.15.19:0/108952750 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fd32c0cf5d0 con 0x7fd33c102300 2024-03-18T15:18:21.605 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.604+0000 7fd33be00640 1 --2- 172.21.15.19:0/108952750 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fd3200a28f0 0x7fd3200a4db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T15:18:21.613 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.612+0000 7fd339600640 1 -- 172.21.15.19:0/108952750 <== mon.1 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd32c09c240 con 0x7fd33c102300 2024-03-18T15:18:21.715 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:18:21 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:18:21.336+0000 7f8fbdfb1200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T15:18:21.715 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:18:21 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:18:21.462+0000 7f8fbdfb1200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T15:18:21.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:21 smithi019 ceph-mon[31522]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-03-18T15:18:21.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:21 smithi019 ceph-mon[31522]: mgrmap e19: x(active, since 11m), standbys: z, y 2024-03-18T15:18:21.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:21 smithi019 ceph-mon[31522]: from='client.? ' entity='client.admin' 2024-03-18T15:18:21.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:21 smithi019 ceph-mon[31522]: from='client.? ' entity='client.admin' 2024-03-18T15:18:21.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:21 smithi033 ceph-mon[36648]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-03-18T15:18:21.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:21 smithi033 ceph-mon[36648]: mgrmap e19: x(active, since 11m), standbys: z, y 2024-03-18T15:18:21.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:21 smithi033 ceph-mon[36648]: from='client.? ' entity='client.admin' 2024-03-18T15:18:21.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:21 smithi033 ceph-mon[36648]: from='client.? ' entity='client.admin' 2024-03-18T15:18:21.806 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.805+0000 7fd33be00640 1 -- 172.21.15.19:0/108952750 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fd3200a28f0 msgr2=0x7fd3200a4db0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.19:6800/2017714236 2024-03-18T15:18:21.806 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.805+0000 7fd33be00640 1 --2- 172.21.15.19:0/108952750 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fd3200a28f0 0x7fd3200a4db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T15:18:21.837 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:21.836+0000 7fd343800640 1 -- 172.21.15.19:0/108952750 --> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7fd33c061c20 con 0x7fd3200a28f0 2024-03-18T15:18:21.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:21 smithi104 ceph-mon[36070]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-03-18T15:18:21.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:21 smithi104 ceph-mon[36070]: mgrmap e19: x(active, since 11m), standbys: z, y 2024-03-18T15:18:21.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:21 smithi104 ceph-mon[36070]: from='client.? ' entity='client.admin' 2024-03-18T15:18:21.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:21 smithi104 ceph-mon[36070]: from='client.? ' entity='client.admin' 2024-03-18T15:18:22.178 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Mar 18 15:18:21 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-y[38034]: 2024-03-18T15:18:21.897+0000 7f9af8653200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T15:18:22.178 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Mar 18 15:18:22 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-y[38034]: 2024-03-18T15:18:22.008+0000 7f9af8653200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T15:18:22.178 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Mar 18 15:18:22 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-y[38034]: 2024-03-18T15:18:22.177+0000 7f9af8653200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T15:18:22.207 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:22.206+0000 7fd33be00640 1 -- 172.21.15.19:0/108952750 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fd3200a28f0 msgr2=0x7fd3200a4db0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.19:6800/2017714236 2024-03-18T15:18:22.207 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:22.206+0000 7fd33be00640 1 --2- 172.21.15.19:0/108952750 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fd3200a28f0 0x7fd3200a4db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-18T15:18:22.244 INFO:journalctl@ceph.mgr.z.smithi104.stdout:Mar 18 15:18:21 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-z[37936]: 2024-03-18T15:18:21.966+0000 7fa02aa11200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T15:18:22.244 INFO:journalctl@ceph.mgr.z.smithi104.stdout:Mar 18 15:18:22 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-z[37936]: 2024-03-18T15:18:22.066+0000 7fa02aa11200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T15:18:22.244 INFO:journalctl@ceph.mgr.z.smithi104.stdout:Mar 18 15:18:22 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-z[37936]: 2024-03-18T15:18:22.243+0000 7fa02aa11200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T15:18:22.506 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Mar 18 15:18:22 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-y[38034]: 2024-03-18T15:18:22.277+0000 7f9af8653200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T15:18:22.507 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Mar 18 15:18:22 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-y[38034]: 2024-03-18T15:18:22.364+0000 7f9af8653200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T15:18:22.608 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:18:22 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:18:22.295+0000 7f8fbdfb1200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T15:18:22.608 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:18:22 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:18:22.424+0000 7f8fbdfb1200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T15:18:22.694 INFO:journalctl@ceph.mgr.z.smithi104.stdout:Mar 18 15:18:22 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-z[37936]: 2024-03-18T15:18:22.362+0000 7fa02aa11200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T15:18:22.694 INFO:journalctl@ceph.mgr.z.smithi104.stdout:Mar 18 15:18:22 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-z[37936]: 2024-03-18T15:18:22.445+0000 7fa02aa11200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T15:18:22.910 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Mar 18 15:18:22 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-y[38034]: 2024-03-18T15:18:22.638+0000 7f9af8653200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T15:18:22.910 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Mar 18 15:18:22 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-y[38034]: 2024-03-18T15:18:22.798+0000 7f9af8653200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T15:18:22.965 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:18:22 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:18:22.606+0000 7f8fbdfb1200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T15:18:22.965 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:18:22 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:18:22.741+0000 7f8fbdfb1200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T15:18:22.965 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:18:22 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:18:22.855+0000 7f8fbdfb1200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T15:18:23.008 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:23.007+0000 7fd33be00640 1 -- 172.21.15.19:0/108952750 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fd3200a28f0 msgr2=0x7fd3200a4db0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.19:6800/2017714236 2024-03-18T15:18:23.008 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:23.007+0000 7fd33be00640 1 --2- 172.21.15.19:0/108952750 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fd3200a28f0 0x7fd3200a4db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-18T15:18:23.114 INFO:journalctl@ceph.mgr.z.smithi104.stdout:Mar 18 15:18:22 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-z[37936]: 2024-03-18T15:18:22.790+0000 7fa02aa11200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T15:18:23.114 INFO:journalctl@ceph.mgr.z.smithi104.stdout:Mar 18 15:18:22 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-z[37936]: 2024-03-18T15:18:22.918+0000 7fa02aa11200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T15:18:23.114 INFO:journalctl@ceph.mgr.z.smithi104.stdout:Mar 18 15:18:23 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-z[37936]: 2024-03-18T15:18:23.005+0000 7fa02aa11200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T15:18:23.114 INFO:journalctl@ceph.mgr.z.smithi104.stdout:Mar 18 15:18:23 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-z[37936]: 2024-03-18T15:18:23.113+0000 7fa02aa11200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T15:18:23.256 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Mar 18 15:18:22 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-y[38034]: 2024-03-18T15:18:22.909+0000 7f9af8653200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T15:18:23.256 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Mar 18 15:18:23 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-y[38034]: 2024-03-18T15:18:23.029+0000 7f9af8653200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T15:18:23.444 INFO:journalctl@ceph.mgr.z.smithi104.stdout:Mar 18 15:18:23 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-z[37936]: 2024-03-18T15:18:23.314+0000 7fa02aa11200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T15:18:23.465 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:18:23 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:18:23.200+0000 7f8fbdfb1200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T15:18:23.465 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:18:23 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:18:23.347+0000 7f8fbdfb1200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T15:18:23.756 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Mar 18 15:18:23 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-y[38034]: 2024-03-18T15:18:23.262+0000 7f9af8653200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T15:18:23.819 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:18:23 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:18:23.464+0000 7f8fbdfb1200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T15:18:23.819 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:18:23 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:18:23.593+0000 7f8fbdfb1200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T15:18:24.215 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:18:23 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:18:23.817+0000 7f8fbdfb1200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T15:18:24.610 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:24.609+0000 7fd33be00640 1 -- 172.21.15.19:0/108952750 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fd3200a28f0 msgr2=0x7fd3200a4db0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.19:6800/2017714236 2024-03-18T15:18:24.610 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:24.609+0000 7fd33be00640 1 --2- 172.21.15.19:0/108952750 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fd3200a28f0 0x7fd3200a4db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-18T15:18:24.756 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Mar 18 15:18:24 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-y[38034]: 2024-03-18T15:18:24.337+0000 7f9af8653200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T15:18:24.756 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Mar 18 15:18:24 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-y[38034]: 2024-03-18T15:18:24.473+0000 7f9af8653200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T15:18:24.944 INFO:journalctl@ceph.mgr.z.smithi104.stdout:Mar 18 15:18:24 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-z[37936]: 2024-03-18T15:18:24.499+0000 7fa02aa11200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T15:18:24.944 INFO:journalctl@ceph.mgr.z.smithi104.stdout:Mar 18 15:18:24 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-z[37936]: 2024-03-18T15:18:24.630+0000 7fa02aa11200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T15:18:25.151 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Mar 18 15:18:24 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-y[38034]: 2024-03-18T15:18:24.832+0000 7f9af8653200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T15:18:25.151 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Mar 18 15:18:24 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-y[38034]: 2024-03-18T15:18:24.925+0000 7f9af8653200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T15:18:25.151 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Mar 18 15:18:25 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-y[38034]: 2024-03-18T15:18:25.150+0000 7f9af8653200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T15:18:25.378 INFO:journalctl@ceph.mgr.z.smithi104.stdout:Mar 18 15:18:25 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-z[37936]: 2024-03-18T15:18:25.032+0000 7fa02aa11200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T15:18:25.378 INFO:journalctl@ceph.mgr.z.smithi104.stdout:Mar 18 15:18:25 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-z[37936]: 2024-03-18T15:18:25.131+0000 7fa02aa11200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T15:18:25.378 INFO:journalctl@ceph.mgr.z.smithi104.stdout:Mar 18 15:18:25 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-z[37936]: 2024-03-18T15:18:25.377+0000 7fa02aa11200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T15:18:25.465 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:18:25 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:18:25.142+0000 7f8fbdfb1200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T15:18:25.465 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:18:25 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:18:25.323+0000 7f8fbdfb1200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T15:18:25.506 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Mar 18 15:18:25 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-y[38034]: 2024-03-18T15:18:25.375+0000 7f9af8653200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T15:18:25.943 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Mar 18 15:18:25 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-y[38034]: 2024-03-18T15:18:25.670+0000 7f9af8653200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T15:18:25.943 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Mar 18 15:18:25 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-y[38034]: 2024-03-18T15:18:25.942+0000 7f9af8653200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T15:18:25.944 INFO:journalctl@ceph.mgr.z.smithi104.stdout:Mar 18 15:18:25 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-z[37936]: 2024-03-18T15:18:25.632+0000 7fa02aa11200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T15:18:26.150 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:18:25 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:18:25.755+0000 7f8fbdfb1200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T15:18:26.151 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:18:25 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:18:25.878+0000 7f8fbdfb1200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T15:18:26.269 INFO:journalctl@ceph.mgr.z.smithi104.stdout:Mar 18 15:18:25 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-z[37936]: 2024-03-18T15:18:25.988+0000 7fa02aa11200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T15:18:26.407 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:18:26 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:18:26.148+0000 7f8fbdfb1200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T15:18:26.506 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Mar 18 15:18:26 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-y[38034]: 2024-03-18T15:18:26.236+0000 7f9af8653200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T15:18:26.694 INFO:journalctl@ceph.mgr.z.smithi104.stdout:Mar 18 15:18:26 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-z[37936]: 2024-03-18T15:18:26.269+0000 7fa02aa11200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T15:18:26.694 INFO:journalctl@ceph.mgr.z.smithi104.stdout:Mar 18 15:18:26 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-z[37936]: 2024-03-18T15:18:26.508+0000 7fa02aa11200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T15:18:26.715 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:18:26 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:18:26.405+0000 7f8fbdfb1200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T15:18:27.126 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:18:26 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:18:26.803+0000 7f8fbdfb1200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T15:18:27.463 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:18:27 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:18:27.140+0000 7f8fbdfb1200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T15:18:27.506 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Mar 18 15:18:27 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-y[38034]: 2024-03-18T15:18:27.206+0000 7f9af8653200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T15:18:27.715 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:18:27 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:18:27.461+0000 7f8fbdfb1200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T15:18:27.814 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:27.813+0000 7fd33be00640 1 -- 172.21.15.19:0/108952750 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fd3200a28f0 msgr2=0x7fd3200a4db0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.19:6800/2017714236 2024-03-18T15:18:27.814 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:27.813+0000 7fd33be00640 1 --2- 172.21.15.19:0/108952750 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fd3200a28f0 0x7fd3200a4db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-03-18T15:18:27.944 INFO:journalctl@ceph.mgr.z.smithi104.stdout:Mar 18 15:18:27 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-z[37936]: 2024-03-18T15:18:27.481+0000 7fa02aa11200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T15:18:28.207 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Mar 18 15:18:27 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-y[38034]: 2024-03-18T15:18:27.878+0000 7f9af8653200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T15:18:28.207 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Mar 18 15:18:27 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-y[38034]: 2024-03-18T15:18:27.982+0000 7f9af8653200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T15:18:28.295 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:28.294+0000 7fd339600640 1 -- 172.21.15.19:0/108952750 <== mon.1 v2:172.21.15.104:3300/0 7 ==== mgrmap(e 20) ==== 137307+0+0 (secure 0 0 0) 0x7fd32c0981b0 con 0x7fd33c102300 2024-03-18T15:18:28.444 INFO:journalctl@ceph.mgr.z.smithi104.stdout:Mar 18 15:18:28 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-z[37936]: 2024-03-18T15:18:28.114+0000 7fa02aa11200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T15:18:28.444 INFO:journalctl@ceph.mgr.z.smithi104.stdout:Mar 18 15:18:28 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-z[37936]: 2024-03-18T15:18:28.226+0000 7fa02aa11200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T15:18:28.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:28 smithi104 ceph-mon[36070]: Standby manager daemon y restarted 2024-03-18T15:18:28.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:28 smithi104 ceph-mon[36070]: Standby manager daemon y started 2024-03-18T15:18:28.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:28 smithi104 ceph-mon[36070]: from='mgr.? 172.21.15.33:0/3040855227' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-18T15:18:28.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:28 smithi104 ceph-mon[36070]: from='mgr.? 172.21.15.33:0/3040855227' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-18T15:18:28.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:28 smithi104 ceph-mon[36070]: from='mgr.? 172.21.15.33:0/3040855227' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-18T15:18:28.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:28 smithi104 ceph-mon[36070]: from='mgr.? 172.21.15.33:0/3040855227' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-18T15:18:28.506 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Mar 18 15:18:28 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-y[38034]: 2024-03-18T15:18:28.206+0000 7f9af8653200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T15:18:28.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:28 smithi033 ceph-mon[36648]: Standby manager daemon y restarted 2024-03-18T15:18:28.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:28 smithi033 ceph-mon[36648]: Standby manager daemon y started 2024-03-18T15:18:28.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:28 smithi033 ceph-mon[36648]: from='mgr.? 172.21.15.33:0/3040855227' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-18T15:18:28.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:28 smithi033 ceph-mon[36648]: from='mgr.? 172.21.15.33:0/3040855227' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-18T15:18:28.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:28 smithi033 ceph-mon[36648]: from='mgr.? 172.21.15.33:0/3040855227' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-18T15:18:28.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:28 smithi033 ceph-mon[36648]: from='mgr.? 172.21.15.33:0/3040855227' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-18T15:18:28.553 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:28 smithi019 ceph-mon[31522]: Standby manager daemon y restarted 2024-03-18T15:18:28.553 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:28 smithi019 ceph-mon[31522]: Standby manager daemon y started 2024-03-18T15:18:28.553 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:28 smithi019 ceph-mon[31522]: from='mgr.? 172.21.15.33:0/3040855227' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-18T15:18:28.553 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:28 smithi019 ceph-mon[31522]: from='mgr.? 172.21.15.33:0/3040855227' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-18T15:18:28.553 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:28 smithi019 ceph-mon[31522]: from='mgr.? 172.21.15.33:0/3040855227' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-18T15:18:28.553 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:28 smithi019 ceph-mon[31522]: from='mgr.? 172.21.15.33:0/3040855227' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-18T15:18:28.943 INFO:journalctl@ceph.mgr.z.smithi104.stdout:Mar 18 15:18:28 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-z[37936]: 2024-03-18T15:18:28.487+0000 7fa02aa11200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T15:18:28.965 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:18:28 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:18:28.552+0000 7f8fbdfb1200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T15:18:29.329 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:29.328+0000 7fd339600640 1 -- 172.21.15.19:0/108952750 <== mon.1 v2:172.21.15.104:3300/0 8 ==== mgrmap(e 21) ==== 137307+0+0 (secure 0 0 0) 0x7fd32c0ccd20 con 0x7fd33c102300 2024-03-18T15:18:29.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:29 smithi104 ceph-mon[36070]: mgrmap e20: x(active, since 11m), standbys: z, y 2024-03-18T15:18:29.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:29 smithi104 ceph-mon[36070]: Standby manager daemon z restarted 2024-03-18T15:18:29.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:29 smithi104 ceph-mon[36070]: Standby manager daemon z started 2024-03-18T15:18:29.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:29 smithi104 ceph-mon[36070]: from='mgr.? 172.21.15.104:0/3387058542' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-18T15:18:29.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:29 smithi104 ceph-mon[36070]: from='mgr.? 172.21.15.104:0/3387058542' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-18T15:18:29.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:29 smithi104 ceph-mon[36070]: from='mgr.? 172.21.15.104:0/3387058542' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-18T15:18:29.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:29 smithi104 ceph-mon[36070]: from='mgr.? 172.21.15.104:0/3387058542' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-18T15:18:29.715 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:18:29 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:18:29.280+0000 7f8fbdfb1200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T15:18:29.715 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:18:29 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:18:29.423+0000 7f8fbdfb1200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T15:18:29.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:29 smithi019 ceph-mon[31522]: mgrmap e20: x(active, since 11m), standbys: z, y 2024-03-18T15:18:29.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:29 smithi019 ceph-mon[31522]: Standby manager daemon z restarted 2024-03-18T15:18:29.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:29 smithi019 ceph-mon[31522]: Standby manager daemon z started 2024-03-18T15:18:29.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:29 smithi019 ceph-mon[31522]: from='mgr.? 172.21.15.104:0/3387058542' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-18T15:18:29.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:29 smithi019 ceph-mon[31522]: from='mgr.? 172.21.15.104:0/3387058542' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-18T15:18:29.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:29 smithi019 ceph-mon[31522]: from='mgr.? 172.21.15.104:0/3387058542' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-18T15:18:29.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:29 smithi019 ceph-mon[31522]: from='mgr.? 172.21.15.104:0/3387058542' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-18T15:18:29.746 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:29.746+0000 7fd339600640 1 -- 172.21.15.19:0/108952750 <== mon.1 v2:172.21.15.104:3300/0 9 ==== mgrmap(e 22) ==== 136975+0+0 (secure 0 0 0) 0x7fd32c099800 con 0x7fd33c102300 2024-03-18T15:18:29.746 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:29.747+0000 7fd339600640 1 -- 172.21.15.19:0/108952750 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fd3200a28f0 msgr2=0x7fd3200a4db0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T15:18:29.747 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:29.747+0000 7fd339600640 1 --2- 172.21.15.19:0/108952750 >> [v2:172.21.15.19:6800/2017714236,v1:172.21.15.19:6801/2017714236] conn(0x7fd3200a28f0 0x7fd3200a4db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:29.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:29 smithi033 ceph-mon[36648]: mgrmap e20: x(active, since 11m), standbys: z, y 2024-03-18T15:18:29.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:29 smithi033 ceph-mon[36648]: Standby manager daemon z restarted 2024-03-18T15:18:29.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:29 smithi033 ceph-mon[36648]: Standby manager daemon z started 2024-03-18T15:18:29.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:29 smithi033 ceph-mon[36648]: from='mgr.? 172.21.15.104:0/3387058542' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-18T15:18:29.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:29 smithi033 ceph-mon[36648]: from='mgr.? 172.21.15.104:0/3387058542' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-18T15:18:29.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:29 smithi033 ceph-mon[36648]: from='mgr.? 172.21.15.104:0/3387058542' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-18T15:18:29.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:29 smithi033 ceph-mon[36648]: from='mgr.? 172.21.15.104:0/3387058542' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-18T15:18:29.989 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:18:29 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:18:29.722+0000 7f8fbdfb1200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T15:18:30.311 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:18:30 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:18:30.052+0000 7f8f0ea00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T15:18:30.311 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:18:30 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:18:30.052+0000 7f8f0ea00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T15:18:30.311 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:18:30 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:18:30.052+0000 7f8f0ea00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T15:18:30.312 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:18:30 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:18:30.053+0000 7f8f0ea00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T15:18:30.312 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:18:30 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:18:30.053+0000 7f8f0ea00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T15:18:30.312 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:18:30 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:18:30.055+0000 7f8f12600700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T15:18:30.312 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:18:30 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:18:30.055+0000 7f8f12600700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T15:18:30.312 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:18:30 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:18:30.055+0000 7f8f12600700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T15:18:30.312 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:18:30 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:18:30.055+0000 7f8f12600700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T15:18:30.312 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:18:30 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:18:30.055+0000 7f8f12600700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T15:18:30.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:30 smithi104 ceph-mon[36070]: mgrmap e21: x(active, since 11m), standbys: y, z 2024-03-18T15:18:30.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:30 smithi104 ceph-mon[36070]: Active manager daemon x restarted 2024-03-18T15:18:30.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:30 smithi104 ceph-mon[36070]: Activating manager daemon x 2024-03-18T15:18:30.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:30 smithi104 ceph-mon[36070]: osdmap e94: 12 total, 12 up, 12 in 2024-03-18T15:18:30.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:30 smithi104 ceph-mon[36070]: mgrmap e22: x(active, starting, since 0.0274691s), standbys: y, z 2024-03-18T15:18:30.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:30 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T15:18:30.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:30 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T15:18:30.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:30 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T15:18:30.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:30 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-18T15:18:30.696 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:30 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-18T15:18:30.696 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:30 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-18T15:18:30.696 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:30 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-18T15:18:30.696 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:30 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-18T15:18:30.696 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:30 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-18T15:18:30.696 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:30 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-18T15:18:30.696 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:30 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-18T15:18:30.696 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:30 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-18T15:18:30.696 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:30 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-18T15:18:30.696 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:30 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-18T15:18:30.696 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:30 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-18T15:18:30.696 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:30 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-18T15:18:30.696 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:30 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-18T15:18:30.696 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:30 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-18T15:18:30.697 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:30 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T15:18:30.697 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:30 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T15:18:30.697 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:30 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T15:18:30.697 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:30 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T15:18:30.697 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:30 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T15:18:30.697 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:30 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T15:18:30.697 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:30 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T15:18:30.697 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:30 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T15:18:30.697 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:30 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T15:18:30.697 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:30 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T15:18:30.697 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:30 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T15:18:30.697 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:30 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T15:18:30.697 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:30 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-18T15:18:30.697 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:30 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-18T15:18:30.697 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:30 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-18T15:18:30.697 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:30 smithi104 ceph-mon[36070]: Manager daemon x is now available 2024-03-18T15:18:30.698 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:30 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:18:30.698 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:30 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:18:30.698 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:30 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-18T15:18:30.698 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:30 smithi104 ceph-mon[36070]: from='mgr.25466 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-18T15:18:30.698 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:30 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-18T15:18:30.698 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:30 smithi104 ceph-mon[36070]: from='mgr.25466 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-18T15:18:30.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:30 smithi019 ceph-mon[31522]: mgrmap e21: x(active, since 11m), standbys: y, z 2024-03-18T15:18:30.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:30 smithi019 ceph-mon[31522]: Active manager daemon x restarted 2024-03-18T15:18:30.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:30 smithi019 ceph-mon[31522]: Activating manager daemon x 2024-03-18T15:18:30.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:30 smithi019 ceph-mon[31522]: osdmap e94: 12 total, 12 up, 12 in 2024-03-18T15:18:30.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:30 smithi019 ceph-mon[31522]: mgrmap e22: x(active, starting, since 0.0274691s), standbys: y, z 2024-03-18T15:18:30.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:30 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T15:18:30.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:30 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T15:18:30.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:30 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T15:18:30.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:30 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-18T15:18:30.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:30 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-18T15:18:30.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:30 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-18T15:18:30.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:30 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-18T15:18:30.717 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:30 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-18T15:18:30.717 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:30 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-18T15:18:30.717 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:30 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-18T15:18:30.717 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:30 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-18T15:18:30.717 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:30 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-18T15:18:30.717 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:30 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-18T15:18:30.717 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:30 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-18T15:18:30.717 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:30 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-18T15:18:30.717 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:30 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-18T15:18:30.717 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:30 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-18T15:18:30.717 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:30 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-18T15:18:30.717 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:30 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T15:18:30.717 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:30 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T15:18:30.717 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:30 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T15:18:30.717 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:30 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T15:18:30.717 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:30 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T15:18:30.718 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:30 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T15:18:30.718 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:30 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T15:18:30.718 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:30 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T15:18:30.718 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:30 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T15:18:30.718 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:30 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T15:18:30.718 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:30 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T15:18:30.718 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:30 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T15:18:30.718 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:30 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-18T15:18:30.718 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:30 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-18T15:18:30.718 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:30 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-18T15:18:30.718 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:30 smithi019 ceph-mon[31522]: Manager daemon x is now available 2024-03-18T15:18:30.718 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:30 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:18:30.718 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:30 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:18:30.718 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:30 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-18T15:18:30.718 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:30 smithi019 ceph-mon[31522]: from='mgr.25466 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-18T15:18:30.718 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:30 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-18T15:18:30.718 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:30 smithi019 ceph-mon[31522]: from='mgr.25466 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-18T15:18:30.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:30 smithi033 ceph-mon[36648]: mgrmap e21: x(active, since 11m), standbys: y, z 2024-03-18T15:18:30.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:30 smithi033 ceph-mon[36648]: Active manager daemon x restarted 2024-03-18T15:18:30.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:30 smithi033 ceph-mon[36648]: Activating manager daemon x 2024-03-18T15:18:30.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:30 smithi033 ceph-mon[36648]: osdmap e94: 12 total, 12 up, 12 in 2024-03-18T15:18:30.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:30 smithi033 ceph-mon[36648]: mgrmap e22: x(active, starting, since 0.0274691s), standbys: y, z 2024-03-18T15:18:30.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:30 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T15:18:30.758 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:30 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T15:18:30.758 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:30 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T15:18:30.758 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:30 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-18T15:18:30.758 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:30 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-18T15:18:30.758 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:30 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-18T15:18:30.758 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:30 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-18T15:18:30.758 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:30 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-18T15:18:30.758 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:30 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-18T15:18:30.758 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:30 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-18T15:18:30.758 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:30 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-18T15:18:30.758 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:30 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-18T15:18:30.758 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:30 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-18T15:18:30.758 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:30 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-18T15:18:30.758 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:30 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-18T15:18:30.758 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:30 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-18T15:18:30.759 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:30 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-18T15:18:30.759 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:30 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-18T15:18:30.759 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:30 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T15:18:30.759 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:30 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T15:18:30.759 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:30 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T15:18:30.759 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:30 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T15:18:30.759 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:30 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T15:18:30.759 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:30 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T15:18:30.759 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:30 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T15:18:30.759 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:30 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T15:18:30.759 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:30 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T15:18:30.759 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:30 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T15:18:30.759 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:30 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T15:18:30.759 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:30 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T15:18:30.759 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:30 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-18T15:18:30.759 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:30 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-18T15:18:30.759 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:30 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-18T15:18:30.760 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:30 smithi033 ceph-mon[36648]: Manager daemon x is now available 2024-03-18T15:18:30.760 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:30 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:18:30.760 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:30 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:18:30.760 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:30 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-18T15:18:30.760 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:30 smithi033 ceph-mon[36648]: from='mgr.25466 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-18T15:18:30.760 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:30 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-18T15:18:30.760 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:30 smithi033 ceph-mon[36648]: from='mgr.25466 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-18T15:18:30.760 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:30.755+0000 7fd339600640 1 -- 172.21.15.19:0/108952750 <== mon.1 v2:172.21.15.104:3300/0 10 ==== mgrmap(e 23) ==== 137102+0+0 (secure 0 0 0) 0x7fd32c0cf850 con 0x7fd33c102300 2024-03-18T15:18:30.760 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:30.757+0000 7fd339600640 1 --2- 172.21.15.19:0/108952750 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7fd3200686e0 0x7fd320068ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:30.760 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:30.757+0000 7fd339600640 1 -- 172.21.15.19:0/108952750 --> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7fd32c090d00 con 0x7fd3200686e0 2024-03-18T15:18:30.764 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:30.764+0000 7fd33be00640 1 --2- 172.21.15.19:0/108952750 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7fd3200686e0 0x7fd320068ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:30.775 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:30.775+0000 7fd33be00640 1 --2- 172.21.15.19:0/108952750 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7fd3200686e0 0x7fd320068ac0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd33c19b170 tx=0x7fd324005f90 comp rx=0 tx=0).ready entity=mgr.25466 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:18:30.866 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:30.866+0000 7fd339600640 1 -- 172.21.15.19:0/108952750 <== mgr.25466 v2:172.21.15.19:6800/3894323675 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fd32c090d00 con 0x7fd3200686e0 2024-03-18T15:18:30.867 INFO:teuthology.orchestra.run.smithi019.stdout:Schedule set for path / 2024-03-18T15:18:30.869 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:30.869+0000 7fd343800640 1 -- 172.21.15.19:0/108952750 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7fd3200686e0 msgr2=0x7fd320068ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:30.869 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:30.869+0000 7fd343800640 1 --2- 172.21.15.19:0/108952750 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7fd3200686e0 0x7fd320068ac0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd33c19b170 tx=0x7fd324005f90 comp rx=0 tx=0).stop 2024-03-18T15:18:30.869 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:30.869+0000 7fd343800640 1 -- 172.21.15.19:0/108952750 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd33c102300 msgr2=0x7fd33c199c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:30.869 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:30.869+0000 7fd343800640 1 --2- 172.21.15.19:0/108952750 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd33c102300 0x7fd33c199c70 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fd32c00eb90 tx=0x7fd32c00c6a0 comp rx=0 tx=0).stop 2024-03-18T15:18:30.869 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:30.869+0000 7fd341800640 1 -- 172.21.15.19:0/108952750 reap_dead start 2024-03-18T15:18:30.869 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:30.869+0000 7fd343800640 1 -- 172.21.15.19:0/108952750 shutdown_connections 2024-03-18T15:18:30.869 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:30.869+0000 7fd343800640 1 -- 172.21.15.19:0/108952750 >> 172.21.15.19:0/108952750 conn(0x7fd33c0fbc90 msgr2=0x7fd33c1018b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:18:30.870 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:30.869+0000 7fd343800640 1 -- 172.21.15.19:0/108952750 shutdown_connections 2024-03-18T15:18:30.870 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:30.869+0000 7fd343800640 1 -- 172.21.15.19:0/108952750 wait complete. 2024-03-18T15:18:30.885 DEBUG:teuthology.orchestra.run.smithi019:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h' 2024-03-18T15:18:31.077 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.077+0000 7f8b19e00640 1 -- 172.21.15.19:0/2248592880 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8b14074240 msgr2=0x7f8b140746c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:31.077 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.077+0000 7f8b19e00640 1 --2- 172.21.15.19:0/2248592880 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8b14074240 0x7f8b140746c0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f8b00009900 tx=0x7f8b0002bc30 comp rx=0 tx=0).stop 2024-03-18T15:18:31.078 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.077+0000 7f8b19e00640 1 -- 172.21.15.19:0/2248592880 shutdown_connections 2024-03-18T15:18:31.078 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.077+0000 7f8b19e00640 1 --2- 172.21.15.19:0/2248592880 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8b141096e0 0x7f8b1410bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:31.078 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.077+0000 7f8b19e00640 1 --2- 172.21.15.19:0/2248592880 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8b14074240 0x7f8b140746c0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:31.078 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.077+0000 7f8b19e00640 1 --2- 172.21.15.19:0/2248592880 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8b14075820 0x7f8b14073c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:31.078 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.077+0000 7f8b19e00640 1 -- 172.21.15.19:0/2248592880 >> 172.21.15.19:0/2248592880 conn(0x7f8b140fcd70 msgr2=0x7f8b140ff1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:18:31.078 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.077+0000 7f8b19e00640 1 -- 172.21.15.19:0/2248592880 shutdown_connections 2024-03-18T15:18:31.078 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.077+0000 7f8b19e00640 1 -- 172.21.15.19:0/2248592880 wait complete. 2024-03-18T15:18:31.079 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.078+0000 7f8b19e00640 1 Processor -- start 2024-03-18T15:18:31.079 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.078+0000 7f8b19e00640 1 -- start start 2024-03-18T15:18:31.079 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.079+0000 7f8b19e00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8b14074240 0x7f8b14199ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:31.080 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.079+0000 7f8b19e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8b14075820 0x7f8b1419a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:31.080 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.079+0000 7f8b19e00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8b141096e0 0x7f8b141a1240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:31.080 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.079+0000 7f8b19e00640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f8b1410f6b0 con 0x7f8b141096e0 2024-03-18T15:18:31.080 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.079+0000 7f8b19e00640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8b1410f530 con 0x7f8b14075820 2024-03-18T15:18:31.080 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.079+0000 7f8b19e00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f8b1410f830 con 0x7f8b14074240 2024-03-18T15:18:31.080 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.079+0000 7f8b12000640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8b14075820 0x7f8b1419a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:31.080 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.079+0000 7f8b12a00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8b14074240 0x7f8b14199ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:31.080 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.079+0000 7f8b12000640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8b14075820 0x7f8b1419a1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:51672/0 (socket says 172.21.15.19:51672) 2024-03-18T15:18:31.080 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.079+0000 7f8b12000640 1 -- 172.21.15.19:0/2194308392 learned_addr learned my addr 172.21.15.19:0/2194308392 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:18:31.080 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.079+0000 7f8b13400640 1 --2- 172.21.15.19:0/2194308392 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8b141096e0 0x7f8b141a1240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:31.080 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.079+0000 7f8b12000640 1 -- 172.21.15.19:0/2194308392 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8b14074240 msgr2=0x7f8b14199ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:31.080 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.079+0000 7f8b12000640 1 --2- 172.21.15.19:0/2194308392 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8b14074240 0x7f8b14199ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:31.080 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.079+0000 7f8b12000640 1 -- 172.21.15.19:0/2194308392 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8b141096e0 msgr2=0x7f8b141a1240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:31.080 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.079+0000 7f8b12000640 1 --2- 172.21.15.19:0/2194308392 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8b141096e0 0x7f8b141a1240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:31.080 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.079+0000 7f8b12000640 1 -- 172.21.15.19:0/2194308392 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b141a18e0 con 0x7f8b14075820 2024-03-18T15:18:31.081 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.080+0000 7f8b12000640 1 --2- 172.21.15.19:0/2194308392 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8b14075820 0x7f8b1419a1e0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f8b00000c00 tx=0x7f8b00005dd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:18:31.081 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.080+0000 7f8b18a00640 1 -- 172.21.15.19:0/2194308392 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b00043020 con 0x7f8b14075820 2024-03-18T15:18:31.081 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.080+0000 7f8b18a00640 1 -- 172.21.15.19:0/2194308392 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b000054e0 con 0x7f8b14075820 2024-03-18T15:18:31.081 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.080+0000 7f8b18a00640 1 -- 172.21.15.19:0/2194308392 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b00003e60 con 0x7f8b14075820 2024-03-18T15:18:31.081 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.080+0000 7f8b19e00640 1 -- 172.21.15.19:0/2194308392 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b141a1b70 con 0x7f8b14075820 2024-03-18T15:18:31.081 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.080+0000 7f8b19e00640 1 -- 172.21.15.19:0/2194308392 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b141a2070 con 0x7f8b14075820 2024-03-18T15:18:31.082 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.082+0000 7f8b19e00640 1 -- 172.21.15.19:0/2194308392 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b14108410 con 0x7f8b14075820 2024-03-18T15:18:31.084 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.084+0000 7f8b18a00640 1 -- 172.21.15.19:0/2194308392 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 23) ==== 137102+0+0 (secure 0 0 0) 0x7f8b00046020 con 0x7f8b14075820 2024-03-18T15:18:31.085 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.086+0000 7f8b18a00640 1 --2- 172.21.15.19:0/2194308392 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f8af80a2730 0x7f8af80a4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:31.085 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.086+0000 7f8b18a00640 1 -- 172.21.15.19:0/2194308392 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f8b000f0cf0 con 0x7f8b14075820 2024-03-18T15:18:31.086 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.086+0000 7f8b12a00640 1 --2- 172.21.15.19:0/2194308392 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f8af80a2730 0x7f8af80a4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:31.086 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.087+0000 7f8b12a00640 1 --2- 172.21.15.19:0/2194308392 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f8af80a2730 0x7f8af80a4bf0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f8b1419ab70 tx=0x7f8b0400b390 comp rx=0 tx=0).ready entity=mgr.25466 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:18:31.090 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.090+0000 7f8b18a00640 1 -- 172.21.15.19:0/2194308392 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8b0004d020 con 0x7f8b14075820 2024-03-18T15:18:31.325 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.325+0000 7f8b19e00640 1 -- 172.21.15.19:0/2194308392 --> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}) -- 0x7f8b14061d80 con 0x7f8af80a2730 2024-03-18T15:18:31.407 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.406+0000 7f8b18a00640 1 -- 172.21.15.19:0/2194308392 <== mgr.25466 v2:172.21.15.19:6800/3894323675 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+26 (secure 0 0 0) 0x7f8b14061d80 con 0x7f8af80a2730 2024-03-18T15:18:31.411 INFO:teuthology.orchestra.run.smithi019.stdout:Retention added to path / 2024-03-18T15:18:31.412 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.408+0000 7f8b19e00640 1 -- 172.21.15.19:0/2194308392 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f8af80a2730 msgr2=0x7f8af80a4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:31.412 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.408+0000 7f8b19e00640 1 --2- 172.21.15.19:0/2194308392 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f8af80a2730 0x7f8af80a4bf0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f8b1419ab70 tx=0x7f8b0400b390 comp rx=0 tx=0).stop 2024-03-18T15:18:31.412 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.408+0000 7f8b19e00640 1 -- 172.21.15.19:0/2194308392 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8b14075820 msgr2=0x7f8b1419a1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:31.412 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.408+0000 7f8b19e00640 1 --2- 172.21.15.19:0/2194308392 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8b14075820 0x7f8b1419a1e0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f8b00000c00 tx=0x7f8b00005dd0 comp rx=0 tx=0).stop 2024-03-18T15:18:31.412 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.408+0000 7f8b19e00640 1 -- 172.21.15.19:0/2194308392 shutdown_connections 2024-03-18T15:18:31.412 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.408+0000 7f8b19e00640 1 --2- 172.21.15.19:0/2194308392 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f8af80a2730 0x7f8af80a4bf0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:31.412 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.408+0000 7f8b19e00640 1 --2- 172.21.15.19:0/2194308392 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8b141096e0 0x7f8b141a1240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:31.413 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.408+0000 7f8b19e00640 1 --2- 172.21.15.19:0/2194308392 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8b14075820 0x7f8b1419a1e0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:31.413 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.408+0000 7f8b19e00640 1 --2- 172.21.15.19:0/2194308392 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8b14074240 0x7f8b14199ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:31.413 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.408+0000 7f8b19e00640 1 -- 172.21.15.19:0/2194308392 >> 172.21.15.19:0/2194308392 conn(0x7f8b140fcd70 msgr2=0x7f8b141055e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:18:31.413 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.408+0000 7f8b19e00640 1 -- 172.21.15.19:0/2194308392 shutdown_connections 2024-03-18T15:18:31.413 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.408+0000 7f8b19e00640 1 -- 172.21.15.19:0/2194308392 wait complete. 2024-03-18T15:18:31.427 DEBUG:teuthology.orchestra.run.smithi019:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-03-18T15:18:31.615 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.614+0000 7f7eeca00640 1 -- 172.21.15.19:0/4009144009 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7ee81020e0 msgr2=0x7f7ee8108970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:31.615 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.614+0000 7f7eeca00640 1 --2- 172.21.15.19:0/4009144009 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7ee81020e0 0x7f7ee8108970 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f7ee0009900 tx=0x7f7ee002bc40 comp rx=0 tx=0).stop 2024-03-18T15:18:31.616 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.614+0000 7f7eeca00640 1 -- 172.21.15.19:0/4009144009 shutdown_connections 2024-03-18T15:18:31.616 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.614+0000 7f7eeca00640 1 --2- 172.21.15.19:0/4009144009 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7ee81020e0 0x7f7ee8108970 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:31.616 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.614+0000 7f7eeca00640 1 --2- 172.21.15.19:0/4009144009 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7ee8101720 0x7f7ee8101ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:31.616 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.614+0000 7f7eeca00640 1 --2- 172.21.15.19:0/4009144009 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7ee8100530 0x7f7ee8100950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:31.616 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.614+0000 7f7eeca00640 1 -- 172.21.15.19:0/4009144009 >> 172.21.15.19:0/4009144009 conn(0x7f7ee80fbac0 msgr2=0x7f7ee80fdee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:18:31.616 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.614+0000 7f7eeca00640 1 -- 172.21.15.19:0/4009144009 shutdown_connections 2024-03-18T15:18:31.616 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.614+0000 7f7eeca00640 1 -- 172.21.15.19:0/4009144009 wait complete. 2024-03-18T15:18:31.616 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.615+0000 7f7eeca00640 1 Processor -- start 2024-03-18T15:18:31.616 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.615+0000 7f7eeca00640 1 -- start start 2024-03-18T15:18:31.616 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.615+0000 7f7eeca00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7ee8100530 0x7f7ee8199af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:31.616 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.615+0000 7f7eeca00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7ee8101720 0x7f7ee819a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:31.618 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.615+0000 7f7eeca00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7ee81020e0 0x7f7ee81a10a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:31.619 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.615+0000 7f7eeca00640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f7ee810c1e0 con 0x7f7ee81020e0 2024-03-18T15:18:31.619 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.615+0000 7f7eeca00640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7ee810c060 con 0x7f7ee8100530 2024-03-18T15:18:31.619 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.615+0000 7f7eeca00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f7ee810c360 con 0x7f7ee8101720 2024-03-18T15:18:31.619 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.616+0000 7f7ee7400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7ee8100530 0x7f7ee8199af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:31.619 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.616+0000 7f7ee6a00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7ee8101720 0x7f7ee819a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:31.619 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.616+0000 7f7ee7400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7ee8100530 0x7f7ee8199af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:51692/0 (socket says 172.21.15.19:51692) 2024-03-18T15:18:31.619 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.616+0000 7f7ee7400640 1 -- 172.21.15.19:0/3488181105 learned_addr learned my addr 172.21.15.19:0/3488181105 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:18:31.619 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.616+0000 7f7ee6a00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7ee8101720 0x7f7ee819a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.19:58172/0 (socket says 172.21.15.19:58172) 2024-03-18T15:18:31.619 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.616+0000 7f7ee7e00640 1 --2- 172.21.15.19:0/3488181105 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7ee81020e0 0x7f7ee81a10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:31.619 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.616+0000 7f7ee7400640 1 -- 172.21.15.19:0/3488181105 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7ee8101720 msgr2=0x7f7ee819a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:31.619 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.616+0000 7f7ee7400640 1 --2- 172.21.15.19:0/3488181105 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7ee8101720 0x7f7ee819a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:31.620 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.616+0000 7f7ee7400640 1 -- 172.21.15.19:0/3488181105 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7ee81020e0 msgr2=0x7f7ee81a10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:31.620 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.616+0000 7f7ee7400640 1 --2- 172.21.15.19:0/3488181105 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7ee81020e0 0x7f7ee81a10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:31.620 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.616+0000 7f7ee7400640 1 -- 172.21.15.19:0/3488181105 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ee81a1710 con 0x7f7ee8100530 2024-03-18T15:18:31.620 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.616+0000 7f7ee7e00640 1 --2- 172.21.15.19:0/3488181105 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7ee81020e0 0x7f7ee81a10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T15:18:31.620 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.616+0000 7f7ee6a00640 1 --2- 172.21.15.19:0/3488181105 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7ee8101720 0x7f7ee819a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:18:31.620 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.616+0000 7f7ee7400640 1 --2- 172.21.15.19:0/3488181105 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7ee8100530 0x7f7ee8199af0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f7ed800ebc0 tx=0x7f7ed800c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:18:31.620 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.616+0000 7f7edfe00640 1 -- 172.21.15.19:0/3488181105 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ed8003b70 con 0x7f7ee8100530 2024-03-18T15:18:31.620 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.617+0000 7f7edfe00640 1 -- 172.21.15.19:0/3488181105 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ed8003d10 con 0x7f7ee8100530 2024-03-18T15:18:31.620 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.617+0000 7f7edfe00640 1 -- 172.21.15.19:0/3488181105 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ed8019980 con 0x7f7ee8100530 2024-03-18T15:18:31.620 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.617+0000 7f7eeca00640 1 -- 172.21.15.19:0/3488181105 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ee81a1a00 con 0x7f7ee8100530 2024-03-18T15:18:31.621 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.617+0000 7f7eeca00640 1 -- 172.21.15.19:0/3488181105 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ee81a1fa0 con 0x7f7ee8100530 2024-03-18T15:18:31.621 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.618+0000 7f7eeca00640 1 -- 172.21.15.19:0/3488181105 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ee804fd90 con 0x7f7ee8100530 2024-03-18T15:18:31.622 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.619+0000 7f7edfe00640 1 -- 172.21.15.19:0/3488181105 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 23) ==== 137102+0+0 (secure 0 0 0) 0x7f7ed8019b20 con 0x7f7ee8100530 2024-03-18T15:18:31.622 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.621+0000 7f7edfe00640 1 --2- 172.21.15.19:0/3488181105 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f7ecc0a2660 0x7f7ecc0a4b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:31.622 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.621+0000 7f7edfe00640 1 -- 172.21.15.19:0/3488181105 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f7ed80cf380 con 0x7f7ee8100530 2024-03-18T15:18:31.622 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.621+0000 7f7ee6a00640 1 --2- 172.21.15.19:0/3488181105 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f7ecc0a2660 0x7f7ecc0a4b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:31.622 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.622+0000 7f7ee6a00640 1 --2- 172.21.15.19:0/3488181105 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f7ecc0a2660 0x7f7ecc0a4b20 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7ed00095f0 tx=0x7f7ed0005f90 comp rx=0 tx=0).ready entity=mgr.25466 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:18:31.630 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.630+0000 7f7edfe00640 1 -- 172.21.15.19:0/3488181105 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7ed809c200 con 0x7f7ee8100530 2024-03-18T15:18:31.769 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.769+0000 7f7edfe00640 1 -- 172.21.15.19:0/3488181105 <== mon.2 v2:172.21.15.33:3300/0 7 ==== mgrmap(e 24) ==== 137410+0+0 (secure 0 0 0) 0x7f7ed8098170 con 0x7f7ee8100530 2024-03-18T15:18:31.865 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.865+0000 7f7eeca00640 1 -- 172.21.15.19:0/3488181105 --> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7f7ee80fdbd0 con 0x7f7ecc0a2660 2024-03-18T15:18:31.880 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.880+0000 7f7edfe00640 1 -- 172.21.15.19:0/3488181105 <== mgr.25466 v2:172.21.15.19:6800/3894323675 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+298 (secure 0 0 0) 0x7f7ee80fdbd0 con 0x7f7ecc0a2660 2024-03-18T15:18:31.880 INFO:teuthology.orchestra.run.smithi019.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-03-18T00:00:00", "created": "2024-03-18T15:18:30", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2024-03-18T15:18:31.883 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.883+0000 7f7eeca00640 1 -- 172.21.15.19:0/3488181105 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f7ecc0a2660 msgr2=0x7f7ecc0a4b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:31.883 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.883+0000 7f7eeca00640 1 --2- 172.21.15.19:0/3488181105 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f7ecc0a2660 0x7f7ecc0a4b20 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7ed00095f0 tx=0x7f7ed0005f90 comp rx=0 tx=0).stop 2024-03-18T15:18:31.883 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.883+0000 7f7eeca00640 1 -- 172.21.15.19:0/3488181105 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7ee8100530 msgr2=0x7f7ee8199af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:31.883 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.883+0000 7f7eeca00640 1 --2- 172.21.15.19:0/3488181105 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7ee8100530 0x7f7ee8199af0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f7ed800ebc0 tx=0x7f7ed800c6a0 comp rx=0 tx=0).stop 2024-03-18T15:18:31.883 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.883+0000 7f7eeca00640 1 -- 172.21.15.19:0/3488181105 shutdown_connections 2024-03-18T15:18:31.884 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.883+0000 7f7eeca00640 1 --2- 172.21.15.19:0/3488181105 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f7ecc0a2660 0x7f7ecc0a4b20 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:31.884 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.883+0000 7f7eeca00640 1 --2- 172.21.15.19:0/3488181105 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7ee81020e0 0x7f7ee81a10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:31.884 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.883+0000 7f7eeca00640 1 --2- 172.21.15.19:0/3488181105 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7ee8101720 0x7f7ee819a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:31.884 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.883+0000 7f7eeca00640 1 --2- 172.21.15.19:0/3488181105 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7ee8100530 0x7f7ee8199af0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:31.884 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.883+0000 7f7eeca00640 1 -- 172.21.15.19:0/3488181105 >> 172.21.15.19:0/3488181105 conn(0x7f7ee80fbac0 msgr2=0x7f7ee80fd4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:18:31.884 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.883+0000 7f7eeca00640 1 -- 172.21.15.19:0/3488181105 shutdown_connections 2024-03-18T15:18:31.884 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:31.883+0000 7f7eeca00640 1 -- 172.21.15.19:0/3488181105 wait complete. 2024-03-18T15:18:31.903 DEBUG:teuthology.orchestra.run.smithi019:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-03-18T15:18:32.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:31 smithi033 ceph-mon[36648]: mgrmap e23: x(active, since 1.02369s), standbys: y, z 2024-03-18T15:18:32.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:31 smithi033 ceph-mon[36648]: [18/Mar/2024:15:18:30] ENGINE Bus STARTING 2024-03-18T15:18:32.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:31 smithi033 ceph-mon[36648]: [18/Mar/2024:15:18:30] ENGINE Serving on http://172.21.15.19:8765 2024-03-18T15:18:32.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:31 smithi033 ceph-mon[36648]: [18/Mar/2024:15:18:31] ENGINE Serving on https://172.21.15.19:7150 2024-03-18T15:18:32.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:31 smithi033 ceph-mon[36648]: [18/Mar/2024:15:18:31] ENGINE Bus STARTED 2024-03-18T15:18:32.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:31 smithi033 ceph-mon[36648]: from='client.25523 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:18:32.102 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:32.101+0000 7f8206200640 1 -- 172.21.15.19:0/607812510 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8200075820 msgr2=0x7f8200073c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:32.102 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:32.101+0000 7f8206200640 1 --2- 172.21.15.19:0/607812510 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8200075820 0x7f8200073c70 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f81f0009cd0 tx=0x7f81f002c0e0 comp rx=0 tx=0).stop 2024-03-18T15:18:32.102 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:32.102+0000 7f8206200640 1 -- 172.21.15.19:0/607812510 shutdown_connections 2024-03-18T15:18:32.102 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:32.102+0000 7f8206200640 1 --2- 172.21.15.19:0/607812510 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f820010dde0 0x7f82001101d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:32.102 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:32.102+0000 7f8206200640 1 --2- 172.21.15.19:0/607812510 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8200074240 0x7f82000746c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:32.102 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:32.102+0000 7f8206200640 1 --2- 172.21.15.19:0/607812510 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8200075820 0x7f8200073c70 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:32.102 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:32.102+0000 7f8206200640 1 -- 172.21.15.19:0/607812510 >> 172.21.15.19:0/607812510 conn(0x7f82000fcda0 msgr2=0x7f82000ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:18:32.103 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:32.102+0000 7f8206200640 1 -- 172.21.15.19:0/607812510 shutdown_connections 2024-03-18T15:18:32.103 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:32.102+0000 7f8206200640 1 -- 172.21.15.19:0/607812510 wait complete. 2024-03-18T15:18:32.104 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:32.103+0000 7f8206200640 1 Processor -- start 2024-03-18T15:18:32.104 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:32.103+0000 7f8206200640 1 -- start start 2024-03-18T15:18:32.104 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:32.104+0000 7f8206200640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8200074240 0x7f820019e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:32.105 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:32.104+0000 7f8206200640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8200075820 0x7f820019e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:32.105 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:32.104+0000 7f8206200640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f820010dde0 0x7f82001a5630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:32.105 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:32.104+0000 7f8206200640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f8200113a60 con 0x7f8200074240 2024-03-18T15:18:32.105 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:32.104+0000 7f8206200640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f82001138e0 con 0x7f8200075820 2024-03-18T15:18:32.105 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:32.104+0000 7f8206200640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f8200113be0 con 0x7f820010dde0 2024-03-18T15:18:32.105 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:32.104+0000 7f81ff400640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8200074240 0x7f820019e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:32.105 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:32.104+0000 7f81fea00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8200075820 0x7f820019e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:32.105 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:32.104+0000 7f81ff400640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8200074240 0x7f820019e080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:53070/0 (socket says 172.21.15.19:53070) 2024-03-18T15:18:32.105 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:32.104+0000 7f81ff400640 1 -- 172.21.15.19:0/2450414808 learned_addr learned my addr 172.21.15.19:0/2450414808 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:18:32.105 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:32.104+0000 7f81fea00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8200075820 0x7f820019e5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:43876/0 (socket says 172.21.15.19:43876) 2024-03-18T15:18:32.105 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:32.104+0000 7f81ffe00640 1 --2- 172.21.15.19:0/2450414808 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f820010dde0 0x7f82001a5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:32.105 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:32.104+0000 7f81ff400640 1 -- 172.21.15.19:0/2450414808 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f820010dde0 msgr2=0x7f82001a5630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:32.105 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:32.105+0000 7f81ff400640 1 --2- 172.21.15.19:0/2450414808 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f820010dde0 0x7f82001a5630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:32.105 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:32.105+0000 7f81ff400640 1 -- 172.21.15.19:0/2450414808 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8200075820 msgr2=0x7f820019e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:32.105 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:32.105+0000 7f81ff400640 1 --2- 172.21.15.19:0/2450414808 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8200075820 0x7f820019e5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:32.106 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:32.105+0000 7f81ff400640 1 -- 172.21.15.19:0/2450414808 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82001a5ca0 con 0x7f8200074240 2024-03-18T15:18:32.106 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:32.105+0000 7f81ff400640 1 --2- 172.21.15.19:0/2450414808 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8200074240 0x7f820019e080 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f81f00325d0 tx=0x7f81f0035ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:18:32.106 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:32.105+0000 7f81f7e00640 1 -- 172.21.15.19:0/2450414808 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81f0039040 con 0x7f8200074240 2024-03-18T15:18:32.106 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:32.105+0000 7f81f7e00640 1 -- 172.21.15.19:0/2450414808 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f81f0044070 con 0x7f8200074240 2024-03-18T15:18:32.107 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:32.105+0000 7f81f7e00640 1 -- 172.21.15.19:0/2450414808 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81f0004440 con 0x7f8200074240 2024-03-18T15:18:32.107 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:32.105+0000 7f8206200640 1 -- 172.21.15.19:0/2450414808 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82001a5f30 con 0x7f8200074240 2024-03-18T15:18:32.107 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:32.106+0000 7f8206200640 1 -- 172.21.15.19:0/2450414808 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82001a6340 con 0x7f8200074240 2024-03-18T15:18:32.107 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:32.107+0000 7f8206200640 1 -- 172.21.15.19:0/2450414808 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8200103fb0 con 0x7f8200074240 2024-03-18T15:18:32.108 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:32.109+0000 7f81f7e00640 1 -- 172.21.15.19:0/2450414808 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 24) ==== 137410+0+0 (secure 0 0 0) 0x7f81f00045e0 con 0x7f8200074240 2024-03-18T15:18:32.110 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:32.110+0000 7f81f7e00640 1 --2- 172.21.15.19:0/2450414808 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f81e40a2a90 0x7f81e40a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:18:32.110 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:32.110+0000 7f81f7e00640 1 -- 172.21.15.19:0/2450414808 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f81f00f1620 con 0x7f8200074240 2024-03-18T15:18:32.110 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:32.111+0000 7f81fea00640 1 --2- 172.21.15.19:0/2450414808 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f81e40a2a90 0x7f81e40a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:18:32.111 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:32.111+0000 7f81fea00640 1 --2- 172.21.15.19:0/2450414808 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f81e40a2a90 0x7f81e40a4f50 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f820019f590 tx=0x7f81ec005f90 comp rx=0 tx=0).ready entity=mgr.25466 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:18:32.116 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:32.116+0000 7f81f7e00640 1 -- 172.21.15.19:0/2450414808 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f81f00be0e0 con 0x7f8200074240 2024-03-18T15:18:32.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:31 smithi104 ceph-mon[36070]: mgrmap e23: x(active, since 1.02369s), standbys: y, z 2024-03-18T15:18:32.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:31 smithi104 ceph-mon[36070]: [18/Mar/2024:15:18:30] ENGINE Bus STARTING 2024-03-18T15:18:32.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:31 smithi104 ceph-mon[36070]: [18/Mar/2024:15:18:30] ENGINE Serving on http://172.21.15.19:8765 2024-03-18T15:18:32.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:31 smithi104 ceph-mon[36070]: [18/Mar/2024:15:18:31] ENGINE Serving on https://172.21.15.19:7150 2024-03-18T15:18:32.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:31 smithi104 ceph-mon[36070]: [18/Mar/2024:15:18:31] ENGINE Bus STARTED 2024-03-18T15:18:32.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:31 smithi104 ceph-mon[36070]: from='client.25523 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:18:32.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:31 smithi019 ceph-mon[31522]: mgrmap e23: x(active, since 1.02369s), standbys: y, z 2024-03-18T15:18:32.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:31 smithi019 ceph-mon[31522]: [18/Mar/2024:15:18:30] ENGINE Bus STARTING 2024-03-18T15:18:32.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:31 smithi019 ceph-mon[31522]: [18/Mar/2024:15:18:30] ENGINE Serving on http://172.21.15.19:8765 2024-03-18T15:18:32.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:31 smithi019 ceph-mon[31522]: [18/Mar/2024:15:18:31] ENGINE Serving on https://172.21.15.19:7150 2024-03-18T15:18:32.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:31 smithi019 ceph-mon[31522]: [18/Mar/2024:15:18:31] ENGINE Bus STARTED 2024-03-18T15:18:32.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:31 smithi019 ceph-mon[31522]: from='client.25523 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:18:32.353 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:32.353+0000 7f8206200640 1 -- 172.21.15.19:0/2450414808 --> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7f8200061c20 con 0x7f81e40a2a90 2024-03-18T15:18:32.366 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:32.366+0000 7f81f7e00640 1 -- 172.21.15.19:0/2450414808 <== mgr.25466 v2:172.21.15.19:6800/3894323675 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7f8200061c20 con 0x7f81e40a2a90 2024-03-18T15:18:32.367 INFO:teuthology.orchestra.run.smithi019.stdout:/ 1m 6m3h 2024-03-18T15:18:32.369 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:32.369+0000 7f8206200640 1 -- 172.21.15.19:0/2450414808 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f81e40a2a90 msgr2=0x7f81e40a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:32.370 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:32.369+0000 7f8206200640 1 --2- 172.21.15.19:0/2450414808 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f81e40a2a90 0x7f81e40a4f50 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f820019f590 tx=0x7f81ec005f90 comp rx=0 tx=0).stop 2024-03-18T15:18:32.370 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:32.369+0000 7f8206200640 1 -- 172.21.15.19:0/2450414808 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8200074240 msgr2=0x7f820019e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:18:32.370 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:32.369+0000 7f8206200640 1 --2- 172.21.15.19:0/2450414808 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8200074240 0x7f820019e080 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f81f00325d0 tx=0x7f81f0035ae0 comp rx=0 tx=0).stop 2024-03-18T15:18:32.370 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:32.369+0000 7f8206200640 1 -- 172.21.15.19:0/2450414808 shutdown_connections 2024-03-18T15:18:32.370 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:32.369+0000 7f8206200640 1 --2- 172.21.15.19:0/2450414808 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f81e40a2a90 0x7f81e40a4f50 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:32.370 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:32.369+0000 7f8206200640 1 --2- 172.21.15.19:0/2450414808 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f820010dde0 0x7f82001a5630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:32.370 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:32.369+0000 7f8206200640 1 --2- 172.21.15.19:0/2450414808 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8200075820 0x7f820019e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:32.370 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:32.369+0000 7f8206200640 1 --2- 172.21.15.19:0/2450414808 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8200074240 0x7f820019e080 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:18:32.370 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:32.370+0000 7f8206200640 1 -- 172.21.15.19:0/2450414808 >> 172.21.15.19:0/2450414808 conn(0x7f82000fcda0 msgr2=0x7f8200101180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:18:32.370 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:32.370+0000 7f8206200640 1 -- 172.21.15.19:0/2450414808 shutdown_connections 2024-03-18T15:18:32.370 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:18:32.370+0000 7f8206200640 1 -- 172.21.15.19:0/2450414808 wait complete. 2024-03-18T15:18:32.388 DEBUG:teuthology.orchestra.run.smithi019:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2024-03-18T15:18:32.471 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-03-18T15:18:32.481 INFO:teuthology.run_tasks:Running task workunit... 2024-03-18T15:18:32.524 INFO:tasks.workunit:Pulling workunits from ref 4e19f704afd9957ea657ab858f9ecf76401fef6f 2024-03-18T15:18:32.525 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-18T15:18:32.525 INFO:tasks.workunit:timeout=3h 2024-03-18T15:18:32.525 INFO:tasks.workunit:cleanup=True 2024-03-18T15:18:32.525 DEBUG:teuthology.orchestra.run.smithi019:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-18T15:18:32.568 INFO:teuthology.orchestra.run.smithi019.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-03-18T15:18:32.568 INFO:teuthology.orchestra.run.smithi019.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-03-18T15:18:32.568 INFO:teuthology.orchestra.run.smithi019.stdout:Device: 76h/118d Inode: 1099511628283 Links: 2 2024-03-18T15:18:32.568 INFO:teuthology.orchestra.run.smithi019.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T15:18:32.568 INFO:teuthology.orchestra.run.smithi019.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-03-18T15:18:32.568 INFO:teuthology.orchestra.run.smithi019.stdout:Access: 2024-03-18 15:15:47.502593830 +0000 2024-03-18T15:18:32.568 INFO:teuthology.orchestra.run.smithi019.stdout:Modify: 2024-03-18 15:15:47.502593830 +0000 2024-03-18T15:18:32.568 INFO:teuthology.orchestra.run.smithi019.stdout:Change: 2024-03-18 15:18:16.962591190 +0000 2024-03-18T15:18:32.569 INFO:teuthology.orchestra.run.smithi019.stdout: Birth: 2024-03-18 15:15:47.502593830 +0000 2024-03-18T15:18:32.569 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-03-18T15:18:32.569 DEBUG:teuthology.orchestra.run.smithi019:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-03-18T15:18:32.704 DEBUG:teuthology.orchestra.run.smithi019:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/batrick/ceph /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 4e19f704afd9957ea657ab858f9ecf76401fef6f 2024-03-18T15:18:32.755 INFO:tasks.workunit.client.0.smithi019.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-18T15:18:32.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:32 smithi019 ceph-mon[31522]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:18:32.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:32 smithi019 ceph-mon[31522]: mgrmap e24: x(active, since 2s), standbys: y, z 2024-03-18T15:18:32.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:32 smithi019 ceph-mon[31522]: from='client.25529 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:18:32.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:32 smithi019 ceph-mon[31522]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:18:32.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:32 smithi019 ceph-mon[31522]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:18:32.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:32 smithi019 ceph-mon[31522]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:18:32.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:32 smithi019 ceph-mon[31522]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:18:32.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:32 smithi019 ceph-mon[31522]: from='client.15435 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:18:33.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:32 smithi104 ceph-mon[36070]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:18:33.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:32 smithi104 ceph-mon[36070]: mgrmap e24: x(active, since 2s), standbys: y, z 2024-03-18T15:18:33.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:32 smithi104 ceph-mon[36070]: from='client.25529 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:18:33.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:32 smithi104 ceph-mon[36070]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:18:33.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:32 smithi104 ceph-mon[36070]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:18:33.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:32 smithi104 ceph-mon[36070]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:18:33.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:32 smithi104 ceph-mon[36070]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:18:33.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:32 smithi104 ceph-mon[36070]: from='client.15435 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:18:33.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:32 smithi033 ceph-mon[36648]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:18:33.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:32 smithi033 ceph-mon[36648]: mgrmap e24: x(active, since 2s), standbys: y, z 2024-03-18T15:18:33.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:32 smithi033 ceph-mon[36648]: from='client.25529 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:18:33.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:32 smithi033 ceph-mon[36648]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:18:33.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:32 smithi033 ceph-mon[36648]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:18:33.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:32 smithi033 ceph-mon[36648]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:18:33.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:32 smithi033 ceph-mon[36648]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:18:33.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:32 smithi033 ceph-mon[36648]: from='client.15435 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:18:34.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:34 smithi104 ceph-mon[36070]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:18:34.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:34 smithi104 ceph-mon[36070]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:18:34.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:34 smithi104 ceph-mon[36070]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:18:34.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:34 smithi104 ceph-mon[36070]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:18:34.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:34 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:18:34.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:34 smithi104 ceph-mon[36070]: from='mgr.25466 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:18:34.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:34 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:18:34.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:34 smithi104 ceph-mon[36070]: from='mgr.25466 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:18:34.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:34 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:18:34.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:34 smithi104 ceph-mon[36070]: from='mgr.25466 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:18:34.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:34 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:18:34.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:34 smithi104 ceph-mon[36070]: from='mgr.25466 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:18:34.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:34 smithi104 ceph-mon[36070]: Adjusting osd_memory_target on smithi033 to 175.2M 2024-03-18T15:18:34.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:34 smithi104 ceph-mon[36070]: Unable to set osd_memory_target on smithi033 to 183744512: error parsing value: Value '183744512' is below minimum 939524096 2024-03-18T15:18:34.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:34 smithi104 ceph-mon[36070]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:18:34.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:34 smithi104 ceph-mon[36070]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:18:34.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:34 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:18:34.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:34 smithi104 ceph-mon[36070]: from='mgr.25466 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:18:34.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:34 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:18:34.696 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:34 smithi104 ceph-mon[36070]: from='mgr.25466 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:18:34.696 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:34 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:18:34.696 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:34 smithi104 ceph-mon[36070]: from='mgr.25466 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:18:34.696 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:34 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:18:34.696 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:34 smithi104 ceph-mon[36070]: from='mgr.25466 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:18:34.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:34 smithi019 ceph-mon[31522]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:18:34.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:34 smithi019 ceph-mon[31522]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:18:34.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:34 smithi019 ceph-mon[31522]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:18:34.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:34 smithi019 ceph-mon[31522]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:18:34.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:34 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:18:34.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:34 smithi019 ceph-mon[31522]: from='mgr.25466 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:18:34.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:34 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:18:34.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:34 smithi019 ceph-mon[31522]: from='mgr.25466 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:18:34.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:34 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:18:34.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:34 smithi019 ceph-mon[31522]: from='mgr.25466 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:18:34.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:34 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:18:34.717 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:34 smithi019 ceph-mon[31522]: from='mgr.25466 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:18:34.717 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:34 smithi019 ceph-mon[31522]: Adjusting osd_memory_target on smithi033 to 175.2M 2024-03-18T15:18:34.717 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:34 smithi019 ceph-mon[31522]: Unable to set osd_memory_target on smithi033 to 183744512: error parsing value: Value '183744512' is below minimum 939524096 2024-03-18T15:18:34.717 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:34 smithi019 ceph-mon[31522]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:18:34.717 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:34 smithi019 ceph-mon[31522]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:18:34.717 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:34 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:18:34.717 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:34 smithi019 ceph-mon[31522]: from='mgr.25466 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:18:34.717 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:34 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:18:34.717 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:34 smithi019 ceph-mon[31522]: from='mgr.25466 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:18:34.717 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:34 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:18:34.717 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:34 smithi019 ceph-mon[31522]: from='mgr.25466 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:18:34.717 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:34 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:18:34.717 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:34 smithi019 ceph-mon[31522]: from='mgr.25466 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:18:34.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:34 smithi033 ceph-mon[36648]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:18:34.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:34 smithi033 ceph-mon[36648]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:18:34.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:34 smithi033 ceph-mon[36648]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:18:34.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:34 smithi033 ceph-mon[36648]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:18:34.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:34 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:18:34.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:34 smithi033 ceph-mon[36648]: from='mgr.25466 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:18:34.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:34 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:18:34.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:34 smithi033 ceph-mon[36648]: from='mgr.25466 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:18:34.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:34 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:18:34.758 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:34 smithi033 ceph-mon[36648]: from='mgr.25466 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:18:34.758 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:34 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:18:34.758 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:34 smithi033 ceph-mon[36648]: from='mgr.25466 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:18:34.758 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:34 smithi033 ceph-mon[36648]: Adjusting osd_memory_target on smithi033 to 175.2M 2024-03-18T15:18:34.758 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:34 smithi033 ceph-mon[36648]: Unable to set osd_memory_target on smithi033 to 183744512: error parsing value: Value '183744512' is below minimum 939524096 2024-03-18T15:18:34.758 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:34 smithi033 ceph-mon[36648]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:18:34.758 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:34 smithi033 ceph-mon[36648]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:18:34.758 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:34 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:18:34.758 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:34 smithi033 ceph-mon[36648]: from='mgr.25466 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:18:34.758 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:34 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:18:34.758 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:34 smithi033 ceph-mon[36648]: from='mgr.25466 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:18:34.758 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:34 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:18:34.758 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:34 smithi033 ceph-mon[36648]: from='mgr.25466 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:18:34.758 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:34 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:18:34.759 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:34 smithi033 ceph-mon[36648]: from='mgr.25466 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:18:35.658 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:35 smithi033 ceph-mon[36648]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-03-18T15:18:35.658 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:35 smithi033 ceph-mon[36648]: Unable to set osd_memory_target on smithi104 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-18T15:18:35.659 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:35 smithi033 ceph-mon[36648]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:18:35.659 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:35 smithi033 ceph-mon[36648]: mgrmap e25: x(active, since 4s), standbys: y, z 2024-03-18T15:18:35.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:35 smithi104 ceph-mon[36070]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-03-18T15:18:35.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:35 smithi104 ceph-mon[36070]: Unable to set osd_memory_target on smithi104 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-18T15:18:35.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:35 smithi104 ceph-mon[36070]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:18:35.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:35 smithi104 ceph-mon[36070]: mgrmap e25: x(active, since 4s), standbys: y, z 2024-03-18T15:18:35.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:35 smithi019 ceph-mon[31522]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-03-18T15:18:35.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:35 smithi019 ceph-mon[31522]: Unable to set osd_memory_target on smithi104 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-18T15:18:35.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:35 smithi019 ceph-mon[31522]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:18:35.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:35 smithi019 ceph-mon[31522]: mgrmap e25: x(active, since 4s), standbys: y, z 2024-03-18T15:18:36.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:36 smithi104 ceph-mon[36070]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:18:36.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:36 smithi104 ceph-mon[36070]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:18:36.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:36 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:18:36.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:36 smithi104 ceph-mon[36070]: from='mgr.25466 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:18:36.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:36 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:18:36.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:36 smithi104 ceph-mon[36070]: from='mgr.25466 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:18:36.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:36 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:18:36.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:36 smithi104 ceph-mon[36070]: from='mgr.25466 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:18:36.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:36 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:18:36.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:36 smithi104 ceph-mon[36070]: from='mgr.25466 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:18:36.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:36 smithi104 ceph-mon[36070]: Adjusting osd_memory_target on smithi019 to 177.8M 2024-03-18T15:18:36.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:36 smithi104 ceph-mon[36070]: Unable to set osd_memory_target on smithi019 to 186497740: error parsing value: Value '186497740' is below minimum 939524096 2024-03-18T15:18:36.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:36 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:18:36.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:36 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:18:36.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:36 smithi104 ceph-mon[36070]: Updating smithi019:/etc/ceph/ceph.conf 2024-03-18T15:18:36.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:36 smithi104 ceph-mon[36070]: Updating smithi033:/etc/ceph/ceph.conf 2024-03-18T15:18:36.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:36 smithi104 ceph-mon[36070]: Updating smithi104:/etc/ceph/ceph.conf 2024-03-18T15:18:36.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:36 smithi019 ceph-mon[31522]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:18:36.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:36 smithi019 ceph-mon[31522]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:18:36.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:36 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:18:36.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:36 smithi019 ceph-mon[31522]: from='mgr.25466 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:18:36.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:36 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:18:36.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:36 smithi019 ceph-mon[31522]: from='mgr.25466 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:18:36.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:36 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:18:36.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:36 smithi019 ceph-mon[31522]: from='mgr.25466 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:18:36.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:36 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:18:36.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:36 smithi019 ceph-mon[31522]: from='mgr.25466 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:18:36.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:36 smithi019 ceph-mon[31522]: Adjusting osd_memory_target on smithi019 to 177.8M 2024-03-18T15:18:36.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:36 smithi019 ceph-mon[31522]: Unable to set osd_memory_target on smithi019 to 186497740: error parsing value: Value '186497740' is below minimum 939524096 2024-03-18T15:18:36.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:36 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:18:36.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:36 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:18:36.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:36 smithi019 ceph-mon[31522]: Updating smithi019:/etc/ceph/ceph.conf 2024-03-18T15:18:36.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:36 smithi019 ceph-mon[31522]: Updating smithi033:/etc/ceph/ceph.conf 2024-03-18T15:18:36.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:36 smithi019 ceph-mon[31522]: Updating smithi104:/etc/ceph/ceph.conf 2024-03-18T15:18:37.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:36 smithi033 ceph-mon[36648]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:18:37.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:36 smithi033 ceph-mon[36648]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:18:37.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:36 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:18:37.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:36 smithi033 ceph-mon[36648]: from='mgr.25466 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:18:37.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:36 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:18:37.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:36 smithi033 ceph-mon[36648]: from='mgr.25466 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:18:37.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:36 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:18:37.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:36 smithi033 ceph-mon[36648]: from='mgr.25466 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:18:37.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:36 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:18:37.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:36 smithi033 ceph-mon[36648]: from='mgr.25466 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:18:37.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:36 smithi033 ceph-mon[36648]: Adjusting osd_memory_target on smithi019 to 177.8M 2024-03-18T15:18:37.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:36 smithi033 ceph-mon[36648]: Unable to set osd_memory_target on smithi019 to 186497740: error parsing value: Value '186497740' is below minimum 939524096 2024-03-18T15:18:37.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:36 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:18:37.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:36 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:18:37.008 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:36 smithi033 ceph-mon[36648]: Updating smithi019:/etc/ceph/ceph.conf 2024-03-18T15:18:37.008 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:36 smithi033 ceph-mon[36648]: Updating smithi033:/etc/ceph/ceph.conf 2024-03-18T15:18:37.008 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:36 smithi033 ceph-mon[36648]: Updating smithi104:/etc/ceph/ceph.conf 2024-03-18T15:18:37.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:37 smithi104 ceph-mon[36070]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:18:37.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:37 smithi104 ceph-mon[36070]: Updating smithi033:/var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T15:18:37.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:37 smithi104 ceph-mon[36070]: Updating smithi104:/var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T15:18:37.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:37 smithi104 ceph-mon[36070]: Updating smithi104:/etc/ceph/ceph.client.admin.keyring 2024-03-18T15:18:37.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:37 smithi104 ceph-mon[36070]: Updating smithi033:/etc/ceph/ceph.client.admin.keyring 2024-03-18T15:18:37.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:37 smithi104 ceph-mon[36070]: Updating smithi019:/var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T15:18:37.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:37 smithi104 ceph-mon[36070]: Updating smithi104:/var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-18T15:18:37.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:37 smithi104 ceph-mon[36070]: Updating smithi033:/var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-18T15:18:37.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:37 smithi104 ceph-mon[36070]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:18:37.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:37 smithi104 ceph-mon[36070]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:18:37.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:37 smithi104 ceph-mon[36070]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:18:37.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:37 smithi104 ceph-mon[36070]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:18:37.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:37 smithi019 ceph-mon[31522]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:18:37.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:37 smithi019 ceph-mon[31522]: Updating smithi033:/var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T15:18:37.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:37 smithi019 ceph-mon[31522]: Updating smithi104:/var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T15:18:37.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:37 smithi019 ceph-mon[31522]: Updating smithi104:/etc/ceph/ceph.client.admin.keyring 2024-03-18T15:18:37.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:37 smithi019 ceph-mon[31522]: Updating smithi033:/etc/ceph/ceph.client.admin.keyring 2024-03-18T15:18:37.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:37 smithi019 ceph-mon[31522]: Updating smithi019:/var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T15:18:37.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:37 smithi019 ceph-mon[31522]: Updating smithi104:/var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-18T15:18:37.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:37 smithi019 ceph-mon[31522]: Updating smithi033:/var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-18T15:18:37.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:37 smithi019 ceph-mon[31522]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:18:37.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:37 smithi019 ceph-mon[31522]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:18:37.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:37 smithi019 ceph-mon[31522]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:18:37.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:37 smithi019 ceph-mon[31522]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:18:38.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:37 smithi033 ceph-mon[36648]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:18:38.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:37 smithi033 ceph-mon[36648]: Updating smithi033:/var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T15:18:38.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:37 smithi033 ceph-mon[36648]: Updating smithi104:/var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T15:18:38.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:37 smithi033 ceph-mon[36648]: Updating smithi104:/etc/ceph/ceph.client.admin.keyring 2024-03-18T15:18:38.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:37 smithi033 ceph-mon[36648]: Updating smithi033:/etc/ceph/ceph.client.admin.keyring 2024-03-18T15:18:38.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:37 smithi033 ceph-mon[36648]: Updating smithi019:/var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T15:18:38.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:37 smithi033 ceph-mon[36648]: Updating smithi104:/var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-18T15:18:38.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:37 smithi033 ceph-mon[36648]: Updating smithi033:/var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-18T15:18:38.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:37 smithi033 ceph-mon[36648]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:18:38.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:37 smithi033 ceph-mon[36648]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:18:38.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:37 smithi033 ceph-mon[36648]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:18:38.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:37 smithi033 ceph-mon[36648]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:18:38.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:38 smithi104 ceph-mon[36070]: Updating smithi019:/etc/ceph/ceph.client.admin.keyring 2024-03-18T15:18:38.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:38 smithi104 ceph-mon[36070]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:18:38.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:38 smithi104 ceph-mon[36070]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:18:38.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:38 smithi104 ceph-mon[36070]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:18:38.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:38 smithi019 ceph-mon[31522]: Updating smithi019:/etc/ceph/ceph.client.admin.keyring 2024-03-18T15:18:38.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:38 smithi019 ceph-mon[31522]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:18:38.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:38 smithi019 ceph-mon[31522]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:18:38.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:38 smithi019 ceph-mon[31522]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:18:39.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:38 smithi033 ceph-mon[36648]: Updating smithi019:/etc/ceph/ceph.client.admin.keyring 2024-03-18T15:18:39.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:38 smithi033 ceph-mon[36648]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:18:39.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:38 smithi033 ceph-mon[36648]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:18:39.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:38 smithi033 ceph-mon[36648]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:18:39.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:39 smithi104 ceph-mon[36070]: Updating smithi019:/var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-18T15:18:39.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:39 smithi104 ceph-mon[36070]: pgmap v7: 129 pgs: 129 active+clean; 1.3 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 133 KiB/s wr, 80 op/s 2024-03-18T15:18:39.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:39 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:18:39.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:39 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:18:39.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:39 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:18:39.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:39 smithi104 ceph-mon[36070]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:18:39.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:39 smithi019 ceph-mon[31522]: Updating smithi019:/var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-18T15:18:39.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:39 smithi019 ceph-mon[31522]: pgmap v7: 129 pgs: 129 active+clean; 1.3 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 133 KiB/s wr, 80 op/s 2024-03-18T15:18:39.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:39 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:18:39.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:39 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:18:39.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:39 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:18:39.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:39 smithi019 ceph-mon[31522]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:18:40.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:39 smithi033 ceph-mon[36648]: Updating smithi019:/var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-18T15:18:40.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:39 smithi033 ceph-mon[36648]: pgmap v7: 129 pgs: 129 active+clean; 1.3 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 133 KiB/s wr, 80 op/s 2024-03-18T15:18:40.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:39 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:18:40.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:39 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:18:40.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:39 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:18:40.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:39 smithi033 ceph-mon[36648]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:18:41.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:41 smithi104 ceph-mon[36070]: pgmap v8: 129 pgs: 129 active+clean; 1.3 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 103 KiB/s wr, 62 op/s 2024-03-18T15:18:41.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:41 smithi019 ceph-mon[31522]: pgmap v8: 129 pgs: 129 active+clean; 1.3 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 103 KiB/s wr, 62 op/s 2024-03-18T15:18:42.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:41 smithi033 ceph-mon[36648]: pgmap v8: 129 pgs: 129 active+clean; 1.3 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 103 KiB/s wr, 62 op/s 2024-03-18T15:18:43.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:43 smithi104 ceph-mon[36070]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 84 KiB/s wr, 55 op/s 2024-03-18T15:18:43.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:43 smithi019 ceph-mon[31522]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 84 KiB/s wr, 55 op/s 2024-03-18T15:18:44.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:43 smithi033 ceph-mon[36648]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 84 KiB/s wr, 55 op/s 2024-03-18T15:18:45.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:45 smithi104 ceph-mon[36070]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 77 KiB/s wr, 50 op/s 2024-03-18T15:18:45.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:45 smithi019 ceph-mon[31522]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 77 KiB/s wr, 50 op/s 2024-03-18T15:18:46.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:45 smithi033 ceph-mon[36648]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 77 KiB/s wr, 50 op/s 2024-03-18T15:18:47.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:47 smithi104 ceph-mon[36070]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 77 KiB/s wr, 51 op/s 2024-03-18T15:18:47.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:47 smithi019 ceph-mon[31522]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 77 KiB/s wr, 51 op/s 2024-03-18T15:18:48.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:47 smithi033 ceph-mon[36648]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 77 KiB/s wr, 51 op/s 2024-03-18T15:18:49.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:49 smithi104 ceph-mon[36070]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 77 KiB/s wr, 55 op/s 2024-03-18T15:18:49.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:49 smithi019 ceph-mon[31522]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 77 KiB/s wr, 55 op/s 2024-03-18T15:18:50.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:49 smithi033 ceph-mon[36648]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 77 KiB/s wr, 55 op/s 2024-03-18T15:18:51.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:51 smithi104 ceph-mon[36070]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:18:51.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:51 smithi019 ceph-mon[31522]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:18:52.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:51 smithi033 ceph-mon[36648]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:18:53.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:53 smithi104 ceph-mon[36070]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:18:53.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:53 smithi019 ceph-mon[31522]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:18:54.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:53 smithi033 ceph-mon[36648]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:18:55.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:55 smithi104 ceph-mon[36070]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:18:55.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:55 smithi019 ceph-mon[31522]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:18:56.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:55 smithi033 ceph-mon[36648]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:18:57.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:57 smithi104 ceph-mon[36070]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-18T15:18:57.964 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:57 smithi019 ceph-mon[31522]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-18T15:18:58.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:57 smithi033 ceph-mon[36648]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-18T15:18:59.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:18:59 smithi104 ceph-mon[36070]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:18:59.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:18:59 smithi019 ceph-mon[31522]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:19:00.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:18:59 smithi033 ceph-mon[36648]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:19:00.715 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:19:00 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:19:00.421+0000 7f8ee9c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T15:19:00.715 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:19:00 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:19:00.421+0000 7f8ee9c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T15:19:00.715 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:19:00 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:19:00.421+0000 7f8ee9c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T15:19:00.715 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:19:00 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:19:00.421+0000 7f8ee9c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T15:19:00.715 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:19:00 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:19:00.421+0000 7f8ee9c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T15:19:01.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:19:01 smithi104 ceph-mon[36070]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:19:01.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:19:01 smithi019 ceph-mon[31522]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:19:02.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:19:01 smithi033 ceph-mon[36648]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:19:03.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:19:03 smithi104 ceph-mon[36070]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-18T15:19:03.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:19:03 smithi104 ceph-mon[36070]: mgrmap e26: x(active, since 32s), standbys: y, z 2024-03-18T15:19:03.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:19:03 smithi019 ceph-mon[31522]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-18T15:19:03.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:19:03 smithi019 ceph-mon[31522]: mgrmap e26: x(active, since 32s), standbys: y, z 2024-03-18T15:19:04.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:19:03 smithi033 ceph-mon[36648]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-18T15:19:04.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:19:03 smithi033 ceph-mon[36648]: mgrmap e26: x(active, since 32s), standbys: y, z 2024-03-18T15:19:05.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:19:05 smithi104 ceph-mon[36070]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-18T15:19:05.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:19:05 smithi019 ceph-mon[31522]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-18T15:19:06.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:19:05 smithi033 ceph-mon[36648]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-18T15:19:07.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:19:07 smithi104 ceph-mon[36070]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 24 op/s 2024-03-18T15:19:07.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:19:07 smithi019 ceph-mon[31522]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 24 op/s 2024-03-18T15:19:08.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:19:07 smithi033 ceph-mon[36648]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 24 op/s 2024-03-18T15:19:09.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:19:09 smithi104 ceph-mon[36070]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 27 op/s 2024-03-18T15:19:09.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:19:09 smithi019 ceph-mon[31522]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 27 op/s 2024-03-18T15:19:10.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:19:09 smithi033 ceph-mon[36648]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 27 op/s 2024-03-18T15:19:11.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:19:11 smithi104 ceph-mon[36070]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-03-18T15:19:11.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:19:11 smithi019 ceph-mon[31522]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-03-18T15:19:12.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:19:11 smithi033 ceph-mon[36648]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-03-18T15:19:13.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:19:13 smithi104 ceph-mon[36070]: pgmap v24: 129 pgs: 129 active+clean; 1.4 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 24 KiB/s wr, 28 op/s 2024-03-18T15:19:13.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:19:13 smithi019 ceph-mon[31522]: pgmap v24: 129 pgs: 129 active+clean; 1.4 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 24 KiB/s wr, 28 op/s 2024-03-18T15:19:14.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:19:13 smithi033 ceph-mon[36648]: pgmap v24: 129 pgs: 129 active+clean; 1.4 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 24 KiB/s wr, 28 op/s 2024-03-18T15:19:15.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:19:15 smithi104 ceph-mon[36070]: pgmap v25: 129 pgs: 129 active+clean; 1.4 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 1.5 KiB/s wr, 8 op/s 2024-03-18T15:19:15.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:19:15 smithi019 ceph-mon[31522]: pgmap v25: 129 pgs: 129 active+clean; 1.4 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 1.5 KiB/s wr, 8 op/s 2024-03-18T15:19:16.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:19:15 smithi033 ceph-mon[36648]: pgmap v25: 129 pgs: 129 active+clean; 1.4 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 1.5 KiB/s wr, 8 op/s 2024-03-18T15:19:17.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:19:17 smithi019 ceph-mon[31522]: pgmap v26: 129 pgs: 129 active+clean; 1.4 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 1.5 KiB/s wr, 9 op/s 2024-03-18T15:19:17.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:19:17 smithi104 ceph-mon[36070]: pgmap v26: 129 pgs: 129 active+clean; 1.4 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 1.5 KiB/s wr, 9 op/s 2024-03-18T15:19:18.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:19:17 smithi033 ceph-mon[36648]: pgmap v26: 129 pgs: 129 active+clean; 1.4 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 1.5 KiB/s wr, 9 op/s 2024-03-18T15:19:19.702 INFO:tasks.workunit.client.0.smithi019.stderr:Note: switching to '4e19f704afd9957ea657ab858f9ecf76401fef6f'. 2024-03-18T15:19:19.702 INFO:tasks.workunit.client.0.smithi019.stderr: 2024-03-18T15:19:19.702 INFO:tasks.workunit.client.0.smithi019.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-18T15:19:19.702 INFO:tasks.workunit.client.0.smithi019.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-18T15:19:19.703 INFO:tasks.workunit.client.0.smithi019.stderr:state without impacting any branches by switching back to a branch. 2024-03-18T15:19:19.703 INFO:tasks.workunit.client.0.smithi019.stderr: 2024-03-18T15:19:19.703 INFO:tasks.workunit.client.0.smithi019.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-18T15:19:19.703 INFO:tasks.workunit.client.0.smithi019.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-18T15:19:19.703 INFO:tasks.workunit.client.0.smithi019.stderr: 2024-03-18T15:19:19.703 INFO:tasks.workunit.client.0.smithi019.stderr: git switch -c 2024-03-18T15:19:19.703 INFO:tasks.workunit.client.0.smithi019.stderr: 2024-03-18T15:19:19.703 INFO:tasks.workunit.client.0.smithi019.stderr:Or undo this operation with: 2024-03-18T15:19:19.703 INFO:tasks.workunit.client.0.smithi019.stderr: 2024-03-18T15:19:19.703 INFO:tasks.workunit.client.0.smithi019.stderr: git switch - 2024-03-18T15:19:19.703 INFO:tasks.workunit.client.0.smithi019.stderr: 2024-03-18T15:19:19.703 INFO:tasks.workunit.client.0.smithi019.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-18T15:19:19.703 INFO:tasks.workunit.client.0.smithi019.stderr: 2024-03-18T15:19:19.703 INFO:tasks.workunit.client.0.smithi019.stderr:HEAD is now at 4e19f704afd ignorelist mon down 2024-03-18T15:19:19.715 DEBUG:teuthology.orchestra.run.smithi019:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-03-18T15:19:19.795 INFO:tasks.workunit.client.0.smithi019.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-18T15:19:19.804 INFO:tasks.workunit.client.0.smithi019.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-18T15:19:19.804 INFO:tasks.workunit.client.0.smithi019.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-18T15:19:19.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:19:19 smithi104 ceph-mon[36070]: pgmap v27: 129 pgs: 129 active+clean; 1.4 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 1.5 KiB/s wr, 12 op/s 2024-03-18T15:19:19.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:19:19 smithi019 ceph-mon[31522]: pgmap v27: 129 pgs: 129 active+clean; 1.4 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 1.5 KiB/s wr, 12 op/s 2024-03-18T15:19:20.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:19:19 smithi033 ceph-mon[36648]: pgmap v27: 129 pgs: 129 active+clean; 1.4 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 1.5 KiB/s wr, 12 op/s 2024-03-18T15:19:20.781 INFO:tasks.workunit.client.0.smithi019.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-18T15:19:20.909 INFO:tasks.workunit.client.0.smithi019.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-18T15:19:20.998 INFO:tasks.workunit.client.0.smithi019.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-18T15:19:21.005 INFO:tasks.workunit.client.0.smithi019.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-18T15:19:21.005 INFO:tasks.workunit.client.0.smithi019.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-18T15:19:21.094 INFO:tasks.workunit.client.0.smithi019.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-18T15:19:21.105 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-03-18T15:19:21.105 DEBUG:teuthology.orchestra.run.smithi019:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-03-18T15:19:21.159 INFO:tasks.workunit:Running workunits matching suites/fsync-tester.sh on client.0... 2024-03-18T15:19:21.160 INFO:tasks.workunit:Running workunit suites/fsync-tester.sh... 2024-03-18T15:19:21.161 DEBUG:teuthology.orchestra.run.smithi019:workunit test suites/fsync-tester.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4e19f704afd9957ea657ab858f9ecf76401fef6f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsync-tester.sh 2024-03-18T15:19:21.235 INFO:tasks.workunit.client.0.smithi019.stderr:+ wget http://download.ceph.com/qa/fsync-tester.c 2024-03-18T15:19:21.266 INFO:tasks.workunit.client.0.smithi019.stderr:--2024-03-18 15:19:21-- http://download.ceph.com/qa/fsync-tester.c 2024-03-18T15:19:21.268 INFO:tasks.workunit.client.0.smithi019.stderr:Resolving download.ceph.com (download.ceph.com)... 158.69.68.124, 2607:5300:201:2000::3:58a1 2024-03-18T15:19:21.305 INFO:tasks.workunit.client.0.smithi019.stderr:Connecting to download.ceph.com (download.ceph.com)|158.69.68.124|:80... connected. 2024-03-18T15:19:21.340 INFO:tasks.workunit.client.0.smithi019.stderr:HTTP request sent, awaiting response... 200 OK 2024-03-18T15:19:21.340 INFO:tasks.workunit.client.0.smithi019.stderr:Length: 3024 (3.0K) [application/octet-stream] 2024-03-18T15:19:21.343 INFO:tasks.workunit.client.0.smithi019.stderr:Saving to: ‘fsync-tester.c’ 2024-03-18T15:19:21.349 INFO:tasks.workunit.client.0.smithi019.stderr: 2024-03-18T15:19:21.349 INFO:tasks.workunit.client.0.smithi019.stderr: 0K .. 100% 122M=0s 2024-03-18T15:19:21.349 INFO:tasks.workunit.client.0.smithi019.stderr: 2024-03-18T15:19:21.349 INFO:tasks.workunit.client.0.smithi019.stderr:2024-03-18 15:19:21 (122 MB/s) - ‘fsync-tester.c’ saved [3024/3024] 2024-03-18T15:19:21.350 INFO:tasks.workunit.client.0.smithi019.stderr: 2024-03-18T15:19:21.351 INFO:tasks.workunit.client.0.smithi019.stderr:+ gcc -D_GNU_SOURCE fsync-tester.c -o fsync-tester 2024-03-18T15:19:21.483 INFO:tasks.workunit.client.0.smithi019.stderr:+ ./fsync-tester 2024-03-18T15:19:21.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:19:21 smithi104 ceph-mon[36070]: pgmap v28: 129 pgs: 129 active+clean; 1.4 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 853 B/s wr, 8 op/s 2024-03-18T15:19:21.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:19:21 smithi019 ceph-mon[31522]: pgmap v28: 129 pgs: 129 active+clean; 1.4 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 853 B/s wr, 8 op/s 2024-03-18T15:19:22.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:19:21 smithi033 ceph-mon[36648]: pgmap v28: 129 pgs: 129 active+clean; 1.4 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 853 B/s wr, 8 op/s 2024-03-18T15:19:23.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:19:23 smithi104 ceph-mon[36070]: pgmap v29: 129 pgs: 129 active+clean; 1.4 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 853 B/s wr, 12 op/s 2024-03-18T15:19:23.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:19:23 smithi019 ceph-mon[31522]: pgmap v29: 129 pgs: 129 active+clean; 1.4 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 853 B/s wr, 12 op/s 2024-03-18T15:19:24.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:19:23 smithi033 ceph-mon[36648]: pgmap v29: 129 pgs: 129 active+clean; 1.4 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 853 B/s wr, 12 op/s 2024-03-18T15:19:24.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:19:24 smithi104 ceph-mon[36070]: pgmap v30: 129 pgs: 129 active+clean; 1.4 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:19:24.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:19:24 smithi104 ceph-mon[36070]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T15:19:24.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:19:24 smithi019 ceph-mon[31522]: pgmap v30: 129 pgs: 129 active+clean; 1.4 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:19:24.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:19:24 smithi019 ceph-mon[31522]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T15:19:25.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:19:24 smithi033 ceph-mon[36648]: pgmap v30: 129 pgs: 129 active+clean; 1.4 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:19:25.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:19:24 smithi033 ceph-mon[36648]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T15:19:27.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:19:26 smithi104 ceph-mon[36070]: pgmap v31: 129 pgs: 129 active+clean; 2.4 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 343 KiB/s wr, 10 op/s 2024-03-18T15:19:27.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:19:26 smithi019 ceph-mon[31522]: pgmap v31: 129 pgs: 129 active+clean; 2.4 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 343 KiB/s wr, 10 op/s 2024-03-18T15:19:27.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:19:26 smithi033 ceph-mon[36648]: pgmap v31: 129 pgs: 129 active+clean; 2.4 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 343 KiB/s wr, 10 op/s 2024-03-18T15:19:29.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:19:28 smithi104 ceph-mon[36070]: pgmap v32: 129 pgs: 129 active+clean; 2.4 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 350 KiB/s wr, 14 op/s 2024-03-18T15:19:29.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:19:28 smithi019 ceph-mon[31522]: pgmap v32: 129 pgs: 129 active+clean; 2.4 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 350 KiB/s wr, 14 op/s 2024-03-18T15:19:29.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:19:28 smithi033 ceph-mon[36648]: pgmap v32: 129 pgs: 129 active+clean; 2.4 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 350 KiB/s wr, 14 op/s 2024-03-18T15:19:31.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:19:30 smithi104 ceph-mon[36070]: pgmap v33: 129 pgs: 129 active+clean; 2.4 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 350 KiB/s wr, 11 op/s 2024-03-18T15:19:31.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:19:30 smithi019 ceph-mon[31522]: pgmap v33: 129 pgs: 129 active+clean; 2.4 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 350 KiB/s wr, 11 op/s 2024-03-18T15:19:31.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:19:30 smithi033 ceph-mon[36648]: pgmap v33: 129 pgs: 129 active+clean; 2.4 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 350 KiB/s wr, 11 op/s 2024-03-18T15:19:33.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:19:32 smithi104 ceph-mon[36070]: pgmap v34: 129 pgs: 129 active+clean; 2.5 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 778 KiB/s wr, 16 op/s 2024-03-18T15:19:33.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:19:32 smithi019 ceph-mon[31522]: pgmap v34: 129 pgs: 129 active+clean; 2.5 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 778 KiB/s wr, 16 op/s 2024-03-18T15:19:33.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:19:32 smithi033 ceph-mon[36648]: pgmap v34: 129 pgs: 129 active+clean; 2.5 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 778 KiB/s wr, 16 op/s 2024-03-18T15:19:35.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:19:34 smithi104 ceph-mon[36070]: pgmap v35: 129 pgs: 129 active+clean; 2.5 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 778 KiB/s wr, 12 op/s 2024-03-18T15:19:35.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:19:34 smithi019 ceph-mon[31522]: pgmap v35: 129 pgs: 129 active+clean; 2.5 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 778 KiB/s wr, 12 op/s 2024-03-18T15:19:35.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:19:34 smithi033 ceph-mon[36648]: pgmap v35: 129 pgs: 129 active+clean; 2.5 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 778 KiB/s wr, 12 op/s 2024-03-18T15:19:37.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:19:36 smithi104 ceph-mon[36070]: pgmap v36: 129 pgs: 129 active+clean; 2.5 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 1.2 MiB/s wr, 15 op/s 2024-03-18T15:19:37.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:19:36 smithi019 ceph-mon[31522]: pgmap v36: 129 pgs: 129 active+clean; 2.5 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 1.2 MiB/s wr, 15 op/s 2024-03-18T15:19:37.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:19:36 smithi033 ceph-mon[36648]: pgmap v36: 129 pgs: 129 active+clean; 2.5 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 1.2 MiB/s wr, 15 op/s 2024-03-18T15:19:39.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:19:38 smithi104 ceph-mon[36070]: pgmap v37: 129 pgs: 129 active+clean; 2.5 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 863 KiB/s wr, 15 op/s 2024-03-18T15:19:39.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:19:38 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:19:39.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:19:38 smithi019 ceph-mon[31522]: pgmap v37: 129 pgs: 129 active+clean; 2.5 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 863 KiB/s wr, 15 op/s 2024-03-18T15:19:39.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:19:38 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:19:39.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:19:38 smithi033 ceph-mon[36648]: pgmap v37: 129 pgs: 129 active+clean; 2.5 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 863 KiB/s wr, 15 op/s 2024-03-18T15:19:39.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:19:38 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:19:41.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:19:40 smithi104 ceph-mon[36070]: pgmap v38: 129 pgs: 129 active+clean; 2.5 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 856 KiB/s wr, 10 op/s 2024-03-18T15:19:41.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:19:40 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:19:41.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:19:40 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:19:41.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:19:40 smithi104 ceph-mon[36070]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:19:41.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:19:40 smithi019 ceph-mon[31522]: pgmap v38: 129 pgs: 129 active+clean; 2.5 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 856 KiB/s wr, 10 op/s 2024-03-18T15:19:41.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:19:40 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:19:41.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:19:40 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:19:41.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:19:40 smithi019 ceph-mon[31522]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:19:41.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:19:40 smithi033 ceph-mon[36648]: pgmap v38: 129 pgs: 129 active+clean; 2.5 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 856 KiB/s wr, 10 op/s 2024-03-18T15:19:41.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:19:40 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:19:41.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:19:40 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:19:41.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:19:40 smithi033 ceph-mon[36648]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:19:43.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:19:42 smithi104 ceph-mon[36070]: pgmap v39: 129 pgs: 129 active+clean; 2.5 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 1.3 MiB/s wr, 16 op/s 2024-03-18T15:19:43.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:19:42 smithi019 ceph-mon[31522]: pgmap v39: 129 pgs: 129 active+clean; 2.5 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 1.3 MiB/s wr, 16 op/s 2024-03-18T15:19:43.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:19:42 smithi033 ceph-mon[36648]: pgmap v39: 129 pgs: 129 active+clean; 2.5 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 1.3 MiB/s wr, 16 op/s 2024-03-18T15:19:45.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:19:44 smithi104 ceph-mon[36070]: pgmap v40: 129 pgs: 129 active+clean; 2.5 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 856 KiB/s wr, 11 op/s 2024-03-18T15:19:45.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:19:44 smithi019 ceph-mon[31522]: pgmap v40: 129 pgs: 129 active+clean; 2.5 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 856 KiB/s wr, 11 op/s 2024-03-18T15:19:45.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:19:44 smithi033 ceph-mon[36648]: pgmap v40: 129 pgs: 129 active+clean; 2.5 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 856 KiB/s wr, 11 op/s 2024-03-18T15:19:47.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:19:46 smithi104 ceph-mon[36070]: pgmap v41: 129 pgs: 129 active+clean; 2.5 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 1.3 MiB/s wr, 16 op/s 2024-03-18T15:19:47.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:19:46 smithi019 ceph-mon[31522]: pgmap v41: 129 pgs: 129 active+clean; 2.5 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 1.3 MiB/s wr, 16 op/s 2024-03-18T15:19:47.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:19:46 smithi033 ceph-mon[36648]: pgmap v41: 129 pgs: 129 active+clean; 2.5 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 1.3 MiB/s wr, 16 op/s 2024-03-18T15:19:49.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:19:48 smithi104 ceph-mon[36070]: pgmap v42: 129 pgs: 129 active+clean; 2.5 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 856 KiB/s wr, 14 op/s 2024-03-18T15:19:49.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:19:48 smithi019 ceph-mon[31522]: pgmap v42: 129 pgs: 129 active+clean; 2.5 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 856 KiB/s wr, 14 op/s 2024-03-18T15:19:49.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:19:48 smithi033 ceph-mon[36648]: pgmap v42: 129 pgs: 129 active+clean; 2.5 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 856 KiB/s wr, 14 op/s 2024-03-18T15:19:51.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:19:50 smithi104 ceph-mon[36070]: pgmap v43: 129 pgs: 129 active+clean; 2.5 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 856 KiB/s wr, 11 op/s 2024-03-18T15:19:51.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:19:50 smithi019 ceph-mon[31522]: pgmap v43: 129 pgs: 129 active+clean; 2.5 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 856 KiB/s wr, 11 op/s 2024-03-18T15:19:51.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:19:50 smithi033 ceph-mon[36648]: pgmap v43: 129 pgs: 129 active+clean; 2.5 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 856 KiB/s wr, 11 op/s 2024-03-18T15:19:53.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:19:52 smithi104 ceph-mon[36070]: pgmap v44: 129 pgs: 129 active+clean; 2.5 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 1.3 MiB/s wr, 16 op/s 2024-03-18T15:19:53.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:19:52 smithi019 ceph-mon[31522]: pgmap v44: 129 pgs: 129 active+clean; 2.5 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 1.3 MiB/s wr, 16 op/s 2024-03-18T15:19:53.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:19:52 smithi033 ceph-mon[36648]: pgmap v44: 129 pgs: 129 active+clean; 2.5 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 1.3 MiB/s wr, 16 op/s 2024-03-18T15:19:55.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:19:54 smithi104 ceph-mon[36070]: pgmap v45: 129 pgs: 129 active+clean; 2.5 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 856 KiB/s wr, 10 op/s 2024-03-18T15:19:55.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:19:54 smithi019 ceph-mon[31522]: pgmap v45: 129 pgs: 129 active+clean; 2.5 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 856 KiB/s wr, 10 op/s 2024-03-18T15:19:55.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:19:54 smithi033 ceph-mon[36648]: pgmap v45: 129 pgs: 129 active+clean; 2.5 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 856 KiB/s wr, 10 op/s 2024-03-18T15:19:57.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:19:56 smithi104 ceph-mon[36070]: pgmap v46: 129 pgs: 129 active+clean; 211 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 19 MiB/s wr, 24 op/s 2024-03-18T15:19:57.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:19:56 smithi019 ceph-mon[31522]: pgmap v46: 129 pgs: 129 active+clean; 211 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 19 MiB/s wr, 24 op/s 2024-03-18T15:19:57.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:19:56 smithi033 ceph-mon[36648]: pgmap v46: 129 pgs: 129 active+clean; 211 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 19 MiB/s wr, 24 op/s 2024-03-18T15:19:59.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:19:58 smithi104 ceph-mon[36070]: pgmap v47: 129 pgs: 129 active+clean; 259 MiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 22 MiB/s wr, 22 op/s 2024-03-18T15:19:59.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:19:58 smithi019 ceph-mon[31522]: pgmap v47: 129 pgs: 129 active+clean; 259 MiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 22 MiB/s wr, 22 op/s 2024-03-18T15:19:59.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:19:58 smithi033 ceph-mon[36648]: pgmap v47: 129 pgs: 129 active+clean; 259 MiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 22 MiB/s wr, 22 op/s 2024-03-18T15:20:01.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:20:00 smithi104 ceph-mon[36070]: pgmap v48: 129 pgs: 129 active+clean; 259 MiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 22 MiB/s wr, 21 op/s 2024-03-18T15:20:01.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:20:00 smithi104 ceph-mon[36070]: overall HEALTH_OK 2024-03-18T15:20:01.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:20:00 smithi019 ceph-mon[31522]: pgmap v48: 129 pgs: 129 active+clean; 259 MiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 22 MiB/s wr, 21 op/s 2024-03-18T15:20:01.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:20:00 smithi019 ceph-mon[31522]: overall HEALTH_OK 2024-03-18T15:20:01.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:20:00 smithi033 ceph-mon[36648]: pgmap v48: 129 pgs: 129 active+clean; 259 MiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 22 MiB/s wr, 21 op/s 2024-03-18T15:20:01.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:20:00 smithi033 ceph-mon[36648]: overall HEALTH_OK 2024-03-18T15:20:03.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:20:02 smithi104 ceph-mon[36070]: pgmap v49: 129 pgs: 129 active+clean; 259 MiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 23 MiB/s wr, 48 op/s 2024-03-18T15:20:03.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:20:02 smithi104 ceph-mon[36070]: mgrmap e27: x(active, since 92s), standbys: y, z 2024-03-18T15:20:03.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:20:02 smithi019 ceph-mon[31522]: pgmap v49: 129 pgs: 129 active+clean; 259 MiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 23 MiB/s wr, 48 op/s 2024-03-18T15:20:03.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:20:02 smithi019 ceph-mon[31522]: mgrmap e27: x(active, since 92s), standbys: y, z 2024-03-18T15:20:03.229 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:20:02 smithi033 ceph-mon[36648]: pgmap v49: 129 pgs: 129 active+clean; 259 MiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 23 MiB/s wr, 48 op/s 2024-03-18T15:20:03.229 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:20:02 smithi033 ceph-mon[36648]: mgrmap e27: x(active, since 92s), standbys: y, z 2024-03-18T15:20:05.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:20:04 smithi104 ceph-mon[36070]: pgmap v50: 129 pgs: 129 active+clean; 259 MiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 22 MiB/s wr, 43 op/s 2024-03-18T15:20:05.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:20:04 smithi019 ceph-mon[31522]: pgmap v50: 129 pgs: 129 active+clean; 259 MiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 22 MiB/s wr, 43 op/s 2024-03-18T15:20:05.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:20:04 smithi033 ceph-mon[36648]: pgmap v50: 129 pgs: 129 active+clean; 259 MiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 22 MiB/s wr, 43 op/s 2024-03-18T15:20:07.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:20:06 smithi019 ceph-mon[31522]: pgmap v51: 129 pgs: 129 active+clean; 260 MiB data, 443 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 23 MiB/s wr, 135 op/s 2024-03-18T15:20:07.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:20:06 smithi033 ceph-mon[36648]: pgmap v51: 129 pgs: 129 active+clean; 260 MiB data, 443 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 23 MiB/s wr, 135 op/s 2024-03-18T15:20:07.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:20:06 smithi104 ceph-mon[36070]: pgmap v51: 129 pgs: 129 active+clean; 260 MiB data, 443 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 23 MiB/s wr, 135 op/s 2024-03-18T15:20:09.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:20:08 smithi019 ceph-mon[31522]: pgmap v52: 129 pgs: 129 active+clean; 260 MiB data, 443 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 5.2 MiB/s wr, 121 op/s 2024-03-18T15:20:09.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:20:08 smithi033 ceph-mon[36648]: pgmap v52: 129 pgs: 129 active+clean; 260 MiB data, 443 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 5.2 MiB/s wr, 121 op/s 2024-03-18T15:20:09.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:20:08 smithi104 ceph-mon[36070]: pgmap v52: 129 pgs: 129 active+clean; 260 MiB data, 443 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 5.2 MiB/s wr, 121 op/s 2024-03-18T15:20:11.206 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:20:10 smithi104 ceph-mon[36070]: pgmap v53: 129 pgs: 129 active+clean; 260 MiB data, 443 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 1.2 MiB/s wr, 118 op/s 2024-03-18T15:20:11.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:20:10 smithi019 ceph-mon[31522]: pgmap v53: 129 pgs: 129 active+clean; 260 MiB data, 443 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 1.2 MiB/s wr, 118 op/s 2024-03-18T15:20:11.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:20:10 smithi033 ceph-mon[36648]: pgmap v53: 129 pgs: 129 active+clean; 260 MiB data, 443 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 1.2 MiB/s wr, 118 op/s 2024-03-18T15:20:13.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:20:12 smithi019 ceph-mon[31522]: pgmap v54: 129 pgs: 129 active+clean; 260 MiB data, 496 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 2.0 MiB/s wr, 211 op/s 2024-03-18T15:20:13.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:20:12 smithi033 ceph-mon[36648]: pgmap v54: 129 pgs: 129 active+clean; 260 MiB data, 496 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 2.0 MiB/s wr, 211 op/s 2024-03-18T15:20:13.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:20:12 smithi104 ceph-mon[36070]: pgmap v54: 129 pgs: 129 active+clean; 260 MiB data, 496 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 2.0 MiB/s wr, 211 op/s 2024-03-18T15:20:15.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:20:14 smithi033 ceph-mon[36648]: pgmap v55: 129 pgs: 129 active+clean; 260 MiB data, 496 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 1.5 MiB/s wr, 184 op/s 2024-03-18T15:20:15.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:20:14 smithi104 ceph-mon[36070]: pgmap v55: 129 pgs: 129 active+clean; 260 MiB data, 496 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 1.5 MiB/s wr, 184 op/s 2024-03-18T15:20:15.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:20:14 smithi019 ceph-mon[31522]: pgmap v55: 129 pgs: 129 active+clean; 260 MiB data, 496 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 1.5 MiB/s wr, 184 op/s 2024-03-18T15:20:17.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:20:16 smithi033 ceph-mon[36648]: pgmap v56: 129 pgs: 129 active+clean; 260 MiB data, 522 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 3.1 MiB/s wr, 289 op/s 2024-03-18T15:20:17.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:20:16 smithi104 ceph-mon[36070]: pgmap v56: 129 pgs: 129 active+clean; 260 MiB data, 522 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 3.1 MiB/s wr, 289 op/s 2024-03-18T15:20:17.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:20:16 smithi019 ceph-mon[31522]: pgmap v56: 129 pgs: 129 active+clean; 260 MiB data, 522 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 3.1 MiB/s wr, 289 op/s 2024-03-18T15:20:19.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:20:18 smithi033 ceph-mon[36648]: pgmap v57: 129 pgs: 129 active+clean; 260 MiB data, 522 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 2.4 MiB/s wr, 198 op/s 2024-03-18T15:20:19.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:20:18 smithi104 ceph-mon[36070]: pgmap v57: 129 pgs: 129 active+clean; 260 MiB data, 522 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 2.4 MiB/s wr, 198 op/s 2024-03-18T15:20:19.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:20:18 smithi019 ceph-mon[31522]: pgmap v57: 129 pgs: 129 active+clean; 260 MiB data, 522 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 2.4 MiB/s wr, 198 op/s 2024-03-18T15:20:21.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:20:20 smithi033 ceph-mon[36648]: pgmap v58: 129 pgs: 129 active+clean; 260 MiB data, 522 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 2.4 MiB/s wr, 198 op/s 2024-03-18T15:20:21.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:20:20 smithi104 ceph-mon[36070]: pgmap v58: 129 pgs: 129 active+clean; 260 MiB data, 522 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 2.4 MiB/s wr, 198 op/s 2024-03-18T15:20:21.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:20:20 smithi019 ceph-mon[31522]: pgmap v58: 129 pgs: 129 active+clean; 260 MiB data, 522 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 2.4 MiB/s wr, 198 op/s 2024-03-18T15:20:22.478 INFO:tasks.workunit.client.0.smithi019.stdout:setting up random write file 2024-03-18T15:20:22.478 INFO:tasks.workunit.client.0.smithi019.stdout:done setting up random write file 2024-03-18T15:20:22.478 INFO:tasks.workunit.client.0.smithi019.stdout:starting fsync run 2024-03-18T15:20:22.478 INFO:tasks.workunit.client.0.smithi019.stdout:write time: 0.0010s fsync time: 0.0410s 2024-03-18T15:20:22.479 INFO:tasks.workunit.client.0.smithi019.stdout:write time: 0.0017s fsync time: 0.0215s 2024-03-18T15:20:22.479 INFO:tasks.workunit.client.0.smithi019.stdout:write time: 0.0008s fsync time: 0.0206s 2024-03-18T15:20:22.479 INFO:tasks.workunit.client.0.smithi019.stdout:write time: 0.0012s fsync time: 0.0305s 2024-03-18T15:20:22.479 INFO:tasks.workunit.client.0.smithi019.stdout:write time: 0.0013s fsync time: 0.0183s 2024-03-18T15:20:22.479 INFO:tasks.workunit.client.0.smithi019.stdout:write time: 0.0012s fsync time: 0.0235s 2024-03-18T15:20:22.479 INFO:tasks.workunit.client.0.smithi019.stdout:write time: 0.0012s fsync time: 0.0289s 2024-03-18T15:20:22.479 INFO:tasks.workunit.client.0.smithi019.stdout:write time: 0.0013s fsync time: 0.0186s 2024-03-18T15:20:22.479 INFO:tasks.workunit.client.0.smithi019.stdout:write time: 0.0012s fsync time: 0.0204s 2024-03-18T15:20:22.479 INFO:tasks.workunit.client.0.smithi019.stdout:write time: 0.0013s fsync time: 0.0184s 2024-03-18T15:20:22.479 INFO:tasks.workunit.client.0.smithi019.stdout:write time: 0.0008s fsync time: 0.0154s 2024-03-18T15:20:22.479 INFO:tasks.workunit.client.0.smithi019.stdout:write time: 0.0012s fsync time: 0.0184s 2024-03-18T15:20:22.479 INFO:tasks.workunit.client.0.smithi019.stdout:write time: 0.0013s fsync time: 0.0182s 2024-03-18T15:20:22.480 INFO:tasks.workunit.client.0.smithi019.stdout:write time: 0.0012s fsync time: 0.0285s 2024-03-18T15:20:22.480 INFO:tasks.workunit.client.0.smithi019.stdout:write time: 0.0012s fsync time: 0.0182s 2024-03-18T15:20:22.480 INFO:tasks.workunit.client.0.smithi019.stdout:write time: 0.0013s fsync time: 0.0174s 2024-03-18T15:20:22.480 INFO:tasks.workunit.client.0.smithi019.stdout:write time: 0.0013s fsync time: 0.0276s 2024-03-18T15:20:22.480 INFO:tasks.workunit.client.0.smithi019.stdout:write time: 0.0005s fsync time: 0.0103s 2024-03-18T15:20:22.480 INFO:tasks.workunit.client.0.smithi019.stdout:write time: 0.0012s fsync time: 0.0253s 2024-03-18T15:20:22.480 INFO:tasks.workunit.client.0.smithi019.stdout:write time: 0.0013s fsync time: 0.0196s 2024-03-18T15:20:22.480 INFO:tasks.workunit.client.0.smithi019.stdout:write time: 0.0013s fsync time: 0.0155s 2024-03-18T15:20:22.480 INFO:tasks.workunit.client.0.smithi019.stdout:write time: 0.0013s fsync time: 0.0202s 2024-03-18T15:20:22.480 INFO:tasks.workunit.client.0.smithi019.stdout:write time: 0.0013s fsync time: 0.0160s 2024-03-18T15:20:22.480 INFO:tasks.workunit.client.0.smithi019.stdout:write time: 0.0013s fsync time: 0.0177s 2024-03-18T15:20:22.480 INFO:tasks.workunit.client.0.smithi019.stdout:write time: 0.0013s fsync time: 0.0207s 2024-03-18T15:20:22.480 INFO:tasks.workunit.client.0.smithi019.stdout:write time: 0.0012s fsync time: 0.0184s 2024-03-18T15:20:22.481 INFO:tasks.workunit.client.0.smithi019.stdout:write time: 0.0013s fsync time: 0.0248s 2024-03-18T15:20:22.481 INFO:tasks.workunit.client.0.smithi019.stdout:write time: 0.0008s fsync time: 0.0168s 2024-03-18T15:20:22.481 INFO:tasks.workunit.client.0.smithi019.stdout:write time: 0.0013s fsync time: 0.0189s 2024-03-18T15:20:22.481 INFO:tasks.workunit.client.0.smithi019.stdout:write time: 0.0012s fsync time: 0.0174s 2024-03-18T15:20:22.481 INFO:tasks.workunit.client.0.smithi019.stdout:write time: 0.0042s fsync time: 0.0881s 2024-03-18T15:20:22.481 INFO:tasks.workunit.client.0.smithi019.stdout:write time: 0.0013s fsync time: 0.0139s 2024-03-18T15:20:22.481 INFO:tasks.workunit.client.0.smithi019.stdout:write time: 0.0013s fsync time: 0.0155s 2024-03-18T15:20:22.481 INFO:tasks.workunit.client.0.smithi019.stdout:write time: 0.0013s fsync time: 0.0176s 2024-03-18T15:20:22.481 INFO:tasks.workunit.client.0.smithi019.stdout:write time: 0.0013s fsync time: 0.0182s 2024-03-18T15:20:22.481 INFO:tasks.workunit.client.0.smithi019.stdout:write time: 0.0012s fsync time: 0.0185s 2024-03-18T15:20:22.481 INFO:tasks.workunit.client.0.smithi019.stdout:write time: 0.0012s fsync time: 0.0173s 2024-03-18T15:20:22.481 INFO:tasks.workunit.client.0.smithi019.stdout:write time: 0.0012s fsync time: 0.0192s 2024-03-18T15:20:22.482 INFO:tasks.workunit.client.0.smithi019.stdout:write time: 0.0016s fsync time: 0.0215s 2024-03-18T15:20:22.482 INFO:tasks.workunit.client.0.smithi019.stdout:write time: 0.0014s fsync time: 0.0186s 2024-03-18T15:20:22.482 INFO:tasks.workunit.client.0.smithi019.stdout:write time: 0.0014s fsync time: 0.0283s 2024-03-18T15:20:22.482 INFO:tasks.workunit.client.0.smithi019.stdout:write time: 0.0014s fsync time: 0.0250s 2024-03-18T15:20:22.482 INFO:tasks.workunit.client.0.smithi019.stdout:write time: 0.0018s fsync time: 0.0258s 2024-03-18T15:20:22.482 INFO:tasks.workunit.client.0.smithi019.stdout:write time: 0.0017s fsync time: 0.0240s 2024-03-18T15:20:22.482 INFO:tasks.workunit.client.0.smithi019.stdout:write time: 0.0017s fsync time: 0.0204s 2024-03-18T15:20:22.482 INFO:tasks.workunit.client.0.smithi019.stdout:write time: 0.0018s fsync time: 0.0213s 2024-03-18T15:20:22.482 INFO:tasks.workunit.client.0.smithi019.stdout:write time: 0.0011s fsync time: 0.0167s 2024-03-18T15:20:22.482 INFO:tasks.workunit.client.0.smithi019.stdout:write time: 0.0019s fsync time: 0.0239s 2024-03-18T15:20:22.482 INFO:tasks.workunit.client.0.smithi019.stdout:write time: 0.0014s fsync time: 0.0178s 2024-03-18T15:20:22.483 INFO:tasks.workunit.client.0.smithi019.stdout:write time: 0.0013s fsync time: 0.0168s 2024-03-18T15:20:22.483 INFO:tasks.workunit.client.0.smithi019.stdout:write time: 0.0020s fsync time: 0.0230s 2024-03-18T15:20:22.483 INFO:tasks.workunit.client.0.smithi019.stdout:write time: 0.0013s fsync time: 0.0224s 2024-03-18T15:20:22.483 INFO:tasks.workunit.client.0.smithi019.stdout:write time: 0.0021s fsync time: 0.0204s 2024-03-18T15:20:22.483 INFO:tasks.workunit.client.0.smithi019.stdout:write time: 0.0014s fsync time: 0.0208s 2024-03-18T15:20:22.483 INFO:tasks.workunit.client.0.smithi019.stdout:write time: 0.0022s fsync time: 0.0225s 2024-03-18T15:20:22.483 INFO:tasks.workunit.client.0.smithi019.stdout:write time: 0.0018s fsync time: 0.0187s 2024-03-18T15:20:22.483 INFO:tasks.workunit.client.0.smithi019.stdout:write time: 0.0017s fsync time: 0.0258s 2024-03-18T15:20:22.483 INFO:tasks.workunit.client.0.smithi019.stdout:write time: 0.0018s fsync time: 0.0170s 2024-03-18T15:20:22.483 INFO:tasks.workunit.client.0.smithi019.stdout:write time: 0.0011s fsync time: 0.0192s 2024-03-18T15:20:22.483 INFO:tasks.workunit.client.0.smithi019.stdout:write time: 0.0023s fsync time: 0.0194s 2024-03-18T15:20:22.483 INFO:tasks.workunit.client.0.smithi019.stdout:run done 60 fsyncs total, killing random writer 2024-03-18T15:20:22.484 INFO:tasks.workunit.client.0.smithi019.stderr:+ echo /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-03-18T15:20:22.484 INFO:tasks.workunit.client.0.smithi019.stdout:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-03-18T15:20:22.484 INFO:tasks.workunit.client.0.smithi019.stderr:+ whereis lsof 2024-03-18T15:20:22.533 INFO:tasks.workunit.client.0.smithi019.stdout:lsof: /usr/bin/lsof /usr/share/man/man1/lsof.1.gz 2024-03-18T15:20:22.533 INFO:tasks.workunit.client.0.smithi019.stderr:+ lsof 2024-03-18T15:20:22.562 INFO:tasks.workunit.client.0.smithi019.stderr:lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/b66b5134a4ea952a92928f21fb342e7679f069674a569961be2c1b90d235945e/merged 2024-03-18T15:20:22.562 INFO:tasks.workunit.client.0.smithi019.stderr: Output information may be incomplete. 2024-03-18T15:20:22.562 INFO:tasks.workunit.client.0.smithi019.stderr:lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/e60a447f27e2255bcf4d845a3c2b10cdce0d123e7f9c1600f01d88156fc78fa6/merged 2024-03-18T15:20:22.562 INFO:tasks.workunit.client.0.smithi019.stderr: Output information may be incomplete. 2024-03-18T15:20:22.562 INFO:tasks.workunit.client.0.smithi019.stderr:lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/fcb443e7538fc4219fea707f22f042e6977d78e6e510f2473c9fc6ed58aa1d34/merged 2024-03-18T15:20:22.562 INFO:tasks.workunit.client.0.smithi019.stderr: Output information may be incomplete. 2024-03-18T15:20:22.565 INFO:tasks.workunit.client.0.smithi019.stderr:lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/e3a32159e383b655b5643b5c40f57f9ae93b443eb771b78e93af30c1cca13282/merged 2024-03-18T15:20:22.565 INFO:tasks.workunit.client.0.smithi019.stderr: Output information may be incomplete. 2024-03-18T15:20:22.566 INFO:tasks.workunit.client.0.smithi019.stderr:lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/c216e79ccf5d731cc08d1498cd6fffe6c6d29d80151efd5cf5140ed9388582c0/merged 2024-03-18T15:20:22.566 INFO:tasks.workunit.client.0.smithi019.stderr: Output information may be incomplete. 2024-03-18T15:20:22.566 INFO:tasks.workunit.client.0.smithi019.stderr:lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/eba1e8dfb65d6f9f3e04558d977da6b5383b5beecbb34d0e5106b89f9e9180e8/merged 2024-03-18T15:20:22.566 INFO:tasks.workunit.client.0.smithi019.stderr: Output information may be incomplete. 2024-03-18T15:20:22.566 INFO:tasks.workunit.client.0.smithi019.stderr:lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/9f4f55fd97845b81e87f9d9c7089affd7ca50033e50a03c261d853276da91aa8/merged 2024-03-18T15:20:22.566 INFO:tasks.workunit.client.0.smithi019.stderr: Output information may be incomplete. 2024-03-18T15:20:22.566 INFO:tasks.workunit.client.0.smithi019.stderr:lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/49d7e6a26ceb5c7a3cd0f5e158afc92b78433a8e69424c1bc233934a6f777e5e/merged 2024-03-18T15:20:22.566 INFO:tasks.workunit.client.0.smithi019.stderr: Output information may be incomplete. 2024-03-18T15:20:22.566 INFO:tasks.workunit.client.0.smithi019.stderr:lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/c6831eb51c280a14b72d1e50e88539fce99785bb58ad511a621d332e0274480c/merged 2024-03-18T15:20:22.566 INFO:tasks.workunit.client.0.smithi019.stderr: Output information may be incomplete. 2024-03-18T15:20:22.566 INFO:tasks.workunit.client.0.smithi019.stderr:lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/c06b6969a39b74e3b6a133480921beb81a0059fb3cdb30846e7535612f53cf3e/merged 2024-03-18T15:20:22.566 INFO:tasks.workunit.client.0.smithi019.stderr: Output information may be incomplete. 2024-03-18T15:20:22.721 INFO:tasks.workunit.client.0.smithi019.stdout:COMMAND PID TID TASKCMD USER FD TYPE DEVICE SIZE/OFF NODE NAME 2024-03-18T15:20:22.721 INFO:tasks.workunit.client.0.smithi019.stdout:systemd 1 root cwd unknown /proc/1/cwd (readlink: Permission denied) 2024-03-18T15:20:22.721 INFO:tasks.workunit.client.0.smithi019.stdout:systemd 1 root rtd unknown /proc/1/root (readlink: Permission denied) 2024-03-18T15:20:22.721 INFO:tasks.workunit.client.0.smithi019.stdout:systemd 1 root txt unknown /proc/1/exe (readlink: Permission denied) 2024-03-18T15:20:22.721 INFO:tasks.workunit.client.0.smithi019.stdout:systemd 1 root NOFD /proc/1/fd (opendir: Permission denied) 2024-03-18T15:20:22.722 INFO:tasks.workunit.client.0.smithi019.stdout:kthreadd 2 root cwd unknown /proc/2/cwd (readlink: Permission denied) 2024-03-18T15:20:22.722 INFO:tasks.workunit.client.0.smithi019.stdout:kthreadd 2 root rtd unknown /proc/2/root (readlink: Permission denied) 2024-03-18T15:20:22.722 INFO:tasks.workunit.client.0.smithi019.stdout:kthreadd 2 root txt unknown /proc/2/exe (readlink: Permission denied) 2024-03-18T15:20:22.722 INFO:tasks.workunit.client.0.smithi019.stdout:kthreadd 2 root NOFD /proc/2/fd (opendir: Permission denied) 2024-03-18T15:20:22.722 INFO:tasks.workunit.client.0.smithi019.stdout:pool_work 3 root cwd unknown /proc/3/cwd (readlink: Permission denied) 2024-03-18T15:20:22.722 INFO:tasks.workunit.client.0.smithi019.stdout:pool_work 3 root rtd unknown /proc/3/root (readlink: Permission denied) 2024-03-18T15:20:22.722 INFO:tasks.workunit.client.0.smithi019.stdout:pool_work 3 root txt unknown /proc/3/exe (readlink: Permission denied) 2024-03-18T15:20:22.722 INFO:tasks.workunit.client.0.smithi019.stdout:pool_work 3 root NOFD /proc/3/fd (opendir: Permission denied) 2024-03-18T15:20:22.722 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 4 root cwd unknown /proc/4/cwd (readlink: Permission denied) 2024-03-18T15:20:22.722 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 4 root rtd unknown /proc/4/root (readlink: Permission denied) 2024-03-18T15:20:22.722 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 4 root txt unknown /proc/4/exe (readlink: Permission denied) 2024-03-18T15:20:22.722 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 4 root NOFD /proc/4/fd (opendir: Permission denied) 2024-03-18T15:20:22.722 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 5 root cwd unknown /proc/5/cwd (readlink: Permission denied) 2024-03-18T15:20:22.723 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 5 root rtd unknown /proc/5/root (readlink: Permission denied) 2024-03-18T15:20:22.723 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 5 root txt unknown /proc/5/exe (readlink: Permission denied) 2024-03-18T15:20:22.723 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 5 root NOFD /proc/5/fd (opendir: Permission denied) 2024-03-18T15:20:22.723 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 6 root cwd unknown /proc/6/cwd (readlink: Permission denied) 2024-03-18T15:20:22.723 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 6 root rtd unknown /proc/6/root (readlink: Permission denied) 2024-03-18T15:20:22.723 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 6 root txt unknown /proc/6/exe (readlink: Permission denied) 2024-03-18T15:20:22.723 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 6 root NOFD /proc/6/fd (opendir: Permission denied) 2024-03-18T15:20:22.723 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 7 root cwd unknown /proc/7/cwd (readlink: Permission denied) 2024-03-18T15:20:22.723 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 7 root rtd unknown /proc/7/root (readlink: Permission denied) 2024-03-18T15:20:22.723 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 7 root txt unknown /proc/7/exe (readlink: Permission denied) 2024-03-18T15:20:22.723 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 7 root NOFD /proc/7/fd (opendir: Permission denied) 2024-03-18T15:20:22.723 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/0 9 root cwd unknown /proc/9/cwd (readlink: Permission denied) 2024-03-18T15:20:22.723 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/0 9 root rtd unknown /proc/9/root (readlink: Permission denied) 2024-03-18T15:20:22.723 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/0 9 root txt unknown /proc/9/exe (readlink: Permission denied) 2024-03-18T15:20:22.723 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/0 9 root NOFD /proc/9/fd (opendir: Permission denied) 2024-03-18T15:20:22.724 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/0 10 root cwd unknown /proc/10/cwd (readlink: Permission denied) 2024-03-18T15:20:22.724 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/0 10 root rtd unknown /proc/10/root (readlink: Permission denied) 2024-03-18T15:20:22.724 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/0 10 root txt unknown /proc/10/exe (readlink: Permission denied) 2024-03-18T15:20:22.724 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/0 10 root NOFD /proc/10/fd (opendir: Permission denied) 2024-03-18T15:20:22.724 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/u 11 root cwd unknown /proc/11/cwd (readlink: Permission denied) 2024-03-18T15:20:22.724 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/u 11 root rtd unknown /proc/11/root (readlink: Permission denied) 2024-03-18T15:20:22.724 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/u 11 root txt unknown /proc/11/exe (readlink: Permission denied) 2024-03-18T15:20:22.724 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/u 11 root NOFD /proc/11/fd (opendir: Permission denied) 2024-03-18T15:20:22.724 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 12 root cwd unknown /proc/12/cwd (readlink: Permission denied) 2024-03-18T15:20:22.724 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 12 root rtd unknown /proc/12/root (readlink: Permission denied) 2024-03-18T15:20:22.724 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 12 root txt unknown /proc/12/exe (readlink: Permission denied) 2024-03-18T15:20:22.724 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 12 root NOFD /proc/12/fd (opendir: Permission denied) 2024-03-18T15:20:22.724 INFO:tasks.workunit.client.0.smithi019.stdout:rcu_tasks 13 root cwd unknown /proc/13/cwd (readlink: Permission denied) 2024-03-18T15:20:22.724 INFO:tasks.workunit.client.0.smithi019.stdout:rcu_tasks 13 root rtd unknown /proc/13/root (readlink: Permission denied) 2024-03-18T15:20:22.725 INFO:tasks.workunit.client.0.smithi019.stdout:rcu_tasks 13 root txt unknown /proc/13/exe (readlink: Permission denied) 2024-03-18T15:20:22.725 INFO:tasks.workunit.client.0.smithi019.stdout:rcu_tasks 13 root NOFD /proc/13/fd (opendir: Permission denied) 2024-03-18T15:20:22.725 INFO:tasks.workunit.client.0.smithi019.stdout:rcu_tasks 14 root cwd unknown /proc/14/cwd (readlink: Permission denied) 2024-03-18T15:20:22.725 INFO:tasks.workunit.client.0.smithi019.stdout:rcu_tasks 14 root rtd unknown /proc/14/root (readlink: Permission denied) 2024-03-18T15:20:22.725 INFO:tasks.workunit.client.0.smithi019.stdout:rcu_tasks 14 root txt unknown /proc/14/exe (readlink: Permission denied) 2024-03-18T15:20:22.725 INFO:tasks.workunit.client.0.smithi019.stdout:rcu_tasks 14 root NOFD /proc/14/fd (opendir: Permission denied) 2024-03-18T15:20:22.725 INFO:tasks.workunit.client.0.smithi019.stdout:rcu_tasks 15 root cwd unknown /proc/15/cwd (readlink: Permission denied) 2024-03-18T15:20:22.725 INFO:tasks.workunit.client.0.smithi019.stdout:rcu_tasks 15 root rtd unknown /proc/15/root (readlink: Permission denied) 2024-03-18T15:20:22.725 INFO:tasks.workunit.client.0.smithi019.stdout:rcu_tasks 15 root txt unknown /proc/15/exe (readlink: Permission denied) 2024-03-18T15:20:22.725 INFO:tasks.workunit.client.0.smithi019.stdout:rcu_tasks 15 root NOFD /proc/15/fd (opendir: Permission denied) 2024-03-18T15:20:22.725 INFO:tasks.workunit.client.0.smithi019.stdout:ksoftirqd 16 root cwd unknown /proc/16/cwd (readlink: Permission denied) 2024-03-18T15:20:22.725 INFO:tasks.workunit.client.0.smithi019.stdout:ksoftirqd 16 root rtd unknown /proc/16/root (readlink: Permission denied) 2024-03-18T15:20:22.725 INFO:tasks.workunit.client.0.smithi019.stdout:ksoftirqd 16 root txt unknown /proc/16/exe (readlink: Permission denied) 2024-03-18T15:20:22.725 INFO:tasks.workunit.client.0.smithi019.stdout:ksoftirqd 16 root NOFD /proc/16/fd (opendir: Permission denied) 2024-03-18T15:20:22.725 INFO:tasks.workunit.client.0.smithi019.stdout:rcu_preem 17 root cwd unknown /proc/17/cwd (readlink: Permission denied) 2024-03-18T15:20:22.726 INFO:tasks.workunit.client.0.smithi019.stdout:rcu_preem 17 root rtd unknown /proc/17/root (readlink: Permission denied) 2024-03-18T15:20:22.726 INFO:tasks.workunit.client.0.smithi019.stdout:rcu_preem 17 root txt unknown /proc/17/exe (readlink: Permission denied) 2024-03-18T15:20:22.726 INFO:tasks.workunit.client.0.smithi019.stdout:rcu_preem 17 root NOFD /proc/17/fd (opendir: Permission denied) 2024-03-18T15:20:22.726 INFO:tasks.workunit.client.0.smithi019.stdout:migration 18 root cwd unknown /proc/18/cwd (readlink: Permission denied) 2024-03-18T15:20:22.726 INFO:tasks.workunit.client.0.smithi019.stdout:migration 18 root rtd unknown /proc/18/root (readlink: Permission denied) 2024-03-18T15:20:22.726 INFO:tasks.workunit.client.0.smithi019.stdout:migration 18 root txt unknown /proc/18/exe (readlink: Permission denied) 2024-03-18T15:20:22.726 INFO:tasks.workunit.client.0.smithi019.stdout:migration 18 root NOFD /proc/18/fd (opendir: Permission denied) 2024-03-18T15:20:22.726 INFO:tasks.workunit.client.0.smithi019.stdout:cpuhp/0 19 root cwd unknown /proc/19/cwd (readlink: Permission denied) 2024-03-18T15:20:22.726 INFO:tasks.workunit.client.0.smithi019.stdout:cpuhp/0 19 root rtd unknown /proc/19/root (readlink: Permission denied) 2024-03-18T15:20:22.726 INFO:tasks.workunit.client.0.smithi019.stdout:cpuhp/0 19 root txt unknown /proc/19/exe (readlink: Permission denied) 2024-03-18T15:20:22.726 INFO:tasks.workunit.client.0.smithi019.stdout:cpuhp/0 19 root NOFD /proc/19/fd (opendir: Permission denied) 2024-03-18T15:20:22.726 INFO:tasks.workunit.client.0.smithi019.stdout:cpuhp/1 20 root cwd unknown /proc/20/cwd (readlink: Permission denied) 2024-03-18T15:20:22.726 INFO:tasks.workunit.client.0.smithi019.stdout:cpuhp/1 20 root rtd unknown /proc/20/root (readlink: Permission denied) 2024-03-18T15:20:22.726 INFO:tasks.workunit.client.0.smithi019.stdout:cpuhp/1 20 root txt unknown /proc/20/exe (readlink: Permission denied) 2024-03-18T15:20:22.727 INFO:tasks.workunit.client.0.smithi019.stdout:cpuhp/1 20 root NOFD /proc/20/fd (opendir: Permission denied) 2024-03-18T15:20:22.727 INFO:tasks.workunit.client.0.smithi019.stdout:migration 21 root cwd unknown /proc/21/cwd (readlink: Permission denied) 2024-03-18T15:20:22.727 INFO:tasks.workunit.client.0.smithi019.stdout:migration 21 root rtd unknown /proc/21/root (readlink: Permission denied) 2024-03-18T15:20:22.727 INFO:tasks.workunit.client.0.smithi019.stdout:migration 21 root txt unknown /proc/21/exe (readlink: Permission denied) 2024-03-18T15:20:22.727 INFO:tasks.workunit.client.0.smithi019.stdout:migration 21 root NOFD /proc/21/fd (opendir: Permission denied) 2024-03-18T15:20:22.727 INFO:tasks.workunit.client.0.smithi019.stdout:ksoftirqd 22 root cwd unknown /proc/22/cwd (readlink: Permission denied) 2024-03-18T15:20:22.727 INFO:tasks.workunit.client.0.smithi019.stdout:ksoftirqd 22 root rtd unknown /proc/22/root (readlink: Permission denied) 2024-03-18T15:20:22.727 INFO:tasks.workunit.client.0.smithi019.stdout:ksoftirqd 22 root txt unknown /proc/22/exe (readlink: Permission denied) 2024-03-18T15:20:22.727 INFO:tasks.workunit.client.0.smithi019.stdout:ksoftirqd 22 root NOFD /proc/22/fd (opendir: Permission denied) 2024-03-18T15:20:22.727 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/1 24 root cwd unknown /proc/24/cwd (readlink: Permission denied) 2024-03-18T15:20:22.727 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/1 24 root rtd unknown /proc/24/root (readlink: Permission denied) 2024-03-18T15:20:22.727 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/1 24 root txt unknown /proc/24/exe (readlink: Permission denied) 2024-03-18T15:20:22.727 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/1 24 root NOFD /proc/24/fd (opendir: Permission denied) 2024-03-18T15:20:22.727 INFO:tasks.workunit.client.0.smithi019.stdout:cpuhp/2 25 root cwd unknown /proc/25/cwd (readlink: Permission denied) 2024-03-18T15:20:22.727 INFO:tasks.workunit.client.0.smithi019.stdout:cpuhp/2 25 root rtd unknown /proc/25/root (readlink: Permission denied) 2024-03-18T15:20:22.728 INFO:tasks.workunit.client.0.smithi019.stdout:cpuhp/2 25 root txt unknown /proc/25/exe (readlink: Permission denied) 2024-03-18T15:20:22.728 INFO:tasks.workunit.client.0.smithi019.stdout:cpuhp/2 25 root NOFD /proc/25/fd (opendir: Permission denied) 2024-03-18T15:20:22.728 INFO:tasks.workunit.client.0.smithi019.stdout:migration 26 root cwd unknown /proc/26/cwd (readlink: Permission denied) 2024-03-18T15:20:22.728 INFO:tasks.workunit.client.0.smithi019.stdout:migration 26 root rtd unknown /proc/26/root (readlink: Permission denied) 2024-03-18T15:20:22.728 INFO:tasks.workunit.client.0.smithi019.stdout:migration 26 root txt unknown /proc/26/exe (readlink: Permission denied) 2024-03-18T15:20:22.728 INFO:tasks.workunit.client.0.smithi019.stdout:migration 26 root NOFD /proc/26/fd (opendir: Permission denied) 2024-03-18T15:20:22.728 INFO:tasks.workunit.client.0.smithi019.stdout:ksoftirqd 27 root cwd unknown /proc/27/cwd (readlink: Permission denied) 2024-03-18T15:20:22.728 INFO:tasks.workunit.client.0.smithi019.stdout:ksoftirqd 27 root rtd unknown /proc/27/root (readlink: Permission denied) 2024-03-18T15:20:22.728 INFO:tasks.workunit.client.0.smithi019.stdout:ksoftirqd 27 root txt unknown /proc/27/exe (readlink: Permission denied) 2024-03-18T15:20:22.728 INFO:tasks.workunit.client.0.smithi019.stdout:ksoftirqd 27 root NOFD /proc/27/fd (opendir: Permission denied) 2024-03-18T15:20:22.728 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/2 29 root cwd unknown /proc/29/cwd (readlink: Permission denied) 2024-03-18T15:20:22.728 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/2 29 root rtd unknown /proc/29/root (readlink: Permission denied) 2024-03-18T15:20:22.728 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/2 29 root txt unknown /proc/29/exe (readlink: Permission denied) 2024-03-18T15:20:22.728 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/2 29 root NOFD /proc/29/fd (opendir: Permission denied) 2024-03-18T15:20:22.728 INFO:tasks.workunit.client.0.smithi019.stdout:cpuhp/3 30 root cwd unknown /proc/30/cwd (readlink: Permission denied) 2024-03-18T15:20:22.728 INFO:tasks.workunit.client.0.smithi019.stdout:cpuhp/3 30 root rtd unknown /proc/30/root (readlink: Permission denied) 2024-03-18T15:20:22.729 INFO:tasks.workunit.client.0.smithi019.stdout:cpuhp/3 30 root txt unknown /proc/30/exe (readlink: Permission denied) 2024-03-18T15:20:22.729 INFO:tasks.workunit.client.0.smithi019.stdout:cpuhp/3 30 root NOFD /proc/30/fd (opendir: Permission denied) 2024-03-18T15:20:22.729 INFO:tasks.workunit.client.0.smithi019.stdout:migration 31 root cwd unknown /proc/31/cwd (readlink: Permission denied) 2024-03-18T15:20:22.729 INFO:tasks.workunit.client.0.smithi019.stdout:migration 31 root rtd unknown /proc/31/root (readlink: Permission denied) 2024-03-18T15:20:22.729 INFO:tasks.workunit.client.0.smithi019.stdout:migration 31 root txt unknown /proc/31/exe (readlink: Permission denied) 2024-03-18T15:20:22.729 INFO:tasks.workunit.client.0.smithi019.stdout:migration 31 root NOFD /proc/31/fd (opendir: Permission denied) 2024-03-18T15:20:22.729 INFO:tasks.workunit.client.0.smithi019.stdout:ksoftirqd 32 root cwd unknown /proc/32/cwd (readlink: Permission denied) 2024-03-18T15:20:22.729 INFO:tasks.workunit.client.0.smithi019.stdout:ksoftirqd 32 root rtd unknown /proc/32/root (readlink: Permission denied) 2024-03-18T15:20:22.729 INFO:tasks.workunit.client.0.smithi019.stdout:ksoftirqd 32 root txt unknown /proc/32/exe (readlink: Permission denied) 2024-03-18T15:20:22.729 INFO:tasks.workunit.client.0.smithi019.stdout:ksoftirqd 32 root NOFD /proc/32/fd (opendir: Permission denied) 2024-03-18T15:20:22.729 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/3 34 root cwd unknown /proc/34/cwd (readlink: Permission denied) 2024-03-18T15:20:22.729 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/3 34 root rtd unknown /proc/34/root (readlink: Permission denied) 2024-03-18T15:20:22.729 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/3 34 root txt unknown /proc/34/exe (readlink: Permission denied) 2024-03-18T15:20:22.729 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/3 34 root NOFD /proc/34/fd (opendir: Permission denied) 2024-03-18T15:20:22.730 INFO:tasks.workunit.client.0.smithi019.stdout:cpuhp/4 35 root cwd unknown /proc/35/cwd (readlink: Permission denied) 2024-03-18T15:20:22.730 INFO:tasks.workunit.client.0.smithi019.stdout:cpuhp/4 35 root rtd unknown /proc/35/root (readlink: Permission denied) 2024-03-18T15:20:22.730 INFO:tasks.workunit.client.0.smithi019.stdout:cpuhp/4 35 root txt unknown /proc/35/exe (readlink: Permission denied) 2024-03-18T15:20:22.730 INFO:tasks.workunit.client.0.smithi019.stdout:cpuhp/4 35 root NOFD /proc/35/fd (opendir: Permission denied) 2024-03-18T15:20:22.730 INFO:tasks.workunit.client.0.smithi019.stdout:migration 36 root cwd unknown /proc/36/cwd (readlink: Permission denied) 2024-03-18T15:20:22.730 INFO:tasks.workunit.client.0.smithi019.stdout:migration 36 root rtd unknown /proc/36/root (readlink: Permission denied) 2024-03-18T15:20:22.730 INFO:tasks.workunit.client.0.smithi019.stdout:migration 36 root txt unknown /proc/36/exe (readlink: Permission denied) 2024-03-18T15:20:22.730 INFO:tasks.workunit.client.0.smithi019.stdout:migration 36 root NOFD /proc/36/fd (opendir: Permission denied) 2024-03-18T15:20:22.730 INFO:tasks.workunit.client.0.smithi019.stdout:ksoftirqd 37 root cwd unknown /proc/37/cwd (readlink: Permission denied) 2024-03-18T15:20:22.730 INFO:tasks.workunit.client.0.smithi019.stdout:ksoftirqd 37 root rtd unknown /proc/37/root (readlink: Permission denied) 2024-03-18T15:20:22.731 INFO:tasks.workunit.client.0.smithi019.stdout:ksoftirqd 37 root txt unknown /proc/37/exe (readlink: Permission denied) 2024-03-18T15:20:22.731 INFO:tasks.workunit.client.0.smithi019.stdout:ksoftirqd 37 root NOFD /proc/37/fd (opendir: Permission denied) 2024-03-18T15:20:22.731 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/4 39 root cwd unknown /proc/39/cwd (readlink: Permission denied) 2024-03-18T15:20:22.731 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/4 39 root rtd unknown /proc/39/root (readlink: Permission denied) 2024-03-18T15:20:22.731 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/4 39 root txt unknown /proc/39/exe (readlink: Permission denied) 2024-03-18T15:20:22.731 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/4 39 root NOFD /proc/39/fd (opendir: Permission denied) 2024-03-18T15:20:22.731 INFO:tasks.workunit.client.0.smithi019.stdout:cpuhp/5 40 root cwd unknown /proc/40/cwd (readlink: Permission denied) 2024-03-18T15:20:22.731 INFO:tasks.workunit.client.0.smithi019.stdout:cpuhp/5 40 root rtd unknown /proc/40/root (readlink: Permission denied) 2024-03-18T15:20:22.731 INFO:tasks.workunit.client.0.smithi019.stdout:cpuhp/5 40 root txt unknown /proc/40/exe (readlink: Permission denied) 2024-03-18T15:20:22.731 INFO:tasks.workunit.client.0.smithi019.stdout:cpuhp/5 40 root NOFD /proc/40/fd (opendir: Permission denied) 2024-03-18T15:20:22.731 INFO:tasks.workunit.client.0.smithi019.stdout:migration 41 root cwd unknown /proc/41/cwd (readlink: Permission denied) 2024-03-18T15:20:22.731 INFO:tasks.workunit.client.0.smithi019.stdout:migration 41 root rtd unknown /proc/41/root (readlink: Permission denied) 2024-03-18T15:20:22.731 INFO:tasks.workunit.client.0.smithi019.stdout:migration 41 root txt unknown /proc/41/exe (readlink: Permission denied) 2024-03-18T15:20:22.732 INFO:tasks.workunit.client.0.smithi019.stdout:migration 41 root NOFD /proc/41/fd (opendir: Permission denied) 2024-03-18T15:20:22.732 INFO:tasks.workunit.client.0.smithi019.stdout:ksoftirqd 42 root cwd unknown /proc/42/cwd (readlink: Permission denied) 2024-03-18T15:20:22.732 INFO:tasks.workunit.client.0.smithi019.stdout:ksoftirqd 42 root rtd unknown /proc/42/root (readlink: Permission denied) 2024-03-18T15:20:22.732 INFO:tasks.workunit.client.0.smithi019.stdout:ksoftirqd 42 root txt unknown /proc/42/exe (readlink: Permission denied) 2024-03-18T15:20:22.732 INFO:tasks.workunit.client.0.smithi019.stdout:ksoftirqd 42 root NOFD /proc/42/fd (opendir: Permission denied) 2024-03-18T15:20:22.732 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/5 44 root cwd unknown /proc/44/cwd (readlink: Permission denied) 2024-03-18T15:20:22.732 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/5 44 root rtd unknown /proc/44/root (readlink: Permission denied) 2024-03-18T15:20:22.732 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/5 44 root txt unknown /proc/44/exe (readlink: Permission denied) 2024-03-18T15:20:22.732 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/5 44 root NOFD /proc/44/fd (opendir: Permission denied) 2024-03-18T15:20:22.732 INFO:tasks.workunit.client.0.smithi019.stdout:cpuhp/6 45 root cwd unknown /proc/45/cwd (readlink: Permission denied) 2024-03-18T15:20:22.732 INFO:tasks.workunit.client.0.smithi019.stdout:cpuhp/6 45 root rtd unknown /proc/45/root (readlink: Permission denied) 2024-03-18T15:20:22.732 INFO:tasks.workunit.client.0.smithi019.stdout:cpuhp/6 45 root txt unknown /proc/45/exe (readlink: Permission denied) 2024-03-18T15:20:22.732 INFO:tasks.workunit.client.0.smithi019.stdout:cpuhp/6 45 root NOFD /proc/45/fd (opendir: Permission denied) 2024-03-18T15:20:22.732 INFO:tasks.workunit.client.0.smithi019.stdout:migration 46 root cwd unknown /proc/46/cwd (readlink: Permission denied) 2024-03-18T15:20:22.732 INFO:tasks.workunit.client.0.smithi019.stdout:migration 46 root rtd unknown /proc/46/root (readlink: Permission denied) 2024-03-18T15:20:22.733 INFO:tasks.workunit.client.0.smithi019.stdout:migration 46 root txt unknown /proc/46/exe (readlink: Permission denied) 2024-03-18T15:20:22.733 INFO:tasks.workunit.client.0.smithi019.stdout:migration 46 root NOFD /proc/46/fd (opendir: Permission denied) 2024-03-18T15:20:22.733 INFO:tasks.workunit.client.0.smithi019.stdout:ksoftirqd 47 root cwd unknown /proc/47/cwd (readlink: Permission denied) 2024-03-18T15:20:22.733 INFO:tasks.workunit.client.0.smithi019.stdout:ksoftirqd 47 root rtd unknown /proc/47/root (readlink: Permission denied) 2024-03-18T15:20:22.733 INFO:tasks.workunit.client.0.smithi019.stdout:ksoftirqd 47 root txt unknown /proc/47/exe (readlink: Permission denied) 2024-03-18T15:20:22.733 INFO:tasks.workunit.client.0.smithi019.stdout:ksoftirqd 47 root NOFD /proc/47/fd (opendir: Permission denied) 2024-03-18T15:20:22.733 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/6 49 root cwd unknown /proc/49/cwd (readlink: Permission denied) 2024-03-18T15:20:22.733 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/6 49 root rtd unknown /proc/49/root (readlink: Permission denied) 2024-03-18T15:20:22.733 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/6 49 root txt unknown /proc/49/exe (readlink: Permission denied) 2024-03-18T15:20:22.733 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/6 49 root NOFD /proc/49/fd (opendir: Permission denied) 2024-03-18T15:20:22.733 INFO:tasks.workunit.client.0.smithi019.stdout:cpuhp/7 50 root cwd unknown /proc/50/cwd (readlink: Permission denied) 2024-03-18T15:20:22.733 INFO:tasks.workunit.client.0.smithi019.stdout:cpuhp/7 50 root rtd unknown /proc/50/root (readlink: Permission denied) 2024-03-18T15:20:22.733 INFO:tasks.workunit.client.0.smithi019.stdout:cpuhp/7 50 root txt unknown /proc/50/exe (readlink: Permission denied) 2024-03-18T15:20:22.734 INFO:tasks.workunit.client.0.smithi019.stdout:cpuhp/7 50 root NOFD /proc/50/fd (opendir: Permission denied) 2024-03-18T15:20:22.734 INFO:tasks.workunit.client.0.smithi019.stdout:migration 51 root cwd unknown /proc/51/cwd (readlink: Permission denied) 2024-03-18T15:20:22.734 INFO:tasks.workunit.client.0.smithi019.stdout:migration 51 root rtd unknown /proc/51/root (readlink: Permission denied) 2024-03-18T15:20:22.734 INFO:tasks.workunit.client.0.smithi019.stdout:migration 51 root txt unknown /proc/51/exe (readlink: Permission denied) 2024-03-18T15:20:22.734 INFO:tasks.workunit.client.0.smithi019.stdout:migration 51 root NOFD /proc/51/fd (opendir: Permission denied) 2024-03-18T15:20:22.734 INFO:tasks.workunit.client.0.smithi019.stdout:ksoftirqd 52 root cwd unknown /proc/52/cwd (readlink: Permission denied) 2024-03-18T15:20:22.734 INFO:tasks.workunit.client.0.smithi019.stdout:ksoftirqd 52 root rtd unknown /proc/52/root (readlink: Permission denied) 2024-03-18T15:20:22.734 INFO:tasks.workunit.client.0.smithi019.stdout:ksoftirqd 52 root txt unknown /proc/52/exe (readlink: Permission denied) 2024-03-18T15:20:22.734 INFO:tasks.workunit.client.0.smithi019.stdout:ksoftirqd 52 root NOFD /proc/52/fd (opendir: Permission denied) 2024-03-18T15:20:22.734 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/7 54 root cwd unknown /proc/54/cwd (readlink: Permission denied) 2024-03-18T15:20:22.734 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/7 54 root rtd unknown /proc/54/root (readlink: Permission denied) 2024-03-18T15:20:22.734 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/7 54 root txt unknown /proc/54/exe (readlink: Permission denied) 2024-03-18T15:20:22.734 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/7 54 root NOFD /proc/54/fd (opendir: Permission denied) 2024-03-18T15:20:22.734 INFO:tasks.workunit.client.0.smithi019.stdout:kdevtmpfs 55 root cwd unknown /proc/55/cwd (readlink: Permission denied) 2024-03-18T15:20:22.735 INFO:tasks.workunit.client.0.smithi019.stdout:kdevtmpfs 55 root rtd unknown /proc/55/root (readlink: Permission denied) 2024-03-18T15:20:22.735 INFO:tasks.workunit.client.0.smithi019.stdout:kdevtmpfs 55 root txt unknown /proc/55/exe (readlink: Permission denied) 2024-03-18T15:20:22.735 INFO:tasks.workunit.client.0.smithi019.stdout:kdevtmpfs 55 root NOFD /proc/55/fd (opendir: Permission denied) 2024-03-18T15:20:22.735 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 56 root cwd unknown /proc/56/cwd (readlink: Permission denied) 2024-03-18T15:20:22.735 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 56 root rtd unknown /proc/56/root (readlink: Permission denied) 2024-03-18T15:20:22.735 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 56 root txt unknown /proc/56/exe (readlink: Permission denied) 2024-03-18T15:20:22.735 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 56 root NOFD /proc/56/fd (opendir: Permission denied) 2024-03-18T15:20:22.735 INFO:tasks.workunit.client.0.smithi019.stdout:kauditd 57 root cwd unknown /proc/57/cwd (readlink: Permission denied) 2024-03-18T15:20:22.735 INFO:tasks.workunit.client.0.smithi019.stdout:kauditd 57 root rtd unknown /proc/57/root (readlink: Permission denied) 2024-03-18T15:20:22.735 INFO:tasks.workunit.client.0.smithi019.stdout:kauditd 57 root txt unknown /proc/57/exe (readlink: Permission denied) 2024-03-18T15:20:22.735 INFO:tasks.workunit.client.0.smithi019.stdout:kauditd 57 root NOFD /proc/57/fd (opendir: Permission denied) 2024-03-18T15:20:22.735 INFO:tasks.workunit.client.0.smithi019.stdout:khungtask 58 root cwd unknown /proc/58/cwd (readlink: Permission denied) 2024-03-18T15:20:22.735 INFO:tasks.workunit.client.0.smithi019.stdout:khungtask 58 root rtd unknown /proc/58/root (readlink: Permission denied) 2024-03-18T15:20:22.735 INFO:tasks.workunit.client.0.smithi019.stdout:khungtask 58 root txt unknown /proc/58/exe (readlink: Permission denied) 2024-03-18T15:20:22.736 INFO:tasks.workunit.client.0.smithi019.stdout:khungtask 58 root NOFD /proc/58/fd (opendir: Permission denied) 2024-03-18T15:20:22.736 INFO:tasks.workunit.client.0.smithi019.stdout:oom_reape 60 root cwd unknown /proc/60/cwd (readlink: Permission denied) 2024-03-18T15:20:22.736 INFO:tasks.workunit.client.0.smithi019.stdout:oom_reape 60 root rtd unknown /proc/60/root (readlink: Permission denied) 2024-03-18T15:20:22.736 INFO:tasks.workunit.client.0.smithi019.stdout:oom_reape 60 root txt unknown /proc/60/exe (readlink: Permission denied) 2024-03-18T15:20:22.736 INFO:tasks.workunit.client.0.smithi019.stdout:oom_reape 60 root NOFD /proc/60/fd (opendir: Permission denied) 2024-03-18T15:20:22.736 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 62 root cwd unknown /proc/62/cwd (readlink: Permission denied) 2024-03-18T15:20:22.736 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 62 root rtd unknown /proc/62/root (readlink: Permission denied) 2024-03-18T15:20:22.736 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 62 root txt unknown /proc/62/exe (readlink: Permission denied) 2024-03-18T15:20:22.736 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 62 root NOFD /proc/62/fd (opendir: Permission denied) 2024-03-18T15:20:22.736 INFO:tasks.workunit.client.0.smithi019.stdout:kcompactd 63 root cwd unknown /proc/63/cwd (readlink: Permission denied) 2024-03-18T15:20:22.736 INFO:tasks.workunit.client.0.smithi019.stdout:kcompactd 63 root rtd unknown /proc/63/root (readlink: Permission denied) 2024-03-18T15:20:22.736 INFO:tasks.workunit.client.0.smithi019.stdout:kcompactd 63 root txt unknown /proc/63/exe (readlink: Permission denied) 2024-03-18T15:20:22.736 INFO:tasks.workunit.client.0.smithi019.stdout:kcompactd 63 root NOFD /proc/63/fd (opendir: Permission denied) 2024-03-18T15:20:22.736 INFO:tasks.workunit.client.0.smithi019.stdout:ksmd 64 root cwd unknown /proc/64/cwd (readlink: Permission denied) 2024-03-18T15:20:22.736 INFO:tasks.workunit.client.0.smithi019.stdout:ksmd 64 root rtd unknown /proc/64/root (readlink: Permission denied) 2024-03-18T15:20:22.737 INFO:tasks.workunit.client.0.smithi019.stdout:ksmd 64 root txt unknown /proc/64/exe (readlink: Permission denied) 2024-03-18T15:20:22.737 INFO:tasks.workunit.client.0.smithi019.stdout:ksmd 64 root NOFD /proc/64/fd (opendir: Permission denied) 2024-03-18T15:20:22.737 INFO:tasks.workunit.client.0.smithi019.stdout:khugepage 65 root cwd unknown /proc/65/cwd (readlink: Permission denied) 2024-03-18T15:20:22.737 INFO:tasks.workunit.client.0.smithi019.stdout:khugepage 65 root rtd unknown /proc/65/root (readlink: Permission denied) 2024-03-18T15:20:22.737 INFO:tasks.workunit.client.0.smithi019.stdout:khugepage 65 root txt unknown /proc/65/exe (readlink: Permission denied) 2024-03-18T15:20:22.737 INFO:tasks.workunit.client.0.smithi019.stdout:khugepage 65 root NOFD /proc/65/fd (opendir: Permission denied) 2024-03-18T15:20:22.737 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 66 root cwd unknown /proc/66/cwd (readlink: Permission denied) 2024-03-18T15:20:22.737 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 66 root rtd unknown /proc/66/root (readlink: Permission denied) 2024-03-18T15:20:22.737 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 66 root txt unknown /proc/66/exe (readlink: Permission denied) 2024-03-18T15:20:22.737 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 66 root NOFD /proc/66/fd (opendir: Permission denied) 2024-03-18T15:20:22.737 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 67 root cwd unknown /proc/67/cwd (readlink: Permission denied) 2024-03-18T15:20:22.737 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 67 root rtd unknown /proc/67/root (readlink: Permission denied) 2024-03-18T15:20:22.737 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 67 root txt unknown /proc/67/exe (readlink: Permission denied) 2024-03-18T15:20:22.737 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 67 root NOFD /proc/67/fd (opendir: Permission denied) 2024-03-18T15:20:22.738 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 68 root cwd unknown /proc/68/cwd (readlink: Permission denied) 2024-03-18T15:20:22.738 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 68 root rtd unknown /proc/68/root (readlink: Permission denied) 2024-03-18T15:20:22.738 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 68 root txt unknown /proc/68/exe (readlink: Permission denied) 2024-03-18T15:20:22.738 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 68 root NOFD /proc/68/fd (opendir: Permission denied) 2024-03-18T15:20:22.738 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 69 root cwd unknown /proc/69/cwd (readlink: Permission denied) 2024-03-18T15:20:22.738 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 69 root rtd unknown /proc/69/root (readlink: Permission denied) 2024-03-18T15:20:22.738 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 69 root txt unknown /proc/69/exe (readlink: Permission denied) 2024-03-18T15:20:22.738 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 69 root NOFD /proc/69/fd (opendir: Permission denied) 2024-03-18T15:20:22.738 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/6 75 root cwd unknown /proc/75/cwd (readlink: Permission denied) 2024-03-18T15:20:22.738 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/6 75 root rtd unknown /proc/75/root (readlink: Permission denied) 2024-03-18T15:20:22.738 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/6 75 root txt unknown /proc/75/exe (readlink: Permission denied) 2024-03-18T15:20:22.738 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/6 75 root NOFD /proc/75/fd (opendir: Permission denied) 2024-03-18T15:20:22.738 INFO:tasks.workunit.client.0.smithi019.stdout:irq/9-acp 77 root cwd unknown /proc/77/cwd (readlink: Permission denied) 2024-03-18T15:20:22.739 INFO:tasks.workunit.client.0.smithi019.stdout:irq/9-acp 77 root rtd unknown /proc/77/root (readlink: Permission denied) 2024-03-18T15:20:22.739 INFO:tasks.workunit.client.0.smithi019.stdout:irq/9-acp 77 root txt unknown /proc/77/exe (readlink: Permission denied) 2024-03-18T15:20:22.739 INFO:tasks.workunit.client.0.smithi019.stdout:irq/9-acp 77 root NOFD /proc/77/fd (opendir: Permission denied) 2024-03-18T15:20:22.739 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 78 root cwd unknown /proc/78/cwd (readlink: Permission denied) 2024-03-18T15:20:22.739 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 78 root rtd unknown /proc/78/root (readlink: Permission denied) 2024-03-18T15:20:22.739 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 78 root txt unknown /proc/78/exe (readlink: Permission denied) 2024-03-18T15:20:22.739 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 78 root NOFD /proc/78/fd (opendir: Permission denied) 2024-03-18T15:20:22.739 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 79 root cwd unknown /proc/79/cwd (readlink: Permission denied) 2024-03-18T15:20:22.739 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 79 root rtd unknown /proc/79/root (readlink: Permission denied) 2024-03-18T15:20:22.739 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 79 root txt unknown /proc/79/exe (readlink: Permission denied) 2024-03-18T15:20:22.739 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 79 root NOFD /proc/79/fd (opendir: Permission denied) 2024-03-18T15:20:22.739 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 80 root cwd unknown /proc/80/cwd (readlink: Permission denied) 2024-03-18T15:20:22.739 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 80 root rtd unknown /proc/80/root (readlink: Permission denied) 2024-03-18T15:20:22.739 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 80 root txt unknown /proc/80/exe (readlink: Permission denied) 2024-03-18T15:20:22.740 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 80 root NOFD /proc/80/fd (opendir: Permission denied) 2024-03-18T15:20:22.740 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 81 root cwd unknown /proc/81/cwd (readlink: Permission denied) 2024-03-18T15:20:22.740 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 81 root rtd unknown /proc/81/root (readlink: Permission denied) 2024-03-18T15:20:22.740 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 81 root txt unknown /proc/81/exe (readlink: Permission denied) 2024-03-18T15:20:22.740 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 81 root NOFD /proc/81/fd (opendir: Permission denied) 2024-03-18T15:20:22.740 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 82 root cwd unknown /proc/82/cwd (readlink: Permission denied) 2024-03-18T15:20:22.740 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 82 root rtd unknown /proc/82/root (readlink: Permission denied) 2024-03-18T15:20:22.740 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 82 root txt unknown /proc/82/exe (readlink: Permission denied) 2024-03-18T15:20:22.740 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 82 root NOFD /proc/82/fd (opendir: Permission denied) 2024-03-18T15:20:22.740 INFO:tasks.workunit.client.0.smithi019.stdout:watchdogd 83 root cwd unknown /proc/83/cwd (readlink: Permission denied) 2024-03-18T15:20:22.740 INFO:tasks.workunit.client.0.smithi019.stdout:watchdogd 83 root rtd unknown /proc/83/root (readlink: Permission denied) 2024-03-18T15:20:22.740 INFO:tasks.workunit.client.0.smithi019.stdout:watchdogd 83 root txt unknown /proc/83/exe (readlink: Permission denied) 2024-03-18T15:20:22.740 INFO:tasks.workunit.client.0.smithi019.stdout:watchdogd 83 root NOFD /proc/83/fd (opendir: Permission denied) 2024-03-18T15:20:22.740 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/0 84 root cwd unknown /proc/84/cwd (readlink: Permission denied) 2024-03-18T15:20:22.740 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/0 84 root rtd unknown /proc/84/root (readlink: Permission denied) 2024-03-18T15:20:22.740 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/0 84 root txt unknown /proc/84/exe (readlink: Permission denied) 2024-03-18T15:20:22.741 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/0 84 root NOFD /proc/84/fd (opendir: Permission denied) 2024-03-18T15:20:22.741 INFO:tasks.workunit.client.0.smithi019.stdout:kswapd0 85 root cwd unknown /proc/85/cwd (readlink: Permission denied) 2024-03-18T15:20:22.741 INFO:tasks.workunit.client.0.smithi019.stdout:kswapd0 85 root rtd unknown /proc/85/root (readlink: Permission denied) 2024-03-18T15:20:22.741 INFO:tasks.workunit.client.0.smithi019.stdout:kswapd0 85 root txt unknown /proc/85/exe (readlink: Permission denied) 2024-03-18T15:20:22.741 INFO:tasks.workunit.client.0.smithi019.stdout:kswapd0 85 root NOFD /proc/85/fd (opendir: Permission denied) 2024-03-18T15:20:22.741 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 87 root cwd unknown /proc/87/cwd (readlink: Permission denied) 2024-03-18T15:20:22.741 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 87 root rtd unknown /proc/87/root (readlink: Permission denied) 2024-03-18T15:20:22.741 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 87 root txt unknown /proc/87/exe (readlink: Permission denied) 2024-03-18T15:20:22.741 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 87 root NOFD /proc/87/fd (opendir: Permission denied) 2024-03-18T15:20:22.741 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 92 root cwd unknown /proc/92/cwd (readlink: Permission denied) 2024-03-18T15:20:22.741 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 92 root rtd unknown /proc/92/root (readlink: Permission denied) 2024-03-18T15:20:22.741 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 92 root txt unknown /proc/92/exe (readlink: Permission denied) 2024-03-18T15:20:22.741 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 92 root NOFD /proc/92/fd (opendir: Permission denied) 2024-03-18T15:20:22.741 INFO:tasks.workunit.client.0.smithi019.stdout:scsi_eh_0 94 root cwd unknown /proc/94/cwd (readlink: Permission denied) 2024-03-18T15:20:22.741 INFO:tasks.workunit.client.0.smithi019.stdout:scsi_eh_0 94 root rtd unknown /proc/94/root (readlink: Permission denied) 2024-03-18T15:20:22.742 INFO:tasks.workunit.client.0.smithi019.stdout:scsi_eh_0 94 root txt unknown /proc/94/exe (readlink: Permission denied) 2024-03-18T15:20:22.742 INFO:tasks.workunit.client.0.smithi019.stdout:scsi_eh_0 94 root NOFD /proc/94/fd (opendir: Permission denied) 2024-03-18T15:20:22.742 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 95 root cwd unknown /proc/95/cwd (readlink: Permission denied) 2024-03-18T15:20:22.742 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 95 root rtd unknown /proc/95/root (readlink: Permission denied) 2024-03-18T15:20:22.742 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 95 root txt unknown /proc/95/exe (readlink: Permission denied) 2024-03-18T15:20:22.742 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 95 root NOFD /proc/95/fd (opendir: Permission denied) 2024-03-18T15:20:22.742 INFO:tasks.workunit.client.0.smithi019.stdout:scsi_eh_1 96 root cwd unknown /proc/96/cwd (readlink: Permission denied) 2024-03-18T15:20:22.742 INFO:tasks.workunit.client.0.smithi019.stdout:scsi_eh_1 96 root rtd unknown /proc/96/root (readlink: Permission denied) 2024-03-18T15:20:22.742 INFO:tasks.workunit.client.0.smithi019.stdout:scsi_eh_1 96 root txt unknown /proc/96/exe (readlink: Permission denied) 2024-03-18T15:20:22.742 INFO:tasks.workunit.client.0.smithi019.stdout:scsi_eh_1 96 root NOFD /proc/96/fd (opendir: Permission denied) 2024-03-18T15:20:22.742 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 97 root cwd unknown /proc/97/cwd (readlink: Permission denied) 2024-03-18T15:20:22.742 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 97 root rtd unknown /proc/97/root (readlink: Permission denied) 2024-03-18T15:20:22.742 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 97 root txt unknown /proc/97/exe (readlink: Permission denied) 2024-03-18T15:20:22.742 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 97 root NOFD /proc/97/fd (opendir: Permission denied) 2024-03-18T15:20:22.743 INFO:tasks.workunit.client.0.smithi019.stdout:scsi_eh_2 98 root cwd unknown /proc/98/cwd (readlink: Permission denied) 2024-03-18T15:20:22.743 INFO:tasks.workunit.client.0.smithi019.stdout:scsi_eh_2 98 root rtd unknown /proc/98/root (readlink: Permission denied) 2024-03-18T15:20:22.743 INFO:tasks.workunit.client.0.smithi019.stdout:scsi_eh_2 98 root txt unknown /proc/98/exe (readlink: Permission denied) 2024-03-18T15:20:22.743 INFO:tasks.workunit.client.0.smithi019.stdout:scsi_eh_2 98 root NOFD /proc/98/fd (opendir: Permission denied) 2024-03-18T15:20:22.743 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 99 root cwd unknown /proc/99/cwd (readlink: Permission denied) 2024-03-18T15:20:22.743 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 99 root rtd unknown /proc/99/root (readlink: Permission denied) 2024-03-18T15:20:22.743 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 99 root txt unknown /proc/99/exe (readlink: Permission denied) 2024-03-18T15:20:22.743 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 99 root NOFD /proc/99/fd (opendir: Permission denied) 2024-03-18T15:20:22.743 INFO:tasks.workunit.client.0.smithi019.stdout:scsi_eh_3 100 root cwd unknown /proc/100/cwd (readlink: Permission denied) 2024-03-18T15:20:22.743 INFO:tasks.workunit.client.0.smithi019.stdout:scsi_eh_3 100 root rtd unknown /proc/100/root (readlink: Permission denied) 2024-03-18T15:20:22.743 INFO:tasks.workunit.client.0.smithi019.stdout:scsi_eh_3 100 root txt unknown /proc/100/exe (readlink: Permission denied) 2024-03-18T15:20:22.743 INFO:tasks.workunit.client.0.smithi019.stdout:scsi_eh_3 100 root NOFD /proc/100/fd (opendir: Permission denied) 2024-03-18T15:20:22.743 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 101 root cwd unknown /proc/101/cwd (readlink: Permission denied) 2024-03-18T15:20:22.743 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 101 root rtd unknown /proc/101/root (readlink: Permission denied) 2024-03-18T15:20:22.743 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 101 root txt unknown /proc/101/exe (readlink: Permission denied) 2024-03-18T15:20:22.743 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 101 root NOFD /proc/101/fd (opendir: Permission denied) 2024-03-18T15:20:22.744 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/u 103 root cwd unknown /proc/103/cwd (readlink: Permission denied) 2024-03-18T15:20:22.744 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/u 103 root rtd unknown /proc/103/root (readlink: Permission denied) 2024-03-18T15:20:22.744 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/u 103 root txt unknown /proc/103/exe (readlink: Permission denied) 2024-03-18T15:20:22.744 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/u 103 root NOFD /proc/103/fd (opendir: Permission denied) 2024-03-18T15:20:22.744 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/u 104 root cwd unknown /proc/104/cwd (readlink: Permission denied) 2024-03-18T15:20:22.744 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/u 104 root rtd unknown /proc/104/root (readlink: Permission denied) 2024-03-18T15:20:22.744 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/u 104 root txt unknown /proc/104/exe (readlink: Permission denied) 2024-03-18T15:20:22.744 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/u 104 root NOFD /proc/104/fd (opendir: Permission denied) 2024-03-18T15:20:22.744 INFO:tasks.workunit.client.0.smithi019.stdout:scsi_eh_4 105 root cwd unknown /proc/105/cwd (readlink: Permission denied) 2024-03-18T15:20:22.744 INFO:tasks.workunit.client.0.smithi019.stdout:scsi_eh_4 105 root rtd unknown /proc/105/root (readlink: Permission denied) 2024-03-18T15:20:22.744 INFO:tasks.workunit.client.0.smithi019.stdout:scsi_eh_4 105 root txt unknown /proc/105/exe (readlink: Permission denied) 2024-03-18T15:20:22.744 INFO:tasks.workunit.client.0.smithi019.stdout:scsi_eh_4 105 root NOFD /proc/105/fd (opendir: Permission denied) 2024-03-18T15:20:22.744 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 106 root cwd unknown /proc/106/cwd (readlink: Permission denied) 2024-03-18T15:20:22.744 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 106 root rtd unknown /proc/106/root (readlink: Permission denied) 2024-03-18T15:20:22.745 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 106 root txt unknown /proc/106/exe (readlink: Permission denied) 2024-03-18T15:20:22.745 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 106 root NOFD /proc/106/fd (opendir: Permission denied) 2024-03-18T15:20:22.745 INFO:tasks.workunit.client.0.smithi019.stdout:scsi_eh_5 107 root cwd unknown /proc/107/cwd (readlink: Permission denied) 2024-03-18T15:20:22.745 INFO:tasks.workunit.client.0.smithi019.stdout:scsi_eh_5 107 root rtd unknown /proc/107/root (readlink: Permission denied) 2024-03-18T15:20:22.745 INFO:tasks.workunit.client.0.smithi019.stdout:scsi_eh_5 107 root txt unknown /proc/107/exe (readlink: Permission denied) 2024-03-18T15:20:22.745 INFO:tasks.workunit.client.0.smithi019.stdout:scsi_eh_5 107 root NOFD /proc/107/fd (opendir: Permission denied) 2024-03-18T15:20:22.745 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 108 root cwd unknown /proc/108/cwd (readlink: Permission denied) 2024-03-18T15:20:22.745 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 108 root rtd unknown /proc/108/root (readlink: Permission denied) 2024-03-18T15:20:22.745 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 108 root txt unknown /proc/108/exe (readlink: Permission denied) 2024-03-18T15:20:22.745 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 108 root NOFD /proc/108/fd (opendir: Permission denied) 2024-03-18T15:20:22.745 INFO:tasks.workunit.client.0.smithi019.stdout:scsi_eh_6 109 root cwd unknown /proc/109/cwd (readlink: Permission denied) 2024-03-18T15:20:22.745 INFO:tasks.workunit.client.0.smithi019.stdout:scsi_eh_6 109 root rtd unknown /proc/109/root (readlink: Permission denied) 2024-03-18T15:20:22.745 INFO:tasks.workunit.client.0.smithi019.stdout:scsi_eh_6 109 root txt unknown /proc/109/exe (readlink: Permission denied) 2024-03-18T15:20:22.746 INFO:tasks.workunit.client.0.smithi019.stdout:scsi_eh_6 109 root NOFD /proc/109/fd (opendir: Permission denied) 2024-03-18T15:20:22.746 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 110 root cwd unknown /proc/110/cwd (readlink: Permission denied) 2024-03-18T15:20:22.746 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 110 root rtd unknown /proc/110/root (readlink: Permission denied) 2024-03-18T15:20:22.746 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 110 root txt unknown /proc/110/exe (readlink: Permission denied) 2024-03-18T15:20:22.746 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 110 root NOFD /proc/110/fd (opendir: Permission denied) 2024-03-18T15:20:22.746 INFO:tasks.workunit.client.0.smithi019.stdout:scsi_eh_7 111 root cwd unknown /proc/111/cwd (readlink: Permission denied) 2024-03-18T15:20:22.746 INFO:tasks.workunit.client.0.smithi019.stdout:scsi_eh_7 111 root rtd unknown /proc/111/root (readlink: Permission denied) 2024-03-18T15:20:22.746 INFO:tasks.workunit.client.0.smithi019.stdout:scsi_eh_7 111 root txt unknown /proc/111/exe (readlink: Permission denied) 2024-03-18T15:20:22.746 INFO:tasks.workunit.client.0.smithi019.stdout:scsi_eh_7 111 root NOFD /proc/111/fd (opendir: Permission denied) 2024-03-18T15:20:22.746 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 112 root cwd unknown /proc/112/cwd (readlink: Permission denied) 2024-03-18T15:20:22.746 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 112 root rtd unknown /proc/112/root (readlink: Permission denied) 2024-03-18T15:20:22.746 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 112 root txt unknown /proc/112/exe (readlink: Permission denied) 2024-03-18T15:20:22.746 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 112 root NOFD /proc/112/fd (opendir: Permission denied) 2024-03-18T15:20:22.746 INFO:tasks.workunit.client.0.smithi019.stdout:scsi_eh_8 113 root cwd unknown /proc/113/cwd (readlink: Permission denied) 2024-03-18T15:20:22.746 INFO:tasks.workunit.client.0.smithi019.stdout:scsi_eh_8 113 root rtd unknown /proc/113/root (readlink: Permission denied) 2024-03-18T15:20:22.747 INFO:tasks.workunit.client.0.smithi019.stdout:scsi_eh_8 113 root txt unknown /proc/113/exe (readlink: Permission denied) 2024-03-18T15:20:22.747 INFO:tasks.workunit.client.0.smithi019.stdout:scsi_eh_8 113 root NOFD /proc/113/fd (opendir: Permission denied) 2024-03-18T15:20:22.747 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 114 root cwd unknown /proc/114/cwd (readlink: Permission denied) 2024-03-18T15:20:22.747 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 114 root rtd unknown /proc/114/root (readlink: Permission denied) 2024-03-18T15:20:22.747 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 114 root txt unknown /proc/114/exe (readlink: Permission denied) 2024-03-18T15:20:22.747 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 114 root NOFD /proc/114/fd (opendir: Permission denied) 2024-03-18T15:20:22.747 INFO:tasks.workunit.client.0.smithi019.stdout:scsi_eh_9 115 root cwd unknown /proc/115/cwd (readlink: Permission denied) 2024-03-18T15:20:22.747 INFO:tasks.workunit.client.0.smithi019.stdout:scsi_eh_9 115 root rtd unknown /proc/115/root (readlink: Permission denied) 2024-03-18T15:20:22.747 INFO:tasks.workunit.client.0.smithi019.stdout:scsi_eh_9 115 root txt unknown /proc/115/exe (readlink: Permission denied) 2024-03-18T15:20:22.747 INFO:tasks.workunit.client.0.smithi019.stdout:scsi_eh_9 115 root NOFD /proc/115/fd (opendir: Permission denied) 2024-03-18T15:20:22.747 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 116 root cwd unknown /proc/116/cwd (readlink: Permission denied) 2024-03-18T15:20:22.747 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 116 root rtd unknown /proc/116/root (readlink: Permission denied) 2024-03-18T15:20:22.747 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 116 root txt unknown /proc/116/exe (readlink: Permission denied) 2024-03-18T15:20:22.747 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 116 root NOFD /proc/116/fd (opendir: Permission denied) 2024-03-18T15:20:22.747 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/u 118 root cwd unknown /proc/118/cwd (readlink: Permission denied) 2024-03-18T15:20:22.748 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/u 118 root rtd unknown /proc/118/root (readlink: Permission denied) 2024-03-18T15:20:22.748 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/u 118 root txt unknown /proc/118/exe (readlink: Permission denied) 2024-03-18T15:20:22.748 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/u 118 root NOFD /proc/118/fd (opendir: Permission denied) 2024-03-18T15:20:22.748 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/u 122 root cwd unknown /proc/122/cwd (readlink: Permission denied) 2024-03-18T15:20:22.748 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/u 122 root rtd unknown /proc/122/root (readlink: Permission denied) 2024-03-18T15:20:22.748 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/u 122 root txt unknown /proc/122/exe (readlink: Permission denied) 2024-03-18T15:20:22.748 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/u 122 root NOFD /proc/122/fd (opendir: Permission denied) 2024-03-18T15:20:22.748 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 125 root cwd unknown /proc/125/cwd (readlink: Permission denied) 2024-03-18T15:20:22.748 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 125 root rtd unknown /proc/125/root (readlink: Permission denied) 2024-03-18T15:20:22.748 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 125 root txt unknown /proc/125/exe (readlink: Permission denied) 2024-03-18T15:20:22.748 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 125 root NOFD /proc/125/fd (opendir: Permission denied) 2024-03-18T15:20:22.748 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 127 root cwd unknown /proc/127/cwd (readlink: Permission denied) 2024-03-18T15:20:22.748 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 127 root rtd unknown /proc/127/root (readlink: Permission denied) 2024-03-18T15:20:22.748 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 127 root txt unknown /proc/127/exe (readlink: Permission denied) 2024-03-18T15:20:22.748 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 127 root NOFD /proc/127/fd (opendir: Permission denied) 2024-03-18T15:20:22.748 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/7 128 root cwd unknown /proc/128/cwd (readlink: Permission denied) 2024-03-18T15:20:22.749 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/7 128 root rtd unknown /proc/128/root (readlink: Permission denied) 2024-03-18T15:20:22.749 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/7 128 root txt unknown /proc/128/exe (readlink: Permission denied) 2024-03-18T15:20:22.749 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/7 128 root NOFD /proc/128/fd (opendir: Permission denied) 2024-03-18T15:20:22.749 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 129 root cwd unknown /proc/129/cwd (readlink: Permission denied) 2024-03-18T15:20:22.749 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 129 root rtd unknown /proc/129/root (readlink: Permission denied) 2024-03-18T15:20:22.749 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 129 root txt unknown /proc/129/exe (readlink: Permission denied) 2024-03-18T15:20:22.749 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 129 root NOFD /proc/129/fd (opendir: Permission denied) 2024-03-18T15:20:22.749 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/3 131 root cwd unknown /proc/131/cwd (readlink: Permission denied) 2024-03-18T15:20:22.749 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/3 131 root rtd unknown /proc/131/root (readlink: Permission denied) 2024-03-18T15:20:22.749 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/3 131 root txt unknown /proc/131/exe (readlink: Permission denied) 2024-03-18T15:20:22.749 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/3 131 root NOFD /proc/131/fd (opendir: Permission denied) 2024-03-18T15:20:22.749 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/u 139 root cwd unknown /proc/139/cwd (readlink: Permission denied) 2024-03-18T15:20:22.749 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/u 139 root rtd unknown /proc/139/root (readlink: Permission denied) 2024-03-18T15:20:22.749 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/u 139 root txt unknown /proc/139/exe (readlink: Permission denied) 2024-03-18T15:20:22.750 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/u 139 root NOFD /proc/139/fd (opendir: Permission denied) 2024-03-18T15:20:22.750 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/2 247 root cwd unknown /proc/247/cwd (readlink: Permission denied) 2024-03-18T15:20:22.750 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/2 247 root rtd unknown /proc/247/root (readlink: Permission denied) 2024-03-18T15:20:22.750 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/2 247 root txt unknown /proc/247/exe (readlink: Permission denied) 2024-03-18T15:20:22.750 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/2 247 root NOFD /proc/247/fd (opendir: Permission denied) 2024-03-18T15:20:22.750 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/4 249 root cwd unknown /proc/249/cwd (readlink: Permission denied) 2024-03-18T15:20:22.750 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/4 249 root rtd unknown /proc/249/root (readlink: Permission denied) 2024-03-18T15:20:22.750 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/4 249 root txt unknown /proc/249/exe (readlink: Permission denied) 2024-03-18T15:20:22.750 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/4 249 root NOFD /proc/249/fd (opendir: Permission denied) 2024-03-18T15:20:22.750 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/1 251 root cwd unknown /proc/251/cwd (readlink: Permission denied) 2024-03-18T15:20:22.750 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/1 251 root rtd unknown /proc/251/root (readlink: Permission denied) 2024-03-18T15:20:22.750 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/1 251 root txt unknown /proc/251/exe (readlink: Permission denied) 2024-03-18T15:20:22.750 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/1 251 root NOFD /proc/251/fd (opendir: Permission denied) 2024-03-18T15:20:22.751 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/5 265 root cwd unknown /proc/265/cwd (readlink: Permission denied) 2024-03-18T15:20:22.751 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/5 265 root rtd unknown /proc/265/root (readlink: Permission denied) 2024-03-18T15:20:22.751 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/5 265 root txt unknown /proc/265/exe (readlink: Permission denied) 2024-03-18T15:20:22.751 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/5 265 root NOFD /proc/265/fd (opendir: Permission denied) 2024-03-18T15:20:22.751 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/5 267 root cwd unknown /proc/267/cwd (readlink: Permission denied) 2024-03-18T15:20:22.751 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/5 267 root rtd unknown /proc/267/root (readlink: Permission denied) 2024-03-18T15:20:22.751 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/5 267 root txt unknown /proc/267/exe (readlink: Permission denied) 2024-03-18T15:20:22.751 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/5 267 root NOFD /proc/267/fd (opendir: Permission denied) 2024-03-18T15:20:22.751 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/6 281 root cwd unknown /proc/281/cwd (readlink: Permission denied) 2024-03-18T15:20:22.751 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/6 281 root rtd unknown /proc/281/root (readlink: Permission denied) 2024-03-18T15:20:22.751 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/6 281 root txt unknown /proc/281/exe (readlink: Permission denied) 2024-03-18T15:20:22.751 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/6 281 root NOFD /proc/281/fd (opendir: Permission denied) 2024-03-18T15:20:22.751 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 400 root cwd unknown /proc/400/cwd (readlink: Permission denied) 2024-03-18T15:20:22.751 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 400 root rtd unknown /proc/400/root (readlink: Permission denied) 2024-03-18T15:20:22.751 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 400 root txt unknown /proc/400/exe (readlink: Permission denied) 2024-03-18T15:20:22.752 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 400 root NOFD /proc/400/fd (opendir: Permission denied) 2024-03-18T15:20:22.752 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 404 root cwd unknown /proc/404/cwd (readlink: Permission denied) 2024-03-18T15:20:22.752 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 404 root rtd unknown /proc/404/root (readlink: Permission denied) 2024-03-18T15:20:22.752 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 404 root txt unknown /proc/404/exe (readlink: Permission denied) 2024-03-18T15:20:22.752 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 404 root NOFD /proc/404/fd (opendir: Permission denied) 2024-03-18T15:20:22.752 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 405 root cwd unknown /proc/405/cwd (readlink: Permission denied) 2024-03-18T15:20:22.752 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 405 root rtd unknown /proc/405/root (readlink: Permission denied) 2024-03-18T15:20:22.752 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 405 root txt unknown /proc/405/exe (readlink: Permission denied) 2024-03-18T15:20:22.752 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 405 root NOFD /proc/405/fd (opendir: Permission denied) 2024-03-18T15:20:22.752 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 433 root cwd unknown /proc/433/cwd (readlink: Permission denied) 2024-03-18T15:20:22.752 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 433 root rtd unknown /proc/433/root (readlink: Permission denied) 2024-03-18T15:20:22.752 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 433 root txt unknown /proc/433/exe (readlink: Permission denied) 2024-03-18T15:20:22.752 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 433 root NOFD /proc/433/fd (opendir: Permission denied) 2024-03-18T15:20:22.752 INFO:tasks.workunit.client.0.smithi019.stdout:jbd2/sda1 445 root cwd unknown /proc/445/cwd (readlink: Permission denied) 2024-03-18T15:20:22.752 INFO:tasks.workunit.client.0.smithi019.stdout:jbd2/sda1 445 root rtd unknown /proc/445/root (readlink: Permission denied) 2024-03-18T15:20:22.753 INFO:tasks.workunit.client.0.smithi019.stdout:jbd2/sda1 445 root txt unknown /proc/445/exe (readlink: Permission denied) 2024-03-18T15:20:22.753 INFO:tasks.workunit.client.0.smithi019.stdout:jbd2/sda1 445 root NOFD /proc/445/fd (opendir: Permission denied) 2024-03-18T15:20:22.753 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 446 root cwd unknown /proc/446/cwd (readlink: Permission denied) 2024-03-18T15:20:22.753 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 446 root rtd unknown /proc/446/root (readlink: Permission denied) 2024-03-18T15:20:22.753 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 446 root txt unknown /proc/446/exe (readlink: Permission denied) 2024-03-18T15:20:22.753 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 446 root NOFD /proc/446/fd (opendir: Permission denied) 2024-03-18T15:20:22.753 INFO:tasks.workunit.client.0.smithi019.stdout:systemd-j 523 root cwd unknown /proc/523/cwd (readlink: Permission denied) 2024-03-18T15:20:22.753 INFO:tasks.workunit.client.0.smithi019.stdout:systemd-j 523 root rtd unknown /proc/523/root (readlink: Permission denied) 2024-03-18T15:20:22.753 INFO:tasks.workunit.client.0.smithi019.stdout:systemd-j 523 root txt unknown /proc/523/exe (readlink: Permission denied) 2024-03-18T15:20:22.753 INFO:tasks.workunit.client.0.smithi019.stdout:systemd-j 523 root NOFD /proc/523/fd (opendir: Permission denied) 2024-03-18T15:20:22.753 INFO:tasks.workunit.client.0.smithi019.stdout:ext4lazyi 529 root cwd unknown /proc/529/cwd (readlink: Permission denied) 2024-03-18T15:20:22.753 INFO:tasks.workunit.client.0.smithi019.stdout:ext4lazyi 529 root rtd unknown /proc/529/root (readlink: Permission denied) 2024-03-18T15:20:22.753 INFO:tasks.workunit.client.0.smithi019.stdout:ext4lazyi 529 root txt unknown /proc/529/exe (readlink: Permission denied) 2024-03-18T15:20:22.753 INFO:tasks.workunit.client.0.smithi019.stdout:ext4lazyi 529 root NOFD /proc/529/fd (opendir: Permission denied) 2024-03-18T15:20:22.753 INFO:tasks.workunit.client.0.smithi019.stdout:systemd-u 543 root cwd unknown /proc/543/cwd (readlink: Permission denied) 2024-03-18T15:20:22.754 INFO:tasks.workunit.client.0.smithi019.stdout:systemd-u 543 root rtd unknown /proc/543/root (readlink: Permission denied) 2024-03-18T15:20:22.754 INFO:tasks.workunit.client.0.smithi019.stdout:systemd-u 543 root txt unknown /proc/543/exe (readlink: Permission denied) 2024-03-18T15:20:22.754 INFO:tasks.workunit.client.0.smithi019.stdout:systemd-u 543 root NOFD /proc/543/fd (opendir: Permission denied) 2024-03-18T15:20:22.754 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 594 root cwd unknown /proc/594/cwd (readlink: Permission denied) 2024-03-18T15:20:22.754 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 594 root rtd unknown /proc/594/root (readlink: Permission denied) 2024-03-18T15:20:22.754 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 594 root txt unknown /proc/594/exe (readlink: Permission denied) 2024-03-18T15:20:22.754 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 594 root NOFD /proc/594/fd (opendir: Permission denied) 2024-03-18T15:20:22.754 INFO:tasks.workunit.client.0.smithi019.stdout:kipmi0 610 root cwd unknown /proc/610/cwd (readlink: Permission denied) 2024-03-18T15:20:22.754 INFO:tasks.workunit.client.0.smithi019.stdout:kipmi0 610 root rtd unknown /proc/610/root (readlink: Permission denied) 2024-03-18T15:20:22.754 INFO:tasks.workunit.client.0.smithi019.stdout:kipmi0 610 root txt unknown /proc/610/exe (readlink: Permission denied) 2024-03-18T15:20:22.754 INFO:tasks.workunit.client.0.smithi019.stdout:kipmi0 610 root NOFD /proc/610/fd (opendir: Permission denied) 2024-03-18T15:20:22.754 INFO:tasks.workunit.client.0.smithi019.stdout:rpcbind 616 rpc cwd unknown /proc/616/cwd (readlink: Permission denied) 2024-03-18T15:20:22.754 INFO:tasks.workunit.client.0.smithi019.stdout:rpcbind 616 rpc rtd unknown /proc/616/root (readlink: Permission denied) 2024-03-18T15:20:22.754 INFO:tasks.workunit.client.0.smithi019.stdout:rpcbind 616 rpc txt unknown /proc/616/exe (readlink: Permission denied) 2024-03-18T15:20:22.754 INFO:tasks.workunit.client.0.smithi019.stdout:rpcbind 616 rpc NOFD /proc/616/fd (opendir: Permission denied) 2024-03-18T15:20:22.755 INFO:tasks.workunit.client.0.smithi019.stdout:auditd 617 root cwd unknown /proc/617/cwd (readlink: Permission denied) 2024-03-18T15:20:22.755 INFO:tasks.workunit.client.0.smithi019.stdout:auditd 617 root rtd unknown /proc/617/root (readlink: Permission denied) 2024-03-18T15:20:22.755 INFO:tasks.workunit.client.0.smithi019.stdout:auditd 617 root txt unknown /proc/617/exe (readlink: Permission denied) 2024-03-18T15:20:22.755 INFO:tasks.workunit.client.0.smithi019.stdout:auditd 617 root NOFD /proc/617/fd (opendir: Permission denied) 2024-03-18T15:20:22.755 INFO:tasks.workunit.client.0.smithi019.stdout:auditd 617 618 auditd root cwd unknown /proc/617/task/618/cwd (readlink: Permission denied) 2024-03-18T15:20:22.755 INFO:tasks.workunit.client.0.smithi019.stdout:auditd 617 618 auditd root rtd unknown /proc/617/task/618/root (readlink: Permission denied) 2024-03-18T15:20:22.755 INFO:tasks.workunit.client.0.smithi019.stdout:auditd 617 618 auditd root txt unknown /proc/617/task/618/exe (readlink: Permission denied) 2024-03-18T15:20:22.755 INFO:tasks.workunit.client.0.smithi019.stdout:auditd 617 618 auditd root NOFD /proc/617/task/618/fd (opendir: Permission denied) 2024-03-18T15:20:22.755 INFO:tasks.workunit.client.0.smithi019.stdout:auditd 617 622 auditd root cwd unknown /proc/617/task/622/cwd (readlink: Permission denied) 2024-03-18T15:20:22.755 INFO:tasks.workunit.client.0.smithi019.stdout:auditd 617 622 auditd root rtd unknown /proc/617/task/622/root (readlink: Permission denied) 2024-03-18T15:20:22.755 INFO:tasks.workunit.client.0.smithi019.stdout:auditd 617 622 auditd root txt unknown /proc/617/task/622/exe (readlink: Permission denied) 2024-03-18T15:20:22.755 INFO:tasks.workunit.client.0.smithi019.stdout:auditd 617 622 auditd root NOFD /proc/617/task/622/fd (opendir: Permission denied) 2024-03-18T15:20:22.755 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 619 root cwd unknown /proc/619/cwd (readlink: Permission denied) 2024-03-18T15:20:22.756 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 619 root rtd unknown /proc/619/root (readlink: Permission denied) 2024-03-18T15:20:22.756 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 619 root txt unknown /proc/619/exe (readlink: Permission denied) 2024-03-18T15:20:22.756 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 619 root NOFD /proc/619/fd (opendir: Permission denied) 2024-03-18T15:20:22.756 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 620 root cwd unknown /proc/620/cwd (readlink: Permission denied) 2024-03-18T15:20:22.756 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 620 root rtd unknown /proc/620/root (readlink: Permission denied) 2024-03-18T15:20:22.756 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 620 root txt unknown /proc/620/exe (readlink: Permission denied) 2024-03-18T15:20:22.756 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 620 root NOFD /proc/620/fd (opendir: Permission denied) 2024-03-18T15:20:22.756 INFO:tasks.workunit.client.0.smithi019.stdout:sedispatc 621 root cwd unknown /proc/621/cwd (readlink: Permission denied) 2024-03-18T15:20:22.756 INFO:tasks.workunit.client.0.smithi019.stdout:sedispatc 621 root rtd unknown /proc/621/root (readlink: Permission denied) 2024-03-18T15:20:22.756 INFO:tasks.workunit.client.0.smithi019.stdout:sedispatc 621 root txt unknown /proc/621/exe (readlink: Permission denied) 2024-03-18T15:20:22.756 INFO:tasks.workunit.client.0.smithi019.stdout:sedispatc 621 root NOFD /proc/621/fd (opendir: Permission denied) 2024-03-18T15:20:22.756 INFO:tasks.workunit.client.0.smithi019.stdout:dbus-brok 658 dbus cwd unknown /proc/658/cwd (readlink: Permission denied) 2024-03-18T15:20:22.756 INFO:tasks.workunit.client.0.smithi019.stdout:dbus-brok 658 dbus rtd unknown /proc/658/root (readlink: Permission denied) 2024-03-18T15:20:22.757 INFO:tasks.workunit.client.0.smithi019.stdout:dbus-brok 658 dbus txt unknown /proc/658/exe (readlink: Permission denied) 2024-03-18T15:20:22.757 INFO:tasks.workunit.client.0.smithi019.stdout:dbus-brok 658 dbus NOFD /proc/658/fd (opendir: Permission denied) 2024-03-18T15:20:22.757 INFO:tasks.workunit.client.0.smithi019.stdout:dbus-brok 659 dbus cwd unknown /proc/659/cwd (readlink: Permission denied) 2024-03-18T15:20:22.757 INFO:tasks.workunit.client.0.smithi019.stdout:dbus-brok 659 dbus rtd unknown /proc/659/root (readlink: Permission denied) 2024-03-18T15:20:22.757 INFO:tasks.workunit.client.0.smithi019.stdout:dbus-brok 659 dbus txt unknown /proc/659/exe (readlink: Permission denied) 2024-03-18T15:20:22.757 INFO:tasks.workunit.client.0.smithi019.stdout:dbus-brok 659 dbus NOFD /proc/659/fd (opendir: Permission denied) 2024-03-18T15:20:22.757 INFO:tasks.workunit.client.0.smithi019.stdout:NetworkMa 660 root cwd unknown /proc/660/cwd (readlink: Permission denied) 2024-03-18T15:20:22.757 INFO:tasks.workunit.client.0.smithi019.stdout:NetworkMa 660 root rtd unknown /proc/660/root (readlink: Permission denied) 2024-03-18T15:20:22.757 INFO:tasks.workunit.client.0.smithi019.stdout:NetworkMa 660 root txt unknown /proc/660/exe (readlink: Permission denied) 2024-03-18T15:20:22.757 INFO:tasks.workunit.client.0.smithi019.stdout:NetworkMa 660 root NOFD /proc/660/fd (opendir: Permission denied) 2024-03-18T15:20:22.757 INFO:tasks.workunit.client.0.smithi019.stdout:NetworkMa 660 679 gmain root cwd unknown /proc/660/task/679/cwd (readlink: Permission denied) 2024-03-18T15:20:22.757 INFO:tasks.workunit.client.0.smithi019.stdout:NetworkMa 660 679 gmain root rtd unknown /proc/660/task/679/root (readlink: Permission denied) 2024-03-18T15:20:22.757 INFO:tasks.workunit.client.0.smithi019.stdout:NetworkMa 660 679 gmain root txt unknown /proc/660/task/679/exe (readlink: Permission denied) 2024-03-18T15:20:22.757 INFO:tasks.workunit.client.0.smithi019.stdout:NetworkMa 660 679 gmain root NOFD /proc/660/task/679/fd (opendir: Permission denied) 2024-03-18T15:20:22.758 INFO:tasks.workunit.client.0.smithi019.stdout:NetworkMa 660 680 gdbus root cwd unknown /proc/660/task/680/cwd (readlink: Permission denied) 2024-03-18T15:20:22.758 INFO:tasks.workunit.client.0.smithi019.stdout:NetworkMa 660 680 gdbus root rtd unknown /proc/660/task/680/root (readlink: Permission denied) 2024-03-18T15:20:22.758 INFO:tasks.workunit.client.0.smithi019.stdout:NetworkMa 660 680 gdbus root txt unknown /proc/660/task/680/exe (readlink: Permission denied) 2024-03-18T15:20:22.758 INFO:tasks.workunit.client.0.smithi019.stdout:NetworkMa 660 680 gdbus root NOFD /proc/660/task/680/fd (opendir: Permission denied) 2024-03-18T15:20:22.758 INFO:tasks.workunit.client.0.smithi019.stdout:irqbalanc 663 root cwd unknown /proc/663/cwd (readlink: Permission denied) 2024-03-18T15:20:22.758 INFO:tasks.workunit.client.0.smithi019.stdout:irqbalanc 663 root rtd unknown /proc/663/root (readlink: Permission denied) 2024-03-18T15:20:22.758 INFO:tasks.workunit.client.0.smithi019.stdout:irqbalanc 663 root txt unknown /proc/663/exe (readlink: Permission denied) 2024-03-18T15:20:22.758 INFO:tasks.workunit.client.0.smithi019.stdout:irqbalanc 663 root NOFD /proc/663/fd (opendir: Permission denied) 2024-03-18T15:20:22.758 INFO:tasks.workunit.client.0.smithi019.stdout:irqbalanc 663 669 gmain root cwd unknown /proc/663/task/669/cwd (readlink: Permission denied) 2024-03-18T15:20:22.758 INFO:tasks.workunit.client.0.smithi019.stdout:irqbalanc 663 669 gmain root rtd unknown /proc/663/task/669/root (readlink: Permission denied) 2024-03-18T15:20:22.758 INFO:tasks.workunit.client.0.smithi019.stdout:irqbalanc 663 669 gmain root txt unknown /proc/663/task/669/exe (readlink: Permission denied) 2024-03-18T15:20:22.758 INFO:tasks.workunit.client.0.smithi019.stdout:irqbalanc 663 669 gmain root NOFD /proc/663/task/669/fd (opendir: Permission denied) 2024-03-18T15:20:22.758 INFO:tasks.workunit.client.0.smithi019.stdout:lsmd 664 libstoragemgmt cwd unknown /proc/664/cwd (readlink: Permission denied) 2024-03-18T15:20:22.758 INFO:tasks.workunit.client.0.smithi019.stdout:lsmd 664 libstoragemgmt rtd unknown /proc/664/root (readlink: Permission denied) 2024-03-18T15:20:22.759 INFO:tasks.workunit.client.0.smithi019.stdout:lsmd 664 libstoragemgmt txt unknown /proc/664/exe (readlink: Permission denied) 2024-03-18T15:20:22.759 INFO:tasks.workunit.client.0.smithi019.stdout:lsmd 664 libstoragemgmt NOFD /proc/664/fd (opendir: Permission denied) 2024-03-18T15:20:22.759 INFO:tasks.workunit.client.0.smithi019.stdout:polkitd 665 polkitd cwd unknown /proc/665/cwd (readlink: Permission denied) 2024-03-18T15:20:22.759 INFO:tasks.workunit.client.0.smithi019.stdout:polkitd 665 polkitd rtd unknown /proc/665/root (readlink: Permission denied) 2024-03-18T15:20:22.759 INFO:tasks.workunit.client.0.smithi019.stdout:polkitd 665 polkitd txt unknown /proc/665/exe (readlink: Permission denied) 2024-03-18T15:20:22.759 INFO:tasks.workunit.client.0.smithi019.stdout:polkitd 665 polkitd NOFD /proc/665/fd (opendir: Permission denied) 2024-03-18T15:20:22.759 INFO:tasks.workunit.client.0.smithi019.stdout:polkitd 665 677 gmain polkitd cwd unknown /proc/665/task/677/cwd (readlink: Permission denied) 2024-03-18T15:20:22.759 INFO:tasks.workunit.client.0.smithi019.stdout:polkitd 665 677 gmain polkitd rtd unknown /proc/665/task/677/root (readlink: Permission denied) 2024-03-18T15:20:22.759 INFO:tasks.workunit.client.0.smithi019.stdout:polkitd 665 677 gmain polkitd txt unknown /proc/665/task/677/exe (readlink: Permission denied) 2024-03-18T15:20:22.759 INFO:tasks.workunit.client.0.smithi019.stdout:polkitd 665 677 gmain polkitd NOFD /proc/665/task/677/fd (opendir: Permission denied) 2024-03-18T15:20:22.759 INFO:tasks.workunit.client.0.smithi019.stdout:polkitd 665 678 gdbus polkitd cwd unknown /proc/665/task/678/cwd (readlink: Permission denied) 2024-03-18T15:20:22.759 INFO:tasks.workunit.client.0.smithi019.stdout:polkitd 665 678 gdbus polkitd rtd unknown /proc/665/task/678/root (readlink: Permission denied) 2024-03-18T15:20:22.759 INFO:tasks.workunit.client.0.smithi019.stdout:polkitd 665 678 gdbus polkitd txt unknown /proc/665/task/678/exe (readlink: Permission denied) 2024-03-18T15:20:22.759 INFO:tasks.workunit.client.0.smithi019.stdout:polkitd 665 678 gdbus polkitd NOFD /proc/665/task/678/fd (opendir: Permission denied) 2024-03-18T15:20:22.759 INFO:tasks.workunit.client.0.smithi019.stdout:polkitd 665 681 JS\x20Hel polkitd cwd unknown /proc/665/task/681/cwd (readlink: Permission denied) 2024-03-18T15:20:22.760 INFO:tasks.workunit.client.0.smithi019.stdout:polkitd 665 681 JS\x20Hel polkitd rtd unknown /proc/665/task/681/root (readlink: Permission denied) 2024-03-18T15:20:22.760 INFO:tasks.workunit.client.0.smithi019.stdout:polkitd 665 681 JS\x20Hel polkitd txt unknown /proc/665/task/681/exe (readlink: Permission denied) 2024-03-18T15:20:22.760 INFO:tasks.workunit.client.0.smithi019.stdout:polkitd 665 681 JS\x20Hel polkitd NOFD /proc/665/task/681/fd (opendir: Permission denied) 2024-03-18T15:20:22.760 INFO:tasks.workunit.client.0.smithi019.stdout:polkitd 665 682 JS\x20Hel polkitd cwd unknown /proc/665/task/682/cwd (readlink: Permission denied) 2024-03-18T15:20:22.760 INFO:tasks.workunit.client.0.smithi019.stdout:polkitd 665 682 JS\x20Hel polkitd rtd unknown /proc/665/task/682/root (readlink: Permission denied) 2024-03-18T15:20:22.760 INFO:tasks.workunit.client.0.smithi019.stdout:polkitd 665 682 JS\x20Hel polkitd txt unknown /proc/665/task/682/exe (readlink: Permission denied) 2024-03-18T15:20:22.760 INFO:tasks.workunit.client.0.smithi019.stdout:polkitd 665 682 JS\x20Hel polkitd NOFD /proc/665/task/682/fd (opendir: Permission denied) 2024-03-18T15:20:22.760 INFO:tasks.workunit.client.0.smithi019.stdout:polkitd 665 683 JS\x20Hel polkitd cwd unknown /proc/665/task/683/cwd (readlink: Permission denied) 2024-03-18T15:20:22.760 INFO:tasks.workunit.client.0.smithi019.stdout:polkitd 665 683 JS\x20Hel polkitd rtd unknown /proc/665/task/683/root (readlink: Permission denied) 2024-03-18T15:20:22.760 INFO:tasks.workunit.client.0.smithi019.stdout:polkitd 665 683 JS\x20Hel polkitd txt unknown /proc/665/task/683/exe (readlink: Permission denied) 2024-03-18T15:20:22.760 INFO:tasks.workunit.client.0.smithi019.stdout:polkitd 665 683 JS\x20Hel polkitd NOFD /proc/665/task/683/fd (opendir: Permission denied) 2024-03-18T15:20:22.760 INFO:tasks.workunit.client.0.smithi019.stdout:polkitd 665 684 JS\x20Hel polkitd cwd unknown /proc/665/task/684/cwd (readlink: Permission denied) 2024-03-18T15:20:22.761 INFO:tasks.workunit.client.0.smithi019.stdout:polkitd 665 684 JS\x20Hel polkitd rtd unknown /proc/665/task/684/root (readlink: Permission denied) 2024-03-18T15:20:22.761 INFO:tasks.workunit.client.0.smithi019.stdout:polkitd 665 684 JS\x20Hel polkitd txt unknown /proc/665/task/684/exe (readlink: Permission denied) 2024-03-18T15:20:22.761 INFO:tasks.workunit.client.0.smithi019.stdout:polkitd 665 684 JS\x20Hel polkitd NOFD /proc/665/task/684/fd (opendir: Permission denied) 2024-03-18T15:20:22.761 INFO:tasks.workunit.client.0.smithi019.stdout:polkitd 665 685 JS\x20Hel polkitd cwd unknown /proc/665/task/685/cwd (readlink: Permission denied) 2024-03-18T15:20:22.761 INFO:tasks.workunit.client.0.smithi019.stdout:polkitd 665 685 JS\x20Hel polkitd rtd unknown /proc/665/task/685/root (readlink: Permission denied) 2024-03-18T15:20:22.761 INFO:tasks.workunit.client.0.smithi019.stdout:polkitd 665 685 JS\x20Hel polkitd txt unknown /proc/665/task/685/exe (readlink: Permission denied) 2024-03-18T15:20:22.761 INFO:tasks.workunit.client.0.smithi019.stdout:polkitd 665 685 JS\x20Hel polkitd NOFD /proc/665/task/685/fd (opendir: Permission denied) 2024-03-18T15:20:22.761 INFO:tasks.workunit.client.0.smithi019.stdout:polkitd 665 686 JS\x20Hel polkitd cwd unknown /proc/665/task/686/cwd (readlink: Permission denied) 2024-03-18T15:20:22.761 INFO:tasks.workunit.client.0.smithi019.stdout:polkitd 665 686 JS\x20Hel polkitd rtd unknown /proc/665/task/686/root (readlink: Permission denied) 2024-03-18T15:20:22.761 INFO:tasks.workunit.client.0.smithi019.stdout:polkitd 665 686 JS\x20Hel polkitd txt unknown /proc/665/task/686/exe (readlink: Permission denied) 2024-03-18T15:20:22.761 INFO:tasks.workunit.client.0.smithi019.stdout:polkitd 665 686 JS\x20Hel polkitd NOFD /proc/665/task/686/fd (opendir: Permission denied) 2024-03-18T15:20:22.761 INFO:tasks.workunit.client.0.smithi019.stdout:polkitd 665 687 JS\x20Hel polkitd cwd unknown /proc/665/task/687/cwd (readlink: Permission denied) 2024-03-18T15:20:22.761 INFO:tasks.workunit.client.0.smithi019.stdout:polkitd 665 687 JS\x20Hel polkitd rtd unknown /proc/665/task/687/root (readlink: Permission denied) 2024-03-18T15:20:22.761 INFO:tasks.workunit.client.0.smithi019.stdout:polkitd 665 687 JS\x20Hel polkitd txt unknown /proc/665/task/687/exe (readlink: Permission denied) 2024-03-18T15:20:22.761 INFO:tasks.workunit.client.0.smithi019.stdout:polkitd 665 687 JS\x20Hel polkitd NOFD /proc/665/task/687/fd (opendir: Permission denied) 2024-03-18T15:20:22.762 INFO:tasks.workunit.client.0.smithi019.stdout:polkitd 665 688 JS\x20Hel polkitd cwd unknown /proc/665/task/688/cwd (readlink: Permission denied) 2024-03-18T15:20:22.762 INFO:tasks.workunit.client.0.smithi019.stdout:polkitd 665 688 JS\x20Hel polkitd rtd unknown /proc/665/task/688/root (readlink: Permission denied) 2024-03-18T15:20:22.762 INFO:tasks.workunit.client.0.smithi019.stdout:polkitd 665 688 JS\x20Hel polkitd txt unknown /proc/665/task/688/exe (readlink: Permission denied) 2024-03-18T15:20:22.762 INFO:tasks.workunit.client.0.smithi019.stdout:polkitd 665 688 JS\x20Hel polkitd NOFD /proc/665/task/688/fd (opendir: Permission denied) 2024-03-18T15:20:22.762 INFO:tasks.workunit.client.0.smithi019.stdout:polkitd 665 691 polkitd polkitd cwd unknown /proc/665/task/691/cwd (readlink: Permission denied) 2024-03-18T15:20:22.762 INFO:tasks.workunit.client.0.smithi019.stdout:polkitd 665 691 polkitd polkitd rtd unknown /proc/665/task/691/root (readlink: Permission denied) 2024-03-18T15:20:22.762 INFO:tasks.workunit.client.0.smithi019.stdout:polkitd 665 691 polkitd polkitd txt unknown /proc/665/task/691/exe (readlink: Permission denied) 2024-03-18T15:20:22.762 INFO:tasks.workunit.client.0.smithi019.stdout:polkitd 665 691 polkitd polkitd NOFD /proc/665/task/691/fd (opendir: Permission denied) 2024-03-18T15:20:22.762 INFO:tasks.workunit.client.0.smithi019.stdout:systemd-l 668 root cwd unknown /proc/668/cwd (readlink: Permission denied) 2024-03-18T15:20:22.762 INFO:tasks.workunit.client.0.smithi019.stdout:systemd-l 668 root rtd unknown /proc/668/root (readlink: Permission denied) 2024-03-18T15:20:22.762 INFO:tasks.workunit.client.0.smithi019.stdout:systemd-l 668 root txt unknown /proc/668/exe (readlink: Permission denied) 2024-03-18T15:20:22.762 INFO:tasks.workunit.client.0.smithi019.stdout:systemd-l 668 root NOFD /proc/668/fd (opendir: Permission denied) 2024-03-18T15:20:22.762 INFO:tasks.workunit.client.0.smithi019.stdout:sshd 706 root cwd unknown /proc/706/cwd (readlink: Permission denied) 2024-03-18T15:20:22.763 INFO:tasks.workunit.client.0.smithi019.stdout:sshd 706 root rtd unknown /proc/706/root (readlink: Permission denied) 2024-03-18T15:20:22.763 INFO:tasks.workunit.client.0.smithi019.stdout:sshd 706 root txt unknown /proc/706/exe (readlink: Permission denied) 2024-03-18T15:20:22.763 INFO:tasks.workunit.client.0.smithi019.stdout:sshd 706 root NOFD /proc/706/fd (opendir: Permission denied) 2024-03-18T15:20:22.763 INFO:tasks.workunit.client.0.smithi019.stdout:tuned 707 root cwd unknown /proc/707/cwd (readlink: Permission denied) 2024-03-18T15:20:22.763 INFO:tasks.workunit.client.0.smithi019.stdout:tuned 707 root rtd unknown /proc/707/root (readlink: Permission denied) 2024-03-18T15:20:22.763 INFO:tasks.workunit.client.0.smithi019.stdout:tuned 707 root txt unknown /proc/707/exe (readlink: Permission denied) 2024-03-18T15:20:22.763 INFO:tasks.workunit.client.0.smithi019.stdout:tuned 707 root NOFD /proc/707/fd (opendir: Permission denied) 2024-03-18T15:20:22.763 INFO:tasks.workunit.client.0.smithi019.stdout:tuned 707 810 tuned root cwd unknown /proc/707/task/810/cwd (readlink: Permission denied) 2024-03-18T15:20:22.763 INFO:tasks.workunit.client.0.smithi019.stdout:tuned 707 810 tuned root rtd unknown /proc/707/task/810/root (readlink: Permission denied) 2024-03-18T15:20:22.763 INFO:tasks.workunit.client.0.smithi019.stdout:tuned 707 810 tuned root txt unknown /proc/707/task/810/exe (readlink: Permission denied) 2024-03-18T15:20:22.763 INFO:tasks.workunit.client.0.smithi019.stdout:tuned 707 810 tuned root NOFD /proc/707/task/810/fd (opendir: Permission denied) 2024-03-18T15:20:22.763 INFO:tasks.workunit.client.0.smithi019.stdout:gssproxy 729 root cwd unknown /proc/729/cwd (readlink: Permission denied) 2024-03-18T15:20:22.763 INFO:tasks.workunit.client.0.smithi019.stdout:gssproxy 729 root rtd unknown /proc/729/root (readlink: Permission denied) 2024-03-18T15:20:22.763 INFO:tasks.workunit.client.0.smithi019.stdout:gssproxy 729 root txt unknown /proc/729/exe (readlink: Permission denied) 2024-03-18T15:20:22.763 INFO:tasks.workunit.client.0.smithi019.stdout:gssproxy 729 root NOFD /proc/729/fd (opendir: Permission denied) 2024-03-18T15:20:22.764 INFO:tasks.workunit.client.0.smithi019.stdout:gssproxy 729 730 gssproxy root cwd unknown /proc/729/task/730/cwd (readlink: Permission denied) 2024-03-18T15:20:22.764 INFO:tasks.workunit.client.0.smithi019.stdout:gssproxy 729 730 gssproxy root rtd unknown /proc/729/task/730/root (readlink: Permission denied) 2024-03-18T15:20:22.764 INFO:tasks.workunit.client.0.smithi019.stdout:gssproxy 729 730 gssproxy root txt unknown /proc/729/task/730/exe (readlink: Permission denied) 2024-03-18T15:20:22.764 INFO:tasks.workunit.client.0.smithi019.stdout:gssproxy 729 730 gssproxy root NOFD /proc/729/task/730/fd (opendir: Permission denied) 2024-03-18T15:20:22.764 INFO:tasks.workunit.client.0.smithi019.stdout:gssproxy 729 731 gssproxy root cwd unknown /proc/729/task/731/cwd (readlink: Permission denied) 2024-03-18T15:20:22.764 INFO:tasks.workunit.client.0.smithi019.stdout:gssproxy 729 731 gssproxy root rtd unknown /proc/729/task/731/root (readlink: Permission denied) 2024-03-18T15:20:22.764 INFO:tasks.workunit.client.0.smithi019.stdout:gssproxy 729 731 gssproxy root txt unknown /proc/729/task/731/exe (readlink: Permission denied) 2024-03-18T15:20:22.764 INFO:tasks.workunit.client.0.smithi019.stdout:gssproxy 729 731 gssproxy root NOFD /proc/729/task/731/fd (opendir: Permission denied) 2024-03-18T15:20:22.764 INFO:tasks.workunit.client.0.smithi019.stdout:gssproxy 729 732 gssproxy root cwd unknown /proc/729/task/732/cwd (readlink: Permission denied) 2024-03-18T15:20:22.764 INFO:tasks.workunit.client.0.smithi019.stdout:gssproxy 729 732 gssproxy root rtd unknown /proc/729/task/732/root (readlink: Permission denied) 2024-03-18T15:20:22.764 INFO:tasks.workunit.client.0.smithi019.stdout:gssproxy 729 732 gssproxy root txt unknown /proc/729/task/732/exe (readlink: Permission denied) 2024-03-18T15:20:22.764 INFO:tasks.workunit.client.0.smithi019.stdout:gssproxy 729 732 gssproxy root NOFD /proc/729/task/732/fd (opendir: Permission denied) 2024-03-18T15:20:22.764 INFO:tasks.workunit.client.0.smithi019.stdout:gssproxy 729 733 gssproxy root cwd unknown /proc/729/task/733/cwd (readlink: Permission denied) 2024-03-18T15:20:22.764 INFO:tasks.workunit.client.0.smithi019.stdout:gssproxy 729 733 gssproxy root rtd unknown /proc/729/task/733/root (readlink: Permission denied) 2024-03-18T15:20:22.765 INFO:tasks.workunit.client.0.smithi019.stdout:gssproxy 729 733 gssproxy root txt unknown /proc/729/task/733/exe (readlink: Permission denied) 2024-03-18T15:20:22.765 INFO:tasks.workunit.client.0.smithi019.stdout:gssproxy 729 733 gssproxy root NOFD /proc/729/task/733/fd (opendir: Permission denied) 2024-03-18T15:20:22.765 INFO:tasks.workunit.client.0.smithi019.stdout:gssproxy 729 734 gssproxy root cwd unknown /proc/729/task/734/cwd (readlink: Permission denied) 2024-03-18T15:20:22.765 INFO:tasks.workunit.client.0.smithi019.stdout:gssproxy 729 734 gssproxy root rtd unknown /proc/729/task/734/root (readlink: Permission denied) 2024-03-18T15:20:22.765 INFO:tasks.workunit.client.0.smithi019.stdout:gssproxy 729 734 gssproxy root txt unknown /proc/729/task/734/exe (readlink: Permission denied) 2024-03-18T15:20:22.765 INFO:tasks.workunit.client.0.smithi019.stdout:gssproxy 729 734 gssproxy root NOFD /proc/729/task/734/fd (opendir: Permission denied) 2024-03-18T15:20:22.765 INFO:tasks.workunit.client.0.smithi019.stdout:atd 736 root cwd unknown /proc/736/cwd (readlink: Permission denied) 2024-03-18T15:20:22.765 INFO:tasks.workunit.client.0.smithi019.stdout:atd 736 root rtd unknown /proc/736/root (readlink: Permission denied) 2024-03-18T15:20:22.765 INFO:tasks.workunit.client.0.smithi019.stdout:atd 736 root txt unknown /proc/736/exe (readlink: Permission denied) 2024-03-18T15:20:22.765 INFO:tasks.workunit.client.0.smithi019.stdout:atd 736 root NOFD /proc/736/fd (opendir: Permission denied) 2024-03-18T15:20:22.765 INFO:tasks.workunit.client.0.smithi019.stdout:crond 737 root cwd unknown /proc/737/cwd (readlink: Permission denied) 2024-03-18T15:20:22.765 INFO:tasks.workunit.client.0.smithi019.stdout:crond 737 root rtd unknown /proc/737/root (readlink: Permission denied) 2024-03-18T15:20:22.765 INFO:tasks.workunit.client.0.smithi019.stdout:crond 737 root txt unknown /proc/737/exe (readlink: Permission denied) 2024-03-18T15:20:22.765 INFO:tasks.workunit.client.0.smithi019.stdout:crond 737 root NOFD /proc/737/fd (opendir: Permission denied) 2024-03-18T15:20:22.765 INFO:tasks.workunit.client.0.smithi019.stdout:nrpe 835 nrpe cwd unknown /proc/835/cwd (readlink: Permission denied) 2024-03-18T15:20:22.766 INFO:tasks.workunit.client.0.smithi019.stdout:nrpe 835 nrpe rtd unknown /proc/835/root (readlink: Permission denied) 2024-03-18T15:20:22.766 INFO:tasks.workunit.client.0.smithi019.stdout:nrpe 835 nrpe txt unknown /proc/835/exe (readlink: Permission denied) 2024-03-18T15:20:22.766 INFO:tasks.workunit.client.0.smithi019.stdout:nrpe 835 nrpe NOFD /proc/835/fd (opendir: Permission denied) 2024-03-18T15:20:22.766 INFO:tasks.workunit.client.0.smithi019.stdout:agetty 1154 root cwd unknown /proc/1154/cwd (readlink: Permission denied) 2024-03-18T15:20:22.766 INFO:tasks.workunit.client.0.smithi019.stdout:agetty 1154 root rtd unknown /proc/1154/root (readlink: Permission denied) 2024-03-18T15:20:22.766 INFO:tasks.workunit.client.0.smithi019.stdout:agetty 1154 root txt unknown /proc/1154/exe (readlink: Permission denied) 2024-03-18T15:20:22.766 INFO:tasks.workunit.client.0.smithi019.stdout:agetty 1154 root NOFD /proc/1154/fd (opendir: Permission denied) 2024-03-18T15:20:22.766 INFO:tasks.workunit.client.0.smithi019.stdout:agetty 1156 root cwd unknown /proc/1156/cwd (readlink: Permission denied) 2024-03-18T15:20:22.766 INFO:tasks.workunit.client.0.smithi019.stdout:agetty 1156 root rtd unknown /proc/1156/root (readlink: Permission denied) 2024-03-18T15:20:22.766 INFO:tasks.workunit.client.0.smithi019.stdout:agetty 1156 root txt unknown /proc/1156/exe (readlink: Permission denied) 2024-03-18T15:20:22.766 INFO:tasks.workunit.client.0.smithi019.stdout:agetty 1156 root NOFD /proc/1156/fd (opendir: Permission denied) 2024-03-18T15:20:22.766 INFO:tasks.workunit.client.0.smithi019.stdout:systemd 4279 ubuntu cwd DIR 8,1 4096 2 / 2024-03-18T15:20:22.766 INFO:tasks.workunit.client.0.smithi019.stdout:systemd 4279 ubuntu rtd DIR 8,1 4096 2 / 2024-03-18T15:20:22.766 INFO:tasks.workunit.client.0.smithi019.stdout:systemd 4279 ubuntu txt REG 8,1 102120 20353 /usr/lib/systemd/systemd 2024-03-18T15:20:22.766 INFO:tasks.workunit.client.0.smithi019.stdout:systemd 4279 ubuntu DEL REG 8,1 1167 /etc/selinux/targeted/contexts/files/file_contexts.bin 2024-03-18T15:20:22.767 INFO:tasks.workunit.client.0.smithi019.stdout:systemd 4279 ubuntu mem REG 8,1 904672 9627 /usr/lib64/libm.so.6 2024-03-18T15:20:22.767 INFO:tasks.workunit.client.0.smithi019.stdout:systemd 4279 ubuntu mem REG 8,1 882376 9699 /usr/lib64/libzstd.so.1.5.1 2024-03-18T15:20:22.767 INFO:tasks.workunit.client.0.smithi019.stdout:systemd 4279 ubuntu DEL REG 8,1 10007 /usr/lib64/libcrypto.so.3.0.7 2024-03-18T15:20:22.767 INFO:tasks.workunit.client.0.smithi019.stdout:systemd 4279 ubuntu mem REG 8,1 1293840 9789 /usr/lib64/libp11-kit.so.0.3.0 2024-03-18T15:20:22.767 INFO:tasks.workunit.client.0.smithi019.stdout:systemd 4279 ubuntu mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-03-18T15:20:22.767 INFO:tasks.workunit.client.0.smithi019.stdout:systemd 4279 ubuntu DEL REG 8,1 1697 /etc/selinux/targeted/contexts/files/file_contexts.homedirs.bin 2024-03-18T15:20:22.767 INFO:tasks.workunit.client.0.smithi019.stdout:systemd 4279 ubuntu mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-03-18T15:20:22.767 INFO:tasks.workunit.client.0.smithi019.stdout:systemd 4279 ubuntu mem REG 8,1 1288504 9883 /usr/lib64/libgcrypt.so.20.4.0 2024-03-18T15:20:22.767 INFO:tasks.workunit.client.0.smithi019.stdout:systemd 4279 ubuntu mem REG 8,1 3645472 49253 /usr/lib64/systemd/libsystemd-shared-252.so 2024-03-18T15:20:22.767 INFO:tasks.workunit.client.0.smithi019.stdout:systemd 4279 ubuntu mem REG 8,1 44784 9785 /usr/lib64/libffi.so.8.1.0 2024-03-18T15:20:22.767 INFO:tasks.workunit.client.0.smithi019.stdout:systemd 4279 ubuntu mem REG 8,1 153600 9809 /usr/lib64/libgpg-error.so.0.32.0 2024-03-18T15:20:22.767 INFO:tasks.workunit.client.0.smithi019.stdout:systemd 4279 ubuntu mem REG 8,1 28552 9804 /usr/lib64/libattr.so.1.1.2501 2024-03-18T15:20:22.767 INFO:tasks.workunit.client.0.smithi019.stdout:systemd 4279 ubuntu mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-03-18T15:20:22.767 INFO:tasks.workunit.client.0.smithi019.stdout:systemd 4279 ubuntu mem REG 8,1 32528 9723 /usr/lib64/libcap-ng.so.0.0.0 2024-03-18T15:20:22.767 INFO:tasks.workunit.client.0.smithi019.stdout:systemd 4279 ubuntu mem REG 8,1 40424 10956 /usr/lib64/libeconf.so.0.4.1 2024-03-18T15:20:22.767 INFO:tasks.workunit.client.0.smithi019.stdout:systemd 4279 ubuntu mem REG 8,1 178728 9694 /usr/lib64/liblzma.so.5.2.5 2024-03-18T15:20:22.768 INFO:tasks.workunit.client.0.smithi019.stdout:systemd 4279 ubuntu mem REG 8,1 2174088 49251 /usr/lib64/systemd/libsystemd-core-252.so 2024-03-18T15:20:22.768 INFO:tasks.workunit.client.0.smithi019.stdout:systemd 4279 ubuntu mem REG 8,1 144120 9867 /usr/lib64/liblz4.so.1.9.3 2024-03-18T15:20:22.768 INFO:tasks.workunit.client.0.smithi019.stdout:systemd 4279 ubuntu mem REG 8,1 201808 9696 /usr/lib64/libcrypt.so.2.0.0 2024-03-18T15:20:22.768 INFO:tasks.workunit.client.0.smithi019.stdout:systemd 4279 ubuntu mem REG 8,1 36296 9702 /usr/lib64/libcap.so.2.48 2024-03-18T15:20:22.768 INFO:tasks.workunit.client.0.smithi019.stdout:systemd 4279 ubuntu DEL REG 8,1 10126 /usr/lib64/libblkid.so.1.1.0 2024-03-18T15:20:22.768 INFO:tasks.workunit.client.0.smithi019.stdout:systemd 4279 ubuntu DEL REG 8,1 9806 /usr/lib64/libacl.so.1.1.2301 2024-03-18T15:20:22.768 INFO:tasks.workunit.client.0.smithi019.stdout:systemd 4279 ubuntu DEL REG 8,1 10156 /usr/lib64/libmount.so.1.1.0 2024-03-18T15:20:22.768 INFO:tasks.workunit.client.0.smithi019.stdout:systemd 4279 ubuntu mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-03-18T15:20:22.768 INFO:tasks.workunit.client.0.smithi019.stdout:systemd 4279 ubuntu mem REG 8,1 107424 10180 /usr/lib64/libkmod.so.2.3.6 2024-03-18T15:20:22.768 INFO:tasks.workunit.client.0.smithi019.stdout:systemd 4279 ubuntu mem REG 8,1 134624 9727 /usr/lib64/libaudit.so.1.0.0 2024-03-18T15:20:22.768 INFO:tasks.workunit.client.0.smithi019.stdout:systemd 4279 ubuntu mem REG 8,1 69472 10965 /usr/lib64/libpam.so.0.85.1 2024-03-18T15:20:22.768 INFO:tasks.workunit.client.0.smithi019.stdout:systemd 4279 ubuntu mem REG 8,1 127680 9972 /usr/lib64/libseccomp.so.2.5.2 2024-03-18T15:20:22.768 INFO:tasks.workunit.client.0.smithi019.stdout:systemd 4279 ubuntu mem REG 8,1 108136 13107 /usr/lib64/libgcc_s-11-20230605.so.1 2024-03-18T15:20:22.768 INFO:tasks.workunit.client.0.smithi019.stdout:systemd 4279 ubuntu mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-03-18T15:20:22.768 INFO:tasks.workunit.client.0.smithi019.stdout:systemd 4279 ubuntu mem REG 0,19 0 19 /sys/fs/selinux/status 2024-03-18T15:20:22.768 INFO:tasks.workunit.client.0.smithi019.stdout:systemd 4279 ubuntu 0r CHR 1,3 0t0 4 /dev/null 2024-03-18T15:20:22.769 INFO:tasks.workunit.client.0.smithi019.stdout:systemd 4279 ubuntu 1u unix 0x0000000000000000 0t0 9055 type=STREAM (CONNECTED) 2024-03-18T15:20:22.769 INFO:tasks.workunit.client.0.smithi019.stdout:systemd 4279 ubuntu 2u unix 0x0000000000000000 0t0 9055 type=STREAM (CONNECTED) 2024-03-18T15:20:22.769 INFO:tasks.workunit.client.0.smithi019.stdout:systemd 4279 ubuntu 3u unix 0x0000000000000000 0t0 9083 type=DGRAM (CONNECTED) 2024-03-18T15:20:22.769 INFO:tasks.workunit.client.0.smithi019.stdout:systemd 4279 ubuntu 4u a_inode 0,14 0 50 [eventpoll:5,6,8,9,11,12,14,15,16,17,19,20,21] 2024-03-18T15:20:22.769 INFO:tasks.workunit.client.0.smithi019.stdout:systemd 4279 ubuntu 5u a_inode 0,14 0 50 [signalfd] 2024-03-18T15:20:22.769 INFO:tasks.workunit.client.0.smithi019.stdout:systemd 4279 ubuntu 6r a_inode 0,14 0 50 inotify 2024-03-18T15:20:22.769 INFO:tasks.workunit.client.0.smithi019.stdout:systemd 4279 ubuntu 7r DIR 0,25 0 3657 /sys/fs/cgroup/user.slice/user-1000.slice/user@1000.service 2024-03-18T15:20:22.769 INFO:tasks.workunit.client.0.smithi019.stdout:systemd 4279 ubuntu 8u a_inode 0,14 0 50 [timerfd] 2024-03-18T15:20:22.769 INFO:tasks.workunit.client.0.smithi019.stdout:systemd 4279 ubuntu 9u a_inode 0,14 0 50 [eventpoll:10,13] 2024-03-18T15:20:22.769 INFO:tasks.workunit.client.0.smithi019.stdout:systemd 4279 ubuntu 10r REG 0,20 0 9072 /proc/4279/mountinfo 2024-03-18T15:20:22.769 INFO:tasks.workunit.client.0.smithi019.stdout:systemd 4279 ubuntu 11r a_inode 0,14 0 50 inotify 2024-03-18T15:20:22.769 INFO:tasks.workunit.client.0.smithi019.stdout:systemd 4279 ubuntu 12u unix 0x0000000000000000 0t0 9105 /run/user/1000/bus type=STREAM (LISTEN) 2024-03-18T15:20:22.769 INFO:tasks.workunit.client.0.smithi019.stdout:systemd 4279 ubuntu 13r a_inode 0,14 0 50 inotify 2024-03-18T15:20:22.769 INFO:tasks.workunit.client.0.smithi019.stdout:systemd 4279 ubuntu 14r REG 0,20 0 4026532078 /proc/swaps 2024-03-18T15:20:22.769 INFO:tasks.workunit.client.0.smithi019.stdout:systemd 4279 ubuntu 15u netlink 0t0 9087 KOBJECT_UEVENT 2024-03-18T15:20:22.770 INFO:tasks.workunit.client.0.smithi019.stdout:systemd 4279 ubuntu 16u unix 0x0000000000000000 0t0 9092 /run/user/1000/systemd/notify type=DGRAM (UNCONNECTED) 2024-03-18T15:20:22.770 INFO:tasks.workunit.client.0.smithi019.stdout:systemd 4279 ubuntu 17u unix 0x0000000000000000 0t0 9093 type=DGRAM (CONNECTED) 2024-03-18T15:20:22.770 INFO:tasks.workunit.client.0.smithi019.stdout:systemd 4279 ubuntu 18u unix 0x0000000000000000 0t0 9094 type=DGRAM (CONNECTED) 2024-03-18T15:20:22.770 INFO:tasks.workunit.client.0.smithi019.stdout:systemd 4279 ubuntu 19u unix 0x0000000000000000 0t0 9095 /run/user/1000/systemd/private type=STREAM (LISTEN) 2024-03-18T15:20:22.770 INFO:tasks.workunit.client.0.smithi019.stdout:systemd 4279 ubuntu 20u unix 0x0000000000000000 0t0 9097 type=STREAM (CONNECTED) 2024-03-18T15:20:22.770 INFO:tasks.workunit.client.0.smithi019.stdout:systemd 4279 ubuntu 21u a_inode 0,14 0 50 [timerfd] 2024-03-18T15:20:22.770 INFO:tasks.workunit.client.0.smithi019.stdout:(sd-pam) 4281 ubuntu cwd unknown /proc/4281/cwd (readlink: Permission denied) 2024-03-18T15:20:22.770 INFO:tasks.workunit.client.0.smithi019.stdout:(sd-pam) 4281 ubuntu rtd unknown /proc/4281/root (readlink: Permission denied) 2024-03-18T15:20:22.770 INFO:tasks.workunit.client.0.smithi019.stdout:(sd-pam) 4281 ubuntu txt unknown /proc/4281/exe (readlink: Permission denied) 2024-03-18T15:20:22.770 INFO:tasks.workunit.client.0.smithi019.stdout:(sd-pam) 4281 ubuntu NOFD /proc/4281/fd (opendir: Permission denied) 2024-03-18T15:20:22.770 INFO:tasks.workunit.client.0.smithi019.stdout:rsyslogd 4524 root cwd unknown /proc/4524/cwd (readlink: Permission denied) 2024-03-18T15:20:22.770 INFO:tasks.workunit.client.0.smithi019.stdout:rsyslogd 4524 root rtd unknown /proc/4524/root (readlink: Permission denied) 2024-03-18T15:20:22.770 INFO:tasks.workunit.client.0.smithi019.stdout:rsyslogd 4524 root txt unknown /proc/4524/exe (readlink: Permission denied) 2024-03-18T15:20:22.770 INFO:tasks.workunit.client.0.smithi019.stdout:rsyslogd 4524 root NOFD /proc/4524/fd (opendir: Permission denied) 2024-03-18T15:20:22.770 INFO:tasks.workunit.client.0.smithi019.stdout:rsyslogd 4524 4526 in:imjour root cwd unknown /proc/4524/task/4526/cwd (readlink: Permission denied) 2024-03-18T15:20:22.771 INFO:tasks.workunit.client.0.smithi019.stdout:rsyslogd 4524 4526 in:imjour root rtd unknown /proc/4524/task/4526/root (readlink: Permission denied) 2024-03-18T15:20:22.771 INFO:tasks.workunit.client.0.smithi019.stdout:rsyslogd 4524 4526 in:imjour root txt unknown /proc/4524/task/4526/exe (readlink: Permission denied) 2024-03-18T15:20:22.771 INFO:tasks.workunit.client.0.smithi019.stdout:rsyslogd 4524 4526 in:imjour root NOFD /proc/4524/task/4526/fd (opendir: Permission denied) 2024-03-18T15:20:22.771 INFO:tasks.workunit.client.0.smithi019.stdout:rsyslogd 4524 4527 rs:main root cwd unknown /proc/4524/task/4527/cwd (readlink: Permission denied) 2024-03-18T15:20:22.771 INFO:tasks.workunit.client.0.smithi019.stdout:rsyslogd 4524 4527 rs:main root rtd unknown /proc/4524/task/4527/root (readlink: Permission denied) 2024-03-18T15:20:22.771 INFO:tasks.workunit.client.0.smithi019.stdout:rsyslogd 4524 4527 rs:main root txt unknown /proc/4524/task/4527/exe (readlink: Permission denied) 2024-03-18T15:20:22.771 INFO:tasks.workunit.client.0.smithi019.stdout:rsyslogd 4524 4527 rs:main root NOFD /proc/4524/task/4527/fd (opendir: Permission denied) 2024-03-18T15:20:22.771 INFO:tasks.workunit.client.0.smithi019.stdout:smartd 15308 root cwd unknown /proc/15308/cwd (readlink: Permission denied) 2024-03-18T15:20:22.771 INFO:tasks.workunit.client.0.smithi019.stdout:smartd 15308 root rtd unknown /proc/15308/root (readlink: Permission denied) 2024-03-18T15:20:22.771 INFO:tasks.workunit.client.0.smithi019.stdout:smartd 15308 root txt unknown /proc/15308/exe (readlink: Permission denied) 2024-03-18T15:20:22.771 INFO:tasks.workunit.client.0.smithi019.stdout:smartd 15308 root NOFD /proc/15308/fd (opendir: Permission denied) 2024-03-18T15:20:22.771 INFO:tasks.workunit.client.0.smithi019.stdout:anacron 19361 root cwd unknown /proc/19361/cwd (readlink: Permission denied) 2024-03-18T15:20:22.771 INFO:tasks.workunit.client.0.smithi019.stdout:anacron 19361 root rtd unknown /proc/19361/root (readlink: Permission denied) 2024-03-18T15:20:22.771 INFO:tasks.workunit.client.0.smithi019.stdout:anacron 19361 root txt unknown /proc/19361/exe (readlink: Permission denied) 2024-03-18T15:20:22.771 INFO:tasks.workunit.client.0.smithi019.stdout:anacron 19361 root NOFD /proc/19361/fd (opendir: Permission denied) 2024-03-18T15:20:22.771 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/3 19461 root cwd unknown /proc/19461/cwd (readlink: Permission denied) 2024-03-18T15:20:22.772 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/3 19461 root rtd unknown /proc/19461/root (readlink: Permission denied) 2024-03-18T15:20:22.772 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/3 19461 root txt unknown /proc/19461/exe (readlink: Permission denied) 2024-03-18T15:20:22.772 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/3 19461 root NOFD /proc/19461/fd (opendir: Permission denied) 2024-03-18T15:20:22.772 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/0 20308 root cwd unknown /proc/20308/cwd (readlink: Permission denied) 2024-03-18T15:20:22.772 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/0 20308 root rtd unknown /proc/20308/root (readlink: Permission denied) 2024-03-18T15:20:22.772 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/0 20308 root txt unknown /proc/20308/exe (readlink: Permission denied) 2024-03-18T15:20:22.772 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/0 20308 root NOFD /proc/20308/fd (opendir: Permission denied) 2024-03-18T15:20:22.772 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 20673 root cwd unknown /proc/20673/cwd (readlink: Permission denied) 2024-03-18T15:20:22.772 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 20673 root rtd unknown /proc/20673/root (readlink: Permission denied) 2024-03-18T15:20:22.772 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 20673 root txt unknown /proc/20673/exe (readlink: Permission denied) 2024-03-18T15:20:22.772 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 20673 root NOFD /proc/20673/fd (opendir: Permission denied) 2024-03-18T15:20:22.772 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 20703 root cwd unknown /proc/20703/cwd (readlink: Permission denied) 2024-03-18T15:20:22.772 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 20703 root rtd unknown /proc/20703/root (readlink: Permission denied) 2024-03-18T15:20:22.772 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 20703 root txt unknown /proc/20703/exe (readlink: Permission denied) 2024-03-18T15:20:22.773 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 20703 root NOFD /proc/20703/fd (opendir: Permission denied) 2024-03-18T15:20:22.773 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 20732 root cwd unknown /proc/20732/cwd (readlink: Permission denied) 2024-03-18T15:20:22.773 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 20732 root rtd unknown /proc/20732/root (readlink: Permission denied) 2024-03-18T15:20:22.773 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 20732 root txt unknown /proc/20732/exe (readlink: Permission denied) 2024-03-18T15:20:22.773 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 20732 root NOFD /proc/20732/fd (opendir: Permission denied) 2024-03-18T15:20:22.773 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 20761 root cwd unknown /proc/20761/cwd (readlink: Permission denied) 2024-03-18T15:20:22.773 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 20761 root rtd unknown /proc/20761/root (readlink: Permission denied) 2024-03-18T15:20:22.773 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 20761 root txt unknown /proc/20761/exe (readlink: Permission denied) 2024-03-18T15:20:22.773 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 20761 root NOFD /proc/20761/fd (opendir: Permission denied) 2024-03-18T15:20:22.773 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 20790 root cwd unknown /proc/20790/cwd (readlink: Permission denied) 2024-03-18T15:20:22.773 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 20790 root rtd unknown /proc/20790/root (readlink: Permission denied) 2024-03-18T15:20:22.773 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 20790 root txt unknown /proc/20790/exe (readlink: Permission denied) 2024-03-18T15:20:22.773 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 20790 root NOFD /proc/20790/fd (opendir: Permission denied) 2024-03-18T15:20:22.773 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 20954 root cwd unknown /proc/20954/cwd (readlink: Permission denied) 2024-03-18T15:20:22.773 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 20954 root rtd unknown /proc/20954/root (readlink: Permission denied) 2024-03-18T15:20:22.774 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 20954 root txt unknown /proc/20954/exe (readlink: Permission denied) 2024-03-18T15:20:22.774 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 20954 root NOFD /proc/20954/fd (opendir: Permission denied) 2024-03-18T15:20:22.774 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 20955 root cwd unknown /proc/20955/cwd (readlink: Permission denied) 2024-03-18T15:20:22.774 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 20955 root rtd unknown /proc/20955/root (readlink: Permission denied) 2024-03-18T15:20:22.774 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 20955 root txt unknown /proc/20955/exe (readlink: Permission denied) 2024-03-18T15:20:22.774 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 20955 root NOFD /proc/20955/fd (opendir: Permission denied) 2024-03-18T15:20:22.774 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 20956 root cwd unknown /proc/20956/cwd (readlink: Permission denied) 2024-03-18T15:20:22.774 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 20956 root rtd unknown /proc/20956/root (readlink: Permission denied) 2024-03-18T15:20:22.774 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 20956 root txt unknown /proc/20956/exe (readlink: Permission denied) 2024-03-18T15:20:22.774 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 20956 root NOFD /proc/20956/fd (opendir: Permission denied) 2024-03-18T15:20:22.774 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 20957 root cwd unknown /proc/20957/cwd (readlink: Permission denied) 2024-03-18T15:20:22.774 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 20957 root rtd unknown /proc/20957/root (readlink: Permission denied) 2024-03-18T15:20:22.774 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 20957 root txt unknown /proc/20957/exe (readlink: Permission denied) 2024-03-18T15:20:22.774 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 20957 root NOFD /proc/20957/fd (opendir: Permission denied) 2024-03-18T15:20:22.775 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 20958 root cwd unknown /proc/20958/cwd (readlink: Permission denied) 2024-03-18T15:20:22.775 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 20958 root rtd unknown /proc/20958/root (readlink: Permission denied) 2024-03-18T15:20:22.775 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 20958 root txt unknown /proc/20958/exe (readlink: Permission denied) 2024-03-18T15:20:22.775 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 20958 root NOFD /proc/20958/fd (opendir: Permission denied) 2024-03-18T15:20:22.775 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 20959 root cwd unknown /proc/20959/cwd (readlink: Permission denied) 2024-03-18T15:20:22.775 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 20959 root rtd unknown /proc/20959/root (readlink: Permission denied) 2024-03-18T15:20:22.775 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 20959 root txt unknown /proc/20959/exe (readlink: Permission denied) 2024-03-18T15:20:22.775 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 20959 root NOFD /proc/20959/fd (opendir: Permission denied) 2024-03-18T15:20:22.775 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 20960 root cwd unknown /proc/20960/cwd (readlink: Permission denied) 2024-03-18T15:20:22.775 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 20960 root rtd unknown /proc/20960/root (readlink: Permission denied) 2024-03-18T15:20:22.775 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 20960 root txt unknown /proc/20960/exe (readlink: Permission denied) 2024-03-18T15:20:22.775 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 20960 root NOFD /proc/20960/fd (opendir: Permission denied) 2024-03-18T15:20:22.775 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 20961 root cwd unknown /proc/20961/cwd (readlink: Permission denied) 2024-03-18T15:20:22.775 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 20961 root rtd unknown /proc/20961/root (readlink: Permission denied) 2024-03-18T15:20:22.775 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 20961 root txt unknown /proc/20961/exe (readlink: Permission denied) 2024-03-18T15:20:22.775 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 20961 root NOFD /proc/20961/fd (opendir: Permission denied) 2024-03-18T15:20:22.776 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 20962 root cwd unknown /proc/20962/cwd (readlink: Permission denied) 2024-03-18T15:20:22.776 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 20962 root rtd unknown /proc/20962/root (readlink: Permission denied) 2024-03-18T15:20:22.776 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 20962 root txt unknown /proc/20962/exe (readlink: Permission denied) 2024-03-18T15:20:22.776 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 20962 root NOFD /proc/20962/fd (opendir: Permission denied) 2024-03-18T15:20:22.776 INFO:tasks.workunit.client.0.smithi019.stdout:xfsaild/d 20963 root cwd unknown /proc/20963/cwd (readlink: Permission denied) 2024-03-18T15:20:22.776 INFO:tasks.workunit.client.0.smithi019.stdout:xfsaild/d 20963 root rtd unknown /proc/20963/root (readlink: Permission denied) 2024-03-18T15:20:22.776 INFO:tasks.workunit.client.0.smithi019.stdout:xfsaild/d 20963 root txt unknown /proc/20963/exe (readlink: Permission denied) 2024-03-18T15:20:22.776 INFO:tasks.workunit.client.0.smithi019.stdout:xfsaild/d 20963 root NOFD /proc/20963/fd (opendir: Permission denied) 2024-03-18T15:20:22.776 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/4 22587 root cwd unknown /proc/22587/cwd (readlink: Permission denied) 2024-03-18T15:20:22.776 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/4 22587 root rtd unknown /proc/22587/root (readlink: Permission denied) 2024-03-18T15:20:22.776 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/4 22587 root txt unknown /proc/22587/exe (readlink: Permission denied) 2024-03-18T15:20:22.776 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/4 22587 root NOFD /proc/22587/fd (opendir: Permission denied) 2024-03-18T15:20:22.776 INFO:tasks.workunit.client.0.smithi019.stdout:sshd 22898 root cwd unknown /proc/22898/cwd (readlink: Permission denied) 2024-03-18T15:20:22.777 INFO:tasks.workunit.client.0.smithi019.stdout:sshd 22898 root rtd unknown /proc/22898/root (readlink: Permission denied) 2024-03-18T15:20:22.777 INFO:tasks.workunit.client.0.smithi019.stdout:sshd 22898 root txt unknown /proc/22898/exe (readlink: Permission denied) 2024-03-18T15:20:22.777 INFO:tasks.workunit.client.0.smithi019.stdout:sshd 22898 root NOFD /proc/22898/fd (opendir: Permission denied) 2024-03-18T15:20:22.777 INFO:tasks.workunit.client.0.smithi019.stdout:sshd 22902 ubuntu cwd unknown /proc/22902/cwd (readlink: Permission denied) 2024-03-18T15:20:22.777 INFO:tasks.workunit.client.0.smithi019.stdout:sshd 22902 ubuntu rtd unknown /proc/22902/root (readlink: Permission denied) 2024-03-18T15:20:22.777 INFO:tasks.workunit.client.0.smithi019.stdout:sshd 22902 ubuntu txt unknown /proc/22902/exe (readlink: Permission denied) 2024-03-18T15:20:22.777 INFO:tasks.workunit.client.0.smithi019.stdout:sshd 22902 ubuntu NOFD /proc/22902/fd (opendir: Permission denied) 2024-03-18T15:20:22.777 INFO:tasks.workunit.client.0.smithi019.stdout:chronyd 22962 chrony cwd unknown /proc/22962/cwd (readlink: Permission denied) 2024-03-18T15:20:22.777 INFO:tasks.workunit.client.0.smithi019.stdout:chronyd 22962 chrony rtd unknown /proc/22962/root (readlink: Permission denied) 2024-03-18T15:20:22.777 INFO:tasks.workunit.client.0.smithi019.stdout:chronyd 22962 chrony txt unknown /proc/22962/exe (readlink: Permission denied) 2024-03-18T15:20:22.777 INFO:tasks.workunit.client.0.smithi019.stdout:chronyd 22962 chrony NOFD /proc/22962/fd (opendir: Permission denied) 2024-03-18T15:20:22.777 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-cras 23312 ceph cwd unknown /proc/23312/cwd (readlink: Permission denied) 2024-03-18T15:20:22.777 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-cras 23312 ceph rtd unknown /proc/23312/root (readlink: Permission denied) 2024-03-18T15:20:22.777 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-cras 23312 ceph txt unknown /proc/23312/exe (readlink: Permission denied) 2024-03-18T15:20:22.777 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-cras 23312 ceph NOFD /proc/23312/fd (opendir: Permission denied) 2024-03-18T15:20:22.778 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/1 24325 root cwd unknown /proc/24325/cwd (readlink: Permission denied) 2024-03-18T15:20:22.778 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/1 24325 root rtd unknown /proc/24325/root (readlink: Permission denied) 2024-03-18T15:20:22.778 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/1 24325 root txt unknown /proc/24325/exe (readlink: Permission denied) 2024-03-18T15:20:22.778 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/1 24325 root NOFD /proc/24325/fd (opendir: Permission denied) 2024-03-18T15:20:22.778 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/7 25037 root cwd unknown /proc/25037/cwd (readlink: Permission denied) 2024-03-18T15:20:22.778 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/7 25037 root rtd unknown /proc/25037/root (readlink: Permission denied) 2024-03-18T15:20:22.778 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/7 25037 root txt unknown /proc/25037/exe (readlink: Permission denied) 2024-03-18T15:20:22.778 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/7 25037 root NOFD /proc/25037/fd (opendir: Permission denied) 2024-03-18T15:20:22.778 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/7 25038 root cwd unknown /proc/25038/cwd (readlink: Permission denied) 2024-03-18T15:20:22.778 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/7 25038 root rtd unknown /proc/25038/root (readlink: Permission denied) 2024-03-18T15:20:22.778 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/7 25038 root txt unknown /proc/25038/exe (readlink: Permission denied) 2024-03-18T15:20:22.778 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/7 25038 root NOFD /proc/25038/fd (opendir: Permission denied) 2024-03-18T15:20:22.778 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/5 27471 root cwd unknown /proc/27471/cwd (readlink: Permission denied) 2024-03-18T15:20:22.778 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/5 27471 root rtd unknown /proc/27471/root (readlink: Permission denied) 2024-03-18T15:20:22.778 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/5 27471 root txt unknown /proc/27471/exe (readlink: Permission denied) 2024-03-18T15:20:22.779 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/5 27471 root NOFD /proc/27471/fd (opendir: Permission denied) 2024-03-18T15:20:22.779 INFO:tasks.workunit.client.0.smithi019.stdout:sudo 30514 root cwd unknown /proc/30514/cwd (readlink: Permission denied) 2024-03-18T15:20:22.779 INFO:tasks.workunit.client.0.smithi019.stdout:sudo 30514 root rtd unknown /proc/30514/root (readlink: Permission denied) 2024-03-18T15:20:22.779 INFO:tasks.workunit.client.0.smithi019.stdout:sudo 30514 root txt unknown /proc/30514/exe (readlink: Permission denied) 2024-03-18T15:20:22.779 INFO:tasks.workunit.client.0.smithi019.stdout:sudo 30514 root NOFD /proc/30514/fd (opendir: Permission denied) 2024-03-18T15:20:22.779 INFO:tasks.workunit.client.0.smithi019.stdout:sudo 30530 root cwd unknown /proc/30530/cwd (readlink: Permission denied) 2024-03-18T15:20:22.779 INFO:tasks.workunit.client.0.smithi019.stdout:sudo 30530 root rtd unknown /proc/30530/root (readlink: Permission denied) 2024-03-18T15:20:22.779 INFO:tasks.workunit.client.0.smithi019.stdout:sudo 30530 root txt unknown /proc/30530/exe (readlink: Permission denied) 2024-03-18T15:20:22.779 INFO:tasks.workunit.client.0.smithi019.stdout:sudo 30530 root NOFD /proc/30530/fd (opendir: Permission denied) 2024-03-18T15:20:22.779 INFO:tasks.workunit.client.0.smithi019.stdout:journalct 30546 root cwd unknown /proc/30546/cwd (readlink: Permission denied) 2024-03-18T15:20:22.779 INFO:tasks.workunit.client.0.smithi019.stdout:journalct 30546 root rtd unknown /proc/30546/root (readlink: Permission denied) 2024-03-18T15:20:22.779 INFO:tasks.workunit.client.0.smithi019.stdout:journalct 30546 root txt unknown /proc/30546/exe (readlink: Permission denied) 2024-03-18T15:20:22.779 INFO:tasks.workunit.client.0.smithi019.stdout:journalct 30546 root NOFD /proc/30546/fd (opendir: Permission denied) 2024-03-18T15:20:22.779 INFO:tasks.workunit.client.0.smithi019.stdout:journalct 30565 root cwd unknown /proc/30565/cwd (readlink: Permission denied) 2024-03-18T15:20:22.779 INFO:tasks.workunit.client.0.smithi019.stdout:journalct 30565 root rtd unknown /proc/30565/root (readlink: Permission denied) 2024-03-18T15:20:22.780 INFO:tasks.workunit.client.0.smithi019.stdout:journalct 30565 root txt unknown /proc/30565/exe (readlink: Permission denied) 2024-03-18T15:20:22.780 INFO:tasks.workunit.client.0.smithi019.stdout:journalct 30565 root NOFD /proc/30565/fd (opendir: Permission denied) 2024-03-18T15:20:22.780 INFO:tasks.workunit.client.0.smithi019.stdout:conmon 31518 root cwd unknown /proc/31518/cwd (readlink: Permission denied) 2024-03-18T15:20:22.780 INFO:tasks.workunit.client.0.smithi019.stdout:conmon 31518 root rtd unknown /proc/31518/root (readlink: Permission denied) 2024-03-18T15:20:22.780 INFO:tasks.workunit.client.0.smithi019.stdout:conmon 31518 root txt unknown /proc/31518/exe (readlink: Permission denied) 2024-03-18T15:20:22.780 INFO:tasks.workunit.client.0.smithi019.stdout:conmon 31518 root NOFD /proc/31518/fd (opendir: Permission denied) 2024-03-18T15:20:22.780 INFO:tasks.workunit.client.0.smithi019.stdout:podman-in 31520 root cwd unknown /proc/31520/cwd (readlink: Permission denied) 2024-03-18T15:20:22.780 INFO:tasks.workunit.client.0.smithi019.stdout:podman-in 31520 root rtd unknown /proc/31520/root (readlink: Permission denied) 2024-03-18T15:20:22.780 INFO:tasks.workunit.client.0.smithi019.stdout:podman-in 31520 root txt unknown /proc/31520/exe (readlink: Permission denied) 2024-03-18T15:20:22.780 INFO:tasks.workunit.client.0.smithi019.stdout:podman-in 31520 root NOFD /proc/31520/fd (opendir: Permission denied) 2024-03-18T15:20:22.780 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 ceph cwd unknown /proc/31522/cwd (readlink: Permission denied) 2024-03-18T15:20:22.780 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 ceph rtd unknown /proc/31522/root (readlink: Permission denied) 2024-03-18T15:20:22.780 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 ceph txt unknown /proc/31522/exe (readlink: Permission denied) 2024-03-18T15:20:22.780 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 ceph NOFD /proc/31522/fd (opendir: Permission denied) 2024-03-18T15:20:22.780 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 31523 log ceph cwd unknown /proc/31522/task/31523/cwd (readlink: Permission denied) 2024-03-18T15:20:22.781 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 31523 log ceph rtd unknown /proc/31522/task/31523/root (readlink: Permission denied) 2024-03-18T15:20:22.781 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 31523 log ceph txt unknown /proc/31522/task/31523/exe (readlink: Permission denied) 2024-03-18T15:20:22.781 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 31523 log ceph NOFD /proc/31522/task/31523/fd (opendir: Permission denied) 2024-03-18T15:20:22.781 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 31524 service ceph cwd unknown /proc/31522/task/31524/cwd (readlink: Permission denied) 2024-03-18T15:20:22.781 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 31524 service ceph rtd unknown /proc/31522/task/31524/root (readlink: Permission denied) 2024-03-18T15:20:22.781 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 31524 service ceph txt unknown /proc/31522/task/31524/exe (readlink: Permission denied) 2024-03-18T15:20:22.781 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 31524 service ceph NOFD /proc/31522/task/31524/fd (opendir: Permission denied) 2024-03-18T15:20:22.781 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 31525 admin_soc ceph cwd unknown /proc/31522/task/31525/cwd (readlink: Permission denied) 2024-03-18T15:20:22.781 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 31525 admin_soc ceph rtd unknown /proc/31522/task/31525/root (readlink: Permission denied) 2024-03-18T15:20:22.781 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 31525 admin_soc ceph txt unknown /proc/31522/task/31525/exe (readlink: Permission denied) 2024-03-18T15:20:22.781 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 31525 admin_soc ceph NOFD /proc/31522/task/31525/fd (opendir: Permission denied) 2024-03-18T15:20:22.781 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 31526 signal_ha ceph cwd unknown /proc/31522/task/31526/cwd (readlink: Permission denied) 2024-03-18T15:20:22.781 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 31526 signal_ha ceph rtd unknown /proc/31522/task/31526/root (readlink: Permission denied) 2024-03-18T15:20:22.781 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 31526 signal_ha ceph txt unknown /proc/31522/task/31526/exe (readlink: Permission denied) 2024-03-18T15:20:22.781 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 31526 signal_ha ceph NOFD /proc/31522/task/31526/fd (opendir: Permission denied) 2024-03-18T15:20:22.782 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 31527 rocksdb:l ceph cwd unknown /proc/31522/task/31527/cwd (readlink: Permission denied) 2024-03-18T15:20:22.782 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 31527 rocksdb:l ceph rtd unknown /proc/31522/task/31527/root (readlink: Permission denied) 2024-03-18T15:20:22.782 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 31527 rocksdb:l ceph txt unknown /proc/31522/task/31527/exe (readlink: Permission denied) 2024-03-18T15:20:22.782 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 31527 rocksdb:l ceph NOFD /proc/31522/task/31527/fd (opendir: Permission denied) 2024-03-18T15:20:22.782 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 31528 rocksdb:h ceph cwd unknown /proc/31522/task/31528/cwd (readlink: Permission denied) 2024-03-18T15:20:22.782 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 31528 rocksdb:h ceph rtd unknown /proc/31522/task/31528/root (readlink: Permission denied) 2024-03-18T15:20:22.782 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 31528 rocksdb:h ceph txt unknown /proc/31522/task/31528/exe (readlink: Permission denied) 2024-03-18T15:20:22.782 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 31528 rocksdb:h ceph NOFD /proc/31522/task/31528/fd (opendir: Permission denied) 2024-03-18T15:20:22.782 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 31544 ceph-mon ceph cwd unknown /proc/31522/task/31544/cwd (readlink: Permission denied) 2024-03-18T15:20:22.782 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 31544 ceph-mon ceph rtd unknown /proc/31522/task/31544/root (readlink: Permission denied) 2024-03-18T15:20:22.782 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 31544 ceph-mon ceph txt unknown /proc/31522/task/31544/exe (readlink: Permission denied) 2024-03-18T15:20:22.782 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 31544 ceph-mon ceph NOFD /proc/31522/task/31544/fd (opendir: Permission denied) 2024-03-18T15:20:22.782 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 31545 fn_monsto ceph cwd unknown /proc/31522/task/31545/cwd (readlink: Permission denied) 2024-03-18T15:20:22.782 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 31545 fn_monsto ceph rtd unknown /proc/31522/task/31545/root (readlink: Permission denied) 2024-03-18T15:20:22.783 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 31545 fn_monsto ceph txt unknown /proc/31522/task/31545/exe (readlink: Permission denied) 2024-03-18T15:20:22.783 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 31545 fn_monsto ceph NOFD /proc/31522/task/31545/fd (opendir: Permission denied) 2024-03-18T15:20:22.783 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 31546 msgr-work ceph cwd unknown /proc/31522/task/31546/cwd (readlink: Permission denied) 2024-03-18T15:20:22.783 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 31546 msgr-work ceph rtd unknown /proc/31522/task/31546/root (readlink: Permission denied) 2024-03-18T15:20:22.783 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 31546 msgr-work ceph txt unknown /proc/31522/task/31546/exe (readlink: Permission denied) 2024-03-18T15:20:22.783 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 31546 msgr-work ceph NOFD /proc/31522/task/31546/fd (opendir: Permission denied) 2024-03-18T15:20:22.783 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 31547 msgr-work ceph cwd unknown /proc/31522/task/31547/cwd (readlink: Permission denied) 2024-03-18T15:20:22.783 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 31547 msgr-work ceph rtd unknown /proc/31522/task/31547/root (readlink: Permission denied) 2024-03-18T15:20:22.783 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 31547 msgr-work ceph txt unknown /proc/31522/task/31547/exe (readlink: Permission denied) 2024-03-18T15:20:22.783 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 31547 msgr-work ceph NOFD /proc/31522/task/31547/fd (opendir: Permission denied) 2024-03-18T15:20:22.783 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 31548 msgr-work ceph cwd unknown /proc/31522/task/31548/cwd (readlink: Permission denied) 2024-03-18T15:20:22.783 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 31548 msgr-work ceph rtd unknown /proc/31522/task/31548/root (readlink: Permission denied) 2024-03-18T15:20:22.783 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 31548 msgr-work ceph txt unknown /proc/31522/task/31548/exe (readlink: Permission denied) 2024-03-18T15:20:22.784 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 31548 msgr-work ceph NOFD /proc/31522/task/31548/fd (opendir: Permission denied) 2024-03-18T15:20:22.784 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 31549 OpHistory ceph cwd unknown /proc/31522/task/31549/cwd (readlink: Permission denied) 2024-03-18T15:20:22.785 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 31549 OpHistory ceph rtd unknown /proc/31522/task/31549/root (readlink: Permission denied) 2024-03-18T15:20:22.785 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 31549 OpHistory ceph txt unknown /proc/31522/task/31549/exe (readlink: Permission denied) 2024-03-18T15:20:22.785 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 31549 OpHistory ceph NOFD /proc/31522/task/31549/fd (opendir: Permission denied) 2024-03-18T15:20:22.785 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 31550 fin ceph cwd unknown /proc/31522/task/31550/cwd (readlink: Permission denied) 2024-03-18T15:20:22.785 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 31550 fin ceph rtd unknown /proc/31522/task/31550/root (readlink: Permission denied) 2024-03-18T15:20:22.785 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 31550 fin ceph txt unknown /proc/31522/task/31550/exe (readlink: Permission denied) 2024-03-18T15:20:22.785 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 31550 fin ceph NOFD /proc/31522/task/31550/fd (opendir: Permission denied) 2024-03-18T15:20:22.785 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 31551 safe_time ceph cwd unknown /proc/31522/task/31551/cwd (readlink: Permission denied) 2024-03-18T15:20:22.785 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 31551 safe_time ceph rtd unknown /proc/31522/task/31551/root (readlink: Permission denied) 2024-03-18T15:20:22.786 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 31551 safe_time ceph txt unknown /proc/31522/task/31551/exe (readlink: Permission denied) 2024-03-18T15:20:22.786 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 31551 safe_time ceph NOFD /proc/31522/task/31551/fd (opendir: Permission denied) 2024-03-18T15:20:22.786 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 31552 cpu_tp ceph cwd unknown /proc/31522/task/31552/cwd (readlink: Permission denied) 2024-03-18T15:20:22.786 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 31552 cpu_tp ceph rtd unknown /proc/31522/task/31552/root (readlink: Permission denied) 2024-03-18T15:20:22.786 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 31552 cpu_tp ceph txt unknown /proc/31522/task/31552/exe (readlink: Permission denied) 2024-03-18T15:20:22.786 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 31552 cpu_tp ceph NOFD /proc/31522/task/31552/fd (opendir: Permission denied) 2024-03-18T15:20:22.786 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 31553 cpu_tp ceph cwd unknown /proc/31522/task/31553/cwd (readlink: Permission denied) 2024-03-18T15:20:22.786 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 31553 cpu_tp ceph rtd unknown /proc/31522/task/31553/root (readlink: Permission denied) 2024-03-18T15:20:22.786 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 31553 cpu_tp ceph txt unknown /proc/31522/task/31553/exe (readlink: Permission denied) 2024-03-18T15:20:22.786 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 31553 cpu_tp ceph NOFD /proc/31522/task/31553/fd (opendir: Permission denied) 2024-03-18T15:20:22.786 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 31554 cpu_tp ceph cwd unknown /proc/31522/task/31554/cwd (readlink: Permission denied) 2024-03-18T15:20:22.786 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 31554 cpu_tp ceph rtd unknown /proc/31522/task/31554/root (readlink: Permission denied) 2024-03-18T15:20:22.786 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 31554 cpu_tp ceph txt unknown /proc/31522/task/31554/exe (readlink: Permission denied) 2024-03-18T15:20:22.786 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 31554 cpu_tp ceph NOFD /proc/31522/task/31554/fd (opendir: Permission denied) 2024-03-18T15:20:22.786 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 31555 cpu_tp ceph cwd unknown /proc/31522/task/31555/cwd (readlink: Permission denied) 2024-03-18T15:20:22.787 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 31555 cpu_tp ceph rtd unknown /proc/31522/task/31555/root (readlink: Permission denied) 2024-03-18T15:20:22.787 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 31555 cpu_tp ceph txt unknown /proc/31522/task/31555/exe (readlink: Permission denied) 2024-03-18T15:20:22.787 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 31555 cpu_tp ceph NOFD /proc/31522/task/31555/fd (opendir: Permission denied) 2024-03-18T15:20:22.787 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 31556 ms_dispat ceph cwd unknown /proc/31522/task/31556/cwd (readlink: Permission denied) 2024-03-18T15:20:22.787 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 31556 ms_dispat ceph rtd unknown /proc/31522/task/31556/root (readlink: Permission denied) 2024-03-18T15:20:22.787 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 31556 ms_dispat ceph txt unknown /proc/31522/task/31556/exe (readlink: Permission denied) 2024-03-18T15:20:22.787 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 31556 ms_dispat ceph NOFD /proc/31522/task/31556/fd (opendir: Permission denied) 2024-03-18T15:20:22.787 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 31557 ms_local ceph cwd unknown /proc/31522/task/31557/cwd (readlink: Permission denied) 2024-03-18T15:20:22.787 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 31557 ms_local ceph rtd unknown /proc/31522/task/31557/root (readlink: Permission denied) 2024-03-18T15:20:22.787 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 31557 ms_local ceph txt unknown /proc/31522/task/31557/exe (readlink: Permission denied) 2024-03-18T15:20:22.787 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 31557 ms_local ceph NOFD /proc/31522/task/31557/fd (opendir: Permission denied) 2024-03-18T15:20:22.787 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 31558 safe_time ceph cwd unknown /proc/31522/task/31558/cwd (readlink: Permission denied) 2024-03-18T15:20:22.787 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 31558 safe_time ceph rtd unknown /proc/31522/task/31558/root (readlink: Permission denied) 2024-03-18T15:20:22.787 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 31558 safe_time ceph txt unknown /proc/31522/task/31558/exe (readlink: Permission denied) 2024-03-18T15:20:22.787 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 31558 safe_time ceph NOFD /proc/31522/task/31558/fd (opendir: Permission denied) 2024-03-18T15:20:22.788 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 31559 ms_dispat ceph cwd unknown /proc/31522/task/31559/cwd (readlink: Permission denied) 2024-03-18T15:20:22.788 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 31559 ms_dispat ceph rtd unknown /proc/31522/task/31559/root (readlink: Permission denied) 2024-03-18T15:20:22.788 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 31559 ms_dispat ceph txt unknown /proc/31522/task/31559/exe (readlink: Permission denied) 2024-03-18T15:20:22.788 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 31559 ms_dispat ceph NOFD /proc/31522/task/31559/fd (opendir: Permission denied) 2024-03-18T15:20:22.788 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 31560 ms_local ceph cwd unknown /proc/31522/task/31560/cwd (readlink: Permission denied) 2024-03-18T15:20:22.788 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 31560 ms_local ceph rtd unknown /proc/31522/task/31560/root (readlink: Permission denied) 2024-03-18T15:20:22.788 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 31560 ms_local ceph txt unknown /proc/31522/task/31560/exe (readlink: Permission denied) 2024-03-18T15:20:22.788 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 31560 ms_local ceph NOFD /proc/31522/task/31560/fd (opendir: Permission denied) 2024-03-18T15:20:22.788 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 67760 rstore_co ceph cwd unknown /proc/31522/task/67760/cwd (readlink: Permission denied) 2024-03-18T15:20:22.788 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 67760 rstore_co ceph rtd unknown /proc/31522/task/67760/root (readlink: Permission denied) 2024-03-18T15:20:22.788 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 67760 rstore_co ceph txt unknown /proc/31522/task/67760/exe (readlink: Permission denied) 2024-03-18T15:20:22.788 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mon 31522 67760 rstore_co ceph NOFD /proc/31522/task/67760/fd (opendir: Permission denied) 2024-03-18T15:20:22.788 INFO:tasks.workunit.client.0.smithi019.stdout:conmon 31750 root cwd unknown /proc/31750/cwd (readlink: Permission denied) 2024-03-18T15:20:22.788 INFO:tasks.workunit.client.0.smithi019.stdout:conmon 31750 root rtd unknown /proc/31750/root (readlink: Permission denied) 2024-03-18T15:20:22.789 INFO:tasks.workunit.client.0.smithi019.stdout:conmon 31750 root txt unknown /proc/31750/exe (readlink: Permission denied) 2024-03-18T15:20:22.789 INFO:tasks.workunit.client.0.smithi019.stdout:conmon 31750 root NOFD /proc/31750/fd (opendir: Permission denied) 2024-03-18T15:20:22.789 INFO:tasks.workunit.client.0.smithi019.stdout:podman-in 31752 root cwd unknown /proc/31752/cwd (readlink: Permission denied) 2024-03-18T15:20:22.789 INFO:tasks.workunit.client.0.smithi019.stdout:podman-in 31752 root rtd unknown /proc/31752/root (readlink: Permission denied) 2024-03-18T15:20:22.789 INFO:tasks.workunit.client.0.smithi019.stdout:podman-in 31752 root txt unknown /proc/31752/exe (readlink: Permission denied) 2024-03-18T15:20:22.789 INFO:tasks.workunit.client.0.smithi019.stdout:podman-in 31752 root NOFD /proc/31752/fd (opendir: Permission denied) 2024-03-18T15:20:22.789 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 ceph cwd unknown /proc/31759/cwd (readlink: Permission denied) 2024-03-18T15:20:22.789 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 ceph rtd unknown /proc/31759/root (readlink: Permission denied) 2024-03-18T15:20:22.789 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 ceph txt unknown /proc/31759/exe (readlink: Permission denied) 2024-03-18T15:20:22.789 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 ceph NOFD /proc/31759/fd (opendir: Permission denied) 2024-03-18T15:20:22.789 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86443 log ceph cwd unknown /proc/31759/task/86443/cwd (readlink: Permission denied) 2024-03-18T15:20:22.789 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86443 log ceph rtd unknown /proc/31759/task/86443/root (readlink: Permission denied) 2024-03-18T15:20:22.789 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86443 log ceph txt unknown /proc/31759/task/86443/exe (readlink: Permission denied) 2024-03-18T15:20:22.789 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86443 log ceph NOFD /proc/31759/task/86443/fd (opendir: Permission denied) 2024-03-18T15:20:22.789 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86445 msgr-work ceph cwd unknown /proc/31759/task/86445/cwd (readlink: Permission denied) 2024-03-18T15:20:22.790 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86445 msgr-work ceph rtd unknown /proc/31759/task/86445/root (readlink: Permission denied) 2024-03-18T15:20:22.790 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86445 msgr-work ceph txt unknown /proc/31759/task/86445/exe (readlink: Permission denied) 2024-03-18T15:20:22.790 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86445 msgr-work ceph NOFD /proc/31759/task/86445/fd (opendir: Permission denied) 2024-03-18T15:20:22.790 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86446 msgr-work ceph cwd unknown /proc/31759/task/86446/cwd (readlink: Permission denied) 2024-03-18T15:20:22.790 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86446 msgr-work ceph rtd unknown /proc/31759/task/86446/root (readlink: Permission denied) 2024-03-18T15:20:22.790 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86446 msgr-work ceph txt unknown /proc/31759/task/86446/exe (readlink: Permission denied) 2024-03-18T15:20:22.790 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86446 msgr-work ceph NOFD /proc/31759/task/86446/fd (opendir: Permission denied) 2024-03-18T15:20:22.790 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86447 msgr-work ceph cwd unknown /proc/31759/task/86447/cwd (readlink: Permission denied) 2024-03-18T15:20:22.790 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86447 msgr-work ceph rtd unknown /proc/31759/task/86447/root (readlink: Permission denied) 2024-03-18T15:20:22.790 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86447 msgr-work ceph txt unknown /proc/31759/task/86447/exe (readlink: Permission denied) 2024-03-18T15:20:22.790 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86447 msgr-work ceph NOFD /proc/31759/task/86447/fd (opendir: Permission denied) 2024-03-18T15:20:22.790 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86451 service ceph cwd unknown /proc/31759/task/86451/cwd (readlink: Permission denied) 2024-03-18T15:20:22.790 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86451 service ceph rtd unknown /proc/31759/task/86451/root (readlink: Permission denied) 2024-03-18T15:20:22.790 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86451 service ceph txt unknown /proc/31759/task/86451/exe (readlink: Permission denied) 2024-03-18T15:20:22.791 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86451 service ceph NOFD /proc/31759/task/86451/fd (opendir: Permission denied) 2024-03-18T15:20:22.791 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86452 admin_soc ceph cwd unknown /proc/31759/task/86452/cwd (readlink: Permission denied) 2024-03-18T15:20:22.791 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86452 admin_soc ceph rtd unknown /proc/31759/task/86452/root (readlink: Permission denied) 2024-03-18T15:20:22.791 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86452 admin_soc ceph txt unknown /proc/31759/task/86452/exe (readlink: Permission denied) 2024-03-18T15:20:22.791 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86452 admin_soc ceph NOFD /proc/31759/task/86452/fd (opendir: Permission denied) 2024-03-18T15:20:22.791 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86453 ceph_time ceph cwd unknown /proc/31759/task/86453/cwd (readlink: Permission denied) 2024-03-18T15:20:22.791 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86453 ceph_time ceph rtd unknown /proc/31759/task/86453/root (readlink: Permission denied) 2024-03-18T15:20:22.791 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86453 ceph_time ceph txt unknown /proc/31759/task/86453/exe (readlink: Permission denied) 2024-03-18T15:20:22.791 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86453 ceph_time ceph NOFD /proc/31759/task/86453/fd (opendir: Permission denied) 2024-03-18T15:20:22.791 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86454 fn_anonym ceph cwd unknown /proc/31759/task/86454/cwd (readlink: Permission denied) 2024-03-18T15:20:22.791 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86454 fn_anonym ceph rtd unknown /proc/31759/task/86454/root (readlink: Permission denied) 2024-03-18T15:20:22.791 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86454 fn_anonym ceph txt unknown /proc/31759/task/86454/exe (readlink: Permission denied) 2024-03-18T15:20:22.791 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86454 fn_anonym ceph NOFD /proc/31759/task/86454/fd (opendir: Permission denied) 2024-03-18T15:20:22.791 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86455 signal_ha ceph cwd unknown /proc/31759/task/86455/cwd (readlink: Permission denied) 2024-03-18T15:20:22.792 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86455 signal_ha ceph rtd unknown /proc/31759/task/86455/root (readlink: Permission denied) 2024-03-18T15:20:22.792 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86455 signal_ha ceph txt unknown /proc/31759/task/86455/exe (readlink: Permission denied) 2024-03-18T15:20:22.792 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86455 signal_ha ceph NOFD /proc/31759/task/86455/fd (opendir: Permission denied) 2024-03-18T15:20:22.792 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86456 mgrsb-fin ceph cwd unknown /proc/31759/task/86456/cwd (readlink: Permission denied) 2024-03-18T15:20:22.792 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86456 mgrsb-fin ceph rtd unknown /proc/31759/task/86456/root (readlink: Permission denied) 2024-03-18T15:20:22.792 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86456 mgrsb-fin ceph txt unknown /proc/31759/task/86456/exe (readlink: Permission denied) 2024-03-18T15:20:22.792 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86456 mgrsb-fin ceph NOFD /proc/31759/task/86456/fd (opendir: Permission denied) 2024-03-18T15:20:22.792 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86457 ms_dispat ceph cwd unknown /proc/31759/task/86457/cwd (readlink: Permission denied) 2024-03-18T15:20:22.792 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86457 ms_dispat ceph rtd unknown /proc/31759/task/86457/root (readlink: Permission denied) 2024-03-18T15:20:22.792 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86457 ms_dispat ceph txt unknown /proc/31759/task/86457/exe (readlink: Permission denied) 2024-03-18T15:20:22.792 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86457 ms_dispat ceph NOFD /proc/31759/task/86457/fd (opendir: Permission denied) 2024-03-18T15:20:22.792 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86458 ms_local ceph cwd unknown /proc/31759/task/86458/cwd (readlink: Permission denied) 2024-03-18T15:20:22.792 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86458 ms_local ceph rtd unknown /proc/31759/task/86458/root (readlink: Permission denied) 2024-03-18T15:20:22.792 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86458 ms_local ceph txt unknown /proc/31759/task/86458/exe (readlink: Permission denied) 2024-03-18T15:20:22.793 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86458 ms_local ceph NOFD /proc/31759/task/86458/fd (opendir: Permission denied) 2024-03-18T15:20:22.793 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86459 io_contex ceph cwd unknown /proc/31759/task/86459/cwd (readlink: Permission denied) 2024-03-18T15:20:22.793 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86459 io_contex ceph rtd unknown /proc/31759/task/86459/root (readlink: Permission denied) 2024-03-18T15:20:22.793 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86459 io_contex ceph txt unknown /proc/31759/task/86459/exe (readlink: Permission denied) 2024-03-18T15:20:22.793 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86459 io_contex ceph NOFD /proc/31759/task/86459/fd (opendir: Permission denied) 2024-03-18T15:20:22.793 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86460 io_contex ceph cwd unknown /proc/31759/task/86460/cwd (readlink: Permission denied) 2024-03-18T15:20:22.793 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86460 io_contex ceph rtd unknown /proc/31759/task/86460/root (readlink: Permission denied) 2024-03-18T15:20:22.793 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86460 io_contex ceph txt unknown /proc/31759/task/86460/exe (readlink: Permission denied) 2024-03-18T15:20:22.793 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86460 io_contex ceph NOFD /proc/31759/task/86460/fd (opendir: Permission denied) 2024-03-18T15:20:22.793 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86461 safe_time ceph cwd unknown /proc/31759/task/86461/cwd (readlink: Permission denied) 2024-03-18T15:20:22.793 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86461 safe_time ceph rtd unknown /proc/31759/task/86461/root (readlink: Permission denied) 2024-03-18T15:20:22.793 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86461 safe_time ceph txt unknown /proc/31759/task/86461/exe (readlink: Permission denied) 2024-03-18T15:20:22.793 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86461 safe_time ceph NOFD /proc/31759/task/86461/fd (opendir: Permission denied) 2024-03-18T15:20:22.793 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86462 safe_time ceph cwd unknown /proc/31759/task/86462/cwd (readlink: Permission denied) 2024-03-18T15:20:22.794 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86462 safe_time ceph rtd unknown /proc/31759/task/86462/root (readlink: Permission denied) 2024-03-18T15:20:22.794 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86462 safe_time ceph txt unknown /proc/31759/task/86462/exe (readlink: Permission denied) 2024-03-18T15:20:22.794 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86462 safe_time ceph NOFD /proc/31759/task/86462/fd (opendir: Permission denied) 2024-03-18T15:20:22.794 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86463 safe_time ceph cwd unknown /proc/31759/task/86463/cwd (readlink: Permission denied) 2024-03-18T15:20:22.794 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86463 safe_time ceph rtd unknown /proc/31759/task/86463/root (readlink: Permission denied) 2024-03-18T15:20:22.794 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86463 safe_time ceph txt unknown /proc/31759/task/86463/exe (readlink: Permission denied) 2024-03-18T15:20:22.794 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86463 safe_time ceph NOFD /proc/31759/task/86463/fd (opendir: Permission denied) 2024-03-18T15:20:22.794 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86464 fn_anonym ceph cwd unknown /proc/31759/task/86464/cwd (readlink: Permission denied) 2024-03-18T15:20:22.794 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86464 fn_anonym ceph rtd unknown /proc/31759/task/86464/root (readlink: Permission denied) 2024-03-18T15:20:22.794 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86464 fn_anonym ceph txt unknown /proc/31759/task/86464/exe (readlink: Permission denied) 2024-03-18T15:20:22.794 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86464 fn_anonym ceph NOFD /proc/31759/task/86464/fd (opendir: Permission denied) 2024-03-18T15:20:22.794 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86465 flusher ceph cwd unknown /proc/31759/task/86465/cwd (readlink: Permission denied) 2024-03-18T15:20:22.794 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86465 flusher ceph rtd unknown /proc/31759/task/86465/root (readlink: Permission denied) 2024-03-18T15:20:22.794 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86465 flusher ceph txt unknown /proc/31759/task/86465/exe (readlink: Permission denied) 2024-03-18T15:20:22.795 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86465 flusher ceph NOFD /proc/31759/task/86465/fd (opendir: Permission denied) 2024-03-18T15:20:22.795 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86466 safe_time ceph cwd unknown /proc/31759/task/86466/cwd (readlink: Permission denied) 2024-03-18T15:20:22.795 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86466 safe_time ceph rtd unknown /proc/31759/task/86466/root (readlink: Permission denied) 2024-03-18T15:20:22.795 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86466 safe_time ceph txt unknown /proc/31759/task/86466/exe (readlink: Permission denied) 2024-03-18T15:20:22.795 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86466 safe_time ceph NOFD /proc/31759/task/86466/fd (opendir: Permission denied) 2024-03-18T15:20:22.795 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86629 mgr-fin ceph cwd unknown /proc/31759/task/86629/cwd (readlink: Permission denied) 2024-03-18T15:20:22.795 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86629 mgr-fin ceph rtd unknown /proc/31759/task/86629/root (readlink: Permission denied) 2024-03-18T15:20:22.795 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86629 mgr-fin ceph txt unknown /proc/31759/task/86629/exe (readlink: Permission denied) 2024-03-18T15:20:22.795 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86629 mgr-fin ceph NOFD /proc/31759/task/86629/fd (opendir: Permission denied) 2024-03-18T15:20:22.795 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86630 ms_dispat ceph cwd unknown /proc/31759/task/86630/cwd (readlink: Permission denied) 2024-03-18T15:20:22.795 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86630 ms_dispat ceph rtd unknown /proc/31759/task/86630/root (readlink: Permission denied) 2024-03-18T15:20:22.795 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86630 ms_dispat ceph txt unknown /proc/31759/task/86630/exe (readlink: Permission denied) 2024-03-18T15:20:22.795 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86630 ms_dispat ceph NOFD /proc/31759/task/86630/fd (opendir: Permission denied) 2024-03-18T15:20:22.795 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86631 ms_local ceph cwd unknown /proc/31759/task/86631/cwd (readlink: Permission denied) 2024-03-18T15:20:22.796 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86631 ms_local ceph rtd unknown /proc/31759/task/86631/root (readlink: Permission denied) 2024-03-18T15:20:22.796 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86631 ms_local ceph txt unknown /proc/31759/task/86631/exe (readlink: Permission denied) 2024-03-18T15:20:22.796 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86631 ms_local ceph NOFD /proc/31759/task/86631/fd (opendir: Permission denied) 2024-03-18T15:20:22.796 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86632 safe_time ceph cwd unknown /proc/31759/task/86632/cwd (readlink: Permission denied) 2024-03-18T15:20:22.796 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86632 safe_time ceph rtd unknown /proc/31759/task/86632/root (readlink: Permission denied) 2024-03-18T15:20:22.796 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86632 safe_time ceph txt unknown /proc/31759/task/86632/exe (readlink: Permission denied) 2024-03-18T15:20:22.796 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86632 safe_time ceph NOFD /proc/31759/task/86632/fd (opendir: Permission denied) 2024-03-18T15:20:22.796 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86633 cmdfin ceph cwd unknown /proc/31759/task/86633/cwd (readlink: Permission denied) 2024-03-18T15:20:22.796 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86633 cmdfin ceph rtd unknown /proc/31759/task/86633/root (readlink: Permission denied) 2024-03-18T15:20:22.796 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86633 cmdfin ceph txt unknown /proc/31759/task/86633/exe (readlink: Permission denied) 2024-03-18T15:20:22.796 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86633 cmdfin ceph NOFD /proc/31759/task/86633/fd (opendir: Permission denied) 2024-03-18T15:20:22.796 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86637 io_contex ceph cwd unknown /proc/31759/task/86637/cwd (readlink: Permission denied) 2024-03-18T15:20:22.796 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86637 io_contex ceph rtd unknown /proc/31759/task/86637/root (readlink: Permission denied) 2024-03-18T15:20:22.796 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86637 io_contex ceph txt unknown /proc/31759/task/86637/exe (readlink: Permission denied) 2024-03-18T15:20:22.796 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86637 io_contex ceph NOFD /proc/31759/task/86637/fd (opendir: Permission denied) 2024-03-18T15:20:22.797 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86638 io_contex ceph cwd unknown /proc/31759/task/86638/cwd (readlink: Permission denied) 2024-03-18T15:20:22.797 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86638 io_contex ceph rtd unknown /proc/31759/task/86638/root (readlink: Permission denied) 2024-03-18T15:20:22.797 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86638 io_contex ceph txt unknown /proc/31759/task/86638/exe (readlink: Permission denied) 2024-03-18T15:20:22.797 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86638 io_contex ceph NOFD /proc/31759/task/86638/fd (opendir: Permission denied) 2024-03-18T15:20:22.797 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86639 ceph_time ceph cwd unknown /proc/31759/task/86639/cwd (readlink: Permission denied) 2024-03-18T15:20:22.797 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86639 ceph_time ceph rtd unknown /proc/31759/task/86639/root (readlink: Permission denied) 2024-03-18T15:20:22.797 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86639 ceph_time ceph txt unknown /proc/31759/task/86639/exe (readlink: Permission denied) 2024-03-18T15:20:22.797 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86639 ceph_time ceph NOFD /proc/31759/task/86639/fd (opendir: Permission denied) 2024-03-18T15:20:22.797 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86640 ms_dispat ceph cwd unknown /proc/31759/task/86640/cwd (readlink: Permission denied) 2024-03-18T15:20:22.797 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86640 ms_dispat ceph rtd unknown /proc/31759/task/86640/root (readlink: Permission denied) 2024-03-18T15:20:22.797 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86640 ms_dispat ceph txt unknown /proc/31759/task/86640/exe (readlink: Permission denied) 2024-03-18T15:20:22.797 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86640 ms_dispat ceph NOFD /proc/31759/task/86640/fd (opendir: Permission denied) 2024-03-18T15:20:22.797 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86641 ms_local ceph cwd unknown /proc/31759/task/86641/cwd (readlink: Permission denied) 2024-03-18T15:20:22.797 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86641 ms_local ceph rtd unknown /proc/31759/task/86641/root (readlink: Permission denied) 2024-03-18T15:20:22.798 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86641 ms_local ceph txt unknown /proc/31759/task/86641/exe (readlink: Permission denied) 2024-03-18T15:20:22.798 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86641 ms_local ceph NOFD /proc/31759/task/86641/fd (opendir: Permission denied) 2024-03-18T15:20:22.798 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86642 safe_time ceph cwd unknown /proc/31759/task/86642/cwd (readlink: Permission denied) 2024-03-18T15:20:22.798 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86642 safe_time ceph rtd unknown /proc/31759/task/86642/root (readlink: Permission denied) 2024-03-18T15:20:22.798 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86642 safe_time ceph txt unknown /proc/31759/task/86642/exe (readlink: Permission denied) 2024-03-18T15:20:22.798 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86642 safe_time ceph NOFD /proc/31759/task/86642/fd (opendir: Permission denied) 2024-03-18T15:20:22.798 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86643 safe_time ceph cwd unknown /proc/31759/task/86643/cwd (readlink: Permission denied) 2024-03-18T15:20:22.798 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86643 safe_time ceph rtd unknown /proc/31759/task/86643/root (readlink: Permission denied) 2024-03-18T15:20:22.798 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86643 safe_time ceph txt unknown /proc/31759/task/86643/exe (readlink: Permission denied) 2024-03-18T15:20:22.798 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86643 safe_time ceph NOFD /proc/31759/task/86643/fd (opendir: Permission denied) 2024-03-18T15:20:22.798 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86644 balancer ceph cwd unknown /proc/31759/task/86644/cwd (readlink: Permission denied) 2024-03-18T15:20:22.798 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86644 balancer ceph rtd unknown /proc/31759/task/86644/root (readlink: Permission denied) 2024-03-18T15:20:22.798 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86644 balancer ceph txt unknown /proc/31759/task/86644/exe (readlink: Permission denied) 2024-03-18T15:20:22.798 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86644 balancer ceph NOFD /proc/31759/task/86644/fd (opendir: Permission denied) 2024-03-18T15:20:22.799 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86645 m-fin-bal ceph cwd unknown /proc/31759/task/86645/cwd (readlink: Permission denied) 2024-03-18T15:20:22.799 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86645 m-fin-bal ceph rtd unknown /proc/31759/task/86645/root (readlink: Permission denied) 2024-03-18T15:20:22.799 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86645 m-fin-bal ceph txt unknown /proc/31759/task/86645/exe (readlink: Permission denied) 2024-03-18T15:20:22.799 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86645 m-fin-bal ceph NOFD /proc/31759/task/86645/fd (opendir: Permission denied) 2024-03-18T15:20:22.799 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86646 mgr-fin ceph cwd unknown /proc/31759/task/86646/cwd (readlink: Permission denied) 2024-03-18T15:20:22.799 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86646 mgr-fin ceph rtd unknown /proc/31759/task/86646/root (readlink: Permission denied) 2024-03-18T15:20:22.799 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86646 mgr-fin ceph txt unknown /proc/31759/task/86646/exe (readlink: Permission denied) 2024-03-18T15:20:22.799 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86646 mgr-fin ceph NOFD /proc/31759/task/86646/fd (opendir: Permission denied) 2024-03-18T15:20:22.799 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86647 mgr-fin ceph cwd unknown /proc/31759/task/86647/cwd (readlink: Permission denied) 2024-03-18T15:20:22.799 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86647 mgr-fin ceph rtd unknown /proc/31759/task/86647/root (readlink: Permission denied) 2024-03-18T15:20:22.799 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86647 mgr-fin ceph txt unknown /proc/31759/task/86647/exe (readlink: Permission denied) 2024-03-18T15:20:22.799 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86647 mgr-fin ceph NOFD /proc/31759/task/86647/fd (opendir: Permission denied) 2024-03-18T15:20:22.799 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86648 mgr-fin ceph cwd unknown /proc/31759/task/86648/cwd (readlink: Permission denied) 2024-03-18T15:20:22.799 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86648 mgr-fin ceph rtd unknown /proc/31759/task/86648/root (readlink: Permission denied) 2024-03-18T15:20:22.799 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86648 mgr-fin ceph txt unknown /proc/31759/task/86648/exe (readlink: Permission denied) 2024-03-18T15:20:22.800 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86648 mgr-fin ceph NOFD /proc/31759/task/86648/fd (opendir: Permission denied) 2024-03-18T15:20:22.800 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86649 mgr-fin ceph cwd unknown /proc/31759/task/86649/cwd (readlink: Permission denied) 2024-03-18T15:20:22.800 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86649 mgr-fin ceph rtd unknown /proc/31759/task/86649/root (readlink: Permission denied) 2024-03-18T15:20:22.800 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86649 mgr-fin ceph txt unknown /proc/31759/task/86649/exe (readlink: Permission denied) 2024-03-18T15:20:22.800 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86649 mgr-fin ceph NOFD /proc/31759/task/86649/fd (opendir: Permission denied) 2024-03-18T15:20:22.800 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86650 mgr-fin ceph cwd unknown /proc/31759/task/86650/cwd (readlink: Permission denied) 2024-03-18T15:20:22.800 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86650 mgr-fin ceph rtd unknown /proc/31759/task/86650/root (readlink: Permission denied) 2024-03-18T15:20:22.800 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86650 mgr-fin ceph txt unknown /proc/31759/task/86650/exe (readlink: Permission denied) 2024-03-18T15:20:22.800 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86650 mgr-fin ceph NOFD /proc/31759/task/86650/fd (opendir: Permission denied) 2024-03-18T15:20:22.800 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86651 mgr-fin ceph cwd unknown /proc/31759/task/86651/cwd (readlink: Permission denied) 2024-03-18T15:20:22.800 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86651 mgr-fin ceph rtd unknown /proc/31759/task/86651/root (readlink: Permission denied) 2024-03-18T15:20:22.800 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86651 mgr-fin ceph txt unknown /proc/31759/task/86651/exe (readlink: Permission denied) 2024-03-18T15:20:22.800 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86651 mgr-fin ceph NOFD /proc/31759/task/86651/fd (opendir: Permission denied) 2024-03-18T15:20:22.800 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86652 mgr-fin ceph cwd unknown /proc/31759/task/86652/cwd (readlink: Permission denied) 2024-03-18T15:20:22.800 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86652 mgr-fin ceph rtd unknown /proc/31759/task/86652/root (readlink: Permission denied) 2024-03-18T15:20:22.801 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86652 mgr-fin ceph txt unknown /proc/31759/task/86652/exe (readlink: Permission denied) 2024-03-18T15:20:22.801 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86652 mgr-fin ceph NOFD /proc/31759/task/86652/fd (opendir: Permission denied) 2024-03-18T15:20:22.801 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86653 mgr-fin ceph cwd unknown /proc/31759/task/86653/cwd (readlink: Permission denied) 2024-03-18T15:20:22.801 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86653 mgr-fin ceph rtd unknown /proc/31759/task/86653/root (readlink: Permission denied) 2024-03-18T15:20:22.801 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86653 mgr-fin ceph txt unknown /proc/31759/task/86653/exe (readlink: Permission denied) 2024-03-18T15:20:22.801 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86653 mgr-fin ceph NOFD /proc/31759/task/86653/fd (opendir: Permission denied) 2024-03-18T15:20:22.801 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86654 mgr-fin ceph cwd unknown /proc/31759/task/86654/cwd (readlink: Permission denied) 2024-03-18T15:20:22.801 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86654 mgr-fin ceph rtd unknown /proc/31759/task/86654/root (readlink: Permission denied) 2024-03-18T15:20:22.801 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86654 mgr-fin ceph txt unknown /proc/31759/task/86654/exe (readlink: Permission denied) 2024-03-18T15:20:22.801 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86654 mgr-fin ceph NOFD /proc/31759/task/86654/fd (opendir: Permission denied) 2024-03-18T15:20:22.801 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86655 mgr-fin ceph cwd unknown /proc/31759/task/86655/cwd (readlink: Permission denied) 2024-03-18T15:20:22.801 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86655 mgr-fin ceph rtd unknown /proc/31759/task/86655/root (readlink: Permission denied) 2024-03-18T15:20:22.801 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86655 mgr-fin ceph txt unknown /proc/31759/task/86655/exe (readlink: Permission denied) 2024-03-18T15:20:22.801 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86655 mgr-fin ceph NOFD /proc/31759/task/86655/fd (opendir: Permission denied) 2024-03-18T15:20:22.801 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86656 mgr-fin ceph cwd unknown /proc/31759/task/86656/cwd (readlink: Permission denied) 2024-03-18T15:20:22.802 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86656 mgr-fin ceph rtd unknown /proc/31759/task/86656/root (readlink: Permission denied) 2024-03-18T15:20:22.802 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86656 mgr-fin ceph txt unknown /proc/31759/task/86656/exe (readlink: Permission denied) 2024-03-18T15:20:22.802 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86656 mgr-fin ceph NOFD /proc/31759/task/86656/fd (opendir: Permission denied) 2024-03-18T15:20:22.802 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86657 mgr-fin ceph cwd unknown /proc/31759/task/86657/cwd (readlink: Permission denied) 2024-03-18T15:20:22.802 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86657 mgr-fin ceph rtd unknown /proc/31759/task/86657/root (readlink: Permission denied) 2024-03-18T15:20:22.802 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86657 mgr-fin ceph txt unknown /proc/31759/task/86657/exe (readlink: Permission denied) 2024-03-18T15:20:22.802 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86657 mgr-fin ceph NOFD /proc/31759/task/86657/fd (opendir: Permission denied) 2024-03-18T15:20:22.802 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86658 mgr-fin ceph cwd unknown /proc/31759/task/86658/cwd (readlink: Permission denied) 2024-03-18T15:20:22.802 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86658 mgr-fin ceph rtd unknown /proc/31759/task/86658/root (readlink: Permission denied) 2024-03-18T15:20:22.802 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86658 mgr-fin ceph txt unknown /proc/31759/task/86658/exe (readlink: Permission denied) 2024-03-18T15:20:22.802 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86658 mgr-fin ceph NOFD /proc/31759/task/86658/fd (opendir: Permission denied) 2024-03-18T15:20:22.802 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86659 mgr-fin ceph cwd unknown /proc/31759/task/86659/cwd (readlink: Permission denied) 2024-03-18T15:20:22.802 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86659 mgr-fin ceph rtd unknown /proc/31759/task/86659/root (readlink: Permission denied) 2024-03-18T15:20:22.803 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86659 mgr-fin ceph txt unknown /proc/31759/task/86659/exe (readlink: Permission denied) 2024-03-18T15:20:22.803 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86659 mgr-fin ceph NOFD /proc/31759/task/86659/fd (opendir: Permission denied) 2024-03-18T15:20:22.803 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86660 mgr-fin ceph cwd unknown /proc/31759/task/86660/cwd (readlink: Permission denied) 2024-03-18T15:20:22.803 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86660 mgr-fin ceph rtd unknown /proc/31759/task/86660/root (readlink: Permission denied) 2024-03-18T15:20:22.803 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86660 mgr-fin ceph txt unknown /proc/31759/task/86660/exe (readlink: Permission denied) 2024-03-18T15:20:22.803 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86660 mgr-fin ceph NOFD /proc/31759/task/86660/fd (opendir: Permission denied) 2024-03-18T15:20:22.803 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86661 mgr-fin ceph cwd unknown /proc/31759/task/86661/cwd (readlink: Permission denied) 2024-03-18T15:20:22.803 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86661 mgr-fin ceph rtd unknown /proc/31759/task/86661/root (readlink: Permission denied) 2024-03-18T15:20:22.803 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86661 mgr-fin ceph txt unknown /proc/31759/task/86661/exe (readlink: Permission denied) 2024-03-18T15:20:22.803 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86661 mgr-fin ceph NOFD /proc/31759/task/86661/fd (opendir: Permission denied) 2024-03-18T15:20:22.803 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86662 cephadm ceph cwd unknown /proc/31759/task/86662/cwd (readlink: Permission denied) 2024-03-18T15:20:22.803 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86662 cephadm ceph rtd unknown /proc/31759/task/86662/root (readlink: Permission denied) 2024-03-18T15:20:22.803 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86662 cephadm ceph txt unknown /proc/31759/task/86662/exe (readlink: Permission denied) 2024-03-18T15:20:22.804 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86662 cephadm ceph NOFD /proc/31759/task/86662/fd (opendir: Permission denied) 2024-03-18T15:20:22.804 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86663 mgr-fin ceph cwd unknown /proc/31759/task/86663/cwd (readlink: Permission denied) 2024-03-18T15:20:22.804 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86663 mgr-fin ceph rtd unknown /proc/31759/task/86663/root (readlink: Permission denied) 2024-03-18T15:20:22.804 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86663 mgr-fin ceph txt unknown /proc/31759/task/86663/exe (readlink: Permission denied) 2024-03-18T15:20:22.804 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86663 mgr-fin ceph NOFD /proc/31759/task/86663/fd (opendir: Permission denied) 2024-03-18T15:20:22.804 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86664 m-fin-cep ceph cwd unknown /proc/31759/task/86664/cwd (readlink: Permission denied) 2024-03-18T15:20:22.804 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86664 m-fin-cep ceph rtd unknown /proc/31759/task/86664/root (readlink: Permission denied) 2024-03-18T15:20:22.804 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86664 m-fin-cep ceph txt unknown /proc/31759/task/86664/exe (readlink: Permission denied) 2024-03-18T15:20:22.804 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86664 m-fin-cep ceph NOFD /proc/31759/task/86664/fd (opendir: Permission denied) 2024-03-18T15:20:22.804 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86665 mgr-fin ceph cwd unknown /proc/31759/task/86665/cwd (readlink: Permission denied) 2024-03-18T15:20:22.804 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86665 mgr-fin ceph rtd unknown /proc/31759/task/86665/root (readlink: Permission denied) 2024-03-18T15:20:22.804 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86665 mgr-fin ceph txt unknown /proc/31759/task/86665/exe (readlink: Permission denied) 2024-03-18T15:20:22.804 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86665 mgr-fin ceph NOFD /proc/31759/task/86665/fd (opendir: Permission denied) 2024-03-18T15:20:22.804 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86666 mgr-fin ceph cwd unknown /proc/31759/task/86666/cwd (readlink: Permission denied) 2024-03-18T15:20:22.804 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86666 mgr-fin ceph rtd unknown /proc/31759/task/86666/root (readlink: Permission denied) 2024-03-18T15:20:22.805 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86666 mgr-fin ceph txt unknown /proc/31759/task/86666/exe (readlink: Permission denied) 2024-03-18T15:20:22.805 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86666 mgr-fin ceph NOFD /proc/31759/task/86666/fd (opendir: Permission denied) 2024-03-18T15:20:22.805 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86667 cephadm ceph cwd unknown /proc/31759/task/86667/cwd (readlink: Permission denied) 2024-03-18T15:20:22.805 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86667 cephadm ceph rtd unknown /proc/31759/task/86667/root (readlink: Permission denied) 2024-03-18T15:20:22.805 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86667 cephadm ceph txt unknown /proc/31759/task/86667/exe (readlink: Permission denied) 2024-03-18T15:20:22.805 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86667 cephadm ceph NOFD /proc/31759/task/86667/fd (opendir: Permission denied) 2024-03-18T15:20:22.805 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86668 mgr-fin ceph cwd unknown /proc/31759/task/86668/cwd (readlink: Permission denied) 2024-03-18T15:20:22.805 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86668 mgr-fin ceph rtd unknown /proc/31759/task/86668/root (readlink: Permission denied) 2024-03-18T15:20:22.805 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86668 mgr-fin ceph txt unknown /proc/31759/task/86668/exe (readlink: Permission denied) 2024-03-18T15:20:22.805 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86668 mgr-fin ceph NOFD /proc/31759/task/86668/fd (opendir: Permission denied) 2024-03-18T15:20:22.805 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86669 crash ceph cwd unknown /proc/31759/task/86669/cwd (readlink: Permission denied) 2024-03-18T15:20:22.805 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86669 crash ceph rtd unknown /proc/31759/task/86669/root (readlink: Permission denied) 2024-03-18T15:20:22.805 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86669 crash ceph txt unknown /proc/31759/task/86669/exe (readlink: Permission denied) 2024-03-18T15:20:22.805 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86669 crash ceph NOFD /proc/31759/task/86669/fd (opendir: Permission denied) 2024-03-18T15:20:22.806 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86670 m-fin-cra ceph cwd unknown /proc/31759/task/86670/cwd (readlink: Permission denied) 2024-03-18T15:20:22.806 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86670 m-fin-cra ceph rtd unknown /proc/31759/task/86670/root (readlink: Permission denied) 2024-03-18T15:20:22.806 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86670 m-fin-cra ceph txt unknown /proc/31759/task/86670/exe (readlink: Permission denied) 2024-03-18T15:20:22.806 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86670 m-fin-cra ceph NOFD /proc/31759/task/86670/fd (opendir: Permission denied) 2024-03-18T15:20:22.806 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86671 mgr-fin ceph cwd unknown /proc/31759/task/86671/cwd (readlink: Permission denied) 2024-03-18T15:20:22.806 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86671 mgr-fin ceph rtd unknown /proc/31759/task/86671/root (readlink: Permission denied) 2024-03-18T15:20:22.806 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86671 mgr-fin ceph txt unknown /proc/31759/task/86671/exe (readlink: Permission denied) 2024-03-18T15:20:22.806 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86671 mgr-fin ceph NOFD /proc/31759/task/86671/fd (opendir: Permission denied) 2024-03-18T15:20:22.806 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86672 dashboard ceph cwd unknown /proc/31759/task/86672/cwd (readlink: Permission denied) 2024-03-18T15:20:22.806 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86672 dashboard ceph rtd unknown /proc/31759/task/86672/root (readlink: Permission denied) 2024-03-18T15:20:22.806 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86672 dashboard ceph txt unknown /proc/31759/task/86672/exe (readlink: Permission denied) 2024-03-18T15:20:22.806 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86672 dashboard ceph NOFD /proc/31759/task/86672/fd (opendir: Permission denied) 2024-03-18T15:20:22.806 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86673 m-fin-das ceph cwd unknown /proc/31759/task/86673/cwd (readlink: Permission denied) 2024-03-18T15:20:22.806 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86673 m-fin-das ceph rtd unknown /proc/31759/task/86673/root (readlink: Permission denied) 2024-03-18T15:20:22.806 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86673 m-fin-das ceph txt unknown /proc/31759/task/86673/exe (readlink: Permission denied) 2024-03-18T15:20:22.807 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86673 m-fin-das ceph NOFD /proc/31759/task/86673/fd (opendir: Permission denied) 2024-03-18T15:20:22.807 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86674 mgr-fin ceph cwd unknown /proc/31759/task/86674/cwd (readlink: Permission denied) 2024-03-18T15:20:22.807 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86674 mgr-fin ceph rtd unknown /proc/31759/task/86674/root (readlink: Permission denied) 2024-03-18T15:20:22.807 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86674 mgr-fin ceph txt unknown /proc/31759/task/86674/exe (readlink: Permission denied) 2024-03-18T15:20:22.807 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86674 mgr-fin ceph NOFD /proc/31759/task/86674/fd (opendir: Permission denied) 2024-03-18T15:20:22.807 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86675 mgr-fin ceph cwd unknown /proc/31759/task/86675/cwd (readlink: Permission denied) 2024-03-18T15:20:22.807 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86675 mgr-fin ceph rtd unknown /proc/31759/task/86675/root (readlink: Permission denied) 2024-03-18T15:20:22.807 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86675 mgr-fin ceph txt unknown /proc/31759/task/86675/exe (readlink: Permission denied) 2024-03-18T15:20:22.807 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86675 mgr-fin ceph NOFD /proc/31759/task/86675/fd (opendir: Permission denied) 2024-03-18T15:20:22.807 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86676 devicehea ceph cwd unknown /proc/31759/task/86676/cwd (readlink: Permission denied) 2024-03-18T15:20:22.807 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86676 devicehea ceph rtd unknown /proc/31759/task/86676/root (readlink: Permission denied) 2024-03-18T15:20:22.807 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86676 devicehea ceph txt unknown /proc/31759/task/86676/exe (readlink: Permission denied) 2024-03-18T15:20:22.807 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86676 devicehea ceph NOFD /proc/31759/task/86676/fd (opendir: Permission denied) 2024-03-18T15:20:22.807 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86677 m-fin-dev ceph cwd unknown /proc/31759/task/86677/cwd (readlink: Permission denied) 2024-03-18T15:20:22.808 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86677 m-fin-dev ceph rtd unknown /proc/31759/task/86677/root (readlink: Permission denied) 2024-03-18T15:20:22.808 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86677 m-fin-dev ceph txt unknown /proc/31759/task/86677/exe (readlink: Permission denied) 2024-03-18T15:20:22.808 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86677 m-fin-dev ceph NOFD /proc/31759/task/86677/fd (opendir: Permission denied) 2024-03-18T15:20:22.808 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86678 mgr-fin ceph cwd unknown /proc/31759/task/86678/cwd (readlink: Permission denied) 2024-03-18T15:20:22.808 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86678 mgr-fin ceph rtd unknown /proc/31759/task/86678/root (readlink: Permission denied) 2024-03-18T15:20:22.808 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86678 mgr-fin ceph txt unknown /proc/31759/task/86678/exe (readlink: Permission denied) 2024-03-18T15:20:22.808 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86678 mgr-fin ceph NOFD /proc/31759/task/86678/fd (opendir: Permission denied) 2024-03-18T15:20:22.808 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86679 mgr-fin ceph cwd unknown /proc/31759/task/86679/cwd (readlink: Permission denied) 2024-03-18T15:20:22.808 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86679 mgr-fin ceph rtd unknown /proc/31759/task/86679/root (readlink: Permission denied) 2024-03-18T15:20:22.808 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86679 mgr-fin ceph txt unknown /proc/31759/task/86679/exe (readlink: Permission denied) 2024-03-18T15:20:22.808 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86679 mgr-fin ceph NOFD /proc/31759/task/86679/fd (opendir: Permission denied) 2024-03-18T15:20:22.808 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86680 mgr-fin ceph cwd unknown /proc/31759/task/86680/cwd (readlink: Permission denied) 2024-03-18T15:20:22.808 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86680 mgr-fin ceph rtd unknown /proc/31759/task/86680/root (readlink: Permission denied) 2024-03-18T15:20:22.808 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86680 mgr-fin ceph txt unknown /proc/31759/task/86680/exe (readlink: Permission denied) 2024-03-18T15:20:22.808 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86680 mgr-fin ceph NOFD /proc/31759/task/86680/fd (opendir: Permission denied) 2024-03-18T15:20:22.809 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86682 m-fin-ios ceph cwd unknown /proc/31759/task/86682/cwd (readlink: Permission denied) 2024-03-18T15:20:22.809 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86682 m-fin-ios ceph rtd unknown /proc/31759/task/86682/root (readlink: Permission denied) 2024-03-18T15:20:22.809 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86682 m-fin-ios ceph txt unknown /proc/31759/task/86682/exe (readlink: Permission denied) 2024-03-18T15:20:22.809 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86682 m-fin-ios ceph NOFD /proc/31759/task/86682/fd (opendir: Permission denied) 2024-03-18T15:20:22.809 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86683 mgr-fin ceph cwd unknown /proc/31759/task/86683/cwd (readlink: Permission denied) 2024-03-18T15:20:22.809 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86683 mgr-fin ceph rtd unknown /proc/31759/task/86683/root (readlink: Permission denied) 2024-03-18T15:20:22.809 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86683 mgr-fin ceph txt unknown /proc/31759/task/86683/exe (readlink: Permission denied) 2024-03-18T15:20:22.809 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86683 mgr-fin ceph NOFD /proc/31759/task/86683/fd (opendir: Permission denied) 2024-03-18T15:20:22.809 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86684 mgr-fin ceph cwd unknown /proc/31759/task/86684/cwd (readlink: Permission denied) 2024-03-18T15:20:22.809 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86684 mgr-fin ceph rtd unknown /proc/31759/task/86684/root (readlink: Permission denied) 2024-03-18T15:20:22.809 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86684 mgr-fin ceph txt unknown /proc/31759/task/86684/exe (readlink: Permission denied) 2024-03-18T15:20:22.809 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86684 mgr-fin ceph NOFD /proc/31759/task/86684/fd (opendir: Permission denied) 2024-03-18T15:20:22.809 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86686 m-fin-nfs ceph cwd unknown /proc/31759/task/86686/cwd (readlink: Permission denied) 2024-03-18T15:20:22.809 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86686 m-fin-nfs ceph rtd unknown /proc/31759/task/86686/root (readlink: Permission denied) 2024-03-18T15:20:22.809 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86686 m-fin-nfs ceph txt unknown /proc/31759/task/86686/exe (readlink: Permission denied) 2024-03-18T15:20:22.810 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86686 m-fin-nfs ceph NOFD /proc/31759/task/86686/fd (opendir: Permission denied) 2024-03-18T15:20:22.810 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86688 m-fin-orc ceph cwd unknown /proc/31759/task/86688/cwd (readlink: Permission denied) 2024-03-18T15:20:22.810 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86688 m-fin-orc ceph rtd unknown /proc/31759/task/86688/root (readlink: Permission denied) 2024-03-18T15:20:22.810 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86688 m-fin-orc ceph txt unknown /proc/31759/task/86688/exe (readlink: Permission denied) 2024-03-18T15:20:22.810 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86688 m-fin-orc ceph NOFD /proc/31759/task/86688/fd (opendir: Permission denied) 2024-03-18T15:20:22.810 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86689 pg_autosc ceph cwd unknown /proc/31759/task/86689/cwd (readlink: Permission denied) 2024-03-18T15:20:22.810 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86689 pg_autosc ceph rtd unknown /proc/31759/task/86689/root (readlink: Permission denied) 2024-03-18T15:20:22.810 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86689 pg_autosc ceph txt unknown /proc/31759/task/86689/exe (readlink: Permission denied) 2024-03-18T15:20:22.810 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86689 pg_autosc ceph NOFD /proc/31759/task/86689/fd (opendir: Permission denied) 2024-03-18T15:20:22.810 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86690 m-fin-pg_ ceph cwd unknown /proc/31759/task/86690/cwd (readlink: Permission denied) 2024-03-18T15:20:22.810 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86690 m-fin-pg_ ceph rtd unknown /proc/31759/task/86690/root (readlink: Permission denied) 2024-03-18T15:20:22.810 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86690 m-fin-pg_ ceph txt unknown /proc/31759/task/86690/exe (readlink: Permission denied) 2024-03-18T15:20:22.810 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86690 m-fin-pg_ ceph NOFD /proc/31759/task/86690/fd (opendir: Permission denied) 2024-03-18T15:20:22.811 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86691 progress ceph cwd unknown /proc/31759/task/86691/cwd (readlink: Permission denied) 2024-03-18T15:20:22.811 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86691 progress ceph rtd unknown /proc/31759/task/86691/root (readlink: Permission denied) 2024-03-18T15:20:22.811 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86691 progress ceph txt unknown /proc/31759/task/86691/exe (readlink: Permission denied) 2024-03-18T15:20:22.811 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86691 progress ceph NOFD /proc/31759/task/86691/fd (opendir: Permission denied) 2024-03-18T15:20:22.811 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86692 m-fin-pro ceph cwd unknown /proc/31759/task/86692/cwd (readlink: Permission denied) 2024-03-18T15:20:22.811 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86692 m-fin-pro ceph rtd unknown /proc/31759/task/86692/root (readlink: Permission denied) 2024-03-18T15:20:22.811 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86692 m-fin-pro ceph txt unknown /proc/31759/task/86692/exe (readlink: Permission denied) 2024-03-18T15:20:22.811 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86692 m-fin-pro ceph NOFD /proc/31759/task/86692/fd (opendir: Permission denied) 2024-03-18T15:20:22.811 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86696 io_contex ceph cwd unknown /proc/31759/task/86696/cwd (readlink: Permission denied) 2024-03-18T15:20:22.811 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86696 io_contex ceph rtd unknown /proc/31759/task/86696/root (readlink: Permission denied) 2024-03-18T15:20:22.811 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86696 io_contex ceph txt unknown /proc/31759/task/86696/exe (readlink: Permission denied) 2024-03-18T15:20:22.811 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86696 io_contex ceph NOFD /proc/31759/task/86696/fd (opendir: Permission denied) 2024-03-18T15:20:22.811 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86697 io_contex ceph cwd unknown /proc/31759/task/86697/cwd (readlink: Permission denied) 2024-03-18T15:20:22.811 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86697 io_contex ceph rtd unknown /proc/31759/task/86697/root (readlink: Permission denied) 2024-03-18T15:20:22.812 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86697 io_contex ceph txt unknown /proc/31759/task/86697/exe (readlink: Permission denied) 2024-03-18T15:20:22.812 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86697 io_contex ceph NOFD /proc/31759/task/86697/fd (opendir: Permission denied) 2024-03-18T15:20:22.812 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86698 ceph_time ceph cwd unknown /proc/31759/task/86698/cwd (readlink: Permission denied) 2024-03-18T15:20:22.812 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86698 ceph_time ceph rtd unknown /proc/31759/task/86698/root (readlink: Permission denied) 2024-03-18T15:20:22.812 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86698 ceph_time ceph txt unknown /proc/31759/task/86698/exe (readlink: Permission denied) 2024-03-18T15:20:22.812 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86698 ceph_time ceph NOFD /proc/31759/task/86698/fd (opendir: Permission denied) 2024-03-18T15:20:22.812 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86699 ms_dispat ceph cwd unknown /proc/31759/task/86699/cwd (readlink: Permission denied) 2024-03-18T15:20:22.812 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86699 ms_dispat ceph rtd unknown /proc/31759/task/86699/root (readlink: Permission denied) 2024-03-18T15:20:22.812 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86699 ms_dispat ceph txt unknown /proc/31759/task/86699/exe (readlink: Permission denied) 2024-03-18T15:20:22.812 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86699 ms_dispat ceph NOFD /proc/31759/task/86699/fd (opendir: Permission denied) 2024-03-18T15:20:22.812 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86700 ms_local ceph cwd unknown /proc/31759/task/86700/cwd (readlink: Permission denied) 2024-03-18T15:20:22.812 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86700 ms_local ceph rtd unknown /proc/31759/task/86700/root (readlink: Permission denied) 2024-03-18T15:20:22.812 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86700 ms_local ceph txt unknown /proc/31759/task/86700/exe (readlink: Permission denied) 2024-03-18T15:20:22.813 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86700 ms_local ceph NOFD /proc/31759/task/86700/fd (opendir: Permission denied) 2024-03-18T15:20:22.813 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86701 safe_time ceph cwd unknown /proc/31759/task/86701/cwd (readlink: Permission denied) 2024-03-18T15:20:22.813 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86701 safe_time ceph rtd unknown /proc/31759/task/86701/root (readlink: Permission denied) 2024-03-18T15:20:22.813 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86701 safe_time ceph txt unknown /proc/31759/task/86701/exe (readlink: Permission denied) 2024-03-18T15:20:22.813 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86701 safe_time ceph NOFD /proc/31759/task/86701/fd (opendir: Permission denied) 2024-03-18T15:20:22.813 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86702 safe_time ceph cwd unknown /proc/31759/task/86702/cwd (readlink: Permission denied) 2024-03-18T15:20:22.813 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86702 safe_time ceph rtd unknown /proc/31759/task/86702/root (readlink: Permission denied) 2024-03-18T15:20:22.813 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86702 safe_time ceph txt unknown /proc/31759/task/86702/exe (readlink: Permission denied) 2024-03-18T15:20:22.813 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86702 safe_time ceph NOFD /proc/31759/task/86702/fd (opendir: Permission denied) 2024-03-18T15:20:22.813 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86703 mgr-fin ceph cwd unknown /proc/31759/task/86703/cwd (readlink: Permission denied) 2024-03-18T15:20:22.813 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86703 mgr-fin ceph rtd unknown /proc/31759/task/86703/root (readlink: Permission denied) 2024-03-18T15:20:22.814 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86703 mgr-fin ceph txt unknown /proc/31759/task/86703/exe (readlink: Permission denied) 2024-03-18T15:20:22.814 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86703 mgr-fin ceph NOFD /proc/31759/task/86703/fd (opendir: Permission denied) 2024-03-18T15:20:22.814 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86705 m-fin-rbd ceph cwd unknown /proc/31759/task/86705/cwd (readlink: Permission denied) 2024-03-18T15:20:22.814 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86705 m-fin-rbd ceph rtd unknown /proc/31759/task/86705/root (readlink: Permission denied) 2024-03-18T15:20:22.814 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86705 m-fin-rbd ceph txt unknown /proc/31759/task/86705/exe (readlink: Permission denied) 2024-03-18T15:20:22.814 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86705 m-fin-rbd ceph NOFD /proc/31759/task/86705/fd (opendir: Permission denied) 2024-03-18T15:20:22.814 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86706 restful ceph cwd unknown /proc/31759/task/86706/cwd (readlink: Permission denied) 2024-03-18T15:20:22.814 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86706 restful ceph rtd unknown /proc/31759/task/86706/root (readlink: Permission denied) 2024-03-18T15:20:22.814 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86706 restful ceph txt unknown /proc/31759/task/86706/exe (readlink: Permission denied) 2024-03-18T15:20:22.814 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86706 restful ceph NOFD /proc/31759/task/86706/fd (opendir: Permission denied) 2024-03-18T15:20:22.814 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86707 m-fin-res ceph cwd unknown /proc/31759/task/86707/cwd (readlink: Permission denied) 2024-03-18T15:20:22.814 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86707 m-fin-res ceph rtd unknown /proc/31759/task/86707/root (readlink: Permission denied) 2024-03-18T15:20:22.814 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86707 m-fin-res ceph txt unknown /proc/31759/task/86707/exe (readlink: Permission denied) 2024-03-18T15:20:22.814 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86707 m-fin-res ceph NOFD /proc/31759/task/86707/fd (opendir: Permission denied) 2024-03-18T15:20:22.815 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86708 mgr-fin ceph cwd unknown /proc/31759/task/86708/cwd (readlink: Permission denied) 2024-03-18T15:20:22.815 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86708 mgr-fin ceph rtd unknown /proc/31759/task/86708/root (readlink: Permission denied) 2024-03-18T15:20:22.815 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86708 mgr-fin ceph txt unknown /proc/31759/task/86708/exe (readlink: Permission denied) 2024-03-18T15:20:22.815 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86708 mgr-fin ceph NOFD /proc/31759/task/86708/fd (opendir: Permission denied) 2024-03-18T15:20:22.815 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86709 mgr-fin ceph cwd unknown /proc/31759/task/86709/cwd (readlink: Permission denied) 2024-03-18T15:20:22.815 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86709 mgr-fin ceph rtd unknown /proc/31759/task/86709/root (readlink: Permission denied) 2024-03-18T15:20:22.815 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86709 mgr-fin ceph txt unknown /proc/31759/task/86709/exe (readlink: Permission denied) 2024-03-18T15:20:22.815 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86709 mgr-fin ceph NOFD /proc/31759/task/86709/fd (opendir: Permission denied) 2024-03-18T15:20:22.815 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86710 mgr-fin ceph cwd unknown /proc/31759/task/86710/cwd (readlink: Permission denied) 2024-03-18T15:20:22.815 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86710 mgr-fin ceph rtd unknown /proc/31759/task/86710/root (readlink: Permission denied) 2024-03-18T15:20:22.815 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86710 mgr-fin ceph txt unknown /proc/31759/task/86710/exe (readlink: Permission denied) 2024-03-18T15:20:22.815 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86710 mgr-fin ceph NOFD /proc/31759/task/86710/fd (opendir: Permission denied) 2024-03-18T15:20:22.815 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86711 mgr-fin ceph cwd unknown /proc/31759/task/86711/cwd (readlink: Permission denied) 2024-03-18T15:20:22.815 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86711 mgr-fin ceph rtd unknown /proc/31759/task/86711/root (readlink: Permission denied) 2024-03-18T15:20:22.815 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86711 mgr-fin ceph txt unknown /proc/31759/task/86711/exe (readlink: Permission denied) 2024-03-18T15:20:22.816 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86711 mgr-fin ceph NOFD /proc/31759/task/86711/fd (opendir: Permission denied) 2024-03-18T15:20:22.816 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86712 mgr-fin ceph cwd unknown /proc/31759/task/86712/cwd (readlink: Permission denied) 2024-03-18T15:20:22.816 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86712 mgr-fin ceph rtd unknown /proc/31759/task/86712/root (readlink: Permission denied) 2024-03-18T15:20:22.816 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86712 mgr-fin ceph txt unknown /proc/31759/task/86712/exe (readlink: Permission denied) 2024-03-18T15:20:22.816 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86712 mgr-fin ceph NOFD /proc/31759/task/86712/fd (opendir: Permission denied) 2024-03-18T15:20:22.816 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86713 mgr-fin ceph cwd unknown /proc/31759/task/86713/cwd (readlink: Permission denied) 2024-03-18T15:20:22.816 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86713 mgr-fin ceph rtd unknown /proc/31759/task/86713/root (readlink: Permission denied) 2024-03-18T15:20:22.816 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86713 mgr-fin ceph txt unknown /proc/31759/task/86713/exe (readlink: Permission denied) 2024-03-18T15:20:22.816 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86713 mgr-fin ceph NOFD /proc/31759/task/86713/fd (opendir: Permission denied) 2024-03-18T15:20:22.816 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86717 io_contex ceph cwd unknown /proc/31759/task/86717/cwd (readlink: Permission denied) 2024-03-18T15:20:22.816 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86717 io_contex ceph rtd unknown /proc/31759/task/86717/root (readlink: Permission denied) 2024-03-18T15:20:22.816 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86717 io_contex ceph txt unknown /proc/31759/task/86717/exe (readlink: Permission denied) 2024-03-18T15:20:22.816 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86717 io_contex ceph NOFD /proc/31759/task/86717/fd (opendir: Permission denied) 2024-03-18T15:20:22.816 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86718 io_contex ceph cwd unknown /proc/31759/task/86718/cwd (readlink: Permission denied) 2024-03-18T15:20:22.816 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86718 io_contex ceph rtd unknown /proc/31759/task/86718/root (readlink: Permission denied) 2024-03-18T15:20:22.817 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86718 io_contex ceph txt unknown /proc/31759/task/86718/exe (readlink: Permission denied) 2024-03-18T15:20:22.817 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86718 io_contex ceph NOFD /proc/31759/task/86718/fd (opendir: Permission denied) 2024-03-18T15:20:22.817 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86719 ceph_time ceph cwd unknown /proc/31759/task/86719/cwd (readlink: Permission denied) 2024-03-18T15:20:22.817 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86719 ceph_time ceph rtd unknown /proc/31759/task/86719/root (readlink: Permission denied) 2024-03-18T15:20:22.817 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86719 ceph_time ceph txt unknown /proc/31759/task/86719/exe (readlink: Permission denied) 2024-03-18T15:20:22.817 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86719 ceph_time ceph NOFD /proc/31759/task/86719/fd (opendir: Permission denied) 2024-03-18T15:20:22.817 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86720 ms_dispat ceph cwd unknown /proc/31759/task/86720/cwd (readlink: Permission denied) 2024-03-18T15:20:22.817 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86720 ms_dispat ceph rtd unknown /proc/31759/task/86720/root (readlink: Permission denied) 2024-03-18T15:20:22.817 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86720 ms_dispat ceph txt unknown /proc/31759/task/86720/exe (readlink: Permission denied) 2024-03-18T15:20:22.817 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86720 ms_dispat ceph NOFD /proc/31759/task/86720/fd (opendir: Permission denied) 2024-03-18T15:20:22.817 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86721 ms_local ceph cwd unknown /proc/31759/task/86721/cwd (readlink: Permission denied) 2024-03-18T15:20:22.817 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86721 ms_local ceph rtd unknown /proc/31759/task/86721/root (readlink: Permission denied) 2024-03-18T15:20:22.817 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86721 ms_local ceph txt unknown /proc/31759/task/86721/exe (readlink: Permission denied) 2024-03-18T15:20:22.817 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86721 ms_local ceph NOFD /proc/31759/task/86721/fd (opendir: Permission denied) 2024-03-18T15:20:22.817 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86722 safe_time ceph cwd unknown /proc/31759/task/86722/cwd (readlink: Permission denied) 2024-03-18T15:20:22.818 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86722 safe_time ceph rtd unknown /proc/31759/task/86722/root (readlink: Permission denied) 2024-03-18T15:20:22.818 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86722 safe_time ceph txt unknown /proc/31759/task/86722/exe (readlink: Permission denied) 2024-03-18T15:20:22.818 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86722 safe_time ceph NOFD /proc/31759/task/86722/fd (opendir: Permission denied) 2024-03-18T15:20:22.818 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86723 safe_time ceph cwd unknown /proc/31759/task/86723/cwd (readlink: Permission denied) 2024-03-18T15:20:22.818 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86723 safe_time ceph rtd unknown /proc/31759/task/86723/root (readlink: Permission denied) 2024-03-18T15:20:22.818 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86723 safe_time ceph txt unknown /proc/31759/task/86723/exe (readlink: Permission denied) 2024-03-18T15:20:22.818 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86723 safe_time ceph NOFD /proc/31759/task/86723/fd (opendir: Permission denied) 2024-03-18T15:20:22.818 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86727 m-fin-sna ceph cwd unknown /proc/31759/task/86727/cwd (readlink: Permission denied) 2024-03-18T15:20:22.818 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86727 m-fin-sna ceph rtd unknown /proc/31759/task/86727/root (readlink: Permission denied) 2024-03-18T15:20:22.818 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86727 m-fin-sna ceph txt unknown /proc/31759/task/86727/exe (readlink: Permission denied) 2024-03-18T15:20:22.818 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86727 m-fin-sna ceph NOFD /proc/31759/task/86727/fd (opendir: Permission denied) 2024-03-18T15:20:22.818 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86729 m-fin-sta ceph cwd unknown /proc/31759/task/86729/cwd (readlink: Permission denied) 2024-03-18T15:20:22.818 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86729 m-fin-sta ceph rtd unknown /proc/31759/task/86729/root (readlink: Permission denied) 2024-03-18T15:20:22.818 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86729 m-fin-sta ceph txt unknown /proc/31759/task/86729/exe (readlink: Permission denied) 2024-03-18T15:20:22.819 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86729 m-fin-sta ceph NOFD /proc/31759/task/86729/fd (opendir: Permission denied) 2024-03-18T15:20:22.819 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86730 telemetry ceph cwd unknown /proc/31759/task/86730/cwd (readlink: Permission denied) 2024-03-18T15:20:22.819 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86730 telemetry ceph rtd unknown /proc/31759/task/86730/root (readlink: Permission denied) 2024-03-18T15:20:22.819 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86730 telemetry ceph txt unknown /proc/31759/task/86730/exe (readlink: Permission denied) 2024-03-18T15:20:22.819 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86730 telemetry ceph NOFD /proc/31759/task/86730/fd (opendir: Permission denied) 2024-03-18T15:20:22.819 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86731 m-fin-tel ceph cwd unknown /proc/31759/task/86731/cwd (readlink: Permission denied) 2024-03-18T15:20:22.819 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86731 m-fin-tel ceph rtd unknown /proc/31759/task/86731/root (readlink: Permission denied) 2024-03-18T15:20:22.819 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86731 m-fin-tel ceph txt unknown /proc/31759/task/86731/exe (readlink: Permission denied) 2024-03-18T15:20:22.819 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86731 m-fin-tel ceph NOFD /proc/31759/task/86731/fd (opendir: Permission denied) 2024-03-18T15:20:22.819 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86732 mgr-fin ceph cwd unknown /proc/31759/task/86732/cwd (readlink: Permission denied) 2024-03-18T15:20:22.819 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86732 mgr-fin ceph rtd unknown /proc/31759/task/86732/root (readlink: Permission denied) 2024-03-18T15:20:22.819 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86732 mgr-fin ceph txt unknown /proc/31759/task/86732/exe (readlink: Permission denied) 2024-03-18T15:20:22.819 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86732 mgr-fin ceph NOFD /proc/31759/task/86732/fd (opendir: Permission denied) 2024-03-18T15:20:22.819 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86736 io_contex ceph cwd unknown /proc/31759/task/86736/cwd (readlink: Permission denied) 2024-03-18T15:20:22.819 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86736 io_contex ceph rtd unknown /proc/31759/task/86736/root (readlink: Permission denied) 2024-03-18T15:20:22.819 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86736 io_contex ceph txt unknown /proc/31759/task/86736/exe (readlink: Permission denied) 2024-03-18T15:20:22.820 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86736 io_contex ceph NOFD /proc/31759/task/86736/fd (opendir: Permission denied) 2024-03-18T15:20:22.820 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86737 io_contex ceph cwd unknown /proc/31759/task/86737/cwd (readlink: Permission denied) 2024-03-18T15:20:22.820 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86737 io_contex ceph rtd unknown /proc/31759/task/86737/root (readlink: Permission denied) 2024-03-18T15:20:22.820 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86737 io_contex ceph txt unknown /proc/31759/task/86737/exe (readlink: Permission denied) 2024-03-18T15:20:22.820 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86737 io_contex ceph NOFD /proc/31759/task/86737/fd (opendir: Permission denied) 2024-03-18T15:20:22.820 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86738 ceph_time ceph cwd unknown /proc/31759/task/86738/cwd (readlink: Permission denied) 2024-03-18T15:20:22.820 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86738 ceph_time ceph rtd unknown /proc/31759/task/86738/root (readlink: Permission denied) 2024-03-18T15:20:22.820 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86738 ceph_time ceph txt unknown /proc/31759/task/86738/exe (readlink: Permission denied) 2024-03-18T15:20:22.820 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86738 ceph_time ceph NOFD /proc/31759/task/86738/fd (opendir: Permission denied) 2024-03-18T15:20:22.820 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86739 ms_dispat ceph cwd unknown /proc/31759/task/86739/cwd (readlink: Permission denied) 2024-03-18T15:20:22.820 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86739 ms_dispat ceph rtd unknown /proc/31759/task/86739/root (readlink: Permission denied) 2024-03-18T15:20:22.820 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86739 ms_dispat ceph txt unknown /proc/31759/task/86739/exe (readlink: Permission denied) 2024-03-18T15:20:22.820 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86739 ms_dispat ceph NOFD /proc/31759/task/86739/fd (opendir: Permission denied) 2024-03-18T15:20:22.820 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86740 ms_local ceph cwd unknown /proc/31759/task/86740/cwd (readlink: Permission denied) 2024-03-18T15:20:22.820 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86740 ms_local ceph rtd unknown /proc/31759/task/86740/root (readlink: Permission denied) 2024-03-18T15:20:22.821 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86740 ms_local ceph txt unknown /proc/31759/task/86740/exe (readlink: Permission denied) 2024-03-18T15:20:22.821 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86740 ms_local ceph NOFD /proc/31759/task/86740/fd (opendir: Permission denied) 2024-03-18T15:20:22.821 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86741 safe_time ceph cwd unknown /proc/31759/task/86741/cwd (readlink: Permission denied) 2024-03-18T15:20:22.821 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86741 safe_time ceph rtd unknown /proc/31759/task/86741/root (readlink: Permission denied) 2024-03-18T15:20:22.821 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86741 safe_time ceph txt unknown /proc/31759/task/86741/exe (readlink: Permission denied) 2024-03-18T15:20:22.821 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86741 safe_time ceph NOFD /proc/31759/task/86741/fd (opendir: Permission denied) 2024-03-18T15:20:22.821 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86742 safe_time ceph cwd unknown /proc/31759/task/86742/cwd (readlink: Permission denied) 2024-03-18T15:20:22.821 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86742 safe_time ceph rtd unknown /proc/31759/task/86742/root (readlink: Permission denied) 2024-03-18T15:20:22.821 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86742 safe_time ceph txt unknown /proc/31759/task/86742/exe (readlink: Permission denied) 2024-03-18T15:20:22.821 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86742 safe_time ceph NOFD /proc/31759/task/86742/fd (opendir: Permission denied) 2024-03-18T15:20:22.821 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86743 mgr-fin ceph cwd unknown /proc/31759/task/86743/cwd (readlink: Permission denied) 2024-03-18T15:20:22.821 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86743 mgr-fin ceph rtd unknown /proc/31759/task/86743/root (readlink: Permission denied) 2024-03-18T15:20:22.821 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86743 mgr-fin ceph txt unknown /proc/31759/task/86743/exe (readlink: Permission denied) 2024-03-18T15:20:22.821 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86743 mgr-fin ceph NOFD /proc/31759/task/86743/fd (opendir: Permission denied) 2024-03-18T15:20:22.821 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86744 mgr-fin ceph cwd unknown /proc/31759/task/86744/cwd (readlink: Permission denied) 2024-03-18T15:20:22.822 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86744 mgr-fin ceph rtd unknown /proc/31759/task/86744/root (readlink: Permission denied) 2024-03-18T15:20:22.822 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86744 mgr-fin ceph txt unknown /proc/31759/task/86744/exe (readlink: Permission denied) 2024-03-18T15:20:22.822 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86744 mgr-fin ceph NOFD /proc/31759/task/86744/fd (opendir: Permission denied) 2024-03-18T15:20:22.822 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86745 mgr-fin ceph cwd unknown /proc/31759/task/86745/cwd (readlink: Permission denied) 2024-03-18T15:20:22.822 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86745 mgr-fin ceph rtd unknown /proc/31759/task/86745/root (readlink: Permission denied) 2024-03-18T15:20:22.822 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86745 mgr-fin ceph txt unknown /proc/31759/task/86745/exe (readlink: Permission denied) 2024-03-18T15:20:22.822 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86745 mgr-fin ceph NOFD /proc/31759/task/86745/fd (opendir: Permission denied) 2024-03-18T15:20:22.822 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86746 mgr-fin ceph cwd unknown /proc/31759/task/86746/cwd (readlink: Permission denied) 2024-03-18T15:20:22.822 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86746 mgr-fin ceph rtd unknown /proc/31759/task/86746/root (readlink: Permission denied) 2024-03-18T15:20:22.822 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86746 mgr-fin ceph txt unknown /proc/31759/task/86746/exe (readlink: Permission denied) 2024-03-18T15:20:22.822 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86746 mgr-fin ceph NOFD /proc/31759/task/86746/fd (opendir: Permission denied) 2024-03-18T15:20:22.822 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86747 mgr-fin ceph cwd unknown /proc/31759/task/86747/cwd (readlink: Permission denied) 2024-03-18T15:20:22.822 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86747 mgr-fin ceph rtd unknown /proc/31759/task/86747/root (readlink: Permission denied) 2024-03-18T15:20:22.822 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86747 mgr-fin ceph txt unknown /proc/31759/task/86747/exe (readlink: Permission denied) 2024-03-18T15:20:22.823 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86747 mgr-fin ceph NOFD /proc/31759/task/86747/fd (opendir: Permission denied) 2024-03-18T15:20:22.823 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86748 mgr-fin ceph cwd unknown /proc/31759/task/86748/cwd (readlink: Permission denied) 2024-03-18T15:20:22.823 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86748 mgr-fin ceph rtd unknown /proc/31759/task/86748/root (readlink: Permission denied) 2024-03-18T15:20:22.823 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86748 mgr-fin ceph txt unknown /proc/31759/task/86748/exe (readlink: Permission denied) 2024-03-18T15:20:22.823 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86748 mgr-fin ceph NOFD /proc/31759/task/86748/fd (opendir: Permission denied) 2024-03-18T15:20:22.823 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86749 mgr-fin ceph cwd unknown /proc/31759/task/86749/cwd (readlink: Permission denied) 2024-03-18T15:20:22.823 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86749 mgr-fin ceph rtd unknown /proc/31759/task/86749/root (readlink: Permission denied) 2024-03-18T15:20:22.823 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86749 mgr-fin ceph txt unknown /proc/31759/task/86749/exe (readlink: Permission denied) 2024-03-18T15:20:22.823 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86749 mgr-fin ceph NOFD /proc/31759/task/86749/fd (opendir: Permission denied) 2024-03-18T15:20:22.823 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86750 mgr-fin ceph cwd unknown /proc/31759/task/86750/cwd (readlink: Permission denied) 2024-03-18T15:20:22.823 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86750 mgr-fin ceph rtd unknown /proc/31759/task/86750/root (readlink: Permission denied) 2024-03-18T15:20:22.823 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86750 mgr-fin ceph txt unknown /proc/31759/task/86750/exe (readlink: Permission denied) 2024-03-18T15:20:22.823 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86750 mgr-fin ceph NOFD /proc/31759/task/86750/fd (opendir: Permission denied) 2024-03-18T15:20:22.823 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86751 mgr-fin ceph cwd unknown /proc/31759/task/86751/cwd (readlink: Permission denied) 2024-03-18T15:20:22.824 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86751 mgr-fin ceph rtd unknown /proc/31759/task/86751/root (readlink: Permission denied) 2024-03-18T15:20:22.824 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86751 mgr-fin ceph txt unknown /proc/31759/task/86751/exe (readlink: Permission denied) 2024-03-18T15:20:22.824 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86751 mgr-fin ceph NOFD /proc/31759/task/86751/fd (opendir: Permission denied) 2024-03-18T15:20:22.824 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86752 mgr-fin ceph cwd unknown /proc/31759/task/86752/cwd (readlink: Permission denied) 2024-03-18T15:20:22.824 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86752 mgr-fin ceph rtd unknown /proc/31759/task/86752/root (readlink: Permission denied) 2024-03-18T15:20:22.824 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86752 mgr-fin ceph txt unknown /proc/31759/task/86752/exe (readlink: Permission denied) 2024-03-18T15:20:22.824 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86752 mgr-fin ceph NOFD /proc/31759/task/86752/fd (opendir: Permission denied) 2024-03-18T15:20:22.824 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86753 mgr-fin ceph cwd unknown /proc/31759/task/86753/cwd (readlink: Permission denied) 2024-03-18T15:20:22.824 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86753 mgr-fin ceph rtd unknown /proc/31759/task/86753/root (readlink: Permission denied) 2024-03-18T15:20:22.824 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86753 mgr-fin ceph txt unknown /proc/31759/task/86753/exe (readlink: Permission denied) 2024-03-18T15:20:22.824 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86753 mgr-fin ceph NOFD /proc/31759/task/86753/fd (opendir: Permission denied) 2024-03-18T15:20:22.824 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86754 mgr-fin ceph cwd unknown /proc/31759/task/86754/cwd (readlink: Permission denied) 2024-03-18T15:20:22.824 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86754 mgr-fin ceph rtd unknown /proc/31759/task/86754/root (readlink: Permission denied) 2024-03-18T15:20:22.824 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86754 mgr-fin ceph txt unknown /proc/31759/task/86754/exe (readlink: Permission denied) 2024-03-18T15:20:22.825 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86754 mgr-fin ceph NOFD /proc/31759/task/86754/fd (opendir: Permission denied) 2024-03-18T15:20:22.825 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86756 m-fin-vol ceph cwd unknown /proc/31759/task/86756/cwd (readlink: Permission denied) 2024-03-18T15:20:22.825 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86756 m-fin-vol ceph rtd unknown /proc/31759/task/86756/root (readlink: Permission denied) 2024-03-18T15:20:22.825 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86756 m-fin-vol ceph txt unknown /proc/31759/task/86756/exe (readlink: Permission denied) 2024-03-18T15:20:22.825 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86756 m-fin-vol ceph NOFD /proc/31759/task/86756/fd (opendir: Permission denied) 2024-03-18T15:20:22.825 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86757 io_contex ceph cwd unknown /proc/31759/task/86757/cwd (readlink: Permission denied) 2024-03-18T15:20:22.825 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86757 io_contex ceph rtd unknown /proc/31759/task/86757/root (readlink: Permission denied) 2024-03-18T15:20:22.825 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86757 io_contex ceph txt unknown /proc/31759/task/86757/exe (readlink: Permission denied) 2024-03-18T15:20:22.825 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86757 io_contex ceph NOFD /proc/31759/task/86757/fd (opendir: Permission denied) 2024-03-18T15:20:22.825 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86758 io_contex ceph cwd unknown /proc/31759/task/86758/cwd (readlink: Permission denied) 2024-03-18T15:20:22.825 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86758 io_contex ceph rtd unknown /proc/31759/task/86758/root (readlink: Permission denied) 2024-03-18T15:20:22.825 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86758 io_contex ceph txt unknown /proc/31759/task/86758/exe (readlink: Permission denied) 2024-03-18T15:20:22.825 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86758 io_contex ceph NOFD /proc/31759/task/86758/fd (opendir: Permission denied) 2024-03-18T15:20:22.825 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86784 dashboard ceph cwd unknown /proc/31759/task/86784/cwd (readlink: Permission denied) 2024-03-18T15:20:22.825 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86784 dashboard ceph rtd unknown /proc/31759/task/86784/root (readlink: Permission denied) 2024-03-18T15:20:22.826 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86784 dashboard ceph txt unknown /proc/31759/task/86784/exe (readlink: Permission denied) 2024-03-18T15:20:22.826 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86784 dashboard ceph NOFD /proc/31759/task/86784/fd (opendir: Permission denied) 2024-03-18T15:20:22.826 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86785 dashboard ceph cwd unknown /proc/31759/task/86785/cwd (readlink: Permission denied) 2024-03-18T15:20:22.826 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86785 dashboard ceph rtd unknown /proc/31759/task/86785/root (readlink: Permission denied) 2024-03-18T15:20:22.826 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86785 dashboard ceph txt unknown /proc/31759/task/86785/exe (readlink: Permission denied) 2024-03-18T15:20:22.826 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86785 dashboard ceph NOFD /proc/31759/task/86785/fd (opendir: Permission denied) 2024-03-18T15:20:22.826 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86786 dashboard ceph cwd unknown /proc/31759/task/86786/cwd (readlink: Permission denied) 2024-03-18T15:20:22.826 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86786 dashboard ceph rtd unknown /proc/31759/task/86786/root (readlink: Permission denied) 2024-03-18T15:20:22.826 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86786 dashboard ceph txt unknown /proc/31759/task/86786/exe (readlink: Permission denied) 2024-03-18T15:20:22.826 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86786 dashboard ceph NOFD /proc/31759/task/86786/fd (opendir: Permission denied) 2024-03-18T15:20:22.826 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86787 dashboard ceph cwd unknown /proc/31759/task/86787/cwd (readlink: Permission denied) 2024-03-18T15:20:22.826 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86787 dashboard ceph rtd unknown /proc/31759/task/86787/root (readlink: Permission denied) 2024-03-18T15:20:22.826 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86787 dashboard ceph txt unknown /proc/31759/task/86787/exe (readlink: Permission denied) 2024-03-18T15:20:22.827 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86787 dashboard ceph NOFD /proc/31759/task/86787/fd (opendir: Permission denied) 2024-03-18T15:20:22.827 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86788 dashboard ceph cwd unknown /proc/31759/task/86788/cwd (readlink: Permission denied) 2024-03-18T15:20:22.827 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86788 dashboard ceph rtd unknown /proc/31759/task/86788/root (readlink: Permission denied) 2024-03-18T15:20:22.827 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86788 dashboard ceph txt unknown /proc/31759/task/86788/exe (readlink: Permission denied) 2024-03-18T15:20:22.827 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86788 dashboard ceph NOFD /proc/31759/task/86788/fd (opendir: Permission denied) 2024-03-18T15:20:22.827 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86789 dashboard ceph cwd unknown /proc/31759/task/86789/cwd (readlink: Permission denied) 2024-03-18T15:20:22.827 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86789 dashboard ceph rtd unknown /proc/31759/task/86789/root (readlink: Permission denied) 2024-03-18T15:20:22.827 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86789 dashboard ceph txt unknown /proc/31759/task/86789/exe (readlink: Permission denied) 2024-03-18T15:20:22.827 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86789 dashboard ceph NOFD /proc/31759/task/86789/fd (opendir: Permission denied) 2024-03-18T15:20:22.827 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86790 dashboard ceph cwd unknown /proc/31759/task/86790/cwd (readlink: Permission denied) 2024-03-18T15:20:22.827 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86790 dashboard ceph rtd unknown /proc/31759/task/86790/root (readlink: Permission denied) 2024-03-18T15:20:22.827 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86790 dashboard ceph txt unknown /proc/31759/task/86790/exe (readlink: Permission denied) 2024-03-18T15:20:22.827 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86790 dashboard ceph NOFD /proc/31759/task/86790/fd (opendir: Permission denied) 2024-03-18T15:20:22.827 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86791 dashboard ceph cwd unknown /proc/31759/task/86791/cwd (readlink: Permission denied) 2024-03-18T15:20:22.828 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86791 dashboard ceph rtd unknown /proc/31759/task/86791/root (readlink: Permission denied) 2024-03-18T15:20:22.828 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86791 dashboard ceph txt unknown /proc/31759/task/86791/exe (readlink: Permission denied) 2024-03-18T15:20:22.828 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86791 dashboard ceph NOFD /proc/31759/task/86791/fd (opendir: Permission denied) 2024-03-18T15:20:22.828 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86792 dashboard ceph cwd unknown /proc/31759/task/86792/cwd (readlink: Permission denied) 2024-03-18T15:20:22.828 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86792 dashboard ceph rtd unknown /proc/31759/task/86792/root (readlink: Permission denied) 2024-03-18T15:20:22.828 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86792 dashboard ceph txt unknown /proc/31759/task/86792/exe (readlink: Permission denied) 2024-03-18T15:20:22.828 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86792 dashboard ceph NOFD /proc/31759/task/86792/fd (opendir: Permission denied) 2024-03-18T15:20:22.828 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86793 dashboard ceph cwd unknown /proc/31759/task/86793/cwd (readlink: Permission denied) 2024-03-18T15:20:22.828 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86793 dashboard ceph rtd unknown /proc/31759/task/86793/root (readlink: Permission denied) 2024-03-18T15:20:22.828 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86793 dashboard ceph txt unknown /proc/31759/task/86793/exe (readlink: Permission denied) 2024-03-18T15:20:22.828 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86793 dashboard ceph NOFD /proc/31759/task/86793/fd (opendir: Permission denied) 2024-03-18T15:20:22.828 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86794 dashboard ceph cwd unknown /proc/31759/task/86794/cwd (readlink: Permission denied) 2024-03-18T15:20:22.828 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86794 dashboard ceph rtd unknown /proc/31759/task/86794/root (readlink: Permission denied) 2024-03-18T15:20:22.828 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86794 dashboard ceph txt unknown /proc/31759/task/86794/exe (readlink: Permission denied) 2024-03-18T15:20:22.828 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86794 dashboard ceph NOFD /proc/31759/task/86794/fd (opendir: Permission denied) 2024-03-18T15:20:22.829 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86797 dashboard ceph cwd unknown /proc/31759/task/86797/cwd (readlink: Permission denied) 2024-03-18T15:20:22.829 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86797 dashboard ceph rtd unknown /proc/31759/task/86797/root (readlink: Permission denied) 2024-03-18T15:20:22.829 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86797 dashboard ceph txt unknown /proc/31759/task/86797/exe (readlink: Permission denied) 2024-03-18T15:20:22.829 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86797 dashboard ceph NOFD /proc/31759/task/86797/fd (opendir: Permission denied) 2024-03-18T15:20:22.829 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86865 mgr-fin ceph cwd unknown /proc/31759/task/86865/cwd (readlink: Permission denied) 2024-03-18T15:20:22.829 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86865 mgr-fin ceph rtd unknown /proc/31759/task/86865/root (readlink: Permission denied) 2024-03-18T15:20:22.829 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86865 mgr-fin ceph txt unknown /proc/31759/task/86865/exe (readlink: Permission denied) 2024-03-18T15:20:22.829 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86865 mgr-fin ceph NOFD /proc/31759/task/86865/fd (opendir: Permission denied) 2024-03-18T15:20:22.829 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86866 mgr-fin ceph cwd unknown /proc/31759/task/86866/cwd (readlink: Permission denied) 2024-03-18T15:20:22.829 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86866 mgr-fin ceph rtd unknown /proc/31759/task/86866/root (readlink: Permission denied) 2024-03-18T15:20:22.829 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86866 mgr-fin ceph txt unknown /proc/31759/task/86866/exe (readlink: Permission denied) 2024-03-18T15:20:22.829 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86866 mgr-fin ceph NOFD /proc/31759/task/86866/fd (opendir: Permission denied) 2024-03-18T15:20:22.829 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86867 mgr-fin ceph cwd unknown /proc/31759/task/86867/cwd (readlink: Permission denied) 2024-03-18T15:20:22.829 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86867 mgr-fin ceph rtd unknown /proc/31759/task/86867/root (readlink: Permission denied) 2024-03-18T15:20:22.830 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86867 mgr-fin ceph txt unknown /proc/31759/task/86867/exe (readlink: Permission denied) 2024-03-18T15:20:22.830 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86867 mgr-fin ceph NOFD /proc/31759/task/86867/fd (opendir: Permission denied) 2024-03-18T15:20:22.830 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86868 mgr-fin ceph cwd unknown /proc/31759/task/86868/cwd (readlink: Permission denied) 2024-03-18T15:20:22.830 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86868 mgr-fin ceph rtd unknown /proc/31759/task/86868/root (readlink: Permission denied) 2024-03-18T15:20:22.830 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86868 mgr-fin ceph txt unknown /proc/31759/task/86868/exe (readlink: Permission denied) 2024-03-18T15:20:22.830 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86868 mgr-fin ceph NOFD /proc/31759/task/86868/fd (opendir: Permission denied) 2024-03-18T15:20:22.830 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86869 mgr-fin ceph cwd unknown /proc/31759/task/86869/cwd (readlink: Permission denied) 2024-03-18T15:20:22.830 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86869 mgr-fin ceph rtd unknown /proc/31759/task/86869/root (readlink: Permission denied) 2024-03-18T15:20:22.830 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86869 mgr-fin ceph txt unknown /proc/31759/task/86869/exe (readlink: Permission denied) 2024-03-18T15:20:22.830 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86869 mgr-fin ceph NOFD /proc/31759/task/86869/fd (opendir: Permission denied) 2024-03-18T15:20:22.830 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86870 mgr-fin ceph cwd unknown /proc/31759/task/86870/cwd (readlink: Permission denied) 2024-03-18T15:20:22.830 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86870 mgr-fin ceph rtd unknown /proc/31759/task/86870/root (readlink: Permission denied) 2024-03-18T15:20:22.830 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86870 mgr-fin ceph txt unknown /proc/31759/task/86870/exe (readlink: Permission denied) 2024-03-18T15:20:22.831 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86870 mgr-fin ceph NOFD /proc/31759/task/86870/fd (opendir: Permission denied) 2024-03-18T15:20:22.831 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86871 mgr-fin ceph cwd unknown /proc/31759/task/86871/cwd (readlink: Permission denied) 2024-03-18T15:20:22.831 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86871 mgr-fin ceph rtd unknown /proc/31759/task/86871/root (readlink: Permission denied) 2024-03-18T15:20:22.831 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86871 mgr-fin ceph txt unknown /proc/31759/task/86871/exe (readlink: Permission denied) 2024-03-18T15:20:22.831 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86871 mgr-fin ceph NOFD /proc/31759/task/86871/fd (opendir: Permission denied) 2024-03-18T15:20:22.831 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86872 mgr-fin ceph cwd unknown /proc/31759/task/86872/cwd (readlink: Permission denied) 2024-03-18T15:20:22.831 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86872 mgr-fin ceph rtd unknown /proc/31759/task/86872/root (readlink: Permission denied) 2024-03-18T15:20:22.831 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86872 mgr-fin ceph txt unknown /proc/31759/task/86872/exe (readlink: Permission denied) 2024-03-18T15:20:22.831 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86872 mgr-fin ceph NOFD /proc/31759/task/86872/fd (opendir: Permission denied) 2024-03-18T15:20:22.831 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86873 mgr-fin ceph cwd unknown /proc/31759/task/86873/cwd (readlink: Permission denied) 2024-03-18T15:20:22.831 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86873 mgr-fin ceph rtd unknown /proc/31759/task/86873/root (readlink: Permission denied) 2024-03-18T15:20:22.831 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86873 mgr-fin ceph txt unknown /proc/31759/task/86873/exe (readlink: Permission denied) 2024-03-18T15:20:22.831 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86873 mgr-fin ceph NOFD /proc/31759/task/86873/fd (opendir: Permission denied) 2024-03-18T15:20:22.831 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86874 mgr-fin ceph cwd unknown /proc/31759/task/86874/cwd (readlink: Permission denied) 2024-03-18T15:20:22.832 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86874 mgr-fin ceph rtd unknown /proc/31759/task/86874/root (readlink: Permission denied) 2024-03-18T15:20:22.832 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86874 mgr-fin ceph txt unknown /proc/31759/task/86874/exe (readlink: Permission denied) 2024-03-18T15:20:22.832 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86874 mgr-fin ceph NOFD /proc/31759/task/86874/fd (opendir: Permission denied) 2024-03-18T15:20:22.832 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86875 mgr-fin ceph cwd unknown /proc/31759/task/86875/cwd (readlink: Permission denied) 2024-03-18T15:20:22.832 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86875 mgr-fin ceph rtd unknown /proc/31759/task/86875/root (readlink: Permission denied) 2024-03-18T15:20:22.832 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86875 mgr-fin ceph txt unknown /proc/31759/task/86875/exe (readlink: Permission denied) 2024-03-18T15:20:22.832 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86875 mgr-fin ceph NOFD /proc/31759/task/86875/fd (opendir: Permission denied) 2024-03-18T15:20:22.832 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86897 mgr-fin ceph cwd unknown /proc/31759/task/86897/cwd (readlink: Permission denied) 2024-03-18T15:20:22.832 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86897 mgr-fin ceph rtd unknown /proc/31759/task/86897/root (readlink: Permission denied) 2024-03-18T15:20:22.832 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86897 mgr-fin ceph txt unknown /proc/31759/task/86897/exe (readlink: Permission denied) 2024-03-18T15:20:22.832 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86897 mgr-fin ceph NOFD /proc/31759/task/86897/fd (opendir: Permission denied) 2024-03-18T15:20:22.832 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86898 mgr-fin ceph cwd unknown /proc/31759/task/86898/cwd (readlink: Permission denied) 2024-03-18T15:20:22.832 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86898 mgr-fin ceph rtd unknown /proc/31759/task/86898/root (readlink: Permission denied) 2024-03-18T15:20:22.832 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86898 mgr-fin ceph txt unknown /proc/31759/task/86898/exe (readlink: Permission denied) 2024-03-18T15:20:22.832 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86898 mgr-fin ceph NOFD /proc/31759/task/86898/fd (opendir: Permission denied) 2024-03-18T15:20:22.832 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86899 mgr-fin ceph cwd unknown /proc/31759/task/86899/cwd (readlink: Permission denied) 2024-03-18T15:20:22.833 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86899 mgr-fin ceph rtd unknown /proc/31759/task/86899/root (readlink: Permission denied) 2024-03-18T15:20:22.833 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86899 mgr-fin ceph txt unknown /proc/31759/task/86899/exe (readlink: Permission denied) 2024-03-18T15:20:22.833 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86899 mgr-fin ceph NOFD /proc/31759/task/86899/fd (opendir: Permission denied) 2024-03-18T15:20:22.833 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86900 mgr-fin ceph cwd unknown /proc/31759/task/86900/cwd (readlink: Permission denied) 2024-03-18T15:20:22.833 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86900 mgr-fin ceph rtd unknown /proc/31759/task/86900/root (readlink: Permission denied) 2024-03-18T15:20:22.833 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86900 mgr-fin ceph txt unknown /proc/31759/task/86900/exe (readlink: Permission denied) 2024-03-18T15:20:22.833 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86900 mgr-fin ceph NOFD /proc/31759/task/86900/fd (opendir: Permission denied) 2024-03-18T15:20:22.833 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86901 mgr-fin ceph cwd unknown /proc/31759/task/86901/cwd (readlink: Permission denied) 2024-03-18T15:20:22.833 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86901 mgr-fin ceph rtd unknown /proc/31759/task/86901/root (readlink: Permission denied) 2024-03-18T15:20:22.833 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86901 mgr-fin ceph txt unknown /proc/31759/task/86901/exe (readlink: Permission denied) 2024-03-18T15:20:22.833 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86901 mgr-fin ceph NOFD /proc/31759/task/86901/fd (opendir: Permission denied) 2024-03-18T15:20:22.833 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86902 mgr-fin ceph cwd unknown /proc/31759/task/86902/cwd (readlink: Permission denied) 2024-03-18T15:20:22.833 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86902 mgr-fin ceph rtd unknown /proc/31759/task/86902/root (readlink: Permission denied) 2024-03-18T15:20:22.834 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86902 mgr-fin ceph txt unknown /proc/31759/task/86902/exe (readlink: Permission denied) 2024-03-18T15:20:22.834 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86902 mgr-fin ceph NOFD /proc/31759/task/86902/fd (opendir: Permission denied) 2024-03-18T15:20:22.834 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86903 mgr-fin ceph cwd unknown /proc/31759/task/86903/cwd (readlink: Permission denied) 2024-03-18T15:20:22.834 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86903 mgr-fin ceph rtd unknown /proc/31759/task/86903/root (readlink: Permission denied) 2024-03-18T15:20:22.834 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86903 mgr-fin ceph txt unknown /proc/31759/task/86903/exe (readlink: Permission denied) 2024-03-18T15:20:22.834 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86903 mgr-fin ceph NOFD /proc/31759/task/86903/fd (opendir: Permission denied) 2024-03-18T15:20:22.834 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86904 mgr-fin ceph cwd unknown /proc/31759/task/86904/cwd (readlink: Permission denied) 2024-03-18T15:20:22.834 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86904 mgr-fin ceph rtd unknown /proc/31759/task/86904/root (readlink: Permission denied) 2024-03-18T15:20:22.834 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86904 mgr-fin ceph txt unknown /proc/31759/task/86904/exe (readlink: Permission denied) 2024-03-18T15:20:22.834 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86904 mgr-fin ceph NOFD /proc/31759/task/86904/fd (opendir: Permission denied) 2024-03-18T15:20:22.834 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86905 mgr-fin ceph cwd unknown /proc/31759/task/86905/cwd (readlink: Permission denied) 2024-03-18T15:20:22.834 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86905 mgr-fin ceph rtd unknown /proc/31759/task/86905/root (readlink: Permission denied) 2024-03-18T15:20:22.834 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86905 mgr-fin ceph txt unknown /proc/31759/task/86905/exe (readlink: Permission denied) 2024-03-18T15:20:22.834 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86905 mgr-fin ceph NOFD /proc/31759/task/86905/fd (opendir: Permission denied) 2024-03-18T15:20:22.835 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86906 mgr-fin ceph cwd unknown /proc/31759/task/86906/cwd (readlink: Permission denied) 2024-03-18T15:20:22.835 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86906 mgr-fin ceph rtd unknown /proc/31759/task/86906/root (readlink: Permission denied) 2024-03-18T15:20:22.835 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86906 mgr-fin ceph txt unknown /proc/31759/task/86906/exe (readlink: Permission denied) 2024-03-18T15:20:22.835 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86906 mgr-fin ceph NOFD /proc/31759/task/86906/fd (opendir: Permission denied) 2024-03-18T15:20:22.835 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86907 mgr-fin ceph cwd unknown /proc/31759/task/86907/cwd (readlink: Permission denied) 2024-03-18T15:20:22.835 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86907 mgr-fin ceph rtd unknown /proc/31759/task/86907/root (readlink: Permission denied) 2024-03-18T15:20:22.835 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86907 mgr-fin ceph txt unknown /proc/31759/task/86907/exe (readlink: Permission denied) 2024-03-18T15:20:22.835 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 86907 mgr-fin ceph NOFD /proc/31759/task/86907/fd (opendir: Permission denied) 2024-03-18T15:20:22.835 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 87055 devicehea ceph cwd unknown /proc/31759/task/87055/cwd (readlink: Permission denied) 2024-03-18T15:20:22.835 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 87055 devicehea ceph rtd unknown /proc/31759/task/87055/root (readlink: Permission denied) 2024-03-18T15:20:22.835 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 87055 devicehea ceph txt unknown /proc/31759/task/87055/exe (readlink: Permission denied) 2024-03-18T15:20:22.835 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 87055 devicehea ceph NOFD /proc/31759/task/87055/fd (opendir: Permission denied) 2024-03-18T15:20:22.835 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 87061 io_contex ceph cwd unknown /proc/31759/task/87061/cwd (readlink: Permission denied) 2024-03-18T15:20:22.835 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 87061 io_contex ceph rtd unknown /proc/31759/task/87061/root (readlink: Permission denied) 2024-03-18T15:20:22.836 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 87061 io_contex ceph txt unknown /proc/31759/task/87061/exe (readlink: Permission denied) 2024-03-18T15:20:22.836 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 87061 io_contex ceph NOFD /proc/31759/task/87061/fd (opendir: Permission denied) 2024-03-18T15:20:22.836 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 87062 io_contex ceph cwd unknown /proc/31759/task/87062/cwd (readlink: Permission denied) 2024-03-18T15:20:22.836 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 87062 io_contex ceph rtd unknown /proc/31759/task/87062/root (readlink: Permission denied) 2024-03-18T15:20:22.836 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 87062 io_contex ceph txt unknown /proc/31759/task/87062/exe (readlink: Permission denied) 2024-03-18T15:20:22.836 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 87062 io_contex ceph NOFD /proc/31759/task/87062/fd (opendir: Permission denied) 2024-03-18T15:20:22.836 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 87063 ceph_time ceph cwd unknown /proc/31759/task/87063/cwd (readlink: Permission denied) 2024-03-18T15:20:22.836 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 87063 ceph_time ceph rtd unknown /proc/31759/task/87063/root (readlink: Permission denied) 2024-03-18T15:20:22.836 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 87063 ceph_time ceph txt unknown /proc/31759/task/87063/exe (readlink: Permission denied) 2024-03-18T15:20:22.836 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 87063 ceph_time ceph NOFD /proc/31759/task/87063/fd (opendir: Permission denied) 2024-03-18T15:20:22.836 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 87064 ms_dispat ceph cwd unknown /proc/31759/task/87064/cwd (readlink: Permission denied) 2024-03-18T15:20:22.836 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 87064 ms_dispat ceph rtd unknown /proc/31759/task/87064/root (readlink: Permission denied) 2024-03-18T15:20:22.836 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 87064 ms_dispat ceph txt unknown /proc/31759/task/87064/exe (readlink: Permission denied) 2024-03-18T15:20:22.836 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 87064 ms_dispat ceph NOFD /proc/31759/task/87064/fd (opendir: Permission denied) 2024-03-18T15:20:22.836 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 87065 ms_local ceph cwd unknown /proc/31759/task/87065/cwd (readlink: Permission denied) 2024-03-18T15:20:22.837 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 87065 ms_local ceph rtd unknown /proc/31759/task/87065/root (readlink: Permission denied) 2024-03-18T15:20:22.837 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 87065 ms_local ceph txt unknown /proc/31759/task/87065/exe (readlink: Permission denied) 2024-03-18T15:20:22.837 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 87065 ms_local ceph NOFD /proc/31759/task/87065/fd (opendir: Permission denied) 2024-03-18T15:20:22.837 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 87066 safe_time ceph cwd unknown /proc/31759/task/87066/cwd (readlink: Permission denied) 2024-03-18T15:20:22.837 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 87066 safe_time ceph rtd unknown /proc/31759/task/87066/root (readlink: Permission denied) 2024-03-18T15:20:22.837 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 87066 safe_time ceph txt unknown /proc/31759/task/87066/exe (readlink: Permission denied) 2024-03-18T15:20:22.837 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 87066 safe_time ceph NOFD /proc/31759/task/87066/fd (opendir: Permission denied) 2024-03-18T15:20:22.837 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 87067 safe_time ceph cwd unknown /proc/31759/task/87067/cwd (readlink: Permission denied) 2024-03-18T15:20:22.837 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 87067 safe_time ceph rtd unknown /proc/31759/task/87067/root (readlink: Permission denied) 2024-03-18T15:20:22.837 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 87067 safe_time ceph txt unknown /proc/31759/task/87067/exe (readlink: Permission denied) 2024-03-18T15:20:22.837 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 87067 safe_time ceph NOFD /proc/31759/task/87067/fd (opendir: Permission denied) 2024-03-18T15:20:22.837 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 88454 ceph_time ceph cwd unknown /proc/31759/task/88454/cwd (readlink: Permission denied) 2024-03-18T15:20:22.837 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 88454 ceph_time ceph rtd unknown /proc/31759/task/88454/root (readlink: Permission denied) 2024-03-18T15:20:22.837 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 88454 ceph_time ceph txt unknown /proc/31759/task/88454/exe (readlink: Permission denied) 2024-03-18T15:20:22.837 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 88454 ceph_time ceph NOFD /proc/31759/task/88454/fd (opendir: Permission denied) 2024-03-18T15:20:22.838 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 88455 fn_anonym ceph cwd unknown /proc/31759/task/88455/cwd (readlink: Permission denied) 2024-03-18T15:20:22.838 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 88455 fn_anonym ceph rtd unknown /proc/31759/task/88455/root (readlink: Permission denied) 2024-03-18T15:20:22.838 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 88455 fn_anonym ceph txt unknown /proc/31759/task/88455/exe (readlink: Permission denied) 2024-03-18T15:20:22.838 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 88455 fn_anonym ceph NOFD /proc/31759/task/88455/fd (opendir: Permission denied) 2024-03-18T15:20:22.838 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 88456 safe_time ceph cwd unknown /proc/31759/task/88456/cwd (readlink: Permission denied) 2024-03-18T15:20:22.838 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 88456 safe_time ceph rtd unknown /proc/31759/task/88456/root (readlink: Permission denied) 2024-03-18T15:20:22.838 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 88456 safe_time ceph txt unknown /proc/31759/task/88456/exe (readlink: Permission denied) 2024-03-18T15:20:22.838 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 88456 safe_time ceph NOFD /proc/31759/task/88456/fd (opendir: Permission denied) 2024-03-18T15:20:22.838 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 88457 fn_anonym ceph cwd unknown /proc/31759/task/88457/cwd (readlink: Permission denied) 2024-03-18T15:20:22.838 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 88457 fn_anonym ceph rtd unknown /proc/31759/task/88457/root (readlink: Permission denied) 2024-03-18T15:20:22.838 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 88457 fn_anonym ceph txt unknown /proc/31759/task/88457/exe (readlink: Permission denied) 2024-03-18T15:20:22.838 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 88457 fn_anonym ceph NOFD /proc/31759/task/88457/fd (opendir: Permission denied) 2024-03-18T15:20:22.838 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 88458 flusher ceph cwd unknown /proc/31759/task/88458/cwd (readlink: Permission denied) 2024-03-18T15:20:22.838 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 88458 flusher ceph rtd unknown /proc/31759/task/88458/root (readlink: Permission denied) 2024-03-18T15:20:22.839 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 88458 flusher ceph txt unknown /proc/31759/task/88458/exe (readlink: Permission denied) 2024-03-18T15:20:22.839 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 88458 flusher ceph NOFD /proc/31759/task/88458/fd (opendir: Permission denied) 2024-03-18T15:20:22.839 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 88459 ms_dispat ceph cwd unknown /proc/31759/task/88459/cwd (readlink: Permission denied) 2024-03-18T15:20:22.839 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 88459 ms_dispat ceph rtd unknown /proc/31759/task/88459/root (readlink: Permission denied) 2024-03-18T15:20:22.839 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 88459 ms_dispat ceph txt unknown /proc/31759/task/88459/exe (readlink: Permission denied) 2024-03-18T15:20:22.839 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 88459 ms_dispat ceph NOFD /proc/31759/task/88459/fd (opendir: Permission denied) 2024-03-18T15:20:22.839 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 88460 ms_local ceph cwd unknown /proc/31759/task/88460/cwd (readlink: Permission denied) 2024-03-18T15:20:22.839 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 88460 ms_local ceph rtd unknown /proc/31759/task/88460/root (readlink: Permission denied) 2024-03-18T15:20:22.839 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 88460 ms_local ceph txt unknown /proc/31759/task/88460/exe (readlink: Permission denied) 2024-03-18T15:20:22.839 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 88460 ms_local ceph NOFD /proc/31759/task/88460/fd (opendir: Permission denied) 2024-03-18T15:20:22.839 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 88461 safe_time ceph cwd unknown /proc/31759/task/88461/cwd (readlink: Permission denied) 2024-03-18T15:20:22.839 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 88461 safe_time ceph rtd unknown /proc/31759/task/88461/root (readlink: Permission denied) 2024-03-18T15:20:22.839 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 88461 safe_time ceph txt unknown /proc/31759/task/88461/exe (readlink: Permission denied) 2024-03-18T15:20:22.840 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 88461 safe_time ceph NOFD /proc/31759/task/88461/fd (opendir: Permission denied) 2024-03-18T15:20:22.840 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 88462 m-fin-sna ceph cwd unknown /proc/31759/task/88462/cwd (readlink: Permission denied) 2024-03-18T15:20:22.840 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 88462 m-fin-sna ceph rtd unknown /proc/31759/task/88462/root (readlink: Permission denied) 2024-03-18T15:20:22.840 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 88462 m-fin-sna ceph txt unknown /proc/31759/task/88462/exe (readlink: Permission denied) 2024-03-18T15:20:22.840 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 88462 m-fin-sna ceph NOFD /proc/31759/task/88462/fd (opendir: Permission denied) 2024-03-18T15:20:22.840 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 88696 m-fin-sna ceph cwd unknown /proc/31759/task/88696/cwd (readlink: Permission denied) 2024-03-18T15:20:22.840 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 88696 m-fin-sna ceph rtd unknown /proc/31759/task/88696/root (readlink: Permission denied) 2024-03-18T15:20:22.840 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 88696 m-fin-sna ceph txt unknown /proc/31759/task/88696/exe (readlink: Permission denied) 2024-03-18T15:20:22.840 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mgr 31759 88696 m-fin-sna ceph NOFD /proc/31759/task/88696/fd (opendir: Permission denied) 2024-03-18T15:20:22.840 INFO:tasks.workunit.client.0.smithi019.stdout:systemd 32889 root cwd unknown /proc/32889/cwd (readlink: Permission denied) 2024-03-18T15:20:22.840 INFO:tasks.workunit.client.0.smithi019.stdout:systemd 32889 root rtd unknown /proc/32889/root (readlink: Permission denied) 2024-03-18T15:20:22.840 INFO:tasks.workunit.client.0.smithi019.stdout:systemd 32889 root txt unknown /proc/32889/exe (readlink: Permission denied) 2024-03-18T15:20:22.840 INFO:tasks.workunit.client.0.smithi019.stdout:systemd 32889 root NOFD /proc/32889/fd (opendir: Permission denied) 2024-03-18T15:20:22.840 INFO:tasks.workunit.client.0.smithi019.stdout:(sd-pam) 32892 root cwd unknown /proc/32892/cwd (readlink: Permission denied) 2024-03-18T15:20:22.840 INFO:tasks.workunit.client.0.smithi019.stdout:(sd-pam) 32892 root rtd unknown /proc/32892/root (readlink: Permission denied) 2024-03-18T15:20:22.841 INFO:tasks.workunit.client.0.smithi019.stdout:(sd-pam) 32892 root txt unknown /proc/32892/exe (readlink: Permission denied) 2024-03-18T15:20:22.841 INFO:tasks.workunit.client.0.smithi019.stdout:(sd-pam) 32892 root NOFD /proc/32892/fd (opendir: Permission denied) 2024-03-18T15:20:22.841 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/2 34252 root cwd unknown /proc/34252/cwd (readlink: Permission denied) 2024-03-18T15:20:22.841 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/2 34252 root rtd unknown /proc/34252/root (readlink: Permission denied) 2024-03-18T15:20:22.841 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/2 34252 root txt unknown /proc/34252/exe (readlink: Permission denied) 2024-03-18T15:20:22.841 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/2 34252 root NOFD /proc/34252/fd (opendir: Permission denied) 2024-03-18T15:20:22.841 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/3 37819 root cwd unknown /proc/37819/cwd (readlink: Permission denied) 2024-03-18T15:20:22.841 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/3 37819 root rtd unknown /proc/37819/root (readlink: Permission denied) 2024-03-18T15:20:22.841 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/3 37819 root txt unknown /proc/37819/exe (readlink: Permission denied) 2024-03-18T15:20:22.841 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/3 37819 root NOFD /proc/37819/fd (opendir: Permission denied) 2024-03-18T15:20:22.841 INFO:tasks.workunit.client.0.smithi019.stdout:conmon 45979 root cwd unknown /proc/45979/cwd (readlink: Permission denied) 2024-03-18T15:20:22.841 INFO:tasks.workunit.client.0.smithi019.stdout:conmon 45979 root rtd unknown /proc/45979/root (readlink: Permission denied) 2024-03-18T15:20:22.841 INFO:tasks.workunit.client.0.smithi019.stdout:conmon 45979 root txt unknown /proc/45979/exe (readlink: Permission denied) 2024-03-18T15:20:22.841 INFO:tasks.workunit.client.0.smithi019.stdout:conmon 45979 root NOFD /proc/45979/fd (opendir: Permission denied) 2024-03-18T15:20:22.842 INFO:tasks.workunit.client.0.smithi019.stdout:podman-in 45981 root cwd unknown /proc/45981/cwd (readlink: Permission denied) 2024-03-18T15:20:22.842 INFO:tasks.workunit.client.0.smithi019.stdout:podman-in 45981 root rtd unknown /proc/45981/root (readlink: Permission denied) 2024-03-18T15:20:22.842 INFO:tasks.workunit.client.0.smithi019.stdout:podman-in 45981 root txt unknown /proc/45981/exe (readlink: Permission denied) 2024-03-18T15:20:22.842 INFO:tasks.workunit.client.0.smithi019.stdout:podman-in 45981 root NOFD /proc/45981/fd (opendir: Permission denied) 2024-03-18T15:20:22.842 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 ceph cwd unknown /proc/45983/cwd (readlink: Permission denied) 2024-03-18T15:20:22.842 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 ceph rtd unknown /proc/45983/root (readlink: Permission denied) 2024-03-18T15:20:22.842 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 ceph txt unknown /proc/45983/exe (readlink: Permission denied) 2024-03-18T15:20:22.842 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 ceph NOFD /proc/45983/fd (opendir: Permission denied) 2024-03-18T15:20:22.842 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 45984 log ceph cwd unknown /proc/45983/task/45984/cwd (readlink: Permission denied) 2024-03-18T15:20:22.842 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 45984 log ceph rtd unknown /proc/45983/task/45984/root (readlink: Permission denied) 2024-03-18T15:20:22.842 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 45984 log ceph txt unknown /proc/45983/task/45984/exe (readlink: Permission denied) 2024-03-18T15:20:22.842 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 45984 log ceph NOFD /proc/45983/task/45984/fd (opendir: Permission denied) 2024-03-18T15:20:22.842 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 45986 msgr-work ceph cwd unknown /proc/45983/task/45986/cwd (readlink: Permission denied) 2024-03-18T15:20:22.843 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 45986 msgr-work ceph rtd unknown /proc/45983/task/45986/root (readlink: Permission denied) 2024-03-18T15:20:22.843 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 45986 msgr-work ceph txt unknown /proc/45983/task/45986/exe (readlink: Permission denied) 2024-03-18T15:20:22.843 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 45986 msgr-work ceph NOFD /proc/45983/task/45986/fd (opendir: Permission denied) 2024-03-18T15:20:22.843 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 45987 msgr-work ceph cwd unknown /proc/45983/task/45987/cwd (readlink: Permission denied) 2024-03-18T15:20:22.843 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 45987 msgr-work ceph rtd unknown /proc/45983/task/45987/root (readlink: Permission denied) 2024-03-18T15:20:22.843 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 45987 msgr-work ceph txt unknown /proc/45983/task/45987/exe (readlink: Permission denied) 2024-03-18T15:20:22.843 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 45987 msgr-work ceph NOFD /proc/45983/task/45987/fd (opendir: Permission denied) 2024-03-18T15:20:22.843 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 45988 msgr-work ceph cwd unknown /proc/45983/task/45988/cwd (readlink: Permission denied) 2024-03-18T15:20:22.843 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 45988 msgr-work ceph rtd unknown /proc/45983/task/45988/root (readlink: Permission denied) 2024-03-18T15:20:22.843 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 45988 msgr-work ceph txt unknown /proc/45983/task/45988/exe (readlink: Permission denied) 2024-03-18T15:20:22.843 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 45988 msgr-work ceph NOFD /proc/45983/task/45988/fd (opendir: Permission denied) 2024-03-18T15:20:22.843 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 45992 service ceph cwd unknown /proc/45983/task/45992/cwd (readlink: Permission denied) 2024-03-18T15:20:22.843 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 45992 service ceph rtd unknown /proc/45983/task/45992/root (readlink: Permission denied) 2024-03-18T15:20:22.843 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 45992 service ceph txt unknown /proc/45983/task/45992/exe (readlink: Permission denied) 2024-03-18T15:20:22.844 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 45992 service ceph NOFD /proc/45983/task/45992/fd (opendir: Permission denied) 2024-03-18T15:20:22.844 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 45993 admin_soc ceph cwd unknown /proc/45983/task/45993/cwd (readlink: Permission denied) 2024-03-18T15:20:22.844 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 45993 admin_soc ceph rtd unknown /proc/45983/task/45993/root (readlink: Permission denied) 2024-03-18T15:20:22.844 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 45993 admin_soc ceph txt unknown /proc/45983/task/45993/exe (readlink: Permission denied) 2024-03-18T15:20:22.844 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 45993 admin_soc ceph NOFD /proc/45983/task/45993/fd (opendir: Permission denied) 2024-03-18T15:20:22.844 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 46136 signal_ha ceph cwd unknown /proc/45983/task/46136/cwd (readlink: Permission denied) 2024-03-18T15:20:22.844 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 46136 signal_ha ceph rtd unknown /proc/45983/task/46136/root (readlink: Permission denied) 2024-03-18T15:20:22.844 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 46136 signal_ha ceph txt unknown /proc/45983/task/46136/exe (readlink: Permission denied) 2024-03-18T15:20:22.844 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 46136 signal_ha ceph NOFD /proc/45983/task/46136/fd (opendir: Permission denied) 2024-03-18T15:20:22.844 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 46137 io_contex ceph cwd unknown /proc/45983/task/46137/cwd (readlink: Permission denied) 2024-03-18T15:20:22.844 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 46137 io_contex ceph rtd unknown /proc/45983/task/46137/root (readlink: Permission denied) 2024-03-18T15:20:22.844 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 46137 io_contex ceph txt unknown /proc/45983/task/46137/exe (readlink: Permission denied) 2024-03-18T15:20:22.844 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 46137 io_contex ceph NOFD /proc/45983/task/46137/fd (opendir: Permission denied) 2024-03-18T15:20:22.844 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 46138 io_contex ceph cwd unknown /proc/45983/task/46138/cwd (readlink: Permission denied) 2024-03-18T15:20:22.844 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 46138 io_contex ceph rtd unknown /proc/45983/task/46138/root (readlink: Permission denied) 2024-03-18T15:20:22.845 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 46138 io_contex ceph txt unknown /proc/45983/task/46138/exe (readlink: Permission denied) 2024-03-18T15:20:22.845 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 46138 io_contex ceph NOFD /proc/45983/task/46138/fd (opendir: Permission denied) 2024-03-18T15:20:22.845 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 46156 OpHistory ceph cwd unknown /proc/45983/task/46156/cwd (readlink: Permission denied) 2024-03-18T15:20:22.845 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 46156 OpHistory ceph rtd unknown /proc/45983/task/46156/root (readlink: Permission denied) 2024-03-18T15:20:22.845 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 46156 OpHistory ceph txt unknown /proc/45983/task/46156/exe (readlink: Permission denied) 2024-03-18T15:20:22.845 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 46156 OpHistory ceph NOFD /proc/45983/task/46156/fd (opendir: Permission denied) 2024-03-18T15:20:22.845 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 46157 ceph_time ceph cwd unknown /proc/45983/task/46157/cwd (readlink: Permission denied) 2024-03-18T15:20:22.845 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 46157 ceph_time ceph rtd unknown /proc/45983/task/46157/root (readlink: Permission denied) 2024-03-18T15:20:22.845 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 46157 ceph_time ceph txt unknown /proc/45983/task/46157/exe (readlink: Permission denied) 2024-03-18T15:20:22.845 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 46157 ceph_time ceph NOFD /proc/45983/task/46157/fd (opendir: Permission denied) 2024-03-18T15:20:22.845 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 46374 ceph-osd ceph cwd unknown /proc/45983/task/46374/cwd (readlink: Permission denied) 2024-03-18T15:20:22.845 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 46374 ceph-osd ceph rtd unknown /proc/45983/task/46374/root (readlink: Permission denied) 2024-03-18T15:20:22.845 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 46374 ceph-osd ceph txt unknown /proc/45983/task/46374/exe (readlink: Permission denied) 2024-03-18T15:20:22.846 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 46374 ceph-osd ceph NOFD /proc/45983/task/46374/fd (opendir: Permission denied) 2024-03-18T15:20:22.846 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 46375 safe_time ceph cwd unknown /proc/45983/task/46375/cwd (readlink: Permission denied) 2024-03-18T15:20:22.846 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 46375 safe_time ceph rtd unknown /proc/45983/task/46375/root (readlink: Permission denied) 2024-03-18T15:20:22.846 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 46375 safe_time ceph txt unknown /proc/45983/task/46375/exe (readlink: Permission denied) 2024-03-18T15:20:22.846 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 46375 safe_time ceph NOFD /proc/45983/task/46375/fd (opendir: Permission denied) 2024-03-18T15:20:22.846 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 46376 safe_time ceph cwd unknown /proc/45983/task/46376/cwd (readlink: Permission denied) 2024-03-18T15:20:22.846 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 46376 safe_time ceph rtd unknown /proc/45983/task/46376/root (readlink: Permission denied) 2024-03-18T15:20:22.846 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 46376 safe_time ceph txt unknown /proc/45983/task/46376/exe (readlink: Permission denied) 2024-03-18T15:20:22.846 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 46376 safe_time ceph NOFD /proc/45983/task/46376/fd (opendir: Permission denied) 2024-03-18T15:20:22.846 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 46377 safe_time ceph cwd unknown /proc/45983/task/46377/cwd (readlink: Permission denied) 2024-03-18T15:20:22.846 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 46377 safe_time ceph rtd unknown /proc/45983/task/46377/root (readlink: Permission denied) 2024-03-18T15:20:22.846 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 46377 safe_time ceph txt unknown /proc/45983/task/46377/exe (readlink: Permission denied) 2024-03-18T15:20:22.846 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 46377 safe_time ceph NOFD /proc/45983/task/46377/fd (opendir: Permission denied) 2024-03-18T15:20:22.846 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 46378 safe_time ceph cwd unknown /proc/45983/task/46378/cwd (readlink: Permission denied) 2024-03-18T15:20:22.847 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 46378 safe_time ceph rtd unknown /proc/45983/task/46378/root (readlink: Permission denied) 2024-03-18T15:20:22.847 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 46378 safe_time ceph txt unknown /proc/45983/task/46378/exe (readlink: Permission denied) 2024-03-18T15:20:22.847 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 46378 safe_time ceph NOFD /proc/45983/task/46378/fd (opendir: Permission denied) 2024-03-18T15:20:22.847 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 46379 fn_anonym ceph cwd unknown /proc/45983/task/46379/cwd (readlink: Permission denied) 2024-03-18T15:20:22.847 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 46379 fn_anonym ceph rtd unknown /proc/45983/task/46379/root (readlink: Permission denied) 2024-03-18T15:20:22.847 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 46379 fn_anonym ceph txt unknown /proc/45983/task/46379/exe (readlink: Permission denied) 2024-03-18T15:20:22.847 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 46379 fn_anonym ceph NOFD /proc/45983/task/46379/fd (opendir: Permission denied) 2024-03-18T15:20:22.847 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 46385 rocksdb:l ceph cwd unknown /proc/45983/task/46385/cwd (readlink: Permission denied) 2024-03-18T15:20:22.847 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 46385 rocksdb:l ceph rtd unknown /proc/45983/task/46385/root (readlink: Permission denied) 2024-03-18T15:20:22.847 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 46385 rocksdb:l ceph txt unknown /proc/45983/task/46385/exe (readlink: Permission denied) 2024-03-18T15:20:22.847 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 46385 rocksdb:l ceph NOFD /proc/45983/task/46385/fd (opendir: Permission denied) 2024-03-18T15:20:22.847 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 46386 rocksdb:l ceph cwd unknown /proc/45983/task/46386/cwd (readlink: Permission denied) 2024-03-18T15:20:22.847 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 46386 rocksdb:l ceph rtd unknown /proc/45983/task/46386/root (readlink: Permission denied) 2024-03-18T15:20:22.847 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 46386 rocksdb:l ceph txt unknown /proc/45983/task/46386/exe (readlink: Permission denied) 2024-03-18T15:20:22.848 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 46386 rocksdb:l ceph NOFD /proc/45983/task/46386/fd (opendir: Permission denied) 2024-03-18T15:20:22.848 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 46387 rocksdb:l ceph cwd unknown /proc/45983/task/46387/cwd (readlink: Permission denied) 2024-03-18T15:20:22.848 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 46387 rocksdb:l ceph rtd unknown /proc/45983/task/46387/root (readlink: Permission denied) 2024-03-18T15:20:22.848 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 46387 rocksdb:l ceph txt unknown /proc/45983/task/46387/exe (readlink: Permission denied) 2024-03-18T15:20:22.848 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 46387 rocksdb:l ceph NOFD /proc/45983/task/46387/fd (opendir: Permission denied) 2024-03-18T15:20:22.848 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 46388 rocksdb:h ceph cwd unknown /proc/45983/task/46388/cwd (readlink: Permission denied) 2024-03-18T15:20:22.848 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 46388 rocksdb:h ceph rtd unknown /proc/45983/task/46388/root (readlink: Permission denied) 2024-03-18T15:20:22.848 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 46388 rocksdb:h ceph txt unknown /proc/45983/task/46388/exe (readlink: Permission denied) 2024-03-18T15:20:22.848 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 46388 rocksdb:h ceph NOFD /proc/45983/task/46388/fd (opendir: Permission denied) 2024-03-18T15:20:22.848 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 46826 bstore_ai ceph cwd unknown /proc/45983/task/46826/cwd (readlink: Permission denied) 2024-03-18T15:20:22.848 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 46826 bstore_ai ceph rtd unknown /proc/45983/task/46826/root (readlink: Permission denied) 2024-03-18T15:20:22.848 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 46826 bstore_ai ceph txt unknown /proc/45983/task/46826/exe (readlink: Permission denied) 2024-03-18T15:20:22.848 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 46826 bstore_ai ceph NOFD /proc/45983/task/46826/fd (opendir: Permission denied) 2024-03-18T15:20:22.848 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47021 bstore_ai ceph cwd unknown /proc/45983/task/47021/cwd (readlink: Permission denied) 2024-03-18T15:20:22.848 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47021 bstore_ai ceph rtd unknown /proc/45983/task/47021/root (readlink: Permission denied) 2024-03-18T15:20:22.849 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47021 bstore_ai ceph txt unknown /proc/45983/task/47021/exe (readlink: Permission denied) 2024-03-18T15:20:22.849 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47021 bstore_ai ceph NOFD /proc/45983/task/47021/fd (opendir: Permission denied) 2024-03-18T15:20:22.849 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47207 ceph-osd ceph cwd unknown /proc/45983/task/47207/cwd (readlink: Permission denied) 2024-03-18T15:20:22.849 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47207 ceph-osd ceph rtd unknown /proc/45983/task/47207/root (readlink: Permission denied) 2024-03-18T15:20:22.849 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47207 ceph-osd ceph txt unknown /proc/45983/task/47207/exe (readlink: Permission denied) 2024-03-18T15:20:22.849 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47207 ceph-osd ceph NOFD /proc/45983/task/47207/fd (opendir: Permission denied) 2024-03-18T15:20:22.849 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47208 cfin ceph cwd unknown /proc/45983/task/47208/cwd (readlink: Permission denied) 2024-03-18T15:20:22.849 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47208 cfin ceph rtd unknown /proc/45983/task/47208/root (readlink: Permission denied) 2024-03-18T15:20:22.849 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47208 cfin ceph txt unknown /proc/45983/task/47208/exe (readlink: Permission denied) 2024-03-18T15:20:22.849 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47208 cfin ceph NOFD /proc/45983/task/47208/fd (opendir: Permission denied) 2024-03-18T15:20:22.849 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47209 bstore_kv ceph cwd unknown /proc/45983/task/47209/cwd (readlink: Permission denied) 2024-03-18T15:20:22.849 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47209 bstore_kv ceph rtd unknown /proc/45983/task/47209/root (readlink: Permission denied) 2024-03-18T15:20:22.849 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47209 bstore_kv ceph txt unknown /proc/45983/task/47209/exe (readlink: Permission denied) 2024-03-18T15:20:22.849 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47209 bstore_kv ceph NOFD /proc/45983/task/47209/fd (opendir: Permission denied) 2024-03-18T15:20:22.850 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47210 bstore_kv ceph cwd unknown /proc/45983/task/47210/cwd (readlink: Permission denied) 2024-03-18T15:20:22.850 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47210 bstore_kv ceph rtd unknown /proc/45983/task/47210/root (readlink: Permission denied) 2024-03-18T15:20:22.850 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47210 bstore_kv ceph txt unknown /proc/45983/task/47210/exe (readlink: Permission denied) 2024-03-18T15:20:22.850 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47210 bstore_kv ceph NOFD /proc/45983/task/47210/fd (opendir: Permission denied) 2024-03-18T15:20:22.850 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47211 bstore_me ceph cwd unknown /proc/45983/task/47211/cwd (readlink: Permission denied) 2024-03-18T15:20:22.850 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47211 bstore_me ceph rtd unknown /proc/45983/task/47211/root (readlink: Permission denied) 2024-03-18T15:20:22.850 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47211 bstore_me ceph txt unknown /proc/45983/task/47211/exe (readlink: Permission denied) 2024-03-18T15:20:22.850 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47211 bstore_me ceph NOFD /proc/45983/task/47211/fd (opendir: Permission denied) 2024-03-18T15:20:22.850 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47215 ms_dispat ceph cwd unknown /proc/45983/task/47215/cwd (readlink: Permission denied) 2024-03-18T15:20:22.850 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47215 ms_dispat ceph rtd unknown /proc/45983/task/47215/root (readlink: Permission denied) 2024-03-18T15:20:22.850 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47215 ms_dispat ceph txt unknown /proc/45983/task/47215/exe (readlink: Permission denied) 2024-03-18T15:20:22.850 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47215 ms_dispat ceph NOFD /proc/45983/task/47215/fd (opendir: Permission denied) 2024-03-18T15:20:22.850 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47216 ms_local ceph cwd unknown /proc/45983/task/47216/cwd (readlink: Permission denied) 2024-03-18T15:20:22.851 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47216 ms_local ceph rtd unknown /proc/45983/task/47216/root (readlink: Permission denied) 2024-03-18T15:20:22.851 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47216 ms_local ceph txt unknown /proc/45983/task/47216/exe (readlink: Permission denied) 2024-03-18T15:20:22.851 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47216 ms_local ceph NOFD /proc/45983/task/47216/fd (opendir: Permission denied) 2024-03-18T15:20:22.851 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47217 safe_time ceph cwd unknown /proc/45983/task/47217/cwd (readlink: Permission denied) 2024-03-18T15:20:22.851 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47217 safe_time ceph rtd unknown /proc/45983/task/47217/root (readlink: Permission denied) 2024-03-18T15:20:22.851 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47217 safe_time ceph txt unknown /proc/45983/task/47217/exe (readlink: Permission denied) 2024-03-18T15:20:22.851 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47217 safe_time ceph NOFD /proc/45983/task/47217/fd (opendir: Permission denied) 2024-03-18T15:20:22.851 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47218 safe_time ceph cwd unknown /proc/45983/task/47218/cwd (readlink: Permission denied) 2024-03-18T15:20:22.851 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47218 safe_time ceph rtd unknown /proc/45983/task/47218/root (readlink: Permission denied) 2024-03-18T15:20:22.851 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47218 safe_time ceph txt unknown /proc/45983/task/47218/exe (readlink: Permission denied) 2024-03-18T15:20:22.851 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47218 safe_time ceph NOFD /proc/45983/task/47218/fd (opendir: Permission denied) 2024-03-18T15:20:22.851 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47219 ms_dispat ceph cwd unknown /proc/45983/task/47219/cwd (readlink: Permission denied) 2024-03-18T15:20:22.851 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47219 ms_dispat ceph rtd unknown /proc/45983/task/47219/root (readlink: Permission denied) 2024-03-18T15:20:22.851 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47219 ms_dispat ceph txt unknown /proc/45983/task/47219/exe (readlink: Permission denied) 2024-03-18T15:20:22.852 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47219 ms_dispat ceph NOFD /proc/45983/task/47219/fd (opendir: Permission denied) 2024-03-18T15:20:22.852 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47220 ms_local ceph cwd unknown /proc/45983/task/47220/cwd (readlink: Permission denied) 2024-03-18T15:20:22.852 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47220 ms_local ceph rtd unknown /proc/45983/task/47220/root (readlink: Permission denied) 2024-03-18T15:20:22.852 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47220 ms_local ceph txt unknown /proc/45983/task/47220/exe (readlink: Permission denied) 2024-03-18T15:20:22.852 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47220 ms_local ceph NOFD /proc/45983/task/47220/fd (opendir: Permission denied) 2024-03-18T15:20:22.852 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47221 ms_dispat ceph cwd unknown /proc/45983/task/47221/cwd (readlink: Permission denied) 2024-03-18T15:20:22.852 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47221 ms_dispat ceph rtd unknown /proc/45983/task/47221/root (readlink: Permission denied) 2024-03-18T15:20:22.852 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47221 ms_dispat ceph txt unknown /proc/45983/task/47221/exe (readlink: Permission denied) 2024-03-18T15:20:22.852 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47221 ms_dispat ceph NOFD /proc/45983/task/47221/fd (opendir: Permission denied) 2024-03-18T15:20:22.852 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47222 ms_local ceph cwd unknown /proc/45983/task/47222/cwd (readlink: Permission denied) 2024-03-18T15:20:22.852 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47222 ms_local ceph rtd unknown /proc/45983/task/47222/root (readlink: Permission denied) 2024-03-18T15:20:22.852 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47222 ms_local ceph txt unknown /proc/45983/task/47222/exe (readlink: Permission denied) 2024-03-18T15:20:22.852 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47222 ms_local ceph NOFD /proc/45983/task/47222/fd (opendir: Permission denied) 2024-03-18T15:20:22.852 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47223 ms_dispat ceph cwd unknown /proc/45983/task/47223/cwd (readlink: Permission denied) 2024-03-18T15:20:22.852 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47223 ms_dispat ceph rtd unknown /proc/45983/task/47223/root (readlink: Permission denied) 2024-03-18T15:20:22.853 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47223 ms_dispat ceph txt unknown /proc/45983/task/47223/exe (readlink: Permission denied) 2024-03-18T15:20:22.853 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47223 ms_dispat ceph NOFD /proc/45983/task/47223/fd (opendir: Permission denied) 2024-03-18T15:20:22.853 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47224 ms_local ceph cwd unknown /proc/45983/task/47224/cwd (readlink: Permission denied) 2024-03-18T15:20:22.853 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47224 ms_local ceph rtd unknown /proc/45983/task/47224/root (readlink: Permission denied) 2024-03-18T15:20:22.853 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47224 ms_local ceph txt unknown /proc/45983/task/47224/exe (readlink: Permission denied) 2024-03-18T15:20:22.853 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47224 ms_local ceph NOFD /proc/45983/task/47224/fd (opendir: Permission denied) 2024-03-18T15:20:22.853 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47225 ms_dispat ceph cwd unknown /proc/45983/task/47225/cwd (readlink: Permission denied) 2024-03-18T15:20:22.853 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47225 ms_dispat ceph rtd unknown /proc/45983/task/47225/root (readlink: Permission denied) 2024-03-18T15:20:22.853 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47225 ms_dispat ceph txt unknown /proc/45983/task/47225/exe (readlink: Permission denied) 2024-03-18T15:20:22.853 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47225 ms_dispat ceph NOFD /proc/45983/task/47225/fd (opendir: Permission denied) 2024-03-18T15:20:22.853 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47226 ms_local ceph cwd unknown /proc/45983/task/47226/cwd (readlink: Permission denied) 2024-03-18T15:20:22.853 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47226 ms_local ceph rtd unknown /proc/45983/task/47226/root (readlink: Permission denied) 2024-03-18T15:20:22.853 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47226 ms_local ceph txt unknown /proc/45983/task/47226/exe (readlink: Permission denied) 2024-03-18T15:20:22.853 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47226 ms_local ceph NOFD /proc/45983/task/47226/fd (opendir: Permission denied) 2024-03-18T15:20:22.854 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47227 ms_dispat ceph cwd unknown /proc/45983/task/47227/cwd (readlink: Permission denied) 2024-03-18T15:20:22.854 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47227 ms_dispat ceph rtd unknown /proc/45983/task/47227/root (readlink: Permission denied) 2024-03-18T15:20:22.854 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47227 ms_dispat ceph txt unknown /proc/45983/task/47227/exe (readlink: Permission denied) 2024-03-18T15:20:22.854 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47227 ms_dispat ceph NOFD /proc/45983/task/47227/fd (opendir: Permission denied) 2024-03-18T15:20:22.854 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47228 ms_local ceph cwd unknown /proc/45983/task/47228/cwd (readlink: Permission denied) 2024-03-18T15:20:22.854 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47228 ms_local ceph rtd unknown /proc/45983/task/47228/root (readlink: Permission denied) 2024-03-18T15:20:22.854 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47228 ms_local ceph txt unknown /proc/45983/task/47228/exe (readlink: Permission denied) 2024-03-18T15:20:22.854 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47228 ms_local ceph NOFD /proc/45983/task/47228/fd (opendir: Permission denied) 2024-03-18T15:20:22.854 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47229 ms_dispat ceph cwd unknown /proc/45983/task/47229/cwd (readlink: Permission denied) 2024-03-18T15:20:22.854 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47229 ms_dispat ceph rtd unknown /proc/45983/task/47229/root (readlink: Permission denied) 2024-03-18T15:20:22.854 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47229 ms_dispat ceph txt unknown /proc/45983/task/47229/exe (readlink: Permission denied) 2024-03-18T15:20:22.854 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47229 ms_dispat ceph NOFD /proc/45983/task/47229/fd (opendir: Permission denied) 2024-03-18T15:20:22.854 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47230 ms_local ceph cwd unknown /proc/45983/task/47230/cwd (readlink: Permission denied) 2024-03-18T15:20:22.854 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47230 ms_local ceph rtd unknown /proc/45983/task/47230/root (readlink: Permission denied) 2024-03-18T15:20:22.855 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47230 ms_local ceph txt unknown /proc/45983/task/47230/exe (readlink: Permission denied) 2024-03-18T15:20:22.855 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47230 ms_local ceph NOFD /proc/45983/task/47230/fd (opendir: Permission denied) 2024-03-18T15:20:22.855 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47231 fn_anonym ceph cwd unknown /proc/45983/task/47231/cwd (readlink: Permission denied) 2024-03-18T15:20:22.855 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47231 fn_anonym ceph rtd unknown /proc/45983/task/47231/root (readlink: Permission denied) 2024-03-18T15:20:22.855 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47231 fn_anonym ceph txt unknown /proc/45983/task/47231/exe (readlink: Permission denied) 2024-03-18T15:20:22.855 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47231 fn_anonym ceph NOFD /proc/45983/task/47231/fd (opendir: Permission denied) 2024-03-18T15:20:22.855 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47232 finisher ceph cwd unknown /proc/45983/task/47232/cwd (readlink: Permission denied) 2024-03-18T15:20:22.855 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47232 finisher ceph rtd unknown /proc/45983/task/47232/root (readlink: Permission denied) 2024-03-18T15:20:22.855 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47232 finisher ceph txt unknown /proc/45983/task/47232/exe (readlink: Permission denied) 2024-03-18T15:20:22.855 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47232 finisher ceph NOFD /proc/45983/task/47232/fd (opendir: Permission denied) 2024-03-18T15:20:22.855 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47233 safe_time ceph cwd unknown /proc/45983/task/47233/cwd (readlink: Permission denied) 2024-03-18T15:20:22.855 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47233 safe_time ceph rtd unknown /proc/45983/task/47233/root (readlink: Permission denied) 2024-03-18T15:20:22.855 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47233 safe_time ceph txt unknown /proc/45983/task/47233/exe (readlink: Permission denied) 2024-03-18T15:20:22.855 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47233 safe_time ceph NOFD /proc/45983/task/47233/fd (opendir: Permission denied) 2024-03-18T15:20:22.856 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47234 safe_time ceph cwd unknown /proc/45983/task/47234/cwd (readlink: Permission denied) 2024-03-18T15:20:22.856 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47234 safe_time ceph rtd unknown /proc/45983/task/47234/root (readlink: Permission denied) 2024-03-18T15:20:22.856 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47234 safe_time ceph txt unknown /proc/45983/task/47234/exe (readlink: Permission denied) 2024-03-18T15:20:22.856 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47234 safe_time ceph NOFD /proc/45983/task/47234/fd (opendir: Permission denied) 2024-03-18T15:20:22.856 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47235 ceph-osd ceph cwd unknown /proc/45983/task/47235/cwd (readlink: Permission denied) 2024-03-18T15:20:22.856 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47235 ceph-osd ceph rtd unknown /proc/45983/task/47235/root (readlink: Permission denied) 2024-03-18T15:20:22.856 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47235 ceph-osd ceph txt unknown /proc/45983/task/47235/exe (readlink: Permission denied) 2024-03-18T15:20:22.856 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47235 ceph-osd ceph NOFD /proc/45983/task/47235/fd (opendir: Permission denied) 2024-03-18T15:20:22.856 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47236 osd_srv_a ceph cwd unknown /proc/45983/task/47236/cwd (readlink: Permission denied) 2024-03-18T15:20:22.856 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47236 osd_srv_a ceph rtd unknown /proc/45983/task/47236/root (readlink: Permission denied) 2024-03-18T15:20:22.856 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47236 osd_srv_a ceph txt unknown /proc/45983/task/47236/exe (readlink: Permission denied) 2024-03-18T15:20:22.856 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47236 osd_srv_a ceph NOFD /proc/45983/task/47236/fd (opendir: Permission denied) 2024-03-18T15:20:22.856 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47237 tp_osd_tp ceph cwd unknown /proc/45983/task/47237/cwd (readlink: Permission denied) 2024-03-18T15:20:22.856 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47237 tp_osd_tp ceph rtd unknown /proc/45983/task/47237/root (readlink: Permission denied) 2024-03-18T15:20:22.857 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47237 tp_osd_tp ceph txt unknown /proc/45983/task/47237/exe (readlink: Permission denied) 2024-03-18T15:20:22.857 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47237 tp_osd_tp ceph NOFD /proc/45983/task/47237/fd (opendir: Permission denied) 2024-03-18T15:20:22.857 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47238 tp_osd_tp ceph cwd unknown /proc/45983/task/47238/cwd (readlink: Permission denied) 2024-03-18T15:20:22.857 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47238 tp_osd_tp ceph rtd unknown /proc/45983/task/47238/root (readlink: Permission denied) 2024-03-18T15:20:22.857 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47238 tp_osd_tp ceph txt unknown /proc/45983/task/47238/exe (readlink: Permission denied) 2024-03-18T15:20:22.857 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47238 tp_osd_tp ceph NOFD /proc/45983/task/47238/fd (opendir: Permission denied) 2024-03-18T15:20:22.857 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47239 tp_osd_tp ceph cwd unknown /proc/45983/task/47239/cwd (readlink: Permission denied) 2024-03-18T15:20:22.857 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47239 tp_osd_tp ceph rtd unknown /proc/45983/task/47239/root (readlink: Permission denied) 2024-03-18T15:20:22.857 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47239 tp_osd_tp ceph txt unknown /proc/45983/task/47239/exe (readlink: Permission denied) 2024-03-18T15:20:22.857 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47239 tp_osd_tp ceph NOFD /proc/45983/task/47239/fd (opendir: Permission denied) 2024-03-18T15:20:22.857 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47240 tp_osd_tp ceph cwd unknown /proc/45983/task/47240/cwd (readlink: Permission denied) 2024-03-18T15:20:22.857 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47240 tp_osd_tp ceph rtd unknown /proc/45983/task/47240/root (readlink: Permission denied) 2024-03-18T15:20:22.857 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47240 tp_osd_tp ceph txt unknown /proc/45983/task/47240/exe (readlink: Permission denied) 2024-03-18T15:20:22.857 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47240 tp_osd_tp ceph NOFD /proc/45983/task/47240/fd (opendir: Permission denied) 2024-03-18T15:20:22.857 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47241 tp_osd_tp ceph cwd unknown /proc/45983/task/47241/cwd (readlink: Permission denied) 2024-03-18T15:20:22.858 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47241 tp_osd_tp ceph rtd unknown /proc/45983/task/47241/root (readlink: Permission denied) 2024-03-18T15:20:22.858 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47241 tp_osd_tp ceph txt unknown /proc/45983/task/47241/exe (readlink: Permission denied) 2024-03-18T15:20:22.858 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47241 tp_osd_tp ceph NOFD /proc/45983/task/47241/fd (opendir: Permission denied) 2024-03-18T15:20:22.858 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47242 tp_osd_tp ceph cwd unknown /proc/45983/task/47242/cwd (readlink: Permission denied) 2024-03-18T15:20:22.858 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47242 tp_osd_tp ceph rtd unknown /proc/45983/task/47242/root (readlink: Permission denied) 2024-03-18T15:20:22.858 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47242 tp_osd_tp ceph txt unknown /proc/45983/task/47242/exe (readlink: Permission denied) 2024-03-18T15:20:22.858 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47242 tp_osd_tp ceph NOFD /proc/45983/task/47242/fd (opendir: Permission denied) 2024-03-18T15:20:22.858 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47243 tp_osd_tp ceph cwd unknown /proc/45983/task/47243/cwd (readlink: Permission denied) 2024-03-18T15:20:22.858 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47243 tp_osd_tp ceph rtd unknown /proc/45983/task/47243/root (readlink: Permission denied) 2024-03-18T15:20:22.858 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47243 tp_osd_tp ceph txt unknown /proc/45983/task/47243/exe (readlink: Permission denied) 2024-03-18T15:20:22.858 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47243 tp_osd_tp ceph NOFD /proc/45983/task/47243/fd (opendir: Permission denied) 2024-03-18T15:20:22.858 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47244 tp_osd_tp ceph cwd unknown /proc/45983/task/47244/cwd (readlink: Permission denied) 2024-03-18T15:20:22.858 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47244 tp_osd_tp ceph rtd unknown /proc/45983/task/47244/root (readlink: Permission denied) 2024-03-18T15:20:22.858 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47244 tp_osd_tp ceph txt unknown /proc/45983/task/47244/exe (readlink: Permission denied) 2024-03-18T15:20:22.859 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47244 tp_osd_tp ceph NOFD /proc/45983/task/47244/fd (opendir: Permission denied) 2024-03-18T15:20:22.859 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47245 tp_osd_tp ceph cwd unknown /proc/45983/task/47245/cwd (readlink: Permission denied) 2024-03-18T15:20:22.859 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47245 tp_osd_tp ceph rtd unknown /proc/45983/task/47245/root (readlink: Permission denied) 2024-03-18T15:20:22.859 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47245 tp_osd_tp ceph txt unknown /proc/45983/task/47245/exe (readlink: Permission denied) 2024-03-18T15:20:22.859 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47245 tp_osd_tp ceph NOFD /proc/45983/task/47245/fd (opendir: Permission denied) 2024-03-18T15:20:22.859 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47246 tp_osd_tp ceph cwd unknown /proc/45983/task/47246/cwd (readlink: Permission denied) 2024-03-18T15:20:22.859 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47246 tp_osd_tp ceph rtd unknown /proc/45983/task/47246/root (readlink: Permission denied) 2024-03-18T15:20:22.859 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47246 tp_osd_tp ceph txt unknown /proc/45983/task/47246/exe (readlink: Permission denied) 2024-03-18T15:20:22.859 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47246 tp_osd_tp ceph NOFD /proc/45983/task/47246/fd (opendir: Permission denied) 2024-03-18T15:20:22.859 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47247 tp_osd_tp ceph cwd unknown /proc/45983/task/47247/cwd (readlink: Permission denied) 2024-03-18T15:20:22.859 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47247 tp_osd_tp ceph rtd unknown /proc/45983/task/47247/root (readlink: Permission denied) 2024-03-18T15:20:22.859 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47247 tp_osd_tp ceph txt unknown /proc/45983/task/47247/exe (readlink: Permission denied) 2024-03-18T15:20:22.859 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47247 tp_osd_tp ceph NOFD /proc/45983/task/47247/fd (opendir: Permission denied) 2024-03-18T15:20:22.859 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47248 tp_osd_tp ceph cwd unknown /proc/45983/task/47248/cwd (readlink: Permission denied) 2024-03-18T15:20:22.859 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47248 tp_osd_tp ceph rtd unknown /proc/45983/task/47248/root (readlink: Permission denied) 2024-03-18T15:20:22.860 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47248 tp_osd_tp ceph txt unknown /proc/45983/task/47248/exe (readlink: Permission denied) 2024-03-18T15:20:22.860 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47248 tp_osd_tp ceph NOFD /proc/45983/task/47248/fd (opendir: Permission denied) 2024-03-18T15:20:22.860 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47249 tp_osd_tp ceph cwd unknown /proc/45983/task/47249/cwd (readlink: Permission denied) 2024-03-18T15:20:22.860 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47249 tp_osd_tp ceph rtd unknown /proc/45983/task/47249/root (readlink: Permission denied) 2024-03-18T15:20:22.860 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47249 tp_osd_tp ceph txt unknown /proc/45983/task/47249/exe (readlink: Permission denied) 2024-03-18T15:20:22.860 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47249 tp_osd_tp ceph NOFD /proc/45983/task/47249/fd (opendir: Permission denied) 2024-03-18T15:20:22.860 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47250 tp_osd_tp ceph cwd unknown /proc/45983/task/47250/cwd (readlink: Permission denied) 2024-03-18T15:20:22.860 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47250 tp_osd_tp ceph rtd unknown /proc/45983/task/47250/root (readlink: Permission denied) 2024-03-18T15:20:22.860 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47250 tp_osd_tp ceph txt unknown /proc/45983/task/47250/exe (readlink: Permission denied) 2024-03-18T15:20:22.860 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47250 tp_osd_tp ceph NOFD /proc/45983/task/47250/fd (opendir: Permission denied) 2024-03-18T15:20:22.860 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47251 tp_osd_tp ceph cwd unknown /proc/45983/task/47251/cwd (readlink: Permission denied) 2024-03-18T15:20:22.860 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47251 tp_osd_tp ceph rtd unknown /proc/45983/task/47251/root (readlink: Permission denied) 2024-03-18T15:20:22.860 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47251 tp_osd_tp ceph txt unknown /proc/45983/task/47251/exe (readlink: Permission denied) 2024-03-18T15:20:22.860 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47251 tp_osd_tp ceph NOFD /proc/45983/task/47251/fd (opendir: Permission denied) 2024-03-18T15:20:22.861 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47252 tp_osd_tp ceph cwd unknown /proc/45983/task/47252/cwd (readlink: Permission denied) 2024-03-18T15:20:22.861 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47252 tp_osd_tp ceph rtd unknown /proc/45983/task/47252/root (readlink: Permission denied) 2024-03-18T15:20:22.861 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47252 tp_osd_tp ceph txt unknown /proc/45983/task/47252/exe (readlink: Permission denied) 2024-03-18T15:20:22.861 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47252 tp_osd_tp ceph NOFD /proc/45983/task/47252/fd (opendir: Permission denied) 2024-03-18T15:20:22.861 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47253 osd_srv_h ceph cwd unknown /proc/45983/task/47253/cwd (readlink: Permission denied) 2024-03-18T15:20:22.861 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47253 osd_srv_h ceph rtd unknown /proc/45983/task/47253/root (readlink: Permission denied) 2024-03-18T15:20:22.861 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47253 osd_srv_h ceph txt unknown /proc/45983/task/47253/exe (readlink: Permission denied) 2024-03-18T15:20:22.861 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 45983 47253 osd_srv_h ceph NOFD /proc/45983/task/47253/fd (opendir: Permission denied) 2024-03-18T15:20:22.861 INFO:tasks.workunit.client.0.smithi019.stdout:sudo 47384 root cwd unknown /proc/47384/cwd (readlink: Permission denied) 2024-03-18T15:20:22.861 INFO:tasks.workunit.client.0.smithi019.stdout:sudo 47384 root rtd unknown /proc/47384/root (readlink: Permission denied) 2024-03-18T15:20:22.861 INFO:tasks.workunit.client.0.smithi019.stdout:sudo 47384 root txt unknown /proc/47384/exe (readlink: Permission denied) 2024-03-18T15:20:22.861 INFO:tasks.workunit.client.0.smithi019.stdout:sudo 47384 root NOFD /proc/47384/fd (opendir: Permission denied) 2024-03-18T15:20:22.861 INFO:tasks.workunit.client.0.smithi019.stdout:journalct 47419 root cwd unknown /proc/47419/cwd (readlink: Permission denied) 2024-03-18T15:20:22.861 INFO:tasks.workunit.client.0.smithi019.stdout:journalct 47419 root rtd unknown /proc/47419/root (readlink: Permission denied) 2024-03-18T15:20:22.862 INFO:tasks.workunit.client.0.smithi019.stdout:journalct 47419 root txt unknown /proc/47419/exe (readlink: Permission denied) 2024-03-18T15:20:22.862 INFO:tasks.workunit.client.0.smithi019.stdout:journalct 47419 root NOFD /proc/47419/fd (opendir: Permission denied) 2024-03-18T15:20:22.862 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/5 50155 root cwd unknown /proc/50155/cwd (readlink: Permission denied) 2024-03-18T15:20:22.862 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/5 50155 root rtd unknown /proc/50155/root (readlink: Permission denied) 2024-03-18T15:20:22.862 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/5 50155 root txt unknown /proc/50155/exe (readlink: Permission denied) 2024-03-18T15:20:22.862 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/5 50155 root NOFD /proc/50155/fd (opendir: Permission denied) 2024-03-18T15:20:22.862 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/6 52050 root cwd unknown /proc/52050/cwd (readlink: Permission denied) 2024-03-18T15:20:22.862 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/6 52050 root rtd unknown /proc/52050/root (readlink: Permission denied) 2024-03-18T15:20:22.862 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/6 52050 root txt unknown /proc/52050/exe (readlink: Permission denied) 2024-03-18T15:20:22.862 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/6 52050 root NOFD /proc/52050/fd (opendir: Permission denied) 2024-03-18T15:20:22.862 INFO:tasks.workunit.client.0.smithi019.stdout:conmon 52790 root cwd unknown /proc/52790/cwd (readlink: Permission denied) 2024-03-18T15:20:22.862 INFO:tasks.workunit.client.0.smithi019.stdout:conmon 52790 root rtd unknown /proc/52790/root (readlink: Permission denied) 2024-03-18T15:20:22.862 INFO:tasks.workunit.client.0.smithi019.stdout:conmon 52790 root txt unknown /proc/52790/exe (readlink: Permission denied) 2024-03-18T15:20:22.863 INFO:tasks.workunit.client.0.smithi019.stdout:conmon 52790 root NOFD /proc/52790/fd (opendir: Permission denied) 2024-03-18T15:20:22.863 INFO:tasks.workunit.client.0.smithi019.stdout:podman-in 52792 root cwd unknown /proc/52792/cwd (readlink: Permission denied) 2024-03-18T15:20:22.863 INFO:tasks.workunit.client.0.smithi019.stdout:podman-in 52792 root rtd unknown /proc/52792/root (readlink: Permission denied) 2024-03-18T15:20:22.863 INFO:tasks.workunit.client.0.smithi019.stdout:podman-in 52792 root txt unknown /proc/52792/exe (readlink: Permission denied) 2024-03-18T15:20:22.863 INFO:tasks.workunit.client.0.smithi019.stdout:podman-in 52792 root NOFD /proc/52792/fd (opendir: Permission denied) 2024-03-18T15:20:22.863 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 ceph cwd unknown /proc/52794/cwd (readlink: Permission denied) 2024-03-18T15:20:22.863 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 ceph rtd unknown /proc/52794/root (readlink: Permission denied) 2024-03-18T15:20:22.863 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 ceph txt unknown /proc/52794/exe (readlink: Permission denied) 2024-03-18T15:20:22.863 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 ceph NOFD /proc/52794/fd (opendir: Permission denied) 2024-03-18T15:20:22.863 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 52795 log ceph cwd unknown /proc/52794/task/52795/cwd (readlink: Permission denied) 2024-03-18T15:20:22.863 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 52795 log ceph rtd unknown /proc/52794/task/52795/root (readlink: Permission denied) 2024-03-18T15:20:22.863 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 52795 log ceph txt unknown /proc/52794/task/52795/exe (readlink: Permission denied) 2024-03-18T15:20:22.863 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 52795 log ceph NOFD /proc/52794/task/52795/fd (opendir: Permission denied) 2024-03-18T15:20:22.863 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 52797 msgr-work ceph cwd unknown /proc/52794/task/52797/cwd (readlink: Permission denied) 2024-03-18T15:20:22.863 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 52797 msgr-work ceph rtd unknown /proc/52794/task/52797/root (readlink: Permission denied) 2024-03-18T15:20:22.864 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 52797 msgr-work ceph txt unknown /proc/52794/task/52797/exe (readlink: Permission denied) 2024-03-18T15:20:22.864 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 52797 msgr-work ceph NOFD /proc/52794/task/52797/fd (opendir: Permission denied) 2024-03-18T15:20:22.864 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 52798 msgr-work ceph cwd unknown /proc/52794/task/52798/cwd (readlink: Permission denied) 2024-03-18T15:20:22.864 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 52798 msgr-work ceph rtd unknown /proc/52794/task/52798/root (readlink: Permission denied) 2024-03-18T15:20:22.864 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 52798 msgr-work ceph txt unknown /proc/52794/task/52798/exe (readlink: Permission denied) 2024-03-18T15:20:22.864 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 52798 msgr-work ceph NOFD /proc/52794/task/52798/fd (opendir: Permission denied) 2024-03-18T15:20:22.864 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 52799 msgr-work ceph cwd unknown /proc/52794/task/52799/cwd (readlink: Permission denied) 2024-03-18T15:20:22.864 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 52799 msgr-work ceph rtd unknown /proc/52794/task/52799/root (readlink: Permission denied) 2024-03-18T15:20:22.864 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 52799 msgr-work ceph txt unknown /proc/52794/task/52799/exe (readlink: Permission denied) 2024-03-18T15:20:22.864 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 52799 msgr-work ceph NOFD /proc/52794/task/52799/fd (opendir: Permission denied) 2024-03-18T15:20:22.864 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 52803 service ceph cwd unknown /proc/52794/task/52803/cwd (readlink: Permission denied) 2024-03-18T15:20:22.864 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 52803 service ceph rtd unknown /proc/52794/task/52803/root (readlink: Permission denied) 2024-03-18T15:20:22.864 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 52803 service ceph txt unknown /proc/52794/task/52803/exe (readlink: Permission denied) 2024-03-18T15:20:22.864 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 52803 service ceph NOFD /proc/52794/task/52803/fd (opendir: Permission denied) 2024-03-18T15:20:22.864 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 52804 admin_soc ceph cwd unknown /proc/52794/task/52804/cwd (readlink: Permission denied) 2024-03-18T15:20:22.865 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 52804 admin_soc ceph rtd unknown /proc/52794/task/52804/root (readlink: Permission denied) 2024-03-18T15:20:22.865 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 52804 admin_soc ceph txt unknown /proc/52794/task/52804/exe (readlink: Permission denied) 2024-03-18T15:20:22.865 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 52804 admin_soc ceph NOFD /proc/52794/task/52804/fd (opendir: Permission denied) 2024-03-18T15:20:22.865 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 52947 signal_ha ceph cwd unknown /proc/52794/task/52947/cwd (readlink: Permission denied) 2024-03-18T15:20:22.865 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 52947 signal_ha ceph rtd unknown /proc/52794/task/52947/root (readlink: Permission denied) 2024-03-18T15:20:22.865 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 52947 signal_ha ceph txt unknown /proc/52794/task/52947/exe (readlink: Permission denied) 2024-03-18T15:20:22.865 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 52947 signal_ha ceph NOFD /proc/52794/task/52947/fd (opendir: Permission denied) 2024-03-18T15:20:22.865 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 52948 io_contex ceph cwd unknown /proc/52794/task/52948/cwd (readlink: Permission denied) 2024-03-18T15:20:22.865 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 52948 io_contex ceph rtd unknown /proc/52794/task/52948/root (readlink: Permission denied) 2024-03-18T15:20:22.865 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 52948 io_contex ceph txt unknown /proc/52794/task/52948/exe (readlink: Permission denied) 2024-03-18T15:20:22.865 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 52948 io_contex ceph NOFD /proc/52794/task/52948/fd (opendir: Permission denied) 2024-03-18T15:20:22.865 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 52949 io_contex ceph cwd unknown /proc/52794/task/52949/cwd (readlink: Permission denied) 2024-03-18T15:20:22.865 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 52949 io_contex ceph rtd unknown /proc/52794/task/52949/root (readlink: Permission denied) 2024-03-18T15:20:22.865 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 52949 io_contex ceph txt unknown /proc/52794/task/52949/exe (readlink: Permission denied) 2024-03-18T15:20:22.866 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 52949 io_contex ceph NOFD /proc/52794/task/52949/fd (opendir: Permission denied) 2024-03-18T15:20:22.866 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 52953 OpHistory ceph cwd unknown /proc/52794/task/52953/cwd (readlink: Permission denied) 2024-03-18T15:20:22.866 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 52953 OpHistory ceph rtd unknown /proc/52794/task/52953/root (readlink: Permission denied) 2024-03-18T15:20:22.866 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 52953 OpHistory ceph txt unknown /proc/52794/task/52953/exe (readlink: Permission denied) 2024-03-18T15:20:22.866 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 52953 OpHistory ceph NOFD /proc/52794/task/52953/fd (opendir: Permission denied) 2024-03-18T15:20:22.866 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 52954 ceph_time ceph cwd unknown /proc/52794/task/52954/cwd (readlink: Permission denied) 2024-03-18T15:20:22.866 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 52954 ceph_time ceph rtd unknown /proc/52794/task/52954/root (readlink: Permission denied) 2024-03-18T15:20:22.866 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 52954 ceph_time ceph txt unknown /proc/52794/task/52954/exe (readlink: Permission denied) 2024-03-18T15:20:22.866 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 52954 ceph_time ceph NOFD /proc/52794/task/52954/fd (opendir: Permission denied) 2024-03-18T15:20:22.866 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 53180 ceph-osd ceph cwd unknown /proc/52794/task/53180/cwd (readlink: Permission denied) 2024-03-18T15:20:22.866 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 53180 ceph-osd ceph rtd unknown /proc/52794/task/53180/root (readlink: Permission denied) 2024-03-18T15:20:22.866 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 53180 ceph-osd ceph txt unknown /proc/52794/task/53180/exe (readlink: Permission denied) 2024-03-18T15:20:22.866 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 53180 ceph-osd ceph NOFD /proc/52794/task/53180/fd (opendir: Permission denied) 2024-03-18T15:20:22.866 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 53181 safe_time ceph cwd unknown /proc/52794/task/53181/cwd (readlink: Permission denied) 2024-03-18T15:20:22.867 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 53181 safe_time ceph rtd unknown /proc/52794/task/53181/root (readlink: Permission denied) 2024-03-18T15:20:22.867 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 53181 safe_time ceph txt unknown /proc/52794/task/53181/exe (readlink: Permission denied) 2024-03-18T15:20:22.867 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 53181 safe_time ceph NOFD /proc/52794/task/53181/fd (opendir: Permission denied) 2024-03-18T15:20:22.867 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 53182 safe_time ceph cwd unknown /proc/52794/task/53182/cwd (readlink: Permission denied) 2024-03-18T15:20:22.867 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 53182 safe_time ceph rtd unknown /proc/52794/task/53182/root (readlink: Permission denied) 2024-03-18T15:20:22.867 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 53182 safe_time ceph txt unknown /proc/52794/task/53182/exe (readlink: Permission denied) 2024-03-18T15:20:22.867 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 53182 safe_time ceph NOFD /proc/52794/task/53182/fd (opendir: Permission denied) 2024-03-18T15:20:22.867 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 53183 safe_time ceph cwd unknown /proc/52794/task/53183/cwd (readlink: Permission denied) 2024-03-18T15:20:22.867 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 53183 safe_time ceph rtd unknown /proc/52794/task/53183/root (readlink: Permission denied) 2024-03-18T15:20:22.867 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 53183 safe_time ceph txt unknown /proc/52794/task/53183/exe (readlink: Permission denied) 2024-03-18T15:20:22.867 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 53183 safe_time ceph NOFD /proc/52794/task/53183/fd (opendir: Permission denied) 2024-03-18T15:20:22.867 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 53184 safe_time ceph cwd unknown /proc/52794/task/53184/cwd (readlink: Permission denied) 2024-03-18T15:20:22.867 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 53184 safe_time ceph rtd unknown /proc/52794/task/53184/root (readlink: Permission denied) 2024-03-18T15:20:22.868 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 53184 safe_time ceph txt unknown /proc/52794/task/53184/exe (readlink: Permission denied) 2024-03-18T15:20:22.868 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 53184 safe_time ceph NOFD /proc/52794/task/53184/fd (opendir: Permission denied) 2024-03-18T15:20:22.868 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 53185 fn_anonym ceph cwd unknown /proc/52794/task/53185/cwd (readlink: Permission denied) 2024-03-18T15:20:22.868 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 53185 fn_anonym ceph rtd unknown /proc/52794/task/53185/root (readlink: Permission denied) 2024-03-18T15:20:22.868 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 53185 fn_anonym ceph txt unknown /proc/52794/task/53185/exe (readlink: Permission denied) 2024-03-18T15:20:22.868 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 53185 fn_anonym ceph NOFD /proc/52794/task/53185/fd (opendir: Permission denied) 2024-03-18T15:20:22.868 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 53191 rocksdb:l ceph cwd unknown /proc/52794/task/53191/cwd (readlink: Permission denied) 2024-03-18T15:20:22.868 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 53191 rocksdb:l ceph rtd unknown /proc/52794/task/53191/root (readlink: Permission denied) 2024-03-18T15:20:22.868 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 53191 rocksdb:l ceph txt unknown /proc/52794/task/53191/exe (readlink: Permission denied) 2024-03-18T15:20:22.868 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 53191 rocksdb:l ceph NOFD /proc/52794/task/53191/fd (opendir: Permission denied) 2024-03-18T15:20:22.868 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 53192 rocksdb:l ceph cwd unknown /proc/52794/task/53192/cwd (readlink: Permission denied) 2024-03-18T15:20:22.868 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 53192 rocksdb:l ceph rtd unknown /proc/52794/task/53192/root (readlink: Permission denied) 2024-03-18T15:20:22.868 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 53192 rocksdb:l ceph txt unknown /proc/52794/task/53192/exe (readlink: Permission denied) 2024-03-18T15:20:22.868 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 53192 rocksdb:l ceph NOFD /proc/52794/task/53192/fd (opendir: Permission denied) 2024-03-18T15:20:22.868 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 53193 rocksdb:l ceph cwd unknown /proc/52794/task/53193/cwd (readlink: Permission denied) 2024-03-18T15:20:22.869 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 53193 rocksdb:l ceph rtd unknown /proc/52794/task/53193/root (readlink: Permission denied) 2024-03-18T15:20:22.869 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 53193 rocksdb:l ceph txt unknown /proc/52794/task/53193/exe (readlink: Permission denied) 2024-03-18T15:20:22.869 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 53193 rocksdb:l ceph NOFD /proc/52794/task/53193/fd (opendir: Permission denied) 2024-03-18T15:20:22.869 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 53194 rocksdb:h ceph cwd unknown /proc/52794/task/53194/cwd (readlink: Permission denied) 2024-03-18T15:20:22.869 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 53194 rocksdb:h ceph rtd unknown /proc/52794/task/53194/root (readlink: Permission denied) 2024-03-18T15:20:22.869 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 53194 rocksdb:h ceph txt unknown /proc/52794/task/53194/exe (readlink: Permission denied) 2024-03-18T15:20:22.869 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 53194 rocksdb:h ceph NOFD /proc/52794/task/53194/fd (opendir: Permission denied) 2024-03-18T15:20:22.869 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 53658 bstore_ai ceph cwd unknown /proc/52794/task/53658/cwd (readlink: Permission denied) 2024-03-18T15:20:22.869 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 53658 bstore_ai ceph rtd unknown /proc/52794/task/53658/root (readlink: Permission denied) 2024-03-18T15:20:22.869 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 53658 bstore_ai ceph txt unknown /proc/52794/task/53658/exe (readlink: Permission denied) 2024-03-18T15:20:22.869 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 53658 bstore_ai ceph NOFD /proc/52794/task/53658/fd (opendir: Permission denied) 2024-03-18T15:20:22.869 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 53855 bstore_ai ceph cwd unknown /proc/52794/task/53855/cwd (readlink: Permission denied) 2024-03-18T15:20:22.869 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 53855 bstore_ai ceph rtd unknown /proc/52794/task/53855/root (readlink: Permission denied) 2024-03-18T15:20:22.869 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 53855 bstore_ai ceph txt unknown /proc/52794/task/53855/exe (readlink: Permission denied) 2024-03-18T15:20:22.869 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 53855 bstore_ai ceph NOFD /proc/52794/task/53855/fd (opendir: Permission denied) 2024-03-18T15:20:22.869 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54041 ceph-osd ceph cwd unknown /proc/52794/task/54041/cwd (readlink: Permission denied) 2024-03-18T15:20:22.870 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54041 ceph-osd ceph rtd unknown /proc/52794/task/54041/root (readlink: Permission denied) 2024-03-18T15:20:22.870 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54041 ceph-osd ceph txt unknown /proc/52794/task/54041/exe (readlink: Permission denied) 2024-03-18T15:20:22.870 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54041 ceph-osd ceph NOFD /proc/52794/task/54041/fd (opendir: Permission denied) 2024-03-18T15:20:22.870 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54042 cfin ceph cwd unknown /proc/52794/task/54042/cwd (readlink: Permission denied) 2024-03-18T15:20:22.870 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54042 cfin ceph rtd unknown /proc/52794/task/54042/root (readlink: Permission denied) 2024-03-18T15:20:22.870 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54042 cfin ceph txt unknown /proc/52794/task/54042/exe (readlink: Permission denied) 2024-03-18T15:20:22.870 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54042 cfin ceph NOFD /proc/52794/task/54042/fd (opendir: Permission denied) 2024-03-18T15:20:22.870 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54043 bstore_kv ceph cwd unknown /proc/52794/task/54043/cwd (readlink: Permission denied) 2024-03-18T15:20:22.870 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54043 bstore_kv ceph rtd unknown /proc/52794/task/54043/root (readlink: Permission denied) 2024-03-18T15:20:22.870 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54043 bstore_kv ceph txt unknown /proc/52794/task/54043/exe (readlink: Permission denied) 2024-03-18T15:20:22.870 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54043 bstore_kv ceph NOFD /proc/52794/task/54043/fd (opendir: Permission denied) 2024-03-18T15:20:22.870 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54044 bstore_kv ceph cwd unknown /proc/52794/task/54044/cwd (readlink: Permission denied) 2024-03-18T15:20:22.871 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54044 bstore_kv ceph rtd unknown /proc/52794/task/54044/root (readlink: Permission denied) 2024-03-18T15:20:22.871 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54044 bstore_kv ceph txt unknown /proc/52794/task/54044/exe (readlink: Permission denied) 2024-03-18T15:20:22.871 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54044 bstore_kv ceph NOFD /proc/52794/task/54044/fd (opendir: Permission denied) 2024-03-18T15:20:22.871 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54045 bstore_me ceph cwd unknown /proc/52794/task/54045/cwd (readlink: Permission denied) 2024-03-18T15:20:22.871 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54045 bstore_me ceph rtd unknown /proc/52794/task/54045/root (readlink: Permission denied) 2024-03-18T15:20:22.871 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54045 bstore_me ceph txt unknown /proc/52794/task/54045/exe (readlink: Permission denied) 2024-03-18T15:20:22.871 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54045 bstore_me ceph NOFD /proc/52794/task/54045/fd (opendir: Permission denied) 2024-03-18T15:20:22.871 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54048 ms_dispat ceph cwd unknown /proc/52794/task/54048/cwd (readlink: Permission denied) 2024-03-18T15:20:22.871 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54048 ms_dispat ceph rtd unknown /proc/52794/task/54048/root (readlink: Permission denied) 2024-03-18T15:20:22.871 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54048 ms_dispat ceph txt unknown /proc/52794/task/54048/exe (readlink: Permission denied) 2024-03-18T15:20:22.871 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54048 ms_dispat ceph NOFD /proc/52794/task/54048/fd (opendir: Permission denied) 2024-03-18T15:20:22.871 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54049 ms_local ceph cwd unknown /proc/52794/task/54049/cwd (readlink: Permission denied) 2024-03-18T15:20:22.871 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54049 ms_local ceph rtd unknown /proc/52794/task/54049/root (readlink: Permission denied) 2024-03-18T15:20:22.871 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54049 ms_local ceph txt unknown /proc/52794/task/54049/exe (readlink: Permission denied) 2024-03-18T15:20:22.872 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54049 ms_local ceph NOFD /proc/52794/task/54049/fd (opendir: Permission denied) 2024-03-18T15:20:22.872 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54050 safe_time ceph cwd unknown /proc/52794/task/54050/cwd (readlink: Permission denied) 2024-03-18T15:20:22.872 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54050 safe_time ceph rtd unknown /proc/52794/task/54050/root (readlink: Permission denied) 2024-03-18T15:20:22.872 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54050 safe_time ceph txt unknown /proc/52794/task/54050/exe (readlink: Permission denied) 2024-03-18T15:20:22.872 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54050 safe_time ceph NOFD /proc/52794/task/54050/fd (opendir: Permission denied) 2024-03-18T15:20:22.872 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54051 safe_time ceph cwd unknown /proc/52794/task/54051/cwd (readlink: Permission denied) 2024-03-18T15:20:22.872 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54051 safe_time ceph rtd unknown /proc/52794/task/54051/root (readlink: Permission denied) 2024-03-18T15:20:22.872 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54051 safe_time ceph txt unknown /proc/52794/task/54051/exe (readlink: Permission denied) 2024-03-18T15:20:22.872 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54051 safe_time ceph NOFD /proc/52794/task/54051/fd (opendir: Permission denied) 2024-03-18T15:20:22.872 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54052 ms_dispat ceph cwd unknown /proc/52794/task/54052/cwd (readlink: Permission denied) 2024-03-18T15:20:22.872 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54052 ms_dispat ceph rtd unknown /proc/52794/task/54052/root (readlink: Permission denied) 2024-03-18T15:20:22.872 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54052 ms_dispat ceph txt unknown /proc/52794/task/54052/exe (readlink: Permission denied) 2024-03-18T15:20:22.872 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54052 ms_dispat ceph NOFD /proc/52794/task/54052/fd (opendir: Permission denied) 2024-03-18T15:20:22.872 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54053 ms_local ceph cwd unknown /proc/52794/task/54053/cwd (readlink: Permission denied) 2024-03-18T15:20:22.872 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54053 ms_local ceph rtd unknown /proc/52794/task/54053/root (readlink: Permission denied) 2024-03-18T15:20:22.873 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54053 ms_local ceph txt unknown /proc/52794/task/54053/exe (readlink: Permission denied) 2024-03-18T15:20:22.873 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54053 ms_local ceph NOFD /proc/52794/task/54053/fd (opendir: Permission denied) 2024-03-18T15:20:22.873 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54054 ms_dispat ceph cwd unknown /proc/52794/task/54054/cwd (readlink: Permission denied) 2024-03-18T15:20:22.873 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54054 ms_dispat ceph rtd unknown /proc/52794/task/54054/root (readlink: Permission denied) 2024-03-18T15:20:22.873 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54054 ms_dispat ceph txt unknown /proc/52794/task/54054/exe (readlink: Permission denied) 2024-03-18T15:20:22.873 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54054 ms_dispat ceph NOFD /proc/52794/task/54054/fd (opendir: Permission denied) 2024-03-18T15:20:22.873 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54055 ms_local ceph cwd unknown /proc/52794/task/54055/cwd (readlink: Permission denied) 2024-03-18T15:20:22.873 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54055 ms_local ceph rtd unknown /proc/52794/task/54055/root (readlink: Permission denied) 2024-03-18T15:20:22.873 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54055 ms_local ceph txt unknown /proc/52794/task/54055/exe (readlink: Permission denied) 2024-03-18T15:20:22.873 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54055 ms_local ceph NOFD /proc/52794/task/54055/fd (opendir: Permission denied) 2024-03-18T15:20:22.873 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54056 ms_dispat ceph cwd unknown /proc/52794/task/54056/cwd (readlink: Permission denied) 2024-03-18T15:20:22.873 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54056 ms_dispat ceph rtd unknown /proc/52794/task/54056/root (readlink: Permission denied) 2024-03-18T15:20:22.873 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54056 ms_dispat ceph txt unknown /proc/52794/task/54056/exe (readlink: Permission denied) 2024-03-18T15:20:22.873 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54056 ms_dispat ceph NOFD /proc/52794/task/54056/fd (opendir: Permission denied) 2024-03-18T15:20:22.873 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54057 ms_local ceph cwd unknown /proc/52794/task/54057/cwd (readlink: Permission denied) 2024-03-18T15:20:22.874 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54057 ms_local ceph rtd unknown /proc/52794/task/54057/root (readlink: Permission denied) 2024-03-18T15:20:22.874 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54057 ms_local ceph txt unknown /proc/52794/task/54057/exe (readlink: Permission denied) 2024-03-18T15:20:22.874 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54057 ms_local ceph NOFD /proc/52794/task/54057/fd (opendir: Permission denied) 2024-03-18T15:20:22.874 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54058 ms_dispat ceph cwd unknown /proc/52794/task/54058/cwd (readlink: Permission denied) 2024-03-18T15:20:22.874 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54058 ms_dispat ceph rtd unknown /proc/52794/task/54058/root (readlink: Permission denied) 2024-03-18T15:20:22.874 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54058 ms_dispat ceph txt unknown /proc/52794/task/54058/exe (readlink: Permission denied) 2024-03-18T15:20:22.874 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54058 ms_dispat ceph NOFD /proc/52794/task/54058/fd (opendir: Permission denied) 2024-03-18T15:20:22.874 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54059 ms_local ceph cwd unknown /proc/52794/task/54059/cwd (readlink: Permission denied) 2024-03-18T15:20:22.874 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54059 ms_local ceph rtd unknown /proc/52794/task/54059/root (readlink: Permission denied) 2024-03-18T15:20:22.874 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54059 ms_local ceph txt unknown /proc/52794/task/54059/exe (readlink: Permission denied) 2024-03-18T15:20:22.874 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54059 ms_local ceph NOFD /proc/52794/task/54059/fd (opendir: Permission denied) 2024-03-18T15:20:22.874 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54060 ms_dispat ceph cwd unknown /proc/52794/task/54060/cwd (readlink: Permission denied) 2024-03-18T15:20:22.874 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54060 ms_dispat ceph rtd unknown /proc/52794/task/54060/root (readlink: Permission denied) 2024-03-18T15:20:22.875 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54060 ms_dispat ceph txt unknown /proc/52794/task/54060/exe (readlink: Permission denied) 2024-03-18T15:20:22.875 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54060 ms_dispat ceph NOFD /proc/52794/task/54060/fd (opendir: Permission denied) 2024-03-18T15:20:22.875 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54061 ms_local ceph cwd unknown /proc/52794/task/54061/cwd (readlink: Permission denied) 2024-03-18T15:20:22.875 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54061 ms_local ceph rtd unknown /proc/52794/task/54061/root (readlink: Permission denied) 2024-03-18T15:20:22.875 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54061 ms_local ceph txt unknown /proc/52794/task/54061/exe (readlink: Permission denied) 2024-03-18T15:20:22.875 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54061 ms_local ceph NOFD /proc/52794/task/54061/fd (opendir: Permission denied) 2024-03-18T15:20:22.875 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54062 ms_dispat ceph cwd unknown /proc/52794/task/54062/cwd (readlink: Permission denied) 2024-03-18T15:20:22.875 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54062 ms_dispat ceph rtd unknown /proc/52794/task/54062/root (readlink: Permission denied) 2024-03-18T15:20:22.875 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54062 ms_dispat ceph txt unknown /proc/52794/task/54062/exe (readlink: Permission denied) 2024-03-18T15:20:22.875 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54062 ms_dispat ceph NOFD /proc/52794/task/54062/fd (opendir: Permission denied) 2024-03-18T15:20:22.875 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54063 ms_local ceph cwd unknown /proc/52794/task/54063/cwd (readlink: Permission denied) 2024-03-18T15:20:22.875 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54063 ms_local ceph rtd unknown /proc/52794/task/54063/root (readlink: Permission denied) 2024-03-18T15:20:22.875 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54063 ms_local ceph txt unknown /proc/52794/task/54063/exe (readlink: Permission denied) 2024-03-18T15:20:22.875 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54063 ms_local ceph NOFD /proc/52794/task/54063/fd (opendir: Permission denied) 2024-03-18T15:20:22.875 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54064 fn_anonym ceph cwd unknown /proc/52794/task/54064/cwd (readlink: Permission denied) 2024-03-18T15:20:22.876 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54064 fn_anonym ceph rtd unknown /proc/52794/task/54064/root (readlink: Permission denied) 2024-03-18T15:20:22.876 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54064 fn_anonym ceph txt unknown /proc/52794/task/54064/exe (readlink: Permission denied) 2024-03-18T15:20:22.876 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54064 fn_anonym ceph NOFD /proc/52794/task/54064/fd (opendir: Permission denied) 2024-03-18T15:20:22.876 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54065 finisher ceph cwd unknown /proc/52794/task/54065/cwd (readlink: Permission denied) 2024-03-18T15:20:22.876 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54065 finisher ceph rtd unknown /proc/52794/task/54065/root (readlink: Permission denied) 2024-03-18T15:20:22.876 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54065 finisher ceph txt unknown /proc/52794/task/54065/exe (readlink: Permission denied) 2024-03-18T15:20:22.876 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54065 finisher ceph NOFD /proc/52794/task/54065/fd (opendir: Permission denied) 2024-03-18T15:20:22.876 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54066 safe_time ceph cwd unknown /proc/52794/task/54066/cwd (readlink: Permission denied) 2024-03-18T15:20:22.876 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54066 safe_time ceph rtd unknown /proc/52794/task/54066/root (readlink: Permission denied) 2024-03-18T15:20:22.876 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54066 safe_time ceph txt unknown /proc/52794/task/54066/exe (readlink: Permission denied) 2024-03-18T15:20:22.876 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54066 safe_time ceph NOFD /proc/52794/task/54066/fd (opendir: Permission denied) 2024-03-18T15:20:22.876 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54067 safe_time ceph cwd unknown /proc/52794/task/54067/cwd (readlink: Permission denied) 2024-03-18T15:20:22.876 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54067 safe_time ceph rtd unknown /proc/52794/task/54067/root (readlink: Permission denied) 2024-03-18T15:20:22.876 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54067 safe_time ceph txt unknown /proc/52794/task/54067/exe (readlink: Permission denied) 2024-03-18T15:20:22.877 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54067 safe_time ceph NOFD /proc/52794/task/54067/fd (opendir: Permission denied) 2024-03-18T15:20:22.877 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54068 ceph-osd ceph cwd unknown /proc/52794/task/54068/cwd (readlink: Permission denied) 2024-03-18T15:20:22.877 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54068 ceph-osd ceph rtd unknown /proc/52794/task/54068/root (readlink: Permission denied) 2024-03-18T15:20:22.877 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54068 ceph-osd ceph txt unknown /proc/52794/task/54068/exe (readlink: Permission denied) 2024-03-18T15:20:22.877 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54068 ceph-osd ceph NOFD /proc/52794/task/54068/fd (opendir: Permission denied) 2024-03-18T15:20:22.877 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54069 osd_srv_a ceph cwd unknown /proc/52794/task/54069/cwd (readlink: Permission denied) 2024-03-18T15:20:22.877 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54069 osd_srv_a ceph rtd unknown /proc/52794/task/54069/root (readlink: Permission denied) 2024-03-18T15:20:22.877 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54069 osd_srv_a ceph txt unknown /proc/52794/task/54069/exe (readlink: Permission denied) 2024-03-18T15:20:22.877 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54069 osd_srv_a ceph NOFD /proc/52794/task/54069/fd (opendir: Permission denied) 2024-03-18T15:20:22.877 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54070 tp_osd_tp ceph cwd unknown /proc/52794/task/54070/cwd (readlink: Permission denied) 2024-03-18T15:20:22.877 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54070 tp_osd_tp ceph rtd unknown /proc/52794/task/54070/root (readlink: Permission denied) 2024-03-18T15:20:22.877 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54070 tp_osd_tp ceph txt unknown /proc/52794/task/54070/exe (readlink: Permission denied) 2024-03-18T15:20:22.877 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54070 tp_osd_tp ceph NOFD /proc/52794/task/54070/fd (opendir: Permission denied) 2024-03-18T15:20:22.877 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54071 tp_osd_tp ceph cwd unknown /proc/52794/task/54071/cwd (readlink: Permission denied) 2024-03-18T15:20:22.877 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54071 tp_osd_tp ceph rtd unknown /proc/52794/task/54071/root (readlink: Permission denied) 2024-03-18T15:20:22.878 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54071 tp_osd_tp ceph txt unknown /proc/52794/task/54071/exe (readlink: Permission denied) 2024-03-18T15:20:22.878 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54071 tp_osd_tp ceph NOFD /proc/52794/task/54071/fd (opendir: Permission denied) 2024-03-18T15:20:22.878 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54072 tp_osd_tp ceph cwd unknown /proc/52794/task/54072/cwd (readlink: Permission denied) 2024-03-18T15:20:22.878 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54072 tp_osd_tp ceph rtd unknown /proc/52794/task/54072/root (readlink: Permission denied) 2024-03-18T15:20:22.878 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54072 tp_osd_tp ceph txt unknown /proc/52794/task/54072/exe (readlink: Permission denied) 2024-03-18T15:20:22.878 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54072 tp_osd_tp ceph NOFD /proc/52794/task/54072/fd (opendir: Permission denied) 2024-03-18T15:20:22.878 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54073 tp_osd_tp ceph cwd unknown /proc/52794/task/54073/cwd (readlink: Permission denied) 2024-03-18T15:20:22.878 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54073 tp_osd_tp ceph rtd unknown /proc/52794/task/54073/root (readlink: Permission denied) 2024-03-18T15:20:22.878 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54073 tp_osd_tp ceph txt unknown /proc/52794/task/54073/exe (readlink: Permission denied) 2024-03-18T15:20:22.878 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54073 tp_osd_tp ceph NOFD /proc/52794/task/54073/fd (opendir: Permission denied) 2024-03-18T15:20:22.878 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54074 tp_osd_tp ceph cwd unknown /proc/52794/task/54074/cwd (readlink: Permission denied) 2024-03-18T15:20:22.878 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54074 tp_osd_tp ceph rtd unknown /proc/52794/task/54074/root (readlink: Permission denied) 2024-03-18T15:20:22.878 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54074 tp_osd_tp ceph txt unknown /proc/52794/task/54074/exe (readlink: Permission denied) 2024-03-18T15:20:22.878 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54074 tp_osd_tp ceph NOFD /proc/52794/task/54074/fd (opendir: Permission denied) 2024-03-18T15:20:22.879 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54075 tp_osd_tp ceph cwd unknown /proc/52794/task/54075/cwd (readlink: Permission denied) 2024-03-18T15:20:22.879 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54075 tp_osd_tp ceph rtd unknown /proc/52794/task/54075/root (readlink: Permission denied) 2024-03-18T15:20:22.879 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54075 tp_osd_tp ceph txt unknown /proc/52794/task/54075/exe (readlink: Permission denied) 2024-03-18T15:20:22.879 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54075 tp_osd_tp ceph NOFD /proc/52794/task/54075/fd (opendir: Permission denied) 2024-03-18T15:20:22.879 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54076 tp_osd_tp ceph cwd unknown /proc/52794/task/54076/cwd (readlink: Permission denied) 2024-03-18T15:20:22.879 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54076 tp_osd_tp ceph rtd unknown /proc/52794/task/54076/root (readlink: Permission denied) 2024-03-18T15:20:22.879 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54076 tp_osd_tp ceph txt unknown /proc/52794/task/54076/exe (readlink: Permission denied) 2024-03-18T15:20:22.879 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54076 tp_osd_tp ceph NOFD /proc/52794/task/54076/fd (opendir: Permission denied) 2024-03-18T15:20:22.879 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54077 tp_osd_tp ceph cwd unknown /proc/52794/task/54077/cwd (readlink: Permission denied) 2024-03-18T15:20:22.879 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54077 tp_osd_tp ceph rtd unknown /proc/52794/task/54077/root (readlink: Permission denied) 2024-03-18T15:20:22.879 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54077 tp_osd_tp ceph txt unknown /proc/52794/task/54077/exe (readlink: Permission denied) 2024-03-18T15:20:22.879 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54077 tp_osd_tp ceph NOFD /proc/52794/task/54077/fd (opendir: Permission denied) 2024-03-18T15:20:22.879 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54078 tp_osd_tp ceph cwd unknown /proc/52794/task/54078/cwd (readlink: Permission denied) 2024-03-18T15:20:22.879 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54078 tp_osd_tp ceph rtd unknown /proc/52794/task/54078/root (readlink: Permission denied) 2024-03-18T15:20:22.879 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54078 tp_osd_tp ceph txt unknown /proc/52794/task/54078/exe (readlink: Permission denied) 2024-03-18T15:20:22.880 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54078 tp_osd_tp ceph NOFD /proc/52794/task/54078/fd (opendir: Permission denied) 2024-03-18T15:20:22.880 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54079 tp_osd_tp ceph cwd unknown /proc/52794/task/54079/cwd (readlink: Permission denied) 2024-03-18T15:20:22.880 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54079 tp_osd_tp ceph rtd unknown /proc/52794/task/54079/root (readlink: Permission denied) 2024-03-18T15:20:22.880 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54079 tp_osd_tp ceph txt unknown /proc/52794/task/54079/exe (readlink: Permission denied) 2024-03-18T15:20:22.880 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54079 tp_osd_tp ceph NOFD /proc/52794/task/54079/fd (opendir: Permission denied) 2024-03-18T15:20:22.880 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54080 tp_osd_tp ceph cwd unknown /proc/52794/task/54080/cwd (readlink: Permission denied) 2024-03-18T15:20:22.880 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54080 tp_osd_tp ceph rtd unknown /proc/52794/task/54080/root (readlink: Permission denied) 2024-03-18T15:20:22.880 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54080 tp_osd_tp ceph txt unknown /proc/52794/task/54080/exe (readlink: Permission denied) 2024-03-18T15:20:22.880 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54080 tp_osd_tp ceph NOFD /proc/52794/task/54080/fd (opendir: Permission denied) 2024-03-18T15:20:22.880 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54081 tp_osd_tp ceph cwd unknown /proc/52794/task/54081/cwd (readlink: Permission denied) 2024-03-18T15:20:22.880 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54081 tp_osd_tp ceph rtd unknown /proc/52794/task/54081/root (readlink: Permission denied) 2024-03-18T15:20:22.880 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54081 tp_osd_tp ceph txt unknown /proc/52794/task/54081/exe (readlink: Permission denied) 2024-03-18T15:20:22.880 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54081 tp_osd_tp ceph NOFD /proc/52794/task/54081/fd (opendir: Permission denied) 2024-03-18T15:20:22.880 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54082 tp_osd_tp ceph cwd unknown /proc/52794/task/54082/cwd (readlink: Permission denied) 2024-03-18T15:20:22.881 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54082 tp_osd_tp ceph rtd unknown /proc/52794/task/54082/root (readlink: Permission denied) 2024-03-18T15:20:22.881 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54082 tp_osd_tp ceph txt unknown /proc/52794/task/54082/exe (readlink: Permission denied) 2024-03-18T15:20:22.881 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54082 tp_osd_tp ceph NOFD /proc/52794/task/54082/fd (opendir: Permission denied) 2024-03-18T15:20:22.881 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54083 tp_osd_tp ceph cwd unknown /proc/52794/task/54083/cwd (readlink: Permission denied) 2024-03-18T15:20:22.881 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54083 tp_osd_tp ceph rtd unknown /proc/52794/task/54083/root (readlink: Permission denied) 2024-03-18T15:20:22.881 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54083 tp_osd_tp ceph txt unknown /proc/52794/task/54083/exe (readlink: Permission denied) 2024-03-18T15:20:22.881 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54083 tp_osd_tp ceph NOFD /proc/52794/task/54083/fd (opendir: Permission denied) 2024-03-18T15:20:22.881 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54084 tp_osd_tp ceph cwd unknown /proc/52794/task/54084/cwd (readlink: Permission denied) 2024-03-18T15:20:22.881 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54084 tp_osd_tp ceph rtd unknown /proc/52794/task/54084/root (readlink: Permission denied) 2024-03-18T15:20:22.881 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54084 tp_osd_tp ceph txt unknown /proc/52794/task/54084/exe (readlink: Permission denied) 2024-03-18T15:20:22.881 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54084 tp_osd_tp ceph NOFD /proc/52794/task/54084/fd (opendir: Permission denied) 2024-03-18T15:20:22.881 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54085 tp_osd_tp ceph cwd unknown /proc/52794/task/54085/cwd (readlink: Permission denied) 2024-03-18T15:20:22.881 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54085 tp_osd_tp ceph rtd unknown /proc/52794/task/54085/root (readlink: Permission denied) 2024-03-18T15:20:22.882 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54085 tp_osd_tp ceph txt unknown /proc/52794/task/54085/exe (readlink: Permission denied) 2024-03-18T15:20:22.882 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54085 tp_osd_tp ceph NOFD /proc/52794/task/54085/fd (opendir: Permission denied) 2024-03-18T15:20:22.882 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54086 osd_srv_h ceph cwd unknown /proc/52794/task/54086/cwd (readlink: Permission denied) 2024-03-18T15:20:22.882 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54086 osd_srv_h ceph rtd unknown /proc/52794/task/54086/root (readlink: Permission denied) 2024-03-18T15:20:22.882 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54086 osd_srv_h ceph txt unknown /proc/52794/task/54086/exe (readlink: Permission denied) 2024-03-18T15:20:22.882 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 52794 54086 osd_srv_h ceph NOFD /proc/52794/task/54086/fd (opendir: Permission denied) 2024-03-18T15:20:22.882 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/u 53846 root cwd unknown /proc/53846/cwd (readlink: Permission denied) 2024-03-18T15:20:22.882 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/u 53846 root rtd unknown /proc/53846/root (readlink: Permission denied) 2024-03-18T15:20:22.882 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/u 53846 root txt unknown /proc/53846/exe (readlink: Permission denied) 2024-03-18T15:20:22.882 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/u 53846 root NOFD /proc/53846/fd (opendir: Permission denied) 2024-03-18T15:20:22.882 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/1 54113 root cwd unknown /proc/54113/cwd (readlink: Permission denied) 2024-03-18T15:20:22.882 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/1 54113 root rtd unknown /proc/54113/root (readlink: Permission denied) 2024-03-18T15:20:22.882 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/1 54113 root txt unknown /proc/54113/exe (readlink: Permission denied) 2024-03-18T15:20:22.882 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/1 54113 root NOFD /proc/54113/fd (opendir: Permission denied) 2024-03-18T15:20:22.883 INFO:tasks.workunit.client.0.smithi019.stdout:sudo 54211 root cwd unknown /proc/54211/cwd (readlink: Permission denied) 2024-03-18T15:20:22.883 INFO:tasks.workunit.client.0.smithi019.stdout:sudo 54211 root rtd unknown /proc/54211/root (readlink: Permission denied) 2024-03-18T15:20:22.883 INFO:tasks.workunit.client.0.smithi019.stdout:sudo 54211 root txt unknown /proc/54211/exe (readlink: Permission denied) 2024-03-18T15:20:22.883 INFO:tasks.workunit.client.0.smithi019.stdout:sudo 54211 root NOFD /proc/54211/fd (opendir: Permission denied) 2024-03-18T15:20:22.883 INFO:tasks.workunit.client.0.smithi019.stdout:journalct 54231 root cwd unknown /proc/54231/cwd (readlink: Permission denied) 2024-03-18T15:20:22.883 INFO:tasks.workunit.client.0.smithi019.stdout:journalct 54231 root rtd unknown /proc/54231/root (readlink: Permission denied) 2024-03-18T15:20:22.883 INFO:tasks.workunit.client.0.smithi019.stdout:journalct 54231 root txt unknown /proc/54231/exe (readlink: Permission denied) 2024-03-18T15:20:22.883 INFO:tasks.workunit.client.0.smithi019.stdout:journalct 54231 root NOFD /proc/54231/fd (opendir: Permission denied) 2024-03-18T15:20:22.883 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/4 56856 root cwd unknown /proc/56856/cwd (readlink: Permission denied) 2024-03-18T15:20:22.883 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/4 56856 root rtd unknown /proc/56856/root (readlink: Permission denied) 2024-03-18T15:20:22.883 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/4 56856 root txt unknown /proc/56856/exe (readlink: Permission denied) 2024-03-18T15:20:22.883 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/4 56856 root NOFD /proc/56856/fd (opendir: Permission denied) 2024-03-18T15:20:22.883 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/0 57500 root cwd unknown /proc/57500/cwd (readlink: Permission denied) 2024-03-18T15:20:22.883 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/0 57500 root rtd unknown /proc/57500/root (readlink: Permission denied) 2024-03-18T15:20:22.884 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/0 57500 root txt unknown /proc/57500/exe (readlink: Permission denied) 2024-03-18T15:20:22.884 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/0 57500 root NOFD /proc/57500/fd (opendir: Permission denied) 2024-03-18T15:20:22.884 INFO:tasks.workunit.client.0.smithi019.stdout:conmon 59600 root cwd unknown /proc/59600/cwd (readlink: Permission denied) 2024-03-18T15:20:22.884 INFO:tasks.workunit.client.0.smithi019.stdout:conmon 59600 root rtd unknown /proc/59600/root (readlink: Permission denied) 2024-03-18T15:20:22.884 INFO:tasks.workunit.client.0.smithi019.stdout:conmon 59600 root txt unknown /proc/59600/exe (readlink: Permission denied) 2024-03-18T15:20:22.884 INFO:tasks.workunit.client.0.smithi019.stdout:conmon 59600 root NOFD /proc/59600/fd (opendir: Permission denied) 2024-03-18T15:20:22.884 INFO:tasks.workunit.client.0.smithi019.stdout:podman-in 59602 root cwd unknown /proc/59602/cwd (readlink: Permission denied) 2024-03-18T15:20:22.884 INFO:tasks.workunit.client.0.smithi019.stdout:podman-in 59602 root rtd unknown /proc/59602/root (readlink: Permission denied) 2024-03-18T15:20:22.884 INFO:tasks.workunit.client.0.smithi019.stdout:podman-in 59602 root txt unknown /proc/59602/exe (readlink: Permission denied) 2024-03-18T15:20:22.884 INFO:tasks.workunit.client.0.smithi019.stdout:podman-in 59602 root NOFD /proc/59602/fd (opendir: Permission denied) 2024-03-18T15:20:22.884 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 ceph cwd unknown /proc/59604/cwd (readlink: Permission denied) 2024-03-18T15:20:22.884 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 ceph rtd unknown /proc/59604/root (readlink: Permission denied) 2024-03-18T15:20:22.884 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 ceph txt unknown /proc/59604/exe (readlink: Permission denied) 2024-03-18T15:20:22.884 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 ceph NOFD /proc/59604/fd (opendir: Permission denied) 2024-03-18T15:20:22.884 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 59607 log ceph cwd unknown /proc/59604/task/59607/cwd (readlink: Permission denied) 2024-03-18T15:20:22.885 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 59607 log ceph rtd unknown /proc/59604/task/59607/root (readlink: Permission denied) 2024-03-18T15:20:22.885 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 59607 log ceph txt unknown /proc/59604/task/59607/exe (readlink: Permission denied) 2024-03-18T15:20:22.885 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 59607 log ceph NOFD /proc/59604/task/59607/fd (opendir: Permission denied) 2024-03-18T15:20:22.885 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 59609 msgr-work ceph cwd unknown /proc/59604/task/59609/cwd (readlink: Permission denied) 2024-03-18T15:20:22.885 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 59609 msgr-work ceph rtd unknown /proc/59604/task/59609/root (readlink: Permission denied) 2024-03-18T15:20:22.885 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 59609 msgr-work ceph txt unknown /proc/59604/task/59609/exe (readlink: Permission denied) 2024-03-18T15:20:22.885 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 59609 msgr-work ceph NOFD /proc/59604/task/59609/fd (opendir: Permission denied) 2024-03-18T15:20:22.885 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 59610 msgr-work ceph cwd unknown /proc/59604/task/59610/cwd (readlink: Permission denied) 2024-03-18T15:20:22.885 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 59610 msgr-work ceph rtd unknown /proc/59604/task/59610/root (readlink: Permission denied) 2024-03-18T15:20:22.885 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 59610 msgr-work ceph txt unknown /proc/59604/task/59610/exe (readlink: Permission denied) 2024-03-18T15:20:22.885 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 59610 msgr-work ceph NOFD /proc/59604/task/59610/fd (opendir: Permission denied) 2024-03-18T15:20:22.885 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 59611 msgr-work ceph cwd unknown /proc/59604/task/59611/cwd (readlink: Permission denied) 2024-03-18T15:20:22.885 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 59611 msgr-work ceph rtd unknown /proc/59604/task/59611/root (readlink: Permission denied) 2024-03-18T15:20:22.885 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 59611 msgr-work ceph txt unknown /proc/59604/task/59611/exe (readlink: Permission denied) 2024-03-18T15:20:22.886 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 59611 msgr-work ceph NOFD /proc/59604/task/59611/fd (opendir: Permission denied) 2024-03-18T15:20:22.886 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 59615 service ceph cwd unknown /proc/59604/task/59615/cwd (readlink: Permission denied) 2024-03-18T15:20:22.886 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 59615 service ceph rtd unknown /proc/59604/task/59615/root (readlink: Permission denied) 2024-03-18T15:20:22.886 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 59615 service ceph txt unknown /proc/59604/task/59615/exe (readlink: Permission denied) 2024-03-18T15:20:22.886 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 59615 service ceph NOFD /proc/59604/task/59615/fd (opendir: Permission denied) 2024-03-18T15:20:22.886 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 59616 admin_soc ceph cwd unknown /proc/59604/task/59616/cwd (readlink: Permission denied) 2024-03-18T15:20:22.886 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 59616 admin_soc ceph rtd unknown /proc/59604/task/59616/root (readlink: Permission denied) 2024-03-18T15:20:22.886 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 59616 admin_soc ceph txt unknown /proc/59604/task/59616/exe (readlink: Permission denied) 2024-03-18T15:20:22.886 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 59616 admin_soc ceph NOFD /proc/59604/task/59616/fd (opendir: Permission denied) 2024-03-18T15:20:22.886 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 59767 signal_ha ceph cwd unknown /proc/59604/task/59767/cwd (readlink: Permission denied) 2024-03-18T15:20:22.886 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 59767 signal_ha ceph rtd unknown /proc/59604/task/59767/root (readlink: Permission denied) 2024-03-18T15:20:22.886 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 59767 signal_ha ceph txt unknown /proc/59604/task/59767/exe (readlink: Permission denied) 2024-03-18T15:20:22.886 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 59767 signal_ha ceph NOFD /proc/59604/task/59767/fd (opendir: Permission denied) 2024-03-18T15:20:22.886 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 59768 io_contex ceph cwd unknown /proc/59604/task/59768/cwd (readlink: Permission denied) 2024-03-18T15:20:22.887 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 59768 io_contex ceph rtd unknown /proc/59604/task/59768/root (readlink: Permission denied) 2024-03-18T15:20:22.887 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 59768 io_contex ceph txt unknown /proc/59604/task/59768/exe (readlink: Permission denied) 2024-03-18T15:20:22.887 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 59768 io_contex ceph NOFD /proc/59604/task/59768/fd (opendir: Permission denied) 2024-03-18T15:20:22.887 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 59769 io_contex ceph cwd unknown /proc/59604/task/59769/cwd (readlink: Permission denied) 2024-03-18T15:20:22.887 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 59769 io_contex ceph rtd unknown /proc/59604/task/59769/root (readlink: Permission denied) 2024-03-18T15:20:22.887 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 59769 io_contex ceph txt unknown /proc/59604/task/59769/exe (readlink: Permission denied) 2024-03-18T15:20:22.887 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 59769 io_contex ceph NOFD /proc/59604/task/59769/fd (opendir: Permission denied) 2024-03-18T15:20:22.887 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 59787 OpHistory ceph cwd unknown /proc/59604/task/59787/cwd (readlink: Permission denied) 2024-03-18T15:20:22.887 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 59787 OpHistory ceph rtd unknown /proc/59604/task/59787/root (readlink: Permission denied) 2024-03-18T15:20:22.887 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 59787 OpHistory ceph txt unknown /proc/59604/task/59787/exe (readlink: Permission denied) 2024-03-18T15:20:22.887 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 59787 OpHistory ceph NOFD /proc/59604/task/59787/fd (opendir: Permission denied) 2024-03-18T15:20:22.887 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 59788 ceph_time ceph cwd unknown /proc/59604/task/59788/cwd (readlink: Permission denied) 2024-03-18T15:20:22.887 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 59788 ceph_time ceph rtd unknown /proc/59604/task/59788/root (readlink: Permission denied) 2024-03-18T15:20:22.887 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 59788 ceph_time ceph txt unknown /proc/59604/task/59788/exe (readlink: Permission denied) 2024-03-18T15:20:22.888 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 59788 ceph_time ceph NOFD /proc/59604/task/59788/fd (opendir: Permission denied) 2024-03-18T15:20:22.888 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 59880 ceph-osd ceph cwd unknown /proc/59604/task/59880/cwd (readlink: Permission denied) 2024-03-18T15:20:22.888 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 59880 ceph-osd ceph rtd unknown /proc/59604/task/59880/root (readlink: Permission denied) 2024-03-18T15:20:22.888 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 59880 ceph-osd ceph txt unknown /proc/59604/task/59880/exe (readlink: Permission denied) 2024-03-18T15:20:22.888 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 59880 ceph-osd ceph NOFD /proc/59604/task/59880/fd (opendir: Permission denied) 2024-03-18T15:20:22.888 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 59902 ceph-osd ceph cwd unknown /proc/59604/task/59902/cwd (readlink: Permission denied) 2024-03-18T15:20:22.888 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 59902 ceph-osd ceph rtd unknown /proc/59604/task/59902/root (readlink: Permission denied) 2024-03-18T15:20:22.888 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 59902 ceph-osd ceph txt unknown /proc/59604/task/59902/exe (readlink: Permission denied) 2024-03-18T15:20:22.888 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 59902 ceph-osd ceph NOFD /proc/59604/task/59902/fd (opendir: Permission denied) 2024-03-18T15:20:22.888 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 59920 ceph-osd ceph cwd unknown /proc/59604/task/59920/cwd (readlink: Permission denied) 2024-03-18T15:20:22.888 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 59920 ceph-osd ceph rtd unknown /proc/59604/task/59920/root (readlink: Permission denied) 2024-03-18T15:20:22.888 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 59920 ceph-osd ceph txt unknown /proc/59604/task/59920/exe (readlink: Permission denied) 2024-03-18T15:20:22.888 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 59920 ceph-osd ceph NOFD /proc/59604/task/59920/fd (opendir: Permission denied) 2024-03-18T15:20:22.888 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 59945 ceph-osd ceph cwd unknown /proc/59604/task/59945/cwd (readlink: Permission denied) 2024-03-18T15:20:22.888 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 59945 ceph-osd ceph rtd unknown /proc/59604/task/59945/root (readlink: Permission denied) 2024-03-18T15:20:22.889 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 59945 ceph-osd ceph txt unknown /proc/59604/task/59945/exe (readlink: Permission denied) 2024-03-18T15:20:22.889 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 59945 ceph-osd ceph NOFD /proc/59604/task/59945/fd (opendir: Permission denied) 2024-03-18T15:20:22.889 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 59951 ceph-osd ceph cwd unknown /proc/59604/task/59951/cwd (readlink: Permission denied) 2024-03-18T15:20:22.889 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 59951 ceph-osd ceph rtd unknown /proc/59604/task/59951/root (readlink: Permission denied) 2024-03-18T15:20:22.889 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 59951 ceph-osd ceph txt unknown /proc/59604/task/59951/exe (readlink: Permission denied) 2024-03-18T15:20:22.889 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 59951 ceph-osd ceph NOFD /proc/59604/task/59951/fd (opendir: Permission denied) 2024-03-18T15:20:22.889 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 59956 ceph-osd ceph cwd unknown /proc/59604/task/59956/cwd (readlink: Permission denied) 2024-03-18T15:20:22.889 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 59956 ceph-osd ceph rtd unknown /proc/59604/task/59956/root (readlink: Permission denied) 2024-03-18T15:20:22.889 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 59956 ceph-osd ceph txt unknown /proc/59604/task/59956/exe (readlink: Permission denied) 2024-03-18T15:20:22.889 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 59956 ceph-osd ceph NOFD /proc/59604/task/59956/fd (opendir: Permission denied) 2024-03-18T15:20:22.889 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 59979 ceph-osd ceph cwd unknown /proc/59604/task/59979/cwd (readlink: Permission denied) 2024-03-18T15:20:22.889 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 59979 ceph-osd ceph rtd unknown /proc/59604/task/59979/root (readlink: Permission denied) 2024-03-18T15:20:22.889 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 59979 ceph-osd ceph txt unknown /proc/59604/task/59979/exe (readlink: Permission denied) 2024-03-18T15:20:22.889 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 59979 ceph-osd ceph NOFD /proc/59604/task/59979/fd (opendir: Permission denied) 2024-03-18T15:20:22.890 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60004 ceph-osd ceph cwd unknown /proc/59604/task/60004/cwd (readlink: Permission denied) 2024-03-18T15:20:22.890 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60004 ceph-osd ceph rtd unknown /proc/59604/task/60004/root (readlink: Permission denied) 2024-03-18T15:20:22.890 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60004 ceph-osd ceph txt unknown /proc/59604/task/60004/exe (readlink: Permission denied) 2024-03-18T15:20:22.890 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60004 ceph-osd ceph NOFD /proc/59604/task/60004/fd (opendir: Permission denied) 2024-03-18T15:20:22.890 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60005 ceph-osd ceph cwd unknown /proc/59604/task/60005/cwd (readlink: Permission denied) 2024-03-18T15:20:22.890 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60005 ceph-osd ceph rtd unknown /proc/59604/task/60005/root (readlink: Permission denied) 2024-03-18T15:20:22.890 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60005 ceph-osd ceph txt unknown /proc/59604/task/60005/exe (readlink: Permission denied) 2024-03-18T15:20:22.890 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60005 ceph-osd ceph NOFD /proc/59604/task/60005/fd (opendir: Permission denied) 2024-03-18T15:20:22.890 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60006 safe_time ceph cwd unknown /proc/59604/task/60006/cwd (readlink: Permission denied) 2024-03-18T15:20:22.890 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60006 safe_time ceph rtd unknown /proc/59604/task/60006/root (readlink: Permission denied) 2024-03-18T15:20:22.890 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60006 safe_time ceph txt unknown /proc/59604/task/60006/exe (readlink: Permission denied) 2024-03-18T15:20:22.890 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60006 safe_time ceph NOFD /proc/59604/task/60006/fd (opendir: Permission denied) 2024-03-18T15:20:22.890 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60007 safe_time ceph cwd unknown /proc/59604/task/60007/cwd (readlink: Permission denied) 2024-03-18T15:20:22.890 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60007 safe_time ceph rtd unknown /proc/59604/task/60007/root (readlink: Permission denied) 2024-03-18T15:20:22.891 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60007 safe_time ceph txt unknown /proc/59604/task/60007/exe (readlink: Permission denied) 2024-03-18T15:20:22.891 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60007 safe_time ceph NOFD /proc/59604/task/60007/fd (opendir: Permission denied) 2024-03-18T15:20:22.891 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60008 safe_time ceph cwd unknown /proc/59604/task/60008/cwd (readlink: Permission denied) 2024-03-18T15:20:22.891 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60008 safe_time ceph rtd unknown /proc/59604/task/60008/root (readlink: Permission denied) 2024-03-18T15:20:22.891 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60008 safe_time ceph txt unknown /proc/59604/task/60008/exe (readlink: Permission denied) 2024-03-18T15:20:22.891 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60008 safe_time ceph NOFD /proc/59604/task/60008/fd (opendir: Permission denied) 2024-03-18T15:20:22.891 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60009 safe_time ceph cwd unknown /proc/59604/task/60009/cwd (readlink: Permission denied) 2024-03-18T15:20:22.891 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60009 safe_time ceph rtd unknown /proc/59604/task/60009/root (readlink: Permission denied) 2024-03-18T15:20:22.891 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60009 safe_time ceph txt unknown /proc/59604/task/60009/exe (readlink: Permission denied) 2024-03-18T15:20:22.891 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60009 safe_time ceph NOFD /proc/59604/task/60009/fd (opendir: Permission denied) 2024-03-18T15:20:22.891 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60010 fn_anonym ceph cwd unknown /proc/59604/task/60010/cwd (readlink: Permission denied) 2024-03-18T15:20:22.891 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60010 fn_anonym ceph rtd unknown /proc/59604/task/60010/root (readlink: Permission denied) 2024-03-18T15:20:22.891 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60010 fn_anonym ceph txt unknown /proc/59604/task/60010/exe (readlink: Permission denied) 2024-03-18T15:20:22.891 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60010 fn_anonym ceph NOFD /proc/59604/task/60010/fd (opendir: Permission denied) 2024-03-18T15:20:22.892 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60014 rocksdb:l ceph cwd unknown /proc/59604/task/60014/cwd (readlink: Permission denied) 2024-03-18T15:20:22.892 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60014 rocksdb:l ceph rtd unknown /proc/59604/task/60014/root (readlink: Permission denied) 2024-03-18T15:20:22.892 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60014 rocksdb:l ceph txt unknown /proc/59604/task/60014/exe (readlink: Permission denied) 2024-03-18T15:20:22.892 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60014 rocksdb:l ceph NOFD /proc/59604/task/60014/fd (opendir: Permission denied) 2024-03-18T15:20:22.892 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60015 rocksdb:l ceph cwd unknown /proc/59604/task/60015/cwd (readlink: Permission denied) 2024-03-18T15:20:22.892 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60015 rocksdb:l ceph rtd unknown /proc/59604/task/60015/root (readlink: Permission denied) 2024-03-18T15:20:22.892 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60015 rocksdb:l ceph txt unknown /proc/59604/task/60015/exe (readlink: Permission denied) 2024-03-18T15:20:22.892 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60015 rocksdb:l ceph NOFD /proc/59604/task/60015/fd (opendir: Permission denied) 2024-03-18T15:20:22.892 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60017 rocksdb:l ceph cwd unknown /proc/59604/task/60017/cwd (readlink: Permission denied) 2024-03-18T15:20:22.892 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60017 rocksdb:l ceph rtd unknown /proc/59604/task/60017/root (readlink: Permission denied) 2024-03-18T15:20:22.892 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60017 rocksdb:l ceph txt unknown /proc/59604/task/60017/exe (readlink: Permission denied) 2024-03-18T15:20:22.892 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60017 rocksdb:l ceph NOFD /proc/59604/task/60017/fd (opendir: Permission denied) 2024-03-18T15:20:22.892 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60018 rocksdb:h ceph cwd unknown /proc/59604/task/60018/cwd (readlink: Permission denied) 2024-03-18T15:20:22.893 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60018 rocksdb:h ceph rtd unknown /proc/59604/task/60018/root (readlink: Permission denied) 2024-03-18T15:20:22.893 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60018 rocksdb:h ceph txt unknown /proc/59604/task/60018/exe (readlink: Permission denied) 2024-03-18T15:20:22.893 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60018 rocksdb:h ceph NOFD /proc/59604/task/60018/fd (opendir: Permission denied) 2024-03-18T15:20:22.893 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60431 bstore_ai ceph cwd unknown /proc/59604/task/60431/cwd (readlink: Permission denied) 2024-03-18T15:20:22.893 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60431 bstore_ai ceph rtd unknown /proc/59604/task/60431/root (readlink: Permission denied) 2024-03-18T15:20:22.893 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60431 bstore_ai ceph txt unknown /proc/59604/task/60431/exe (readlink: Permission denied) 2024-03-18T15:20:22.893 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60431 bstore_ai ceph NOFD /proc/59604/task/60431/fd (opendir: Permission denied) 2024-03-18T15:20:22.893 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60658 bstore_ai ceph cwd unknown /proc/59604/task/60658/cwd (readlink: Permission denied) 2024-03-18T15:20:22.893 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60658 bstore_ai ceph rtd unknown /proc/59604/task/60658/root (readlink: Permission denied) 2024-03-18T15:20:22.893 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60658 bstore_ai ceph txt unknown /proc/59604/task/60658/exe (readlink: Permission denied) 2024-03-18T15:20:22.893 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60658 bstore_ai ceph NOFD /proc/59604/task/60658/fd (opendir: Permission denied) 2024-03-18T15:20:22.893 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60857 ceph-osd ceph cwd unknown /proc/59604/task/60857/cwd (readlink: Permission denied) 2024-03-18T15:20:22.893 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60857 ceph-osd ceph rtd unknown /proc/59604/task/60857/root (readlink: Permission denied) 2024-03-18T15:20:22.893 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60857 ceph-osd ceph txt unknown /proc/59604/task/60857/exe (readlink: Permission denied) 2024-03-18T15:20:22.894 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60857 ceph-osd ceph NOFD /proc/59604/task/60857/fd (opendir: Permission denied) 2024-03-18T15:20:22.894 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60858 cfin ceph cwd unknown /proc/59604/task/60858/cwd (readlink: Permission denied) 2024-03-18T15:20:22.894 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60858 cfin ceph rtd unknown /proc/59604/task/60858/root (readlink: Permission denied) 2024-03-18T15:20:22.894 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60858 cfin ceph txt unknown /proc/59604/task/60858/exe (readlink: Permission denied) 2024-03-18T15:20:22.894 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60858 cfin ceph NOFD /proc/59604/task/60858/fd (opendir: Permission denied) 2024-03-18T15:20:22.894 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60859 bstore_kv ceph cwd unknown /proc/59604/task/60859/cwd (readlink: Permission denied) 2024-03-18T15:20:22.894 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60859 bstore_kv ceph rtd unknown /proc/59604/task/60859/root (readlink: Permission denied) 2024-03-18T15:20:22.894 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60859 bstore_kv ceph txt unknown /proc/59604/task/60859/exe (readlink: Permission denied) 2024-03-18T15:20:22.894 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60859 bstore_kv ceph NOFD /proc/59604/task/60859/fd (opendir: Permission denied) 2024-03-18T15:20:22.894 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60860 bstore_kv ceph cwd unknown /proc/59604/task/60860/cwd (readlink: Permission denied) 2024-03-18T15:20:22.894 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60860 bstore_kv ceph rtd unknown /proc/59604/task/60860/root (readlink: Permission denied) 2024-03-18T15:20:22.894 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60860 bstore_kv ceph txt unknown /proc/59604/task/60860/exe (readlink: Permission denied) 2024-03-18T15:20:22.894 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60860 bstore_kv ceph NOFD /proc/59604/task/60860/fd (opendir: Permission denied) 2024-03-18T15:20:22.894 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60861 bstore_me ceph cwd unknown /proc/59604/task/60861/cwd (readlink: Permission denied) 2024-03-18T15:20:22.895 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60861 bstore_me ceph rtd unknown /proc/59604/task/60861/root (readlink: Permission denied) 2024-03-18T15:20:22.895 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60861 bstore_me ceph txt unknown /proc/59604/task/60861/exe (readlink: Permission denied) 2024-03-18T15:20:22.895 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60861 bstore_me ceph NOFD /proc/59604/task/60861/fd (opendir: Permission denied) 2024-03-18T15:20:22.895 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60872 ms_dispat ceph cwd unknown /proc/59604/task/60872/cwd (readlink: Permission denied) 2024-03-18T15:20:22.895 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60872 ms_dispat ceph rtd unknown /proc/59604/task/60872/root (readlink: Permission denied) 2024-03-18T15:20:22.895 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60872 ms_dispat ceph txt unknown /proc/59604/task/60872/exe (readlink: Permission denied) 2024-03-18T15:20:22.895 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60872 ms_dispat ceph NOFD /proc/59604/task/60872/fd (opendir: Permission denied) 2024-03-18T15:20:22.895 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60873 ms_local ceph cwd unknown /proc/59604/task/60873/cwd (readlink: Permission denied) 2024-03-18T15:20:22.895 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60873 ms_local ceph rtd unknown /proc/59604/task/60873/root (readlink: Permission denied) 2024-03-18T15:20:22.895 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60873 ms_local ceph txt unknown /proc/59604/task/60873/exe (readlink: Permission denied) 2024-03-18T15:20:22.895 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60873 ms_local ceph NOFD /proc/59604/task/60873/fd (opendir: Permission denied) 2024-03-18T15:20:22.895 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60874 safe_time ceph cwd unknown /proc/59604/task/60874/cwd (readlink: Permission denied) 2024-03-18T15:20:22.895 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60874 safe_time ceph rtd unknown /proc/59604/task/60874/root (readlink: Permission denied) 2024-03-18T15:20:22.895 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60874 safe_time ceph txt unknown /proc/59604/task/60874/exe (readlink: Permission denied) 2024-03-18T15:20:22.895 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60874 safe_time ceph NOFD /proc/59604/task/60874/fd (opendir: Permission denied) 2024-03-18T15:20:22.895 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60875 safe_time ceph cwd unknown /proc/59604/task/60875/cwd (readlink: Permission denied) 2024-03-18T15:20:22.896 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60875 safe_time ceph rtd unknown /proc/59604/task/60875/root (readlink: Permission denied) 2024-03-18T15:20:22.896 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60875 safe_time ceph txt unknown /proc/59604/task/60875/exe (readlink: Permission denied) 2024-03-18T15:20:22.896 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60875 safe_time ceph NOFD /proc/59604/task/60875/fd (opendir: Permission denied) 2024-03-18T15:20:22.896 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60876 ms_dispat ceph cwd unknown /proc/59604/task/60876/cwd (readlink: Permission denied) 2024-03-18T15:20:22.896 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60876 ms_dispat ceph rtd unknown /proc/59604/task/60876/root (readlink: Permission denied) 2024-03-18T15:20:22.896 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60876 ms_dispat ceph txt unknown /proc/59604/task/60876/exe (readlink: Permission denied) 2024-03-18T15:20:22.896 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60876 ms_dispat ceph NOFD /proc/59604/task/60876/fd (opendir: Permission denied) 2024-03-18T15:20:22.896 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60877 ms_local ceph cwd unknown /proc/59604/task/60877/cwd (readlink: Permission denied) 2024-03-18T15:20:22.896 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60877 ms_local ceph rtd unknown /proc/59604/task/60877/root (readlink: Permission denied) 2024-03-18T15:20:22.896 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60877 ms_local ceph txt unknown /proc/59604/task/60877/exe (readlink: Permission denied) 2024-03-18T15:20:22.896 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60877 ms_local ceph NOFD /proc/59604/task/60877/fd (opendir: Permission denied) 2024-03-18T15:20:22.896 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60878 ms_dispat ceph cwd unknown /proc/59604/task/60878/cwd (readlink: Permission denied) 2024-03-18T15:20:22.896 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60878 ms_dispat ceph rtd unknown /proc/59604/task/60878/root (readlink: Permission denied) 2024-03-18T15:20:22.897 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60878 ms_dispat ceph txt unknown /proc/59604/task/60878/exe (readlink: Permission denied) 2024-03-18T15:20:22.897 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60878 ms_dispat ceph NOFD /proc/59604/task/60878/fd (opendir: Permission denied) 2024-03-18T15:20:22.897 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60879 ms_local ceph cwd unknown /proc/59604/task/60879/cwd (readlink: Permission denied) 2024-03-18T15:20:22.897 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60879 ms_local ceph rtd unknown /proc/59604/task/60879/root (readlink: Permission denied) 2024-03-18T15:20:22.897 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60879 ms_local ceph txt unknown /proc/59604/task/60879/exe (readlink: Permission denied) 2024-03-18T15:20:22.897 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60879 ms_local ceph NOFD /proc/59604/task/60879/fd (opendir: Permission denied) 2024-03-18T15:20:22.897 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60880 ms_dispat ceph cwd unknown /proc/59604/task/60880/cwd (readlink: Permission denied) 2024-03-18T15:20:22.897 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60880 ms_dispat ceph rtd unknown /proc/59604/task/60880/root (readlink: Permission denied) 2024-03-18T15:20:22.897 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60880 ms_dispat ceph txt unknown /proc/59604/task/60880/exe (readlink: Permission denied) 2024-03-18T15:20:22.897 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60880 ms_dispat ceph NOFD /proc/59604/task/60880/fd (opendir: Permission denied) 2024-03-18T15:20:22.897 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60881 ms_local ceph cwd unknown /proc/59604/task/60881/cwd (readlink: Permission denied) 2024-03-18T15:20:22.897 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60881 ms_local ceph rtd unknown /proc/59604/task/60881/root (readlink: Permission denied) 2024-03-18T15:20:22.897 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60881 ms_local ceph txt unknown /proc/59604/task/60881/exe (readlink: Permission denied) 2024-03-18T15:20:22.897 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60881 ms_local ceph NOFD /proc/59604/task/60881/fd (opendir: Permission denied) 2024-03-18T15:20:22.898 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60882 ms_dispat ceph cwd unknown /proc/59604/task/60882/cwd (readlink: Permission denied) 2024-03-18T15:20:22.898 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60882 ms_dispat ceph rtd unknown /proc/59604/task/60882/root (readlink: Permission denied) 2024-03-18T15:20:22.898 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60882 ms_dispat ceph txt unknown /proc/59604/task/60882/exe (readlink: Permission denied) 2024-03-18T15:20:22.898 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60882 ms_dispat ceph NOFD /proc/59604/task/60882/fd (opendir: Permission denied) 2024-03-18T15:20:22.898 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60883 ms_local ceph cwd unknown /proc/59604/task/60883/cwd (readlink: Permission denied) 2024-03-18T15:20:22.898 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60883 ms_local ceph rtd unknown /proc/59604/task/60883/root (readlink: Permission denied) 2024-03-18T15:20:22.898 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60883 ms_local ceph txt unknown /proc/59604/task/60883/exe (readlink: Permission denied) 2024-03-18T15:20:22.898 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60883 ms_local ceph NOFD /proc/59604/task/60883/fd (opendir: Permission denied) 2024-03-18T15:20:22.898 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60884 ms_dispat ceph cwd unknown /proc/59604/task/60884/cwd (readlink: Permission denied) 2024-03-18T15:20:22.898 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60884 ms_dispat ceph rtd unknown /proc/59604/task/60884/root (readlink: Permission denied) 2024-03-18T15:20:22.898 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60884 ms_dispat ceph txt unknown /proc/59604/task/60884/exe (readlink: Permission denied) 2024-03-18T15:20:22.898 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60884 ms_dispat ceph NOFD /proc/59604/task/60884/fd (opendir: Permission denied) 2024-03-18T15:20:22.898 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60885 ms_local ceph cwd unknown /proc/59604/task/60885/cwd (readlink: Permission denied) 2024-03-18T15:20:22.898 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60885 ms_local ceph rtd unknown /proc/59604/task/60885/root (readlink: Permission denied) 2024-03-18T15:20:22.898 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60885 ms_local ceph txt unknown /proc/59604/task/60885/exe (readlink: Permission denied) 2024-03-18T15:20:22.899 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60885 ms_local ceph NOFD /proc/59604/task/60885/fd (opendir: Permission denied) 2024-03-18T15:20:22.899 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60886 ms_dispat ceph cwd unknown /proc/59604/task/60886/cwd (readlink: Permission denied) 2024-03-18T15:20:22.899 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60886 ms_dispat ceph rtd unknown /proc/59604/task/60886/root (readlink: Permission denied) 2024-03-18T15:20:22.899 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60886 ms_dispat ceph txt unknown /proc/59604/task/60886/exe (readlink: Permission denied) 2024-03-18T15:20:22.899 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60886 ms_dispat ceph NOFD /proc/59604/task/60886/fd (opendir: Permission denied) 2024-03-18T15:20:22.899 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60887 ms_local ceph cwd unknown /proc/59604/task/60887/cwd (readlink: Permission denied) 2024-03-18T15:20:22.899 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60887 ms_local ceph rtd unknown /proc/59604/task/60887/root (readlink: Permission denied) 2024-03-18T15:20:22.899 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60887 ms_local ceph txt unknown /proc/59604/task/60887/exe (readlink: Permission denied) 2024-03-18T15:20:22.899 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60887 ms_local ceph NOFD /proc/59604/task/60887/fd (opendir: Permission denied) 2024-03-18T15:20:22.899 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60889 fn_anonym ceph cwd unknown /proc/59604/task/60889/cwd (readlink: Permission denied) 2024-03-18T15:20:22.899 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60889 fn_anonym ceph rtd unknown /proc/59604/task/60889/root (readlink: Permission denied) 2024-03-18T15:20:22.899 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60889 fn_anonym ceph txt unknown /proc/59604/task/60889/exe (readlink: Permission denied) 2024-03-18T15:20:22.899 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60889 fn_anonym ceph NOFD /proc/59604/task/60889/fd (opendir: Permission denied) 2024-03-18T15:20:22.899 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60890 finisher ceph cwd unknown /proc/59604/task/60890/cwd (readlink: Permission denied) 2024-03-18T15:20:22.900 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60890 finisher ceph rtd unknown /proc/59604/task/60890/root (readlink: Permission denied) 2024-03-18T15:20:22.900 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60890 finisher ceph txt unknown /proc/59604/task/60890/exe (readlink: Permission denied) 2024-03-18T15:20:22.900 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60890 finisher ceph NOFD /proc/59604/task/60890/fd (opendir: Permission denied) 2024-03-18T15:20:22.900 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60891 safe_time ceph cwd unknown /proc/59604/task/60891/cwd (readlink: Permission denied) 2024-03-18T15:20:22.900 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60891 safe_time ceph rtd unknown /proc/59604/task/60891/root (readlink: Permission denied) 2024-03-18T15:20:22.900 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60891 safe_time ceph txt unknown /proc/59604/task/60891/exe (readlink: Permission denied) 2024-03-18T15:20:22.900 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60891 safe_time ceph NOFD /proc/59604/task/60891/fd (opendir: Permission denied) 2024-03-18T15:20:22.900 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60892 safe_time ceph cwd unknown /proc/59604/task/60892/cwd (readlink: Permission denied) 2024-03-18T15:20:22.900 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60892 safe_time ceph rtd unknown /proc/59604/task/60892/root (readlink: Permission denied) 2024-03-18T15:20:22.900 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60892 safe_time ceph txt unknown /proc/59604/task/60892/exe (readlink: Permission denied) 2024-03-18T15:20:22.900 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60892 safe_time ceph NOFD /proc/59604/task/60892/fd (opendir: Permission denied) 2024-03-18T15:20:22.900 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60894 ceph-osd ceph cwd unknown /proc/59604/task/60894/cwd (readlink: Permission denied) 2024-03-18T15:20:22.900 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60894 ceph-osd ceph rtd unknown /proc/59604/task/60894/root (readlink: Permission denied) 2024-03-18T15:20:22.900 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60894 ceph-osd ceph txt unknown /proc/59604/task/60894/exe (readlink: Permission denied) 2024-03-18T15:20:22.901 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60894 ceph-osd ceph NOFD /proc/59604/task/60894/fd (opendir: Permission denied) 2024-03-18T15:20:22.901 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60895 osd_srv_a ceph cwd unknown /proc/59604/task/60895/cwd (readlink: Permission denied) 2024-03-18T15:20:22.901 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60895 osd_srv_a ceph rtd unknown /proc/59604/task/60895/root (readlink: Permission denied) 2024-03-18T15:20:22.901 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60895 osd_srv_a ceph txt unknown /proc/59604/task/60895/exe (readlink: Permission denied) 2024-03-18T15:20:22.901 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60895 osd_srv_a ceph NOFD /proc/59604/task/60895/fd (opendir: Permission denied) 2024-03-18T15:20:22.901 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60897 tp_osd_tp ceph cwd unknown /proc/59604/task/60897/cwd (readlink: Permission denied) 2024-03-18T15:20:22.901 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60897 tp_osd_tp ceph rtd unknown /proc/59604/task/60897/root (readlink: Permission denied) 2024-03-18T15:20:22.901 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60897 tp_osd_tp ceph txt unknown /proc/59604/task/60897/exe (readlink: Permission denied) 2024-03-18T15:20:22.901 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60897 tp_osd_tp ceph NOFD /proc/59604/task/60897/fd (opendir: Permission denied) 2024-03-18T15:20:22.901 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60898 tp_osd_tp ceph cwd unknown /proc/59604/task/60898/cwd (readlink: Permission denied) 2024-03-18T15:20:22.901 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60898 tp_osd_tp ceph rtd unknown /proc/59604/task/60898/root (readlink: Permission denied) 2024-03-18T15:20:22.901 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60898 tp_osd_tp ceph txt unknown /proc/59604/task/60898/exe (readlink: Permission denied) 2024-03-18T15:20:22.901 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60898 tp_osd_tp ceph NOFD /proc/59604/task/60898/fd (opendir: Permission denied) 2024-03-18T15:20:22.901 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60899 tp_osd_tp ceph cwd unknown /proc/59604/task/60899/cwd (readlink: Permission denied) 2024-03-18T15:20:22.902 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60899 tp_osd_tp ceph rtd unknown /proc/59604/task/60899/root (readlink: Permission denied) 2024-03-18T15:20:22.902 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60899 tp_osd_tp ceph txt unknown /proc/59604/task/60899/exe (readlink: Permission denied) 2024-03-18T15:20:22.902 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60899 tp_osd_tp ceph NOFD /proc/59604/task/60899/fd (opendir: Permission denied) 2024-03-18T15:20:22.902 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60900 tp_osd_tp ceph cwd unknown /proc/59604/task/60900/cwd (readlink: Permission denied) 2024-03-18T15:20:22.902 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60900 tp_osd_tp ceph rtd unknown /proc/59604/task/60900/root (readlink: Permission denied) 2024-03-18T15:20:22.902 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60900 tp_osd_tp ceph txt unknown /proc/59604/task/60900/exe (readlink: Permission denied) 2024-03-18T15:20:22.902 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60900 tp_osd_tp ceph NOFD /proc/59604/task/60900/fd (opendir: Permission denied) 2024-03-18T15:20:22.902 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60901 tp_osd_tp ceph cwd unknown /proc/59604/task/60901/cwd (readlink: Permission denied) 2024-03-18T15:20:22.902 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60901 tp_osd_tp ceph rtd unknown /proc/59604/task/60901/root (readlink: Permission denied) 2024-03-18T15:20:22.902 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60901 tp_osd_tp ceph txt unknown /proc/59604/task/60901/exe (readlink: Permission denied) 2024-03-18T15:20:22.902 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60901 tp_osd_tp ceph NOFD /proc/59604/task/60901/fd (opendir: Permission denied) 2024-03-18T15:20:22.902 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60902 tp_osd_tp ceph cwd unknown /proc/59604/task/60902/cwd (readlink: Permission denied) 2024-03-18T15:20:22.902 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60902 tp_osd_tp ceph rtd unknown /proc/59604/task/60902/root (readlink: Permission denied) 2024-03-18T15:20:22.902 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60902 tp_osd_tp ceph txt unknown /proc/59604/task/60902/exe (readlink: Permission denied) 2024-03-18T15:20:22.903 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60902 tp_osd_tp ceph NOFD /proc/59604/task/60902/fd (opendir: Permission denied) 2024-03-18T15:20:22.903 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60903 tp_osd_tp ceph cwd unknown /proc/59604/task/60903/cwd (readlink: Permission denied) 2024-03-18T15:20:22.903 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60903 tp_osd_tp ceph rtd unknown /proc/59604/task/60903/root (readlink: Permission denied) 2024-03-18T15:20:22.903 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60903 tp_osd_tp ceph txt unknown /proc/59604/task/60903/exe (readlink: Permission denied) 2024-03-18T15:20:22.903 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60903 tp_osd_tp ceph NOFD /proc/59604/task/60903/fd (opendir: Permission denied) 2024-03-18T15:20:22.903 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60904 tp_osd_tp ceph cwd unknown /proc/59604/task/60904/cwd (readlink: Permission denied) 2024-03-18T15:20:22.903 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60904 tp_osd_tp ceph rtd unknown /proc/59604/task/60904/root (readlink: Permission denied) 2024-03-18T15:20:22.903 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60904 tp_osd_tp ceph txt unknown /proc/59604/task/60904/exe (readlink: Permission denied) 2024-03-18T15:20:22.903 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60904 tp_osd_tp ceph NOFD /proc/59604/task/60904/fd (opendir: Permission denied) 2024-03-18T15:20:22.903 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60905 tp_osd_tp ceph cwd unknown /proc/59604/task/60905/cwd (readlink: Permission denied) 2024-03-18T15:20:22.903 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60905 tp_osd_tp ceph rtd unknown /proc/59604/task/60905/root (readlink: Permission denied) 2024-03-18T15:20:22.903 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60905 tp_osd_tp ceph txt unknown /proc/59604/task/60905/exe (readlink: Permission denied) 2024-03-18T15:20:22.903 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60905 tp_osd_tp ceph NOFD /proc/59604/task/60905/fd (opendir: Permission denied) 2024-03-18T15:20:22.903 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60906 tp_osd_tp ceph cwd unknown /proc/59604/task/60906/cwd (readlink: Permission denied) 2024-03-18T15:20:22.904 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60906 tp_osd_tp ceph rtd unknown /proc/59604/task/60906/root (readlink: Permission denied) 2024-03-18T15:20:22.904 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60906 tp_osd_tp ceph txt unknown /proc/59604/task/60906/exe (readlink: Permission denied) 2024-03-18T15:20:22.904 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60906 tp_osd_tp ceph NOFD /proc/59604/task/60906/fd (opendir: Permission denied) 2024-03-18T15:20:22.904 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60907 tp_osd_tp ceph cwd unknown /proc/59604/task/60907/cwd (readlink: Permission denied) 2024-03-18T15:20:22.904 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60907 tp_osd_tp ceph rtd unknown /proc/59604/task/60907/root (readlink: Permission denied) 2024-03-18T15:20:22.904 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60907 tp_osd_tp ceph txt unknown /proc/59604/task/60907/exe (readlink: Permission denied) 2024-03-18T15:20:22.904 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60907 tp_osd_tp ceph NOFD /proc/59604/task/60907/fd (opendir: Permission denied) 2024-03-18T15:20:22.904 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60908 tp_osd_tp ceph cwd unknown /proc/59604/task/60908/cwd (readlink: Permission denied) 2024-03-18T15:20:22.904 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60908 tp_osd_tp ceph rtd unknown /proc/59604/task/60908/root (readlink: Permission denied) 2024-03-18T15:20:22.904 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60908 tp_osd_tp ceph txt unknown /proc/59604/task/60908/exe (readlink: Permission denied) 2024-03-18T15:20:22.904 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60908 tp_osd_tp ceph NOFD /proc/59604/task/60908/fd (opendir: Permission denied) 2024-03-18T15:20:22.904 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60909 tp_osd_tp ceph cwd unknown /proc/59604/task/60909/cwd (readlink: Permission denied) 2024-03-18T15:20:22.904 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60909 tp_osd_tp ceph rtd unknown /proc/59604/task/60909/root (readlink: Permission denied) 2024-03-18T15:20:22.904 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60909 tp_osd_tp ceph txt unknown /proc/59604/task/60909/exe (readlink: Permission denied) 2024-03-18T15:20:22.905 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60909 tp_osd_tp ceph NOFD /proc/59604/task/60909/fd (opendir: Permission denied) 2024-03-18T15:20:22.905 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60910 tp_osd_tp ceph cwd unknown /proc/59604/task/60910/cwd (readlink: Permission denied) 2024-03-18T15:20:22.905 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60910 tp_osd_tp ceph rtd unknown /proc/59604/task/60910/root (readlink: Permission denied) 2024-03-18T15:20:22.905 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60910 tp_osd_tp ceph txt unknown /proc/59604/task/60910/exe (readlink: Permission denied) 2024-03-18T15:20:22.905 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60910 tp_osd_tp ceph NOFD /proc/59604/task/60910/fd (opendir: Permission denied) 2024-03-18T15:20:22.905 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60911 tp_osd_tp ceph cwd unknown /proc/59604/task/60911/cwd (readlink: Permission denied) 2024-03-18T15:20:22.905 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60911 tp_osd_tp ceph rtd unknown /proc/59604/task/60911/root (readlink: Permission denied) 2024-03-18T15:20:22.905 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60911 tp_osd_tp ceph txt unknown /proc/59604/task/60911/exe (readlink: Permission denied) 2024-03-18T15:20:22.905 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60911 tp_osd_tp ceph NOFD /proc/59604/task/60911/fd (opendir: Permission denied) 2024-03-18T15:20:22.905 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60912 tp_osd_tp ceph cwd unknown /proc/59604/task/60912/cwd (readlink: Permission denied) 2024-03-18T15:20:22.905 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60912 tp_osd_tp ceph rtd unknown /proc/59604/task/60912/root (readlink: Permission denied) 2024-03-18T15:20:22.905 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60912 tp_osd_tp ceph txt unknown /proc/59604/task/60912/exe (readlink: Permission denied) 2024-03-18T15:20:22.905 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60912 tp_osd_tp ceph NOFD /proc/59604/task/60912/fd (opendir: Permission denied) 2024-03-18T15:20:22.905 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60913 osd_srv_h ceph cwd unknown /proc/59604/task/60913/cwd (readlink: Permission denied) 2024-03-18T15:20:22.906 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60913 osd_srv_h ceph rtd unknown /proc/59604/task/60913/root (readlink: Permission denied) 2024-03-18T15:20:22.906 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60913 osd_srv_h ceph txt unknown /proc/59604/task/60913/exe (readlink: Permission denied) 2024-03-18T15:20:22.906 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 59604 60913 osd_srv_h ceph NOFD /proc/59604/task/60913/fd (opendir: Permission denied) 2024-03-18T15:20:22.906 INFO:tasks.workunit.client.0.smithi019.stdout:sudo 61121 root cwd unknown /proc/61121/cwd (readlink: Permission denied) 2024-03-18T15:20:22.906 INFO:tasks.workunit.client.0.smithi019.stdout:sudo 61121 root rtd unknown /proc/61121/root (readlink: Permission denied) 2024-03-18T15:20:22.906 INFO:tasks.workunit.client.0.smithi019.stdout:sudo 61121 root txt unknown /proc/61121/exe (readlink: Permission denied) 2024-03-18T15:20:22.906 INFO:tasks.workunit.client.0.smithi019.stdout:sudo 61121 root NOFD /proc/61121/fd (opendir: Permission denied) 2024-03-18T15:20:22.906 INFO:tasks.workunit.client.0.smithi019.stdout:journalct 61151 root cwd unknown /proc/61151/cwd (readlink: Permission denied) 2024-03-18T15:20:22.906 INFO:tasks.workunit.client.0.smithi019.stdout:journalct 61151 root rtd unknown /proc/61151/root (readlink: Permission denied) 2024-03-18T15:20:22.906 INFO:tasks.workunit.client.0.smithi019.stdout:journalct 61151 root txt unknown /proc/61151/exe (readlink: Permission denied) 2024-03-18T15:20:22.906 INFO:tasks.workunit.client.0.smithi019.stdout:journalct 61151 root NOFD /proc/61151/fd (opendir: Permission denied) 2024-03-18T15:20:22.906 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/7 64348 root cwd unknown /proc/64348/cwd (readlink: Permission denied) 2024-03-18T15:20:22.906 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/7 64348 root rtd unknown /proc/64348/root (readlink: Permission denied) 2024-03-18T15:20:22.906 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/7 64348 root txt unknown /proc/64348/exe (readlink: Permission denied) 2024-03-18T15:20:22.907 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/7 64348 root NOFD /proc/64348/fd (opendir: Permission denied) 2024-03-18T15:20:22.907 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/2 65602 root cwd unknown /proc/65602/cwd (readlink: Permission denied) 2024-03-18T15:20:22.907 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/2 65602 root rtd unknown /proc/65602/root (readlink: Permission denied) 2024-03-18T15:20:22.907 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/2 65602 root txt unknown /proc/65602/exe (readlink: Permission denied) 2024-03-18T15:20:22.907 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/2 65602 root NOFD /proc/65602/fd (opendir: Permission denied) 2024-03-18T15:20:22.907 INFO:tasks.workunit.client.0.smithi019.stdout:conmon 66333 root cwd unknown /proc/66333/cwd (readlink: Permission denied) 2024-03-18T15:20:22.907 INFO:tasks.workunit.client.0.smithi019.stdout:conmon 66333 root rtd unknown /proc/66333/root (readlink: Permission denied) 2024-03-18T15:20:22.907 INFO:tasks.workunit.client.0.smithi019.stdout:conmon 66333 root txt unknown /proc/66333/exe (readlink: Permission denied) 2024-03-18T15:20:22.907 INFO:tasks.workunit.client.0.smithi019.stdout:conmon 66333 root NOFD /proc/66333/fd (opendir: Permission denied) 2024-03-18T15:20:22.907 INFO:tasks.workunit.client.0.smithi019.stdout:podman-in 66335 root cwd unknown /proc/66335/cwd (readlink: Permission denied) 2024-03-18T15:20:22.907 INFO:tasks.workunit.client.0.smithi019.stdout:podman-in 66335 root rtd unknown /proc/66335/root (readlink: Permission denied) 2024-03-18T15:20:22.907 INFO:tasks.workunit.client.0.smithi019.stdout:podman-in 66335 root txt unknown /proc/66335/exe (readlink: Permission denied) 2024-03-18T15:20:22.908 INFO:tasks.workunit.client.0.smithi019.stdout:podman-in 66335 root NOFD /proc/66335/fd (opendir: Permission denied) 2024-03-18T15:20:22.908 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 ceph cwd unknown /proc/66337/cwd (readlink: Permission denied) 2024-03-18T15:20:22.908 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 ceph rtd unknown /proc/66337/root (readlink: Permission denied) 2024-03-18T15:20:22.908 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 ceph txt unknown /proc/66337/exe (readlink: Permission denied) 2024-03-18T15:20:22.908 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 ceph NOFD /proc/66337/fd (opendir: Permission denied) 2024-03-18T15:20:22.908 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66338 log ceph cwd unknown /proc/66337/task/66338/cwd (readlink: Permission denied) 2024-03-18T15:20:22.908 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66338 log ceph rtd unknown /proc/66337/task/66338/root (readlink: Permission denied) 2024-03-18T15:20:22.908 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66338 log ceph txt unknown /proc/66337/task/66338/exe (readlink: Permission denied) 2024-03-18T15:20:22.908 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66338 log ceph NOFD /proc/66337/task/66338/fd (opendir: Permission denied) 2024-03-18T15:20:22.908 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66340 msgr-work ceph cwd unknown /proc/66337/task/66340/cwd (readlink: Permission denied) 2024-03-18T15:20:22.908 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66340 msgr-work ceph rtd unknown /proc/66337/task/66340/root (readlink: Permission denied) 2024-03-18T15:20:22.908 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66340 msgr-work ceph txt unknown /proc/66337/task/66340/exe (readlink: Permission denied) 2024-03-18T15:20:22.908 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66340 msgr-work ceph NOFD /proc/66337/task/66340/fd (opendir: Permission denied) 2024-03-18T15:20:22.908 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66341 msgr-work ceph cwd unknown /proc/66337/task/66341/cwd (readlink: Permission denied) 2024-03-18T15:20:22.909 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66341 msgr-work ceph rtd unknown /proc/66337/task/66341/root (readlink: Permission denied) 2024-03-18T15:20:22.909 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66341 msgr-work ceph txt unknown /proc/66337/task/66341/exe (readlink: Permission denied) 2024-03-18T15:20:22.909 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66341 msgr-work ceph NOFD /proc/66337/task/66341/fd (opendir: Permission denied) 2024-03-18T15:20:22.909 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66342 msgr-work ceph cwd unknown /proc/66337/task/66342/cwd (readlink: Permission denied) 2024-03-18T15:20:22.909 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66342 msgr-work ceph rtd unknown /proc/66337/task/66342/root (readlink: Permission denied) 2024-03-18T15:20:22.909 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66342 msgr-work ceph txt unknown /proc/66337/task/66342/exe (readlink: Permission denied) 2024-03-18T15:20:22.909 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66342 msgr-work ceph NOFD /proc/66337/task/66342/fd (opendir: Permission denied) 2024-03-18T15:20:22.909 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66346 service ceph cwd unknown /proc/66337/task/66346/cwd (readlink: Permission denied) 2024-03-18T15:20:22.909 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66346 service ceph rtd unknown /proc/66337/task/66346/root (readlink: Permission denied) 2024-03-18T15:20:22.909 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66346 service ceph txt unknown /proc/66337/task/66346/exe (readlink: Permission denied) 2024-03-18T15:20:22.909 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66346 service ceph NOFD /proc/66337/task/66346/fd (opendir: Permission denied) 2024-03-18T15:20:22.909 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66347 admin_soc ceph cwd unknown /proc/66337/task/66347/cwd (readlink: Permission denied) 2024-03-18T15:20:22.909 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66347 admin_soc ceph rtd unknown /proc/66337/task/66347/root (readlink: Permission denied) 2024-03-18T15:20:22.909 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66347 admin_soc ceph txt unknown /proc/66337/task/66347/exe (readlink: Permission denied) 2024-03-18T15:20:22.909 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66347 admin_soc ceph NOFD /proc/66337/task/66347/fd (opendir: Permission denied) 2024-03-18T15:20:22.910 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66492 signal_ha ceph cwd unknown /proc/66337/task/66492/cwd (readlink: Permission denied) 2024-03-18T15:20:22.910 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66492 signal_ha ceph rtd unknown /proc/66337/task/66492/root (readlink: Permission denied) 2024-03-18T15:20:22.910 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66492 signal_ha ceph txt unknown /proc/66337/task/66492/exe (readlink: Permission denied) 2024-03-18T15:20:22.910 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66492 signal_ha ceph NOFD /proc/66337/task/66492/fd (opendir: Permission denied) 2024-03-18T15:20:22.910 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66493 io_contex ceph cwd unknown /proc/66337/task/66493/cwd (readlink: Permission denied) 2024-03-18T15:20:22.910 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66493 io_contex ceph rtd unknown /proc/66337/task/66493/root (readlink: Permission denied) 2024-03-18T15:20:22.910 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66493 io_contex ceph txt unknown /proc/66337/task/66493/exe (readlink: Permission denied) 2024-03-18T15:20:22.910 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66493 io_contex ceph NOFD /proc/66337/task/66493/fd (opendir: Permission denied) 2024-03-18T15:20:22.910 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66494 io_contex ceph cwd unknown /proc/66337/task/66494/cwd (readlink: Permission denied) 2024-03-18T15:20:22.910 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66494 io_contex ceph rtd unknown /proc/66337/task/66494/root (readlink: Permission denied) 2024-03-18T15:20:22.910 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66494 io_contex ceph txt unknown /proc/66337/task/66494/exe (readlink: Permission denied) 2024-03-18T15:20:22.910 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66494 io_contex ceph NOFD /proc/66337/task/66494/fd (opendir: Permission denied) 2024-03-18T15:20:22.910 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66501 OpHistory ceph cwd unknown /proc/66337/task/66501/cwd (readlink: Permission denied) 2024-03-18T15:20:22.910 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66501 OpHistory ceph rtd unknown /proc/66337/task/66501/root (readlink: Permission denied) 2024-03-18T15:20:22.911 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66501 OpHistory ceph txt unknown /proc/66337/task/66501/exe (readlink: Permission denied) 2024-03-18T15:20:22.911 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66501 OpHistory ceph NOFD /proc/66337/task/66501/fd (opendir: Permission denied) 2024-03-18T15:20:22.911 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66502 ceph_time ceph cwd unknown /proc/66337/task/66502/cwd (readlink: Permission denied) 2024-03-18T15:20:22.911 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66502 ceph_time ceph rtd unknown /proc/66337/task/66502/root (readlink: Permission denied) 2024-03-18T15:20:22.911 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66502 ceph_time ceph txt unknown /proc/66337/task/66502/exe (readlink: Permission denied) 2024-03-18T15:20:22.911 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66502 ceph_time ceph NOFD /proc/66337/task/66502/fd (opendir: Permission denied) 2024-03-18T15:20:22.911 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66613 ceph-osd ceph cwd unknown /proc/66337/task/66613/cwd (readlink: Permission denied) 2024-03-18T15:20:22.911 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66613 ceph-osd ceph rtd unknown /proc/66337/task/66613/root (readlink: Permission denied) 2024-03-18T15:20:22.911 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66613 ceph-osd ceph txt unknown /proc/66337/task/66613/exe (readlink: Permission denied) 2024-03-18T15:20:22.911 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66613 ceph-osd ceph NOFD /proc/66337/task/66613/fd (opendir: Permission denied) 2024-03-18T15:20:22.911 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66633 ceph-osd ceph cwd unknown /proc/66337/task/66633/cwd (readlink: Permission denied) 2024-03-18T15:20:22.911 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66633 ceph-osd ceph rtd unknown /proc/66337/task/66633/root (readlink: Permission denied) 2024-03-18T15:20:22.911 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66633 ceph-osd ceph txt unknown /proc/66337/task/66633/exe (readlink: Permission denied) 2024-03-18T15:20:22.911 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66633 ceph-osd ceph NOFD /proc/66337/task/66633/fd (opendir: Permission denied) 2024-03-18T15:20:22.912 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66667 ceph-osd ceph cwd unknown /proc/66337/task/66667/cwd (readlink: Permission denied) 2024-03-18T15:20:22.912 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66667 ceph-osd ceph rtd unknown /proc/66337/task/66667/root (readlink: Permission denied) 2024-03-18T15:20:22.912 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66667 ceph-osd ceph txt unknown /proc/66337/task/66667/exe (readlink: Permission denied) 2024-03-18T15:20:22.912 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66667 ceph-osd ceph NOFD /proc/66337/task/66667/fd (opendir: Permission denied) 2024-03-18T15:20:22.912 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66676 ceph-osd ceph cwd unknown /proc/66337/task/66676/cwd (readlink: Permission denied) 2024-03-18T15:20:22.912 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66676 ceph-osd ceph rtd unknown /proc/66337/task/66676/root (readlink: Permission denied) 2024-03-18T15:20:22.912 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66676 ceph-osd ceph txt unknown /proc/66337/task/66676/exe (readlink: Permission denied) 2024-03-18T15:20:22.912 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66676 ceph-osd ceph NOFD /proc/66337/task/66676/fd (opendir: Permission denied) 2024-03-18T15:20:22.912 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66695 ceph-osd ceph cwd unknown /proc/66337/task/66695/cwd (readlink: Permission denied) 2024-03-18T15:20:22.912 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66695 ceph-osd ceph rtd unknown /proc/66337/task/66695/root (readlink: Permission denied) 2024-03-18T15:20:22.912 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66695 ceph-osd ceph txt unknown /proc/66337/task/66695/exe (readlink: Permission denied) 2024-03-18T15:20:22.912 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66695 ceph-osd ceph NOFD /proc/66337/task/66695/fd (opendir: Permission denied) 2024-03-18T15:20:22.912 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66711 ceph-osd ceph cwd unknown /proc/66337/task/66711/cwd (readlink: Permission denied) 2024-03-18T15:20:22.912 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66711 ceph-osd ceph rtd unknown /proc/66337/task/66711/root (readlink: Permission denied) 2024-03-18T15:20:22.913 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66711 ceph-osd ceph txt unknown /proc/66337/task/66711/exe (readlink: Permission denied) 2024-03-18T15:20:22.913 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66711 ceph-osd ceph NOFD /proc/66337/task/66711/fd (opendir: Permission denied) 2024-03-18T15:20:22.913 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66737 ceph-osd ceph cwd unknown /proc/66337/task/66737/cwd (readlink: Permission denied) 2024-03-18T15:20:22.913 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66737 ceph-osd ceph rtd unknown /proc/66337/task/66737/root (readlink: Permission denied) 2024-03-18T15:20:22.913 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66737 ceph-osd ceph txt unknown /proc/66337/task/66737/exe (readlink: Permission denied) 2024-03-18T15:20:22.913 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66737 ceph-osd ceph NOFD /proc/66337/task/66737/fd (opendir: Permission denied) 2024-03-18T15:20:22.913 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66758 ceph-osd ceph cwd unknown /proc/66337/task/66758/cwd (readlink: Permission denied) 2024-03-18T15:20:22.913 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66758 ceph-osd ceph rtd unknown /proc/66337/task/66758/root (readlink: Permission denied) 2024-03-18T15:20:22.913 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66758 ceph-osd ceph txt unknown /proc/66337/task/66758/exe (readlink: Permission denied) 2024-03-18T15:20:22.913 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66758 ceph-osd ceph NOFD /proc/66337/task/66758/fd (opendir: Permission denied) 2024-03-18T15:20:22.913 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66759 ceph-osd ceph cwd unknown /proc/66337/task/66759/cwd (readlink: Permission denied) 2024-03-18T15:20:22.913 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66759 ceph-osd ceph rtd unknown /proc/66337/task/66759/root (readlink: Permission denied) 2024-03-18T15:20:22.913 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66759 ceph-osd ceph txt unknown /proc/66337/task/66759/exe (readlink: Permission denied) 2024-03-18T15:20:22.913 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66759 ceph-osd ceph NOFD /proc/66337/task/66759/fd (opendir: Permission denied) 2024-03-18T15:20:22.913 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66760 safe_time ceph cwd unknown /proc/66337/task/66760/cwd (readlink: Permission denied) 2024-03-18T15:20:22.914 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66760 safe_time ceph rtd unknown /proc/66337/task/66760/root (readlink: Permission denied) 2024-03-18T15:20:22.914 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66760 safe_time ceph txt unknown /proc/66337/task/66760/exe (readlink: Permission denied) 2024-03-18T15:20:22.914 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66760 safe_time ceph NOFD /proc/66337/task/66760/fd (opendir: Permission denied) 2024-03-18T15:20:22.914 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66761 safe_time ceph cwd unknown /proc/66337/task/66761/cwd (readlink: Permission denied) 2024-03-18T15:20:22.914 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66761 safe_time ceph rtd unknown /proc/66337/task/66761/root (readlink: Permission denied) 2024-03-18T15:20:22.914 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66761 safe_time ceph txt unknown /proc/66337/task/66761/exe (readlink: Permission denied) 2024-03-18T15:20:22.914 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66761 safe_time ceph NOFD /proc/66337/task/66761/fd (opendir: Permission denied) 2024-03-18T15:20:22.914 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66762 safe_time ceph cwd unknown /proc/66337/task/66762/cwd (readlink: Permission denied) 2024-03-18T15:20:22.914 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66762 safe_time ceph rtd unknown /proc/66337/task/66762/root (readlink: Permission denied) 2024-03-18T15:20:22.914 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66762 safe_time ceph txt unknown /proc/66337/task/66762/exe (readlink: Permission denied) 2024-03-18T15:20:22.914 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66762 safe_time ceph NOFD /proc/66337/task/66762/fd (opendir: Permission denied) 2024-03-18T15:20:22.914 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66763 safe_time ceph cwd unknown /proc/66337/task/66763/cwd (readlink: Permission denied) 2024-03-18T15:20:22.914 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66763 safe_time ceph rtd unknown /proc/66337/task/66763/root (readlink: Permission denied) 2024-03-18T15:20:22.914 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66763 safe_time ceph txt unknown /proc/66337/task/66763/exe (readlink: Permission denied) 2024-03-18T15:20:22.915 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66763 safe_time ceph NOFD /proc/66337/task/66763/fd (opendir: Permission denied) 2024-03-18T15:20:22.915 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66764 fn_anonym ceph cwd unknown /proc/66337/task/66764/cwd (readlink: Permission denied) 2024-03-18T15:20:22.915 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66764 fn_anonym ceph rtd unknown /proc/66337/task/66764/root (readlink: Permission denied) 2024-03-18T15:20:22.915 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66764 fn_anonym ceph txt unknown /proc/66337/task/66764/exe (readlink: Permission denied) 2024-03-18T15:20:22.915 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66764 fn_anonym ceph NOFD /proc/66337/task/66764/fd (opendir: Permission denied) 2024-03-18T15:20:22.915 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66769 rocksdb:l ceph cwd unknown /proc/66337/task/66769/cwd (readlink: Permission denied) 2024-03-18T15:20:22.915 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66769 rocksdb:l ceph rtd unknown /proc/66337/task/66769/root (readlink: Permission denied) 2024-03-18T15:20:22.915 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66769 rocksdb:l ceph txt unknown /proc/66337/task/66769/exe (readlink: Permission denied) 2024-03-18T15:20:22.915 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66769 rocksdb:l ceph NOFD /proc/66337/task/66769/fd (opendir: Permission denied) 2024-03-18T15:20:22.915 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66770 rocksdb:l ceph cwd unknown /proc/66337/task/66770/cwd (readlink: Permission denied) 2024-03-18T15:20:22.915 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66770 rocksdb:l ceph rtd unknown /proc/66337/task/66770/root (readlink: Permission denied) 2024-03-18T15:20:22.915 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66770 rocksdb:l ceph txt unknown /proc/66337/task/66770/exe (readlink: Permission denied) 2024-03-18T15:20:22.915 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66770 rocksdb:l ceph NOFD /proc/66337/task/66770/fd (opendir: Permission denied) 2024-03-18T15:20:22.916 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66771 rocksdb:l ceph cwd unknown /proc/66337/task/66771/cwd (readlink: Permission denied) 2024-03-18T15:20:22.916 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66771 rocksdb:l ceph rtd unknown /proc/66337/task/66771/root (readlink: Permission denied) 2024-03-18T15:20:22.916 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66771 rocksdb:l ceph txt unknown /proc/66337/task/66771/exe (readlink: Permission denied) 2024-03-18T15:20:22.916 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66771 rocksdb:l ceph NOFD /proc/66337/task/66771/fd (opendir: Permission denied) 2024-03-18T15:20:22.916 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66772 rocksdb:h ceph cwd unknown /proc/66337/task/66772/cwd (readlink: Permission denied) 2024-03-18T15:20:22.916 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66772 rocksdb:h ceph rtd unknown /proc/66337/task/66772/root (readlink: Permission denied) 2024-03-18T15:20:22.916 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66772 rocksdb:h ceph txt unknown /proc/66337/task/66772/exe (readlink: Permission denied) 2024-03-18T15:20:22.916 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 66772 rocksdb:h ceph NOFD /proc/66337/task/66772/fd (opendir: Permission denied) 2024-03-18T15:20:22.916 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67221 bstore_ai ceph cwd unknown /proc/66337/task/67221/cwd (readlink: Permission denied) 2024-03-18T15:20:22.916 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67221 bstore_ai ceph rtd unknown /proc/66337/task/67221/root (readlink: Permission denied) 2024-03-18T15:20:22.916 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67221 bstore_ai ceph txt unknown /proc/66337/task/67221/exe (readlink: Permission denied) 2024-03-18T15:20:22.916 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67221 bstore_ai ceph NOFD /proc/66337/task/67221/fd (opendir: Permission denied) 2024-03-18T15:20:22.916 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67407 bstore_ai ceph cwd unknown /proc/66337/task/67407/cwd (readlink: Permission denied) 2024-03-18T15:20:22.916 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67407 bstore_ai ceph rtd unknown /proc/66337/task/67407/root (readlink: Permission denied) 2024-03-18T15:20:22.917 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67407 bstore_ai ceph txt unknown /proc/66337/task/67407/exe (readlink: Permission denied) 2024-03-18T15:20:22.917 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67407 bstore_ai ceph NOFD /proc/66337/task/67407/fd (opendir: Permission denied) 2024-03-18T15:20:22.917 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67590 ceph-osd ceph cwd unknown /proc/66337/task/67590/cwd (readlink: Permission denied) 2024-03-18T15:20:22.917 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67590 ceph-osd ceph rtd unknown /proc/66337/task/67590/root (readlink: Permission denied) 2024-03-18T15:20:22.917 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67590 ceph-osd ceph txt unknown /proc/66337/task/67590/exe (readlink: Permission denied) 2024-03-18T15:20:22.917 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67590 ceph-osd ceph NOFD /proc/66337/task/67590/fd (opendir: Permission denied) 2024-03-18T15:20:22.917 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67591 cfin ceph cwd unknown /proc/66337/task/67591/cwd (readlink: Permission denied) 2024-03-18T15:20:22.917 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67591 cfin ceph rtd unknown /proc/66337/task/67591/root (readlink: Permission denied) 2024-03-18T15:20:22.917 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67591 cfin ceph txt unknown /proc/66337/task/67591/exe (readlink: Permission denied) 2024-03-18T15:20:22.917 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67591 cfin ceph NOFD /proc/66337/task/67591/fd (opendir: Permission denied) 2024-03-18T15:20:22.917 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67592 bstore_kv ceph cwd unknown /proc/66337/task/67592/cwd (readlink: Permission denied) 2024-03-18T15:20:22.917 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67592 bstore_kv ceph rtd unknown /proc/66337/task/67592/root (readlink: Permission denied) 2024-03-18T15:20:22.917 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67592 bstore_kv ceph txt unknown /proc/66337/task/67592/exe (readlink: Permission denied) 2024-03-18T15:20:22.917 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67592 bstore_kv ceph NOFD /proc/66337/task/67592/fd (opendir: Permission denied) 2024-03-18T15:20:22.917 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67593 bstore_kv ceph cwd unknown /proc/66337/task/67593/cwd (readlink: Permission denied) 2024-03-18T15:20:22.918 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67593 bstore_kv ceph rtd unknown /proc/66337/task/67593/root (readlink: Permission denied) 2024-03-18T15:20:22.918 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67593 bstore_kv ceph txt unknown /proc/66337/task/67593/exe (readlink: Permission denied) 2024-03-18T15:20:22.918 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67593 bstore_kv ceph NOFD /proc/66337/task/67593/fd (opendir: Permission denied) 2024-03-18T15:20:22.918 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67594 bstore_me ceph cwd unknown /proc/66337/task/67594/cwd (readlink: Permission denied) 2024-03-18T15:20:22.918 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67594 bstore_me ceph rtd unknown /proc/66337/task/67594/root (readlink: Permission denied) 2024-03-18T15:20:22.918 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67594 bstore_me ceph txt unknown /proc/66337/task/67594/exe (readlink: Permission denied) 2024-03-18T15:20:22.918 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67594 bstore_me ceph NOFD /proc/66337/task/67594/fd (opendir: Permission denied) 2024-03-18T15:20:22.918 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67598 ms_dispat ceph cwd unknown /proc/66337/task/67598/cwd (readlink: Permission denied) 2024-03-18T15:20:22.918 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67598 ms_dispat ceph rtd unknown /proc/66337/task/67598/root (readlink: Permission denied) 2024-03-18T15:20:22.918 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67598 ms_dispat ceph txt unknown /proc/66337/task/67598/exe (readlink: Permission denied) 2024-03-18T15:20:22.918 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67598 ms_dispat ceph NOFD /proc/66337/task/67598/fd (opendir: Permission denied) 2024-03-18T15:20:22.918 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67599 ms_local ceph cwd unknown /proc/66337/task/67599/cwd (readlink: Permission denied) 2024-03-18T15:20:22.918 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67599 ms_local ceph rtd unknown /proc/66337/task/67599/root (readlink: Permission denied) 2024-03-18T15:20:22.919 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67599 ms_local ceph txt unknown /proc/66337/task/67599/exe (readlink: Permission denied) 2024-03-18T15:20:22.919 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67599 ms_local ceph NOFD /proc/66337/task/67599/fd (opendir: Permission denied) 2024-03-18T15:20:22.919 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67600 safe_time ceph cwd unknown /proc/66337/task/67600/cwd (readlink: Permission denied) 2024-03-18T15:20:22.919 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67600 safe_time ceph rtd unknown /proc/66337/task/67600/root (readlink: Permission denied) 2024-03-18T15:20:22.919 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67600 safe_time ceph txt unknown /proc/66337/task/67600/exe (readlink: Permission denied) 2024-03-18T15:20:22.919 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67600 safe_time ceph NOFD /proc/66337/task/67600/fd (opendir: Permission denied) 2024-03-18T15:20:22.919 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67601 safe_time ceph cwd unknown /proc/66337/task/67601/cwd (readlink: Permission denied) 2024-03-18T15:20:22.919 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67601 safe_time ceph rtd unknown /proc/66337/task/67601/root (readlink: Permission denied) 2024-03-18T15:20:22.919 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67601 safe_time ceph txt unknown /proc/66337/task/67601/exe (readlink: Permission denied) 2024-03-18T15:20:22.919 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67601 safe_time ceph NOFD /proc/66337/task/67601/fd (opendir: Permission denied) 2024-03-18T15:20:22.919 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67602 ms_dispat ceph cwd unknown /proc/66337/task/67602/cwd (readlink: Permission denied) 2024-03-18T15:20:22.919 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67602 ms_dispat ceph rtd unknown /proc/66337/task/67602/root (readlink: Permission denied) 2024-03-18T15:20:22.919 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67602 ms_dispat ceph txt unknown /proc/66337/task/67602/exe (readlink: Permission denied) 2024-03-18T15:20:22.919 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67602 ms_dispat ceph NOFD /proc/66337/task/67602/fd (opendir: Permission denied) 2024-03-18T15:20:22.919 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67603 ms_local ceph cwd unknown /proc/66337/task/67603/cwd (readlink: Permission denied) 2024-03-18T15:20:22.920 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67603 ms_local ceph rtd unknown /proc/66337/task/67603/root (readlink: Permission denied) 2024-03-18T15:20:22.920 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67603 ms_local ceph txt unknown /proc/66337/task/67603/exe (readlink: Permission denied) 2024-03-18T15:20:22.920 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67603 ms_local ceph NOFD /proc/66337/task/67603/fd (opendir: Permission denied) 2024-03-18T15:20:22.920 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67604 ms_dispat ceph cwd unknown /proc/66337/task/67604/cwd (readlink: Permission denied) 2024-03-18T15:20:22.920 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67604 ms_dispat ceph rtd unknown /proc/66337/task/67604/root (readlink: Permission denied) 2024-03-18T15:20:22.920 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67604 ms_dispat ceph txt unknown /proc/66337/task/67604/exe (readlink: Permission denied) 2024-03-18T15:20:22.920 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67604 ms_dispat ceph NOFD /proc/66337/task/67604/fd (opendir: Permission denied) 2024-03-18T15:20:22.920 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67605 ms_local ceph cwd unknown /proc/66337/task/67605/cwd (readlink: Permission denied) 2024-03-18T15:20:22.920 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67605 ms_local ceph rtd unknown /proc/66337/task/67605/root (readlink: Permission denied) 2024-03-18T15:20:22.920 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67605 ms_local ceph txt unknown /proc/66337/task/67605/exe (readlink: Permission denied) 2024-03-18T15:20:22.920 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67605 ms_local ceph NOFD /proc/66337/task/67605/fd (opendir: Permission denied) 2024-03-18T15:20:22.920 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67606 ms_dispat ceph cwd unknown /proc/66337/task/67606/cwd (readlink: Permission denied) 2024-03-18T15:20:22.920 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67606 ms_dispat ceph rtd unknown /proc/66337/task/67606/root (readlink: Permission denied) 2024-03-18T15:20:22.920 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67606 ms_dispat ceph txt unknown /proc/66337/task/67606/exe (readlink: Permission denied) 2024-03-18T15:20:22.920 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67606 ms_dispat ceph NOFD /proc/66337/task/67606/fd (opendir: Permission denied) 2024-03-18T15:20:22.921 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67607 ms_local ceph cwd unknown /proc/66337/task/67607/cwd (readlink: Permission denied) 2024-03-18T15:20:22.921 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67607 ms_local ceph rtd unknown /proc/66337/task/67607/root (readlink: Permission denied) 2024-03-18T15:20:22.921 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67607 ms_local ceph txt unknown /proc/66337/task/67607/exe (readlink: Permission denied) 2024-03-18T15:20:22.921 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67607 ms_local ceph NOFD /proc/66337/task/67607/fd (opendir: Permission denied) 2024-03-18T15:20:22.921 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67608 ms_dispat ceph cwd unknown /proc/66337/task/67608/cwd (readlink: Permission denied) 2024-03-18T15:20:22.921 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67608 ms_dispat ceph rtd unknown /proc/66337/task/67608/root (readlink: Permission denied) 2024-03-18T15:20:22.921 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67608 ms_dispat ceph txt unknown /proc/66337/task/67608/exe (readlink: Permission denied) 2024-03-18T15:20:22.921 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67608 ms_dispat ceph NOFD /proc/66337/task/67608/fd (opendir: Permission denied) 2024-03-18T15:20:22.921 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67609 ms_local ceph cwd unknown /proc/66337/task/67609/cwd (readlink: Permission denied) 2024-03-18T15:20:22.921 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67609 ms_local ceph rtd unknown /proc/66337/task/67609/root (readlink: Permission denied) 2024-03-18T15:20:22.921 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67609 ms_local ceph txt unknown /proc/66337/task/67609/exe (readlink: Permission denied) 2024-03-18T15:20:22.921 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67609 ms_local ceph NOFD /proc/66337/task/67609/fd (opendir: Permission denied) 2024-03-18T15:20:22.921 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67610 ms_dispat ceph cwd unknown /proc/66337/task/67610/cwd (readlink: Permission denied) 2024-03-18T15:20:22.922 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67610 ms_dispat ceph rtd unknown /proc/66337/task/67610/root (readlink: Permission denied) 2024-03-18T15:20:22.922 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67610 ms_dispat ceph txt unknown /proc/66337/task/67610/exe (readlink: Permission denied) 2024-03-18T15:20:22.922 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67610 ms_dispat ceph NOFD /proc/66337/task/67610/fd (opendir: Permission denied) 2024-03-18T15:20:22.922 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67611 ms_local ceph cwd unknown /proc/66337/task/67611/cwd (readlink: Permission denied) 2024-03-18T15:20:22.922 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67611 ms_local ceph rtd unknown /proc/66337/task/67611/root (readlink: Permission denied) 2024-03-18T15:20:22.922 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67611 ms_local ceph txt unknown /proc/66337/task/67611/exe (readlink: Permission denied) 2024-03-18T15:20:22.922 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67611 ms_local ceph NOFD /proc/66337/task/67611/fd (opendir: Permission denied) 2024-03-18T15:20:22.922 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67612 ms_dispat ceph cwd unknown /proc/66337/task/67612/cwd (readlink: Permission denied) 2024-03-18T15:20:22.922 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67612 ms_dispat ceph rtd unknown /proc/66337/task/67612/root (readlink: Permission denied) 2024-03-18T15:20:22.922 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67612 ms_dispat ceph txt unknown /proc/66337/task/67612/exe (readlink: Permission denied) 2024-03-18T15:20:22.922 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67612 ms_dispat ceph NOFD /proc/66337/task/67612/fd (opendir: Permission denied) 2024-03-18T15:20:22.922 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67613 ms_local ceph cwd unknown /proc/66337/task/67613/cwd (readlink: Permission denied) 2024-03-18T15:20:22.922 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67613 ms_local ceph rtd unknown /proc/66337/task/67613/root (readlink: Permission denied) 2024-03-18T15:20:22.922 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67613 ms_local ceph txt unknown /proc/66337/task/67613/exe (readlink: Permission denied) 2024-03-18T15:20:22.923 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67613 ms_local ceph NOFD /proc/66337/task/67613/fd (opendir: Permission denied) 2024-03-18T15:20:22.923 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67614 fn_anonym ceph cwd unknown /proc/66337/task/67614/cwd (readlink: Permission denied) 2024-03-18T15:20:22.923 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67614 fn_anonym ceph rtd unknown /proc/66337/task/67614/root (readlink: Permission denied) 2024-03-18T15:20:22.923 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67614 fn_anonym ceph txt unknown /proc/66337/task/67614/exe (readlink: Permission denied) 2024-03-18T15:20:22.923 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67614 fn_anonym ceph NOFD /proc/66337/task/67614/fd (opendir: Permission denied) 2024-03-18T15:20:22.923 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67615 finisher ceph cwd unknown /proc/66337/task/67615/cwd (readlink: Permission denied) 2024-03-18T15:20:22.923 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67615 finisher ceph rtd unknown /proc/66337/task/67615/root (readlink: Permission denied) 2024-03-18T15:20:22.923 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67615 finisher ceph txt unknown /proc/66337/task/67615/exe (readlink: Permission denied) 2024-03-18T15:20:22.923 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67615 finisher ceph NOFD /proc/66337/task/67615/fd (opendir: Permission denied) 2024-03-18T15:20:22.923 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67616 safe_time ceph cwd unknown /proc/66337/task/67616/cwd (readlink: Permission denied) 2024-03-18T15:20:22.923 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67616 safe_time ceph rtd unknown /proc/66337/task/67616/root (readlink: Permission denied) 2024-03-18T15:20:22.923 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67616 safe_time ceph txt unknown /proc/66337/task/67616/exe (readlink: Permission denied) 2024-03-18T15:20:22.923 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67616 safe_time ceph NOFD /proc/66337/task/67616/fd (opendir: Permission denied) 2024-03-18T15:20:22.923 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67617 safe_time ceph cwd unknown /proc/66337/task/67617/cwd (readlink: Permission denied) 2024-03-18T15:20:22.924 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67617 safe_time ceph rtd unknown /proc/66337/task/67617/root (readlink: Permission denied) 2024-03-18T15:20:22.924 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67617 safe_time ceph txt unknown /proc/66337/task/67617/exe (readlink: Permission denied) 2024-03-18T15:20:22.924 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67617 safe_time ceph NOFD /proc/66337/task/67617/fd (opendir: Permission denied) 2024-03-18T15:20:22.924 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67618 ceph-osd ceph cwd unknown /proc/66337/task/67618/cwd (readlink: Permission denied) 2024-03-18T15:20:22.924 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67618 ceph-osd ceph rtd unknown /proc/66337/task/67618/root (readlink: Permission denied) 2024-03-18T15:20:22.924 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67618 ceph-osd ceph txt unknown /proc/66337/task/67618/exe (readlink: Permission denied) 2024-03-18T15:20:22.924 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67618 ceph-osd ceph NOFD /proc/66337/task/67618/fd (opendir: Permission denied) 2024-03-18T15:20:22.924 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67619 osd_srv_a ceph cwd unknown /proc/66337/task/67619/cwd (readlink: Permission denied) 2024-03-18T15:20:22.924 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67619 osd_srv_a ceph rtd unknown /proc/66337/task/67619/root (readlink: Permission denied) 2024-03-18T15:20:22.924 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67619 osd_srv_a ceph txt unknown /proc/66337/task/67619/exe (readlink: Permission denied) 2024-03-18T15:20:22.924 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67619 osd_srv_a ceph NOFD /proc/66337/task/67619/fd (opendir: Permission denied) 2024-03-18T15:20:22.924 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67620 tp_osd_tp ceph cwd unknown /proc/66337/task/67620/cwd (readlink: Permission denied) 2024-03-18T15:20:22.924 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67620 tp_osd_tp ceph rtd unknown /proc/66337/task/67620/root (readlink: Permission denied) 2024-03-18T15:20:22.924 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67620 tp_osd_tp ceph txt unknown /proc/66337/task/67620/exe (readlink: Permission denied) 2024-03-18T15:20:22.925 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67620 tp_osd_tp ceph NOFD /proc/66337/task/67620/fd (opendir: Permission denied) 2024-03-18T15:20:22.925 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67621 tp_osd_tp ceph cwd unknown /proc/66337/task/67621/cwd (readlink: Permission denied) 2024-03-18T15:20:22.925 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67621 tp_osd_tp ceph rtd unknown /proc/66337/task/67621/root (readlink: Permission denied) 2024-03-18T15:20:22.925 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67621 tp_osd_tp ceph txt unknown /proc/66337/task/67621/exe (readlink: Permission denied) 2024-03-18T15:20:22.925 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67621 tp_osd_tp ceph NOFD /proc/66337/task/67621/fd (opendir: Permission denied) 2024-03-18T15:20:22.925 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67622 tp_osd_tp ceph cwd unknown /proc/66337/task/67622/cwd (readlink: Permission denied) 2024-03-18T15:20:22.925 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67622 tp_osd_tp ceph rtd unknown /proc/66337/task/67622/root (readlink: Permission denied) 2024-03-18T15:20:22.925 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67622 tp_osd_tp ceph txt unknown /proc/66337/task/67622/exe (readlink: Permission denied) 2024-03-18T15:20:22.925 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67622 tp_osd_tp ceph NOFD /proc/66337/task/67622/fd (opendir: Permission denied) 2024-03-18T15:20:22.925 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67623 tp_osd_tp ceph cwd unknown /proc/66337/task/67623/cwd (readlink: Permission denied) 2024-03-18T15:20:22.925 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67623 tp_osd_tp ceph rtd unknown /proc/66337/task/67623/root (readlink: Permission denied) 2024-03-18T15:20:22.925 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67623 tp_osd_tp ceph txt unknown /proc/66337/task/67623/exe (readlink: Permission denied) 2024-03-18T15:20:22.925 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67623 tp_osd_tp ceph NOFD /proc/66337/task/67623/fd (opendir: Permission denied) 2024-03-18T15:20:22.925 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67624 tp_osd_tp ceph cwd unknown /proc/66337/task/67624/cwd (readlink: Permission denied) 2024-03-18T15:20:22.925 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67624 tp_osd_tp ceph rtd unknown /proc/66337/task/67624/root (readlink: Permission denied) 2024-03-18T15:20:22.926 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67624 tp_osd_tp ceph txt unknown /proc/66337/task/67624/exe (readlink: Permission denied) 2024-03-18T15:20:22.926 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67624 tp_osd_tp ceph NOFD /proc/66337/task/67624/fd (opendir: Permission denied) 2024-03-18T15:20:22.926 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67625 tp_osd_tp ceph cwd unknown /proc/66337/task/67625/cwd (readlink: Permission denied) 2024-03-18T15:20:22.926 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67625 tp_osd_tp ceph rtd unknown /proc/66337/task/67625/root (readlink: Permission denied) 2024-03-18T15:20:22.926 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67625 tp_osd_tp ceph txt unknown /proc/66337/task/67625/exe (readlink: Permission denied) 2024-03-18T15:20:22.926 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67625 tp_osd_tp ceph NOFD /proc/66337/task/67625/fd (opendir: Permission denied) 2024-03-18T15:20:22.926 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67626 tp_osd_tp ceph cwd unknown /proc/66337/task/67626/cwd (readlink: Permission denied) 2024-03-18T15:20:22.926 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67626 tp_osd_tp ceph rtd unknown /proc/66337/task/67626/root (readlink: Permission denied) 2024-03-18T15:20:22.926 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67626 tp_osd_tp ceph txt unknown /proc/66337/task/67626/exe (readlink: Permission denied) 2024-03-18T15:20:22.926 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67626 tp_osd_tp ceph NOFD /proc/66337/task/67626/fd (opendir: Permission denied) 2024-03-18T15:20:22.926 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67627 tp_osd_tp ceph cwd unknown /proc/66337/task/67627/cwd (readlink: Permission denied) 2024-03-18T15:20:22.926 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67627 tp_osd_tp ceph rtd unknown /proc/66337/task/67627/root (readlink: Permission denied) 2024-03-18T15:20:22.926 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67627 tp_osd_tp ceph txt unknown /proc/66337/task/67627/exe (readlink: Permission denied) 2024-03-18T15:20:22.926 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67627 tp_osd_tp ceph NOFD /proc/66337/task/67627/fd (opendir: Permission denied) 2024-03-18T15:20:22.927 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67628 tp_osd_tp ceph cwd unknown /proc/66337/task/67628/cwd (readlink: Permission denied) 2024-03-18T15:20:22.927 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67628 tp_osd_tp ceph rtd unknown /proc/66337/task/67628/root (readlink: Permission denied) 2024-03-18T15:20:22.927 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67628 tp_osd_tp ceph txt unknown /proc/66337/task/67628/exe (readlink: Permission denied) 2024-03-18T15:20:22.927 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67628 tp_osd_tp ceph NOFD /proc/66337/task/67628/fd (opendir: Permission denied) 2024-03-18T15:20:22.927 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67629 tp_osd_tp ceph cwd unknown /proc/66337/task/67629/cwd (readlink: Permission denied) 2024-03-18T15:20:22.927 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67629 tp_osd_tp ceph rtd unknown /proc/66337/task/67629/root (readlink: Permission denied) 2024-03-18T15:20:22.927 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67629 tp_osd_tp ceph txt unknown /proc/66337/task/67629/exe (readlink: Permission denied) 2024-03-18T15:20:22.927 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67629 tp_osd_tp ceph NOFD /proc/66337/task/67629/fd (opendir: Permission denied) 2024-03-18T15:20:22.927 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67630 tp_osd_tp ceph cwd unknown /proc/66337/task/67630/cwd (readlink: Permission denied) 2024-03-18T15:20:22.927 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67630 tp_osd_tp ceph rtd unknown /proc/66337/task/67630/root (readlink: Permission denied) 2024-03-18T15:20:22.927 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67630 tp_osd_tp ceph txt unknown /proc/66337/task/67630/exe (readlink: Permission denied) 2024-03-18T15:20:22.927 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67630 tp_osd_tp ceph NOFD /proc/66337/task/67630/fd (opendir: Permission denied) 2024-03-18T15:20:22.927 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67631 tp_osd_tp ceph cwd unknown /proc/66337/task/67631/cwd (readlink: Permission denied) 2024-03-18T15:20:22.927 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67631 tp_osd_tp ceph rtd unknown /proc/66337/task/67631/root (readlink: Permission denied) 2024-03-18T15:20:22.927 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67631 tp_osd_tp ceph txt unknown /proc/66337/task/67631/exe (readlink: Permission denied) 2024-03-18T15:20:22.927 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67631 tp_osd_tp ceph NOFD /proc/66337/task/67631/fd (opendir: Permission denied) 2024-03-18T15:20:22.927 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67632 tp_osd_tp ceph cwd unknown /proc/66337/task/67632/cwd (readlink: Permission denied) 2024-03-18T15:20:22.928 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67632 tp_osd_tp ceph rtd unknown /proc/66337/task/67632/root (readlink: Permission denied) 2024-03-18T15:20:22.928 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67632 tp_osd_tp ceph txt unknown /proc/66337/task/67632/exe (readlink: Permission denied) 2024-03-18T15:20:22.928 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67632 tp_osd_tp ceph NOFD /proc/66337/task/67632/fd (opendir: Permission denied) 2024-03-18T15:20:22.928 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67633 tp_osd_tp ceph cwd unknown /proc/66337/task/67633/cwd (readlink: Permission denied) 2024-03-18T15:20:22.928 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67633 tp_osd_tp ceph rtd unknown /proc/66337/task/67633/root (readlink: Permission denied) 2024-03-18T15:20:22.928 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67633 tp_osd_tp ceph txt unknown /proc/66337/task/67633/exe (readlink: Permission denied) 2024-03-18T15:20:22.928 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67633 tp_osd_tp ceph NOFD /proc/66337/task/67633/fd (opendir: Permission denied) 2024-03-18T15:20:22.928 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67634 tp_osd_tp ceph cwd unknown /proc/66337/task/67634/cwd (readlink: Permission denied) 2024-03-18T15:20:22.928 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67634 tp_osd_tp ceph rtd unknown /proc/66337/task/67634/root (readlink: Permission denied) 2024-03-18T15:20:22.928 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67634 tp_osd_tp ceph txt unknown /proc/66337/task/67634/exe (readlink: Permission denied) 2024-03-18T15:20:22.928 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67634 tp_osd_tp ceph NOFD /proc/66337/task/67634/fd (opendir: Permission denied) 2024-03-18T15:20:22.928 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67635 tp_osd_tp ceph cwd unknown /proc/66337/task/67635/cwd (readlink: Permission denied) 2024-03-18T15:20:22.928 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67635 tp_osd_tp ceph rtd unknown /proc/66337/task/67635/root (readlink: Permission denied) 2024-03-18T15:20:22.928 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67635 tp_osd_tp ceph txt unknown /proc/66337/task/67635/exe (readlink: Permission denied) 2024-03-18T15:20:22.928 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67635 tp_osd_tp ceph NOFD /proc/66337/task/67635/fd (opendir: Permission denied) 2024-03-18T15:20:22.928 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67636 osd_srv_h ceph cwd unknown /proc/66337/task/67636/cwd (readlink: Permission denied) 2024-03-18T15:20:22.929 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67636 osd_srv_h ceph rtd unknown /proc/66337/task/67636/root (readlink: Permission denied) 2024-03-18T15:20:22.929 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67636 osd_srv_h ceph txt unknown /proc/66337/task/67636/exe (readlink: Permission denied) 2024-03-18T15:20:22.929 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-osd 66337 67636 osd_srv_h ceph NOFD /proc/66337/task/67636/fd (opendir: Permission denied) 2024-03-18T15:20:22.929 INFO:tasks.workunit.client.0.smithi019.stdout:sudo 67867 root cwd unknown /proc/67867/cwd (readlink: Permission denied) 2024-03-18T15:20:22.929 INFO:tasks.workunit.client.0.smithi019.stdout:sudo 67867 root rtd unknown /proc/67867/root (readlink: Permission denied) 2024-03-18T15:20:22.929 INFO:tasks.workunit.client.0.smithi019.stdout:sudo 67867 root txt unknown /proc/67867/exe (readlink: Permission denied) 2024-03-18T15:20:22.929 INFO:tasks.workunit.client.0.smithi019.stdout:sudo 67867 root NOFD /proc/67867/fd (opendir: Permission denied) 2024-03-18T15:20:22.929 INFO:tasks.workunit.client.0.smithi019.stdout:journalct 67899 root cwd unknown /proc/67899/cwd (readlink: Permission denied) 2024-03-18T15:20:22.929 INFO:tasks.workunit.client.0.smithi019.stdout:journalct 67899 root rtd unknown /proc/67899/root (readlink: Permission denied) 2024-03-18T15:20:22.929 INFO:tasks.workunit.client.0.smithi019.stdout:journalct 67899 root txt unknown /proc/67899/exe (readlink: Permission denied) 2024-03-18T15:20:22.929 INFO:tasks.workunit.client.0.smithi019.stdout:journalct 67899 root NOFD /proc/67899/fd (opendir: Permission denied) 2024-03-18T15:20:22.929 INFO:tasks.workunit.client.0.smithi019.stdout:sudo 71311 root cwd unknown /proc/71311/cwd (readlink: Permission denied) 2024-03-18T15:20:22.929 INFO:tasks.workunit.client.0.smithi019.stdout:sudo 71311 root rtd unknown /proc/71311/root (readlink: Permission denied) 2024-03-18T15:20:22.929 INFO:tasks.workunit.client.0.smithi019.stdout:sudo 71311 root txt unknown /proc/71311/exe (readlink: Permission denied) 2024-03-18T15:20:22.930 INFO:tasks.workunit.client.0.smithi019.stdout:sudo 71311 root NOFD /proc/71311/fd (opendir: Permission denied) 2024-03-18T15:20:22.930 INFO:tasks.workunit.client.0.smithi019.stdout:sudo 71327 root cwd unknown /proc/71327/cwd (readlink: Permission denied) 2024-03-18T15:20:22.930 INFO:tasks.workunit.client.0.smithi019.stdout:sudo 71327 root rtd unknown /proc/71327/root (readlink: Permission denied) 2024-03-18T15:20:22.930 INFO:tasks.workunit.client.0.smithi019.stdout:sudo 71327 root txt unknown /proc/71327/exe (readlink: Permission denied) 2024-03-18T15:20:22.930 INFO:tasks.workunit.client.0.smithi019.stdout:sudo 71327 root NOFD /proc/71327/fd (opendir: Permission denied) 2024-03-18T15:20:22.930 INFO:tasks.workunit.client.0.smithi019.stdout:journalct 71350 root cwd unknown /proc/71350/cwd (readlink: Permission denied) 2024-03-18T15:20:22.930 INFO:tasks.workunit.client.0.smithi019.stdout:journalct 71350 root rtd unknown /proc/71350/root (readlink: Permission denied) 2024-03-18T15:20:22.930 INFO:tasks.workunit.client.0.smithi019.stdout:journalct 71350 root txt unknown /proc/71350/exe (readlink: Permission denied) 2024-03-18T15:20:22.930 INFO:tasks.workunit.client.0.smithi019.stdout:journalct 71350 root NOFD /proc/71350/fd (opendir: Permission denied) 2024-03-18T15:20:22.930 INFO:tasks.workunit.client.0.smithi019.stdout:sudo 71356 root cwd unknown /proc/71356/cwd (readlink: Permission denied) 2024-03-18T15:20:22.930 INFO:tasks.workunit.client.0.smithi019.stdout:sudo 71356 root rtd unknown /proc/71356/root (readlink: Permission denied) 2024-03-18T15:20:22.930 INFO:tasks.workunit.client.0.smithi019.stdout:sudo 71356 root txt unknown /proc/71356/exe (readlink: Permission denied) 2024-03-18T15:20:22.931 INFO:tasks.workunit.client.0.smithi019.stdout:sudo 71356 root NOFD /proc/71356/fd (opendir: Permission denied) 2024-03-18T15:20:22.931 INFO:tasks.workunit.client.0.smithi019.stdout:journalct 71367 root cwd unknown /proc/71367/cwd (readlink: Permission denied) 2024-03-18T15:20:22.931 INFO:tasks.workunit.client.0.smithi019.stdout:journalct 71367 root rtd unknown /proc/71367/root (readlink: Permission denied) 2024-03-18T15:20:22.931 INFO:tasks.workunit.client.0.smithi019.stdout:journalct 71367 root txt unknown /proc/71367/exe (readlink: Permission denied) 2024-03-18T15:20:22.931 INFO:tasks.workunit.client.0.smithi019.stdout:journalct 71367 root NOFD /proc/71367/fd (opendir: Permission denied) 2024-03-18T15:20:22.931 INFO:tasks.workunit.client.0.smithi019.stdout:sudo 71377 root cwd unknown /proc/71377/cwd (readlink: Permission denied) 2024-03-18T15:20:22.931 INFO:tasks.workunit.client.0.smithi019.stdout:sudo 71377 root rtd unknown /proc/71377/root (readlink: Permission denied) 2024-03-18T15:20:22.931 INFO:tasks.workunit.client.0.smithi019.stdout:sudo 71377 root txt unknown /proc/71377/exe (readlink: Permission denied) 2024-03-18T15:20:22.931 INFO:tasks.workunit.client.0.smithi019.stdout:sudo 71377 root NOFD /proc/71377/fd (opendir: Permission denied) 2024-03-18T15:20:22.931 INFO:tasks.workunit.client.0.smithi019.stdout:journalct 71384 root cwd unknown /proc/71384/cwd (readlink: Permission denied) 2024-03-18T15:20:22.931 INFO:tasks.workunit.client.0.smithi019.stdout:journalct 71384 root rtd unknown /proc/71384/root (readlink: Permission denied) 2024-03-18T15:20:22.931 INFO:tasks.workunit.client.0.smithi019.stdout:journalct 71384 root txt unknown /proc/71384/exe (readlink: Permission denied) 2024-03-18T15:20:22.931 INFO:tasks.workunit.client.0.smithi019.stdout:journalct 71384 root NOFD /proc/71384/fd (opendir: Permission denied) 2024-03-18T15:20:22.931 INFO:tasks.workunit.client.0.smithi019.stdout:journalct 71398 root cwd unknown /proc/71398/cwd (readlink: Permission denied) 2024-03-18T15:20:22.932 INFO:tasks.workunit.client.0.smithi019.stdout:journalct 71398 root rtd unknown /proc/71398/root (readlink: Permission denied) 2024-03-18T15:20:22.932 INFO:tasks.workunit.client.0.smithi019.stdout:journalct 71398 root txt unknown /proc/71398/exe (readlink: Permission denied) 2024-03-18T15:20:22.932 INFO:tasks.workunit.client.0.smithi019.stdout:journalct 71398 root NOFD /proc/71398/fd (opendir: Permission denied) 2024-03-18T15:20:22.932 INFO:tasks.workunit.client.0.smithi019.stdout:conmon 71933 root cwd unknown /proc/71933/cwd (readlink: Permission denied) 2024-03-18T15:20:22.932 INFO:tasks.workunit.client.0.smithi019.stdout:conmon 71933 root rtd unknown /proc/71933/root (readlink: Permission denied) 2024-03-18T15:20:22.932 INFO:tasks.workunit.client.0.smithi019.stdout:conmon 71933 root txt unknown /proc/71933/exe (readlink: Permission denied) 2024-03-18T15:20:22.932 INFO:tasks.workunit.client.0.smithi019.stdout:conmon 71933 root NOFD /proc/71933/fd (opendir: Permission denied) 2024-03-18T15:20:22.932 INFO:tasks.workunit.client.0.smithi019.stdout:podman-in 71935 root cwd unknown /proc/71935/cwd (readlink: Permission denied) 2024-03-18T15:20:22.932 INFO:tasks.workunit.client.0.smithi019.stdout:podman-in 71935 root rtd unknown /proc/71935/root (readlink: Permission denied) 2024-03-18T15:20:22.932 INFO:tasks.workunit.client.0.smithi019.stdout:podman-in 71935 root txt unknown /proc/71935/exe (readlink: Permission denied) 2024-03-18T15:20:22.932 INFO:tasks.workunit.client.0.smithi019.stdout:podman-in 71935 root NOFD /proc/71935/fd (opendir: Permission denied) 2024-03-18T15:20:22.932 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 71937 ceph cwd unknown /proc/71937/cwd (readlink: Permission denied) 2024-03-18T15:20:22.932 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 71937 ceph rtd unknown /proc/71937/root (readlink: Permission denied) 2024-03-18T15:20:22.932 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 71937 ceph txt unknown /proc/71937/exe (readlink: Permission denied) 2024-03-18T15:20:22.932 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 71937 ceph NOFD /proc/71937/fd (opendir: Permission denied) 2024-03-18T15:20:22.932 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 71937 71938 log ceph cwd unknown /proc/71937/task/71938/cwd (readlink: Permission denied) 2024-03-18T15:20:22.933 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 71937 71938 log ceph rtd unknown /proc/71937/task/71938/root (readlink: Permission denied) 2024-03-18T15:20:22.933 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 71937 71938 log ceph txt unknown /proc/71937/task/71938/exe (readlink: Permission denied) 2024-03-18T15:20:22.933 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 71937 71938 log ceph NOFD /proc/71937/task/71938/fd (opendir: Permission denied) 2024-03-18T15:20:22.933 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 71937 71940 msgr-work ceph cwd unknown /proc/71937/task/71940/cwd (readlink: Permission denied) 2024-03-18T15:20:22.933 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 71937 71940 msgr-work ceph rtd unknown /proc/71937/task/71940/root (readlink: Permission denied) 2024-03-18T15:20:22.933 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 71937 71940 msgr-work ceph txt unknown /proc/71937/task/71940/exe (readlink: Permission denied) 2024-03-18T15:20:22.933 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 71937 71940 msgr-work ceph NOFD /proc/71937/task/71940/fd (opendir: Permission denied) 2024-03-18T15:20:22.933 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 71937 71941 msgr-work ceph cwd unknown /proc/71937/task/71941/cwd (readlink: Permission denied) 2024-03-18T15:20:22.933 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 71937 71941 msgr-work ceph rtd unknown /proc/71937/task/71941/root (readlink: Permission denied) 2024-03-18T15:20:22.933 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 71937 71941 msgr-work ceph txt unknown /proc/71937/task/71941/exe (readlink: Permission denied) 2024-03-18T15:20:22.933 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 71937 71941 msgr-work ceph NOFD /proc/71937/task/71941/fd (opendir: Permission denied) 2024-03-18T15:20:22.933 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 71937 71942 msgr-work ceph cwd unknown /proc/71937/task/71942/cwd (readlink: Permission denied) 2024-03-18T15:20:22.933 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 71937 71942 msgr-work ceph rtd unknown /proc/71937/task/71942/root (readlink: Permission denied) 2024-03-18T15:20:22.933 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 71937 71942 msgr-work ceph txt unknown /proc/71937/task/71942/exe (readlink: Permission denied) 2024-03-18T15:20:22.933 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 71937 71942 msgr-work ceph NOFD /proc/71937/task/71942/fd (opendir: Permission denied) 2024-03-18T15:20:22.934 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 71937 71946 service ceph cwd unknown /proc/71937/task/71946/cwd (readlink: Permission denied) 2024-03-18T15:20:22.934 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 71937 71946 service ceph rtd unknown /proc/71937/task/71946/root (readlink: Permission denied) 2024-03-18T15:20:22.934 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 71937 71946 service ceph txt unknown /proc/71937/task/71946/exe (readlink: Permission denied) 2024-03-18T15:20:22.934 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 71937 71946 service ceph NOFD /proc/71937/task/71946/fd (opendir: Permission denied) 2024-03-18T15:20:22.934 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 71937 71947 admin_soc ceph cwd unknown /proc/71937/task/71947/cwd (readlink: Permission denied) 2024-03-18T15:20:22.934 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 71937 71947 admin_soc ceph rtd unknown /proc/71937/task/71947/root (readlink: Permission denied) 2024-03-18T15:20:22.934 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 71937 71947 admin_soc ceph txt unknown /proc/71937/task/71947/exe (readlink: Permission denied) 2024-03-18T15:20:22.934 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 71937 71947 admin_soc ceph NOFD /proc/71937/task/71947/fd (opendir: Permission denied) 2024-03-18T15:20:22.934 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 71937 71948 signal_ha ceph cwd unknown /proc/71937/task/71948/cwd (readlink: Permission denied) 2024-03-18T15:20:22.934 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 71937 71948 signal_ha ceph rtd unknown /proc/71937/task/71948/root (readlink: Permission denied) 2024-03-18T15:20:22.934 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 71937 71948 signal_ha ceph txt unknown /proc/71937/task/71948/exe (readlink: Permission denied) 2024-03-18T15:20:22.934 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 71937 71948 signal_ha ceph NOFD /proc/71937/task/71948/fd (opendir: Permission denied) 2024-03-18T15:20:22.934 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 71937 71949 io_contex ceph cwd unknown /proc/71937/task/71949/cwd (readlink: Permission denied) 2024-03-18T15:20:22.935 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 71937 71949 io_contex ceph rtd unknown /proc/71937/task/71949/root (readlink: Permission denied) 2024-03-18T15:20:22.935 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 71937 71949 io_contex ceph txt unknown /proc/71937/task/71949/exe (readlink: Permission denied) 2024-03-18T15:20:22.935 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 71937 71949 io_contex ceph NOFD /proc/71937/task/71949/fd (opendir: Permission denied) 2024-03-18T15:20:22.935 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 71937 71950 io_contex ceph cwd unknown /proc/71937/task/71950/cwd (readlink: Permission denied) 2024-03-18T15:20:22.935 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 71937 71950 io_contex ceph rtd unknown /proc/71937/task/71950/root (readlink: Permission denied) 2024-03-18T15:20:22.935 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 71937 71950 io_contex ceph txt unknown /proc/71937/task/71950/exe (readlink: Permission denied) 2024-03-18T15:20:22.935 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 71937 71950 io_contex ceph NOFD /proc/71937/task/71950/fd (opendir: Permission denied) 2024-03-18T15:20:22.935 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 71937 71951 ms_dispat ceph cwd unknown /proc/71937/task/71951/cwd (readlink: Permission denied) 2024-03-18T15:20:22.935 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 71937 71951 ms_dispat ceph rtd unknown /proc/71937/task/71951/root (readlink: Permission denied) 2024-03-18T15:20:22.935 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 71937 71951 ms_dispat ceph txt unknown /proc/71937/task/71951/exe (readlink: Permission denied) 2024-03-18T15:20:22.935 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 71937 71951 ms_dispat ceph NOFD /proc/71937/task/71951/fd (opendir: Permission denied) 2024-03-18T15:20:22.935 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 71937 71952 ms_local ceph cwd unknown /proc/71937/task/71952/cwd (readlink: Permission denied) 2024-03-18T15:20:22.935 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 71937 71952 ms_local ceph rtd unknown /proc/71937/task/71952/root (readlink: Permission denied) 2024-03-18T15:20:22.935 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 71937 71952 ms_local ceph txt unknown /proc/71937/task/71952/exe (readlink: Permission denied) 2024-03-18T15:20:22.935 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 71937 71952 ms_local ceph NOFD /proc/71937/task/71952/fd (opendir: Permission denied) 2024-03-18T15:20:22.935 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 71937 71953 safe_time ceph cwd unknown /proc/71937/task/71953/cwd (readlink: Permission denied) 2024-03-18T15:20:22.936 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 71937 71953 safe_time ceph rtd unknown /proc/71937/task/71953/root (readlink: Permission denied) 2024-03-18T15:20:22.936 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 71937 71953 safe_time ceph txt unknown /proc/71937/task/71953/exe (readlink: Permission denied) 2024-03-18T15:20:22.936 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 71937 71953 safe_time ceph NOFD /proc/71937/task/71953/fd (opendir: Permission denied) 2024-03-18T15:20:22.936 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 71937 71954 safe_time ceph cwd unknown /proc/71937/task/71954/cwd (readlink: Permission denied) 2024-03-18T15:20:22.936 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 71937 71954 safe_time ceph rtd unknown /proc/71937/task/71954/root (readlink: Permission denied) 2024-03-18T15:20:22.936 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 71937 71954 safe_time ceph txt unknown /proc/71937/task/71954/exe (readlink: Permission denied) 2024-03-18T15:20:22.936 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 71937 71954 safe_time ceph NOFD /proc/71937/task/71954/fd (opendir: Permission denied) 2024-03-18T15:20:22.936 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 71937 71955 ceph-mds ceph cwd unknown /proc/71937/task/71955/cwd (readlink: Permission denied) 2024-03-18T15:20:22.936 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 71937 71955 ceph-mds ceph rtd unknown /proc/71937/task/71955/root (readlink: Permission denied) 2024-03-18T15:20:22.936 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 71937 71955 ceph-mds ceph txt unknown /proc/71937/task/71955/exe (readlink: Permission denied) 2024-03-18T15:20:22.936 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 71937 71955 ceph-mds ceph NOFD /proc/71937/task/71955/fd (opendir: Permission denied) 2024-03-18T15:20:22.936 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 71937 71956 safe_time ceph cwd unknown /proc/71937/task/71956/cwd (readlink: Permission denied) 2024-03-18T15:20:22.936 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 71937 71956 safe_time ceph rtd unknown /proc/71937/task/71956/root (readlink: Permission denied) 2024-03-18T15:20:22.936 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 71937 71956 safe_time ceph txt unknown /proc/71937/task/71956/exe (readlink: Permission denied) 2024-03-18T15:20:22.936 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 71937 71956 safe_time ceph NOFD /proc/71937/task/71956/fd (opendir: Permission denied) 2024-03-18T15:20:22.937 INFO:tasks.workunit.client.0.smithi019.stdout:conmon 72277 root cwd unknown /proc/72277/cwd (readlink: Permission denied) 2024-03-18T15:20:22.937 INFO:tasks.workunit.client.0.smithi019.stdout:conmon 72277 root rtd unknown /proc/72277/root (readlink: Permission denied) 2024-03-18T15:20:22.937 INFO:tasks.workunit.client.0.smithi019.stdout:conmon 72277 root txt unknown /proc/72277/exe (readlink: Permission denied) 2024-03-18T15:20:22.937 INFO:tasks.workunit.client.0.smithi019.stdout:conmon 72277 root NOFD /proc/72277/fd (opendir: Permission denied) 2024-03-18T15:20:22.937 INFO:tasks.workunit.client.0.smithi019.stdout:podman-in 72279 root cwd unknown /proc/72279/cwd (readlink: Permission denied) 2024-03-18T15:20:22.937 INFO:tasks.workunit.client.0.smithi019.stdout:podman-in 72279 root rtd unknown /proc/72279/root (readlink: Permission denied) 2024-03-18T15:20:22.937 INFO:tasks.workunit.client.0.smithi019.stdout:podman-in 72279 root txt unknown /proc/72279/exe (readlink: Permission denied) 2024-03-18T15:20:22.937 INFO:tasks.workunit.client.0.smithi019.stdout:podman-in 72279 root NOFD /proc/72279/fd (opendir: Permission denied) 2024-03-18T15:20:22.937 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 ceph cwd unknown /proc/72297/cwd (readlink: Permission denied) 2024-03-18T15:20:22.937 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 ceph rtd unknown /proc/72297/root (readlink: Permission denied) 2024-03-18T15:20:22.937 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 ceph txt unknown /proc/72297/exe (readlink: Permission denied) 2024-03-18T15:20:22.937 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 ceph NOFD /proc/72297/fd (opendir: Permission denied) 2024-03-18T15:20:22.937 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 72301 log ceph cwd unknown /proc/72297/task/72301/cwd (readlink: Permission denied) 2024-03-18T15:20:22.937 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 72301 log ceph rtd unknown /proc/72297/task/72301/root (readlink: Permission denied) 2024-03-18T15:20:22.937 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 72301 log ceph txt unknown /proc/72297/task/72301/exe (readlink: Permission denied) 2024-03-18T15:20:22.938 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 72301 log ceph NOFD /proc/72297/task/72301/fd (opendir: Permission denied) 2024-03-18T15:20:22.938 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 72303 msgr-work ceph cwd unknown /proc/72297/task/72303/cwd (readlink: Permission denied) 2024-03-18T15:20:22.938 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 72303 msgr-work ceph rtd unknown /proc/72297/task/72303/root (readlink: Permission denied) 2024-03-18T15:20:22.938 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 72303 msgr-work ceph txt unknown /proc/72297/task/72303/exe (readlink: Permission denied) 2024-03-18T15:20:22.938 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 72303 msgr-work ceph NOFD /proc/72297/task/72303/fd (opendir: Permission denied) 2024-03-18T15:20:22.938 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 72304 msgr-work ceph cwd unknown /proc/72297/task/72304/cwd (readlink: Permission denied) 2024-03-18T15:20:22.938 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 72304 msgr-work ceph rtd unknown /proc/72297/task/72304/root (readlink: Permission denied) 2024-03-18T15:20:22.938 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 72304 msgr-work ceph txt unknown /proc/72297/task/72304/exe (readlink: Permission denied) 2024-03-18T15:20:22.938 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 72304 msgr-work ceph NOFD /proc/72297/task/72304/fd (opendir: Permission denied) 2024-03-18T15:20:22.938 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 72305 msgr-work ceph cwd unknown /proc/72297/task/72305/cwd (readlink: Permission denied) 2024-03-18T15:20:22.938 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 72305 msgr-work ceph rtd unknown /proc/72297/task/72305/root (readlink: Permission denied) 2024-03-18T15:20:22.938 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 72305 msgr-work ceph txt unknown /proc/72297/task/72305/exe (readlink: Permission denied) 2024-03-18T15:20:22.938 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 72305 msgr-work ceph NOFD /proc/72297/task/72305/fd (opendir: Permission denied) 2024-03-18T15:20:22.938 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 72310 service ceph cwd unknown /proc/72297/task/72310/cwd (readlink: Permission denied) 2024-03-18T15:20:22.938 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 72310 service ceph rtd unknown /proc/72297/task/72310/root (readlink: Permission denied) 2024-03-18T15:20:22.939 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 72310 service ceph txt unknown /proc/72297/task/72310/exe (readlink: Permission denied) 2024-03-18T15:20:22.939 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 72310 service ceph NOFD /proc/72297/task/72310/fd (opendir: Permission denied) 2024-03-18T15:20:22.939 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 72311 admin_soc ceph cwd unknown /proc/72297/task/72311/cwd (readlink: Permission denied) 2024-03-18T15:20:22.939 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 72311 admin_soc ceph rtd unknown /proc/72297/task/72311/root (readlink: Permission denied) 2024-03-18T15:20:22.939 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 72311 admin_soc ceph txt unknown /proc/72297/task/72311/exe (readlink: Permission denied) 2024-03-18T15:20:22.939 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 72311 admin_soc ceph NOFD /proc/72297/task/72311/fd (opendir: Permission denied) 2024-03-18T15:20:22.939 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 72312 signal_ha ceph cwd unknown /proc/72297/task/72312/cwd (readlink: Permission denied) 2024-03-18T15:20:22.939 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 72312 signal_ha ceph rtd unknown /proc/72297/task/72312/root (readlink: Permission denied) 2024-03-18T15:20:22.939 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 72312 signal_ha ceph txt unknown /proc/72297/task/72312/exe (readlink: Permission denied) 2024-03-18T15:20:22.939 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 72312 signal_ha ceph NOFD /proc/72297/task/72312/fd (opendir: Permission denied) 2024-03-18T15:20:22.939 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 72313 io_contex ceph cwd unknown /proc/72297/task/72313/cwd (readlink: Permission denied) 2024-03-18T15:20:22.939 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 72313 io_contex ceph rtd unknown /proc/72297/task/72313/root (readlink: Permission denied) 2024-03-18T15:20:22.939 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 72313 io_contex ceph txt unknown /proc/72297/task/72313/exe (readlink: Permission denied) 2024-03-18T15:20:22.939 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 72313 io_contex ceph NOFD /proc/72297/task/72313/fd (opendir: Permission denied) 2024-03-18T15:20:22.939 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 72314 io_contex ceph cwd unknown /proc/72297/task/72314/cwd (readlink: Permission denied) 2024-03-18T15:20:22.939 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 72314 io_contex ceph rtd unknown /proc/72297/task/72314/root (readlink: Permission denied) 2024-03-18T15:20:22.940 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 72314 io_contex ceph txt unknown /proc/72297/task/72314/exe (readlink: Permission denied) 2024-03-18T15:20:22.940 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 72314 io_contex ceph NOFD /proc/72297/task/72314/fd (opendir: Permission denied) 2024-03-18T15:20:22.940 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 72315 ms_dispat ceph cwd unknown /proc/72297/task/72315/cwd (readlink: Permission denied) 2024-03-18T15:20:22.940 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 72315 ms_dispat ceph rtd unknown /proc/72297/task/72315/root (readlink: Permission denied) 2024-03-18T15:20:22.940 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 72315 ms_dispat ceph txt unknown /proc/72297/task/72315/exe (readlink: Permission denied) 2024-03-18T15:20:22.940 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 72315 ms_dispat ceph NOFD /proc/72297/task/72315/fd (opendir: Permission denied) 2024-03-18T15:20:22.940 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 72316 ms_local ceph cwd unknown /proc/72297/task/72316/cwd (readlink: Permission denied) 2024-03-18T15:20:22.940 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 72316 ms_local ceph rtd unknown /proc/72297/task/72316/root (readlink: Permission denied) 2024-03-18T15:20:22.940 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 72316 ms_local ceph txt unknown /proc/72297/task/72316/exe (readlink: Permission denied) 2024-03-18T15:20:22.940 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 72316 ms_local ceph NOFD /proc/72297/task/72316/fd (opendir: Permission denied) 2024-03-18T15:20:22.940 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 72317 safe_time ceph cwd unknown /proc/72297/task/72317/cwd (readlink: Permission denied) 2024-03-18T15:20:22.940 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 72317 safe_time ceph rtd unknown /proc/72297/task/72317/root (readlink: Permission denied) 2024-03-18T15:20:22.940 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 72317 safe_time ceph txt unknown /proc/72297/task/72317/exe (readlink: Permission denied) 2024-03-18T15:20:22.940 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 72317 safe_time ceph NOFD /proc/72297/task/72317/fd (opendir: Permission denied) 2024-03-18T15:20:22.940 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 72318 safe_time ceph cwd unknown /proc/72297/task/72318/cwd (readlink: Permission denied) 2024-03-18T15:20:22.941 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 72318 safe_time ceph rtd unknown /proc/72297/task/72318/root (readlink: Permission denied) 2024-03-18T15:20:22.941 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 72318 safe_time ceph txt unknown /proc/72297/task/72318/exe (readlink: Permission denied) 2024-03-18T15:20:22.941 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 72318 safe_time ceph NOFD /proc/72297/task/72318/fd (opendir: Permission denied) 2024-03-18T15:20:22.941 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 72319 ceph-mds ceph cwd unknown /proc/72297/task/72319/cwd (readlink: Permission denied) 2024-03-18T15:20:22.941 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 72319 ceph-mds ceph rtd unknown /proc/72297/task/72319/root (readlink: Permission denied) 2024-03-18T15:20:22.941 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 72319 ceph-mds ceph txt unknown /proc/72297/task/72319/exe (readlink: Permission denied) 2024-03-18T15:20:22.941 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 72319 ceph-mds ceph NOFD /proc/72297/task/72319/fd (opendir: Permission denied) 2024-03-18T15:20:22.941 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 72334 safe_time ceph cwd unknown /proc/72297/task/72334/cwd (readlink: Permission denied) 2024-03-18T15:20:22.941 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 72334 safe_time ceph rtd unknown /proc/72297/task/72334/root (readlink: Permission denied) 2024-03-18T15:20:22.941 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 72334 safe_time ceph txt unknown /proc/72297/task/72334/exe (readlink: Permission denied) 2024-03-18T15:20:22.941 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 72334 safe_time ceph NOFD /proc/72297/task/72334/fd (opendir: Permission denied) 2024-03-18T15:20:22.941 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 76571 ceph_time ceph cwd unknown /proc/72297/task/76571/cwd (readlink: Permission denied) 2024-03-18T15:20:22.941 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 76571 ceph_time ceph rtd unknown /proc/72297/task/76571/root (readlink: Permission denied) 2024-03-18T15:20:22.941 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 76571 ceph_time ceph txt unknown /proc/72297/task/76571/exe (readlink: Permission denied) 2024-03-18T15:20:22.941 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 76571 ceph_time ceph NOFD /proc/72297/task/76571/fd (opendir: Permission denied) 2024-03-18T15:20:22.941 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 76574 OpHistory ceph cwd unknown /proc/72297/task/76574/cwd (readlink: Permission denied) 2024-03-18T15:20:22.942 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 76574 OpHistory ceph rtd unknown /proc/72297/task/76574/root (readlink: Permission denied) 2024-03-18T15:20:22.942 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 76574 OpHistory ceph txt unknown /proc/72297/task/76574/exe (readlink: Permission denied) 2024-03-18T15:20:22.942 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 76574 OpHistory ceph NOFD /proc/72297/task/76574/fd (opendir: Permission denied) 2024-03-18T15:20:22.942 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 76578 ms_dispat ceph cwd unknown /proc/72297/task/76578/cwd (readlink: Permission denied) 2024-03-18T15:20:22.942 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 76578 ms_dispat ceph rtd unknown /proc/72297/task/76578/root (readlink: Permission denied) 2024-03-18T15:20:22.942 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 76578 ms_dispat ceph txt unknown /proc/72297/task/76578/exe (readlink: Permission denied) 2024-03-18T15:20:22.942 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 76578 ms_dispat ceph NOFD /proc/72297/task/76578/fd (opendir: Permission denied) 2024-03-18T15:20:22.942 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 76580 ms_dispat ceph cwd unknown /proc/72297/task/76580/cwd (readlink: Permission denied) 2024-03-18T15:20:22.942 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 76580 ms_dispat ceph rtd unknown /proc/72297/task/76580/root (readlink: Permission denied) 2024-03-18T15:20:22.942 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 76580 ms_dispat ceph txt unknown /proc/72297/task/76580/exe (readlink: Permission denied) 2024-03-18T15:20:22.942 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 76580 ms_dispat ceph NOFD /proc/72297/task/76580/fd (opendir: Permission denied) 2024-03-18T15:20:22.942 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 76583 mds_rank_ ceph cwd unknown /proc/72297/task/76583/cwd (readlink: Permission denied) 2024-03-18T15:20:22.943 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 76583 mds_rank_ ceph rtd unknown /proc/72297/task/76583/root (readlink: Permission denied) 2024-03-18T15:20:22.943 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 76583 mds_rank_ ceph txt unknown /proc/72297/task/76583/exe (readlink: Permission denied) 2024-03-18T15:20:22.943 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 76583 mds_rank_ ceph NOFD /proc/72297/task/76583/fd (opendir: Permission denied) 2024-03-18T15:20:22.943 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 76584 PQ_Finish ceph cwd unknown /proc/72297/task/76584/cwd (readlink: Permission denied) 2024-03-18T15:20:22.943 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 76584 PQ_Finish ceph rtd unknown /proc/72297/task/76584/root (readlink: Permission denied) 2024-03-18T15:20:22.943 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 76584 PQ_Finish ceph txt unknown /proc/72297/task/76584/exe (readlink: Permission denied) 2024-03-18T15:20:22.943 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 76584 PQ_Finish ceph NOFD /proc/72297/task/76584/fd (opendir: Permission denied) 2024-03-18T15:20:22.943 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 76585 safe_time ceph cwd unknown /proc/72297/task/76585/cwd (readlink: Permission denied) 2024-03-18T15:20:22.943 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 76585 safe_time ceph rtd unknown /proc/72297/task/76585/root (readlink: Permission denied) 2024-03-18T15:20:22.943 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 76585 safe_time ceph txt unknown /proc/72297/task/76585/exe (readlink: Permission denied) 2024-03-18T15:20:22.943 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 76585 safe_time ceph NOFD /proc/72297/task/76585/fd (opendir: Permission denied) 2024-03-18T15:20:22.943 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 76586 MR_Finish ceph cwd unknown /proc/72297/task/76586/cwd (readlink: Permission denied) 2024-03-18T15:20:22.943 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 76586 MR_Finish ceph rtd unknown /proc/72297/task/76586/root (readlink: Permission denied) 2024-03-18T15:20:22.943 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 76586 MR_Finish ceph txt unknown /proc/72297/task/76586/exe (readlink: Permission denied) 2024-03-18T15:20:22.943 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 76586 MR_Finish ceph NOFD /proc/72297/task/76586/fd (opendir: Permission denied) 2024-03-18T15:20:22.944 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 76588 md_submit ceph cwd unknown /proc/72297/task/76588/cwd (readlink: Permission denied) 2024-03-18T15:20:22.944 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 76588 md_submit ceph rtd unknown /proc/72297/task/76588/root (readlink: Permission denied) 2024-03-18T15:20:22.944 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 76588 md_submit ceph txt unknown /proc/72297/task/76588/exe (readlink: Permission denied) 2024-03-18T15:20:22.944 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72297 76588 md_submit ceph NOFD /proc/72297/task/76588/fd (opendir: Permission denied) 2024-03-18T15:20:22.944 INFO:tasks.workunit.client.0.smithi019.stdout:conmon 72921 root cwd unknown /proc/72921/cwd (readlink: Permission denied) 2024-03-18T15:20:22.944 INFO:tasks.workunit.client.0.smithi019.stdout:conmon 72921 root rtd unknown /proc/72921/root (readlink: Permission denied) 2024-03-18T15:20:22.944 INFO:tasks.workunit.client.0.smithi019.stdout:conmon 72921 root txt unknown /proc/72921/exe (readlink: Permission denied) 2024-03-18T15:20:22.944 INFO:tasks.workunit.client.0.smithi019.stdout:conmon 72921 root NOFD /proc/72921/fd (opendir: Permission denied) 2024-03-18T15:20:22.944 INFO:tasks.workunit.client.0.smithi019.stdout:podman-in 72923 root cwd unknown /proc/72923/cwd (readlink: Permission denied) 2024-03-18T15:20:22.944 INFO:tasks.workunit.client.0.smithi019.stdout:podman-in 72923 root rtd unknown /proc/72923/root (readlink: Permission denied) 2024-03-18T15:20:22.944 INFO:tasks.workunit.client.0.smithi019.stdout:podman-in 72923 root txt unknown /proc/72923/exe (readlink: Permission denied) 2024-03-18T15:20:22.944 INFO:tasks.workunit.client.0.smithi019.stdout:podman-in 72923 root NOFD /proc/72923/fd (opendir: Permission denied) 2024-03-18T15:20:22.944 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72925 ceph cwd unknown /proc/72925/cwd (readlink: Permission denied) 2024-03-18T15:20:22.944 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72925 ceph rtd unknown /proc/72925/root (readlink: Permission denied) 2024-03-18T15:20:22.944 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72925 ceph txt unknown /proc/72925/exe (readlink: Permission denied) 2024-03-18T15:20:22.945 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72925 ceph NOFD /proc/72925/fd (opendir: Permission denied) 2024-03-18T15:20:22.945 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72925 72926 log ceph cwd unknown /proc/72925/task/72926/cwd (readlink: Permission denied) 2024-03-18T15:20:22.945 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72925 72926 log ceph rtd unknown /proc/72925/task/72926/root (readlink: Permission denied) 2024-03-18T15:20:22.945 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72925 72926 log ceph txt unknown /proc/72925/task/72926/exe (readlink: Permission denied) 2024-03-18T15:20:22.945 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72925 72926 log ceph NOFD /proc/72925/task/72926/fd (opendir: Permission denied) 2024-03-18T15:20:22.945 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72925 72928 msgr-work ceph cwd unknown /proc/72925/task/72928/cwd (readlink: Permission denied) 2024-03-18T15:20:22.945 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72925 72928 msgr-work ceph rtd unknown /proc/72925/task/72928/root (readlink: Permission denied) 2024-03-18T15:20:22.945 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72925 72928 msgr-work ceph txt unknown /proc/72925/task/72928/exe (readlink: Permission denied) 2024-03-18T15:20:22.945 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72925 72928 msgr-work ceph NOFD /proc/72925/task/72928/fd (opendir: Permission denied) 2024-03-18T15:20:22.945 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72925 72929 msgr-work ceph cwd unknown /proc/72925/task/72929/cwd (readlink: Permission denied) 2024-03-18T15:20:22.945 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72925 72929 msgr-work ceph rtd unknown /proc/72925/task/72929/root (readlink: Permission denied) 2024-03-18T15:20:22.945 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72925 72929 msgr-work ceph txt unknown /proc/72925/task/72929/exe (readlink: Permission denied) 2024-03-18T15:20:22.945 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72925 72929 msgr-work ceph NOFD /proc/72925/task/72929/fd (opendir: Permission denied) 2024-03-18T15:20:22.945 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72925 72930 msgr-work ceph cwd unknown /proc/72925/task/72930/cwd (readlink: Permission denied) 2024-03-18T15:20:22.946 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72925 72930 msgr-work ceph rtd unknown /proc/72925/task/72930/root (readlink: Permission denied) 2024-03-18T15:20:22.946 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72925 72930 msgr-work ceph txt unknown /proc/72925/task/72930/exe (readlink: Permission denied) 2024-03-18T15:20:22.946 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72925 72930 msgr-work ceph NOFD /proc/72925/task/72930/fd (opendir: Permission denied) 2024-03-18T15:20:22.946 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72925 72934 service ceph cwd unknown /proc/72925/task/72934/cwd (readlink: Permission denied) 2024-03-18T15:20:22.946 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72925 72934 service ceph rtd unknown /proc/72925/task/72934/root (readlink: Permission denied) 2024-03-18T15:20:22.946 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72925 72934 service ceph txt unknown /proc/72925/task/72934/exe (readlink: Permission denied) 2024-03-18T15:20:22.946 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72925 72934 service ceph NOFD /proc/72925/task/72934/fd (opendir: Permission denied) 2024-03-18T15:20:22.946 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72925 72935 admin_soc ceph cwd unknown /proc/72925/task/72935/cwd (readlink: Permission denied) 2024-03-18T15:20:22.946 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72925 72935 admin_soc ceph rtd unknown /proc/72925/task/72935/root (readlink: Permission denied) 2024-03-18T15:20:22.946 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72925 72935 admin_soc ceph txt unknown /proc/72925/task/72935/exe (readlink: Permission denied) 2024-03-18T15:20:22.946 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72925 72935 admin_soc ceph NOFD /proc/72925/task/72935/fd (opendir: Permission denied) 2024-03-18T15:20:22.946 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72925 72936 signal_ha ceph cwd unknown /proc/72925/task/72936/cwd (readlink: Permission denied) 2024-03-18T15:20:22.946 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72925 72936 signal_ha ceph rtd unknown /proc/72925/task/72936/root (readlink: Permission denied) 2024-03-18T15:20:22.946 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72925 72936 signal_ha ceph txt unknown /proc/72925/task/72936/exe (readlink: Permission denied) 2024-03-18T15:20:22.947 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72925 72936 signal_ha ceph NOFD /proc/72925/task/72936/fd (opendir: Permission denied) 2024-03-18T15:20:22.947 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72925 72937 io_contex ceph cwd unknown /proc/72925/task/72937/cwd (readlink: Permission denied) 2024-03-18T15:20:22.947 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72925 72937 io_contex ceph rtd unknown /proc/72925/task/72937/root (readlink: Permission denied) 2024-03-18T15:20:22.947 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72925 72937 io_contex ceph txt unknown /proc/72925/task/72937/exe (readlink: Permission denied) 2024-03-18T15:20:22.947 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72925 72937 io_contex ceph NOFD /proc/72925/task/72937/fd (opendir: Permission denied) 2024-03-18T15:20:22.947 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72925 72938 io_contex ceph cwd unknown /proc/72925/task/72938/cwd (readlink: Permission denied) 2024-03-18T15:20:22.947 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72925 72938 io_contex ceph rtd unknown /proc/72925/task/72938/root (readlink: Permission denied) 2024-03-18T15:20:22.947 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72925 72938 io_contex ceph txt unknown /proc/72925/task/72938/exe (readlink: Permission denied) 2024-03-18T15:20:22.947 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72925 72938 io_contex ceph NOFD /proc/72925/task/72938/fd (opendir: Permission denied) 2024-03-18T15:20:22.947 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72925 72939 ms_dispat ceph cwd unknown /proc/72925/task/72939/cwd (readlink: Permission denied) 2024-03-18T15:20:22.947 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72925 72939 ms_dispat ceph rtd unknown /proc/72925/task/72939/root (readlink: Permission denied) 2024-03-18T15:20:22.947 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72925 72939 ms_dispat ceph txt unknown /proc/72925/task/72939/exe (readlink: Permission denied) 2024-03-18T15:20:22.947 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72925 72939 ms_dispat ceph NOFD /proc/72925/task/72939/fd (opendir: Permission denied) 2024-03-18T15:20:22.947 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72925 72940 ms_local ceph cwd unknown /proc/72925/task/72940/cwd (readlink: Permission denied) 2024-03-18T15:20:22.947 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72925 72940 ms_local ceph rtd unknown /proc/72925/task/72940/root (readlink: Permission denied) 2024-03-18T15:20:22.948 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72925 72940 ms_local ceph txt unknown /proc/72925/task/72940/exe (readlink: Permission denied) 2024-03-18T15:20:22.948 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72925 72940 ms_local ceph NOFD /proc/72925/task/72940/fd (opendir: Permission denied) 2024-03-18T15:20:22.948 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72925 72941 safe_time ceph cwd unknown /proc/72925/task/72941/cwd (readlink: Permission denied) 2024-03-18T15:20:22.948 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72925 72941 safe_time ceph rtd unknown /proc/72925/task/72941/root (readlink: Permission denied) 2024-03-18T15:20:22.948 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72925 72941 safe_time ceph txt unknown /proc/72925/task/72941/exe (readlink: Permission denied) 2024-03-18T15:20:22.948 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72925 72941 safe_time ceph NOFD /proc/72925/task/72941/fd (opendir: Permission denied) 2024-03-18T15:20:22.948 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72925 72942 safe_time ceph cwd unknown /proc/72925/task/72942/cwd (readlink: Permission denied) 2024-03-18T15:20:22.948 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72925 72942 safe_time ceph rtd unknown /proc/72925/task/72942/root (readlink: Permission denied) 2024-03-18T15:20:22.948 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72925 72942 safe_time ceph txt unknown /proc/72925/task/72942/exe (readlink: Permission denied) 2024-03-18T15:20:22.948 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72925 72942 safe_time ceph NOFD /proc/72925/task/72942/fd (opendir: Permission denied) 2024-03-18T15:20:22.948 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72925 72943 ceph-mds ceph cwd unknown /proc/72925/task/72943/cwd (readlink: Permission denied) 2024-03-18T15:20:22.948 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72925 72943 ceph-mds ceph rtd unknown /proc/72925/task/72943/root (readlink: Permission denied) 2024-03-18T15:20:22.948 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72925 72943 ceph-mds ceph txt unknown /proc/72925/task/72943/exe (readlink: Permission denied) 2024-03-18T15:20:22.948 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72925 72943 ceph-mds ceph NOFD /proc/72925/task/72943/fd (opendir: Permission denied) 2024-03-18T15:20:22.948 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72925 73015 safe_time ceph cwd unknown /proc/72925/task/73015/cwd (readlink: Permission denied) 2024-03-18T15:20:22.948 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72925 73015 safe_time ceph rtd unknown /proc/72925/task/73015/root (readlink: Permission denied) 2024-03-18T15:20:22.949 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72925 73015 safe_time ceph txt unknown /proc/72925/task/73015/exe (readlink: Permission denied) 2024-03-18T15:20:22.949 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 72925 73015 safe_time ceph NOFD /proc/72925/task/73015/fd (opendir: Permission denied) 2024-03-18T15:20:22.949 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/3 72982 root cwd unknown /proc/72982/cwd (readlink: Permission denied) 2024-03-18T15:20:22.949 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/3 72982 root rtd unknown /proc/72982/root (readlink: Permission denied) 2024-03-18T15:20:22.949 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/3 72982 root txt unknown /proc/72982/exe (readlink: Permission denied) 2024-03-18T15:20:22.949 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/3 72982 root NOFD /proc/72982/fd (opendir: Permission denied) 2024-03-18T15:20:22.949 INFO:tasks.workunit.client.0.smithi019.stdout:conmon 73284 root cwd unknown /proc/73284/cwd (readlink: Permission denied) 2024-03-18T15:20:22.949 INFO:tasks.workunit.client.0.smithi019.stdout:conmon 73284 root rtd unknown /proc/73284/root (readlink: Permission denied) 2024-03-18T15:20:22.949 INFO:tasks.workunit.client.0.smithi019.stdout:conmon 73284 root txt unknown /proc/73284/exe (readlink: Permission denied) 2024-03-18T15:20:22.949 INFO:tasks.workunit.client.0.smithi019.stdout:conmon 73284 root NOFD /proc/73284/fd (opendir: Permission denied) 2024-03-18T15:20:22.949 INFO:tasks.workunit.client.0.smithi019.stdout:podman-in 73286 root cwd unknown /proc/73286/cwd (readlink: Permission denied) 2024-03-18T15:20:22.949 INFO:tasks.workunit.client.0.smithi019.stdout:podman-in 73286 root rtd unknown /proc/73286/root (readlink: Permission denied) 2024-03-18T15:20:22.949 INFO:tasks.workunit.client.0.smithi019.stdout:podman-in 73286 root txt unknown /proc/73286/exe (readlink: Permission denied) 2024-03-18T15:20:22.949 INFO:tasks.workunit.client.0.smithi019.stdout:podman-in 73286 root NOFD /proc/73286/fd (opendir: Permission denied) 2024-03-18T15:20:22.949 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 ceph cwd unknown /proc/73307/cwd (readlink: Permission denied) 2024-03-18T15:20:22.950 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 ceph rtd unknown /proc/73307/root (readlink: Permission denied) 2024-03-18T15:20:22.950 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 ceph txt unknown /proc/73307/exe (readlink: Permission denied) 2024-03-18T15:20:22.950 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 ceph NOFD /proc/73307/fd (opendir: Permission denied) 2024-03-18T15:20:22.950 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 73308 log ceph cwd unknown /proc/73307/task/73308/cwd (readlink: Permission denied) 2024-03-18T15:20:22.950 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 73308 log ceph rtd unknown /proc/73307/task/73308/root (readlink: Permission denied) 2024-03-18T15:20:22.950 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 73308 log ceph txt unknown /proc/73307/task/73308/exe (readlink: Permission denied) 2024-03-18T15:20:22.950 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 73308 log ceph NOFD /proc/73307/task/73308/fd (opendir: Permission denied) 2024-03-18T15:20:22.950 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 73310 msgr-work ceph cwd unknown /proc/73307/task/73310/cwd (readlink: Permission denied) 2024-03-18T15:20:22.950 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 73310 msgr-work ceph rtd unknown /proc/73307/task/73310/root (readlink: Permission denied) 2024-03-18T15:20:22.950 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 73310 msgr-work ceph txt unknown /proc/73307/task/73310/exe (readlink: Permission denied) 2024-03-18T15:20:22.950 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 73310 msgr-work ceph NOFD /proc/73307/task/73310/fd (opendir: Permission denied) 2024-03-18T15:20:22.950 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 73311 msgr-work ceph cwd unknown /proc/73307/task/73311/cwd (readlink: Permission denied) 2024-03-18T15:20:22.950 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 73311 msgr-work ceph rtd unknown /proc/73307/task/73311/root (readlink: Permission denied) 2024-03-18T15:20:22.950 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 73311 msgr-work ceph txt unknown /proc/73307/task/73311/exe (readlink: Permission denied) 2024-03-18T15:20:22.951 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 73311 msgr-work ceph NOFD /proc/73307/task/73311/fd (opendir: Permission denied) 2024-03-18T15:20:22.951 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 73312 msgr-work ceph cwd unknown /proc/73307/task/73312/cwd (readlink: Permission denied) 2024-03-18T15:20:22.951 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 73312 msgr-work ceph rtd unknown /proc/73307/task/73312/root (readlink: Permission denied) 2024-03-18T15:20:22.951 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 73312 msgr-work ceph txt unknown /proc/73307/task/73312/exe (readlink: Permission denied) 2024-03-18T15:20:22.951 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 73312 msgr-work ceph NOFD /proc/73307/task/73312/fd (opendir: Permission denied) 2024-03-18T15:20:22.951 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 73316 service ceph cwd unknown /proc/73307/task/73316/cwd (readlink: Permission denied) 2024-03-18T15:20:22.951 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 73316 service ceph rtd unknown /proc/73307/task/73316/root (readlink: Permission denied) 2024-03-18T15:20:22.951 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 73316 service ceph txt unknown /proc/73307/task/73316/exe (readlink: Permission denied) 2024-03-18T15:20:22.951 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 73316 service ceph NOFD /proc/73307/task/73316/fd (opendir: Permission denied) 2024-03-18T15:20:22.951 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 73317 admin_soc ceph cwd unknown /proc/73307/task/73317/cwd (readlink: Permission denied) 2024-03-18T15:20:22.951 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 73317 admin_soc ceph rtd unknown /proc/73307/task/73317/root (readlink: Permission denied) 2024-03-18T15:20:22.951 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 73317 admin_soc ceph txt unknown /proc/73307/task/73317/exe (readlink: Permission denied) 2024-03-18T15:20:22.951 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 73317 admin_soc ceph NOFD /proc/73307/task/73317/fd (opendir: Permission denied) 2024-03-18T15:20:22.951 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 73318 signal_ha ceph cwd unknown /proc/73307/task/73318/cwd (readlink: Permission denied) 2024-03-18T15:20:22.952 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 73318 signal_ha ceph rtd unknown /proc/73307/task/73318/root (readlink: Permission denied) 2024-03-18T15:20:22.952 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 73318 signal_ha ceph txt unknown /proc/73307/task/73318/exe (readlink: Permission denied) 2024-03-18T15:20:22.952 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 73318 signal_ha ceph NOFD /proc/73307/task/73318/fd (opendir: Permission denied) 2024-03-18T15:20:22.952 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 73319 io_contex ceph cwd unknown /proc/73307/task/73319/cwd (readlink: Permission denied) 2024-03-18T15:20:22.952 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 73319 io_contex ceph rtd unknown /proc/73307/task/73319/root (readlink: Permission denied) 2024-03-18T15:20:22.952 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 73319 io_contex ceph txt unknown /proc/73307/task/73319/exe (readlink: Permission denied) 2024-03-18T15:20:22.952 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 73319 io_contex ceph NOFD /proc/73307/task/73319/fd (opendir: Permission denied) 2024-03-18T15:20:22.952 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 73320 io_contex ceph cwd unknown /proc/73307/task/73320/cwd (readlink: Permission denied) 2024-03-18T15:20:22.952 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 73320 io_contex ceph rtd unknown /proc/73307/task/73320/root (readlink: Permission denied) 2024-03-18T15:20:22.952 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 73320 io_contex ceph txt unknown /proc/73307/task/73320/exe (readlink: Permission denied) 2024-03-18T15:20:22.952 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 73320 io_contex ceph NOFD /proc/73307/task/73320/fd (opendir: Permission denied) 2024-03-18T15:20:22.952 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 73321 ms_dispat ceph cwd unknown /proc/73307/task/73321/cwd (readlink: Permission denied) 2024-03-18T15:20:22.952 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 73321 ms_dispat ceph rtd unknown /proc/73307/task/73321/root (readlink: Permission denied) 2024-03-18T15:20:22.952 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 73321 ms_dispat ceph txt unknown /proc/73307/task/73321/exe (readlink: Permission denied) 2024-03-18T15:20:22.952 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 73321 ms_dispat ceph NOFD /proc/73307/task/73321/fd (opendir: Permission denied) 2024-03-18T15:20:22.953 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 73322 ms_local ceph cwd unknown /proc/73307/task/73322/cwd (readlink: Permission denied) 2024-03-18T15:20:22.953 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 73322 ms_local ceph rtd unknown /proc/73307/task/73322/root (readlink: Permission denied) 2024-03-18T15:20:22.953 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 73322 ms_local ceph txt unknown /proc/73307/task/73322/exe (readlink: Permission denied) 2024-03-18T15:20:22.953 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 73322 ms_local ceph NOFD /proc/73307/task/73322/fd (opendir: Permission denied) 2024-03-18T15:20:22.953 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 73323 safe_time ceph cwd unknown /proc/73307/task/73323/cwd (readlink: Permission denied) 2024-03-18T15:20:22.953 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 73323 safe_time ceph rtd unknown /proc/73307/task/73323/root (readlink: Permission denied) 2024-03-18T15:20:22.953 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 73323 safe_time ceph txt unknown /proc/73307/task/73323/exe (readlink: Permission denied) 2024-03-18T15:20:22.953 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 73323 safe_time ceph NOFD /proc/73307/task/73323/fd (opendir: Permission denied) 2024-03-18T15:20:22.953 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 73324 safe_time ceph cwd unknown /proc/73307/task/73324/cwd (readlink: Permission denied) 2024-03-18T15:20:22.953 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 73324 safe_time ceph rtd unknown /proc/73307/task/73324/root (readlink: Permission denied) 2024-03-18T15:20:22.953 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 73324 safe_time ceph txt unknown /proc/73307/task/73324/exe (readlink: Permission denied) 2024-03-18T15:20:22.953 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 73324 safe_time ceph NOFD /proc/73307/task/73324/fd (opendir: Permission denied) 2024-03-18T15:20:22.953 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 73325 ceph-mds ceph cwd unknown /proc/73307/task/73325/cwd (readlink: Permission denied) 2024-03-18T15:20:22.953 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 73325 ceph-mds ceph rtd unknown /proc/73307/task/73325/root (readlink: Permission denied) 2024-03-18T15:20:22.953 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 73325 ceph-mds ceph txt unknown /proc/73307/task/73325/exe (readlink: Permission denied) 2024-03-18T15:20:22.954 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 73325 ceph-mds ceph NOFD /proc/73307/task/73325/fd (opendir: Permission denied) 2024-03-18T15:20:22.954 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 73332 safe_time ceph cwd unknown /proc/73307/task/73332/cwd (readlink: Permission denied) 2024-03-18T15:20:22.954 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 73332 safe_time ceph rtd unknown /proc/73307/task/73332/root (readlink: Permission denied) 2024-03-18T15:20:22.954 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 73332 safe_time ceph txt unknown /proc/73307/task/73332/exe (readlink: Permission denied) 2024-03-18T15:20:22.954 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 73332 safe_time ceph NOFD /proc/73307/task/73332/fd (opendir: Permission denied) 2024-03-18T15:20:22.954 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 76569 ceph_time ceph cwd unknown /proc/73307/task/76569/cwd (readlink: Permission denied) 2024-03-18T15:20:22.954 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 76569 ceph_time ceph rtd unknown /proc/73307/task/76569/root (readlink: Permission denied) 2024-03-18T15:20:22.954 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 76569 ceph_time ceph txt unknown /proc/73307/task/76569/exe (readlink: Permission denied) 2024-03-18T15:20:22.954 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 76569 ceph_time ceph NOFD /proc/73307/task/76569/fd (opendir: Permission denied) 2024-03-18T15:20:22.954 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 76570 OpHistory ceph cwd unknown /proc/73307/task/76570/cwd (readlink: Permission denied) 2024-03-18T15:20:22.954 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 76570 OpHistory ceph rtd unknown /proc/73307/task/76570/root (readlink: Permission denied) 2024-03-18T15:20:22.954 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 76570 OpHistory ceph txt unknown /proc/73307/task/76570/exe (readlink: Permission denied) 2024-03-18T15:20:22.954 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 76570 OpHistory ceph NOFD /proc/73307/task/76570/fd (opendir: Permission denied) 2024-03-18T15:20:22.954 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 76572 ms_dispat ceph cwd unknown /proc/73307/task/76572/cwd (readlink: Permission denied) 2024-03-18T15:20:22.955 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 76572 ms_dispat ceph rtd unknown /proc/73307/task/76572/root (readlink: Permission denied) 2024-03-18T15:20:22.955 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 76572 ms_dispat ceph txt unknown /proc/73307/task/76572/exe (readlink: Permission denied) 2024-03-18T15:20:22.955 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 76572 ms_dispat ceph NOFD /proc/73307/task/76572/fd (opendir: Permission denied) 2024-03-18T15:20:22.955 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 76573 ms_dispat ceph cwd unknown /proc/73307/task/76573/cwd (readlink: Permission denied) 2024-03-18T15:20:22.955 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 76573 ms_dispat ceph rtd unknown /proc/73307/task/76573/root (readlink: Permission denied) 2024-03-18T15:20:22.955 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 76573 ms_dispat ceph txt unknown /proc/73307/task/76573/exe (readlink: Permission denied) 2024-03-18T15:20:22.955 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 76573 ms_dispat ceph NOFD /proc/73307/task/76573/fd (opendir: Permission denied) 2024-03-18T15:20:22.955 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 76575 mds_rank_ ceph cwd unknown /proc/73307/task/76575/cwd (readlink: Permission denied) 2024-03-18T15:20:22.955 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 76575 mds_rank_ ceph rtd unknown /proc/73307/task/76575/root (readlink: Permission denied) 2024-03-18T15:20:22.955 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 76575 mds_rank_ ceph txt unknown /proc/73307/task/76575/exe (readlink: Permission denied) 2024-03-18T15:20:22.955 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 76575 mds_rank_ ceph NOFD /proc/73307/task/76575/fd (opendir: Permission denied) 2024-03-18T15:20:22.955 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 76576 PQ_Finish ceph cwd unknown /proc/73307/task/76576/cwd (readlink: Permission denied) 2024-03-18T15:20:22.955 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 76576 PQ_Finish ceph rtd unknown /proc/73307/task/76576/root (readlink: Permission denied) 2024-03-18T15:20:22.955 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 76576 PQ_Finish ceph txt unknown /proc/73307/task/76576/exe (readlink: Permission denied) 2024-03-18T15:20:22.956 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 76576 PQ_Finish ceph NOFD /proc/73307/task/76576/fd (opendir: Permission denied) 2024-03-18T15:20:22.956 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 76577 safe_time ceph cwd unknown /proc/73307/task/76577/cwd (readlink: Permission denied) 2024-03-18T15:20:22.956 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 76577 safe_time ceph rtd unknown /proc/73307/task/76577/root (readlink: Permission denied) 2024-03-18T15:20:22.956 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 76577 safe_time ceph txt unknown /proc/73307/task/76577/exe (readlink: Permission denied) 2024-03-18T15:20:22.956 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 76577 safe_time ceph NOFD /proc/73307/task/76577/fd (opendir: Permission denied) 2024-03-18T15:20:22.956 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 76579 MR_Finish ceph cwd unknown /proc/73307/task/76579/cwd (readlink: Permission denied) 2024-03-18T15:20:22.956 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 76579 MR_Finish ceph rtd unknown /proc/73307/task/76579/root (readlink: Permission denied) 2024-03-18T15:20:22.956 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 76579 MR_Finish ceph txt unknown /proc/73307/task/76579/exe (readlink: Permission denied) 2024-03-18T15:20:22.956 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 76579 MR_Finish ceph NOFD /proc/73307/task/76579/fd (opendir: Permission denied) 2024-03-18T15:20:22.956 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 76582 md_submit ceph cwd unknown /proc/73307/task/76582/cwd (readlink: Permission denied) 2024-03-18T15:20:22.956 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 76582 md_submit ceph rtd unknown /proc/73307/task/76582/root (readlink: Permission denied) 2024-03-18T15:20:22.956 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 76582 md_submit ceph txt unknown /proc/73307/task/76582/exe (readlink: Permission denied) 2024-03-18T15:20:22.956 INFO:tasks.workunit.client.0.smithi019.stdout:ceph-mds 73307 76582 md_submit ceph NOFD /proc/73307/task/76582/fd (opendir: Permission denied) 2024-03-18T15:20:22.956 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/u 73352 root cwd unknown /proc/73352/cwd (readlink: Permission denied) 2024-03-18T15:20:22.957 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/u 73352 root rtd unknown /proc/73352/root (readlink: Permission denied) 2024-03-18T15:20:22.957 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/u 73352 root txt unknown /proc/73352/exe (readlink: Permission denied) 2024-03-18T15:20:22.957 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/u 73352 root NOFD /proc/73352/fd (opendir: Permission denied) 2024-03-18T15:20:22.957 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/4 78317 root cwd unknown /proc/78317/cwd (readlink: Permission denied) 2024-03-18T15:20:22.957 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/4 78317 root rtd unknown /proc/78317/root (readlink: Permission denied) 2024-03-18T15:20:22.957 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/4 78317 root txt unknown /proc/78317/exe (readlink: Permission denied) 2024-03-18T15:20:22.957 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/4 78317 root NOFD /proc/78317/fd (opendir: Permission denied) 2024-03-18T15:20:22.957 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/5 79773 root cwd unknown /proc/79773/cwd (readlink: Permission denied) 2024-03-18T15:20:22.957 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/5 79773 root rtd unknown /proc/79773/root (readlink: Permission denied) 2024-03-18T15:20:22.957 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/5 79773 root txt unknown /proc/79773/exe (readlink: Permission denied) 2024-03-18T15:20:22.957 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/5 79773 root NOFD /proc/79773/fd (opendir: Permission denied) 2024-03-18T15:20:22.957 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/6 81692 root cwd unknown /proc/81692/cwd (readlink: Permission denied) 2024-03-18T15:20:22.957 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/6 81692 root rtd unknown /proc/81692/root (readlink: Permission denied) 2024-03-18T15:20:22.958 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/6 81692 root txt unknown /proc/81692/exe (readlink: Permission denied) 2024-03-18T15:20:22.958 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/6 81692 root NOFD /proc/81692/fd (opendir: Permission denied) 2024-03-18T15:20:22.958 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/0 85227 root cwd unknown /proc/85227/cwd (readlink: Permission denied) 2024-03-18T15:20:22.958 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/0 85227 root rtd unknown /proc/85227/root (readlink: Permission denied) 2024-03-18T15:20:22.958 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/0 85227 root txt unknown /proc/85227/exe (readlink: Permission denied) 2024-03-18T15:20:22.958 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/0 85227 root NOFD /proc/85227/fd (opendir: Permission denied) 2024-03-18T15:20:22.958 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/6 85372 root cwd unknown /proc/85372/cwd (readlink: Permission denied) 2024-03-18T15:20:22.958 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/6 85372 root rtd unknown /proc/85372/root (readlink: Permission denied) 2024-03-18T15:20:22.958 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/6 85372 root txt unknown /proc/85372/exe (readlink: Permission denied) 2024-03-18T15:20:22.958 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/6 85372 root NOFD /proc/85372/fd (opendir: Permission denied) 2024-03-18T15:20:22.958 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 86161 root cwd unknown /proc/86161/cwd (readlink: Permission denied) 2024-03-18T15:20:22.958 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 86161 root rtd unknown /proc/86161/root (readlink: Permission denied) 2024-03-18T15:20:22.958 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 86161 root txt unknown /proc/86161/exe (readlink: Permission denied) 2024-03-18T15:20:22.958 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 86161 root NOFD /proc/86161/fd (opendir: Permission denied) 2024-03-18T15:20:22.959 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 86176 root cwd unknown /proc/86176/cwd (readlink: Permission denied) 2024-03-18T15:20:22.959 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 86176 root rtd unknown /proc/86176/root (readlink: Permission denied) 2024-03-18T15:20:22.959 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 86176 root txt unknown /proc/86176/exe (readlink: Permission denied) 2024-03-18T15:20:22.959 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 86176 root NOFD /proc/86176/fd (opendir: Permission denied) 2024-03-18T15:20:22.959 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 86177 root cwd unknown /proc/86177/cwd (readlink: Permission denied) 2024-03-18T15:20:22.959 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 86177 root rtd unknown /proc/86177/root (readlink: Permission denied) 2024-03-18T15:20:22.959 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 86177 root txt unknown /proc/86177/exe (readlink: Permission denied) 2024-03-18T15:20:22.959 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/R 86177 root NOFD /proc/86177/fd (opendir: Permission denied) 2024-03-18T15:20:22.959 INFO:tasks.workunit.client.0.smithi019.stdout:sshd 86724 root cwd unknown /proc/86724/cwd (readlink: Permission denied) 2024-03-18T15:20:22.959 INFO:tasks.workunit.client.0.smithi019.stdout:sshd 86724 root rtd unknown /proc/86724/root (readlink: Permission denied) 2024-03-18T15:20:22.959 INFO:tasks.workunit.client.0.smithi019.stdout:sshd 86724 root txt unknown /proc/86724/exe (readlink: Permission denied) 2024-03-18T15:20:22.959 INFO:tasks.workunit.client.0.smithi019.stdout:sshd 86724 root NOFD /proc/86724/fd (opendir: Permission denied) 2024-03-18T15:20:22.959 INFO:tasks.workunit.client.0.smithi019.stdout:sshd 86798 root cwd unknown /proc/86798/cwd (readlink: Permission denied) 2024-03-18T15:20:22.960 INFO:tasks.workunit.client.0.smithi019.stdout:sshd 86798 root rtd unknown /proc/86798/root (readlink: Permission denied) 2024-03-18T15:20:22.960 INFO:tasks.workunit.client.0.smithi019.stdout:sshd 86798 root txt unknown /proc/86798/exe (readlink: Permission denied) 2024-03-18T15:20:22.960 INFO:tasks.workunit.client.0.smithi019.stdout:sshd 86798 root NOFD /proc/86798/fd (opendir: Permission denied) 2024-03-18T15:20:22.960 INFO:tasks.workunit.client.0.smithi019.stdout:timeout 88537 ubuntu cwd DIR 0,118 4 1099511627782 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-18T15:20:22.960 INFO:tasks.workunit.client.0.smithi019.stdout:timeout 88537 ubuntu rtd DIR 8,1 4096 2 / 2024-03-18T15:20:22.960 INFO:tasks.workunit.client.0.smithi019.stdout:timeout 88537 ubuntu txt REG 8,1 40944 10098 /usr/bin/timeout 2024-03-18T15:20:22.960 INFO:tasks.workunit.client.0.smithi019.stdout:timeout 88537 ubuntu mem REG 8,1 2586930 17848 /usr/lib/locale/en_US.utf8/LC_COLLATE 2024-03-18T15:20:22.960 INFO:tasks.workunit.client.0.smithi019.stdout:timeout 88537 ubuntu mem REG 8,1 346132 30861 /usr/lib/locale/C.utf8/LC_CTYPE 2024-03-18T15:20:22.960 INFO:tasks.workunit.client.0.smithi019.stdout:timeout 88537 ubuntu mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-03-18T15:20:22.960 INFO:tasks.workunit.client.0.smithi019.stdout:timeout 88537 ubuntu mem REG 8,1 54 17851 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2024-03-18T15:20:22.960 INFO:tasks.workunit.client.0.smithi019.stdout:timeout 88537 ubuntu mem REG 8,1 3284 30787 /usr/lib/locale/en_US.utf8/LC_TIME 2024-03-18T15:20:22.960 INFO:tasks.workunit.client.0.smithi019.stdout:timeout 88537 ubuntu mem REG 8,1 286 30784 /usr/lib/locale/en_US.utf8/LC_MONETARY 2024-03-18T15:20:22.960 INFO:tasks.workunit.client.0.smithi019.stdout:timeout 88537 ubuntu mem REG 8,1 57 28351 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2024-03-18T15:20:22.960 INFO:tasks.workunit.client.0.smithi019.stdout:timeout 88537 ubuntu mem REG 8,1 34 30837 /usr/lib/locale/en_US.utf8/LC_PAPER 2024-03-18T15:20:22.961 INFO:tasks.workunit.client.0.smithi019.stdout:timeout 88537 ubuntu mem REG 8,1 77 17850 /usr/lib/locale/en_US.utf8/LC_NAME 2024-03-18T15:20:22.961 INFO:tasks.workunit.client.0.smithi019.stdout:timeout 88537 ubuntu mem REG 8,1 167 30779 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2024-03-18T15:20:22.961 INFO:tasks.workunit.client.0.smithi019.stdout:timeout 88537 ubuntu mem REG 8,1 59 30786 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2024-03-18T15:20:22.961 INFO:tasks.workunit.client.0.smithi019.stdout:timeout 88537 ubuntu mem REG 8,1 23 30783 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2024-03-18T15:20:22.961 INFO:tasks.workunit.client.0.smithi019.stdout:timeout 88537 ubuntu mem REG 8,1 26988 29960 /usr/lib64/gconv/gconv-modules.cache 2024-03-18T15:20:22.961 INFO:tasks.workunit.client.0.smithi019.stdout:timeout 88537 ubuntu mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-03-18T15:20:22.961 INFO:tasks.workunit.client.0.smithi019.stdout:timeout 88537 ubuntu mem REG 8,1 369 30782 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2024-03-18T15:20:22.961 INFO:tasks.workunit.client.0.smithi019.stdout:timeout 88537 ubuntu 0r FIFO 0,13 0t0 225350 pipe 2024-03-18T15:20:22.961 INFO:tasks.workunit.client.0.smithi019.stdout:timeout 88537 ubuntu 1w FIFO 0,13 0t0 225351 pipe 2024-03-18T15:20:22.961 INFO:tasks.workunit.client.0.smithi019.stdout:timeout 88537 ubuntu 2w FIFO 0,13 0t0 225352 pipe 2024-03-18T15:20:22.961 INFO:tasks.workunit.client.0.smithi019.stdout:fsync-tes 88555 ubuntu cwd DIR 0,118 4 1099511627782 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-18T15:20:22.961 INFO:tasks.workunit.client.0.smithi019.stdout:fsync-tes 88555 ubuntu rtd DIR 8,1 4096 2 / 2024-03-18T15:20:22.961 INFO:tasks.workunit.client.0.smithi019.stdout:fsync-tes 88555 ubuntu txt REG 8,1 1388920 9667 /usr/bin/bash 2024-03-18T15:20:22.961 INFO:tasks.workunit.client.0.smithi019.stdout:fsync-tes 88555 ubuntu mem REG 8,1 2586930 17848 /usr/lib/locale/en_US.utf8/LC_COLLATE 2024-03-18T15:20:22.962 INFO:tasks.workunit.client.0.smithi019.stdout:fsync-tes 88555 ubuntu mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-03-18T15:20:22.962 INFO:tasks.workunit.client.0.smithi019.stdout:fsync-tes 88555 ubuntu mem REG 8,1 346132 30861 /usr/lib/locale/C.utf8/LC_CTYPE 2024-03-18T15:20:22.962 INFO:tasks.workunit.client.0.smithi019.stdout:fsync-tes 88555 ubuntu mem REG 8,1 54 17851 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2024-03-18T15:20:22.962 INFO:tasks.workunit.client.0.smithi019.stdout:fsync-tes 88555 ubuntu mem REG 8,1 3284 30787 /usr/lib/locale/en_US.utf8/LC_TIME 2024-03-18T15:20:22.962 INFO:tasks.workunit.client.0.smithi019.stdout:fsync-tes 88555 ubuntu mem REG 8,1 286 30784 /usr/lib/locale/en_US.utf8/LC_MONETARY 2024-03-18T15:20:22.962 INFO:tasks.workunit.client.0.smithi019.stdout:fsync-tes 88555 ubuntu mem REG 8,1 57 28351 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2024-03-18T15:20:22.962 INFO:tasks.workunit.client.0.smithi019.stdout:fsync-tes 88555 ubuntu mem REG 8,1 34 30837 /usr/lib/locale/en_US.utf8/LC_PAPER 2024-03-18T15:20:22.962 INFO:tasks.workunit.client.0.smithi019.stdout:fsync-tes 88555 ubuntu mem REG 8,1 77 17850 /usr/lib/locale/en_US.utf8/LC_NAME 2024-03-18T15:20:22.962 INFO:tasks.workunit.client.0.smithi019.stdout:fsync-tes 88555 ubuntu mem REG 8,1 167 30779 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2024-03-18T15:20:22.962 INFO:tasks.workunit.client.0.smithi019.stdout:fsync-tes 88555 ubuntu mem REG 8,1 59 30786 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2024-03-18T15:20:22.962 INFO:tasks.workunit.client.0.smithi019.stdout:fsync-tes 88555 ubuntu mem REG 8,1 195080 9664 /usr/lib64/libtinfo.so.6.2 2024-03-18T15:20:22.962 INFO:tasks.workunit.client.0.smithi019.stdout:fsync-tes 88555 ubuntu mem REG 8,1 23 30783 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2024-03-18T15:20:22.962 INFO:tasks.workunit.client.0.smithi019.stdout:fsync-tes 88555 ubuntu mem REG 8,1 26988 29960 /usr/lib64/gconv/gconv-modules.cache 2024-03-18T15:20:22.963 INFO:tasks.workunit.client.0.smithi019.stdout:fsync-tes 88555 ubuntu mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-03-18T15:20:22.963 INFO:tasks.workunit.client.0.smithi019.stdout:fsync-tes 88555 ubuntu mem REG 8,1 369 30782 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2024-03-18T15:20:22.963 INFO:tasks.workunit.client.0.smithi019.stdout:fsync-tes 88555 ubuntu 0r FIFO 0,13 0t0 225350 pipe 2024-03-18T15:20:22.963 INFO:tasks.workunit.client.0.smithi019.stdout:fsync-tes 88555 ubuntu 1w FIFO 0,13 0t0 225351 pipe 2024-03-18T15:20:22.963 INFO:tasks.workunit.client.0.smithi019.stdout:fsync-tes 88555 ubuntu 2w FIFO 0,13 0t0 225352 pipe 2024-03-18T15:20:22.963 INFO:tasks.workunit.client.0.smithi019.stdout:fsync-tes 88555 ubuntu 255r REG 8,1 207 792864 /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsync-tester.sh 2024-03-18T15:20:22.963 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/2 88564 root cwd unknown /proc/88564/cwd (readlink: Permission denied) 2024-03-18T15:20:22.963 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/2 88564 root rtd unknown /proc/88564/root (readlink: Permission denied) 2024-03-18T15:20:22.963 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/2 88564 root txt unknown /proc/88564/exe (readlink: Permission denied) 2024-03-18T15:20:22.963 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/2 88564 root NOFD /proc/88564/fd (opendir: Permission denied) 2024-03-18T15:20:22.963 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/2 88565 root cwd unknown /proc/88565/cwd (readlink: Permission denied) 2024-03-18T15:20:22.963 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/2 88565 root rtd unknown /proc/88565/root (readlink: Permission denied) 2024-03-18T15:20:22.963 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/2 88565 root txt unknown /proc/88565/exe (readlink: Permission denied) 2024-03-18T15:20:22.963 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/2 88565 root NOFD /proc/88565/fd (opendir: Permission denied) 2024-03-18T15:20:22.964 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/3 88694 root cwd unknown /proc/88694/cwd (readlink: Permission denied) 2024-03-18T15:20:22.964 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/3 88694 root rtd unknown /proc/88694/root (readlink: Permission denied) 2024-03-18T15:20:22.964 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/3 88694 root txt unknown /proc/88694/exe (readlink: Permission denied) 2024-03-18T15:20:22.964 INFO:tasks.workunit.client.0.smithi019.stdout:kworker/3 88694 root NOFD /proc/88694/fd (opendir: Permission denied) 2024-03-18T15:20:22.964 INFO:tasks.workunit.client.0.smithi019.stdout:lsof 88698 ubuntu cwd DIR 0,118 4 1099511627782 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-18T15:20:22.964 INFO:tasks.workunit.client.0.smithi019.stdout:lsof 88698 ubuntu rtd DIR 8,1 4096 2 / 2024-03-18T15:20:22.964 INFO:tasks.workunit.client.0.smithi019.stdout:lsof 88698 ubuntu txt REG 8,1 179232 12343 /usr/bin/lsof 2024-03-18T15:20:22.964 INFO:tasks.workunit.client.0.smithi019.stdout:lsof 88698 ubuntu mem REG 8,1 4487176 2250 /usr/lib64/libcrypto.so.3.0.7 2024-03-18T15:20:22.964 INFO:tasks.workunit.client.0.smithi019.stdout:lsof 88698 ubuntu mem REG 8,1 346132 30861 /usr/lib/locale/C.utf8/LC_CTYPE 2024-03-18T15:20:22.964 INFO:tasks.workunit.client.0.smithi019.stdout:lsof 88698 ubuntu mem REG 8,1 906256 11403 /usr/lib64/libkrb5.so.3.3 2024-03-18T15:20:22.964 INFO:tasks.workunit.client.0.smithi019.stdout:lsof 88698 ubuntu mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-03-18T15:20:22.964 INFO:tasks.workunit.client.0.smithi019.stdout:lsof 88698 ubuntu mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-03-18T15:20:22.964 INFO:tasks.workunit.client.0.smithi019.stdout:lsof 88698 ubuntu mem REG 8,1 70512 9633 /usr/lib64/libresolv.so.2 2024-03-18T15:20:22.964 INFO:tasks.workunit.client.0.smithi019.stdout:lsof 88698 ubuntu mem REG 8,1 23992 9760 /usr/lib64/libkeyutils.so.1.10 2024-03-18T15:20:22.965 INFO:tasks.workunit.client.0.smithi019.stdout:lsof 88698 ubuntu mem REG 8,1 66192 11405 /usr/lib64/libkrb5support.so.0.1 2024-03-18T15:20:22.965 INFO:tasks.workunit.client.0.smithi019.stdout:lsof 88698 ubuntu mem REG 8,1 24400 9711 /usr/lib64/libcom_err.so.2.1 2024-03-18T15:20:22.965 INFO:tasks.workunit.client.0.smithi019.stdout:lsof 88698 ubuntu mem REG 8,1 98992 11397 /usr/lib64/libk5crypto.so.3.1 2024-03-18T15:20:22.965 INFO:tasks.workunit.client.0.smithi019.stdout:lsof 88698 ubuntu mem REG 8,1 358448 11393 /usr/lib64/libgssapi_krb5.so.2.2 2024-03-18T15:20:22.965 INFO:tasks.workunit.client.0.smithi019.stdout:lsof 88698 ubuntu mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-03-18T15:20:22.965 INFO:tasks.workunit.client.0.smithi019.stdout:lsof 88698 ubuntu mem REG 8,1 196112 11431 /usr/lib64/libtirpc.so.3.0.0 2024-03-18T15:20:22.965 INFO:tasks.workunit.client.0.smithi019.stdout:lsof 88698 ubuntu mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-03-18T15:20:22.965 INFO:tasks.workunit.client.0.smithi019.stdout:lsof 88698 ubuntu mem REG 8,1 26988 29960 /usr/lib64/gconv/gconv-modules.cache 2024-03-18T15:20:22.965 INFO:tasks.workunit.client.0.smithi019.stdout:lsof 88698 ubuntu mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-03-18T15:20:22.965 INFO:tasks.workunit.client.0.smithi019.stdout:lsof 88698 ubuntu 0r FIFO 0,13 0t0 225350 pipe 2024-03-18T15:20:22.965 INFO:tasks.workunit.client.0.smithi019.stdout:lsof 88698 ubuntu 1w FIFO 0,13 0t0 225351 pipe 2024-03-18T15:20:22.965 INFO:tasks.workunit.client.0.smithi019.stdout:lsof 88698 ubuntu 2w FIFO 0,13 0t0 225352 pipe 2024-03-18T15:20:22.965 INFO:tasks.workunit.client.0.smithi019.stdout:lsof 88698 ubuntu 3r DIR 0,20 0 1 /proc 2024-03-18T15:20:22.965 INFO:tasks.workunit.client.0.smithi019.stdout:lsof 88698 ubuntu 4r DIR 0,20 7 221487 /proc/88698/fd 2024-03-18T15:20:22.966 INFO:tasks.workunit.client.0.smithi019.stdout:lsof 88698 ubuntu 5w FIFO 0,13 0t0 221492 pipe 2024-03-18T15:20:22.966 INFO:tasks.workunit.client.0.smithi019.stdout:lsof 88698 ubuntu 6r FIFO 0,13 0t0 221493 pipe 2024-03-18T15:20:22.966 INFO:tasks.workunit.client.0.smithi019.stdout:lsof 88699 ubuntu cwd DIR 0,118 4 1099511627782 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-18T15:20:22.966 INFO:tasks.workunit.client.0.smithi019.stdout:lsof 88699 ubuntu rtd DIR 8,1 4096 2 / 2024-03-18T15:20:22.966 INFO:tasks.workunit.client.0.smithi019.stdout:lsof 88699 ubuntu txt REG 8,1 179232 12343 /usr/bin/lsof 2024-03-18T15:20:22.966 INFO:tasks.workunit.client.0.smithi019.stdout:lsof 88699 ubuntu mem REG 8,1 4487176 2250 /usr/lib64/libcrypto.so.3.0.7 2024-03-18T15:20:22.966 INFO:tasks.workunit.client.0.smithi019.stdout:lsof 88699 ubuntu mem REG 8,1 346132 30861 /usr/lib/locale/C.utf8/LC_CTYPE 2024-03-18T15:20:22.966 INFO:tasks.workunit.client.0.smithi019.stdout:lsof 88699 ubuntu mem REG 8,1 906256 11403 /usr/lib64/libkrb5.so.3.3 2024-03-18T15:20:22.966 INFO:tasks.workunit.client.0.smithi019.stdout:lsof 88699 ubuntu mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-03-18T15:20:22.966 INFO:tasks.workunit.client.0.smithi019.stdout:lsof 88699 ubuntu mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-03-18T15:20:22.966 INFO:tasks.workunit.client.0.smithi019.stdout:lsof 88699 ubuntu mem REG 8,1 70512 9633 /usr/lib64/libresolv.so.2 2024-03-18T15:20:22.966 INFO:tasks.workunit.client.0.smithi019.stdout:lsof 88699 ubuntu mem REG 8,1 23992 9760 /usr/lib64/libkeyutils.so.1.10 2024-03-18T15:20:22.966 INFO:tasks.workunit.client.0.smithi019.stdout:lsof 88699 ubuntu mem REG 8,1 66192 11405 /usr/lib64/libkrb5support.so.0.1 2024-03-18T15:20:22.966 INFO:tasks.workunit.client.0.smithi019.stdout:lsof 88699 ubuntu mem REG 8,1 24400 9711 /usr/lib64/libcom_err.so.2.1 2024-03-18T15:20:22.967 INFO:tasks.workunit.client.0.smithi019.stdout:lsof 88699 ubuntu mem REG 8,1 98992 11397 /usr/lib64/libk5crypto.so.3.1 2024-03-18T15:20:22.967 INFO:tasks.workunit.client.0.smithi019.stdout:lsof 88699 ubuntu mem REG 8,1 358448 11393 /usr/lib64/libgssapi_krb5.so.2.2 2024-03-18T15:20:22.967 INFO:tasks.workunit.client.0.smithi019.stdout:lsof 88699 ubuntu mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-03-18T15:20:22.967 INFO:tasks.workunit.client.0.smithi019.stdout:lsof 88699 ubuntu mem REG 8,1 196112 11431 /usr/lib64/libtirpc.so.3.0.0 2024-03-18T15:20:22.967 INFO:tasks.workunit.client.0.smithi019.stdout:lsof 88699 ubuntu mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-03-18T15:20:22.967 INFO:tasks.workunit.client.0.smithi019.stdout:lsof 88699 ubuntu mem REG 8,1 26988 29960 /usr/lib64/gconv/gconv-modules.cache 2024-03-18T15:20:22.967 INFO:tasks.workunit.client.0.smithi019.stdout:lsof 88699 ubuntu mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-03-18T15:20:22.967 INFO:tasks.workunit.client.0.smithi019.stdout:lsof 88699 ubuntu 4r FIFO 0,13 0t0 221492 pipe 2024-03-18T15:20:22.967 INFO:tasks.workunit.client.0.smithi019.stdout:lsof 88699 ubuntu 7w FIFO 0,13 0t0 221493 pipe 2024-03-18T15:20:22.969 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-18T15:20:22.969 DEBUG:teuthology.orchestra.run.smithi019:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-18T15:20:23.110 INFO:tasks.workunit:Stopping ['suites/fsync-tester.sh'] on client.0... 2024-03-18T15:20:23.110 DEBUG:teuthology.orchestra.run.smithi019:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-03-18T15:20:23.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:20:22 smithi033 ceph-mon[36648]: pgmap v59: 129 pgs: 129 active+clean; 260 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 3.2 MiB/s wr, 292 op/s 2024-03-18T15:20:23.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:20:22 smithi104 ceph-mon[36070]: pgmap v59: 129 pgs: 129 active+clean; 260 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 3.2 MiB/s wr, 292 op/s 2024-03-18T15:20:23.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:20:22 smithi019 ceph-mon[31522]: pgmap v59: 129 pgs: 129 active+clean; 260 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 3.2 MiB/s wr, 292 op/s 2024-03-18T15:20:24.250 DEBUG:teuthology.parallel:result is None 2024-03-18T15:20:24.250 DEBUG:teuthology.orchestra.run.smithi019:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-03-18T15:20:24.404 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-03-18T15:20:24.404 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2024-03-18T15:20:24.415 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2024-03-18T15:20:24.415 INFO:teuthology.task.exec:Executing custom commands... 2024-03-18T15:20:24.415 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi019.front.sepia.ceph.com 2024-03-18T15:20:24.415 DEBUG:teuthology.orchestra.run.smithi019:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2024-03-18T15:20:24.588 DEBUG:teuthology.orchestra.run.smithi019:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2024-03-18T15:20:25.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:20:24 smithi033 ceph-mon[36648]: pgmap v60: 129 pgs: 129 active+clean; 260 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 2.4 MiB/s wr, 199 op/s 2024-03-18T15:20:25.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:20:24 smithi104 ceph-mon[36070]: pgmap v60: 129 pgs: 129 active+clean; 260 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 2.4 MiB/s wr, 199 op/s 2024-03-18T15:20:25.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:20:24 smithi019 ceph-mon[31522]: pgmap v60: 129 pgs: 129 active+clean; 260 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 2.4 MiB/s wr, 199 op/s 2024-03-18T15:20:27.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:20:27 smithi033 ceph-mon[36648]: pgmap v61: 129 pgs: 129 active+clean; 259 MiB data, 453 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 MiB/s wr, 587 op/s 2024-03-18T15:20:27.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:20:27 smithi104 ceph-mon[36070]: pgmap v61: 129 pgs: 129 active+clean; 259 MiB data, 453 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 MiB/s wr, 587 op/s 2024-03-18T15:20:27.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:20:27 smithi019 ceph-mon[31522]: pgmap v61: 129 pgs: 129 active+clean; 259 MiB data, 453 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 MiB/s wr, 587 op/s 2024-03-18T15:20:29.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:20:29 smithi104 ceph-mon[36070]: pgmap v62: 129 pgs: 129 active+clean; 259 MiB data, 453 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 20 MiB/s wr, 481 op/s 2024-03-18T15:20:29.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:20:29 smithi104 ceph-mon[36070]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T15:20:29.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:20:29 smithi019 ceph-mon[31522]: pgmap v62: 129 pgs: 129 active+clean; 259 MiB data, 453 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 20 MiB/s wr, 481 op/s 2024-03-18T15:20:29.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:20:29 smithi019 ceph-mon[31522]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T15:20:29.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:20:29 smithi033 ceph-mon[36648]: pgmap v62: 129 pgs: 129 active+clean; 259 MiB data, 453 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 20 MiB/s wr, 481 op/s 2024-03-18T15:20:29.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:20:29 smithi033 ceph-mon[36648]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T15:20:31.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:20:31 smithi104 ceph-mon[36070]: pgmap v63: 129 pgs: 129 active+clean; 259 MiB data, 453 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 20 MiB/s wr, 481 op/s 2024-03-18T15:20:31.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:20:31 smithi019 ceph-mon[31522]: pgmap v63: 129 pgs: 129 active+clean; 259 MiB data, 453 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 20 MiB/s wr, 481 op/s 2024-03-18T15:20:31.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:20:31 smithi033 ceph-mon[36648]: pgmap v63: 129 pgs: 129 active+clean; 259 MiB data, 453 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 20 MiB/s wr, 481 op/s 2024-03-18T15:20:33.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:20:33 smithi104 ceph-mon[36070]: pgmap v64: 129 pgs: 129 active+clean; 275 MiB data, 458 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 21 MiB/s wr, 817 op/s 2024-03-18T15:20:33.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:20:33 smithi019 ceph-mon[31522]: pgmap v64: 129 pgs: 129 active+clean; 275 MiB data, 458 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 21 MiB/s wr, 817 op/s 2024-03-18T15:20:33.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:20:33 smithi033 ceph-mon[36648]: pgmap v64: 129 pgs: 129 active+clean; 275 MiB data, 458 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 21 MiB/s wr, 817 op/s 2024-03-18T15:20:35.308 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:20:35 smithi033 ceph-mon[36648]: pgmap v65: 129 pgs: 129 active+clean; 275 MiB data, 458 MiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 21 MiB/s wr, 723 op/s 2024-03-18T15:20:35.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:20:35 smithi104 ceph-mon[36070]: pgmap v65: 129 pgs: 129 active+clean; 275 MiB data, 458 MiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 21 MiB/s wr, 723 op/s 2024-03-18T15:20:35.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:20:35 smithi019 ceph-mon[31522]: pgmap v65: 129 pgs: 129 active+clean; 275 MiB data, 458 MiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 21 MiB/s wr, 723 op/s 2024-03-18T15:20:37.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:20:37 smithi104 ceph-mon[36070]: pgmap v66: 129 pgs: 129 active+clean; 263 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 21 MiB/s wr, 751 op/s 2024-03-18T15:20:37.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:20:37 smithi019 ceph-mon[31522]: pgmap v66: 129 pgs: 129 active+clean; 263 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 21 MiB/s wr, 751 op/s 2024-03-18T15:20:37.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:20:37 smithi033 ceph-mon[36648]: pgmap v66: 129 pgs: 129 active+clean; 263 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 21 MiB/s wr, 751 op/s 2024-03-18T15:20:39.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:20:39 smithi104 ceph-mon[36070]: pgmap v67: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.5 MiB/s wr, 378 op/s 2024-03-18T15:20:39.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:20:39 smithi019 ceph-mon[31522]: pgmap v67: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.5 MiB/s wr, 378 op/s 2024-03-18T15:20:39.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:20:39 smithi033 ceph-mon[36648]: pgmap v67: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.5 MiB/s wr, 378 op/s 2024-03-18T15:20:41.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:20:41 smithi104 ceph-mon[36070]: pgmap v68: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.5 MiB/s wr, 378 op/s 2024-03-18T15:20:41.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:20:41 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:20:41.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:20:41 smithi019 ceph-mon[31522]: pgmap v68: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.5 MiB/s wr, 378 op/s 2024-03-18T15:20:41.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:20:41 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:20:41.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:20:41 smithi033 ceph-mon[36648]: pgmap v68: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.5 MiB/s wr, 378 op/s 2024-03-18T15:20:41.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:20:41 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:20:42.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:20:42 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:20:42.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:20:42 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:20:42.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:20:42 smithi104 ceph-mon[36070]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:20:42.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:20:42 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:20:42.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:20:42 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:20:42.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:20:42 smithi019 ceph-mon[31522]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:20:42.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:20:42 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:20:42.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:20:42 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:20:42.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:20:42 smithi033 ceph-mon[36648]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:20:43.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:20:43 smithi104 ceph-mon[36070]: pgmap v69: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 1.5 MiB/s wr, 383 op/s 2024-03-18T15:20:43.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:20:43 smithi019 ceph-mon[31522]: pgmap v69: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 1.5 MiB/s wr, 383 op/s 2024-03-18T15:20:43.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:20:43 smithi033 ceph-mon[36648]: pgmap v69: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 1.5 MiB/s wr, 383 op/s 2024-03-18T15:20:45.300 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:20:45 smithi019 ceph-mon[31522]: pgmap v70: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 150 KiB/s wr, 47 op/s 2024-03-18T15:20:45.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:20:45 smithi104 ceph-mon[36070]: pgmap v70: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 150 KiB/s wr, 47 op/s 2024-03-18T15:20:45.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:20:45 smithi033 ceph-mon[36648]: pgmap v70: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 150 KiB/s wr, 47 op/s 2024-03-18T15:20:47.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:20:47 smithi019 ceph-mon[31522]: pgmap v71: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 150 KiB/s wr, 51 op/s 2024-03-18T15:20:47.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:20:47 smithi104 ceph-mon[36070]: pgmap v71: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 150 KiB/s wr, 51 op/s 2024-03-18T15:20:47.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:20:47 smithi033 ceph-mon[36648]: pgmap v71: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 150 KiB/s wr, 51 op/s 2024-03-18T15:20:49.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:20:49 smithi104 ceph-mon[36070]: pgmap v72: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 62 KiB/s wr, 23 op/s 2024-03-18T15:20:49.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:20:49 smithi019 ceph-mon[31522]: pgmap v72: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 62 KiB/s wr, 23 op/s 2024-03-18T15:20:49.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:20:49 smithi033 ceph-mon[36648]: pgmap v72: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 62 KiB/s wr, 23 op/s 2024-03-18T15:20:51.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:20:51 smithi104 ceph-mon[36070]: pgmap v73: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-18T15:20:51.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:20:51 smithi019 ceph-mon[31522]: pgmap v73: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-18T15:20:51.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:20:51 smithi033 ceph-mon[36648]: pgmap v73: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-18T15:20:53.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:20:53 smithi104 ceph-mon[36070]: pgmap v74: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-18T15:20:53.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:20:53 smithi019 ceph-mon[31522]: pgmap v74: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-18T15:20:53.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:20:53 smithi033 ceph-mon[36648]: pgmap v74: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-18T15:20:55.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:20:55 smithi104 ceph-mon[36070]: pgmap v75: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:20:55.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:20:55 smithi019 ceph-mon[31522]: pgmap v75: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:20:55.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:20:55 smithi033 ceph-mon[36648]: pgmap v75: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:20:57.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:20:57 smithi104 ceph-mon[36070]: pgmap v76: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-03-18T15:20:57.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:20:57 smithi019 ceph-mon[31522]: pgmap v76: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-03-18T15:20:57.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:20:57 smithi033 ceph-mon[36648]: pgmap v76: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-03-18T15:20:59.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:20:59 smithi104 ceph-mon[36070]: pgmap v77: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-18T15:20:59.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:20:59 smithi019 ceph-mon[31522]: pgmap v77: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-18T15:20:59.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:20:59 smithi033 ceph-mon[36648]: pgmap v77: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-18T15:21:01.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:21:01 smithi104 ceph-mon[36070]: pgmap v78: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:21:01.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:21:01 smithi019 ceph-mon[31522]: pgmap v78: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:21:01.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:21:01 smithi033 ceph-mon[36648]: pgmap v78: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:21:03.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:21:03 smithi104 ceph-mon[36070]: pgmap v79: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-18T15:21:03.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:21:03 smithi019 ceph-mon[31522]: pgmap v79: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-18T15:21:03.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:21:03 smithi033 ceph-mon[36648]: pgmap v79: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-18T15:21:05.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:21:05 smithi104 ceph-mon[36070]: pgmap v80: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-18T15:21:05.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:21:05 smithi019 ceph-mon[31522]: pgmap v80: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-18T15:21:05.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:21:05 smithi033 ceph-mon[36648]: pgmap v80: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-18T15:21:07.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:21:07 smithi104 ceph-mon[36070]: pgmap v81: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-18T15:21:07.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:21:07 smithi019 ceph-mon[31522]: pgmap v81: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-18T15:21:07.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:21:07 smithi033 ceph-mon[36648]: pgmap v81: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-18T15:21:09.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:21:09 smithi104 ceph-mon[36070]: pgmap v82: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-18T15:21:09.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:21:09 smithi019 ceph-mon[31522]: pgmap v82: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-18T15:21:09.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:21:09 smithi033 ceph-mon[36648]: pgmap v82: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-18T15:21:11.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:21:11 smithi104 ceph-mon[36070]: pgmap v83: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-18T15:21:11.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:21:11 smithi019 ceph-mon[31522]: pgmap v83: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-18T15:21:11.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:21:11 smithi033 ceph-mon[36648]: pgmap v83: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-18T15:21:13.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:21:13 smithi104 ceph-mon[36070]: pgmap v84: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-18T15:21:13.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:21:13 smithi019 ceph-mon[31522]: pgmap v84: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-18T15:21:13.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:21:13 smithi033 ceph-mon[36648]: pgmap v84: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-18T15:21:15.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:21:15 smithi104 ceph-mon[36070]: pgmap v85: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-18T15:21:15.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:21:15 smithi019 ceph-mon[31522]: pgmap v85: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-18T15:21:15.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:21:15 smithi033 ceph-mon[36648]: pgmap v85: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-18T15:21:17.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:21:17 smithi104 ceph-mon[36070]: pgmap v86: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 170 B/s wr, 12 op/s 2024-03-18T15:21:17.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:21:17 smithi019 ceph-mon[31522]: pgmap v86: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 170 B/s wr, 12 op/s 2024-03-18T15:21:17.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:21:17 smithi033 ceph-mon[36648]: pgmap v86: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 170 B/s wr, 12 op/s 2024-03-18T15:21:19.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:21:19 smithi104 ceph-mon[36070]: pgmap v87: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-18T15:21:19.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:21:19 smithi019 ceph-mon[31522]: pgmap v87: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-18T15:21:19.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:21:19 smithi033 ceph-mon[36648]: pgmap v87: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-18T15:21:21.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:21:21 smithi104 ceph-mon[36070]: pgmap v88: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-18T15:21:21.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:21:21 smithi019 ceph-mon[31522]: pgmap v88: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-18T15:21:21.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:21:21 smithi033 ceph-mon[36648]: pgmap v88: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-18T15:21:23.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:21:23 smithi104 ceph-mon[36070]: pgmap v89: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-18T15:21:23.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:21:23 smithi019 ceph-mon[31522]: pgmap v89: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-18T15:21:23.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:21:23 smithi033 ceph-mon[36648]: pgmap v89: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-18T15:21:25.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:21:25 smithi104 ceph-mon[36070]: pgmap v90: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-03-18T15:21:25.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:21:25 smithi019 ceph-mon[31522]: pgmap v90: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-03-18T15:21:25.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:21:25 smithi033 ceph-mon[36648]: pgmap v90: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-03-18T15:21:27.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:21:27 smithi104 ceph-mon[36070]: pgmap v91: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:21:27.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:21:27 smithi019 ceph-mon[31522]: pgmap v91: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:21:27.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:21:27 smithi033 ceph-mon[36648]: pgmap v91: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:21:29.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:21:29 smithi104 ceph-mon[36070]: pgmap v92: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:21:29.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:21:29 smithi019 ceph-mon[31522]: pgmap v92: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:21:29.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:21:29 smithi033 ceph-mon[36648]: pgmap v92: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:21:31.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:21:31 smithi104 ceph-mon[36070]: pgmap v93: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:21:31.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:21:31 smithi104 ceph-mon[36070]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T15:21:31.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:21:31 smithi019 ceph-mon[31522]: pgmap v93: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:21:31.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:21:31 smithi019 ceph-mon[31522]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T15:21:31.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:21:31 smithi033 ceph-mon[36648]: pgmap v93: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:21:31.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:21:31 smithi033 ceph-mon[36648]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T15:21:33.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:21:33 smithi104 ceph-mon[36070]: pgmap v94: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:21:33.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:21:33 smithi019 ceph-mon[31522]: pgmap v94: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:21:33.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:21:33 smithi033 ceph-mon[36648]: pgmap v94: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:21:35.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:21:35 smithi104 ceph-mon[36070]: pgmap v95: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:21:35.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:21:35 smithi019 ceph-mon[31522]: pgmap v95: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:21:35.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:21:35 smithi033 ceph-mon[36648]: pgmap v95: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:21:37.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:21:37 smithi104 ceph-mon[36070]: pgmap v96: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:21:37.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:21:37 smithi019 ceph-mon[31522]: pgmap v96: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:21:37.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:21:37 smithi033 ceph-mon[36648]: pgmap v96: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:21:39.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:21:39 smithi019 ceph-mon[31522]: pgmap v97: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:21:39.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:21:39 smithi033 ceph-mon[36648]: pgmap v97: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:21:39.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:21:39 smithi104 ceph-mon[36070]: pgmap v97: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:21:41.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:21:41 smithi019 ceph-mon[31522]: pgmap v98: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:21:41.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:21:41 smithi033 ceph-mon[36648]: pgmap v98: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:21:41.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:21:41 smithi104 ceph-mon[36070]: pgmap v98: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:21:42.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:21:42 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:21:42.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:21:42 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:21:42.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:21:42 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:21:43.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:21:43 smithi019 ceph-mon[31522]: pgmap v99: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:21:43.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:21:43 smithi033 ceph-mon[36648]: pgmap v99: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:21:43.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:21:43 smithi104 ceph-mon[36070]: pgmap v99: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:21:44.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:21:44 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:21:44.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:21:44 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:21:44.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:21:44 smithi019 ceph-mon[31522]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:21:44.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:21:44 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:21:44.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:21:44 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:21:44.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:21:44 smithi033 ceph-mon[36648]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:21:44.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:21:44 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:21:44.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:21:44 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:21:44.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:21:44 smithi104 ceph-mon[36070]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:21:45.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:21:45 smithi033 ceph-mon[36648]: pgmap v100: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:21:45.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:21:45 smithi104 ceph-mon[36070]: pgmap v100: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:21:45.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:21:45 smithi019 ceph-mon[31522]: pgmap v100: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:21:47.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:21:47 smithi033 ceph-mon[36648]: pgmap v101: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:21:47.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:21:47 smithi104 ceph-mon[36070]: pgmap v101: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:21:47.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:21:47 smithi019 ceph-mon[31522]: pgmap v101: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:21:49.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:21:49 smithi033 ceph-mon[36648]: pgmap v102: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:21:49.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:21:49 smithi104 ceph-mon[36070]: pgmap v102: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:21:49.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:21:49 smithi019 ceph-mon[31522]: pgmap v102: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:21:51.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:21:51 smithi033 ceph-mon[36648]: pgmap v103: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:21:51.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:21:51 smithi104 ceph-mon[36070]: pgmap v103: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:21:51.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:21:51 smithi019 ceph-mon[31522]: pgmap v103: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:21:53.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:21:53 smithi033 ceph-mon[36648]: pgmap v104: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:21:53.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:21:53 smithi104 ceph-mon[36070]: pgmap v104: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:21:53.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:21:53 smithi019 ceph-mon[31522]: pgmap v104: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:21:55.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:21:55 smithi033 ceph-mon[36648]: pgmap v105: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:21:55.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:21:55 smithi104 ceph-mon[36070]: pgmap v105: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:21:55.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:21:55 smithi019 ceph-mon[31522]: pgmap v105: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:21:57.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:21:57 smithi104 ceph-mon[36070]: pgmap v106: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:21:57.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:21:57 smithi019 ceph-mon[31522]: pgmap v106: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:21:58.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:21:57 smithi033 ceph-mon[36648]: pgmap v106: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:21:59.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:21:59 smithi104 ceph-mon[36070]: pgmap v107: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:21:59.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:21:59 smithi019 ceph-mon[31522]: pgmap v107: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:22:00.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:21:59 smithi033 ceph-mon[36648]: pgmap v107: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:22:01.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:22:01 smithi104 ceph-mon[36070]: pgmap v108: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:22:01.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:22:01 smithi019 ceph-mon[31522]: pgmap v108: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:22:02.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:22:01 smithi033 ceph-mon[36648]: pgmap v108: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:22:03.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:22:03 smithi104 ceph-mon[36070]: pgmap v109: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 0 B/s wr, 14 op/s 2024-03-18T15:22:03.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:22:03 smithi019 ceph-mon[31522]: pgmap v109: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 0 B/s wr, 14 op/s 2024-03-18T15:22:04.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:22:03 smithi033 ceph-mon[36648]: pgmap v109: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 0 B/s wr, 14 op/s 2024-03-18T15:22:05.808 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:22:05 smithi033 ceph-mon[36648]: pgmap v110: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-18T15:22:05.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:22:05 smithi104 ceph-mon[36070]: pgmap v110: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-18T15:22:05.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:22:05 smithi019 ceph-mon[31522]: pgmap v110: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-18T15:22:07.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:22:07 smithi104 ceph-mon[36070]: pgmap v111: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-18T15:22:07.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:22:07 smithi019 ceph-mon[31522]: pgmap v111: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-18T15:22:08.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:22:07 smithi033 ceph-mon[36648]: pgmap v111: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-18T15:22:09.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:22:09 smithi104 ceph-mon[36070]: pgmap v112: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-18T15:22:09.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:22:09 smithi019 ceph-mon[31522]: pgmap v112: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-18T15:22:10.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:22:09 smithi033 ceph-mon[36648]: pgmap v112: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-18T15:22:11.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:22:11 smithi104 ceph-mon[36070]: pgmap v113: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-18T15:22:11.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:22:11 smithi019 ceph-mon[31522]: pgmap v113: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-18T15:22:12.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:22:11 smithi033 ceph-mon[36648]: pgmap v113: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-18T15:22:13.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:22:13 smithi104 ceph-mon[36070]: pgmap v114: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-18T15:22:13.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:22:13 smithi019 ceph-mon[31522]: pgmap v114: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-18T15:22:14.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:22:13 smithi033 ceph-mon[36648]: pgmap v114: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-18T15:22:15.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:22:15 smithi104 ceph-mon[36070]: pgmap v115: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 21 op/s 2024-03-18T15:22:15.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:22:15 smithi019 ceph-mon[31522]: pgmap v115: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 21 op/s 2024-03-18T15:22:16.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:22:15 smithi033 ceph-mon[36648]: pgmap v115: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 21 op/s 2024-03-18T15:22:17.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:22:17 smithi104 ceph-mon[36070]: pgmap v116: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 25 op/s 2024-03-18T15:22:17.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:22:17 smithi019 ceph-mon[31522]: pgmap v116: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 25 op/s 2024-03-18T15:22:18.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:22:17 smithi033 ceph-mon[36648]: pgmap v116: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 25 op/s 2024-03-18T15:22:19.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:22:19 smithi104 ceph-mon[36070]: pgmap v117: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-18T15:22:19.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:22:19 smithi019 ceph-mon[31522]: pgmap v117: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-18T15:22:20.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:22:19 smithi033 ceph-mon[36648]: pgmap v117: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-18T15:22:21.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:22:21 smithi104 ceph-mon[36070]: pgmap v118: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-18T15:22:21.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:22:21 smithi019 ceph-mon[31522]: pgmap v118: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-18T15:22:22.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:22:21 smithi033 ceph-mon[36648]: pgmap v118: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-18T15:22:23.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:22:23 smithi104 ceph-mon[36070]: pgmap v119: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-18T15:22:23.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:22:23 smithi019 ceph-mon[31522]: pgmap v119: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-18T15:22:24.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:22:23 smithi033 ceph-mon[36648]: pgmap v119: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-18T15:22:25.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:22:25 smithi104 ceph-mon[36070]: pgmap v120: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:22:25.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:22:25 smithi019 ceph-mon[31522]: pgmap v120: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:22:26.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:22:25 smithi033 ceph-mon[36648]: pgmap v120: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:22:27.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:22:27 smithi104 ceph-mon[36070]: pgmap v121: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:22:27.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:22:27 smithi019 ceph-mon[31522]: pgmap v121: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:22:28.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:22:27 smithi033 ceph-mon[36648]: pgmap v121: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:22:29.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:22:29 smithi104 ceph-mon[36070]: pgmap v122: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:22:29.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:22:29 smithi019 ceph-mon[31522]: pgmap v122: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:22:30.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:22:29 smithi033 ceph-mon[36648]: pgmap v122: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:22:31.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:22:31 smithi104 ceph-mon[36070]: pgmap v123: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:22:31.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:22:31 smithi019 ceph-mon[31522]: pgmap v123: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:22:32.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:22:31 smithi033 ceph-mon[36648]: pgmap v123: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:22:33.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:22:33 smithi104 ceph-mon[36070]: pgmap v124: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:22:33.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:22:33 smithi104 ceph-mon[36070]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T15:22:33.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:22:33 smithi019 ceph-mon[31522]: pgmap v124: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:22:33.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:22:33 smithi019 ceph-mon[31522]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T15:22:34.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:22:33 smithi033 ceph-mon[36648]: pgmap v124: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:22:34.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:22:33 smithi033 ceph-mon[36648]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T15:22:35.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:22:35 smithi104 ceph-mon[36070]: pgmap v125: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:22:35.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:22:35 smithi019 ceph-mon[31522]: pgmap v125: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:22:36.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:22:35 smithi033 ceph-mon[36648]: pgmap v125: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:22:37.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:22:37 smithi104 ceph-mon[36070]: pgmap v126: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:22:37.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:22:37 smithi019 ceph-mon[31522]: pgmap v126: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:22:38.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:22:37 smithi033 ceph-mon[36648]: pgmap v126: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:22:39.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:22:39 smithi104 ceph-mon[36070]: pgmap v127: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:22:39.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:22:39 smithi019 ceph-mon[31522]: pgmap v127: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:22:40.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:22:39 smithi033 ceph-mon[36648]: pgmap v127: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:22:41.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:22:41 smithi104 ceph-mon[36070]: pgmap v128: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:22:41.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:22:41 smithi019 ceph-mon[31522]: pgmap v128: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:22:42.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:22:41 smithi033 ceph-mon[36648]: pgmap v128: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:22:43.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:22:43 smithi104 ceph-mon[36070]: pgmap v129: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:22:43.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:22:43 smithi019 ceph-mon[31522]: pgmap v129: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:22:44.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:22:43 smithi033 ceph-mon[36648]: pgmap v129: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:22:44.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:22:44 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:22:44.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:22:44 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:22:45.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:22:44 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:22:45.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:22:45 smithi104 ceph-mon[36070]: pgmap v130: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:22:45.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:22:45 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:22:45.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:22:45 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:22:45.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:22:45 smithi104 ceph-mon[36070]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:22:45.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:22:45 smithi019 ceph-mon[31522]: pgmap v130: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:22:45.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:22:45 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:22:45.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:22:45 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:22:45.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:22:45 smithi019 ceph-mon[31522]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:22:46.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:22:45 smithi033 ceph-mon[36648]: pgmap v130: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:22:46.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:22:45 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:22:46.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:22:45 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:22:46.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:22:45 smithi033 ceph-mon[36648]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:22:47.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:22:47 smithi104 ceph-mon[36070]: pgmap v131: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:22:47.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:22:47 smithi019 ceph-mon[31522]: pgmap v131: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:22:48.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:22:47 smithi033 ceph-mon[36648]: pgmap v131: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:22:49.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:22:49 smithi104 ceph-mon[36070]: pgmap v132: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:22:49.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:22:49 smithi019 ceph-mon[31522]: pgmap v132: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:22:50.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:22:49 smithi033 ceph-mon[36648]: pgmap v132: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:22:51.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:22:51 smithi104 ceph-mon[36070]: pgmap v133: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:22:51.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:22:51 smithi019 ceph-mon[31522]: pgmap v133: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:22:52.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:22:51 smithi033 ceph-mon[36648]: pgmap v133: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:22:53.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:22:53 smithi104 ceph-mon[36070]: pgmap v134: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:22:53.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:22:53 smithi019 ceph-mon[31522]: pgmap v134: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:22:54.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:22:53 smithi033 ceph-mon[36648]: pgmap v134: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:22:55.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:22:55 smithi104 ceph-mon[36070]: pgmap v135: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-03-18T15:22:55.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:22:55 smithi019 ceph-mon[31522]: pgmap v135: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-03-18T15:22:56.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:22:55 smithi033 ceph-mon[36648]: pgmap v135: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-03-18T15:22:57.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:22:57 smithi104 ceph-mon[36070]: pgmap v136: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:22:57.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:22:57 smithi019 ceph-mon[31522]: pgmap v136: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:22:58.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:22:57 smithi033 ceph-mon[36648]: pgmap v136: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:22:59.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:22:59 smithi104 ceph-mon[36070]: pgmap v137: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-03-18T15:22:59.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:22:59 smithi019 ceph-mon[31522]: pgmap v137: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-03-18T15:23:00.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:22:59 smithi033 ceph-mon[36648]: pgmap v137: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-03-18T15:23:01.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:23:01 smithi104 ceph-mon[36070]: pgmap v138: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-03-18T15:23:01.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:23:01 smithi019 ceph-mon[31522]: pgmap v138: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-03-18T15:23:02.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:23:01 smithi033 ceph-mon[36648]: pgmap v138: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-03-18T15:23:03.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:23:03 smithi104 ceph-mon[36070]: pgmap v139: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:23:03.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:23:03 smithi019 ceph-mon[31522]: pgmap v139: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:23:04.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:23:03 smithi033 ceph-mon[36648]: pgmap v139: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:23:05.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:23:05 smithi104 ceph-mon[36070]: pgmap v140: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:23:05.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:23:05 smithi019 ceph-mon[31522]: pgmap v140: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:23:06.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:23:05 smithi033 ceph-mon[36648]: pgmap v140: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:23:06.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:23:06 smithi019 ceph-mon[31522]: pgmap v141: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-18T15:23:07.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:23:06 smithi033 ceph-mon[36648]: pgmap v141: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-18T15:23:07.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:23:06 smithi104 ceph-mon[36070]: pgmap v141: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-18T15:23:09.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:23:08 smithi019 ceph-mon[31522]: pgmap v142: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-18T15:23:09.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:23:08 smithi033 ceph-mon[36648]: pgmap v142: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-18T15:23:09.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:23:08 smithi104 ceph-mon[36070]: pgmap v142: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-18T15:23:11.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:23:10 smithi019 ceph-mon[31522]: pgmap v143: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-18T15:23:11.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:23:10 smithi033 ceph-mon[36648]: pgmap v143: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-18T15:23:11.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:23:10 smithi104 ceph-mon[36070]: pgmap v143: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-18T15:23:13.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:23:12 smithi019 ceph-mon[31522]: pgmap v144: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-18T15:23:13.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:23:12 smithi033 ceph-mon[36648]: pgmap v144: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-18T15:23:13.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:23:12 smithi104 ceph-mon[36070]: pgmap v144: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-18T15:23:15.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:23:14 smithi019 ceph-mon[31522]: pgmap v145: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-18T15:23:15.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:23:14 smithi033 ceph-mon[36648]: pgmap v145: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-18T15:23:15.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:23:14 smithi104 ceph-mon[36070]: pgmap v145: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-18T15:23:17.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:23:16 smithi033 ceph-mon[36648]: pgmap v146: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-18T15:23:17.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:23:16 smithi104 ceph-mon[36070]: pgmap v146: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-18T15:23:17.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:23:16 smithi019 ceph-mon[31522]: pgmap v146: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-18T15:23:19.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:23:18 smithi033 ceph-mon[36648]: pgmap v147: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-18T15:23:19.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:23:18 smithi104 ceph-mon[36070]: pgmap v147: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-18T15:23:19.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:23:18 smithi019 ceph-mon[31522]: pgmap v147: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-18T15:23:21.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:23:20 smithi033 ceph-mon[36648]: pgmap v148: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-18T15:23:21.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:23:20 smithi033 ceph-mon[36648]: evicting unresponsive client smithi019:x (25459), after 300.124 seconds 2024-03-18T15:23:21.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:23:20 smithi033 ceph-mon[36648]: Evicting (and blocklisting) client session 25459 (172.21.15.19:0/1895666842) 2024-03-18T15:23:21.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:23:20 smithi033 ceph-mon[36648]: from='mds.? [v2:172.21.15.33:6836/160187378,v1:172.21.15.33:6837/160187378]' entity='mds.h' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.19:0/1895666842"}]: dispatch 2024-03-18T15:23:21.257 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:23:20 smithi033 ceph-mon[36648]: from='mds.? ' entity='mds.h' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.19:0/1895666842"}]: dispatch 2024-03-18T15:23:21.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:23:20 smithi104 ceph-mon[36070]: pgmap v148: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-18T15:23:21.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:23:20 smithi104 ceph-mon[36070]: evicting unresponsive client smithi019:x (25459), after 300.124 seconds 2024-03-18T15:23:21.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:23:20 smithi104 ceph-mon[36070]: Evicting (and blocklisting) client session 25459 (172.21.15.19:0/1895666842) 2024-03-18T15:23:21.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:23:20 smithi104 ceph-mon[36070]: from='mds.? [v2:172.21.15.33:6836/160187378,v1:172.21.15.33:6837/160187378]' entity='mds.h' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.19:0/1895666842"}]: dispatch 2024-03-18T15:23:21.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:23:20 smithi104 ceph-mon[36070]: from='mds.? ' entity='mds.h' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.19:0/1895666842"}]: dispatch 2024-03-18T15:23:21.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:23:20 smithi019 ceph-mon[31522]: pgmap v148: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-18T15:23:21.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:23:20 smithi019 ceph-mon[31522]: evicting unresponsive client smithi019:x (25459), after 300.124 seconds 2024-03-18T15:23:21.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:23:20 smithi019 ceph-mon[31522]: Evicting (and blocklisting) client session 25459 (172.21.15.19:0/1895666842) 2024-03-18T15:23:21.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:23:20 smithi019 ceph-mon[31522]: from='mds.? [v2:172.21.15.33:6836/160187378,v1:172.21.15.33:6837/160187378]' entity='mds.h' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.19:0/1895666842"}]: dispatch 2024-03-18T15:23:21.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:23:20 smithi019 ceph-mon[31522]: from='mds.? ' entity='mds.h' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.19:0/1895666842"}]: dispatch 2024-03-18T15:23:22.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:23:21 smithi033 ceph-mon[36648]: from='mds.? ' entity='mds.h' cmd='[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.19:0/1895666842"}]': finished 2024-03-18T15:23:22.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:23:21 smithi033 ceph-mon[36648]: osdmap e95: 12 total, 12 up, 12 in 2024-03-18T15:23:22.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:23:21 smithi104 ceph-mon[36070]: from='mds.? ' entity='mds.h' cmd='[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.19:0/1895666842"}]': finished 2024-03-18T15:23:22.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:23:21 smithi104 ceph-mon[36070]: osdmap e95: 12 total, 12 up, 12 in 2024-03-18T15:23:22.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:23:21 smithi019 ceph-mon[31522]: from='mds.? ' entity='mds.h' cmd='[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.19:0/1895666842"}]': finished 2024-03-18T15:23:22.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:23:21 smithi019 ceph-mon[31522]: osdmap e95: 12 total, 12 up, 12 in 2024-03-18T15:23:23.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:23:22 smithi033 ceph-mon[36648]: pgmap v150: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T15:23:23.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:23:22 smithi104 ceph-mon[36070]: pgmap v150: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T15:23:23.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:23:22 smithi019 ceph-mon[31522]: pgmap v150: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T15:23:25.256 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:23:25 smithi033 ceph-mon[36648]: pgmap v151: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T15:23:25.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:23:25 smithi104 ceph-mon[36070]: pgmap v151: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T15:23:25.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:23:25 smithi019 ceph-mon[31522]: pgmap v151: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T15:23:27.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:23:27 smithi104 ceph-mon[36070]: pgmap v152: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 716 B/s wr, 11 op/s 2024-03-18T15:23:27.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:23:27 smithi019 ceph-mon[31522]: pgmap v152: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 716 B/s wr, 11 op/s 2024-03-18T15:23:27.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:23:27 smithi033 ceph-mon[36648]: pgmap v152: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 716 B/s wr, 11 op/s 2024-03-18T15:23:29.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:23:29 smithi104 ceph-mon[36070]: pgmap v153: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 716 B/s wr, 11 op/s 2024-03-18T15:23:29.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:23:29 smithi019 ceph-mon[31522]: pgmap v153: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 716 B/s wr, 11 op/s 2024-03-18T15:23:29.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:23:29 smithi033 ceph-mon[36648]: pgmap v153: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 716 B/s wr, 11 op/s 2024-03-18T15:23:31.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:23:31 smithi104 ceph-mon[36070]: pgmap v154: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 716 B/s wr, 11 op/s 2024-03-18T15:23:31.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:23:31 smithi019 ceph-mon[31522]: pgmap v154: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 716 B/s wr, 11 op/s 2024-03-18T15:23:31.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:23:31 smithi033 ceph-mon[36648]: pgmap v154: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 716 B/s wr, 11 op/s 2024-03-18T15:23:33.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:23:33 smithi104 ceph-mon[36070]: pgmap v155: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 656 B/s wr, 10 op/s 2024-03-18T15:23:33.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:23:33 smithi019 ceph-mon[31522]: pgmap v155: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 656 B/s wr, 10 op/s 2024-03-18T15:23:33.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:23:33 smithi033 ceph-mon[36648]: pgmap v155: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 656 B/s wr, 10 op/s 2024-03-18T15:23:35.308 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:23:35 smithi033 ceph-mon[36648]: pgmap v156: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 597 B/s wr, 9 op/s 2024-03-18T15:23:35.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:23:35 smithi104 ceph-mon[36070]: pgmap v156: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 597 B/s wr, 9 op/s 2024-03-18T15:23:35.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:23:35 smithi019 ceph-mon[31522]: pgmap v156: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 597 B/s wr, 9 op/s 2024-03-18T15:23:37.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:23:37 smithi104 ceph-mon[36070]: pgmap v157: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 597 B/s wr, 13 op/s 2024-03-18T15:23:37.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:23:37 smithi019 ceph-mon[31522]: pgmap v157: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 597 B/s wr, 13 op/s 2024-03-18T15:23:37.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:23:37 smithi033 ceph-mon[36648]: pgmap v157: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 597 B/s wr, 13 op/s 2024-03-18T15:23:39.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:23:39 smithi104 ceph-mon[36070]: pgmap v158: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:23:39.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:23:39 smithi019 ceph-mon[31522]: pgmap v158: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:23:39.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:23:39 smithi033 ceph-mon[36648]: pgmap v158: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:23:41.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:23:41 smithi104 ceph-mon[36070]: pgmap v159: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:23:41.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:23:41 smithi019 ceph-mon[31522]: pgmap v159: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:23:41.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:23:41 smithi033 ceph-mon[36648]: pgmap v159: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:23:43.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:23:43 smithi104 ceph-mon[36070]: pgmap v160: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:23:43.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:23:43 smithi019 ceph-mon[31522]: pgmap v160: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:23:43.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:23:43 smithi033 ceph-mon[36648]: pgmap v160: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:23:45.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:23:45 smithi104 ceph-mon[36070]: pgmap v161: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:23:45.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:23:45 smithi019 ceph-mon[31522]: pgmap v161: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:23:45.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:23:45 smithi033 ceph-mon[36648]: pgmap v161: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:23:46.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:23:46 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:23:46.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:23:46 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:23:46.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:23:46 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:23:47.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:23:47 smithi104 ceph-mon[36070]: pgmap v162: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:23:47.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:23:47 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:23:47.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:23:47 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:23:47.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:23:47 smithi104 ceph-mon[36070]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:23:47.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:23:47 smithi019 ceph-mon[31522]: pgmap v162: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:23:47.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:23:47 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:23:47.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:23:47 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:23:47.466 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:23:47 smithi019 ceph-mon[31522]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:23:47.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:23:47 smithi033 ceph-mon[36648]: pgmap v162: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:23:47.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:23:47 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:23:47.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:23:47 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:23:47.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:23:47 smithi033 ceph-mon[36648]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:23:49.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:23:49 smithi104 ceph-mon[36070]: pgmap v163: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:23:49.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:23:49 smithi019 ceph-mon[31522]: pgmap v163: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:23:49.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:23:49 smithi033 ceph-mon[36648]: pgmap v163: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:23:51.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:23:51 smithi104 ceph-mon[36070]: pgmap v164: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:23:51.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:23:51 smithi019 ceph-mon[31522]: pgmap v164: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:23:51.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:23:51 smithi033 ceph-mon[36648]: pgmap v164: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:23:53.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:23:53 smithi104 ceph-mon[36070]: pgmap v165: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:23:53.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:23:53 smithi019 ceph-mon[31522]: pgmap v165: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:23:53.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:23:53 smithi033 ceph-mon[36648]: pgmap v165: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:23:55.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:23:55 smithi104 ceph-mon[36070]: pgmap v166: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:23:55.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:23:55 smithi019 ceph-mon[31522]: pgmap v166: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:23:55.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:23:55 smithi033 ceph-mon[36648]: pgmap v166: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:23:57.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:23:57 smithi104 ceph-mon[36070]: pgmap v167: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:23:57.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:23:57 smithi019 ceph-mon[31522]: pgmap v167: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:23:57.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:23:57 smithi033 ceph-mon[36648]: pgmap v167: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:23:59.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:23:59 smithi104 ceph-mon[36070]: pgmap v168: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:23:59.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:23:59 smithi019 ceph-mon[31522]: pgmap v168: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:23:59.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:23:59 smithi033 ceph-mon[36648]: pgmap v168: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:24:01.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:24:01 smithi104 ceph-mon[36070]: pgmap v169: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:24:01.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:24:01 smithi019 ceph-mon[31522]: pgmap v169: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:24:01.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:24:01 smithi033 ceph-mon[36648]: pgmap v169: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:24:03.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:24:03 smithi104 ceph-mon[36070]: pgmap v170: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-18T15:24:03.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:24:03 smithi019 ceph-mon[31522]: pgmap v170: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-18T15:24:03.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:24:03 smithi033 ceph-mon[36648]: pgmap v170: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-18T15:24:05.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:24:05 smithi104 ceph-mon[36070]: pgmap v171: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-18T15:24:05.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:24:05 smithi019 ceph-mon[31522]: pgmap v171: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-18T15:24:05.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:24:05 smithi033 ceph-mon[36648]: pgmap v171: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-18T15:24:07.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:24:07 smithi104 ceph-mon[36070]: pgmap v172: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-18T15:24:07.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:24:07 smithi019 ceph-mon[31522]: pgmap v172: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-18T15:24:07.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:24:07 smithi033 ceph-mon[36648]: pgmap v172: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-18T15:24:09.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:24:09 smithi104 ceph-mon[36070]: pgmap v173: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-18T15:24:09.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:24:09 smithi019 ceph-mon[31522]: pgmap v173: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-18T15:24:09.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:24:09 smithi033 ceph-mon[36648]: pgmap v173: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-18T15:24:11.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:24:11 smithi104 ceph-mon[36070]: pgmap v174: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-18T15:24:11.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:24:11 smithi019 ceph-mon[31522]: pgmap v174: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-18T15:24:11.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:24:11 smithi033 ceph-mon[36648]: pgmap v174: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-18T15:24:13.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:24:13 smithi104 ceph-mon[36070]: pgmap v175: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-18T15:24:13.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:24:13 smithi019 ceph-mon[31522]: pgmap v175: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-18T15:24:13.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:24:13 smithi033 ceph-mon[36648]: pgmap v175: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-18T15:24:15.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:24:15 smithi104 ceph-mon[36070]: pgmap v176: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-18T15:24:15.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:24:15 smithi019 ceph-mon[31522]: pgmap v176: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-18T15:24:15.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:24:15 smithi033 ceph-mon[36648]: pgmap v176: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-18T15:24:17.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:24:17 smithi104 ceph-mon[36070]: pgmap v177: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 170 B/s wr, 12 op/s 2024-03-18T15:24:17.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:24:17 smithi019 ceph-mon[31522]: pgmap v177: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 170 B/s wr, 12 op/s 2024-03-18T15:24:17.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:24:17 smithi033 ceph-mon[36648]: pgmap v177: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 170 B/s wr, 12 op/s 2024-03-18T15:24:19.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:24:19 smithi104 ceph-mon[36070]: pgmap v178: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-18T15:24:19.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:24:19 smithi019 ceph-mon[31522]: pgmap v178: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-18T15:24:19.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:24:19 smithi033 ceph-mon[36648]: pgmap v178: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-18T15:24:21.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:24:21 smithi104 ceph-mon[36070]: pgmap v179: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-18T15:24:21.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:24:21 smithi019 ceph-mon[31522]: pgmap v179: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-18T15:24:21.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:24:21 smithi033 ceph-mon[36648]: pgmap v179: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-18T15:24:23.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:24:23 smithi104 ceph-mon[36070]: pgmap v180: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-18T15:24:23.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:24:23 smithi019 ceph-mon[31522]: pgmap v180: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-18T15:24:23.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:24:23 smithi033 ceph-mon[36648]: pgmap v180: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-18T15:24:25.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:24:25 smithi104 ceph-mon[36070]: pgmap v181: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:24:25.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:24:25 smithi019 ceph-mon[31522]: pgmap v181: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:24:25.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:24:25 smithi033 ceph-mon[36648]: pgmap v181: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:24:27.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:24:27 smithi104 ceph-mon[36070]: pgmap v182: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:24:27.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:24:27 smithi019 ceph-mon[31522]: pgmap v182: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:24:27.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:24:27 smithi033 ceph-mon[36648]: pgmap v182: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:24:29.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:24:29 smithi104 ceph-mon[36070]: pgmap v183: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:24:29.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:24:29 smithi019 ceph-mon[31522]: pgmap v183: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:24:29.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:24:29 smithi033 ceph-mon[36648]: pgmap v183: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:24:31.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:24:31 smithi104 ceph-mon[36070]: pgmap v184: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:24:31.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:24:31 smithi019 ceph-mon[31522]: pgmap v184: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:24:31.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:24:31 smithi033 ceph-mon[36648]: pgmap v184: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:24:33.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:24:33 smithi104 ceph-mon[36070]: pgmap v185: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:24:33.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:24:33 smithi019 ceph-mon[31522]: pgmap v185: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:24:33.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:24:33 smithi033 ceph-mon[36648]: pgmap v185: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:24:35.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:24:35 smithi104 ceph-mon[36070]: pgmap v186: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:24:35.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:24:35 smithi019 ceph-mon[31522]: pgmap v186: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:24:35.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:24:35 smithi033 ceph-mon[36648]: pgmap v186: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:24:37.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:24:37 smithi104 ceph-mon[36070]: pgmap v187: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:24:37.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:24:37 smithi019 ceph-mon[31522]: pgmap v187: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:24:37.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:24:37 smithi033 ceph-mon[36648]: pgmap v187: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:24:39.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:24:39 smithi104 ceph-mon[36070]: pgmap v188: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:24:39.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:24:39 smithi019 ceph-mon[31522]: pgmap v188: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:24:39.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:24:39 smithi033 ceph-mon[36648]: pgmap v188: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:24:41.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:24:41 smithi019 ceph-mon[31522]: pgmap v189: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:24:41.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:24:41 smithi033 ceph-mon[36648]: pgmap v189: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:24:41.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:24:41 smithi104 ceph-mon[36070]: pgmap v189: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:24:43.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:24:43 smithi019 ceph-mon[31522]: pgmap v190: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:24:43.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:24:43 smithi033 ceph-mon[36648]: pgmap v190: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:24:43.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:24:43 smithi104 ceph-mon[36070]: pgmap v190: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:24:45.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:24:45 smithi019 ceph-mon[31522]: pgmap v191: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:24:45.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:24:45 smithi033 ceph-mon[36648]: pgmap v191: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:24:45.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:24:45 smithi104 ceph-mon[36070]: pgmap v191: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:24:47.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:24:47 smithi019 ceph-mon[31522]: pgmap v192: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:24:47.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:24:47 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:24:47.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:24:47 smithi033 ceph-mon[36648]: pgmap v192: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:24:47.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:24:47 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:24:47.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:24:47 smithi104 ceph-mon[36070]: pgmap v192: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:24:47.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:24:47 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:24:49.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:24:49 smithi019 ceph-mon[31522]: pgmap v193: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:24:49.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:24:49 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:24:49.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:24:49 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:24:49.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:24:49 smithi019 ceph-mon[31522]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:24:49.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:24:49 smithi033 ceph-mon[36648]: pgmap v193: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:24:49.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:24:49 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:24:49.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:24:49 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:24:49.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:24:49 smithi033 ceph-mon[36648]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:24:49.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:24:49 smithi104 ceph-mon[36070]: pgmap v193: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:24:49.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:24:49 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:24:49.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:24:49 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:24:49.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:24:49 smithi104 ceph-mon[36070]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:24:51.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:24:51 smithi033 ceph-mon[36648]: pgmap v194: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:24:51.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:24:51 smithi104 ceph-mon[36070]: pgmap v194: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:24:51.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:24:51 smithi019 ceph-mon[31522]: pgmap v194: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:24:53.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:24:53 smithi033 ceph-mon[36648]: pgmap v195: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:24:53.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:24:53 smithi104 ceph-mon[36070]: pgmap v195: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:24:53.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:24:53 smithi019 ceph-mon[31522]: pgmap v195: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:24:55.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:24:55 smithi033 ceph-mon[36648]: pgmap v196: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:24:55.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:24:55 smithi104 ceph-mon[36070]: pgmap v196: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:24:55.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:24:55 smithi019 ceph-mon[31522]: pgmap v196: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:24:57.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:24:57 smithi033 ceph-mon[36648]: pgmap v197: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:24:57.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:24:57 smithi104 ceph-mon[36070]: pgmap v197: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:24:57.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:24:57 smithi019 ceph-mon[31522]: pgmap v197: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:24:59.495 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:24:59 smithi104 ceph-mon[36070]: pgmap v198: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:24:59.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:24:59 smithi033 ceph-mon[36648]: pgmap v198: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:24:59.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:24:59 smithi019 ceph-mon[31522]: pgmap v198: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:25:01.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:25:01 smithi033 ceph-mon[36648]: pgmap v199: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:25:01.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:25:01 smithi104 ceph-mon[36070]: pgmap v199: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:25:01.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:25:01 smithi019 ceph-mon[31522]: pgmap v199: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:25:03.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:25:03 smithi033 ceph-mon[36648]: pgmap v200: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-18T15:25:03.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:25:03 smithi104 ceph-mon[36070]: pgmap v200: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-18T15:25:03.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:25:03 smithi019 ceph-mon[31522]: pgmap v200: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-18T15:25:05.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:25:05 smithi033 ceph-mon[36648]: pgmap v201: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-18T15:25:05.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:25:05 smithi104 ceph-mon[36070]: pgmap v201: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-18T15:25:05.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:25:05 smithi019 ceph-mon[31522]: pgmap v201: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-18T15:25:07.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:25:07 smithi033 ceph-mon[36648]: pgmap v202: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-18T15:25:07.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:25:07 smithi104 ceph-mon[36070]: pgmap v202: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-18T15:25:07.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:25:07 smithi019 ceph-mon[31522]: pgmap v202: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-18T15:25:09.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:25:09 smithi104 ceph-mon[36070]: pgmap v203: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-18T15:25:09.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:25:09 smithi019 ceph-mon[31522]: pgmap v203: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-18T15:25:09.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:25:09 smithi033 ceph-mon[36648]: pgmap v203: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-18T15:25:11.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:25:11 smithi104 ceph-mon[36070]: pgmap v204: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-18T15:25:11.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:25:11 smithi104 ceph-mon[36070]: osdmap e96: 12 total, 12 up, 12 in 2024-03-18T15:25:11.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:25:11 smithi019 ceph-mon[31522]: pgmap v204: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-18T15:25:11.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:25:11 smithi019 ceph-mon[31522]: osdmap e96: 12 total, 12 up, 12 in 2024-03-18T15:25:11.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:25:11 smithi033 ceph-mon[36648]: pgmap v204: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-18T15:25:11.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:25:11 smithi033 ceph-mon[36648]: osdmap e96: 12 total, 12 up, 12 in 2024-03-18T15:25:13.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:25:13 smithi104 ceph-mon[36070]: pgmap v206: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-18T15:25:13.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:25:13 smithi019 ceph-mon[31522]: pgmap v206: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-18T15:25:13.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:25:13 smithi033 ceph-mon[36648]: pgmap v206: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-18T15:25:15.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:25:15 smithi104 ceph-mon[36070]: pgmap v207: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-18T15:25:15.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:25:15 smithi019 ceph-mon[31522]: pgmap v207: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-18T15:25:15.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:25:15 smithi033 ceph-mon[36648]: pgmap v207: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-18T15:25:17.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:25:17 smithi104 ceph-mon[36070]: pgmap v208: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-18T15:25:17.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:25:17 smithi104 ceph-mon[36070]: osdmap e97: 12 total, 12 up, 12 in 2024-03-18T15:25:17.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:25:17 smithi019 ceph-mon[31522]: pgmap v208: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-18T15:25:17.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:25:17 smithi019 ceph-mon[31522]: osdmap e97: 12 total, 12 up, 12 in 2024-03-18T15:25:17.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:25:17 smithi033 ceph-mon[36648]: pgmap v208: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-18T15:25:17.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:25:17 smithi033 ceph-mon[36648]: osdmap e97: 12 total, 12 up, 12 in 2024-03-18T15:25:19.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:25:19 smithi104 ceph-mon[36070]: pgmap v210: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2024-03-18T15:25:19.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:25:19 smithi019 ceph-mon[31522]: pgmap v210: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2024-03-18T15:25:19.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:25:19 smithi033 ceph-mon[36648]: pgmap v210: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2024-03-18T15:25:21.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:25:21 smithi104 ceph-mon[36070]: pgmap v211: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 211 B/s wr, 10 op/s 2024-03-18T15:25:21.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:25:21 smithi019 ceph-mon[31522]: pgmap v211: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 211 B/s wr, 10 op/s 2024-03-18T15:25:21.758 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:25:21 smithi033 ceph-mon[36648]: pgmap v211: 129 pgs: 129 active+clean; 259 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 211 B/s wr, 10 op/s 2024-03-18T15:25:23.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:25:23 smithi104 ceph-mon[36070]: pgmap v212: 129 pgs: 129 active+clean; 259 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 9 op/s 2024-03-18T15:25:23.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:25:23 smithi019 ceph-mon[31522]: pgmap v212: 129 pgs: 129 active+clean; 259 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 9 op/s 2024-03-18T15:25:23.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:25:23 smithi033 ceph-mon[36648]: pgmap v212: 129 pgs: 129 active+clean; 259 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 9 op/s 2024-03-18T15:25:25.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:25:25 smithi104 ceph-mon[36070]: pgmap v213: 129 pgs: 129 active+clean; 259 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 9 op/s 2024-03-18T15:25:25.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:25:25 smithi019 ceph-mon[31522]: pgmap v213: 129 pgs: 129 active+clean; 259 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 9 op/s 2024-03-18T15:25:25.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:25:25 smithi033 ceph-mon[36648]: pgmap v213: 129 pgs: 129 active+clean; 259 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 9 op/s 2024-03-18T15:25:27.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:25:27 smithi104 ceph-mon[36070]: pgmap v214: 129 pgs: 129 active+clean; 259 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-03-18T15:25:27.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:25:27 smithi019 ceph-mon[31522]: pgmap v214: 129 pgs: 129 active+clean; 259 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-03-18T15:25:27.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:25:27 smithi033 ceph-mon[36648]: pgmap v214: 129 pgs: 129 active+clean; 259 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-03-18T15:25:29.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:25:29 smithi104 ceph-mon[36070]: pgmap v215: 129 pgs: 129 active+clean; 259 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T15:25:29.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:25:29 smithi019 ceph-mon[31522]: pgmap v215: 129 pgs: 129 active+clean; 259 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T15:25:29.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:25:29 smithi033 ceph-mon[36648]: pgmap v215: 129 pgs: 129 active+clean; 259 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T15:25:31.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:25:31 smithi104 ceph-mon[36070]: pgmap v216: 129 pgs: 129 active+clean; 259 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-03-18T15:25:31.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:25:31 smithi019 ceph-mon[31522]: pgmap v216: 129 pgs: 129 active+clean; 259 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-03-18T15:25:31.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:25:31 smithi033 ceph-mon[36648]: pgmap v216: 129 pgs: 129 active+clean; 259 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-03-18T15:25:33.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:25:33 smithi104 ceph-mon[36070]: pgmap v217: 129 pgs: 129 active+clean; 259 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:25:33.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:25:33 smithi019 ceph-mon[31522]: pgmap v217: 129 pgs: 129 active+clean; 259 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:25:33.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:25:33 smithi033 ceph-mon[36648]: pgmap v217: 129 pgs: 129 active+clean; 259 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:25:35.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:25:35 smithi104 ceph-mon[36070]: pgmap v218: 129 pgs: 129 active+clean; 259 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:25:35.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:25:35 smithi019 ceph-mon[31522]: pgmap v218: 129 pgs: 129 active+clean; 259 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:25:35.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:25:35 smithi033 ceph-mon[36648]: pgmap v218: 129 pgs: 129 active+clean; 259 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:25:37.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:25:37 smithi104 ceph-mon[36070]: pgmap v219: 129 pgs: 129 active+clean; 259 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:25:37.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:25:37 smithi019 ceph-mon[31522]: pgmap v219: 129 pgs: 129 active+clean; 259 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:25:37.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:25:37 smithi033 ceph-mon[36648]: pgmap v219: 129 pgs: 129 active+clean; 259 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:25:39.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:25:39 smithi104 ceph-mon[36070]: pgmap v220: 129 pgs: 129 active+clean; 259 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:25:39.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:25:39 smithi019 ceph-mon[31522]: pgmap v220: 129 pgs: 129 active+clean; 259 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:25:39.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:25:39 smithi033 ceph-mon[36648]: pgmap v220: 129 pgs: 129 active+clean; 259 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:25:41.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:25:41 smithi104 ceph-mon[36070]: pgmap v221: 129 pgs: 129 active+clean; 259 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:25:41.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:25:41 smithi019 ceph-mon[31522]: pgmap v221: 129 pgs: 129 active+clean; 259 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:25:41.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:25:41 smithi033 ceph-mon[36648]: pgmap v221: 129 pgs: 129 active+clean; 259 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:25:43.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:25:43 smithi104 ceph-mon[36070]: pgmap v222: 129 pgs: 129 active+clean; 259 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:25:43.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:25:43 smithi019 ceph-mon[31522]: pgmap v222: 129 pgs: 129 active+clean; 259 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:25:43.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:25:43 smithi033 ceph-mon[36648]: pgmap v222: 129 pgs: 129 active+clean; 259 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:25:45.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:25:45 smithi104 ceph-mon[36070]: pgmap v223: 129 pgs: 129 active+clean; 259 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:25:45.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:25:45 smithi019 ceph-mon[31522]: pgmap v223: 129 pgs: 129 active+clean; 259 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:25:45.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:25:45 smithi033 ceph-mon[36648]: pgmap v223: 129 pgs: 129 active+clean; 259 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:25:47.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:25:47 smithi104 ceph-mon[36070]: pgmap v224: 129 pgs: 129 active+clean; 259 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:25:47.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:25:47 smithi019 ceph-mon[31522]: pgmap v224: 129 pgs: 129 active+clean; 259 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:25:47.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:25:47 smithi033 ceph-mon[36648]: pgmap v224: 129 pgs: 129 active+clean; 259 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:25:49.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:25:49 smithi104 ceph-mon[36070]: pgmap v225: 129 pgs: 129 active+clean; 259 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:25:49.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:25:49 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:25:49.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:25:49 smithi019 ceph-mon[31522]: pgmap v225: 129 pgs: 129 active+clean; 259 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:25:49.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:25:49 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:25:49.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:25:49 smithi033 ceph-mon[36648]: pgmap v225: 129 pgs: 129 active+clean; 259 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:25:49.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:25:49 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:25:50.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:25:50 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:25:50.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:25:50 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:25:50.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:25:50 smithi104 ceph-mon[36070]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:25:50.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:25:50 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:25:50.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:25:50 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:25:50.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:25:50 smithi019 ceph-mon[31522]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:25:50.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:25:50 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:25:50.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:25:50 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:25:50.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:25:50 smithi033 ceph-mon[36648]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:25:51.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:25:51 smithi104 ceph-mon[36070]: pgmap v226: 129 pgs: 129 active+clean; 259 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:25:51.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:25:51 smithi019 ceph-mon[31522]: pgmap v226: 129 pgs: 129 active+clean; 259 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:25:51.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:25:51 smithi033 ceph-mon[36648]: pgmap v226: 129 pgs: 129 active+clean; 259 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:25:53.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:25:53 smithi104 ceph-mon[36070]: pgmap v227: 129 pgs: 129 active+clean; 259 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:25:53.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:25:53 smithi019 ceph-mon[31522]: pgmap v227: 129 pgs: 129 active+clean; 259 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:25:53.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:25:53 smithi033 ceph-mon[36648]: pgmap v227: 129 pgs: 129 active+clean; 259 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:25:55.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:25:55 smithi104 ceph-mon[36070]: pgmap v228: 129 pgs: 129 active+clean; 259 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:25:55.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:25:55 smithi019 ceph-mon[31522]: pgmap v228: 129 pgs: 129 active+clean; 259 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:25:55.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:25:55 smithi033 ceph-mon[36648]: pgmap v228: 129 pgs: 129 active+clean; 259 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:25:57.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:25:57 smithi104 ceph-mon[36070]: pgmap v229: 129 pgs: 129 active+clean; 259 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:25:57.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:25:57 smithi019 ceph-mon[31522]: pgmap v229: 129 pgs: 129 active+clean; 259 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:25:57.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:25:57 smithi033 ceph-mon[36648]: pgmap v229: 129 pgs: 129 active+clean; 259 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:25:59.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:25:59 smithi104 ceph-mon[36070]: pgmap v230: 129 pgs: 129 active+clean; 259 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:25:59.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:25:59 smithi019 ceph-mon[31522]: pgmap v230: 129 pgs: 129 active+clean; 259 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:25:59.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:25:59 smithi033 ceph-mon[36648]: pgmap v230: 129 pgs: 129 active+clean; 259 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:26:01.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:26:01 smithi104 ceph-mon[36070]: pgmap v231: 129 pgs: 129 active+clean; 259 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:26:01.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:26:01 smithi019 ceph-mon[31522]: pgmap v231: 129 pgs: 129 active+clean; 259 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:26:01.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:26:01 smithi033 ceph-mon[36648]: pgmap v231: 129 pgs: 129 active+clean; 259 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:26:03.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:26:03 smithi104 ceph-mon[36070]: pgmap v232: 129 pgs: 129 active+clean; 259 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-18T15:26:03.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:26:03 smithi019 ceph-mon[31522]: pgmap v232: 129 pgs: 129 active+clean; 259 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-18T15:26:03.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:26:03 smithi033 ceph-mon[36648]: pgmap v232: 129 pgs: 129 active+clean; 259 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-18T15:26:05.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:26:05 smithi104 ceph-mon[36070]: pgmap v233: 129 pgs: 129 active+clean; 259 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-18T15:26:05.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:26:05 smithi019 ceph-mon[31522]: pgmap v233: 129 pgs: 129 active+clean; 259 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-18T15:26:05.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:26:05 smithi033 ceph-mon[36648]: pgmap v233: 129 pgs: 129 active+clean; 259 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-18T15:26:07.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:26:07 smithi104 ceph-mon[36070]: pgmap v234: 129 pgs: 129 active+clean; 259 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-18T15:26:07.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:26:07 smithi019 ceph-mon[31522]: pgmap v234: 129 pgs: 129 active+clean; 259 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-18T15:26:07.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:26:07 smithi033 ceph-mon[36648]: pgmap v234: 129 pgs: 129 active+clean; 259 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-18T15:26:09.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:26:09 smithi104 ceph-mon[36070]: pgmap v235: 129 pgs: 129 active+clean; 259 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-18T15:26:09.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:26:09 smithi019 ceph-mon[31522]: pgmap v235: 129 pgs: 129 active+clean; 259 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-18T15:26:09.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:26:09 smithi033 ceph-mon[36648]: pgmap v235: 129 pgs: 129 active+clean; 259 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-18T15:26:10.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:26:10 smithi104 ceph-mon[36070]: osdmap e98: 12 total, 12 up, 12 in 2024-03-18T15:26:10.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:26:10 smithi019 ceph-mon[31522]: osdmap e98: 12 total, 12 up, 12 in 2024-03-18T15:26:10.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:26:10 smithi033 ceph-mon[36648]: osdmap e98: 12 total, 12 up, 12 in 2024-03-18T15:26:11.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:26:11 smithi019 ceph-mon[31522]: pgmap v237: 129 pgs: 129 active+clean; 259 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-18T15:26:11.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:26:11 smithi033 ceph-mon[36648]: pgmap v237: 129 pgs: 129 active+clean; 259 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-18T15:26:11.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:26:11 smithi104 ceph-mon[36070]: pgmap v237: 129 pgs: 129 active+clean; 259 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-18T15:26:13.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:26:13 smithi019 ceph-mon[31522]: pgmap v238: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-18T15:26:13.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:26:13 smithi033 ceph-mon[36648]: pgmap v238: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-18T15:26:13.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:26:13 smithi104 ceph-mon[36070]: pgmap v238: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-18T15:26:15.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:26:15 smithi019 ceph-mon[31522]: pgmap v239: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-18T15:26:15.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:26:15 smithi033 ceph-mon[36648]: pgmap v239: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-18T15:26:15.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:26:15 smithi104 ceph-mon[36070]: pgmap v239: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-18T15:26:17.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:26:17 smithi033 ceph-mon[36648]: pgmap v240: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-18T15:26:17.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:26:17 smithi033 ceph-mon[36648]: osdmap e99: 12 total, 12 up, 12 in 2024-03-18T15:26:17.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:26:17 smithi104 ceph-mon[36070]: pgmap v240: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-18T15:26:17.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:26:17 smithi104 ceph-mon[36070]: osdmap e99: 12 total, 12 up, 12 in 2024-03-18T15:26:17.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:26:17 smithi019 ceph-mon[31522]: pgmap v240: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-18T15:26:17.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:26:17 smithi019 ceph-mon[31522]: osdmap e99: 12 total, 12 up, 12 in 2024-03-18T15:26:19.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:26:19 smithi033 ceph-mon[36648]: pgmap v242: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 239 B/s wr, 12 op/s 2024-03-18T15:26:19.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:26:19 smithi104 ceph-mon[36070]: pgmap v242: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 239 B/s wr, 12 op/s 2024-03-18T15:26:19.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:26:19 smithi019 ceph-mon[31522]: pgmap v242: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 239 B/s wr, 12 op/s 2024-03-18T15:26:21.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:26:21 smithi033 ceph-mon[36648]: pgmap v243: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-18T15:26:21.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:26:21 smithi104 ceph-mon[36070]: pgmap v243: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-18T15:26:21.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:26:21 smithi019 ceph-mon[31522]: pgmap v243: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-18T15:26:23.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:26:23 smithi033 ceph-mon[36648]: pgmap v244: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-18T15:26:23.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:26:23 smithi104 ceph-mon[36070]: pgmap v244: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-18T15:26:23.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:26:23 smithi019 ceph-mon[31522]: pgmap v244: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-18T15:26:25.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:26:25 smithi033 ceph-mon[36648]: pgmap v245: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-18T15:26:25.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:26:25 smithi104 ceph-mon[36070]: pgmap v245: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-18T15:26:25.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:26:25 smithi019 ceph-mon[31522]: pgmap v245: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-18T15:26:27.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:26:27 smithi033 ceph-mon[36648]: pgmap v246: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T15:26:27.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:26:27 smithi104 ceph-mon[36070]: pgmap v246: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T15:26:27.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:26:27 smithi019 ceph-mon[31522]: pgmap v246: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T15:26:29.745 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:26:29 smithi104 ceph-mon[36070]: pgmap v247: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T15:26:29.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:26:29 smithi033 ceph-mon[36648]: pgmap v247: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T15:26:29.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:26:29 smithi019 ceph-mon[31522]: pgmap v247: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T15:26:31.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:26:31 smithi033 ceph-mon[36648]: pgmap v248: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:26:31.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:26:31 smithi104 ceph-mon[36070]: pgmap v248: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:26:31.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:26:31 smithi019 ceph-mon[31522]: pgmap v248: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:26:33.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:26:33 smithi033 ceph-mon[36648]: pgmap v249: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:26:33.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:26:33 smithi104 ceph-mon[36070]: pgmap v249: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:26:33.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:26:33 smithi019 ceph-mon[31522]: pgmap v249: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:26:35.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:26:35 smithi033 ceph-mon[36648]: pgmap v250: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:26:35.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:26:35 smithi104 ceph-mon[36070]: pgmap v250: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:26:35.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:26:35 smithi019 ceph-mon[31522]: pgmap v250: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:26:37.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:26:37 smithi104 ceph-mon[36070]: pgmap v251: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:26:37.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:26:37 smithi019 ceph-mon[31522]: pgmap v251: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:26:38.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:26:37 smithi033 ceph-mon[36648]: pgmap v251: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:26:39.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:26:39 smithi104 ceph-mon[36070]: pgmap v252: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:26:39.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:26:39 smithi019 ceph-mon[31522]: pgmap v252: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:26:40.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:26:39 smithi033 ceph-mon[36648]: pgmap v252: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:26:41.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:26:41 smithi104 ceph-mon[36070]: pgmap v253: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:26:41.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:26:41 smithi019 ceph-mon[31522]: pgmap v253: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:26:42.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:26:41 smithi033 ceph-mon[36648]: pgmap v253: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:26:43.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:26:43 smithi104 ceph-mon[36070]: pgmap v254: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:26:43.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:26:43 smithi019 ceph-mon[31522]: pgmap v254: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:26:44.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:26:43 smithi033 ceph-mon[36648]: pgmap v254: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:26:45.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:26:45 smithi104 ceph-mon[36070]: pgmap v255: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:26:45.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:26:45 smithi019 ceph-mon[31522]: pgmap v255: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:26:46.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:26:45 smithi033 ceph-mon[36648]: pgmap v255: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:26:47.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:26:47 smithi104 ceph-mon[36070]: pgmap v256: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:26:47.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:26:47 smithi019 ceph-mon[31522]: pgmap v256: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:26:48.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:26:47 smithi033 ceph-mon[36648]: pgmap v256: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:26:49.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:26:49 smithi104 ceph-mon[36070]: pgmap v257: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:26:49.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:26:49 smithi019 ceph-mon[31522]: pgmap v257: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:26:50.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:26:49 smithi033 ceph-mon[36648]: pgmap v257: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:26:50.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:26:50 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:26:50.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:26:50 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:26:51.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:26:50 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:26:51.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:26:51 smithi104 ceph-mon[36070]: pgmap v258: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:26:51.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:26:51 smithi019 ceph-mon[31522]: pgmap v258: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:26:52.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:26:51 smithi033 ceph-mon[36648]: pgmap v258: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:26:52.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:26:52 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:26:52.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:26:52 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:26:52.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:26:52 smithi104 ceph-mon[36070]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:26:52.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:26:52 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:26:52.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:26:52 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:26:52.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:26:52 smithi019 ceph-mon[31522]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:26:53.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:26:52 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:26:53.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:26:52 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:26:53.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:26:52 smithi033 ceph-mon[36648]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:26:53.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:26:53 smithi104 ceph-mon[36070]: pgmap v259: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:26:53.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:26:53 smithi019 ceph-mon[31522]: pgmap v259: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:26:54.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:26:53 smithi033 ceph-mon[36648]: pgmap v259: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:26:55.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:26:55 smithi104 ceph-mon[36070]: pgmap v260: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:26:55.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:26:55 smithi019 ceph-mon[31522]: pgmap v260: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:26:56.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:26:55 smithi033 ceph-mon[36648]: pgmap v260: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:26:57.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:26:57 smithi104 ceph-mon[36070]: pgmap v261: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:26:57.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:26:57 smithi019 ceph-mon[31522]: pgmap v261: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:26:58.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:26:57 smithi033 ceph-mon[36648]: pgmap v261: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:26:59.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:26:59 smithi104 ceph-mon[36070]: pgmap v262: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:26:59.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:26:59 smithi019 ceph-mon[31522]: pgmap v262: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:27:00.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:26:59 smithi033 ceph-mon[36648]: pgmap v262: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:27:01.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:27:01 smithi104 ceph-mon[36070]: pgmap v263: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:27:01.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:27:01 smithi019 ceph-mon[31522]: pgmap v263: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:27:02.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:27:01 smithi033 ceph-mon[36648]: pgmap v263: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:27:03.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:27:03 smithi104 ceph-mon[36070]: pgmap v264: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-18T15:27:03.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:27:03 smithi019 ceph-mon[31522]: pgmap v264: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-18T15:27:04.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:27:03 smithi033 ceph-mon[36648]: pgmap v264: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-18T15:27:05.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:27:05 smithi104 ceph-mon[36070]: pgmap v265: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-18T15:27:05.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:27:05 smithi019 ceph-mon[31522]: pgmap v265: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-18T15:27:06.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:27:05 smithi033 ceph-mon[36648]: pgmap v265: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-18T15:27:07.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:27:07 smithi104 ceph-mon[36070]: pgmap v266: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-18T15:27:07.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:27:07 smithi019 ceph-mon[31522]: pgmap v266: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-18T15:27:08.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:27:07 smithi033 ceph-mon[36648]: pgmap v266: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-18T15:27:09.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:27:09 smithi104 ceph-mon[36070]: pgmap v267: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-18T15:27:09.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:27:09 smithi019 ceph-mon[31522]: pgmap v267: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-18T15:27:10.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:27:09 smithi033 ceph-mon[36648]: pgmap v267: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-18T15:27:11.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:27:10 smithi033 ceph-mon[36648]: osdmap e100: 12 total, 12 up, 12 in 2024-03-18T15:27:11.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:27:10 smithi104 ceph-mon[36070]: osdmap e100: 12 total, 12 up, 12 in 2024-03-18T15:27:11.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:27:10 smithi019 ceph-mon[31522]: osdmap e100: 12 total, 12 up, 12 in 2024-03-18T15:27:12.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:27:11 smithi033 ceph-mon[36648]: pgmap v269: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-18T15:27:12.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:27:11 smithi104 ceph-mon[36070]: pgmap v269: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-18T15:27:12.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:27:11 smithi019 ceph-mon[31522]: pgmap v269: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-18T15:27:14.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:27:13 smithi033 ceph-mon[36648]: pgmap v270: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 27 KiB/s wr, 12 op/s 2024-03-18T15:27:14.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:27:13 smithi104 ceph-mon[36070]: pgmap v270: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 27 KiB/s wr, 12 op/s 2024-03-18T15:27:14.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:27:13 smithi019 ceph-mon[31522]: pgmap v270: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 27 KiB/s wr, 12 op/s 2024-03-18T15:27:16.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:27:15 smithi033 ceph-mon[36648]: pgmap v271: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 27 KiB/s wr, 12 op/s 2024-03-18T15:27:16.017 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:27:15 smithi019 ceph-mon[31522]: pgmap v271: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 27 KiB/s wr, 12 op/s 2024-03-18T15:27:16.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:27:15 smithi104 ceph-mon[36070]: pgmap v271: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 27 KiB/s wr, 12 op/s 2024-03-18T15:27:18.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:27:17 smithi033 ceph-mon[36648]: pgmap v272: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-18T15:27:18.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:27:17 smithi033 ceph-mon[36648]: osdmap e101: 12 total, 12 up, 12 in 2024-03-18T15:27:18.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:27:17 smithi104 ceph-mon[36070]: pgmap v272: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-18T15:27:18.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:27:17 smithi104 ceph-mon[36070]: osdmap e101: 12 total, 12 up, 12 in 2024-03-18T15:27:18.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:27:17 smithi019 ceph-mon[31522]: pgmap v272: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-18T15:27:18.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:27:17 smithi019 ceph-mon[31522]: osdmap e101: 12 total, 12 up, 12 in 2024-03-18T15:27:20.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:27:19 smithi033 ceph-mon[36648]: pgmap v274: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 246 B/s wr, 12 op/s 2024-03-18T15:27:20.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:27:19 smithi104 ceph-mon[36070]: pgmap v274: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 246 B/s wr, 12 op/s 2024-03-18T15:27:20.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:27:19 smithi019 ceph-mon[31522]: pgmap v274: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 246 B/s wr, 12 op/s 2024-03-18T15:27:22.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:27:21 smithi033 ceph-mon[36648]: pgmap v275: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-18T15:27:22.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:27:21 smithi104 ceph-mon[36070]: pgmap v275: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-18T15:27:22.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:27:21 smithi019 ceph-mon[31522]: pgmap v275: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-18T15:27:23.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:27:22 smithi033 ceph-mon[36648]: pgmap v276: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-18T15:27:23.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:27:22 smithi104 ceph-mon[36070]: pgmap v276: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-18T15:27:23.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:27:22 smithi019 ceph-mon[31522]: pgmap v276: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-18T15:27:25.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:27:25 smithi104 ceph-mon[36070]: pgmap v277: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-18T15:27:25.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:27:25 smithi019 ceph-mon[31522]: pgmap v277: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-18T15:27:25.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:27:25 smithi033 ceph-mon[36648]: pgmap v277: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-18T15:27:27.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:27:27 smithi104 ceph-mon[36070]: pgmap v278: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T15:27:27.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:27:27 smithi019 ceph-mon[31522]: pgmap v278: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T15:27:27.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:27:27 smithi033 ceph-mon[36648]: pgmap v278: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T15:27:29.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:27:29 smithi104 ceph-mon[36070]: pgmap v279: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T15:27:29.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:27:29 smithi019 ceph-mon[31522]: pgmap v279: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T15:27:29.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:27:29 smithi033 ceph-mon[36648]: pgmap v279: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T15:27:31.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:27:31 smithi104 ceph-mon[36070]: pgmap v280: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:27:31.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:27:31 smithi019 ceph-mon[31522]: pgmap v280: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:27:31.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:27:31 smithi033 ceph-mon[36648]: pgmap v280: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:27:33.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:27:33 smithi104 ceph-mon[36070]: pgmap v281: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:27:33.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:27:33 smithi019 ceph-mon[31522]: pgmap v281: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:27:33.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:27:33 smithi033 ceph-mon[36648]: pgmap v281: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:27:35.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:27:35 smithi104 ceph-mon[36070]: pgmap v282: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:27:35.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:27:35 smithi019 ceph-mon[31522]: pgmap v282: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:27:35.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:27:35 smithi033 ceph-mon[36648]: pgmap v282: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:27:37.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:27:37 smithi104 ceph-mon[36070]: pgmap v283: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:27:37.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:27:37 smithi019 ceph-mon[31522]: pgmap v283: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:27:37.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:27:37 smithi033 ceph-mon[36648]: pgmap v283: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:27:39.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:27:39 smithi104 ceph-mon[36070]: pgmap v284: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:27:39.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:27:39 smithi019 ceph-mon[31522]: pgmap v284: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:27:39.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:27:39 smithi033 ceph-mon[36648]: pgmap v284: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:27:41.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:27:41 smithi104 ceph-mon[36070]: pgmap v285: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:27:41.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:27:41 smithi019 ceph-mon[31522]: pgmap v285: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:27:41.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:27:41 smithi033 ceph-mon[36648]: pgmap v285: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:27:43.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:27:43 smithi104 ceph-mon[36070]: pgmap v286: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:27:43.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:27:43 smithi019 ceph-mon[31522]: pgmap v286: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:27:43.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:27:43 smithi033 ceph-mon[36648]: pgmap v286: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:27:45.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:27:45 smithi104 ceph-mon[36070]: pgmap v287: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:27:45.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:27:45 smithi019 ceph-mon[31522]: pgmap v287: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:27:45.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:27:45 smithi033 ceph-mon[36648]: pgmap v287: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:27:47.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:27:47 smithi104 ceph-mon[36070]: pgmap v288: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:27:47.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:27:47 smithi019 ceph-mon[31522]: pgmap v288: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:27:47.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:27:47 smithi033 ceph-mon[36648]: pgmap v288: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:27:49.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:27:49 smithi104 ceph-mon[36070]: pgmap v289: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:27:49.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:27:49 smithi019 ceph-mon[31522]: pgmap v289: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:27:49.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:27:49 smithi033 ceph-mon[36648]: pgmap v289: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:27:51.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:27:51 smithi104 ceph-mon[36070]: pgmap v290: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:27:51.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:27:51 smithi019 ceph-mon[31522]: pgmap v290: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:27:51.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:27:51 smithi033 ceph-mon[36648]: pgmap v290: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:27:52.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:27:52 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:27:52.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:27:52 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:27:52.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:27:52 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:27:53.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:27:53 smithi104 ceph-mon[36070]: pgmap v291: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:27:53.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:27:53 smithi019 ceph-mon[31522]: pgmap v291: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:27:53.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:27:53 smithi033 ceph-mon[36648]: pgmap v291: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:27:54.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:27:54 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:27:54.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:27:54 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:27:54.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:27:54 smithi104 ceph-mon[36070]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:27:54.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:27:54 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:27:54.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:27:54 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:27:54.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:27:54 smithi019 ceph-mon[31522]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:27:54.506 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:27:54 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:27:54.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:27:54 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:27:54.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:27:54 smithi033 ceph-mon[36648]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:27:55.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:27:55 smithi104 ceph-mon[36070]: pgmap v292: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:27:55.718 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:27:55 smithi019 ceph-mon[31522]: pgmap v292: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:27:55.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:27:55 smithi033 ceph-mon[36648]: pgmap v292: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:27:57.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:27:57 smithi104 ceph-mon[36070]: pgmap v293: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:27:57.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:27:57 smithi019 ceph-mon[31522]: pgmap v293: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:27:57.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:27:57 smithi033 ceph-mon[36648]: pgmap v293: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:27:59.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:27:59 smithi104 ceph-mon[36070]: pgmap v294: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:27:59.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:27:59 smithi019 ceph-mon[31522]: pgmap v294: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:27:59.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:27:59 smithi033 ceph-mon[36648]: pgmap v294: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:28:01.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:28:01 smithi104 ceph-mon[36070]: pgmap v295: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:28:01.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:28:01 smithi019 ceph-mon[31522]: pgmap v295: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:28:01.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:28:01 smithi033 ceph-mon[36648]: pgmap v295: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:28:03.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:28:03 smithi104 ceph-mon[36070]: pgmap v296: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 0 B/s wr, 14 op/s 2024-03-18T15:28:03.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:28:03 smithi019 ceph-mon[31522]: pgmap v296: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 0 B/s wr, 14 op/s 2024-03-18T15:28:03.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:28:03 smithi033 ceph-mon[36648]: pgmap v296: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 0 B/s wr, 14 op/s 2024-03-18T15:28:05.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:28:05 smithi104 ceph-mon[36070]: pgmap v297: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-18T15:28:05.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:28:05 smithi019 ceph-mon[31522]: pgmap v297: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-18T15:28:05.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:28:05 smithi033 ceph-mon[36648]: pgmap v297: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-18T15:28:07.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:28:07 smithi104 ceph-mon[36070]: pgmap v298: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-18T15:28:07.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:28:07 smithi019 ceph-mon[31522]: pgmap v298: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-18T15:28:07.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:28:07 smithi033 ceph-mon[36648]: pgmap v298: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-18T15:28:09.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:28:09 smithi104 ceph-mon[36070]: pgmap v299: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-18T15:28:09.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:28:09 smithi019 ceph-mon[31522]: pgmap v299: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-18T15:28:09.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:28:09 smithi033 ceph-mon[36648]: pgmap v299: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-18T15:28:10.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:28:10 smithi104 ceph-mon[36070]: osdmap e102: 12 total, 12 up, 12 in 2024-03-18T15:28:10.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:28:10 smithi019 ceph-mon[31522]: osdmap e102: 12 total, 12 up, 12 in 2024-03-18T15:28:10.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:28:10 smithi033 ceph-mon[36648]: osdmap e102: 12 total, 12 up, 12 in 2024-03-18T15:28:11.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:28:11 smithi104 ceph-mon[36070]: pgmap v301: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-18T15:28:11.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:28:11 smithi019 ceph-mon[31522]: pgmap v301: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-18T15:28:11.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:28:11 smithi033 ceph-mon[36648]: pgmap v301: 129 pgs: 129 active+clean; 1.7 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-18T15:28:13.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:28:13 smithi104 ceph-mon[36070]: pgmap v302: 129 pgs: 129 active+clean; 1.7 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2024-03-18T15:28:13.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:28:13 smithi019 ceph-mon[31522]: pgmap v302: 129 pgs: 129 active+clean; 1.7 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2024-03-18T15:28:13.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:28:13 smithi033 ceph-mon[36648]: pgmap v302: 129 pgs: 129 active+clean; 1.7 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2024-03-18T15:28:15.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:28:15 smithi104 ceph-mon[36070]: pgmap v303: 129 pgs: 129 active+clean; 1.7 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2024-03-18T15:28:15.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:28:15 smithi019 ceph-mon[31522]: pgmap v303: 129 pgs: 129 active+clean; 1.7 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2024-03-18T15:28:15.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:28:15 smithi033 ceph-mon[36648]: pgmap v303: 129 pgs: 129 active+clean; 1.7 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2024-03-18T15:28:17.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:28:17 smithi104 ceph-mon[36070]: pgmap v304: 129 pgs: 129 active+clean; 1.7 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-18T15:28:17.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:28:17 smithi104 ceph-mon[36070]: osdmap e103: 12 total, 12 up, 12 in 2024-03-18T15:28:17.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:28:17 smithi019 ceph-mon[31522]: pgmap v304: 129 pgs: 129 active+clean; 1.7 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-18T15:28:17.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:28:17 smithi019 ceph-mon[31522]: osdmap e103: 12 total, 12 up, 12 in 2024-03-18T15:28:17.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:28:17 smithi033 ceph-mon[36648]: pgmap v304: 129 pgs: 129 active+clean; 1.7 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-18T15:28:17.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:28:17 smithi033 ceph-mon[36648]: osdmap e103: 12 total, 12 up, 12 in 2024-03-18T15:28:19.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:28:19 smithi104 ceph-mon[36070]: pgmap v306: 129 pgs: 129 active+clean; 1.7 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 236 B/s wr, 11 op/s 2024-03-18T15:28:19.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:28:19 smithi019 ceph-mon[31522]: pgmap v306: 129 pgs: 129 active+clean; 1.7 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 236 B/s wr, 11 op/s 2024-03-18T15:28:19.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:28:19 smithi033 ceph-mon[36648]: pgmap v306: 129 pgs: 129 active+clean; 1.7 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 236 B/s wr, 11 op/s 2024-03-18T15:28:21.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:28:21 smithi104 ceph-mon[36070]: pgmap v307: 129 pgs: 129 active+clean; 1.7 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-18T15:28:21.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:28:21 smithi019 ceph-mon[31522]: pgmap v307: 129 pgs: 129 active+clean; 1.7 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-18T15:28:21.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:28:21 smithi033 ceph-mon[36648]: pgmap v307: 129 pgs: 129 active+clean; 1.7 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-18T15:28:23.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:28:23 smithi104 ceph-mon[36070]: pgmap v308: 129 pgs: 129 active+clean; 1.7 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-18T15:28:23.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:28:23 smithi019 ceph-mon[31522]: pgmap v308: 129 pgs: 129 active+clean; 1.7 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-18T15:28:23.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:28:23 smithi033 ceph-mon[36648]: pgmap v308: 129 pgs: 129 active+clean; 1.7 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-18T15:28:25.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:28:25 smithi104 ceph-mon[36070]: pgmap v309: 129 pgs: 129 active+clean; 1.7 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-18T15:28:25.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:28:25 smithi019 ceph-mon[31522]: pgmap v309: 129 pgs: 129 active+clean; 1.7 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-18T15:28:25.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:28:25 smithi033 ceph-mon[36648]: pgmap v309: 129 pgs: 129 active+clean; 1.7 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-18T15:28:27.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:28:27 smithi104 ceph-mon[36070]: pgmap v310: 129 pgs: 129 active+clean; 1.7 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-03-18T15:28:27.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:28:27 smithi019 ceph-mon[31522]: pgmap v310: 129 pgs: 129 active+clean; 1.7 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-03-18T15:28:27.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:28:27 smithi033 ceph-mon[36648]: pgmap v310: 129 pgs: 129 active+clean; 1.7 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-03-18T15:28:29.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:28:29 smithi104 ceph-mon[36070]: pgmap v311: 129 pgs: 129 active+clean; 1.7 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-03-18T15:28:29.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:28:29 smithi019 ceph-mon[31522]: pgmap v311: 129 pgs: 129 active+clean; 1.7 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-03-18T15:28:29.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:28:29 smithi033 ceph-mon[36648]: pgmap v311: 129 pgs: 129 active+clean; 1.7 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-03-18T15:28:31.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:28:31 smithi104 ceph-mon[36070]: pgmap v312: 129 pgs: 129 active+clean; 1.7 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-03-18T15:28:31.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:28:31 smithi019 ceph-mon[31522]: pgmap v312: 129 pgs: 129 active+clean; 1.7 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-03-18T15:28:31.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:28:31 smithi033 ceph-mon[36648]: pgmap v312: 129 pgs: 129 active+clean; 1.7 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-03-18T15:28:32.730 DEBUG:teuthology.orchestra.run.smithi019:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-03-18T15:28:32.964 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:32.959+0000 7f0224800640 1 -- 172.21.15.19:0/3480084589 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f021c10d0f0 msgr2=0x7f021c114cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:28:32.964 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:32.959+0000 7f0224800640 1 --2- 172.21.15.19:0/3480084589 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f021c10d0f0 0x7f021c114cc0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f0218009ca0 tx=0x7f021802c100 comp rx=0 tx=0).stop 2024-03-18T15:28:32.964 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:32.959+0000 7f0224800640 1 -- 172.21.15.19:0/3480084589 shutdown_connections 2024-03-18T15:28:32.964 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:32.959+0000 7f0224800640 1 --2- 172.21.15.19:0/3480084589 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f021c10d0f0 0x7f021c114cc0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:28:32.964 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:32.959+0000 7f0224800640 1 --2- 172.21.15.19:0/3480084589 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f021c1089d0 0x7f021c10ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:28:32.964 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:32.959+0000 7f0224800640 1 --2- 172.21.15.19:0/3480084589 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f021c108020 0x7f021c108400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:28:32.964 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:32.959+0000 7f0224800640 1 -- 172.21.15.19:0/3480084589 >> 172.21.15.19:0/3480084589 conn(0x7f021c075d50 msgr2=0x7f021c078170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:28:32.964 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:32.959+0000 7f0224800640 1 -- 172.21.15.19:0/3480084589 shutdown_connections 2024-03-18T15:28:32.965 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:32.959+0000 7f0224800640 1 -- 172.21.15.19:0/3480084589 wait complete. 2024-03-18T15:28:32.965 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:32.960+0000 7f0224800640 1 Processor -- start 2024-03-18T15:28:32.965 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:32.960+0000 7f0224800640 1 -- start start 2024-03-18T15:28:32.965 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:32.961+0000 7f0224800640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f021c108020 0x7f021c19e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:28:32.965 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:32.961+0000 7f0224800640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f021c10d0f0 0x7f021c19eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:28:32.965 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:32.961+0000 7f0224800640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f021c1a2eb0 0x7f021c1a3360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:28:32.965 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:32.961+0000 7f0224800640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f021c118190 con 0x7f021c10d0f0 2024-03-18T15:28:32.965 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:32.961+0000 7f0224800640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f021c118010 con 0x7f021c1a2eb0 2024-03-18T15:28:32.965 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:32.961+0000 7f0224800640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f021c118310 con 0x7f021c108020 2024-03-18T15:28:32.965 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:32.961+0000 7f0221400640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f021c10d0f0 0x7f021c19eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:28:32.965 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:32.961+0000 7f0221400640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f021c10d0f0 0x7f021c19eb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:43902/0 (socket says 172.21.15.19:43902) 2024-03-18T15:28:32.965 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:32.961+0000 7f0221400640 1 -- 172.21.15.19:0/3959729288 learned_addr learned my addr 172.21.15.19:0/3959729288 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:28:32.966 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:32.961+0000 7f0222800640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f021c1a2eb0 0x7f021c1a3360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:28:32.966 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:32.961+0000 7f0221e00640 1 --2- 172.21.15.19:0/3959729288 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f021c108020 0x7f021c19e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:28:32.966 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:32.961+0000 7f0222800640 1 -- 172.21.15.19:0/3959729288 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f021c108020 msgr2=0x7f021c19e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:28:32.966 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:32.961+0000 7f0222800640 1 --2- 172.21.15.19:0/3959729288 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f021c108020 0x7f021c19e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:28:32.966 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:32.962+0000 7f0222800640 1 -- 172.21.15.19:0/3959729288 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f021c10d0f0 msgr2=0x7f021c19eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:28:32.966 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:32.962+0000 7f0222800640 1 --2- 172.21.15.19:0/3959729288 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f021c10d0f0 0x7f021c19eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:28:32.966 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:32.962+0000 7f0222800640 1 -- 172.21.15.19:0/3959729288 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f021c1a39a0 con 0x7f021c1a2eb0 2024-03-18T15:28:32.966 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:32.962+0000 7f0222800640 1 --2- 172.21.15.19:0/3959729288 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f021c1a2eb0 0x7f021c1a3360 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f0218009bf0 tx=0x7f0218035d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:28:32.966 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:32.962+0000 7f0216a00640 1 -- 172.21.15.19:0/3959729288 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0218039040 con 0x7f021c1a2eb0 2024-03-18T15:28:32.966 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:32.962+0000 7f0224800640 1 -- 172.21.15.19:0/3959729288 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f021c1a3c30 con 0x7f021c1a2eb0 2024-03-18T15:28:32.966 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:32.962+0000 7f0216a00640 1 -- 172.21.15.19:0/3959729288 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0218044070 con 0x7f021c1a2eb0 2024-03-18T15:28:32.966 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:32.962+0000 7f0216a00640 1 -- 172.21.15.19:0/3959729288 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02180043c0 con 0x7f021c1a2eb0 2024-03-18T15:28:32.966 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:32.962+0000 7f0224800640 1 -- 172.21.15.19:0/3959729288 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f021c1ab3f0 con 0x7f021c1a2eb0 2024-03-18T15:28:32.967 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:32.963+0000 7f0224800640 1 -- 172.21.15.19:0/3959729288 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f021c04f540 con 0x7f021c1a2eb0 2024-03-18T15:28:32.967 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:32.966+0000 7f0216a00640 1 -- 172.21.15.19:0/3959729288 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 27) ==== 137421+0+0 (secure 0 0 0) 0x7f0218071bc0 con 0x7f021c1a2eb0 2024-03-18T15:28:32.968 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:32.967+0000 7f0216a00640 1 --2- 172.21.15.19:0/3959729288 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f02040a2a10 0x7f02040a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:28:32.968 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:32.967+0000 7f0216a00640 1 -- 172.21.15.19:0/3959729288 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9597+0+0 (secure 0 0 0) 0x7f02180f1780 con 0x7f021c1a2eb0 2024-03-18T15:28:32.968 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:32.967+0000 7f0221e00640 1 --2- 172.21.15.19:0/3959729288 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f02040a2a10 0x7f02040a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:28:32.969 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:32.968+0000 7f0221e00640 1 --2- 172.21.15.19:0/3959729288 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f02040a2a10 0x7f02040a4ed0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f0210005d00 tx=0x7f021000b410 comp rx=0 tx=0).ready entity=mgr.25466 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:28:32.976 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:32.975+0000 7f0216a00640 1 -- 172.21.15.19:0/3959729288 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f02180be1c0 con 0x7f021c1a2eb0 2024-03-18T15:28:33.203 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.202+0000 7f0224800640 1 -- 172.21.15.19:0/3959729288 --> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7f021c1089d0 con 0x7f02040a2a10 2024-03-18T15:28:33.218 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.217+0000 7f0216a00640 1 -- 172.21.15.19:0/3959729288 <== mgr.25466 v2:172.21.15.19:6800/3894323675 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+350 (secure 0 0 0) 0x7f021c1089d0 con 0x7f02040a2a10 2024-03-18T15:28:33.218 INFO:teuthology.orchestra.run.smithi019.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-03-18T00:00:00", "created": "2024-03-18T15:18:30", "first": "2024-03-18T15:19:00", "last": "2024-03-18T15:28:00", "last_pruned": "2024-03-18T15:28:01", "created_count": 10, "pruned_count": 4, "active": true} 2024-03-18T15:28:33.221 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.219+0000 7f0224800640 1 -- 172.21.15.19:0/3959729288 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f02040a2a10 msgr2=0x7f02040a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:28:33.221 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.219+0000 7f0224800640 1 --2- 172.21.15.19:0/3959729288 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f02040a2a10 0x7f02040a4ed0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f0210005d00 tx=0x7f021000b410 comp rx=0 tx=0).stop 2024-03-18T15:28:33.221 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.219+0000 7f0224800640 1 -- 172.21.15.19:0/3959729288 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f021c1a2eb0 msgr2=0x7f021c1a3360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:28:33.221 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.220+0000 7f0224800640 1 --2- 172.21.15.19:0/3959729288 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f021c1a2eb0 0x7f021c1a3360 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f0218009bf0 tx=0x7f0218035d10 comp rx=0 tx=0).stop 2024-03-18T15:28:33.221 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.220+0000 7f0224800640 1 -- 172.21.15.19:0/3959729288 shutdown_connections 2024-03-18T15:28:33.221 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.220+0000 7f0224800640 1 --2- 172.21.15.19:0/3959729288 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f02040a2a10 0x7f02040a4ed0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:28:33.221 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.220+0000 7f0224800640 1 --2- 172.21.15.19:0/3959729288 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f021c1a2eb0 0x7f021c1a3360 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:28:33.221 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.220+0000 7f0224800640 1 --2- 172.21.15.19:0/3959729288 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f021c10d0f0 0x7f021c19eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:28:33.221 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.220+0000 7f0224800640 1 --2- 172.21.15.19:0/3959729288 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f021c108020 0x7f021c19e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:28:33.221 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.220+0000 7f0224800640 1 -- 172.21.15.19:0/3959729288 >> 172.21.15.19:0/3959729288 conn(0x7f021c075d50 msgr2=0x7f021c076870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:28:33.221 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.220+0000 7f0224800640 1 -- 172.21.15.19:0/3959729288 shutdown_connections 2024-03-18T15:28:33.221 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.220+0000 7f0224800640 1 -- 172.21.15.19:0/3959729288 wait complete. 2024-03-18T15:28:33.243 DEBUG:teuthology.orchestra.run.smithi019:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-03-18T15:28:33.464 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.460+0000 7f793ea00640 1 -- 172.21.15.19:0/888550551 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f79381039c0 msgr2=0x7f7938103e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:28:33.465 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.460+0000 7f793ea00640 1 --2- 172.21.15.19:0/888550551 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f79381039c0 0x7f7938103e40 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f7924009930 tx=0x7f792402bc30 comp rx=0 tx=0).stop 2024-03-18T15:28:33.465 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.460+0000 7f793ea00640 1 -- 172.21.15.19:0/888550551 shutdown_connections 2024-03-18T15:28:33.465 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.460+0000 7f793ea00640 1 --2- 172.21.15.19:0/888550551 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7938104380 0x7f7938110890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:28:33.465 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.460+0000 7f793ea00640 1 --2- 172.21.15.19:0/888550551 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f79381039c0 0x7f7938103e40 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:28:33.465 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.460+0000 7f793ea00640 1 --2- 172.21.15.19:0/888550551 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f79381099d0 0x7f7938109db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:28:33.465 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.460+0000 7f793ea00640 1 -- 172.21.15.19:0/888550551 >> 172.21.15.19:0/888550551 conn(0x7f79380ff490 msgr2=0x7f79381018b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:28:33.465 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.460+0000 7f793ea00640 1 -- 172.21.15.19:0/888550551 shutdown_connections 2024-03-18T15:28:33.465 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.460+0000 7f793ea00640 1 -- 172.21.15.19:0/888550551 wait complete. 2024-03-18T15:28:33.466 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.461+0000 7f793ea00640 1 Processor -- start 2024-03-18T15:28:33.466 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.461+0000 7f793ea00640 1 -- start start 2024-03-18T15:28:33.466 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.461+0000 7f793ea00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f79381039c0 0x7f793819a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:28:33.466 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.461+0000 7f793ea00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7938104380 0x7f793819a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:28:33.466 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.461+0000 7f793ea00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f79381099d0 0x7f793819eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:28:33.466 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.461+0000 7f793ea00640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f7938113e30 con 0x7f79381099d0 2024-03-18T15:28:33.466 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.462+0000 7f793ea00640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7938113cb0 con 0x7f79381039c0 2024-03-18T15:28:33.466 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.462+0000 7f793ea00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f7938113fb0 con 0x7f7938104380 2024-03-18T15:28:33.466 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.462+0000 7f7937e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f79381039c0 0x7f793819a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:28:33.466 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.462+0000 7f7937e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f79381039c0 0x7f793819a1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:57248/0 (socket says 172.21.15.19:57248) 2024-03-18T15:28:33.466 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.462+0000 7f7937e00640 1 -- 172.21.15.19:0/1710373653 learned_addr learned my addr 172.21.15.19:0/1710373653 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:28:33.466 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.462+0000 7f7937400640 1 --2- 172.21.15.19:0/1710373653 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7938104380 0x7f793819a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:28:33.467 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.462+0000 7f793ca00640 1 --2- 172.21.15.19:0/1710373653 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f79381099d0 0x7f793819eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:28:33.467 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.462+0000 7f7937e00640 1 -- 172.21.15.19:0/1710373653 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7938104380 msgr2=0x7f793819a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:28:33.467 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.462+0000 7f7937e00640 1 --2- 172.21.15.19:0/1710373653 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7938104380 0x7f793819a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:28:33.467 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.462+0000 7f7937e00640 1 -- 172.21.15.19:0/1710373653 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f79381099d0 msgr2=0x7f793819eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:28:33.467 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.462+0000 7f7937e00640 1 --2- 172.21.15.19:0/1710373653 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f79381099d0 0x7f793819eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:28:33.467 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.462+0000 7f7937e00640 1 -- 172.21.15.19:0/1710373653 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f793819f1e0 con 0x7f79381039c0 2024-03-18T15:28:33.467 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.462+0000 7f7937e00640 1 --2- 172.21.15.19:0/1710373653 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f79381039c0 0x7f793819a1f0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f792800e9f0 tx=0x7f792800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:28:33.467 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.463+0000 7f7934c00640 1 -- 172.21.15.19:0/1710373653 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f792800cd80 con 0x7f79381039c0 2024-03-18T15:28:33.467 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.463+0000 7f793ea00640 1 -- 172.21.15.19:0/1710373653 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f793819f4d0 con 0x7f79381039c0 2024-03-18T15:28:33.468 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.463+0000 7f7934c00640 1 -- 172.21.15.19:0/1710373653 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7928010910 con 0x7f79381039c0 2024-03-18T15:28:33.468 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.463+0000 7f7934c00640 1 -- 172.21.15.19:0/1710373653 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7928018990 con 0x7f79381039c0 2024-03-18T15:28:33.468 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.463+0000 7f793ea00640 1 -- 172.21.15.19:0/1710373653 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79381a6d60 con 0x7f79381039c0 2024-03-18T15:28:33.469 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.464+0000 7f793ea00640 1 -- 172.21.15.19:0/1710373653 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f793804f540 con 0x7f79381039c0 2024-03-18T15:28:33.469 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.466+0000 7f7934c00640 1 -- 172.21.15.19:0/1710373653 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 27) ==== 137421+0+0 (secure 0 0 0) 0x7f7928010ac0 con 0x7f79381039c0 2024-03-18T15:28:33.469 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.467+0000 7f7934c00640 1 --2- 172.21.15.19:0/1710373653 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f791c0a2a90 0x7f791c0a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:28:33.469 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.467+0000 7f7934c00640 1 -- 172.21.15.19:0/1710373653 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9597+0+0 (secure 0 0 0) 0x7f79280cf500 con 0x7f79381039c0 2024-03-18T15:28:33.469 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.467+0000 7f7937400640 1 --2- 172.21.15.19:0/1710373653 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f791c0a2a90 0x7f791c0a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:28:33.469 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.468+0000 7f7937400640 1 --2- 172.21.15.19:0/1710373653 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f791c0a2a90 0x7f791c0a4f50 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f7924009900 tx=0x7f792402df90 comp rx=0 tx=0).ready entity=mgr.25466 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:28:33.472 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.471+0000 7f7934c00640 1 -- 172.21.15.19:0/1710373653 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f792809b780 con 0x7f79381039c0 2024-03-18T15:28:33.692 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.691+0000 7f793ea00640 1 -- 172.21.15.19:0/1710373653 --> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7f79381000c0 con 0x7f791c0a2a90 2024-03-18T15:28:33.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:28:33 smithi104 ceph-mon[36070]: pgmap v313: 129 pgs: 129 active+clean; 1.7 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:28:33.704 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.703+0000 7f7934c00640 1 -- 172.21.15.19:0/1710373653 <== mgr.25466 v2:172.21.15.19:6800/3894323675 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7f79381000c0 con 0x7f791c0a2a90 2024-03-18T15:28:33.704 INFO:teuthology.orchestra.run.smithi019.stdout:/ 1m 6m3h 2024-03-18T15:28:33.707 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.705+0000 7f793ea00640 1 -- 172.21.15.19:0/1710373653 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f791c0a2a90 msgr2=0x7f791c0a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:28:33.708 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.705+0000 7f793ea00640 1 --2- 172.21.15.19:0/1710373653 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f791c0a2a90 0x7f791c0a4f50 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f7924009900 tx=0x7f792402df90 comp rx=0 tx=0).stop 2024-03-18T15:28:33.708 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.705+0000 7f793ea00640 1 -- 172.21.15.19:0/1710373653 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f79381039c0 msgr2=0x7f793819a1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:28:33.708 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.705+0000 7f793ea00640 1 --2- 172.21.15.19:0/1710373653 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f79381039c0 0x7f793819a1f0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f792800e9f0 tx=0x7f792800eeb0 comp rx=0 tx=0).stop 2024-03-18T15:28:33.708 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.706+0000 7f793ea00640 1 -- 172.21.15.19:0/1710373653 shutdown_connections 2024-03-18T15:28:33.708 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.706+0000 7f793ea00640 1 --2- 172.21.15.19:0/1710373653 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f791c0a2a90 0x7f791c0a4f50 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:28:33.708 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.706+0000 7f793ea00640 1 --2- 172.21.15.19:0/1710373653 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f79381099d0 0x7f793819eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:28:33.709 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.706+0000 7f793ea00640 1 --2- 172.21.15.19:0/1710373653 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7938104380 0x7f793819a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:28:33.709 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.706+0000 7f793ea00640 1 --2- 172.21.15.19:0/1710373653 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f79381039c0 0x7f793819a1f0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:28:33.709 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.706+0000 7f793ea00640 1 -- 172.21.15.19:0/1710373653 >> 172.21.15.19:0/1710373653 conn(0x7f79380ff490 msgr2=0x7f79380ff9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:28:33.709 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.706+0000 7f793ea00640 1 -- 172.21.15.19:0/1710373653 shutdown_connections 2024-03-18T15:28:33.709 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.706+0000 7f793ea00640 1 -- 172.21.15.19:0/1710373653 wait complete. 2024-03-18T15:28:33.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:28:33 smithi019 ceph-mon[31522]: pgmap v313: 129 pgs: 129 active+clean; 1.7 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:28:33.725 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-03-18T15:28:33.737 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T15:28:33.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:28:33 smithi033 ceph-mon[36648]: pgmap v313: 129 pgs: 129 active+clean; 1.7 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:28:33.961 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.959+0000 7f59e6600640 1 -- 172.21.15.19:0/3871046701 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f59e0103050 msgr2=0x7f59e010b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:28:33.961 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.959+0000 7f59e6600640 1 --2- 172.21.15.19:0/3871046701 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f59e0103050 0x7f59e010b530 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f59d0009ca0 tx=0x7f59d002c190 comp rx=0 tx=0).stop 2024-03-18T15:28:33.961 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.959+0000 7f59e6600640 1 -- 172.21.15.19:0/3871046701 shutdown_connections 2024-03-18T15:28:33.961 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.960+0000 7f59e6600640 1 --2- 172.21.15.19:0/3871046701 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f59e010bb50 0x7f59e010dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:28:33.961 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.960+0000 7f59e6600640 1 --2- 172.21.15.19:0/3871046701 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f59e0103050 0x7f59e010b530 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:28:33.961 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.960+0000 7f59e6600640 1 --2- 172.21.15.19:0/3871046701 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f59e0102730 0x7f59e0102b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:28:33.961 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.960+0000 7f59e6600640 1 -- 172.21.15.19:0/3871046701 >> 172.21.15.19:0/3871046701 conn(0x7f59e00fbf70 msgr2=0x7f59e00fe390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:28:33.962 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.960+0000 7f59e6600640 1 -- 172.21.15.19:0/3871046701 shutdown_connections 2024-03-18T15:28:33.962 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.960+0000 7f59e6600640 1 -- 172.21.15.19:0/3871046701 wait complete. 2024-03-18T15:28:33.963 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.961+0000 7f59e6600640 1 Processor -- start 2024-03-18T15:28:33.963 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.961+0000 7f59e6600640 1 -- start start 2024-03-18T15:28:33.963 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.962+0000 7f59e6600640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f59e0102730 0x7f59e0100380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:28:33.963 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.962+0000 7f59e6600640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f59e0103050 0x7f59e01008c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:28:33.964 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.962+0000 7f59e6600640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f59e010bb50 0x7f59e0100e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:28:33.964 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.962+0000 7f59e6600640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f59e0111930 con 0x7f59e010bb50 2024-03-18T15:28:33.964 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.962+0000 7f59e6600640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f59e01117b0 con 0x7f59e0102730 2024-03-18T15:28:33.964 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.962+0000 7f59e6600640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f59e0111ab0 con 0x7f59e0103050 2024-03-18T15:28:33.964 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.962+0000 7f59e5200640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f59e0102730 0x7f59e0100380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:28:33.964 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.962+0000 7f59e5200640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f59e0102730 0x7f59e0100380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:57256/0 (socket says 172.21.15.19:57256) 2024-03-18T15:28:33.964 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.962+0000 7f59e5200640 1 -- 172.21.15.19:0/1018571867 learned_addr learned my addr 172.21.15.19:0/1018571867 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:28:33.964 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.962+0000 7f59dfe00640 1 --2- 172.21.15.19:0/1018571867 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f59e0103050 0x7f59e01008c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:28:33.964 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.962+0000 7f59e5c00640 1 --2- 172.21.15.19:0/1018571867 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f59e010bb50 0x7f59e0100e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:28:33.964 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.962+0000 7f59e5200640 1 -- 172.21.15.19:0/1018571867 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f59e0103050 msgr2=0x7f59e01008c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:28:33.964 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.962+0000 7f59e5200640 1 --2- 172.21.15.19:0/1018571867 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f59e0103050 0x7f59e01008c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:28:33.964 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.962+0000 7f59e5200640 1 -- 172.21.15.19:0/1018571867 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f59e010bb50 msgr2=0x7f59e0100e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:28:33.964 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.962+0000 7f59e5200640 1 --2- 172.21.15.19:0/1018571867 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f59e010bb50 0x7f59e0100e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:28:33.964 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.962+0000 7f59e5200640 1 -- 172.21.15.19:0/1018571867 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59e01a62f0 con 0x7f59e0102730 2024-03-18T15:28:33.965 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.963+0000 7f59e5200640 1 --2- 172.21.15.19:0/1018571867 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f59e0102730 0x7f59e0100380 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f59c800e9f0 tx=0x7f59c800edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:28:33.965 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.963+0000 7f59dd600640 1 -- 172.21.15.19:0/1018571867 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59c800cd80 con 0x7f59e0102730 2024-03-18T15:28:33.965 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.963+0000 7f59e6600640 1 -- 172.21.15.19:0/1018571867 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59e01a65e0 con 0x7f59e0102730 2024-03-18T15:28:33.965 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.963+0000 7f59dd600640 1 -- 172.21.15.19:0/1018571867 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f59c8005500 con 0x7f59e0102730 2024-03-18T15:28:33.966 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.963+0000 7f59dd600640 1 -- 172.21.15.19:0/1018571867 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59c8010690 con 0x7f59e0102730 2024-03-18T15:28:33.966 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.963+0000 7f59e6600640 1 -- 172.21.15.19:0/1018571867 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59e01a6ab0 con 0x7f59e0102730 2024-03-18T15:28:33.966 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.964+0000 7f59e6600640 1 -- 172.21.15.19:0/1018571867 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59e00684f0 con 0x7f59e0102730 2024-03-18T15:28:33.967 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.966+0000 7f59dd600640 1 -- 172.21.15.19:0/1018571867 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 27) ==== 137421+0+0 (secure 0 0 0) 0x7f59c80046c0 con 0x7f59e0102730 2024-03-18T15:28:33.968 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.968+0000 7f59dd600640 1 --2- 172.21.15.19:0/1018571867 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f59c40a29a0 0x7f59c40a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:28:33.969 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.968+0000 7f59dd600640 1 -- 172.21.15.19:0/1018571867 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9597+0+0 (secure 0 0 0) 0x7f59c80cf750 con 0x7f59e0102730 2024-03-18T15:28:33.969 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.968+0000 7f59dfe00640 1 --2- 172.21.15.19:0/1018571867 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f59c40a29a0 0x7f59c40a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:28:33.969 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.969+0000 7f59dfe00640 1 --2- 172.21.15.19:0/1018571867 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f59c40a29a0 0x7f59c40a4e60 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f59d0004af0 tx=0x7f59d0005c50 comp rx=0 tx=0).ready entity=mgr.25466 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:28:33.974 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:33.974+0000 7f59dd600640 1 -- 172.21.15.19:0/1018571867 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f59c809c020 con 0x7f59e0102730 2024-03-18T15:28:34.180 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.179+0000 7f59e6600640 1 -- 172.21.15.19:0/1018571867 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f59e0106c20 con 0x7f59e0102730 2024-03-18T15:28:34.182 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.181+0000 7f59dd600640 1 -- 172.21.15.19:0/1018571867 <== mon.2 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v103) ==== 74+0+25433 (secure 0 0 0) 0x7f59c809b920 con 0x7f59e0102730 2024-03-18T15:28:34.183 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-18T15:28:34.183 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":103,"fsid":"d4dc2ba6-e538-11ee-95c9-87774f69a715","created":"2024-03-18T15:05:57.538551+0000","modified":"2024-03-18T15:28:17.267028+0000","last_up_change":"2024-03-18T15:14:12.009522+0000","last_in_change":"2024-03-18T15:13:52.555387+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T15:09:56.843349+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T15:15:08.941047+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T15:15:10.044671+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"102","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":102,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T15:15:13.621916+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"102","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":102,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"37b5a5cd-27ae-4e77-9b05-774b817b2ee9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6802","nonce":402333948},{"type":"v1","addr":"172.21.15.19:6803","nonce":402333948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6804","nonce":402333948},{"type":"v1","addr":"172.21.15.19:6805","nonce":402333948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6808","nonce":402333948},{"type":"v1","addr":"172.21.15.19:6809","nonce":402333948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6806","nonce":402333948},{"type":"v1","addr":"172.21.15.19:6807","nonce":402333948}]},"public_addr":"172.21.15.19:6803/402333948","cluster_addr":"172.21.15.19:6805/402333948","heartbeat_back_addr":"172.21.15.19:6809/402333948","heartbeat_front_addr":"172.21.15.19:6807/402333948","state":["exists","up"]},{"osd":1,"uuid":"3a1134e8-ede7-4dbd-9be1-eefd2cbb43bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1951880113},{"type":"v1","addr":"172.21.15.33:6801","nonce":1951880113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1951880113},{"type":"v1","addr":"172.21.15.33:6803","nonce":1951880113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1951880113},{"type":"v1","addr":"172.21.15.33:6807","nonce":1951880113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1951880113},{"type":"v1","addr":"172.21.15.33:6805","nonce":1951880113}]},"public_addr":"172.21.15.33:6801/1951880113","cluster_addr":"172.21.15.33:6803/1951880113","heartbeat_back_addr":"172.21.15.33:6807/1951880113","heartbeat_front_addr":"172.21.15.33:6805/1951880113","state":["exists","up"]},{"osd":2,"uuid":"f383a164-a3a0-49fd-a9e8-9c2178d8cfb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":1004727949},{"type":"v1","addr":"172.21.15.104:6801","nonce":1004727949}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":1004727949},{"type":"v1","addr":"172.21.15.104:6803","nonce":1004727949}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":1004727949},{"type":"v1","addr":"172.21.15.104:6807","nonce":1004727949}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":1004727949},{"type":"v1","addr":"172.21.15.104:6805","nonce":1004727949}]},"public_addr":"172.21.15.104:6801/1004727949","cluster_addr":"172.21.15.104:6803/1004727949","heartbeat_back_addr":"172.21.15.104:6807/1004727949","heartbeat_front_addr":"172.21.15.104:6805/1004727949","state":["exists","up"]},{"osd":3,"uuid":"c409e116-28bf-45dd-a791-539af22abe1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6810","nonce":3571525854},{"type":"v1","addr":"172.21.15.19:6811","nonce":3571525854}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6812","nonce":3571525854},{"type":"v1","addr":"172.21.15.19:6813","nonce":3571525854}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6816","nonce":3571525854},{"type":"v1","addr":"172.21.15.19:6817","nonce":3571525854}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6814","nonce":3571525854},{"type":"v1","addr":"172.21.15.19:6815","nonce":3571525854}]},"public_addr":"172.21.15.19:6811/3571525854","cluster_addr":"172.21.15.19:6813/3571525854","heartbeat_back_addr":"172.21.15.19:6817/3571525854","heartbeat_front_addr":"172.21.15.19:6815/3571525854","state":["exists","up"]},{"osd":4,"uuid":"696eb719-0b3e-43ae-b5f1-ccd73a3b2843","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":624901749},{"type":"v1","addr":"172.21.15.33:6809","nonce":624901749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":624901749},{"type":"v1","addr":"172.21.15.33:6811","nonce":624901749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":624901749},{"type":"v1","addr":"172.21.15.33:6815","nonce":624901749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":624901749},{"type":"v1","addr":"172.21.15.33:6813","nonce":624901749}]},"public_addr":"172.21.15.33:6809/624901749","cluster_addr":"172.21.15.33:6811/624901749","heartbeat_back_addr":"172.21.15.33:6815/624901749","heartbeat_front_addr":"172.21.15.33:6813/624901749","state":["exists","up"]},{"osd":5,"uuid":"21789779-8bb9-42c7-b9ee-437985f11e3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":1258208297},{"type":"v1","addr":"172.21.15.104:6809","nonce":1258208297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":1258208297},{"type":"v1","addr":"172.21.15.104:6811","nonce":1258208297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":1258208297},{"type":"v1","addr":"172.21.15.104:6815","nonce":1258208297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":1258208297},{"type":"v1","addr":"172.21.15.104:6813","nonce":1258208297}]},"public_addr":"172.21.15.104:6809/1258208297","cluster_addr":"172.21.15.104:6811/1258208297","heartbeat_back_addr":"172.21.15.104:6815/1258208297","heartbeat_front_addr":"172.21.15.104:6813/1258208297","state":["exists","up"]},{"osd":6,"uuid":"0e92a2ef-43cd-487c-9203-e219158aef8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6818","nonce":2782482301},{"type":"v1","addr":"172.21.15.19:6819","nonce":2782482301}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6820","nonce":2782482301},{"type":"v1","addr":"172.21.15.19:6821","nonce":2782482301}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6824","nonce":2782482301},{"type":"v1","addr":"172.21.15.19:6825","nonce":2782482301}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6822","nonce":2782482301},{"type":"v1","addr":"172.21.15.19:6823","nonce":2782482301}]},"public_addr":"172.21.15.19:6819/2782482301","cluster_addr":"172.21.15.19:6821/2782482301","heartbeat_back_addr":"172.21.15.19:6825/2782482301","heartbeat_front_addr":"172.21.15.19:6823/2782482301","state":["exists","up"]},{"osd":7,"uuid":"a16e39a1-3685-491d-b376-3e80a2084072","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":4238160988},{"type":"v1","addr":"172.21.15.33:6817","nonce":4238160988}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":4238160988},{"type":"v1","addr":"172.21.15.33:6819","nonce":4238160988}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":4238160988},{"type":"v1","addr":"172.21.15.33:6823","nonce":4238160988}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":4238160988},{"type":"v1","addr":"172.21.15.33:6821","nonce":4238160988}]},"public_addr":"172.21.15.33:6817/4238160988","cluster_addr":"172.21.15.33:6819/4238160988","heartbeat_back_addr":"172.21.15.33:6823/4238160988","heartbeat_front_addr":"172.21.15.33:6821/4238160988","state":["exists","up"]},{"osd":8,"uuid":"d5c3fa92-cd5d-4349-9c7d-d5666bacde16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":1014779813},{"type":"v1","addr":"172.21.15.104:6817","nonce":1014779813}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":1014779813},{"type":"v1","addr":"172.21.15.104:6819","nonce":1014779813}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":1014779813},{"type":"v1","addr":"172.21.15.104:6823","nonce":1014779813}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":1014779813},{"type":"v1","addr":"172.21.15.104:6821","nonce":1014779813}]},"public_addr":"172.21.15.104:6817/1014779813","cluster_addr":"172.21.15.104:6819/1014779813","heartbeat_back_addr":"172.21.15.104:6823/1014779813","heartbeat_front_addr":"172.21.15.104:6821/1014779813","state":["exists","up"]},{"osd":9,"uuid":"eed2e9b2-26d3-4c46-88ec-34b5596fe991","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6826","nonce":2641151009},{"type":"v1","addr":"172.21.15.19:6827","nonce":2641151009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6828","nonce":2641151009},{"type":"v1","addr":"172.21.15.19:6829","nonce":2641151009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6832","nonce":2641151009},{"type":"v1","addr":"172.21.15.19:6833","nonce":2641151009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6830","nonce":2641151009},{"type":"v1","addr":"172.21.15.19:6831","nonce":2641151009}]},"public_addr":"172.21.15.19:6827/2641151009","cluster_addr":"172.21.15.19:6829/2641151009","heartbeat_back_addr":"172.21.15.19:6833/2641151009","heartbeat_front_addr":"172.21.15.19:6831/2641151009","state":["exists","up"]},{"osd":10,"uuid":"f3c35ea5-fbe7-4dc4-93c5-b92d1a714db3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":1346058869},{"type":"v1","addr":"172.21.15.33:6825","nonce":1346058869}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":1346058869},{"type":"v1","addr":"172.21.15.33:6827","nonce":1346058869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":1346058869},{"type":"v1","addr":"172.21.15.33:6831","nonce":1346058869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":1346058869},{"type":"v1","addr":"172.21.15.33:6829","nonce":1346058869}]},"public_addr":"172.21.15.33:6825/1346058869","cluster_addr":"172.21.15.33:6827/1346058869","heartbeat_back_addr":"172.21.15.33:6831/1346058869","heartbeat_front_addr":"172.21.15.33:6829/1346058869","state":["exists","up"]},{"osd":11,"uuid":"c8b51571-f807-4b62-90b6-7529e09e59da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":3759498281},{"type":"v1","addr":"172.21.15.104:6825","nonce":3759498281}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":3759498281},{"type":"v1","addr":"172.21.15.104:6827","nonce":3759498281}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":3759498281},{"type":"v1","addr":"172.21.15.104:6831","nonce":3759498281}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":3759498281},{"type":"v1","addr":"172.21.15.104:6829","nonce":3759498281}]},"public_addr":"172.21.15.104:6825/3759498281","cluster_addr":"172.21.15.104:6827/3759498281","heartbeat_back_addr":"172.21.15.104:6831/3759498281","heartbeat_front_addr":"172.21.15.104:6829/3759498281","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:09:05.267795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:09:30.234068+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:09:53.870232+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:10:26.471296+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:10:53.180341+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:11:17.801361+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:11:51.364816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:12:18.306408+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:12:43.978744+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:13:18.213964+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:13:45.437544+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:14:10.148580+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":3,"to":6}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"3.37","mappings":[{"from":5,"to":8}]},{"pgid":"3.3a","mappings":[{"from":4,"to":7}]},{"pgid":"4.8","mappings":[{"from":5,"to":8}]},{"pgid":"4.f","mappings":[{"from":9,"to":10}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.19:0/1895666842":"2024-03-18T16:23:20.095509+0000","172.21.15.19:6801/3264337803":"2024-03-19T15:06:38.688405+0000","172.21.15.19:6800/3264337803":"2024-03-19T15:06:38.688405+0000","172.21.15.19:0/2793980440":"2024-03-19T15:06:38.688405+0000","172.21.15.19:6801/2776008962":"2024-03-19T15:07:14.150983+0000","172.21.15.19:0/2316969005":"2024-03-19T15:07:14.150983+0000","172.21.15.19:0/84203799":"2024-03-19T15:18:29.726021+0000","172.21.15.19:6800/2017714236":"2024-03-19T15:18:29.726021+0000","172.21.15.19:0/2737223213":"2024-03-19T15:07:14.150983+0000","172.21.15.19:0/3860783274":"2024-03-19T15:07:14.150983+0000","172.21.15.19:6800/2776008962":"2024-03-19T15:07:14.150983+0000","172.21.15.19:0/1024556841":"2024-03-19T15:18:29.726021+0000","172.21.15.19:0/3166705295":"2024-03-19T15:06:38.688405+0000","172.21.15.19:6801/2017714236":"2024-03-19T15:18:29.726021+0000","172.21.15.19:0/176699025":"2024-03-19T15:06:38.688405+0000","172.21.15.19:0/2730531676":"2024-03-19T15:18:29.726021+0000","172.21.15.19:0/2353244552":"2024-03-19T15:18:29.726021+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]},{"pool":4,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T15:28:34.185 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.184+0000 7f59e6600640 1 -- 172.21.15.19:0/1018571867 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f59c40a29a0 msgr2=0x7f59c40a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:28:34.185 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.184+0000 7f59e6600640 1 --2- 172.21.15.19:0/1018571867 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f59c40a29a0 0x7f59c40a4e60 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f59d0004af0 tx=0x7f59d0005c50 comp rx=0 tx=0).stop 2024-03-18T15:28:34.185 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.184+0000 7f59e6600640 1 -- 172.21.15.19:0/1018571867 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f59e0102730 msgr2=0x7f59e0100380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:28:34.185 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.184+0000 7f59e6600640 1 --2- 172.21.15.19:0/1018571867 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f59e0102730 0x7f59e0100380 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f59c800e9f0 tx=0x7f59c800edf0 comp rx=0 tx=0).stop 2024-03-18T15:28:34.185 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.184+0000 7f59e6600640 1 -- 172.21.15.19:0/1018571867 shutdown_connections 2024-03-18T15:28:34.185 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.184+0000 7f59e6600640 1 --2- 172.21.15.19:0/1018571867 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f59c40a29a0 0x7f59c40a4e60 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:28:34.185 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.184+0000 7f59e6600640 1 --2- 172.21.15.19:0/1018571867 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f59e010bb50 0x7f59e0100e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:28:34.185 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.184+0000 7f59e6600640 1 --2- 172.21.15.19:0/1018571867 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f59e0103050 0x7f59e01008c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:28:34.185 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.184+0000 7f59e6600640 1 --2- 172.21.15.19:0/1018571867 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f59e0102730 0x7f59e0100380 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:28:34.185 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.184+0000 7f59e6600640 1 -- 172.21.15.19:0/1018571867 >> 172.21.15.19:0/1018571867 conn(0x7f59e00fbf70 msgr2=0x7f59e00fcc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:28:34.186 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.184+0000 7f59e6600640 1 -- 172.21.15.19:0/1018571867 shutdown_connections 2024-03-18T15:28:34.186 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.184+0000 7f59e6600640 1 -- 172.21.15.19:0/1018571867 wait complete. 2024-03-18T15:28:34.206 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T15:09:56.843349+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T15:15:08.941047+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T15:15:10.044671+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '102', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 15, 'snap_epoch': 102, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-18T15:15:13.621916+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '102', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 15, 'snap_epoch': 102, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-18T15:28:34.207 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T15:28:34.442 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.441+0000 7f4143e00640 1 -- 172.21.15.19:0/2493306899 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4144107f90 msgr2=0x7f4144108370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:28:34.442 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.441+0000 7f4143e00640 1 --2- 172.21.15.19:0/2493306899 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4144107f90 0x7f4144108370 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f4134009900 tx=0x7f413402bc40 comp rx=0 tx=0).stop 2024-03-18T15:28:34.442 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.441+0000 7f4143e00640 1 -- 172.21.15.19:0/2493306899 shutdown_connections 2024-03-18T15:28:34.442 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.441+0000 7f4143e00640 1 --2- 172.21.15.19:0/2493306899 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f414410cfe0 0x7f4144114bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:28:34.443 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.441+0000 7f4143e00640 1 --2- 172.21.15.19:0/2493306899 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4144108940 0x7f414410c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:28:34.443 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.441+0000 7f4143e00640 1 --2- 172.21.15.19:0/2493306899 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4144107f90 0x7f4144108370 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:28:34.443 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.441+0000 7f4143e00640 1 -- 172.21.15.19:0/2493306899 >> 172.21.15.19:0/2493306899 conn(0x7f4144077600 msgr2=0x7f4144106600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:28:34.443 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.441+0000 7f4143e00640 1 -- 172.21.15.19:0/2493306899 shutdown_connections 2024-03-18T15:28:34.443 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.441+0000 7f4143e00640 1 -- 172.21.15.19:0/2493306899 wait complete. 2024-03-18T15:28:34.444 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.442+0000 7f4143e00640 1 Processor -- start 2024-03-18T15:28:34.444 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.442+0000 7f4143e00640 1 -- start start 2024-03-18T15:28:34.444 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.443+0000 7f4143e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4144107f90 0x7f414419e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:28:34.445 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.443+0000 7f4143e00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f414410cfe0 0x7f414419eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:28:34.445 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.443+0000 7f4143e00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f41441a2eb0 0x7f41441a3360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:28:34.445 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.443+0000 7f4143e00640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f41441180f0 con 0x7f41441a2eb0 2024-03-18T15:28:34.445 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.443+0000 7f4143e00640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4144117f70 con 0x7f4144107f90 2024-03-18T15:28:34.445 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.443+0000 7f4143e00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f4144118270 con 0x7f414410cfe0 2024-03-18T15:28:34.445 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.443+0000 7f4142a00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4144107f90 0x7f414419e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:28:34.445 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.443+0000 7f4142a00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4144107f90 0x7f414419e5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:57270/0 (socket says 172.21.15.19:57270) 2024-03-18T15:28:34.445 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.443+0000 7f413be00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f414410cfe0 0x7f414419eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:28:34.445 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.443+0000 7f4142a00640 1 -- 172.21.15.19:0/3040652893 learned_addr learned my addr 172.21.15.19:0/3040652893 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:28:34.445 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.443+0000 7f4143400640 1 --2- 172.21.15.19:0/3040652893 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f41441a2eb0 0x7f41441a3360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:28:34.445 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.443+0000 7f4142a00640 1 -- 172.21.15.19:0/3040652893 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f414410cfe0 msgr2=0x7f414419eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:28:34.445 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.443+0000 7f4142a00640 1 --2- 172.21.15.19:0/3040652893 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f414410cfe0 0x7f414419eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:28:34.445 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.443+0000 7f4142a00640 1 -- 172.21.15.19:0/3040652893 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f41441a2eb0 msgr2=0x7f41441a3360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:28:34.445 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.443+0000 7f4142a00640 1 --2- 172.21.15.19:0/3040652893 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f41441a2eb0 0x7f41441a3360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:28:34.446 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.443+0000 7f4142a00640 1 -- 172.21.15.19:0/3040652893 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41441a39a0 con 0x7f4144107f90 2024-03-18T15:28:34.446 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.444+0000 7f4142a00640 1 --2- 172.21.15.19:0/3040652893 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4144107f90 0x7f414419e5f0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f4134005ab0 tx=0x7f4134004360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:28:34.447 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.446+0000 7f413b400640 1 -- 172.21.15.19:0/3040652893 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4134041070 con 0x7f4144107f90 2024-03-18T15:28:34.447 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.446+0000 7f413b400640 1 -- 172.21.15.19:0/3040652893 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f413402cca0 con 0x7f4144107f90 2024-03-18T15:28:34.447 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.446+0000 7f413b400640 1 -- 172.21.15.19:0/3040652893 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4134003e50 con 0x7f4144107f90 2024-03-18T15:28:34.447 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.446+0000 7f4143e00640 1 -- 172.21.15.19:0/3040652893 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41441a3c30 con 0x7f4144107f90 2024-03-18T15:28:34.447 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.446+0000 7f4143e00640 1 -- 172.21.15.19:0/3040652893 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41441ab3f0 con 0x7f4144107f90 2024-03-18T15:28:34.449 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.447+0000 7f4143e00640 1 -- 172.21.15.19:0/3040652893 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4144068940 con 0x7f4144107f90 2024-03-18T15:28:34.450 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.449+0000 7f413b400640 1 -- 172.21.15.19:0/3040652893 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 27) ==== 137421+0+0 (secure 0 0 0) 0x7f4134044070 con 0x7f4144107f90 2024-03-18T15:28:34.451 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.450+0000 7f413b400640 1 --2- 172.21.15.19:0/3040652893 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f41280a2b90 0x7f41280a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:28:34.452 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.450+0000 7f413b400640 1 -- 172.21.15.19:0/3040652893 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9597+0+0 (secure 0 0 0) 0x7f41340f0c40 con 0x7f4144107f90 2024-03-18T15:28:34.452 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.451+0000 7f413be00640 1 --2- 172.21.15.19:0/3040652893 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f41280a2b90 0x7f41280a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:28:34.452 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.451+0000 7f413be00640 1 --2- 172.21.15.19:0/3040652893 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f41280a2b90 0x7f41280a5050 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f41440785b0 tx=0x7f412c005f90 comp rx=0 tx=0).ready entity=mgr.25466 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:28:34.456 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.454+0000 7f413b400640 1 -- 172.21.15.19:0/3040652893 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f41340bd620 con 0x7f4144107f90 2024-03-18T15:28:34.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:28:34 smithi019 ceph-mon[31522]: from='client.25553 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:28:34.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:28:34 smithi019 ceph-mon[31522]: from='client.25559 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:28:34.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:28:34 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/1018571867' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:28:34.657 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.656+0000 7f4143e00640 1 -- 172.21.15.19:0/3040652893 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f4144108940 con 0x7f4144107f90 2024-03-18T15:28:34.657 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.657+0000 7f413b400640 1 -- 172.21.15.19:0/3040652893 <== mon.2 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v103) ==== 93+0+10 (secure 0 0 0) 0x7f41340bcf20 con 0x7f4144107f90 2024-03-18T15:28:34.658 INFO:teuthology.orchestra.run.smithi019.stdout:pg_num: 1 2024-03-18T15:28:34.660 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.659+0000 7f4143e00640 1 -- 172.21.15.19:0/3040652893 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f41280a2b90 msgr2=0x7f41280a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:28:34.661 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.659+0000 7f4143e00640 1 --2- 172.21.15.19:0/3040652893 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f41280a2b90 0x7f41280a5050 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f41440785b0 tx=0x7f412c005f90 comp rx=0 tx=0).stop 2024-03-18T15:28:34.661 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.659+0000 7f4143e00640 1 -- 172.21.15.19:0/3040652893 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4144107f90 msgr2=0x7f414419e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:28:34.661 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.659+0000 7f4143e00640 1 --2- 172.21.15.19:0/3040652893 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4144107f90 0x7f414419e5f0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f4134005ab0 tx=0x7f4134004360 comp rx=0 tx=0).stop 2024-03-18T15:28:34.661 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.659+0000 7f4143e00640 1 -- 172.21.15.19:0/3040652893 shutdown_connections 2024-03-18T15:28:34.661 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.659+0000 7f4143e00640 1 --2- 172.21.15.19:0/3040652893 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f41280a2b90 0x7f41280a5050 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:28:34.661 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.659+0000 7f4143e00640 1 --2- 172.21.15.19:0/3040652893 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f41441a2eb0 0x7f41441a3360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:28:34.661 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.659+0000 7f4143e00640 1 --2- 172.21.15.19:0/3040652893 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f414410cfe0 0x7f414419eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:28:34.661 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.659+0000 7f4143e00640 1 --2- 172.21.15.19:0/3040652893 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4144107f90 0x7f414419e5f0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:28:34.661 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.659+0000 7f4143e00640 1 -- 172.21.15.19:0/3040652893 >> 172.21.15.19:0/3040652893 conn(0x7f4144077600 msgr2=0x7f4144079230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:28:34.661 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.659+0000 7f4143e00640 1 -- 172.21.15.19:0/3040652893 shutdown_connections 2024-03-18T15:28:34.661 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.660+0000 7f4143e00640 1 -- 172.21.15.19:0/3040652893 wait complete. 2024-03-18T15:28:34.680 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T15:28:34.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:28:34 smithi104 ceph-mon[36070]: from='client.25553 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:28:34.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:28:34 smithi104 ceph-mon[36070]: from='client.25559 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:28:34.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:28:34 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/1018571867' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:28:34.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:28:34 smithi033 ceph-mon[36648]: from='client.25553 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:28:34.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:28:34 smithi033 ceph-mon[36648]: from='client.25559 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:28:34.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:28:34 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/1018571867' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:28:34.910 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.909+0000 7f1b3ea00640 1 -- 172.21.15.19:0/2652415311 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1b3810dfb0 msgr2=0x7f1b38110400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:28:34.910 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.909+0000 7f1b3ea00640 1 --2- 172.21.15.19:0/2652415311 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1b3810dfb0 0x7f1b38110400 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f1b30009900 tx=0x7f1b3002bcc0 comp rx=0 tx=0).stop 2024-03-18T15:28:34.911 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.909+0000 7f1b3ea00640 1 -- 172.21.15.19:0/2652415311 shutdown_connections 2024-03-18T15:28:34.911 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.909+0000 7f1b3ea00640 1 --2- 172.21.15.19:0/2652415311 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1b3810dfb0 0x7f1b38110400 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:28:34.911 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.909+0000 7f1b3ea00640 1 --2- 172.21.15.19:0/2652415311 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1b38074530 0x7f1b3810da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:28:34.911 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.909+0000 7f1b3ea00640 1 --2- 172.21.15.19:0/2652415311 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1b38073c10 0x7f1b38073ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:28:34.911 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.909+0000 7f1b3ea00640 1 -- 172.21.15.19:0/2652415311 >> 172.21.15.19:0/2652415311 conn(0x7f1b38078be0 msgr2=0x7f1b38078ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:28:34.911 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.909+0000 7f1b3ea00640 1 -- 172.21.15.19:0/2652415311 shutdown_connections 2024-03-18T15:28:34.911 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.909+0000 7f1b3ea00640 1 -- 172.21.15.19:0/2652415311 wait complete. 2024-03-18T15:28:34.912 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.910+0000 7f1b3ea00640 1 Processor -- start 2024-03-18T15:28:34.912 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.910+0000 7f1b3ea00640 1 -- start start 2024-03-18T15:28:34.912 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.911+0000 7f1b3ea00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1b38073c10 0x7f1b3819a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:28:34.912 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.911+0000 7f1b3ea00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1b38074530 0x7f1b3819a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:28:34.913 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.911+0000 7f1b3ea00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1b3810dfb0 0x7f1b3819eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:28:34.913 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.911+0000 7f1b3ea00640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f1b38113da0 con 0x7f1b38074530 2024-03-18T15:28:34.913 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.911+0000 7f1b3ea00640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1b38113c20 con 0x7f1b3810dfb0 2024-03-18T15:28:34.913 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.911+0000 7f1b3ea00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f1b38113f20 con 0x7f1b38073c10 2024-03-18T15:28:34.913 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.911+0000 7f1b3ca00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1b3810dfb0 0x7f1b3819eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:28:34.913 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.911+0000 7f1b37400640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1b38074530 0x7f1b3819a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:28:34.913 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.911+0000 7f1b3ca00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1b3810dfb0 0x7f1b3819eac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:57280/0 (socket says 172.21.15.19:57280) 2024-03-18T15:28:34.913 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.911+0000 7f1b3ca00640 1 -- 172.21.15.19:0/2473865982 learned_addr learned my addr 172.21.15.19:0/2473865982 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:28:34.913 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.911+0000 7f1b37e00640 1 --2- 172.21.15.19:0/2473865982 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1b38073c10 0x7f1b3819a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:28:34.913 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.912+0000 7f1b3ca00640 1 -- 172.21.15.19:0/2473865982 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1b38073c10 msgr2=0x7f1b3819a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:28:34.913 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.912+0000 7f1b3ca00640 1 --2- 172.21.15.19:0/2473865982 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1b38073c10 0x7f1b3819a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:28:34.913 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.912+0000 7f1b3ca00640 1 -- 172.21.15.19:0/2473865982 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1b38074530 msgr2=0x7f1b3819a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:28:34.913 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.912+0000 7f1b3ca00640 1 --2- 172.21.15.19:0/2473865982 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1b38074530 0x7f1b3819a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:28:34.913 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.912+0000 7f1b3ca00640 1 -- 172.21.15.19:0/2473865982 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b3819f1f0 con 0x7f1b3810dfb0 2024-03-18T15:28:34.915 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.912+0000 7f1b3ca00640 1 --2- 172.21.15.19:0/2473865982 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1b3810dfb0 0x7f1b3819eac0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f1b3002c200 tx=0x7f1b3002c870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:28:34.915 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.912+0000 7f1b34c00640 1 -- 172.21.15.19:0/2473865982 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b30042020 con 0x7f1b3810dfb0 2024-03-18T15:28:34.915 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.912+0000 7f1b34c00640 1 -- 172.21.15.19:0/2473865982 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1b300044c0 con 0x7f1b3810dfb0 2024-03-18T15:28:34.915 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.912+0000 7f1b3ea00640 1 -- 172.21.15.19:0/2473865982 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b3819f480 con 0x7f1b3810dfb0 2024-03-18T15:28:34.915 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.912+0000 7f1b34c00640 1 -- 172.21.15.19:0/2473865982 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b30003b20 con 0x7f1b3810dfb0 2024-03-18T15:28:34.915 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.912+0000 7f1b3ea00640 1 -- 172.21.15.19:0/2473865982 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b381a6d70 con 0x7f1b3810dfb0 2024-03-18T15:28:34.915 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.913+0000 7f1b3ea00640 1 -- 172.21.15.19:0/2473865982 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b38068a20 con 0x7f1b3810dfb0 2024-03-18T15:28:34.916 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.915+0000 7f1b34c00640 1 -- 172.21.15.19:0/2473865982 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 27) ==== 137421+0+0 (secure 0 0 0) 0x7f1b30045020 con 0x7f1b3810dfb0 2024-03-18T15:28:34.917 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.917+0000 7f1b34c00640 1 --2- 172.21.15.19:0/2473865982 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f1b1c0a29f0 0x7f1b1c0a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:28:34.918 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.917+0000 7f1b34c00640 1 -- 172.21.15.19:0/2473865982 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9597+0+0 (secure 0 0 0) 0x7f1b300f0cc0 con 0x7f1b3810dfb0 2024-03-18T15:28:34.918 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.917+0000 7f1b37e00640 1 --2- 172.21.15.19:0/2473865982 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f1b1c0a29f0 0x7f1b1c0a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:28:34.918 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.918+0000 7f1b37e00640 1 --2- 172.21.15.19:0/2473865982 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f1b1c0a29f0 0x7f1b1c0a4eb0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f1b28005fd0 tx=0x7f1b28009500 comp rx=0 tx=0).ready entity=mgr.25466 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:28:34.922 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:34.921+0000 7f1b34c00640 1 -- 172.21.15.19:0/2473865982 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1b300f7080 con 0x7f1b3810dfb0 2024-03-18T15:28:35.126 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.125+0000 7f1b3ea00640 1 -- 172.21.15.19:0/2473865982 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f1b38109130 con 0x7f1b3810dfb0 2024-03-18T15:28:35.126 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.126+0000 7f1b34c00640 1 -- 172.21.15.19:0/2473865982 <== mon.2 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v103) ==== 104+0+11 (secure 0 0 0) 0x7f1b3004c020 con 0x7f1b3810dfb0 2024-03-18T15:28:35.127 INFO:teuthology.orchestra.run.smithi019.stdout:pg_num: 32 2024-03-18T15:28:35.130 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.128+0000 7f1b3ea00640 1 -- 172.21.15.19:0/2473865982 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f1b1c0a29f0 msgr2=0x7f1b1c0a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:28:35.130 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.128+0000 7f1b3ea00640 1 --2- 172.21.15.19:0/2473865982 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f1b1c0a29f0 0x7f1b1c0a4eb0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f1b28005fd0 tx=0x7f1b28009500 comp rx=0 tx=0).stop 2024-03-18T15:28:35.130 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.128+0000 7f1b3ea00640 1 -- 172.21.15.19:0/2473865982 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1b3810dfb0 msgr2=0x7f1b3819eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:28:35.130 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.128+0000 7f1b3ea00640 1 --2- 172.21.15.19:0/2473865982 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1b3810dfb0 0x7f1b3819eac0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f1b3002c200 tx=0x7f1b3002c870 comp rx=0 tx=0).stop 2024-03-18T15:28:35.130 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.129+0000 7f1b3ea00640 1 -- 172.21.15.19:0/2473865982 shutdown_connections 2024-03-18T15:28:35.130 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.129+0000 7f1b3ea00640 1 --2- 172.21.15.19:0/2473865982 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f1b1c0a29f0 0x7f1b1c0a4eb0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:28:35.130 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.129+0000 7f1b3ea00640 1 --2- 172.21.15.19:0/2473865982 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1b3810dfb0 0x7f1b3819eac0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:28:35.130 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.129+0000 7f1b3ea00640 1 --2- 172.21.15.19:0/2473865982 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1b38074530 0x7f1b3819a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:28:35.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.129+0000 7f1b3ea00640 1 --2- 172.21.15.19:0/2473865982 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1b38073c10 0x7f1b3819a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:28:35.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.129+0000 7f1b3ea00640 1 -- 172.21.15.19:0/2473865982 >> 172.21.15.19:0/2473865982 conn(0x7f1b38078be0 msgr2=0x7f1b380fe660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:28:35.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.129+0000 7f1b3ea00640 1 -- 172.21.15.19:0/2473865982 shutdown_connections 2024-03-18T15:28:35.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.129+0000 7f1b3ea00640 1 -- 172.21.15.19:0/2473865982 wait complete. 2024-03-18T15:28:35.150 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T15:28:35.372 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.370+0000 7f1277200640 1 -- 172.21.15.19:0/2749059660 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1278102810 msgr2=0x7f1278102bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:28:35.372 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.370+0000 7f1277200640 1 --2- 172.21.15.19:0/2749059660 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1278102810 0x7f1278102bf0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f1268009900 tx=0x7f126802bcd0 comp rx=0 tx=0).stop 2024-03-18T15:28:35.372 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.371+0000 7f1277200640 1 -- 172.21.15.19:0/2749059660 shutdown_connections 2024-03-18T15:28:35.372 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.371+0000 7f1277200640 1 --2- 172.21.15.19:0/2749059660 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f127810bc30 0x7f127810e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:28:35.372 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.371+0000 7f1277200640 1 --2- 172.21.15.19:0/2749059660 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1278103130 0x7f127810b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:28:35.372 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.371+0000 7f1277200640 1 --2- 172.21.15.19:0/2749059660 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1278102810 0x7f1278102bf0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:28:35.372 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.371+0000 7f1277200640 1 -- 172.21.15.19:0/2749059660 >> 172.21.15.19:0/2749059660 conn(0x7f12780fc030 msgr2=0x7f12780fe450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:28:35.372 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.371+0000 7f1277200640 1 -- 172.21.15.19:0/2749059660 shutdown_connections 2024-03-18T15:28:35.373 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.371+0000 7f1277200640 1 -- 172.21.15.19:0/2749059660 wait complete. 2024-03-18T15:28:35.375 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.372+0000 7f1277200640 1 Processor -- start 2024-03-18T15:28:35.375 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.372+0000 7f1277200640 1 -- start start 2024-03-18T15:28:35.375 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.373+0000 7f1277200640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1278102810 0x7f127819a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:28:35.375 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.373+0000 7f1277200640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1278103130 0x7f127819a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:28:35.375 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.373+0000 7f1277200640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f127810bc30 0x7f127819e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:28:35.375 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.373+0000 7f1277200640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f1278111a30 con 0x7f1278103130 2024-03-18T15:28:35.375 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.373+0000 7f1277200640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f12781118b0 con 0x7f127810bc30 2024-03-18T15:28:35.375 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.373+0000 7f1277200640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f1278111bb0 con 0x7f1278102810 2024-03-18T15:28:35.375 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.373+0000 7f1275e00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1278102810 0x7f127819a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:28:35.375 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.373+0000 7f1275e00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1278102810 0x7f127819a0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.19:40148/0 (socket says 172.21.15.19:40148) 2024-03-18T15:28:35.376 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.373+0000 7f1275e00640 1 -- 172.21.15.19:0/81568607 learned_addr learned my addr 172.21.15.19:0/81568607 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:28:35.376 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.373+0000 7f1276800640 1 --2- 172.21.15.19:0/81568607 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f127810bc30 0x7f127819e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:28:35.376 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.373+0000 7f1275400640 1 --2- 172.21.15.19:0/81568607 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1278103130 0x7f127819a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:28:35.376 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.373+0000 7f1275e00640 1 -- 172.21.15.19:0/81568607 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f127810bc30 msgr2=0x7f127819e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:28:35.376 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.373+0000 7f1275e00640 1 --2- 172.21.15.19:0/81568607 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f127810bc30 0x7f127819e9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:28:35.376 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.373+0000 7f1275e00640 1 -- 172.21.15.19:0/81568607 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1278103130 msgr2=0x7f127819a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:28:35.376 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.373+0000 7f1275e00640 1 --2- 172.21.15.19:0/81568607 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1278103130 0x7f127819a630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:28:35.376 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.373+0000 7f1275e00640 1 -- 172.21.15.19:0/81568607 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f127819f0e0 con 0x7f1278102810 2024-03-18T15:28:35.376 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.374+0000 7f1275e00640 1 --2- 172.21.15.19:0/81568607 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1278102810 0x7f127819a0f0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f1268005940 tx=0x7f126802cc40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:28:35.376 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.374+0000 7f126ea00640 1 -- 172.21.15.19:0/81568607 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1268041070 con 0x7f1278102810 2024-03-18T15:28:35.376 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.374+0000 7f126ea00640 1 -- 172.21.15.19:0/81568607 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1268004450 con 0x7f1278102810 2024-03-18T15:28:35.376 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.374+0000 7f1277200640 1 -- 172.21.15.19:0/81568607 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f127819f370 con 0x7f1278102810 2024-03-18T15:28:35.376 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.374+0000 7f126ea00640 1 -- 172.21.15.19:0/81568607 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1268003df0 con 0x7f1278102810 2024-03-18T15:28:35.376 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.374+0000 7f1277200640 1 -- 172.21.15.19:0/81568607 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12781a6c60 con 0x7f1278102810 2024-03-18T15:28:35.377 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.375+0000 7f1277200640 1 -- 172.21.15.19:0/81568607 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12780684f0 con 0x7f1278102810 2024-03-18T15:28:35.378 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.378+0000 7f126ea00640 1 -- 172.21.15.19:0/81568607 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 27) ==== 137421+0+0 (secure 0 0 0) 0x7f1268044070 con 0x7f1278102810 2024-03-18T15:28:35.379 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.379+0000 7f126ea00640 1 --2- 172.21.15.19:0/81568607 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f125c0a29f0 0x7f125c0a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:28:35.380 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.379+0000 7f126ea00640 1 -- 172.21.15.19:0/81568607 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9597+0+0 (secure 0 0 0) 0x7f12680f0eb0 con 0x7f1278102810 2024-03-18T15:28:35.380 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.379+0000 7f1275400640 1 --2- 172.21.15.19:0/81568607 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f125c0a29f0 0x7f125c0a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:28:35.381 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.380+0000 7f1275400640 1 --2- 172.21.15.19:0/81568607 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f125c0a29f0 0x7f125c0a4eb0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f12640058c0 tx=0x7f1264005850 comp rx=0 tx=0).ready entity=mgr.25466 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:28:35.387 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.386+0000 7f126ea00640 1 -- 172.21.15.19:0/81568607 <== mon.1 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f126811f740 con 0x7f1278102810 2024-03-18T15:28:35.591 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.590+0000 7f1277200640 1 -- 172.21.15.19:0/81568607 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f12781a6e30 con 0x7f1278102810 2024-03-18T15:28:35.592 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.591+0000 7f126ea00640 1 -- 172.21.15.19:0/81568607 <== mon.1 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v103) ==== 100+0+11 (secure 0 0 0) 0x7f12680bd8f0 con 0x7f1278102810 2024-03-18T15:28:35.592 INFO:teuthology.orchestra.run.smithi019.stdout:pg_num: 64 2024-03-18T15:28:35.595 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.593+0000 7f1277200640 1 -- 172.21.15.19:0/81568607 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f125c0a29f0 msgr2=0x7f125c0a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:28:35.595 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.593+0000 7f1277200640 1 --2- 172.21.15.19:0/81568607 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f125c0a29f0 0x7f125c0a4eb0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f12640058c0 tx=0x7f1264005850 comp rx=0 tx=0).stop 2024-03-18T15:28:35.595 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.593+0000 7f1277200640 1 -- 172.21.15.19:0/81568607 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1278102810 msgr2=0x7f127819a0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:28:35.595 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.593+0000 7f1277200640 1 --2- 172.21.15.19:0/81568607 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1278102810 0x7f127819a0f0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f1268005940 tx=0x7f126802cc40 comp rx=0 tx=0).stop 2024-03-18T15:28:35.595 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.594+0000 7f1277200640 1 -- 172.21.15.19:0/81568607 shutdown_connections 2024-03-18T15:28:35.595 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.594+0000 7f1277200640 1 --2- 172.21.15.19:0/81568607 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f125c0a29f0 0x7f125c0a4eb0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:28:35.595 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.594+0000 7f1277200640 1 --2- 172.21.15.19:0/81568607 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f127810bc30 0x7f127819e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:28:35.595 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.594+0000 7f1277200640 1 --2- 172.21.15.19:0/81568607 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1278103130 0x7f127819a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:28:35.595 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.594+0000 7f1277200640 1 --2- 172.21.15.19:0/81568607 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1278102810 0x7f127819a0f0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:28:35.595 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.594+0000 7f1277200640 1 -- 172.21.15.19:0/81568607 >> 172.21.15.19:0/81568607 conn(0x7f12780fc030 msgr2=0x7f12780fe420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:28:35.596 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.594+0000 7f1277200640 1 -- 172.21.15.19:0/81568607 shutdown_connections 2024-03-18T15:28:35.596 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.594+0000 7f1277200640 1 -- 172.21.15.19:0/81568607 wait complete. 2024-03-18T15:28:35.615 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-18T15:28:35.702 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:28:35 smithi019 ceph-mon[31522]: pgmap v314: 129 pgs: 129 active+clean; 1.7 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:28:35.702 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:28:35 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/3040652893' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T15:28:35.702 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:28:35 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/2473865982' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T15:28:35.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:28:35 smithi033 ceph-mon[36648]: pgmap v314: 129 pgs: 129 active+clean; 1.7 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:28:35.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:28:35 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/3040652893' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T15:28:35.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:28:35 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/2473865982' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T15:28:35.841 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.840+0000 7f3981800640 1 -- 172.21.15.19:0/3701004560 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f397c073bb0 msgr2=0x7f397c073f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:28:35.841 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.840+0000 7f3981800640 1 --2- 172.21.15.19:0/3701004560 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f397c073bb0 0x7f397c073f90 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f396c009900 tx=0x7f396c02bcd0 comp rx=0 tx=0).stop 2024-03-18T15:28:35.842 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.840+0000 7f3981800640 1 -- 172.21.15.19:0/3701004560 shutdown_connections 2024-03-18T15:28:35.842 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.840+0000 7f3981800640 1 --2- 172.21.15.19:0/3701004560 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f397c10def0 0x7f397c110300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:28:35.842 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.840+0000 7f3981800640 1 --2- 172.21.15.19:0/3701004560 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f397c0744d0 0x7f397c10d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:28:35.842 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.840+0000 7f3981800640 1 --2- 172.21.15.19:0/3701004560 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f397c073bb0 0x7f397c073f90 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:28:35.842 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.840+0000 7f3981800640 1 -- 172.21.15.19:0/3701004560 >> 172.21.15.19:0/3701004560 conn(0x7f397c0fd140 msgr2=0x7f397c0ff560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:28:35.842 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.840+0000 7f3981800640 1 -- 172.21.15.19:0/3701004560 shutdown_connections 2024-03-18T15:28:35.842 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.840+0000 7f3981800640 1 -- 172.21.15.19:0/3701004560 wait complete. 2024-03-18T15:28:35.843 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.841+0000 7f3981800640 1 Processor -- start 2024-03-18T15:28:35.843 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.841+0000 7f3981800640 1 -- start start 2024-03-18T15:28:35.843 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.842+0000 7f3981800640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f397c073bb0 0x7f397c19a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:28:35.844 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.842+0000 7f3981800640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f397c0744d0 0x7f397c19a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:28:35.844 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.842+0000 7f3981800640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f397c10def0 0x7f397c19ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:28:35.844 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.842+0000 7f3981800640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f397c113cc0 con 0x7f397c10def0 2024-03-18T15:28:35.844 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.842+0000 7f3981800640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f397c113b40 con 0x7f397c073bb0 2024-03-18T15:28:35.844 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.842+0000 7f3981800640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f397c113e40 con 0x7f397c0744d0 2024-03-18T15:28:35.844 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.842+0000 7f397be00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f397c073bb0 0x7f397c19a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:28:35.844 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.842+0000 7f397be00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f397c073bb0 0x7f397c19a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:57312/0 (socket says 172.21.15.19:57312) 2024-03-18T15:28:35.844 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.842+0000 7f397be00640 1 -- 172.21.15.19:0/1438829839 learned_addr learned my addr 172.21.15.19:0/1438829839 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:28:35.844 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.842+0000 7f397b400640 1 --2- 172.21.15.19:0/1438829839 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f397c0744d0 0x7f397c19a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:28:35.844 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.842+0000 7f3980e00640 1 --2- 172.21.15.19:0/1438829839 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f397c10def0 0x7f397c19ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:28:35.844 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.842+0000 7f397be00640 1 -- 172.21.15.19:0/1438829839 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f397c0744d0 msgr2=0x7f397c19a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:28:35.844 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.842+0000 7f397be00640 1 --2- 172.21.15.19:0/1438829839 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f397c0744d0 0x7f397c19a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:28:35.844 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.842+0000 7f397be00640 1 -- 172.21.15.19:0/1438829839 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f397c10def0 msgr2=0x7f397c19ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:28:35.844 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.842+0000 7f397be00640 1 --2- 172.21.15.19:0/1438829839 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f397c10def0 0x7f397c19ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:28:35.845 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.842+0000 7f397be00640 1 -- 172.21.15.19:0/1438829839 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f397c19f1a0 con 0x7f397c073bb0 2024-03-18T15:28:35.845 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.843+0000 7f397be00640 1 --2- 172.21.15.19:0/1438829839 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f397c073bb0 0x7f397c19a200 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f396c000c00 tx=0x7f396c004360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:28:35.845 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.843+0000 7f3978c00640 1 -- 172.21.15.19:0/1438829839 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f396c042020 con 0x7f397c073bb0 2024-03-18T15:28:35.845 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.843+0000 7f3978c00640 1 -- 172.21.15.19:0/1438829839 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f396c0054e0 con 0x7f397c073bb0 2024-03-18T15:28:35.846 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.843+0000 7f3981800640 1 -- 172.21.15.19:0/1438829839 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f397c19f430 con 0x7f397c073bb0 2024-03-18T15:28:35.846 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.843+0000 7f3978c00640 1 -- 172.21.15.19:0/1438829839 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f396c003b20 con 0x7f397c073bb0 2024-03-18T15:28:35.846 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.843+0000 7f3981800640 1 -- 172.21.15.19:0/1438829839 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f397c1a6d20 con 0x7f397c073bb0 2024-03-18T15:28:35.846 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.845+0000 7f3981800640 1 -- 172.21.15.19:0/1438829839 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f397c068940 con 0x7f397c073bb0 2024-03-18T15:28:35.847 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.846+0000 7f3978c00640 1 -- 172.21.15.19:0/1438829839 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 27) ==== 137421+0+0 (secure 0 0 0) 0x7f396c045020 con 0x7f397c073bb0 2024-03-18T15:28:35.848 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.847+0000 7f3978c00640 1 --2- 172.21.15.19:0/1438829839 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f39600a2970 0x7f39600a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:28:35.848 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.847+0000 7f3978c00640 1 -- 172.21.15.19:0/1438829839 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9597+0+0 (secure 0 0 0) 0x7f396c0f0db0 con 0x7f397c073bb0 2024-03-18T15:28:35.848 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.848+0000 7f397b400640 1 --2- 172.21.15.19:0/1438829839 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f39600a2970 0x7f39600a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:28:35.849 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.848+0000 7f397b400640 1 --2- 172.21.15.19:0/1438829839 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f39600a2970 0x7f39600a4e30 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f397c0ff7d0 tx=0x7f3964005f90 comp rx=0 tx=0).ready entity=mgr.25466 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:28:35.852 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:35.851+0000 7f3978c00640 1 -- 172.21.15.19:0/1438829839 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f396c0f5070 con 0x7f397c073bb0 2024-03-18T15:28:35.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:28:35 smithi104 ceph-mon[36070]: pgmap v314: 129 pgs: 129 active+clean; 1.7 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:28:35.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:28:35 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/3040652893' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T15:28:35.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:28:35 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/2473865982' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T15:28:36.056 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:36.055+0000 7f3981800640 1 -- 172.21.15.19:0/1438829839 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f397c109070 con 0x7f397c073bb0 2024-03-18T15:28:36.057 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:36.056+0000 7f3978c00640 1 -- 172.21.15.19:0/1438829839 <== mon.2 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v103) ==== 103+0+11 (secure 0 0 0) 0x7f396c04c020 con 0x7f397c073bb0 2024-03-18T15:28:36.057 INFO:teuthology.orchestra.run.smithi019.stdout:pg_num: 32 2024-03-18T15:28:36.060 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:36.059+0000 7f3981800640 1 -- 172.21.15.19:0/1438829839 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f39600a2970 msgr2=0x7f39600a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:28:36.060 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:36.059+0000 7f3981800640 1 --2- 172.21.15.19:0/1438829839 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f39600a2970 0x7f39600a4e30 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f397c0ff7d0 tx=0x7f3964005f90 comp rx=0 tx=0).stop 2024-03-18T15:28:36.060 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:36.059+0000 7f3981800640 1 -- 172.21.15.19:0/1438829839 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f397c073bb0 msgr2=0x7f397c19a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:28:36.061 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:36.059+0000 7f3981800640 1 --2- 172.21.15.19:0/1438829839 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f397c073bb0 0x7f397c19a200 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f396c000c00 tx=0x7f396c004360 comp rx=0 tx=0).stop 2024-03-18T15:28:36.061 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:36.059+0000 7f3981800640 1 -- 172.21.15.19:0/1438829839 shutdown_connections 2024-03-18T15:28:36.061 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:36.059+0000 7f3981800640 1 --2- 172.21.15.19:0/1438829839 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f39600a2970 0x7f39600a4e30 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:28:36.061 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:36.059+0000 7f3981800640 1 --2- 172.21.15.19:0/1438829839 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f397c10def0 0x7f397c19ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:28:36.061 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:36.059+0000 7f3981800640 1 --2- 172.21.15.19:0/1438829839 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f397c0744d0 0x7f397c19a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:28:36.061 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:36.059+0000 7f3981800640 1 --2- 172.21.15.19:0/1438829839 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f397c073bb0 0x7f397c19a200 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:28:36.061 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:36.059+0000 7f3981800640 1 -- 172.21.15.19:0/1438829839 >> 172.21.15.19:0/1438829839 conn(0x7f397c0fd140 msgr2=0x7f397c0fe8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:28:36.061 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:36.060+0000 7f3981800640 1 -- 172.21.15.19:0/1438829839 shutdown_connections 2024-03-18T15:28:36.061 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:36.060+0000 7f3981800640 1 -- 172.21.15.19:0/1438829839 wait complete. 2024-03-18T15:28:36.080 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-03-18T15:28:36.309 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:36.307+0000 7f9c25000640 1 -- 172.21.15.19:0/2429412210 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9c20104350 msgr2=0x7f9c20110860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:28:36.309 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:36.307+0000 7f9c25000640 1 --2- 172.21.15.19:0/2429412210 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9c20104350 0x7f9c20110860 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f9c18009900 tx=0x7f9c1802bc40 comp rx=0 tx=0).stop 2024-03-18T15:28:36.309 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:36.308+0000 7f9c25000640 1 -- 172.21.15.19:0/2429412210 shutdown_connections 2024-03-18T15:28:36.309 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:36.308+0000 7f9c25000640 1 --2- 172.21.15.19:0/2429412210 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9c20104350 0x7f9c20110860 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:28:36.309 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:36.308+0000 7f9c25000640 1 --2- 172.21.15.19:0/2429412210 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9c20103990 0x7f9c20103e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:28:36.309 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:36.308+0000 7f9c25000640 1 --2- 172.21.15.19:0/2429412210 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c201099a0 0x7f9c20109d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:28:36.309 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:36.308+0000 7f9c25000640 1 -- 172.21.15.19:0/2429412210 >> 172.21.15.19:0/2429412210 conn(0x7f9c200ff460 msgr2=0x7f9c20101880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:28:36.309 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:36.308+0000 7f9c25000640 1 -- 172.21.15.19:0/2429412210 shutdown_connections 2024-03-18T15:28:36.309 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:36.308+0000 7f9c25000640 1 -- 172.21.15.19:0/2429412210 wait complete. 2024-03-18T15:28:36.312 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:36.309+0000 7f9c25000640 1 Processor -- start 2024-03-18T15:28:36.312 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:36.309+0000 7f9c25000640 1 -- start start 2024-03-18T15:28:36.312 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:36.309+0000 7f9c25000640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c20103990 0x7f9c2019a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:28:36.312 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:36.309+0000 7f9c25000640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9c20104350 0x7f9c2019a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:28:36.312 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:36.309+0000 7f9c25000640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9c201099a0 0x7f9c2019eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:28:36.312 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:36.309+0000 7f9c25000640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f9c20113e00 con 0x7f9c201099a0 2024-03-18T15:28:36.312 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:36.309+0000 7f9c25000640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9c20113c80 con 0x7f9c20104350 2024-03-18T15:28:36.312 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:36.309+0000 7f9c25000640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f9c20113f80 con 0x7f9c20103990 2024-03-18T15:28:36.312 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:36.310+0000 7f9c1de00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c20103990 0x7f9c2019a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:28:36.312 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:36.310+0000 7f9c1d400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9c20104350 0x7f9c2019a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:28:36.312 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:36.310+0000 7f9c1de00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c20103990 0x7f9c2019a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.19:40166/0 (socket says 172.21.15.19:40166) 2024-03-18T15:28:36.313 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:36.310+0000 7f9c1de00640 1 -- 172.21.15.19:0/2368939074 learned_addr learned my addr 172.21.15.19:0/2368939074 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:28:36.313 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:36.310+0000 7f9c1e800640 1 --2- 172.21.15.19:0/2368939074 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9c201099a0 0x7f9c2019eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:28:36.313 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:36.310+0000 7f9c1de00640 1 -- 172.21.15.19:0/2368939074 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9c20104350 msgr2=0x7f9c2019a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:28:36.313 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:36.310+0000 7f9c1de00640 1 --2- 172.21.15.19:0/2368939074 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9c20104350 0x7f9c2019a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:28:36.313 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:36.310+0000 7f9c1de00640 1 -- 172.21.15.19:0/2368939074 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9c201099a0 msgr2=0x7f9c2019eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:28:36.313 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:36.310+0000 7f9c1de00640 1 --2- 172.21.15.19:0/2368939074 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9c201099a0 0x7f9c2019eb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:28:36.313 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:36.310+0000 7f9c1de00640 1 -- 172.21.15.19:0/2368939074 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c2019f230 con 0x7f9c20103990 2024-03-18T15:28:36.313 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:36.310+0000 7f9c1de00640 1 --2- 172.21.15.19:0/2368939074 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c20103990 0x7f9c2019a240 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f9c1000ebc0 tx=0x7f9c1000c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:28:36.313 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:36.310+0000 7f9c16a00640 1 -- 172.21.15.19:0/2368939074 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c10010630 con 0x7f9c20103990 2024-03-18T15:28:36.313 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:36.311+0000 7f9c25000640 1 -- 172.21.15.19:0/2368939074 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c2019f520 con 0x7f9c20103990 2024-03-18T15:28:36.313 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:36.311+0000 7f9c16a00640 1 -- 172.21.15.19:0/2368939074 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9c10010cb0 con 0x7f9c20103990 2024-03-18T15:28:36.313 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:36.311+0000 7f9c16a00640 1 -- 172.21.15.19:0/2368939074 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c10018bd0 con 0x7f9c20103990 2024-03-18T15:28:36.313 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:36.311+0000 7f9c25000640 1 -- 172.21.15.19:0/2368939074 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c201a6db0 con 0x7f9c20103990 2024-03-18T15:28:36.314 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:36.312+0000 7f9c25000640 1 -- 172.21.15.19:0/2368939074 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c20068a20 con 0x7f9c20103990 2024-03-18T15:28:36.315 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:36.314+0000 7f9c16a00640 1 -- 172.21.15.19:0/2368939074 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 27) ==== 137421+0+0 (secure 0 0 0) 0x7f9c100107d0 con 0x7f9c20103990 2024-03-18T15:28:36.316 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:36.315+0000 7f9c16a00640 1 --2- 172.21.15.19:0/2368939074 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f9c040a2970 0x7f9c040a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:28:36.317 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:36.315+0000 7f9c16a00640 1 -- 172.21.15.19:0/2368939074 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9597+0+0 (secure 0 0 0) 0x7f9c100cfaa0 con 0x7f9c20103990 2024-03-18T15:28:36.317 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:36.315+0000 7f9c1d400640 1 --2- 172.21.15.19:0/2368939074 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f9c040a2970 0x7f9c040a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:28:36.317 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:36.316+0000 7f9c1d400640 1 --2- 172.21.15.19:0/2368939074 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f9c040a2970 0x7f9c040a4e30 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f9c20069430 tx=0x7f9c0c005f90 comp rx=0 tx=0).ready entity=mgr.25466 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:28:36.322 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:36.322+0000 7f9c16a00640 1 -- 172.21.15.19:0/2368939074 <== mon.1 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9c1009bd20 con 0x7f9c20103990 2024-03-18T15:28:36.578 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:36.577+0000 7f9c25000640 1 -- 172.21.15.19:0/2368939074 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7f9c201a7090 con 0x7f9c20103990 2024-03-18T15:28:36.589 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:36.588+0000 7f9c16a00640 1 -- 172.21.15.19:0/2368939074 <== mon.1 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v27) ==== 81+0+740360 (secure 0 0 0) 0x7f9c1009fdc0 con 0x7f9c20103990 2024-03-18T15:28:36.601 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-18T15:28:36.601 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-03-18T15:28:36.602 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 27, 2024-03-18T15:28:36.602 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.602 INFO:teuthology.orchestra.run.smithi019.stdout: "active_gid": 25466, 2024-03-18T15:28:36.602 INFO:teuthology.orchestra.run.smithi019.stdout: "active_name": "x", 2024-03-18T15:28:36.602 INFO:teuthology.orchestra.run.smithi019.stdout: "active_addrs": { 2024-03-18T15:28:36.602 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-03-18T15:28:36.602 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:36.602 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-03-18T15:28:36.602 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6800", 2024-03-18T15:28:36.602 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 3894323675 2024-03-18T15:28:36.602 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.602 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:36.602 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-03-18T15:28:36.603 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:6801", 2024-03-18T15:28:36.603 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 3894323675 2024-03-18T15:28:36.603 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:36.603 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-03-18T15:28:36.603 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.603 INFO:teuthology.orchestra.run.smithi019.stdout: "active_addr": "172.21.15.19:6801/3894323675", 2024-03-18T15:28:36.603 INFO:teuthology.orchestra.run.smithi019.stdout: "active_change": "2024-03-18T15:18:29.726606+0000", 2024-03-18T15:28:36.603 INFO:teuthology.orchestra.run.smithi019.stdout: "active_mgr_features": 4540701547738038271, 2024-03-18T15:28:36.603 INFO:teuthology.orchestra.run.smithi019.stdout: "available": true, 2024-03-18T15:28:36.603 INFO:teuthology.orchestra.run.smithi019.stdout: "standbys": [ 2024-03-18T15:28:36.603 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:36.603 INFO:teuthology.orchestra.run.smithi019.stdout: "gid": 25457, 2024-03-18T15:28:36.603 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "y", 2024-03-18T15:28:36.603 INFO:teuthology.orchestra.run.smithi019.stdout: "mgr_features": 4540701547738038271, 2024-03-18T15:28:36.604 INFO:teuthology.orchestra.run.smithi019.stdout: "available_modules": [ 2024-03-18T15:28:36.604 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:36.604 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "alerts", 2024-03-18T15:28:36.604 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": true, 2024-03-18T15:28:36.604 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "", 2024-03-18T15:28:36.604 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:36.604 INFO:teuthology.orchestra.run.smithi019.stdout: "interval": { 2024-03-18T15:28:36.604 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "interval", 2024-03-18T15:28:36.604 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "secs", 2024-03-18T15:28:36.605 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.605 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.605 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "60", 2024-03-18T15:28:36.605 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.605 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.605 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.605 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "How frequently to reexamine health status", 2024-03-18T15:28:36.605 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.605 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.605 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.605 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.606 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:36.606 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:36.606 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.606 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.606 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.606 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:36.606 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.606 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.606 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:36.606 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:36.606 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:36.606 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:36.607 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:36.607 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:36.607 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:36.607 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:36.607 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.607 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.607 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.607 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.607 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.607 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:36.607 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:36.607 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.607 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.608 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.608 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:36.608 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.608 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.608 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.608 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.608 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.608 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.608 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.608 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.608 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:36.608 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:36.608 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.608 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.609 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.609 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:36.609 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.609 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.609 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:36.609 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:36.609 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:36.609 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:36.609 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:36.609 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:36.609 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:36.609 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:36.609 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.609 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.610 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.610 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.610 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.610 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:36.610 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:36.610 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.610 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.610 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.610 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:36.610 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.610 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.610 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.610 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.610 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.611 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.611 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.611 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.611 INFO:teuthology.orchestra.run.smithi019.stdout: "smtp_destination": { 2024-03-18T15:28:36.611 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "smtp_destination", 2024-03-18T15:28:36.611 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.611 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.611 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.611 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:36.611 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.611 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.611 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.611 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Email address to send alerts to", 2024-03-18T15:28:36.611 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.611 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.612 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.612 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.612 INFO:teuthology.orchestra.run.smithi019.stdout: "smtp_from_name": { 2024-03-18T15:28:36.612 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "smtp_from_name", 2024-03-18T15:28:36.612 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.612 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.612 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.612 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "Ceph", 2024-03-18T15:28:36.612 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.612 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.612 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.612 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Email From: name", 2024-03-18T15:28:36.612 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.612 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.612 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.613 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.613 INFO:teuthology.orchestra.run.smithi019.stdout: "smtp_host": { 2024-03-18T15:28:36.613 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "smtp_host", 2024-03-18T15:28:36.613 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.613 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.613 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.613 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:36.613 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.613 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.613 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.613 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "SMTP server", 2024-03-18T15:28:36.613 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.613 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.613 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.614 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.614 INFO:teuthology.orchestra.run.smithi019.stdout: "smtp_password": { 2024-03-18T15:28:36.614 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "smtp_password", 2024-03-18T15:28:36.614 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.614 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.614 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.614 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:36.614 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.614 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.614 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.614 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Password to authenticate with", 2024-03-18T15:28:36.614 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.614 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.614 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.614 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.615 INFO:teuthology.orchestra.run.smithi019.stdout: "smtp_port": { 2024-03-18T15:28:36.615 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "smtp_port", 2024-03-18T15:28:36.615 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:36.615 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.615 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.615 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "465", 2024-03-18T15:28:36.615 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.615 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.615 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.615 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "SMTP port", 2024-03-18T15:28:36.615 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.615 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.615 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.615 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.615 INFO:teuthology.orchestra.run.smithi019.stdout: "smtp_sender": { 2024-03-18T15:28:36.616 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "smtp_sender", 2024-03-18T15:28:36.616 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.616 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.616 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.616 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:36.616 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.616 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.616 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.616 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "SMTP envelope sender", 2024-03-18T15:28:36.616 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.616 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.616 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.617 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.617 INFO:teuthology.orchestra.run.smithi019.stdout: "smtp_ssl": { 2024-03-18T15:28:36.617 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "smtp_ssl", 2024-03-18T15:28:36.617 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.617 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.617 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.617 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:36.617 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.617 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.617 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.617 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-18T15:28:36.617 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.617 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.617 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.618 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.618 INFO:teuthology.orchestra.run.smithi019.stdout: "smtp_user": { 2024-03-18T15:28:36.618 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "smtp_user", 2024-03-18T15:28:36.618 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.618 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.618 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.618 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:36.618 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.618 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.618 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.618 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "User to authenticate as", 2024-03-18T15:28:36.619 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.619 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.619 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.619 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:36.619 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:36.619 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.619 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:36.619 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "balancer", 2024-03-18T15:28:36.619 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": true, 2024-03-18T15:28:36.619 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "", 2024-03-18T15:28:36.619 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:36.619 INFO:teuthology.orchestra.run.smithi019.stdout: "active": { 2024-03-18T15:28:36.619 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "active", 2024-03-18T15:28:36.619 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.620 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.620 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.620 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:36.620 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.620 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.620 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.620 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "automatically balance PGs across cluster", 2024-03-18T15:28:36.620 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.620 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.620 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.620 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.620 INFO:teuthology.orchestra.run.smithi019.stdout: "begin_time": { 2024-03-18T15:28:36.620 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "begin_time", 2024-03-18T15:28:36.620 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.621 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.621 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.621 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "0000", 2024-03-18T15:28:36.621 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.621 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.621 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.621 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "beginning time of day to automatically balance", 2024-03-18T15:28:36.621 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-18T15:28:36.621 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.621 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.621 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.621 INFO:teuthology.orchestra.run.smithi019.stdout: "begin_weekday": { 2024-03-18T15:28:36.621 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "begin_weekday", 2024-03-18T15:28:36.622 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "uint", 2024-03-18T15:28:36.622 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.622 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.622 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "0", 2024-03-18T15:28:36.622 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "0", 2024-03-18T15:28:36.622 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "6", 2024-03-18T15:28:36.622 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.622 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-18T15:28:36.622 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-18T15:28:36.622 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.622 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.622 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.622 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_compat_max_iterations": { 2024-03-18T15:28:36.623 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "crush_compat_max_iterations", 2024-03-18T15:28:36.623 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "uint", 2024-03-18T15:28:36.623 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.623 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.623 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "25", 2024-03-18T15:28:36.623 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "1", 2024-03-18T15:28:36.623 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "250", 2024-03-18T15:28:36.623 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.623 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-18T15:28:36.623 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.623 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.623 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.623 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.624 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_compat_metrics": { 2024-03-18T15:28:36.624 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "crush_compat_metrics", 2024-03-18T15:28:36.624 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.624 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.624 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.624 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "pgs,objects,bytes", 2024-03-18T15:28:36.624 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.624 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.624 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.624 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-18T15:28:36.624 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-18T15:28:36.624 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.624 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.624 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.625 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_compat_step": { 2024-03-18T15:28:36.625 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "crush_compat_step", 2024-03-18T15:28:36.625 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "float", 2024-03-18T15:28:36.625 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.625 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.625 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "0.5", 2024-03-18T15:28:36.625 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "0.001", 2024-03-18T15:28:36.625 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "0.999", 2024-03-18T15:28:36.625 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.625 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "aggressiveness of optimization", 2024-03-18T15:28:36.625 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-18T15:28:36.625 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.625 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.625 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.625 INFO:teuthology.orchestra.run.smithi019.stdout: "end_time": { 2024-03-18T15:28:36.626 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "end_time", 2024-03-18T15:28:36.626 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.626 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.626 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.626 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "2359", 2024-03-18T15:28:36.626 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.626 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.626 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.626 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "ending time of day to automatically balance", 2024-03-18T15:28:36.626 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-18T15:28:36.626 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.626 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.626 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.626 INFO:teuthology.orchestra.run.smithi019.stdout: "end_weekday": { 2024-03-18T15:28:36.627 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "end_weekday", 2024-03-18T15:28:36.627 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "uint", 2024-03-18T15:28:36.627 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.627 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.627 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "0", 2024-03-18T15:28:36.627 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "0", 2024-03-18T15:28:36.627 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "6", 2024-03-18T15:28:36.627 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.627 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-18T15:28:36.627 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-18T15:28:36.627 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.627 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.627 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.627 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:36.628 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:36.628 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.628 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.628 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.628 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:36.628 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.628 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.628 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:36.628 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:36.628 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:36.628 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:36.628 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:36.628 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:36.628 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:36.629 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:36.629 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.629 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.629 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.629 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.629 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.629 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:36.629 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:36.629 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.629 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.629 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.629 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:36.630 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.630 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.630 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.630 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.630 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.630 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.630 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.630 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.630 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:36.630 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:36.630 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.630 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.630 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.631 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:36.631 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.631 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.631 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:36.631 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:36.631 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:36.631 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:36.631 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:36.631 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:36.631 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:36.631 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:36.631 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.631 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.632 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.632 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.632 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.632 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:36.632 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:36.632 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.632 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.632 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.632 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:36.632 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.632 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.632 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.632 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.632 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.633 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.633 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.633 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.633 INFO:teuthology.orchestra.run.smithi019.stdout: "min_score": { 2024-03-18T15:28:36.633 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "min_score", 2024-03-18T15:28:36.633 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "float", 2024-03-18T15:28:36.633 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.633 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.633 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "0", 2024-03-18T15:28:36.633 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.633 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.633 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.633 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-18T15:28:36.634 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.634 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.634 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.634 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.634 INFO:teuthology.orchestra.run.smithi019.stdout: "mode": { 2024-03-18T15:28:36.634 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "mode", 2024-03-18T15:28:36.634 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.634 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.634 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.634 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "upmap", 2024-03-18T15:28:36.634 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.634 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.634 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:36.634 INFO:teuthology.orchestra.run.smithi019.stdout: "crush-compat", 2024-03-18T15:28:36.635 INFO:teuthology.orchestra.run.smithi019.stdout: "none", 2024-03-18T15:28:36.635 INFO:teuthology.orchestra.run.smithi019.stdout: "read", 2024-03-18T15:28:36.635 INFO:teuthology.orchestra.run.smithi019.stdout: "upmap", 2024-03-18T15:28:36.635 INFO:teuthology.orchestra.run.smithi019.stdout: "upmap-read" 2024-03-18T15:28:36.635 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:36.635 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Balancer mode", 2024-03-18T15:28:36.635 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.635 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.635 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.635 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.635 INFO:teuthology.orchestra.run.smithi019.stdout: "pool_ids": { 2024-03-18T15:28:36.635 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "pool_ids", 2024-03-18T15:28:36.635 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.635 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.636 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.636 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:36.636 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.636 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.636 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.636 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-18T15:28:36.636 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.636 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.636 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.636 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.636 INFO:teuthology.orchestra.run.smithi019.stdout: "sleep_interval": { 2024-03-18T15:28:36.637 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "sleep_interval", 2024-03-18T15:28:36.637 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "secs", 2024-03-18T15:28:36.637 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.637 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.637 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "60", 2024-03-18T15:28:36.637 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.637 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.637 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.637 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-18T15:28:36.637 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.638 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.638 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.638 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.638 INFO:teuthology.orchestra.run.smithi019.stdout: "upmap_max_deviation": { 2024-03-18T15:28:36.638 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "upmap_max_deviation", 2024-03-18T15:28:36.638 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:36.638 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.638 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.638 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "5", 2024-03-18T15:28:36.638 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "1", 2024-03-18T15:28:36.638 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.638 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.638 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-18T15:28:36.639 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-18T15:28:36.639 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.639 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.639 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.639 INFO:teuthology.orchestra.run.smithi019.stdout: "upmap_max_optimizations": { 2024-03-18T15:28:36.639 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "upmap_max_optimizations", 2024-03-18T15:28:36.639 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "uint", 2024-03-18T15:28:36.639 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.639 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.639 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "10", 2024-03-18T15:28:36.639 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.639 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.639 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.640 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-18T15:28:36.640 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.640 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.640 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.640 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:36.640 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:36.640 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.640 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:36.640 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "cephadm", 2024-03-18T15:28:36.640 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": true, 2024-03-18T15:28:36.640 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "", 2024-03-18T15:28:36.640 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:36.641 INFO:teuthology.orchestra.run.smithi019.stdout: "agent_down_multiplier": { 2024-03-18T15:28:36.641 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "agent_down_multiplier", 2024-03-18T15:28:36.641 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "float", 2024-03-18T15:28:36.641 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.641 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.642 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "3.0", 2024-03-18T15:28:36.642 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.642 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.642 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.642 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-18T15:28:36.642 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.642 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.642 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.642 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.642 INFO:teuthology.orchestra.run.smithi019.stdout: "agent_refresh_rate": { 2024-03-18T15:28:36.642 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "agent_refresh_rate", 2024-03-18T15:28:36.642 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "secs", 2024-03-18T15:28:36.642 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.642 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.643 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "20", 2024-03-18T15:28:36.643 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.643 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.643 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.643 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-18T15:28:36.643 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.643 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.643 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.643 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.643 INFO:teuthology.orchestra.run.smithi019.stdout: "agent_starting_port": { 2024-03-18T15:28:36.643 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "agent_starting_port", 2024-03-18T15:28:36.643 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:36.643 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.644 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.644 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "4721", 2024-03-18T15:28:36.644 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.644 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.644 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.644 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-18T15:28:36.644 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.644 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.644 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.644 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.644 INFO:teuthology.orchestra.run.smithi019.stdout: "allow_ptrace": { 2024-03-18T15:28:36.644 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "allow_ptrace", 2024-03-18T15:28:36.645 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.645 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.645 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.645 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:36.645 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.645 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.645 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.645 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-18T15:28:36.645 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-18T15:28:36.645 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.645 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.645 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.645 INFO:teuthology.orchestra.run.smithi019.stdout: "autotune_interval": { 2024-03-18T15:28:36.645 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "autotune_interval", 2024-03-18T15:28:36.646 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "secs", 2024-03-18T15:28:36.646 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.646 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.646 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "600", 2024-03-18T15:28:36.646 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.646 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.646 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.646 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-18T15:28:36.646 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.646 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.646 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.646 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.650 INFO:teuthology.orchestra.run.smithi019.stdout: "autotune_memory_target_ratio": { 2024-03-18T15:28:36.650 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "autotune_memory_target_ratio", 2024-03-18T15:28:36.650 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "float", 2024-03-18T15:28:36.650 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.650 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.650 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "0.7", 2024-03-18T15:28:36.651 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.651 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.651 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.651 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-18T15:28:36.651 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.651 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.651 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.651 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.651 INFO:teuthology.orchestra.run.smithi019.stdout: "cephadm_log_destination": { 2024-03-18T15:28:36.651 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "cephadm_log_destination", 2024-03-18T15:28:36.651 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.651 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.651 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.652 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:36.652 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.652 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.652 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:36.652 INFO:teuthology.orchestra.run.smithi019.stdout: "file", 2024-03-18T15:28:36.652 INFO:teuthology.orchestra.run.smithi019.stdout: "file,syslog", 2024-03-18T15:28:36.652 INFO:teuthology.orchestra.run.smithi019.stdout: "syslog" 2024-03-18T15:28:36.652 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:36.652 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-18T15:28:36.652 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.652 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.652 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.652 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.652 INFO:teuthology.orchestra.run.smithi019.stdout: "cgroups_split": { 2024-03-18T15:28:36.653 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "cgroups_split", 2024-03-18T15:28:36.653 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.653 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.653 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.653 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:36.653 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.653 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.653 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.653 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-18T15:28:36.653 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.653 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.653 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.653 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.654 INFO:teuthology.orchestra.run.smithi019.stdout: "config_checks_enabled": { 2024-03-18T15:28:36.654 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "config_checks_enabled", 2024-03-18T15:28:36.654 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.654 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.654 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.654 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:36.654 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.654 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.654 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.654 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-18T15:28:36.654 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.654 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.654 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.655 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.655 INFO:teuthology.orchestra.run.smithi019.stdout: "config_dashboard": { 2024-03-18T15:28:36.655 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "config_dashboard", 2024-03-18T15:28:36.655 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.655 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.655 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.655 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:36.655 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.655 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.655 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.655 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-18T15:28:36.655 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.655 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.655 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.655 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.656 INFO:teuthology.orchestra.run.smithi019.stdout: "container_image_alertmanager": { 2024-03-18T15:28:36.656 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "container_image_alertmanager", 2024-03-18T15:28:36.656 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.656 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.656 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.656 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-18T15:28:36.656 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.656 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.656 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.656 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Prometheus container image", 2024-03-18T15:28:36.656 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.656 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.656 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.657 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.657 INFO:teuthology.orchestra.run.smithi019.stdout: "container_image_base": { 2024-03-18T15:28:36.657 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "container_image_base", 2024-03-18T15:28:36.657 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.657 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.657 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.657 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-18T15:28:36.657 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.657 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.657 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.657 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Container image name, without the tag", 2024-03-18T15:28:36.657 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.657 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.657 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.657 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.658 INFO:teuthology.orchestra.run.smithi019.stdout: "container_image_elasticsearch": { 2024-03-18T15:28:36.658 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "container_image_elasticsearch", 2024-03-18T15:28:36.658 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.658 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.658 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.658 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-18T15:28:36.658 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.658 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.658 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.658 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "elasticsearch container image", 2024-03-18T15:28:36.658 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.658 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.659 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.659 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.659 INFO:teuthology.orchestra.run.smithi019.stdout: "container_image_grafana": { 2024-03-18T15:28:36.659 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "container_image_grafana", 2024-03-18T15:28:36.659 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.659 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.659 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.659 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-18T15:28:36.659 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.659 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.659 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.659 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Prometheus container image", 2024-03-18T15:28:36.659 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.660 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.660 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.660 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.660 INFO:teuthology.orchestra.run.smithi019.stdout: "container_image_haproxy": { 2024-03-18T15:28:36.660 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "container_image_haproxy", 2024-03-18T15:28:36.660 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.660 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.660 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.660 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-18T15:28:36.660 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.660 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.661 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.661 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "HAproxy container image", 2024-03-18T15:28:36.661 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.661 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.661 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.661 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.661 INFO:teuthology.orchestra.run.smithi019.stdout: "container_image_jaeger_agent": { 2024-03-18T15:28:36.661 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "container_image_jaeger_agent", 2024-03-18T15:28:36.661 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.661 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.661 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.661 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-18T15:28:36.661 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.661 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.662 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.662 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Jaeger agent container image", 2024-03-18T15:28:36.662 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.662 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.662 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.662 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.662 INFO:teuthology.orchestra.run.smithi019.stdout: "container_image_jaeger_collector": { 2024-03-18T15:28:36.662 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "container_image_jaeger_collector", 2024-03-18T15:28:36.662 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.662 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.662 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.662 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-18T15:28:36.662 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.663 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.663 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.663 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Jaeger collector container image", 2024-03-18T15:28:36.663 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.663 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.663 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.663 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.663 INFO:teuthology.orchestra.run.smithi019.stdout: "container_image_jaeger_query": { 2024-03-18T15:28:36.663 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "container_image_jaeger_query", 2024-03-18T15:28:36.663 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.663 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.663 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.663 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-18T15:28:36.663 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.664 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.664 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.664 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Jaeger query container image", 2024-03-18T15:28:36.664 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.664 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.664 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.664 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.664 INFO:teuthology.orchestra.run.smithi019.stdout: "container_image_keepalived": { 2024-03-18T15:28:36.664 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "container_image_keepalived", 2024-03-18T15:28:36.664 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.664 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.664 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.664 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-18T15:28:36.665 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.665 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.665 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.665 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Keepalived container image", 2024-03-18T15:28:36.665 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.665 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.665 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.665 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.665 INFO:teuthology.orchestra.run.smithi019.stdout: "container_image_loki": { 2024-03-18T15:28:36.665 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "container_image_loki", 2024-03-18T15:28:36.665 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.665 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.665 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.665 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-18T15:28:36.665 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.666 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.666 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.666 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Loki container image", 2024-03-18T15:28:36.666 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.666 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.666 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.666 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.666 INFO:teuthology.orchestra.run.smithi019.stdout: "container_image_node_exporter": { 2024-03-18T15:28:36.666 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "container_image_node_exporter", 2024-03-18T15:28:36.666 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.666 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.666 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.666 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-18T15:28:36.667 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.667 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.667 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.667 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Prometheus container image", 2024-03-18T15:28:36.667 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.667 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.667 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.667 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.667 INFO:teuthology.orchestra.run.smithi019.stdout: "container_image_nvmeof": { 2024-03-18T15:28:36.667 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "container_image_nvmeof", 2024-03-18T15:28:36.667 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.667 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.667 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.668 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-18T15:28:36.668 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.668 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.668 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.668 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Nvme-of container image", 2024-03-18T15:28:36.668 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.668 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.668 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.668 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.668 INFO:teuthology.orchestra.run.smithi019.stdout: "container_image_prometheus": { 2024-03-18T15:28:36.668 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "container_image_prometheus", 2024-03-18T15:28:36.668 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.668 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.668 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.669 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-18T15:28:36.669 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.669 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.669 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.669 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Prometheus container image", 2024-03-18T15:28:36.669 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.669 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.669 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.669 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.669 INFO:teuthology.orchestra.run.smithi019.stdout: "container_image_promtail": { 2024-03-18T15:28:36.669 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "container_image_promtail", 2024-03-18T15:28:36.669 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.669 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.669 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.669 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-18T15:28:36.670 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.670 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.670 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.670 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Promtail container image", 2024-03-18T15:28:36.670 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.670 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.671 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.671 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.671 INFO:teuthology.orchestra.run.smithi019.stdout: "container_image_snmp_gateway": { 2024-03-18T15:28:36.671 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "container_image_snmp_gateway", 2024-03-18T15:28:36.672 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.672 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.672 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.672 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-18T15:28:36.672 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.672 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.672 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.672 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "SNMP Gateway container image", 2024-03-18T15:28:36.672 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.672 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.672 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.672 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.672 INFO:teuthology.orchestra.run.smithi019.stdout: "container_init": { 2024-03-18T15:28:36.672 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "container_init", 2024-03-18T15:28:36.673 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.673 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.673 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.673 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:36.673 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.673 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.673 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.673 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Run podman/docker with `--init`", 2024-03-18T15:28:36.673 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.673 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.673 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.673 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.673 INFO:teuthology.orchestra.run.smithi019.stdout: "daemon_cache_timeout": { 2024-03-18T15:28:36.673 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "daemon_cache_timeout", 2024-03-18T15:28:36.674 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "secs", 2024-03-18T15:28:36.674 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.674 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.674 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "600", 2024-03-18T15:28:36.674 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.674 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.674 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.674 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-18T15:28:36.674 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.674 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.674 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.674 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.675 INFO:teuthology.orchestra.run.smithi019.stdout: "default_cephadm_command_timeout": { 2024-03-18T15:28:36.675 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "default_cephadm_command_timeout", 2024-03-18T15:28:36.675 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:36.675 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.675 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.675 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "900", 2024-03-18T15:28:36.675 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.675 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.675 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.675 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-18T15:28:36.675 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.675 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.676 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.676 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.676 INFO:teuthology.orchestra.run.smithi019.stdout: "default_registry": { 2024-03-18T15:28:36.676 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "default_registry", 2024-03-18T15:28:36.676 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.676 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.676 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.676 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "docker.io", 2024-03-18T15:28:36.676 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.676 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.676 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.676 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-18T15:28:36.676 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.676 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.677 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.677 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.677 INFO:teuthology.orchestra.run.smithi019.stdout: "device_cache_timeout": { 2024-03-18T15:28:36.677 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "device_cache_timeout", 2024-03-18T15:28:36.677 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "secs", 2024-03-18T15:28:36.677 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.677 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.677 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "1800", 2024-03-18T15:28:36.677 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.677 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.677 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.677 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "seconds to cache device inventory", 2024-03-18T15:28:36.677 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.677 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.678 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.678 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.678 INFO:teuthology.orchestra.run.smithi019.stdout: "device_enhanced_scan": { 2024-03-18T15:28:36.678 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "device_enhanced_scan", 2024-03-18T15:28:36.678 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.678 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.678 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.678 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:36.678 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.678 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.678 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.678 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-18T15:28:36.678 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.678 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.679 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.679 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.679 INFO:teuthology.orchestra.run.smithi019.stdout: "facts_cache_timeout": { 2024-03-18T15:28:36.679 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "facts_cache_timeout", 2024-03-18T15:28:36.679 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "secs", 2024-03-18T15:28:36.679 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.679 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.679 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "60", 2024-03-18T15:28:36.679 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.679 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.679 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.679 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "seconds to cache host facts data", 2024-03-18T15:28:36.679 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.679 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.680 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.680 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.680 INFO:teuthology.orchestra.run.smithi019.stdout: "host_check_interval": { 2024-03-18T15:28:36.680 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "host_check_interval", 2024-03-18T15:28:36.680 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "secs", 2024-03-18T15:28:36.680 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.680 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.680 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "600", 2024-03-18T15:28:36.680 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.680 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.680 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.680 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "how frequently to perform a host check", 2024-03-18T15:28:36.680 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.680 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.681 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.681 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.681 INFO:teuthology.orchestra.run.smithi019.stdout: "hw_monitoring": { 2024-03-18T15:28:36.681 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "hw_monitoring", 2024-03-18T15:28:36.681 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.681 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.681 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.681 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:36.681 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.681 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.681 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.681 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-18T15:28:36.681 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.681 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.681 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.682 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.682 INFO:teuthology.orchestra.run.smithi019.stdout: "inventory_list_all": { 2024-03-18T15:28:36.682 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "inventory_list_all", 2024-03-18T15:28:36.682 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.682 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.682 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.682 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:36.682 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.682 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.682 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.682 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-18T15:28:36.682 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.682 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.682 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.683 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.683 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:36.683 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:36.683 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.683 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.683 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.683 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:36.683 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.683 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.683 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:36.683 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:36.683 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:36.683 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:36.684 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:36.684 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:36.684 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:36.684 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:36.684 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.684 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.684 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.684 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.684 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.684 INFO:teuthology.orchestra.run.smithi019.stdout: "log_refresh_metadata": { 2024-03-18T15:28:36.684 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_refresh_metadata", 2024-03-18T15:28:36.684 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.684 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.684 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.684 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:36.685 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.685 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.685 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.685 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-18T15:28:36.685 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.685 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.685 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.685 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.685 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:36.685 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:36.685 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.685 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.685 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.685 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:36.686 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.686 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.686 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.686 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-18T15:28:36.686 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.686 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.686 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.686 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.686 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:36.686 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:36.686 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.686 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.686 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.687 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:36.687 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.687 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.687 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:36.687 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:36.687 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:36.687 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:36.687 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:36.687 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:36.687 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:36.687 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:36.687 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.687 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.687 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.688 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.688 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.688 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:36.688 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:36.688 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.688 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.688 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.688 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:36.688 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.689 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.689 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.689 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.689 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.689 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.689 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.689 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.689 INFO:teuthology.orchestra.run.smithi019.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-18T15:28:36.689 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-18T15:28:36.689 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.689 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.689 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.689 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:36.689 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.690 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.690 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.690 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-18T15:28:36.690 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.690 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.690 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.690 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.690 INFO:teuthology.orchestra.run.smithi019.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-18T15:28:36.690 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-18T15:28:36.690 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.690 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.690 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.690 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "*", 2024-03-18T15:28:36.690 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.691 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.691 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.691 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-18T15:28:36.691 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.691 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.691 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.691 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.691 INFO:teuthology.orchestra.run.smithi019.stdout: "max_count_per_host": { 2024-03-18T15:28:36.691 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "max_count_per_host", 2024-03-18T15:28:36.691 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:36.691 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.691 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.691 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "10", 2024-03-18T15:28:36.691 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.692 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.692 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.692 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "max number of daemons per service per host", 2024-03-18T15:28:36.692 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.692 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.692 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.692 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.692 INFO:teuthology.orchestra.run.smithi019.stdout: "max_osd_draining_count": { 2024-03-18T15:28:36.692 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "max_osd_draining_count", 2024-03-18T15:28:36.692 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:36.692 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.692 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.692 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "10", 2024-03-18T15:28:36.692 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.693 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.693 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.693 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-18T15:28:36.693 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.693 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.693 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.693 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.693 INFO:teuthology.orchestra.run.smithi019.stdout: "migration_current": { 2024-03-18T15:28:36.693 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "migration_current", 2024-03-18T15:28:36.693 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:36.693 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.693 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.693 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:36.693 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.694 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.694 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.694 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "internal - do not modify", 2024-03-18T15:28:36.694 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.694 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.694 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.694 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.694 INFO:teuthology.orchestra.run.smithi019.stdout: "mode": { 2024-03-18T15:28:36.694 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "mode", 2024-03-18T15:28:36.694 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.694 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.694 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.694 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "root", 2024-03-18T15:28:36.695 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.695 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.695 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:36.695 INFO:teuthology.orchestra.run.smithi019.stdout: "cephadm-package", 2024-03-18T15:28:36.695 INFO:teuthology.orchestra.run.smithi019.stdout: "root" 2024-03-18T15:28:36.695 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:36.695 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "mode for remote execution of cephadm", 2024-03-18T15:28:36.695 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.695 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.695 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.695 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.695 INFO:teuthology.orchestra.run.smithi019.stdout: "oob_default_addr": { 2024-03-18T15:28:36.695 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "oob_default_addr", 2024-03-18T15:28:36.695 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.695 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.696 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.696 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "169.254.1.1", 2024-03-18T15:28:36.696 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.696 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.696 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.696 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-18T15:28:36.696 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.696 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.696 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.696 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.696 INFO:teuthology.orchestra.run.smithi019.stdout: "prometheus_alerts_path": { 2024-03-18T15:28:36.696 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "prometheus_alerts_path", 2024-03-18T15:28:36.696 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.696 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.697 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.697 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-18T15:28:36.697 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.697 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.697 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.697 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-18T15:28:36.697 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.697 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.697 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.697 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.697 INFO:teuthology.orchestra.run.smithi019.stdout: "registry_insecure": { 2024-03-18T15:28:36.697 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "registry_insecure", 2024-03-18T15:28:36.697 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.697 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.697 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.698 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:36.698 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.698 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.698 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.698 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-18T15:28:36.698 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.698 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.698 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.698 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.698 INFO:teuthology.orchestra.run.smithi019.stdout: "registry_password": { 2024-03-18T15:28:36.698 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "registry_password", 2024-03-18T15:28:36.698 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.698 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.698 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.699 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:36.699 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.699 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.699 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.699 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-18T15:28:36.699 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.699 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.699 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.699 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.699 INFO:teuthology.orchestra.run.smithi019.stdout: "registry_url": { 2024-03-18T15:28:36.699 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "registry_url", 2024-03-18T15:28:36.700 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.700 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.700 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.700 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:36.700 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.700 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.700 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.700 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-18T15:28:36.700 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.700 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.700 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.700 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.700 INFO:teuthology.orchestra.run.smithi019.stdout: "registry_username": { 2024-03-18T15:28:36.700 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "registry_username", 2024-03-18T15:28:36.701 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.701 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.701 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.701 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:36.701 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.701 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.701 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.701 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-18T15:28:36.701 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.701 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.701 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.701 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.701 INFO:teuthology.orchestra.run.smithi019.stdout: "secure_monitoring_stack": { 2024-03-18T15:28:36.702 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "secure_monitoring_stack", 2024-03-18T15:28:36.702 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.702 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.702 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.702 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:36.702 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.702 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.702 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.702 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-18T15:28:36.702 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.702 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.702 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.702 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.702 INFO:teuthology.orchestra.run.smithi019.stdout: "service_discovery_port": { 2024-03-18T15:28:36.703 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "service_discovery_port", 2024-03-18T15:28:36.703 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:36.703 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.703 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.703 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "8765", 2024-03-18T15:28:36.703 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.703 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.703 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.703 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "cephadm service discovery port", 2024-03-18T15:28:36.703 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.703 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.703 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.703 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.703 INFO:teuthology.orchestra.run.smithi019.stdout: "ssh_config_file": { 2024-03-18T15:28:36.704 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "ssh_config_file", 2024-03-18T15:28:36.704 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.704 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.704 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.704 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:36.704 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.704 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.704 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.704 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-18T15:28:36.704 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.704 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.704 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.704 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.704 INFO:teuthology.orchestra.run.smithi019.stdout: "use_agent": { 2024-03-18T15:28:36.705 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "use_agent", 2024-03-18T15:28:36.705 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.705 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.705 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.705 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:36.705 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.705 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.705 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.705 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-18T15:28:36.705 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.705 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.705 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.705 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.705 INFO:teuthology.orchestra.run.smithi019.stdout: "use_repo_digest": { 2024-03-18T15:28:36.705 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "use_repo_digest", 2024-03-18T15:28:36.706 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.706 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.706 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.706 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:36.706 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.706 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.706 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.706 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-18T15:28:36.706 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.706 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.706 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.706 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.706 INFO:teuthology.orchestra.run.smithi019.stdout: "warn_on_failed_host_check": { 2024-03-18T15:28:36.706 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "warn_on_failed_host_check", 2024-03-18T15:28:36.707 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.707 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.707 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.707 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:36.707 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.707 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.707 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.707 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "raise a health warning if the host check fails", 2024-03-18T15:28:36.707 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.707 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.707 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.707 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.707 INFO:teuthology.orchestra.run.smithi019.stdout: "warn_on_stray_daemons": { 2024-03-18T15:28:36.707 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "warn_on_stray_daemons", 2024-03-18T15:28:36.708 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.708 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.708 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.708 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:36.708 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.708 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.708 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.708 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-18T15:28:36.708 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.708 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.708 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.708 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.708 INFO:teuthology.orchestra.run.smithi019.stdout: "warn_on_stray_hosts": { 2024-03-18T15:28:36.708 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "warn_on_stray_hosts", 2024-03-18T15:28:36.708 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.709 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.709 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.709 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:36.709 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.709 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.709 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.709 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-18T15:28:36.709 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.709 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.709 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.709 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:36.709 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:36.709 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.709 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:36.709 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "crash", 2024-03-18T15:28:36.710 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": true, 2024-03-18T15:28:36.710 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "", 2024-03-18T15:28:36.710 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:36.710 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:36.710 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:36.710 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.710 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.711 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.711 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:36.711 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.711 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.711 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:36.711 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:36.711 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:36.711 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:36.711 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:36.711 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:36.711 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:36.711 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:36.711 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.711 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.711 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.712 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.712 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.712 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:36.712 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:36.712 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.712 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.712 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.712 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:36.712 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.712 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.712 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.712 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.712 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.713 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.713 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.713 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.713 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:36.713 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:36.713 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.713 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.713 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.713 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:36.713 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.713 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.713 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:36.713 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:36.714 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:36.714 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:36.714 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:36.714 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:36.714 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:36.714 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:36.714 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.714 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.714 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.714 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.714 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.714 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:36.714 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:36.714 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.715 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.715 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.715 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:36.715 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.715 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.715 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.715 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.715 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.715 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.715 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.715 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.715 INFO:teuthology.orchestra.run.smithi019.stdout: "retain_interval": { 2024-03-18T15:28:36.715 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "retain_interval", 2024-03-18T15:28:36.715 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "secs", 2024-03-18T15:28:36.715 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.716 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.716 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "31536000", 2024-03-18T15:28:36.716 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.716 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.716 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.716 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-18T15:28:36.716 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.716 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.716 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.716 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.716 INFO:teuthology.orchestra.run.smithi019.stdout: "warn_recent_interval": { 2024-03-18T15:28:36.716 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "warn_recent_interval", 2024-03-18T15:28:36.716 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "secs", 2024-03-18T15:28:36.717 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.717 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.717 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "1209600", 2024-03-18T15:28:36.717 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.717 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.717 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.717 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-18T15:28:36.717 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.717 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.717 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.717 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:36.717 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:36.717 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.717 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:36.718 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "dashboard", 2024-03-18T15:28:36.718 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": true, 2024-03-18T15:28:36.718 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "", 2024-03-18T15:28:36.718 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:36.718 INFO:teuthology.orchestra.run.smithi019.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-18T15:28:36.718 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-18T15:28:36.718 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:36.718 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.718 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.718 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "10", 2024-03-18T15:28:36.718 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.718 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.718 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.718 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.719 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.719 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.719 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.719 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.719 INFO:teuthology.orchestra.run.smithi019.stdout: "ALERTMANAGER_API_HOST": { 2024-03-18T15:28:36.719 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-18T15:28:36.719 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.719 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.719 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.719 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:36.719 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.719 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.719 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.720 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.720 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.720 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.720 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.720 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.720 INFO:teuthology.orchestra.run.smithi019.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-18T15:28:36.720 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-18T15:28:36.720 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.720 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.720 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.720 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:36.720 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.721 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.721 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.721 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.721 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.721 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.721 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.721 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.721 INFO:teuthology.orchestra.run.smithi019.stdout: "AUDIT_API_ENABLED": { 2024-03-18T15:28:36.721 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "AUDIT_API_ENABLED", 2024-03-18T15:28:36.721 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.722 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.722 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.722 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:36.722 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.722 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.722 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.722 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.722 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.722 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.722 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.722 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.722 INFO:teuthology.orchestra.run.smithi019.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-18T15:28:36.722 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-18T15:28:36.722 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.723 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.723 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.723 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:36.723 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.723 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.723 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.723 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.723 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.724 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.724 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.724 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.724 INFO:teuthology.orchestra.run.smithi019.stdout: "ENABLE_BROWSABLE_API": { 2024-03-18T15:28:36.724 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-18T15:28:36.724 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.724 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.724 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.724 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:36.724 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.724 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.724 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.725 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.725 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.725 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.725 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.725 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.725 INFO:teuthology.orchestra.run.smithi019.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-18T15:28:36.725 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-18T15:28:36.725 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.725 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.725 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.725 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:36.725 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.725 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.725 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.726 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.726 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.726 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.726 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.726 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.726 INFO:teuthology.orchestra.run.smithi019.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-18T15:28:36.726 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-18T15:28:36.726 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.726 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.726 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.726 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:36.726 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.726 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.726 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.726 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.727 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.727 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.727 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.727 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.727 INFO:teuthology.orchestra.run.smithi019.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-18T15:28:36.727 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-18T15:28:36.727 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.727 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.728 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.728 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:36.728 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.728 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.728 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.728 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.728 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.729 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.729 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.729 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.729 INFO:teuthology.orchestra.run.smithi019.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-18T15:28:36.729 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-18T15:28:36.729 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.729 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.729 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.729 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:36.729 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.729 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.729 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.729 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.729 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.730 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.730 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.730 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.730 INFO:teuthology.orchestra.run.smithi019.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-18T15:28:36.730 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-18T15:28:36.730 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.730 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.730 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.730 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:36.730 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.730 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.730 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.730 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.730 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.731 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.731 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.731 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.731 INFO:teuthology.orchestra.run.smithi019.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-18T15:28:36.731 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-18T15:28:36.731 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.731 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.731 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.731 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:36.731 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.731 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.731 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.731 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.731 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.732 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.732 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.732 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.732 INFO:teuthology.orchestra.run.smithi019.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-18T15:28:36.732 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-18T15:28:36.732 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.732 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.732 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.733 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:36.733 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.733 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.733 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.733 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.733 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.733 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.733 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.733 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.733 INFO:teuthology.orchestra.run.smithi019.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-18T15:28:36.733 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-18T15:28:36.733 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.733 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.734 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.734 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:36.734 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.734 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.734 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.734 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.734 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.734 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.734 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.734 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.734 INFO:teuthology.orchestra.run.smithi019.stdout: "GRAFANA_API_PASSWORD": { 2024-03-18T15:28:36.734 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-18T15:28:36.734 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.734 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.735 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.735 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "admin", 2024-03-18T15:28:36.735 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.735 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.735 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.735 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.735 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.735 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.735 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.735 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.735 INFO:teuthology.orchestra.run.smithi019.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-18T15:28:36.735 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-18T15:28:36.735 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.735 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.736 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.736 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:36.736 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.736 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.736 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.736 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.736 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.736 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.736 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.736 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.736 INFO:teuthology.orchestra.run.smithi019.stdout: "GRAFANA_API_URL": { 2024-03-18T15:28:36.736 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "GRAFANA_API_URL", 2024-03-18T15:28:36.736 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.736 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.736 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.737 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:36.737 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.737 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.737 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.737 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.737 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.737 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.737 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.737 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.737 INFO:teuthology.orchestra.run.smithi019.stdout: "GRAFANA_API_USERNAME": { 2024-03-18T15:28:36.737 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-18T15:28:36.737 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.737 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.737 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.737 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "admin", 2024-03-18T15:28:36.738 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.738 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.738 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.738 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.738 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.738 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.738 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.738 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.738 INFO:teuthology.orchestra.run.smithi019.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-18T15:28:36.738 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-18T15:28:36.738 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.738 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.738 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.738 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:36.739 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.739 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.739 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.739 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.739 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.739 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.739 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.739 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.739 INFO:teuthology.orchestra.run.smithi019.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-18T15:28:36.739 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-18T15:28:36.739 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.739 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.739 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.739 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:36.740 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.740 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.740 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.740 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.740 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.740 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.740 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.740 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.740 INFO:teuthology.orchestra.run.smithi019.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-18T15:28:36.740 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-18T15:28:36.740 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.740 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.740 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.740 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:36.740 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.741 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.741 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.741 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.741 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.741 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.741 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.741 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.741 INFO:teuthology.orchestra.run.smithi019.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-18T15:28:36.741 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-18T15:28:36.741 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.741 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.741 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.741 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:36.742 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.742 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.742 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.742 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.742 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.742 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.742 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.742 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.742 INFO:teuthology.orchestra.run.smithi019.stdout: "MULTICLUSTER_CONFIG": { 2024-03-18T15:28:36.742 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-18T15:28:36.742 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.742 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.742 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.742 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "{}", 2024-03-18T15:28:36.743 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.743 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.743 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.743 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.743 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.743 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.743 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.743 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.743 INFO:teuthology.orchestra.run.smithi019.stdout: "PROMETHEUS_API_HOST": { 2024-03-18T15:28:36.743 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-18T15:28:36.743 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.743 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.743 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.743 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:36.744 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.744 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.744 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.744 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.744 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.744 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.744 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.744 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.744 INFO:teuthology.orchestra.run.smithi019.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-18T15:28:36.744 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-18T15:28:36.744 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.744 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.744 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.744 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:36.745 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.745 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.745 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.745 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.745 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.745 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.745 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.745 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.745 INFO:teuthology.orchestra.run.smithi019.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-18T15:28:36.745 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-18T15:28:36.745 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.745 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.745 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.745 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:36.746 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.746 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.746 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.746 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.746 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.746 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.746 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.746 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.746 INFO:teuthology.orchestra.run.smithi019.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-18T15:28:36.746 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-18T15:28:36.746 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.746 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.746 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.746 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:36.747 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.747 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.747 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.747 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.747 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.747 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.747 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.747 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.747 INFO:teuthology.orchestra.run.smithi019.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-18T15:28:36.747 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-18T15:28:36.747 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.747 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.747 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.747 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:36.748 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.748 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.748 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.748 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.748 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.748 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.748 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.748 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.748 INFO:teuthology.orchestra.run.smithi019.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-18T15:28:36.748 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-18T15:28:36.748 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.748 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.748 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.748 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:36.748 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.749 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.749 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.749 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.749 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.749 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.749 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.749 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.749 INFO:teuthology.orchestra.run.smithi019.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-18T15:28:36.749 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-18T15:28:36.749 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.749 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.749 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.749 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:36.749 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.750 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.750 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.750 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.750 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.750 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.750 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.750 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.750 INFO:teuthology.orchestra.run.smithi019.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-18T15:28:36.750 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-18T15:28:36.750 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.750 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.750 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.750 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:36.750 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.750 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.751 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.751 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.751 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.751 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.751 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.751 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.751 INFO:teuthology.orchestra.run.smithi019.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-18T15:28:36.751 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-18T15:28:36.751 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.751 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.751 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.751 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:36.751 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.751 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.752 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.752 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.752 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.752 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.752 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.752 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.752 INFO:teuthology.orchestra.run.smithi019.stdout: "PWD_POLICY_ENABLED": { 2024-03-18T15:28:36.752 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-18T15:28:36.752 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.752 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.752 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.752 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:36.752 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.752 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.752 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.753 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.753 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.753 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.753 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.753 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.753 INFO:teuthology.orchestra.run.smithi019.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-18T15:28:36.753 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-18T15:28:36.753 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.753 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.753 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.753 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-18T15:28:36.753 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.753 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.753 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.753 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.754 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.754 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.754 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.754 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.754 INFO:teuthology.orchestra.run.smithi019.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-18T15:28:36.754 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-18T15:28:36.754 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:36.754 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.754 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.754 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "10", 2024-03-18T15:28:36.754 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.755 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.755 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.755 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.755 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.755 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.755 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.755 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.755 INFO:teuthology.orchestra.run.smithi019.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-18T15:28:36.755 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-18T15:28:36.755 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:36.755 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.755 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.755 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "8", 2024-03-18T15:28:36.755 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.756 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.756 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.756 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.756 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.756 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.756 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.756 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.756 INFO:teuthology.orchestra.run.smithi019.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-18T15:28:36.756 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-18T15:28:36.756 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:36.756 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.756 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.756 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "45", 2024-03-18T15:28:36.756 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.757 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.757 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.757 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.757 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.757 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.757 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.757 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.757 INFO:teuthology.orchestra.run.smithi019.stdout: "RGW_API_ACCESS_KEY": { 2024-03-18T15:28:36.757 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-18T15:28:36.757 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.757 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.757 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.757 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:36.757 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.757 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.758 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.758 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.758 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.758 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.758 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.758 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.758 INFO:teuthology.orchestra.run.smithi019.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-18T15:28:36.758 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-18T15:28:36.758 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.758 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.758 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.758 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "admin", 2024-03-18T15:28:36.758 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.758 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.759 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.759 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.759 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.759 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.759 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.759 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.759 INFO:teuthology.orchestra.run.smithi019.stdout: "RGW_API_SECRET_KEY": { 2024-03-18T15:28:36.759 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-18T15:28:36.759 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.759 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.759 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.759 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:36.759 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.759 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.760 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.760 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.760 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.760 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.760 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.760 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.760 INFO:teuthology.orchestra.run.smithi019.stdout: "RGW_API_SSL_VERIFY": { 2024-03-18T15:28:36.760 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-18T15:28:36.760 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.760 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.760 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.760 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:36.760 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.760 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.760 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.761 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.761 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.761 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.761 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.761 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.761 INFO:teuthology.orchestra.run.smithi019.stdout: "UNSAFE_TLS_v1_2": { 2024-03-18T15:28:36.761 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-18T15:28:36.761 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.761 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.761 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.761 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:36.761 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.761 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.761 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.762 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.762 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.762 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.762 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.762 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.762 INFO:teuthology.orchestra.run.smithi019.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-18T15:28:36.762 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-18T15:28:36.762 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:36.762 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.762 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.762 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "0", 2024-03-18T15:28:36.762 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.762 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.762 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.763 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.763 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.763 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.763 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.763 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.763 INFO:teuthology.orchestra.run.smithi019.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-18T15:28:36.763 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-18T15:28:36.763 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:36.763 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.763 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.763 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "10", 2024-03-18T15:28:36.763 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.763 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.763 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.763 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.764 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.764 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.764 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.764 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.764 INFO:teuthology.orchestra.run.smithi019.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-18T15:28:36.764 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-18T15:28:36.764 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:36.764 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.764 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.764 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "5", 2024-03-18T15:28:36.764 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.764 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.764 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.764 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.765 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.765 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.765 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.765 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.765 INFO:teuthology.orchestra.run.smithi019.stdout: "cross_origin_url": { 2024-03-18T15:28:36.765 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "cross_origin_url", 2024-03-18T15:28:36.765 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.765 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.765 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.765 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:36.765 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.765 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.765 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.765 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.765 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.766 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.766 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.766 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.766 INFO:teuthology.orchestra.run.smithi019.stdout: "crt_file": { 2024-03-18T15:28:36.766 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "crt_file", 2024-03-18T15:28:36.766 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.766 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.766 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.766 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:36.766 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.766 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.766 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.766 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.766 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.767 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.767 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.767 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.767 INFO:teuthology.orchestra.run.smithi019.stdout: "debug": { 2024-03-18T15:28:36.767 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "debug", 2024-03-18T15:28:36.767 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.767 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.767 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.767 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:36.767 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.767 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.767 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.767 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Enable/disable debug options", 2024-03-18T15:28:36.768 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.768 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.768 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.768 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.768 INFO:teuthology.orchestra.run.smithi019.stdout: "jwt_token_ttl": { 2024-03-18T15:28:36.768 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "jwt_token_ttl", 2024-03-18T15:28:36.768 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:36.768 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.768 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.768 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "28800", 2024-03-18T15:28:36.768 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.768 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.768 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.768 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.768 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.769 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.769 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.769 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.769 INFO:teuthology.orchestra.run.smithi019.stdout: "key_file": { 2024-03-18T15:28:36.769 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "key_file", 2024-03-18T15:28:36.769 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.769 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.769 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.769 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:36.769 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.769 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.769 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.769 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.769 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.770 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.770 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.770 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.770 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:36.770 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:36.770 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.770 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.770 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.770 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:36.770 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.770 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.770 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:36.770 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:36.770 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:36.771 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:36.771 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:36.771 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:36.771 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:36.771 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:36.771 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.771 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.771 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.771 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.771 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.771 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:36.771 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:36.771 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.771 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.772 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.772 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:36.772 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.772 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.772 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.772 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.772 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.772 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.772 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.772 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.772 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:36.772 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:36.772 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.772 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.773 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.773 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:36.773 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.773 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.773 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:36.773 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:36.773 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:36.773 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:36.773 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:36.773 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:36.773 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:36.773 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:36.773 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.774 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.774 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.774 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.774 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.774 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:36.774 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:36.774 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.774 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.774 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.774 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:36.774 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.774 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.774 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.774 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.774 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.775 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.775 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.775 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.775 INFO:teuthology.orchestra.run.smithi019.stdout: "motd": { 2024-03-18T15:28:36.775 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "motd", 2024-03-18T15:28:36.775 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.775 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.775 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.775 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:36.775 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.775 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.775 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.775 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "The message of the day", 2024-03-18T15:28:36.775 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.776 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.776 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.776 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.776 INFO:teuthology.orchestra.run.smithi019.stdout: "redirect_resolve_ip_addr": { 2024-03-18T15:28:36.776 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "redirect_resolve_ip_addr", 2024-03-18T15:28:36.776 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.776 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.776 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.776 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:36.776 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.776 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.776 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.776 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.776 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.776 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.777 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.777 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.777 INFO:teuthology.orchestra.run.smithi019.stdout: "server_addr": { 2024-03-18T15:28:36.777 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "server_addr", 2024-03-18T15:28:36.777 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.777 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.777 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.777 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "::", 2024-03-18T15:28:36.777 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.777 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.777 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.777 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.777 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.777 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.778 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.778 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.778 INFO:teuthology.orchestra.run.smithi019.stdout: "server_port": { 2024-03-18T15:28:36.778 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "server_port", 2024-03-18T15:28:36.778 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:36.778 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.778 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.778 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "8080", 2024-03-18T15:28:36.778 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.778 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.778 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.778 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.778 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.778 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.778 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.779 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.779 INFO:teuthology.orchestra.run.smithi019.stdout: "ssl": { 2024-03-18T15:28:36.779 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "ssl", 2024-03-18T15:28:36.779 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.779 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.779 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.779 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:36.779 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.779 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.779 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.779 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.779 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.779 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.779 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.780 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.780 INFO:teuthology.orchestra.run.smithi019.stdout: "ssl_server_port": { 2024-03-18T15:28:36.780 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "ssl_server_port", 2024-03-18T15:28:36.780 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:36.780 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.780 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.780 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "8443", 2024-03-18T15:28:36.780 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.780 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.780 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.780 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.780 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.780 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.780 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.780 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.781 INFO:teuthology.orchestra.run.smithi019.stdout: "standby_behaviour": { 2024-03-18T15:28:36.781 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "standby_behaviour", 2024-03-18T15:28:36.781 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.781 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.781 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.781 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "redirect", 2024-03-18T15:28:36.781 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.781 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.781 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:36.781 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:36.781 INFO:teuthology.orchestra.run.smithi019.stdout: "redirect" 2024-03-18T15:28:36.781 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:36.781 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.782 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.782 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.782 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.782 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.782 INFO:teuthology.orchestra.run.smithi019.stdout: "standby_error_status_code": { 2024-03-18T15:28:36.782 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "standby_error_status_code", 2024-03-18T15:28:36.782 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:36.782 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.782 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.782 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "500", 2024-03-18T15:28:36.782 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "400", 2024-03-18T15:28:36.782 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "599", 2024-03-18T15:28:36.782 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.782 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.783 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.783 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.783 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.783 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.783 INFO:teuthology.orchestra.run.smithi019.stdout: "url_prefix": { 2024-03-18T15:28:36.783 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "url_prefix", 2024-03-18T15:28:36.783 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.783 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.783 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.783 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:36.783 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.783 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.783 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.783 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.784 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.784 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.784 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.784 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:36.784 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:36.784 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.784 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:36.784 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "devicehealth", 2024-03-18T15:28:36.784 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": true, 2024-03-18T15:28:36.784 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "", 2024-03-18T15:28:36.784 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:36.784 INFO:teuthology.orchestra.run.smithi019.stdout: "enable_monitoring": { 2024-03-18T15:28:36.784 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "enable_monitoring", 2024-03-18T15:28:36.784 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.785 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.785 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.785 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:36.785 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.785 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.785 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.785 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "monitor device health metrics", 2024-03-18T15:28:36.785 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.785 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.785 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.785 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.785 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:36.785 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:36.785 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.786 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.786 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.786 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:36.786 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.786 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.786 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:36.786 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:36.786 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:36.786 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:36.786 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:36.786 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:36.786 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:36.786 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:36.786 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.786 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.787 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.787 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.787 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.787 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:36.787 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:36.787 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.787 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.787 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.787 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:36.787 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.787 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.787 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.787 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.787 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.788 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.788 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.788 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.788 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:36.788 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:36.788 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.788 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.788 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.788 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:36.788 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.788 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.788 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:36.788 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:36.788 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:36.788 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:36.789 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:36.789 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:36.789 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:36.789 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:36.789 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.789 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.789 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.789 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.789 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.789 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:36.789 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:36.789 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.789 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.789 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.790 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:36.790 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.790 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.790 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.790 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.790 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.790 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.790 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.790 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.790 INFO:teuthology.orchestra.run.smithi019.stdout: "mark_out_threshold": { 2024-03-18T15:28:36.790 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "mark_out_threshold", 2024-03-18T15:28:36.790 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "secs", 2024-03-18T15:28:36.790 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.790 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.790 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "2419200", 2024-03-18T15:28:36.791 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.791 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.791 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.791 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-18T15:28:36.791 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.791 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.791 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.791 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.791 INFO:teuthology.orchestra.run.smithi019.stdout: "pool_name": { 2024-03-18T15:28:36.791 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "pool_name", 2024-03-18T15:28:36.791 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.791 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.791 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.791 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "device_health_metrics", 2024-03-18T15:28:36.792 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.792 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.792 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.792 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-18T15:28:36.792 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.792 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.792 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.792 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.792 INFO:teuthology.orchestra.run.smithi019.stdout: "retention_period": { 2024-03-18T15:28:36.792 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "retention_period", 2024-03-18T15:28:36.792 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "secs", 2024-03-18T15:28:36.792 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.792 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.792 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "15552000", 2024-03-18T15:28:36.793 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.793 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.793 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.793 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "how long to retain device health metrics", 2024-03-18T15:28:36.793 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.793 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.793 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.793 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.793 INFO:teuthology.orchestra.run.smithi019.stdout: "scrape_frequency": { 2024-03-18T15:28:36.793 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "scrape_frequency", 2024-03-18T15:28:36.793 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "secs", 2024-03-18T15:28:36.793 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.793 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.793 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "86400", 2024-03-18T15:28:36.794 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.794 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.794 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.794 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-18T15:28:36.794 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.794 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.794 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.794 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.794 INFO:teuthology.orchestra.run.smithi019.stdout: "self_heal": { 2024-03-18T15:28:36.794 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "self_heal", 2024-03-18T15:28:36.794 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.794 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.794 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.795 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:36.795 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.795 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.795 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.795 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-18T15:28:36.795 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.795 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.795 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.795 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.795 INFO:teuthology.orchestra.run.smithi019.stdout: "sleep_interval": { 2024-03-18T15:28:36.795 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "sleep_interval", 2024-03-18T15:28:36.795 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "secs", 2024-03-18T15:28:36.795 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.795 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.796 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "600", 2024-03-18T15:28:36.796 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.796 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.796 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.796 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "how frequently to wake up and check device health", 2024-03-18T15:28:36.796 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.796 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.796 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.796 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.796 INFO:teuthology.orchestra.run.smithi019.stdout: "warn_threshold": { 2024-03-18T15:28:36.796 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "warn_threshold", 2024-03-18T15:28:36.796 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "secs", 2024-03-18T15:28:36.796 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.796 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.796 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "7257600", 2024-03-18T15:28:36.797 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.797 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.797 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.797 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-18T15:28:36.797 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.797 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.797 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.797 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:36.797 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:36.797 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.797 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:36.797 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "diskprediction_local", 2024-03-18T15:28:36.797 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": true, 2024-03-18T15:28:36.797 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "", 2024-03-18T15:28:36.798 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:36.798 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:36.798 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:36.798 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.798 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.798 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.798 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:36.798 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.798 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.798 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:36.798 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:36.798 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:36.798 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:36.798 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:36.798 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:36.799 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:36.799 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:36.799 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.799 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.799 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.799 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.799 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.799 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:36.799 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:36.799 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.799 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.799 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.799 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:36.799 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.800 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.800 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.800 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.800 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.800 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.800 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.800 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.800 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:36.800 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:36.800 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.800 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.800 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.800 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:36.800 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.801 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.801 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:36.801 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:36.801 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:36.801 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:36.801 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:36.801 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:36.801 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:36.801 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:36.801 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.801 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.801 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.801 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.801 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.801 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:36.802 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:36.802 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.802 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.802 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.802 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:36.802 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.802 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.802 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.802 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.802 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.802 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.802 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.802 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.802 INFO:teuthology.orchestra.run.smithi019.stdout: "predict_interval": { 2024-03-18T15:28:36.803 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "predict_interval", 2024-03-18T15:28:36.803 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.803 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.803 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.803 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "86400", 2024-03-18T15:28:36.803 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.803 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.803 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.803 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.803 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.803 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.803 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.803 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.803 INFO:teuthology.orchestra.run.smithi019.stdout: "predictor_model": { 2024-03-18T15:28:36.804 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "predictor_model", 2024-03-18T15:28:36.804 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.804 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.804 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.804 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "prophetstor", 2024-03-18T15:28:36.804 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.804 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.804 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.804 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.804 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.804 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.804 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.804 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.804 INFO:teuthology.orchestra.run.smithi019.stdout: "sleep_interval": { 2024-03-18T15:28:36.804 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "sleep_interval", 2024-03-18T15:28:36.805 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.805 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.805 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.805 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "600", 2024-03-18T15:28:36.805 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.805 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.805 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.805 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.805 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.805 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.805 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.805 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:36.805 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:36.805 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.805 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:36.806 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "influx", 2024-03-18T15:28:36.806 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": false, 2024-03-18T15:28:36.806 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "influxdb python module not found", 2024-03-18T15:28:36.806 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:36.806 INFO:teuthology.orchestra.run.smithi019.stdout: "batch_size": { 2024-03-18T15:28:36.806 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "batch_size", 2024-03-18T15:28:36.806 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:36.806 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.806 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.806 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "5000", 2024-03-18T15:28:36.806 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.806 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.806 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.806 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-18T15:28:36.807 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.807 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.807 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.807 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.807 INFO:teuthology.orchestra.run.smithi019.stdout: "database": { 2024-03-18T15:28:36.807 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "database", 2024-03-18T15:28:36.807 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.807 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.807 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.807 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "ceph", 2024-03-18T15:28:36.807 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.807 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.807 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.808 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-18T15:28:36.808 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.808 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.808 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.808 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.808 INFO:teuthology.orchestra.run.smithi019.stdout: "hostname": { 2024-03-18T15:28:36.808 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "hostname", 2024-03-18T15:28:36.808 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.808 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.808 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.808 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:36.808 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.808 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.808 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.809 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "InfluxDB server hostname", 2024-03-18T15:28:36.809 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.809 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.809 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.809 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.809 INFO:teuthology.orchestra.run.smithi019.stdout: "interval": { 2024-03-18T15:28:36.809 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "interval", 2024-03-18T15:28:36.809 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "secs", 2024-03-18T15:28:36.809 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.809 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.809 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "30", 2024-03-18T15:28:36.809 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "5", 2024-03-18T15:28:36.809 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.809 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.810 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-18T15:28:36.810 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.810 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.810 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.810 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.810 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:36.810 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:36.810 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.810 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.810 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.810 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:36.810 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.810 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.810 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:36.811 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:36.811 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:36.811 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:36.811 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:36.811 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:36.811 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:36.811 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:36.811 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.811 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.811 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.811 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.811 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.811 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:36.811 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:36.811 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.812 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.812 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.812 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:36.812 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.812 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.812 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.812 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.812 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.812 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.812 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.812 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.812 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:36.812 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:36.812 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.813 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.813 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.813 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:36.813 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.813 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.813 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:36.813 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:36.813 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:36.813 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:36.813 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:36.813 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:36.813 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:36.813 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:36.813 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.814 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.814 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.814 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.814 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.814 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:36.814 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:36.814 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.814 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.814 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.814 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:36.814 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.814 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.814 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.815 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.815 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.815 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.815 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.815 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.815 INFO:teuthology.orchestra.run.smithi019.stdout: "password": { 2024-03-18T15:28:36.815 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "password", 2024-03-18T15:28:36.815 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.815 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.815 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.815 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:36.815 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.815 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.815 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.816 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "password of InfluxDB server user", 2024-03-18T15:28:36.816 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.816 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.816 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.816 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.816 INFO:teuthology.orchestra.run.smithi019.stdout: "port": { 2024-03-18T15:28:36.816 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "port", 2024-03-18T15:28:36.816 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:36.816 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.816 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.817 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "8086", 2024-03-18T15:28:36.817 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.817 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.817 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.817 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "InfluxDB server port", 2024-03-18T15:28:36.817 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.817 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.817 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.817 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.817 INFO:teuthology.orchestra.run.smithi019.stdout: "ssl": { 2024-03-18T15:28:36.817 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "ssl", 2024-03-18T15:28:36.817 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.817 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.817 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.818 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "false", 2024-03-18T15:28:36.818 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.818 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.818 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.818 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-18T15:28:36.818 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.818 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.818 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.818 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.818 INFO:teuthology.orchestra.run.smithi019.stdout: "threads": { 2024-03-18T15:28:36.818 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "threads", 2024-03-18T15:28:36.818 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:36.818 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.818 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.819 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "5", 2024-03-18T15:28:36.819 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "1", 2024-03-18T15:28:36.819 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "32", 2024-03-18T15:28:36.819 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.819 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-18T15:28:36.819 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.819 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.819 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.819 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.819 INFO:teuthology.orchestra.run.smithi019.stdout: "username": { 2024-03-18T15:28:36.819 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "username", 2024-03-18T15:28:36.819 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.819 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.819 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.819 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:36.820 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.820 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.820 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.820 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "username of InfluxDB server user", 2024-03-18T15:28:36.820 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.820 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.820 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.820 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.820 INFO:teuthology.orchestra.run.smithi019.stdout: "verify_ssl": { 2024-03-18T15:28:36.820 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "verify_ssl", 2024-03-18T15:28:36.820 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.820 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.820 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.821 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "true", 2024-03-18T15:28:36.821 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.821 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.821 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.821 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-18T15:28:36.821 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.821 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.821 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.821 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:36.821 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:36.821 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.821 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:36.821 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "insights", 2024-03-18T15:28:36.821 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": true, 2024-03-18T15:28:36.822 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "", 2024-03-18T15:28:36.822 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:36.822 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:36.822 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:36.822 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.822 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.822 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.822 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:36.822 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.822 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.822 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:36.822 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:36.822 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:36.822 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:36.823 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:36.823 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:36.823 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:36.823 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:36.823 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.823 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.823 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.823 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.823 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.823 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:36.823 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:36.823 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.823 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.823 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.823 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:36.824 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.824 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.824 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.824 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.824 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.824 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.824 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.824 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.824 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:36.824 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:36.824 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.824 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.824 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.825 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:36.825 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.825 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.825 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:36.825 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:36.825 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:36.825 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:36.825 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:36.825 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:36.825 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:36.825 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:36.825 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.825 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.825 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.825 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.826 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.826 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:36.826 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:36.826 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.826 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.826 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.826 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:36.826 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.826 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.826 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.826 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.826 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.826 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.826 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.827 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:36.827 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:36.827 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.827 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:36.827 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "iostat", 2024-03-18T15:28:36.827 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": true, 2024-03-18T15:28:36.827 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "", 2024-03-18T15:28:36.827 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:36.827 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:36.827 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:36.827 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.827 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.827 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.828 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:36.828 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.828 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.828 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:36.828 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:36.828 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:36.828 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:36.828 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:36.828 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:36.828 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:36.828 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:36.828 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.828 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.828 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.829 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.829 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.829 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:36.829 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:36.829 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.829 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.829 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.829 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:36.829 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.829 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.829 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.829 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.829 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.829 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.829 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.830 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.830 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:36.830 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:36.830 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.830 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.830 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.830 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:36.830 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.830 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.830 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:36.830 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:36.830 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:36.830 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:36.830 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:36.831 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:36.831 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:36.831 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:36.831 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.831 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.831 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.831 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.831 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.831 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:36.831 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:36.831 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.831 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.831 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.831 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:36.831 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.832 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.832 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.832 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.832 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.832 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.832 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.832 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:36.832 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:36.832 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.832 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:36.832 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "k8sevents", 2024-03-18T15:28:36.832 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": true, 2024-03-18T15:28:36.832 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "", 2024-03-18T15:28:36.832 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:36.833 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph_event_retention_days": { 2024-03-18T15:28:36.833 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "ceph_event_retention_days", 2024-03-18T15:28:36.833 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:36.833 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.833 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.833 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "7", 2024-03-18T15:28:36.833 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.833 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.833 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.833 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-18T15:28:36.833 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.833 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.833 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.833 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.834 INFO:teuthology.orchestra.run.smithi019.stdout: "config_check_secs": { 2024-03-18T15:28:36.834 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "config_check_secs", 2024-03-18T15:28:36.834 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:36.834 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.834 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.834 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "10", 2024-03-18T15:28:36.834 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "10", 2024-03-18T15:28:36.834 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.834 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.834 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-18T15:28:36.834 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.834 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.835 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.835 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.835 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:36.835 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:36.835 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.835 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.835 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.835 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:36.835 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.835 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.835 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:36.835 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:36.835 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:36.835 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:36.836 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:36.836 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:36.836 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:36.836 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:36.836 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.836 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.836 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.836 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.836 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.836 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:36.836 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:36.836 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.836 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.836 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.836 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:36.837 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.837 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.837 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.837 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.837 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.837 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.837 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.837 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.837 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:36.837 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:36.837 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.837 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.837 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.837 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:36.838 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.838 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.838 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:36.838 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:36.838 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:36.838 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:36.838 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:36.838 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:36.838 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:36.838 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:36.838 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.838 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.838 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.838 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.839 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.839 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:36.839 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:36.839 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.839 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.839 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.839 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:36.839 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.839 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.839 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.839 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.839 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.839 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.839 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.839 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:36.840 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:36.840 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.840 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:36.840 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "localpool", 2024-03-18T15:28:36.840 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": true, 2024-03-18T15:28:36.840 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "", 2024-03-18T15:28:36.840 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:36.840 INFO:teuthology.orchestra.run.smithi019.stdout: "failure_domain": { 2024-03-18T15:28:36.840 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "failure_domain", 2024-03-18T15:28:36.840 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.840 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.840 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.840 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "host", 2024-03-18T15:28:36.840 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.841 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.841 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.841 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "failure domain for any created local pool", 2024-03-18T15:28:36.841 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-18T15:28:36.841 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.841 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.841 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.841 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:36.841 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:36.841 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.841 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.841 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.841 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:36.841 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.841 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.842 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:36.842 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:36.842 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:36.842 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:36.842 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:36.842 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:36.842 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:36.842 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:36.842 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.842 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.842 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.842 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.842 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.842 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:36.843 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:36.843 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.843 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.843 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.843 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:36.843 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.843 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.843 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.843 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.843 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.843 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.843 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.843 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.843 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:36.844 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:36.844 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.844 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.844 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.844 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:36.844 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.844 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.844 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:36.844 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:36.844 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:36.844 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:36.844 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:36.844 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:36.844 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:36.844 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:36.845 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.845 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.845 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.845 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.845 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.845 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:36.845 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:36.845 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.845 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.845 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.845 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:36.845 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.845 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.845 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.845 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.846 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.846 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.846 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.846 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.846 INFO:teuthology.orchestra.run.smithi019.stdout: "min_size": { 2024-03-18T15:28:36.846 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "min_size", 2024-03-18T15:28:36.846 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:36.846 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.846 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.846 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:36.846 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.846 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.846 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.846 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "default min_size for any created local pool", 2024-03-18T15:28:36.847 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-18T15:28:36.847 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.847 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.847 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.847 INFO:teuthology.orchestra.run.smithi019.stdout: "num_rep": { 2024-03-18T15:28:36.847 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "num_rep", 2024-03-18T15:28:36.847 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:36.847 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.847 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.847 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "3", 2024-03-18T15:28:36.847 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.847 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.847 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.847 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "default replica count for any created local pool", 2024-03-18T15:28:36.848 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.848 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.848 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.848 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.848 INFO:teuthology.orchestra.run.smithi019.stdout: "pg_num": { 2024-03-18T15:28:36.848 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "pg_num", 2024-03-18T15:28:36.848 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:36.848 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.848 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.848 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "128", 2024-03-18T15:28:36.848 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.848 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.848 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.848 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "default pg_num for any created local pool", 2024-03-18T15:28:36.849 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.849 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.849 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.849 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.849 INFO:teuthology.orchestra.run.smithi019.stdout: "prefix": { 2024-03-18T15:28:36.849 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "prefix", 2024-03-18T15:28:36.849 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.849 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.849 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.849 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:36.849 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.849 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.849 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.849 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "name prefix for any created local pool", 2024-03-18T15:28:36.850 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.850 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.850 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.850 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.850 INFO:teuthology.orchestra.run.smithi019.stdout: "subtree": { 2024-03-18T15:28:36.850 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "subtree", 2024-03-18T15:28:36.850 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.850 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.850 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.850 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "rack", 2024-03-18T15:28:36.850 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.850 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.850 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.850 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-18T15:28:36.851 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-18T15:28:36.853 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.853 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.854 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:36.854 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:36.854 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.854 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:36.854 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "mds_autoscaler", 2024-03-18T15:28:36.854 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": true, 2024-03-18T15:28:36.854 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "", 2024-03-18T15:28:36.854 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:36.854 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:36.854 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:36.854 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.854 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.854 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.855 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:36.855 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.855 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.855 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:36.855 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:36.855 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:36.855 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:36.855 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:36.855 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:36.855 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:36.855 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:36.855 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.855 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.855 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.855 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.856 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.856 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:36.856 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:36.856 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.856 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.856 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.856 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:36.856 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.856 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.856 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.856 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.856 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.856 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.856 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.857 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.857 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:36.857 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:36.857 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.857 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.857 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.857 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:36.857 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.857 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.857 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:36.857 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:36.857 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:36.857 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:36.857 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:36.857 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:36.858 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:36.858 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:36.858 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.858 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.858 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.858 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.858 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.858 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:36.858 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:36.858 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.858 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.858 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.858 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:36.858 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.859 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.859 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.859 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.859 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.859 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.859 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.859 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:36.859 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:36.859 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.859 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:36.859 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "mirroring", 2024-03-18T15:28:36.859 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": true, 2024-03-18T15:28:36.859 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "", 2024-03-18T15:28:36.859 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:36.860 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:36.860 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:36.860 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.860 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.860 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.860 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:36.860 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.860 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.860 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:36.860 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:36.860 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:36.860 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:36.860 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:36.860 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:36.860 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:36.861 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:36.861 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.861 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.861 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.861 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.861 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.861 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:36.861 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:36.861 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.861 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.861 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.861 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:36.861 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.861 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.861 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.862 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.862 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.862 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.862 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.862 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.862 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:36.862 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:36.862 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.862 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.862 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.862 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:36.862 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.862 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.862 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:36.863 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:36.863 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:36.863 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:36.863 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:36.863 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:36.863 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:36.863 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:36.863 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.863 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.863 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.863 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.863 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.863 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:36.863 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:36.864 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.864 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.864 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.864 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:36.864 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.864 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.864 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.864 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.864 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.864 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.864 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.864 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:36.865 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:36.865 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.865 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:36.865 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "nfs", 2024-03-18T15:28:36.865 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": true, 2024-03-18T15:28:36.865 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "", 2024-03-18T15:28:36.865 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:36.865 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:36.865 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:36.865 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.865 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.865 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.865 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:36.865 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.866 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.866 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:36.866 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:36.866 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:36.866 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:36.866 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:36.866 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:36.866 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:36.866 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:36.866 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.866 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.866 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.866 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.866 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.867 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:36.867 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:36.867 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.867 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.867 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.867 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:36.867 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.867 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.867 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.867 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.867 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.867 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.867 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.867 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.868 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:36.868 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:36.868 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.868 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.868 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.868 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:36.868 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.868 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.868 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:36.868 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:36.868 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:36.868 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:36.868 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:36.868 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:36.868 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:36.869 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:36.869 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.869 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.869 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.869 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.869 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.869 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:36.869 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:36.869 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.869 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.869 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.869 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:36.869 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.869 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.870 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.870 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.870 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.870 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.870 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.870 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:36.870 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:36.870 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.870 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:36.870 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "orchestrator", 2024-03-18T15:28:36.870 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": true, 2024-03-18T15:28:36.870 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "", 2024-03-18T15:28:36.870 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:36.870 INFO:teuthology.orchestra.run.smithi019.stdout: "fail_fs": { 2024-03-18T15:28:36.871 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "fail_fs", 2024-03-18T15:28:36.871 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.871 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.871 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.871 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:36.871 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.871 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.871 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.871 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-18T15:28:36.871 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.871 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.871 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.871 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.871 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:36.871 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:36.872 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.872 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.872 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.872 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:36.872 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.872 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.872 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:36.872 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:36.872 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:36.872 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:36.872 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:36.872 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:36.872 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:36.872 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:36.872 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.873 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.873 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.873 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.873 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.873 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:36.873 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:36.873 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.873 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.873 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.873 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:36.873 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.873 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.873 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.873 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.874 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.874 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.874 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.874 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.874 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:36.874 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:36.874 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.874 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.874 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.874 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:36.874 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.874 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.874 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:36.874 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:36.875 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:36.875 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:36.875 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:36.875 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:36.875 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:36.875 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:36.875 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.875 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.875 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.875 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.875 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.875 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:36.875 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:36.875 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.875 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.876 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.876 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:36.876 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.876 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.876 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.876 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.877 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.877 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.877 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.877 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.877 INFO:teuthology.orchestra.run.smithi019.stdout: "orchestrator": { 2024-03-18T15:28:36.877 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "orchestrator", 2024-03-18T15:28:36.877 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.877 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.877 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.877 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:36.877 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.877 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.877 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:36.878 INFO:teuthology.orchestra.run.smithi019.stdout: "cephadm", 2024-03-18T15:28:36.878 INFO:teuthology.orchestra.run.smithi019.stdout: "rook", 2024-03-18T15:28:36.878 INFO:teuthology.orchestra.run.smithi019.stdout: "test_orchestrator" 2024-03-18T15:28:36.878 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:36.878 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Orchestrator backend", 2024-03-18T15:28:36.878 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.878 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.878 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.878 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:36.878 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:36.878 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.879 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:36.879 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "osd_perf_query", 2024-03-18T15:28:36.879 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": true, 2024-03-18T15:28:36.879 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "", 2024-03-18T15:28:36.879 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:36.879 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:36.879 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:36.879 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.879 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.879 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.879 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:36.879 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.879 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.880 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:36.880 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:36.880 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:36.880 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:36.880 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:36.880 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:36.880 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:36.880 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:36.880 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.880 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.880 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.880 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.880 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.880 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:36.880 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:36.881 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.881 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.881 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.881 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:36.881 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.881 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.881 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.881 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.881 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.881 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.881 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.881 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.881 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:36.881 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:36.882 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.882 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.882 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.882 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:36.882 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.882 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.882 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:36.882 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:36.882 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:36.882 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:36.882 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:36.882 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:36.882 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:36.882 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:36.883 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.883 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.883 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.883 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.883 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.883 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:36.883 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:36.883 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.883 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.883 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.883 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:36.883 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.883 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.883 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.883 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.884 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.884 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.884 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.884 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:36.884 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:36.884 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.884 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:36.884 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "osd_support", 2024-03-18T15:28:36.884 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": true, 2024-03-18T15:28:36.884 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "", 2024-03-18T15:28:36.884 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:36.884 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:36.884 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:36.885 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.885 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.885 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.885 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:36.885 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.885 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.885 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:36.885 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:36.885 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:36.885 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:36.885 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:36.885 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:36.885 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:36.885 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:36.885 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.886 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.886 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.886 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.886 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.886 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:36.886 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:36.886 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.886 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.886 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.886 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:36.886 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.886 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.886 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.886 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.887 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.887 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.887 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.887 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.887 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:36.887 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:36.887 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.887 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.887 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.887 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:36.887 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.887 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.887 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:36.887 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:36.887 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:36.888 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:36.888 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:36.888 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:36.888 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:36.888 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:36.888 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.888 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.888 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.888 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.888 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.888 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:36.888 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:36.888 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.888 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.888 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.889 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:36.889 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.889 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.889 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.889 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.889 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.889 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.889 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.889 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:36.889 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:36.889 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.889 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:36.889 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "pg_autoscaler", 2024-03-18T15:28:36.889 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": true, 2024-03-18T15:28:36.890 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "", 2024-03-18T15:28:36.890 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:36.890 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:36.890 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:36.890 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.890 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.890 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.890 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:36.890 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.890 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.890 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:36.890 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:36.890 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:36.890 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:36.890 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:36.891 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:36.891 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:36.891 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:36.891 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.891 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.891 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.891 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.891 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.891 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:36.891 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:36.891 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.891 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.891 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.891 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:36.891 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.892 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.892 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.892 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.892 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.892 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.892 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.892 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.892 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:36.892 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:36.892 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.892 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.892 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.892 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:36.892 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.893 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.893 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:36.893 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:36.893 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:36.893 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:36.893 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:36.893 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:36.893 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:36.893 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:36.893 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.893 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.893 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.894 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.894 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.894 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:36.894 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:36.894 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.894 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.894 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.894 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:36.894 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.894 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.894 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.894 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.894 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.894 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.895 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.895 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.895 INFO:teuthology.orchestra.run.smithi019.stdout: "sleep_interval": { 2024-03-18T15:28:36.895 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "sleep_interval", 2024-03-18T15:28:36.895 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "secs", 2024-03-18T15:28:36.895 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.895 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.895 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "60", 2024-03-18T15:28:36.895 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.895 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.895 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.895 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.895 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.895 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.896 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.896 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.896 INFO:teuthology.orchestra.run.smithi019.stdout: "threshold": { 2024-03-18T15:28:36.896 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "threshold", 2024-03-18T15:28:36.896 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "float", 2024-03-18T15:28:36.896 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.896 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.896 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "3.0", 2024-03-18T15:28:36.896 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "1.0", 2024-03-18T15:28:36.896 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.896 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.896 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "scaling threshold", 2024-03-18T15:28:36.896 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-18T15:28:36.896 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.896 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.897 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:36.897 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:36.897 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.897 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:36.897 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "progress", 2024-03-18T15:28:36.897 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": true, 2024-03-18T15:28:36.897 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "", 2024-03-18T15:28:36.897 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:36.897 INFO:teuthology.orchestra.run.smithi019.stdout: "allow_pg_recovery_event": { 2024-03-18T15:28:36.897 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "allow_pg_recovery_event", 2024-03-18T15:28:36.897 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.897 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.897 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.897 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:36.898 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.898 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.898 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.898 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-18T15:28:36.898 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.898 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.898 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.898 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.898 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled": { 2024-03-18T15:28:36.898 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "enabled", 2024-03-18T15:28:36.898 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.898 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.898 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.898 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:36.899 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.899 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.899 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.899 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.899 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.899 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.899 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.899 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.899 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:36.899 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:36.899 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.899 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.899 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.899 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:36.900 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.900 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.900 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:36.900 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:36.900 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:36.900 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:36.900 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:36.900 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:36.900 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:36.900 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:36.900 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.900 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.900 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.900 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.900 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.901 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:36.901 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:36.901 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.901 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.901 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.901 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:36.901 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.901 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.901 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.901 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.901 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.901 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.901 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.901 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.901 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:36.902 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:36.902 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.902 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.902 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.902 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:36.902 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.902 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.902 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:36.902 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:36.902 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:36.902 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:36.902 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:36.902 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:36.902 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:36.903 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:36.903 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.903 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.903 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.903 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.903 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.903 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:36.903 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:36.903 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.903 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.903 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.903 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:36.903 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.903 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.903 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.904 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.904 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.904 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.904 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.904 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.904 INFO:teuthology.orchestra.run.smithi019.stdout: "max_completed_events": { 2024-03-18T15:28:36.904 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "max_completed_events", 2024-03-18T15:28:36.904 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:36.904 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.904 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.904 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "50", 2024-03-18T15:28:36.904 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.904 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.904 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.904 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "number of past completed events to remember", 2024-03-18T15:28:36.905 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.905 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.905 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.905 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.905 INFO:teuthology.orchestra.run.smithi019.stdout: "sleep_interval": { 2024-03-18T15:28:36.905 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "sleep_interval", 2024-03-18T15:28:36.905 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "secs", 2024-03-18T15:28:36.905 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.905 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.905 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "5", 2024-03-18T15:28:36.905 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.905 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.905 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.905 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "how long the module is going to sleep", 2024-03-18T15:28:36.905 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.906 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.906 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.906 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:36.906 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:36.906 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.906 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:36.906 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "prometheus", 2024-03-18T15:28:36.906 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": true, 2024-03-18T15:28:36.906 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "", 2024-03-18T15:28:36.906 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:36.906 INFO:teuthology.orchestra.run.smithi019.stdout: "cache": { 2024-03-18T15:28:36.906 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "cache", 2024-03-18T15:28:36.907 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.907 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.907 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.907 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:36.907 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.907 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.907 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.907 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.907 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.907 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.907 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.907 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.907 INFO:teuthology.orchestra.run.smithi019.stdout: "exclude_perf_counters": { 2024-03-18T15:28:36.907 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "exclude_perf_counters", 2024-03-18T15:28:36.908 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.908 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.908 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.908 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:36.908 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.908 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.908 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.908 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-18T15:28:36.908 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-18T15:28:36.908 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.908 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.908 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.908 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:36.908 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:36.909 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.909 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.909 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.909 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:36.909 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.909 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.909 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:36.909 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:36.909 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:36.909 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:36.909 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:36.909 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:36.909 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:36.909 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:36.909 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.910 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.910 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.910 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.910 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.910 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:36.910 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:36.910 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.910 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.910 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.910 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:36.910 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.910 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.910 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.910 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.911 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.911 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.911 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.911 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.911 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:36.911 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:36.911 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.911 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.911 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.911 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:36.911 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.911 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.911 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:36.911 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:36.911 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:36.912 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:36.912 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:36.912 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:36.912 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:36.912 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:36.912 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.912 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.912 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.912 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.912 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.912 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:36.912 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:36.912 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.913 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.913 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.913 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:36.913 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.913 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.913 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.913 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.913 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.913 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.913 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.913 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.913 INFO:teuthology.orchestra.run.smithi019.stdout: "rbd_stats_pools": { 2024-03-18T15:28:36.913 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "rbd_stats_pools", 2024-03-18T15:28:36.913 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.913 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.914 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.914 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:36.914 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.914 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.914 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.914 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.914 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.914 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.914 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.914 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.914 INFO:teuthology.orchestra.run.smithi019.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-18T15:28:36.914 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-18T15:28:36.914 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:36.914 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.915 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.915 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "300", 2024-03-18T15:28:36.915 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.915 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.915 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.915 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.915 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.915 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.915 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.915 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.915 INFO:teuthology.orchestra.run.smithi019.stdout: "scrape_interval": { 2024-03-18T15:28:36.915 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "scrape_interval", 2024-03-18T15:28:36.915 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "float", 2024-03-18T15:28:36.915 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.916 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.916 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "15.0", 2024-03-18T15:28:36.916 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.916 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.916 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.916 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.916 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.916 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.916 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.916 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.916 INFO:teuthology.orchestra.run.smithi019.stdout: "server_addr": { 2024-03-18T15:28:36.916 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "server_addr", 2024-03-18T15:28:36.916 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.916 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.916 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.917 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "::", 2024-03-18T15:28:36.917 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.917 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.917 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.917 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-18T15:28:36.917 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.917 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.917 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.917 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.917 INFO:teuthology.orchestra.run.smithi019.stdout: "server_port": { 2024-03-18T15:28:36.917 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "server_port", 2024-03-18T15:28:36.917 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:36.917 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.917 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.918 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "9283", 2024-03-18T15:28:36.918 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.918 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.918 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.918 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-18T15:28:36.918 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.918 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.918 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.918 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.918 INFO:teuthology.orchestra.run.smithi019.stdout: "stale_cache_strategy": { 2024-03-18T15:28:36.918 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "stale_cache_strategy", 2024-03-18T15:28:36.918 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.918 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.918 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.919 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "log", 2024-03-18T15:28:36.919 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.919 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.919 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.919 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.919 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.919 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.919 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.919 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.919 INFO:teuthology.orchestra.run.smithi019.stdout: "standby_behaviour": { 2024-03-18T15:28:36.919 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "standby_behaviour", 2024-03-18T15:28:36.919 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.919 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.920 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.920 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "default", 2024-03-18T15:28:36.920 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.920 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.920 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:36.920 INFO:teuthology.orchestra.run.smithi019.stdout: "default", 2024-03-18T15:28:36.920 INFO:teuthology.orchestra.run.smithi019.stdout: "error" 2024-03-18T15:28:36.920 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:36.920 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.920 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.920 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.920 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.920 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.920 INFO:teuthology.orchestra.run.smithi019.stdout: "standby_error_status_code": { 2024-03-18T15:28:36.921 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "standby_error_status_code", 2024-03-18T15:28:36.921 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:36.921 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.921 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.921 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "500", 2024-03-18T15:28:36.921 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "400", 2024-03-18T15:28:36.921 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "599", 2024-03-18T15:28:36.921 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.921 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.921 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.921 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.921 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.921 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:36.921 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:36.921 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.922 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:36.922 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "rbd_support", 2024-03-18T15:28:36.922 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": true, 2024-03-18T15:28:36.922 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "", 2024-03-18T15:28:36.922 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:36.922 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:36.922 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:36.922 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.922 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.922 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.922 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:36.922 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.922 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.923 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:36.923 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:36.923 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:36.923 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:36.923 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:36.923 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:36.923 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:36.923 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:36.923 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.923 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.923 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.923 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.923 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.923 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:36.923 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:36.924 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.924 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.924 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.924 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:36.924 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.924 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.924 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.924 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.924 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.924 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.924 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.924 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.924 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:36.924 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:36.925 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.925 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.925 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.925 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:36.925 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.925 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.925 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:36.925 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:36.925 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:36.925 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:36.925 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:36.925 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:36.925 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:36.925 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:36.925 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.926 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.926 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.926 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.926 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.926 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:36.926 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:36.926 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.926 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.926 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.926 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:36.926 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.926 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.926 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.926 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.926 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.927 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.927 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.927 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.927 INFO:teuthology.orchestra.run.smithi019.stdout: "max_concurrent_snap_create": { 2024-03-18T15:28:36.927 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "max_concurrent_snap_create", 2024-03-18T15:28:36.927 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:36.927 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.927 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.927 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "10", 2024-03-18T15:28:36.927 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.927 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.927 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.927 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.927 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.928 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.928 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.928 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.928 INFO:teuthology.orchestra.run.smithi019.stdout: "mirror_snapshot_schedule": { 2024-03-18T15:28:36.928 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "mirror_snapshot_schedule", 2024-03-18T15:28:36.928 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.928 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.928 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.928 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:36.928 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.928 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.928 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.928 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.928 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.928 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.929 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.929 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.929 INFO:teuthology.orchestra.run.smithi019.stdout: "trash_purge_schedule": { 2024-03-18T15:28:36.929 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "trash_purge_schedule", 2024-03-18T15:28:36.929 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.929 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.929 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.929 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:36.929 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.929 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.929 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.929 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.929 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.929 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.929 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.930 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:36.930 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:36.930 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.930 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:36.930 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "restful", 2024-03-18T15:28:36.930 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": true, 2024-03-18T15:28:36.930 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "", 2024-03-18T15:28:36.930 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:36.930 INFO:teuthology.orchestra.run.smithi019.stdout: "enable_auth": { 2024-03-18T15:28:36.930 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "enable_auth", 2024-03-18T15:28:36.930 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.930 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.930 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.930 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:36.931 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.931 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.931 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.931 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.931 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.931 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.931 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.931 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.931 INFO:teuthology.orchestra.run.smithi019.stdout: "key_file": { 2024-03-18T15:28:36.931 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "key_file", 2024-03-18T15:28:36.931 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.931 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.931 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.931 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:36.931 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.932 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.932 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.932 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.932 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.932 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.932 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.932 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.932 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:36.932 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:36.932 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.933 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.933 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.933 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:36.933 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.933 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.933 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:36.933 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:36.933 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:36.933 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:36.933 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:36.933 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:36.933 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:36.933 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:36.933 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.934 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.934 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.934 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.934 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.934 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:36.934 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:36.934 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.934 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.934 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.934 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:36.934 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.934 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.934 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.935 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.935 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.935 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.935 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.935 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.935 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:36.935 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:36.935 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.935 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.935 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.935 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:36.935 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.935 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.935 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:36.935 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:36.936 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:36.936 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:36.936 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:36.936 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:36.936 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:36.936 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:36.936 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.936 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.936 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.936 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.936 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.936 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:36.936 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:36.936 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.936 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.937 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.937 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:36.937 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.937 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.937 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.937 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.937 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.937 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.937 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.937 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.937 INFO:teuthology.orchestra.run.smithi019.stdout: "server_addr": { 2024-03-18T15:28:36.937 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "server_addr", 2024-03-18T15:28:36.937 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.937 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.937 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.938 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:36.938 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.938 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.938 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.938 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.938 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.938 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.938 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.938 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.938 INFO:teuthology.orchestra.run.smithi019.stdout: "server_port": { 2024-03-18T15:28:36.938 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "server_port", 2024-03-18T15:28:36.938 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.938 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.938 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.939 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:36.939 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.939 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.939 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.939 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.939 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.939 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.939 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.939 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:36.939 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:36.939 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.939 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:36.939 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "rgw", 2024-03-18T15:28:36.939 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": true, 2024-03-18T15:28:36.939 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "", 2024-03-18T15:28:36.940 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:36.940 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:36.940 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:36.940 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.940 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.940 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.940 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:36.940 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.940 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.940 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:36.940 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:36.940 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:36.940 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:36.940 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:36.940 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:36.941 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:36.941 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:36.941 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.941 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.941 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.941 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.941 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.941 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:36.941 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:36.941 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.941 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.941 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.941 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:36.941 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.941 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.942 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.942 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.942 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.942 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.942 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.942 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.942 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:36.942 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:36.942 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.942 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.942 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.942 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:36.942 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.942 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.943 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:36.943 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:36.943 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:36.943 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:36.943 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:36.943 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:36.943 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:36.943 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:36.943 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.943 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.943 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.943 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.943 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.943 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:36.943 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:36.944 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.944 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.944 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.944 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:36.944 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.944 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.944 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.944 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.944 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.944 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.944 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.944 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:36.944 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:36.944 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.944 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:36.945 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "rook", 2024-03-18T15:28:36.945 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": true, 2024-03-18T15:28:36.945 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "", 2024-03-18T15:28:36.945 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:36.945 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:36.945 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:36.945 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.945 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.945 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.945 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:36.945 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.945 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.945 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:36.945 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:36.945 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:36.946 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:36.946 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:36.946 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:36.946 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:36.946 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:36.946 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.946 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.946 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.946 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.946 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.946 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:36.946 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:36.946 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.947 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.947 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.947 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:36.947 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.947 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.947 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.947 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.947 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.947 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.947 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.947 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.947 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:36.947 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:36.947 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.948 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.948 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.948 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:36.948 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.948 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.948 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:36.948 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:36.948 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:36.948 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:36.948 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:36.948 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:36.948 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:36.948 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:36.948 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.948 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.949 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.949 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.949 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.949 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:36.949 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:36.949 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.949 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.949 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.949 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:36.949 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.949 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.949 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.949 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.949 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.950 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.950 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.950 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.950 INFO:teuthology.orchestra.run.smithi019.stdout: "storage_class": { 2024-03-18T15:28:36.950 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "storage_class", 2024-03-18T15:28:36.950 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.950 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.950 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.950 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "local", 2024-03-18T15:28:36.950 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.950 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.950 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.950 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-18T15:28:36.950 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.951 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.951 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.951 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:36.951 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:36.951 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.951 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:36.951 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "selftest", 2024-03-18T15:28:36.951 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": true, 2024-03-18T15:28:36.951 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "", 2024-03-18T15:28:36.951 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:36.951 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:36.951 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:36.951 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.951 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.952 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.952 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:36.952 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.952 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.952 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:36.952 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:36.952 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:36.952 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:36.952 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:36.952 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:36.952 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:36.952 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:36.952 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.952 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.953 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.953 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.953 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.953 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:36.953 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:36.953 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.953 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.953 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.953 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:36.953 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.953 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.953 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.953 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.953 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.954 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.954 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.954 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.954 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:36.954 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:36.954 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.954 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.954 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.954 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:36.954 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.954 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.954 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:36.954 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:36.954 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:36.955 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:36.955 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:36.955 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:36.955 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:36.955 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:36.955 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.955 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.955 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.955 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.955 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.955 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:36.955 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:36.955 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.955 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.956 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.956 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:36.956 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.956 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.956 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.956 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.956 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.956 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.956 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.956 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.956 INFO:teuthology.orchestra.run.smithi019.stdout: "roption1": { 2024-03-18T15:28:36.956 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "roption1", 2024-03-18T15:28:36.956 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.956 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.957 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.957 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:36.957 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.957 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.957 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.957 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.957 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.957 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.957 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.957 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.957 INFO:teuthology.orchestra.run.smithi019.stdout: "roption2": { 2024-03-18T15:28:36.957 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "roption2", 2024-03-18T15:28:36.957 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.957 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.957 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.958 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "xyz", 2024-03-18T15:28:36.958 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.958 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.958 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.958 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.958 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.958 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.958 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.958 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.958 INFO:teuthology.orchestra.run.smithi019.stdout: "rwoption1": { 2024-03-18T15:28:36.958 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "rwoption1", 2024-03-18T15:28:36.958 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.958 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.958 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.958 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:36.959 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.959 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.959 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.959 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.959 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.959 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.959 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.959 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.959 INFO:teuthology.orchestra.run.smithi019.stdout: "rwoption2": { 2024-03-18T15:28:36.959 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "rwoption2", 2024-03-18T15:28:36.959 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:36.959 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.959 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.959 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:36.960 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.960 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.960 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.960 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.960 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.960 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.960 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.960 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.960 INFO:teuthology.orchestra.run.smithi019.stdout: "rwoption3": { 2024-03-18T15:28:36.960 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "rwoption3", 2024-03-18T15:28:36.960 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "float", 2024-03-18T15:28:36.960 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.960 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.961 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:36.961 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.961 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.961 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.961 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.961 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.961 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.961 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.961 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.961 INFO:teuthology.orchestra.run.smithi019.stdout: "rwoption4": { 2024-03-18T15:28:36.961 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "rwoption4", 2024-03-18T15:28:36.961 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.961 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.961 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.962 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:36.962 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.962 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.962 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.962 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.962 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.962 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.962 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.962 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.962 INFO:teuthology.orchestra.run.smithi019.stdout: "rwoption5": { 2024-03-18T15:28:36.962 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "rwoption5", 2024-03-18T15:28:36.962 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.962 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.962 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.963 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:36.963 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.963 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.963 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.963 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.963 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.963 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.963 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.963 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.963 INFO:teuthology.orchestra.run.smithi019.stdout: "rwoption6": { 2024-03-18T15:28:36.963 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "rwoption6", 2024-03-18T15:28:36.963 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.963 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.963 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.964 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:36.964 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.964 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.964 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.964 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.964 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.964 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.964 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.964 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.964 INFO:teuthology.orchestra.run.smithi019.stdout: "rwoption7": { 2024-03-18T15:28:36.964 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "rwoption7", 2024-03-18T15:28:36.964 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:36.964 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.964 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.965 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:36.965 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "1", 2024-03-18T15:28:36.965 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "42", 2024-03-18T15:28:36.965 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.965 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.965 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.965 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.965 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.965 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.965 INFO:teuthology.orchestra.run.smithi019.stdout: "testkey": { 2024-03-18T15:28:36.965 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "testkey", 2024-03-18T15:28:36.965 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.965 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.965 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.966 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:36.966 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.966 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.966 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.966 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.966 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.966 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.966 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.966 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.966 INFO:teuthology.orchestra.run.smithi019.stdout: "testlkey": { 2024-03-18T15:28:36.966 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "testlkey", 2024-03-18T15:28:36.966 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.966 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.966 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.967 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:36.967 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.967 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.967 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.967 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.967 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.967 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.967 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.967 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.967 INFO:teuthology.orchestra.run.smithi019.stdout: "testnewline": { 2024-03-18T15:28:36.967 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "testnewline", 2024-03-18T15:28:36.967 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.967 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.967 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.967 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:36.968 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.968 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.968 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.968 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.968 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.968 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.968 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.968 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:36.968 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:36.968 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.968 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:36.968 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "snap_schedule", 2024-03-18T15:28:36.968 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": true, 2024-03-18T15:28:36.968 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "", 2024-03-18T15:28:36.968 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:36.969 INFO:teuthology.orchestra.run.smithi019.stdout: "allow_m_granularity": { 2024-03-18T15:28:36.969 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "allow_m_granularity", 2024-03-18T15:28:36.969 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.969 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.969 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.969 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:36.969 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.969 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.969 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.969 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "allow minute scheduled snapshots", 2024-03-18T15:28:36.969 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.969 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.969 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.969 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.970 INFO:teuthology.orchestra.run.smithi019.stdout: "dump_on_update": { 2024-03-18T15:28:36.970 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "dump_on_update", 2024-03-18T15:28:36.970 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.970 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.970 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.970 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:36.970 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.970 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.970 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.970 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "dump database to debug log on update", 2024-03-18T15:28:36.970 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.970 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.970 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.970 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.970 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:36.971 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:36.971 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.971 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.971 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.971 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:36.971 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.971 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.971 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:36.971 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:36.971 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:36.971 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:36.971 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:36.971 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:36.971 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:36.971 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:36.972 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.972 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.972 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.972 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.972 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.972 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:36.972 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:36.972 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.972 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.972 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.972 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:36.972 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.972 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.972 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.973 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.973 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.973 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.973 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.973 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.973 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:36.973 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:36.973 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.973 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.973 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.973 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:36.973 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.973 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.974 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:36.974 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:36.974 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:36.974 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:36.974 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:36.974 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:36.974 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:36.974 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:36.974 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.974 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.974 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.974 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.974 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.974 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:36.975 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:36.975 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.975 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.975 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.975 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:36.975 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.975 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.975 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.975 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.975 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.975 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.975 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.975 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:36.975 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:36.976 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.976 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:36.976 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "stats", 2024-03-18T15:28:36.976 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": true, 2024-03-18T15:28:36.976 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "", 2024-03-18T15:28:36.976 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:36.976 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:36.976 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:36.976 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.976 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.976 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.976 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:36.976 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.976 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.976 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:36.977 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:36.977 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:36.977 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:36.977 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:36.977 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:36.977 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:36.977 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:36.977 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.977 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.977 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.977 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.977 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.977 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:36.977 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:36.978 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.978 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.978 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.978 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:36.978 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.978 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.978 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.978 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.978 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.978 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.978 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.978 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.978 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:36.978 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:36.978 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.979 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.979 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.979 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:36.979 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.979 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.979 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:36.979 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:36.979 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:36.979 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:36.979 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:36.979 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:36.979 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:36.979 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:36.979 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.980 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.980 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.980 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.980 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.980 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:36.980 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:36.980 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.980 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.980 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.980 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:36.980 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.980 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.980 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.980 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.980 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.981 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.981 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.981 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:36.981 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:36.981 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.981 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:36.981 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "status", 2024-03-18T15:28:36.981 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": true, 2024-03-18T15:28:36.981 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "", 2024-03-18T15:28:36.981 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:36.981 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:36.981 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:36.981 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.981 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.981 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.982 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:36.982 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.982 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.982 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:36.982 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:36.982 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:36.982 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:36.982 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:36.982 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:36.982 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:36.982 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:36.982 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.982 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.982 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.983 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.983 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.983 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:36.983 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:36.983 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.983 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.983 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.983 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:36.983 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.983 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.983 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.983 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.983 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.983 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.983 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.984 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.984 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:36.984 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:36.984 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.984 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.984 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.984 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:36.984 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.984 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.984 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:36.984 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:36.984 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:36.984 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:36.984 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:36.985 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:36.985 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:36.985 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:36.985 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.985 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.985 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.985 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.985 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.985 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:36.985 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:36.985 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.985 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.985 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.985 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:36.985 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.986 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.986 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.986 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.986 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.986 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.986 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.986 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:36.986 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:36.986 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.986 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:36.986 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "telegraf", 2024-03-18T15:28:36.986 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": true, 2024-03-18T15:28:36.986 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "", 2024-03-18T15:28:36.986 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:36.987 INFO:teuthology.orchestra.run.smithi019.stdout: "address": { 2024-03-18T15:28:36.987 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "address", 2024-03-18T15:28:36.987 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.987 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.987 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.987 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-18T15:28:36.987 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.987 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.987 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.987 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.987 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.987 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.987 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.988 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.988 INFO:teuthology.orchestra.run.smithi019.stdout: "interval": { 2024-03-18T15:28:36.988 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "interval", 2024-03-18T15:28:36.988 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "secs", 2024-03-18T15:28:36.988 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.988 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.988 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "15", 2024-03-18T15:28:36.988 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.988 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.988 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.988 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.988 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.988 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.989 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.989 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.989 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:36.989 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:36.989 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.989 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.989 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.989 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:36.989 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.989 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.989 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:36.989 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:36.989 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:36.989 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:36.989 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:36.990 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:36.990 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:36.990 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:36.990 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.990 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.990 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.990 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.990 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.990 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:36.990 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:36.990 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.990 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.990 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.990 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:36.991 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.991 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.991 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.991 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.991 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.991 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.991 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.991 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.991 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:36.991 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:36.991 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.991 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.991 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.991 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:36.991 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.992 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.992 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:36.992 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:36.992 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:36.992 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:36.992 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:36.992 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:36.992 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:36.992 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:36.992 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.992 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.992 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.992 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.992 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.992 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:36.993 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:36.993 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.993 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.993 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:36.993 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:36.993 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.993 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.993 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.993 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:36.993 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.993 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.993 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.993 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:36.993 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:36.994 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.994 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:36.994 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "telemetry", 2024-03-18T15:28:36.994 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": true, 2024-03-18T15:28:36.994 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "", 2024-03-18T15:28:36.994 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:36.994 INFO:teuthology.orchestra.run.smithi019.stdout: "channel_basic": { 2024-03-18T15:28:36.994 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "channel_basic", 2024-03-18T15:28:36.994 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.994 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.994 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.994 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:36.994 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.994 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.995 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.995 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-18T15:28:36.995 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.995 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.995 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.995 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.995 INFO:teuthology.orchestra.run.smithi019.stdout: "channel_crash": { 2024-03-18T15:28:36.995 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "channel_crash", 2024-03-18T15:28:36.995 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.995 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.995 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.995 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:36.995 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.995 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.996 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.996 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-18T15:28:36.996 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.996 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.996 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.996 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.996 INFO:teuthology.orchestra.run.smithi019.stdout: "channel_device": { 2024-03-18T15:28:36.996 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "channel_device", 2024-03-18T15:28:36.996 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.996 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.996 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.996 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:36.996 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.996 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.996 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.997 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-18T15:28:36.997 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.997 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.997 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.997 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.997 INFO:teuthology.orchestra.run.smithi019.stdout: "channel_ident": { 2024-03-18T15:28:36.997 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "channel_ident", 2024-03-18T15:28:36.997 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.997 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.997 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.997 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:36.997 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.997 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.997 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.997 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-18T15:28:36.998 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.998 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.998 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.998 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.998 INFO:teuthology.orchestra.run.smithi019.stdout: "channel_perf": { 2024-03-18T15:28:36.998 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "channel_perf", 2024-03-18T15:28:36.998 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:36.998 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.998 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.998 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:36.998 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.998 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.998 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.998 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-18T15:28:36.999 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:36.999 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:36.999 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:36.999 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:36.999 INFO:teuthology.orchestra.run.smithi019.stdout: "contact": { 2024-03-18T15:28:36.999 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "contact", 2024-03-18T15:28:36.999 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:36.999 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:36.999 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:36.999 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:36.999 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:36.999 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:36.999 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:36.999 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.000 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.000 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.000 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.000 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.000 INFO:teuthology.orchestra.run.smithi019.stdout: "description": { 2024-03-18T15:28:37.000 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "description", 2024-03-18T15:28:37.000 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.000 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.000 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.000 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.000 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.000 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.000 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.001 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.001 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.001 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.001 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.001 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.001 INFO:teuthology.orchestra.run.smithi019.stdout: "device_url": { 2024-03-18T15:28:37.001 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "device_url", 2024-03-18T15:28:37.001 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.001 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.001 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.001 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-18T15:28:37.001 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.001 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.001 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.002 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.002 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.002 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.002 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.002 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.002 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled": { 2024-03-18T15:28:37.002 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "enabled", 2024-03-18T15:28:37.002 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.002 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.002 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.002 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.002 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.002 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.002 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.003 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.003 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.003 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.003 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.003 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.003 INFO:teuthology.orchestra.run.smithi019.stdout: "interval": { 2024-03-18T15:28:37.003 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "interval", 2024-03-18T15:28:37.003 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:37.003 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.003 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.003 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "24", 2024-03-18T15:28:37.003 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "8", 2024-03-18T15:28:37.003 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.003 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.004 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.004 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.004 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.004 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.004 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.004 INFO:teuthology.orchestra.run.smithi019.stdout: "last_opt_revision": { 2024-03-18T15:28:37.004 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "last_opt_revision", 2024-03-18T15:28:37.004 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:37.004 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.004 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.004 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "1", 2024-03-18T15:28:37.004 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.004 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.004 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.005 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.005 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.005 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.005 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.005 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.005 INFO:teuthology.orchestra.run.smithi019.stdout: "leaderboard": { 2024-03-18T15:28:37.005 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "leaderboard", 2024-03-18T15:28:37.005 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.005 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.005 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.005 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.005 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.005 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.005 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.006 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.006 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.006 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.006 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.006 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.006 INFO:teuthology.orchestra.run.smithi019.stdout: "leaderboard_description": { 2024-03-18T15:28:37.006 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "leaderboard_description", 2024-03-18T15:28:37.006 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.006 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.006 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.006 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.006 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.006 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.006 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.007 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.007 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.007 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.007 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.007 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.007 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:37.007 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:37.007 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.007 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.007 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.007 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.007 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.007 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.007 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.007 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.008 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.008 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.008 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.008 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.008 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.008 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.008 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.008 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.008 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.008 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.008 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.008 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:37.008 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:37.008 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.008 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.009 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.009 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.009 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.009 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.009 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.009 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.009 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.009 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.009 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.009 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.009 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:37.009 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:37.009 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.009 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.010 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.010 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:37.010 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.010 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.010 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.010 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.010 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.010 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.010 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.010 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.010 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.010 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.010 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.010 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.011 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.011 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.011 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.011 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:37.011 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:37.011 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.011 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.011 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.011 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.011 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.011 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.011 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.011 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.011 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.011 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.012 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.012 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.012 INFO:teuthology.orchestra.run.smithi019.stdout: "organization": { 2024-03-18T15:28:37.012 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "organization", 2024-03-18T15:28:37.012 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.012 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.012 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.012 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.012 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.012 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.012 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.012 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.012 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.012 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.012 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.013 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.013 INFO:teuthology.orchestra.run.smithi019.stdout: "proxy": { 2024-03-18T15:28:37.013 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "proxy", 2024-03-18T15:28:37.013 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.013 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.013 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.013 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.013 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.013 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.013 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.013 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.013 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.013 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.014 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.014 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.014 INFO:teuthology.orchestra.run.smithi019.stdout: "url": { 2024-03-18T15:28:37.014 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "url", 2024-03-18T15:28:37.014 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.014 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.014 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.014 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-18T15:28:37.014 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.014 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.014 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.014 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.014 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.014 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.015 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.015 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.015 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.015 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.015 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:37.015 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "test_orchestrator", 2024-03-18T15:28:37.015 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": true, 2024-03-18T15:28:37.015 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "", 2024-03-18T15:28:37.015 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:37.015 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:37.015 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:37.015 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.015 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.015 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.016 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.016 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.016 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.016 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.016 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.016 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.016 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.016 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.016 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.016 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.016 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.016 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.016 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.016 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.016 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.017 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.017 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:37.017 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:37.017 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.017 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.017 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.017 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.017 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.017 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.017 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.017 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.017 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.017 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.017 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.018 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.018 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:37.018 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:37.018 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.018 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.018 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.018 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:37.018 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.018 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.018 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.018 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.018 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.018 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.018 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.019 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.019 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.019 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.019 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.019 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.019 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.019 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.019 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.019 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:37.019 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:37.019 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.019 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.019 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.019 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.019 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.020 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.020 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.020 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.020 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.020 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.020 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.020 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.020 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.020 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.020 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:37.020 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "volumes", 2024-03-18T15:28:37.020 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": true, 2024-03-18T15:28:37.020 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "", 2024-03-18T15:28:37.020 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:37.021 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:37.021 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:37.021 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.021 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.021 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.021 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.021 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.021 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.021 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.021 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.021 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.021 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.021 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.021 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.021 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.022 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.022 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.022 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.022 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.022 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.022 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.022 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:37.022 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:37.022 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.022 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.022 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.022 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.022 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.022 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.023 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.023 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.023 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.023 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.023 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.023 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.023 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:37.023 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:37.023 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.023 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.023 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.023 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:37.023 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.023 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.024 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.024 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.024 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.024 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.024 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.024 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.024 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.024 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.024 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.024 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.024 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.024 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.024 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.024 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:37.024 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:37.025 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.025 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.025 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.025 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.025 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.025 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.025 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.025 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.025 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.025 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.025 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.025 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.025 INFO:teuthology.orchestra.run.smithi019.stdout: "max_concurrent_clones": { 2024-03-18T15:28:37.025 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "max_concurrent_clones", 2024-03-18T15:28:37.026 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:37.026 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.026 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.026 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "4", 2024-03-18T15:28:37.026 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.026 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.026 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.026 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-18T15:28:37.026 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.026 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.026 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.026 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.026 INFO:teuthology.orchestra.run.smithi019.stdout: "periodic_async_work": { 2024-03-18T15:28:37.026 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "periodic_async_work", 2024-03-18T15:28:37.027 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.027 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.027 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.027 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.027 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.027 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.027 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.027 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Periodically check for async work", 2024-03-18T15:28:37.027 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.027 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.027 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.027 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.027 INFO:teuthology.orchestra.run.smithi019.stdout: "snapshot_clone_delay": { 2024-03-18T15:28:37.027 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "snapshot_clone_delay", 2024-03-18T15:28:37.028 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:37.028 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.028 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.028 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "0", 2024-03-18T15:28:37.028 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.028 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.028 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.028 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-18T15:28:37.028 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.028 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.028 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.028 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.028 INFO:teuthology.orchestra.run.smithi019.stdout: "snapshot_clone_no_wait": { 2024-03-18T15:28:37.028 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "snapshot_clone_no_wait", 2024-03-18T15:28:37.029 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.029 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.029 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.029 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:37.029 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.029 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.029 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.029 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-18T15:28:37.029 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.029 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.029 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.029 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.029 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.029 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.030 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:37.030 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "zabbix", 2024-03-18T15:28:37.030 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": true, 2024-03-18T15:28:37.030 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "", 2024-03-18T15:28:37.030 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:37.030 INFO:teuthology.orchestra.run.smithi019.stdout: "discovery_interval": { 2024-03-18T15:28:37.030 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "discovery_interval", 2024-03-18T15:28:37.030 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "uint", 2024-03-18T15:28:37.030 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.030 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.030 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "100", 2024-03-18T15:28:37.030 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.030 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.030 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.031 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.031 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.031 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.031 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.031 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.031 INFO:teuthology.orchestra.run.smithi019.stdout: "identifier": { 2024-03-18T15:28:37.031 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "identifier", 2024-03-18T15:28:37.031 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.031 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.031 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.031 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.031 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.031 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.031 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.031 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.032 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.032 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.032 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.032 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.032 INFO:teuthology.orchestra.run.smithi019.stdout: "interval": { 2024-03-18T15:28:37.032 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "interval", 2024-03-18T15:28:37.032 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "secs", 2024-03-18T15:28:37.032 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.032 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.032 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "60", 2024-03-18T15:28:37.032 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.032 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.032 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.032 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.033 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.033 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.033 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.033 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.033 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:37.033 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:37.033 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.033 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.033 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.033 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.033 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.033 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.033 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.033 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.034 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.034 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.034 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.034 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.034 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.034 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.034 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.034 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.034 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.034 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.034 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.034 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:37.034 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:37.034 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.035 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.035 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.035 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.035 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.035 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.035 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.035 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.035 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.035 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.035 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.035 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.035 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:37.035 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:37.035 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.035 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.036 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.036 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:37.036 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.036 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.036 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.036 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.036 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.036 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.036 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.036 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.036 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.036 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.036 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.036 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.037 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.037 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.037 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.037 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:37.037 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:37.037 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.037 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.037 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.037 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.037 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.037 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.037 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.037 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.037 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.037 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.038 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.038 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.038 INFO:teuthology.orchestra.run.smithi019.stdout: "zabbix_host": { 2024-03-18T15:28:37.038 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "zabbix_host", 2024-03-18T15:28:37.038 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.038 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.038 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.038 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.038 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.038 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.038 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.038 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.038 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.038 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.039 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.039 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.039 INFO:teuthology.orchestra.run.smithi019.stdout: "zabbix_port": { 2024-03-18T15:28:37.039 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "zabbix_port", 2024-03-18T15:28:37.039 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:37.039 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.039 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.039 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "10051", 2024-03-18T15:28:37.039 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.039 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.039 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.039 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.039 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.039 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.039 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.040 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.040 INFO:teuthology.orchestra.run.smithi019.stdout: "zabbix_sender": { 2024-03-18T15:28:37.040 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "zabbix_sender", 2024-03-18T15:28:37.040 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.040 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.040 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.040 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-18T15:28:37.040 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.040 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.040 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.040 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.040 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.040 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.040 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.041 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.041 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.041 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.041 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-03-18T15:28:37.041 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.041 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:37.041 INFO:teuthology.orchestra.run.smithi019.stdout: "gid": 25477, 2024-03-18T15:28:37.041 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "z", 2024-03-18T15:28:37.041 INFO:teuthology.orchestra.run.smithi019.stdout: "mgr_features": 4540701547738038271, 2024-03-18T15:28:37.041 INFO:teuthology.orchestra.run.smithi019.stdout: "available_modules": [ 2024-03-18T15:28:37.041 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:37.041 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "alerts", 2024-03-18T15:28:37.041 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": true, 2024-03-18T15:28:37.041 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "", 2024-03-18T15:28:37.042 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:37.042 INFO:teuthology.orchestra.run.smithi019.stdout: "interval": { 2024-03-18T15:28:37.042 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "interval", 2024-03-18T15:28:37.042 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "secs", 2024-03-18T15:28:37.042 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.042 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.042 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "60", 2024-03-18T15:28:37.042 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.042 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.042 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.042 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "How frequently to reexamine health status", 2024-03-18T15:28:37.042 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.042 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.042 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.043 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.043 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:37.043 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:37.043 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.043 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.043 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.043 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.043 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.043 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.043 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.043 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.043 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.043 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.043 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.044 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.044 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.044 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.044 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.044 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.044 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.044 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.044 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.044 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:37.044 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:37.044 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.044 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.044 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.044 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.045 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.045 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.045 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.045 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.045 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.045 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.045 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.045 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.045 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:37.045 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:37.045 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.045 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.045 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.045 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:37.046 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.046 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.046 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.046 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.046 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.046 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.046 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.046 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.046 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.046 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.046 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.046 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.046 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.046 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.047 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.047 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:37.047 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:37.047 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.047 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.047 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.047 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.047 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.047 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.047 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.047 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.047 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.047 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.047 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.047 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.048 INFO:teuthology.orchestra.run.smithi019.stdout: "smtp_destination": { 2024-03-18T15:28:37.048 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "smtp_destination", 2024-03-18T15:28:37.048 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.048 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.048 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.048 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.048 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.048 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.048 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.048 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Email address to send alerts to", 2024-03-18T15:28:37.048 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.048 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.048 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.048 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.049 INFO:teuthology.orchestra.run.smithi019.stdout: "smtp_from_name": { 2024-03-18T15:28:37.049 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "smtp_from_name", 2024-03-18T15:28:37.049 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.049 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.049 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.049 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "Ceph", 2024-03-18T15:28:37.049 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.049 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.049 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.049 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Email From: name", 2024-03-18T15:28:37.049 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.049 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.049 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.049 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.049 INFO:teuthology.orchestra.run.smithi019.stdout: "smtp_host": { 2024-03-18T15:28:37.050 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "smtp_host", 2024-03-18T15:28:37.050 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.050 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.050 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.050 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.050 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.050 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.050 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.050 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "SMTP server", 2024-03-18T15:28:37.050 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.050 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.050 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.050 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.050 INFO:teuthology.orchestra.run.smithi019.stdout: "smtp_password": { 2024-03-18T15:28:37.051 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "smtp_password", 2024-03-18T15:28:37.051 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.051 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.051 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.051 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.051 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.051 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.051 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.051 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Password to authenticate with", 2024-03-18T15:28:37.051 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.051 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.051 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.051 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.051 INFO:teuthology.orchestra.run.smithi019.stdout: "smtp_port": { 2024-03-18T15:28:37.051 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "smtp_port", 2024-03-18T15:28:37.052 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:37.052 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.052 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.052 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "465", 2024-03-18T15:28:37.052 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.052 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.052 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.052 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "SMTP port", 2024-03-18T15:28:37.052 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.052 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.052 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.052 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.052 INFO:teuthology.orchestra.run.smithi019.stdout: "smtp_sender": { 2024-03-18T15:28:37.052 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "smtp_sender", 2024-03-18T15:28:37.052 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.053 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.053 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.053 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.053 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.053 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.053 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.053 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "SMTP envelope sender", 2024-03-18T15:28:37.053 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.053 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.053 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.053 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.053 INFO:teuthology.orchestra.run.smithi019.stdout: "smtp_ssl": { 2024-03-18T15:28:37.053 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "smtp_ssl", 2024-03-18T15:28:37.053 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.054 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.054 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.054 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:37.054 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.054 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.054 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.054 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-18T15:28:37.054 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.054 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.054 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.054 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.054 INFO:teuthology.orchestra.run.smithi019.stdout: "smtp_user": { 2024-03-18T15:28:37.055 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "smtp_user", 2024-03-18T15:28:37.055 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.057 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.058 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.058 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.058 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.058 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.058 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.058 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "User to authenticate as", 2024-03-18T15:28:37.058 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.058 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.058 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.058 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.058 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.058 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.058 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:37.058 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "balancer", 2024-03-18T15:28:37.059 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": true, 2024-03-18T15:28:37.059 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "", 2024-03-18T15:28:37.059 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:37.059 INFO:teuthology.orchestra.run.smithi019.stdout: "active": { 2024-03-18T15:28:37.059 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "active", 2024-03-18T15:28:37.059 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.059 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.059 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.059 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:37.059 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.059 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.059 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.059 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "automatically balance PGs across cluster", 2024-03-18T15:28:37.059 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.060 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.060 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.060 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.060 INFO:teuthology.orchestra.run.smithi019.stdout: "begin_time": { 2024-03-18T15:28:37.060 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "begin_time", 2024-03-18T15:28:37.060 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.060 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.060 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.060 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "0000", 2024-03-18T15:28:37.060 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.060 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.060 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.060 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "beginning time of day to automatically balance", 2024-03-18T15:28:37.060 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-18T15:28:37.061 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.061 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.061 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.061 INFO:teuthology.orchestra.run.smithi019.stdout: "begin_weekday": { 2024-03-18T15:28:37.061 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "begin_weekday", 2024-03-18T15:28:37.061 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "uint", 2024-03-18T15:28:37.061 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.061 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.061 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "0", 2024-03-18T15:28:37.061 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "0", 2024-03-18T15:28:37.061 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "6", 2024-03-18T15:28:37.061 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.061 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-18T15:28:37.061 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-18T15:28:37.062 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.062 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.062 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.062 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_compat_max_iterations": { 2024-03-18T15:28:37.062 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "crush_compat_max_iterations", 2024-03-18T15:28:37.062 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "uint", 2024-03-18T15:28:37.062 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.062 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.062 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "25", 2024-03-18T15:28:37.062 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "1", 2024-03-18T15:28:37.062 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "250", 2024-03-18T15:28:37.062 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.062 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-18T15:28:37.062 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.063 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.063 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.063 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.063 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_compat_metrics": { 2024-03-18T15:28:37.063 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "crush_compat_metrics", 2024-03-18T15:28:37.063 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.063 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.063 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.063 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "pgs,objects,bytes", 2024-03-18T15:28:37.063 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.063 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.063 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.063 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-18T15:28:37.063 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-18T15:28:37.064 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.064 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.064 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.064 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_compat_step": { 2024-03-18T15:28:37.064 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "crush_compat_step", 2024-03-18T15:28:37.064 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "float", 2024-03-18T15:28:37.064 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.064 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.064 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "0.5", 2024-03-18T15:28:37.064 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "0.001", 2024-03-18T15:28:37.064 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "0.999", 2024-03-18T15:28:37.064 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.064 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "aggressiveness of optimization", 2024-03-18T15:28:37.064 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-18T15:28:37.065 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.065 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.065 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.065 INFO:teuthology.orchestra.run.smithi019.stdout: "end_time": { 2024-03-18T15:28:37.065 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "end_time", 2024-03-18T15:28:37.065 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.065 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.065 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.065 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "2359", 2024-03-18T15:28:37.065 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.065 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.065 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.065 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "ending time of day to automatically balance", 2024-03-18T15:28:37.065 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-18T15:28:37.065 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.066 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.066 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.066 INFO:teuthology.orchestra.run.smithi019.stdout: "end_weekday": { 2024-03-18T15:28:37.066 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "end_weekday", 2024-03-18T15:28:37.066 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "uint", 2024-03-18T15:28:37.066 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.066 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.066 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "0", 2024-03-18T15:28:37.066 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "0", 2024-03-18T15:28:37.066 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "6", 2024-03-18T15:28:37.066 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.066 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-18T15:28:37.066 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-18T15:28:37.066 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.067 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.067 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.067 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:37.067 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:37.067 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.067 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.067 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.067 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.067 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.067 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.067 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.067 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.067 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.067 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.067 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.068 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.068 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.068 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.068 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.068 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.068 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.068 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.068 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.068 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:37.068 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:37.068 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.068 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.068 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.068 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.068 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.069 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.069 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.069 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.069 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.069 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.069 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.069 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.069 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:37.069 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:37.069 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.069 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.069 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.069 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:37.069 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.069 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.070 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.070 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.070 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.070 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.070 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.070 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.070 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.070 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.070 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.070 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.070 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.070 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.071 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.071 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:37.071 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:37.071 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.071 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.071 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.071 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.071 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.071 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.071 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.071 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.071 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.071 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.071 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.071 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.072 INFO:teuthology.orchestra.run.smithi019.stdout: "min_score": { 2024-03-18T15:28:37.072 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "min_score", 2024-03-18T15:28:37.072 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "float", 2024-03-18T15:28:37.072 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.072 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.072 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "0", 2024-03-18T15:28:37.072 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.072 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.072 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.072 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-18T15:28:37.072 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.072 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.072 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.072 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.073 INFO:teuthology.orchestra.run.smithi019.stdout: "mode": { 2024-03-18T15:28:37.073 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "mode", 2024-03-18T15:28:37.073 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.073 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.073 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.073 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "upmap", 2024-03-18T15:28:37.073 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.073 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.073 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.073 INFO:teuthology.orchestra.run.smithi019.stdout: "crush-compat", 2024-03-18T15:28:37.073 INFO:teuthology.orchestra.run.smithi019.stdout: "none", 2024-03-18T15:28:37.073 INFO:teuthology.orchestra.run.smithi019.stdout: "read", 2024-03-18T15:28:37.073 INFO:teuthology.orchestra.run.smithi019.stdout: "upmap", 2024-03-18T15:28:37.073 INFO:teuthology.orchestra.run.smithi019.stdout: "upmap-read" 2024-03-18T15:28:37.074 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.074 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Balancer mode", 2024-03-18T15:28:37.074 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.074 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.074 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.074 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.074 INFO:teuthology.orchestra.run.smithi019.stdout: "pool_ids": { 2024-03-18T15:28:37.074 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "pool_ids", 2024-03-18T15:28:37.074 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.074 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.074 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.074 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.074 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.074 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.075 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.075 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-18T15:28:37.075 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.075 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.075 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.075 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.075 INFO:teuthology.orchestra.run.smithi019.stdout: "sleep_interval": { 2024-03-18T15:28:37.075 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "sleep_interval", 2024-03-18T15:28:37.075 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "secs", 2024-03-18T15:28:37.075 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.075 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.075 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "60", 2024-03-18T15:28:37.075 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.075 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.076 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.076 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-18T15:28:37.076 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.076 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.076 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.076 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.076 INFO:teuthology.orchestra.run.smithi019.stdout: "upmap_max_deviation": { 2024-03-18T15:28:37.076 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "upmap_max_deviation", 2024-03-18T15:28:37.076 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:37.076 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.076 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.076 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "5", 2024-03-18T15:28:37.076 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "1", 2024-03-18T15:28:37.076 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.076 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.077 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-18T15:28:37.077 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-18T15:28:37.077 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.077 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.077 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.077 INFO:teuthology.orchestra.run.smithi019.stdout: "upmap_max_optimizations": { 2024-03-18T15:28:37.077 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "upmap_max_optimizations", 2024-03-18T15:28:37.077 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "uint", 2024-03-18T15:28:37.077 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.077 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.077 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "10", 2024-03-18T15:28:37.077 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.077 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.077 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.078 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-18T15:28:37.078 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.078 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.078 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.078 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.078 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.078 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.078 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:37.078 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "cephadm", 2024-03-18T15:28:37.078 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": true, 2024-03-18T15:28:37.078 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "", 2024-03-18T15:28:37.078 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:37.078 INFO:teuthology.orchestra.run.smithi019.stdout: "agent_down_multiplier": { 2024-03-18T15:28:37.078 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "agent_down_multiplier", 2024-03-18T15:28:37.078 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "float", 2024-03-18T15:28:37.079 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.079 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.079 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "3.0", 2024-03-18T15:28:37.079 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.079 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.079 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.079 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-18T15:28:37.079 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.079 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.079 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.079 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.079 INFO:teuthology.orchestra.run.smithi019.stdout: "agent_refresh_rate": { 2024-03-18T15:28:37.079 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "agent_refresh_rate", 2024-03-18T15:28:37.079 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "secs", 2024-03-18T15:28:37.080 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.080 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.080 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "20", 2024-03-18T15:28:37.080 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.080 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.080 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.080 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-18T15:28:37.080 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.080 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.080 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.080 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.080 INFO:teuthology.orchestra.run.smithi019.stdout: "agent_starting_port": { 2024-03-18T15:28:37.080 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "agent_starting_port", 2024-03-18T15:28:37.080 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:37.080 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.081 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.081 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "4721", 2024-03-18T15:28:37.081 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.081 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.081 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.081 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-18T15:28:37.081 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.081 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.081 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.081 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.081 INFO:teuthology.orchestra.run.smithi019.stdout: "allow_ptrace": { 2024-03-18T15:28:37.081 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "allow_ptrace", 2024-03-18T15:28:37.081 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.081 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.081 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.082 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.082 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.082 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.082 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.082 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-18T15:28:37.082 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-18T15:28:37.082 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.082 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.082 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.082 INFO:teuthology.orchestra.run.smithi019.stdout: "autotune_interval": { 2024-03-18T15:28:37.082 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "autotune_interval", 2024-03-18T15:28:37.082 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "secs", 2024-03-18T15:28:37.083 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.083 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.083 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "600", 2024-03-18T15:28:37.083 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.083 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.083 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.083 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-18T15:28:37.083 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.083 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.083 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.083 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.083 INFO:teuthology.orchestra.run.smithi019.stdout: "autotune_memory_target_ratio": { 2024-03-18T15:28:37.083 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "autotune_memory_target_ratio", 2024-03-18T15:28:37.083 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "float", 2024-03-18T15:28:37.084 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.084 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.084 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "0.7", 2024-03-18T15:28:37.084 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.084 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.084 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.084 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-18T15:28:37.084 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.084 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.084 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.084 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.084 INFO:teuthology.orchestra.run.smithi019.stdout: "cephadm_log_destination": { 2024-03-18T15:28:37.084 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "cephadm_log_destination", 2024-03-18T15:28:37.084 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.085 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.085 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.085 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.085 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.085 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.085 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.085 INFO:teuthology.orchestra.run.smithi019.stdout: "file", 2024-03-18T15:28:37.085 INFO:teuthology.orchestra.run.smithi019.stdout: "file,syslog", 2024-03-18T15:28:37.085 INFO:teuthology.orchestra.run.smithi019.stdout: "syslog" 2024-03-18T15:28:37.085 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.085 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-18T15:28:37.085 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.085 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.085 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.085 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.086 INFO:teuthology.orchestra.run.smithi019.stdout: "cgroups_split": { 2024-03-18T15:28:37.086 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "cgroups_split", 2024-03-18T15:28:37.086 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.086 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.086 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.086 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:37.086 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.086 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.086 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.086 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-18T15:28:37.086 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.086 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.086 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.087 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.087 INFO:teuthology.orchestra.run.smithi019.stdout: "config_checks_enabled": { 2024-03-18T15:28:37.087 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "config_checks_enabled", 2024-03-18T15:28:37.087 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.087 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.087 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.087 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.087 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.087 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.087 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.087 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-18T15:28:37.087 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.087 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.087 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.087 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.088 INFO:teuthology.orchestra.run.smithi019.stdout: "config_dashboard": { 2024-03-18T15:28:37.088 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "config_dashboard", 2024-03-18T15:28:37.088 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.088 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.088 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.088 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:37.088 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.088 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.088 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.088 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-18T15:28:37.088 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.088 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.088 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.088 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.089 INFO:teuthology.orchestra.run.smithi019.stdout: "container_image_alertmanager": { 2024-03-18T15:28:37.089 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "container_image_alertmanager", 2024-03-18T15:28:37.089 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.089 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.089 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.089 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-18T15:28:37.089 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.089 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.089 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.089 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Prometheus container image", 2024-03-18T15:28:37.089 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.089 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.089 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.089 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.089 INFO:teuthology.orchestra.run.smithi019.stdout: "container_image_base": { 2024-03-18T15:28:37.090 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "container_image_base", 2024-03-18T15:28:37.090 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.090 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.090 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.090 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-18T15:28:37.090 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.090 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.090 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.090 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Container image name, without the tag", 2024-03-18T15:28:37.090 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.090 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.090 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.090 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.090 INFO:teuthology.orchestra.run.smithi019.stdout: "container_image_elasticsearch": { 2024-03-18T15:28:37.090 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "container_image_elasticsearch", 2024-03-18T15:28:37.091 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.091 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.091 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.091 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-18T15:28:37.091 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.091 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.091 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.091 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "elasticsearch container image", 2024-03-18T15:28:37.091 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.091 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.091 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.091 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.091 INFO:teuthology.orchestra.run.smithi019.stdout: "container_image_grafana": { 2024-03-18T15:28:37.091 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "container_image_grafana", 2024-03-18T15:28:37.092 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.092 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.092 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.092 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-18T15:28:37.092 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.092 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.092 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.092 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Prometheus container image", 2024-03-18T15:28:37.092 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.092 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.092 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.092 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.092 INFO:teuthology.orchestra.run.smithi019.stdout: "container_image_haproxy": { 2024-03-18T15:28:37.092 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "container_image_haproxy", 2024-03-18T15:28:37.092 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.093 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.093 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.093 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-18T15:28:37.093 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.093 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.093 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.093 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "HAproxy container image", 2024-03-18T15:28:37.093 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.093 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.093 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.093 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.093 INFO:teuthology.orchestra.run.smithi019.stdout: "container_image_jaeger_agent": { 2024-03-18T15:28:37.093 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "container_image_jaeger_agent", 2024-03-18T15:28:37.093 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.094 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.094 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.094 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-18T15:28:37.094 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.094 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.094 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.094 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Jaeger agent container image", 2024-03-18T15:28:37.094 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.094 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.094 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.094 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.094 INFO:teuthology.orchestra.run.smithi019.stdout: "container_image_jaeger_collector": { 2024-03-18T15:28:37.094 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "container_image_jaeger_collector", 2024-03-18T15:28:37.095 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.095 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.095 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.095 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-18T15:28:37.095 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.095 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.095 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.095 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Jaeger collector container image", 2024-03-18T15:28:37.095 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.095 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.095 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.095 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.095 INFO:teuthology.orchestra.run.smithi019.stdout: "container_image_jaeger_query": { 2024-03-18T15:28:37.095 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "container_image_jaeger_query", 2024-03-18T15:28:37.096 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.096 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.096 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.096 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-18T15:28:37.096 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.096 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.096 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.096 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Jaeger query container image", 2024-03-18T15:28:37.096 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.096 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.096 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.096 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.096 INFO:teuthology.orchestra.run.smithi019.stdout: "container_image_keepalived": { 2024-03-18T15:28:37.096 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "container_image_keepalived", 2024-03-18T15:28:37.097 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.097 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.097 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.097 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-18T15:28:37.097 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.097 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.097 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.097 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Keepalived container image", 2024-03-18T15:28:37.097 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.097 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.097 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.097 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.097 INFO:teuthology.orchestra.run.smithi019.stdout: "container_image_loki": { 2024-03-18T15:28:37.097 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "container_image_loki", 2024-03-18T15:28:37.097 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.098 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.098 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.098 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-18T15:28:37.098 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.098 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.098 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.098 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Loki container image", 2024-03-18T15:28:37.098 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.098 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.098 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.098 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.098 INFO:teuthology.orchestra.run.smithi019.stdout: "container_image_node_exporter": { 2024-03-18T15:28:37.098 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "container_image_node_exporter", 2024-03-18T15:28:37.099 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.099 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.099 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.099 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-18T15:28:37.099 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.099 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.099 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.099 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Prometheus container image", 2024-03-18T15:28:37.099 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.099 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.099 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.099 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.099 INFO:teuthology.orchestra.run.smithi019.stdout: "container_image_nvmeof": { 2024-03-18T15:28:37.099 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "container_image_nvmeof", 2024-03-18T15:28:37.099 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.100 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.100 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.100 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-18T15:28:37.100 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.100 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.100 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.100 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Nvme-of container image", 2024-03-18T15:28:37.100 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.100 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.100 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.100 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.100 INFO:teuthology.orchestra.run.smithi019.stdout: "container_image_prometheus": { 2024-03-18T15:28:37.100 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "container_image_prometheus", 2024-03-18T15:28:37.100 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.101 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.101 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.101 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-18T15:28:37.101 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.101 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.101 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.101 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Prometheus container image", 2024-03-18T15:28:37.101 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.101 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.101 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.101 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.101 INFO:teuthology.orchestra.run.smithi019.stdout: "container_image_promtail": { 2024-03-18T15:28:37.101 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "container_image_promtail", 2024-03-18T15:28:37.101 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.102 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.102 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.102 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-18T15:28:37.102 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.102 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.102 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.102 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Promtail container image", 2024-03-18T15:28:37.102 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.102 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.102 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.102 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.102 INFO:teuthology.orchestra.run.smithi019.stdout: "container_image_snmp_gateway": { 2024-03-18T15:28:37.102 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "container_image_snmp_gateway", 2024-03-18T15:28:37.102 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.103 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.103 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.103 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-18T15:28:37.103 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.103 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.103 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.103 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "SNMP Gateway container image", 2024-03-18T15:28:37.103 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.103 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.103 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.103 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.103 INFO:teuthology.orchestra.run.smithi019.stdout: "container_init": { 2024-03-18T15:28:37.103 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "container_init", 2024-03-18T15:28:37.103 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.103 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.104 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.104 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:37.104 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.104 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.104 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.104 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Run podman/docker with `--init`", 2024-03-18T15:28:37.104 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.104 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.104 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.104 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.104 INFO:teuthology.orchestra.run.smithi019.stdout: "daemon_cache_timeout": { 2024-03-18T15:28:37.104 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "daemon_cache_timeout", 2024-03-18T15:28:37.104 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "secs", 2024-03-18T15:28:37.104 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.104 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.105 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "600", 2024-03-18T15:28:37.105 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.105 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.105 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.105 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-18T15:28:37.105 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.105 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.105 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.105 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.105 INFO:teuthology.orchestra.run.smithi019.stdout: "default_cephadm_command_timeout": { 2024-03-18T15:28:37.105 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "default_cephadm_command_timeout", 2024-03-18T15:28:37.105 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:37.105 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.105 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.105 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "900", 2024-03-18T15:28:37.106 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.106 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.106 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.106 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-18T15:28:37.106 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.106 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.106 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.106 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.106 INFO:teuthology.orchestra.run.smithi019.stdout: "default_registry": { 2024-03-18T15:28:37.106 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "default_registry", 2024-03-18T15:28:37.106 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.106 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.107 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.107 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "docker.io", 2024-03-18T15:28:37.107 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.107 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.107 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.107 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-18T15:28:37.107 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.107 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.107 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.107 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.107 INFO:teuthology.orchestra.run.smithi019.stdout: "device_cache_timeout": { 2024-03-18T15:28:37.107 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "device_cache_timeout", 2024-03-18T15:28:37.107 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "secs", 2024-03-18T15:28:37.107 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.108 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.108 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "1800", 2024-03-18T15:28:37.108 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.108 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.108 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.108 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "seconds to cache device inventory", 2024-03-18T15:28:37.108 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.108 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.108 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.108 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.108 INFO:teuthology.orchestra.run.smithi019.stdout: "device_enhanced_scan": { 2024-03-18T15:28:37.108 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "device_enhanced_scan", 2024-03-18T15:28:37.108 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.108 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.108 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.109 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.109 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.109 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.109 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.109 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-18T15:28:37.109 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.109 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.109 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.109 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.109 INFO:teuthology.orchestra.run.smithi019.stdout: "facts_cache_timeout": { 2024-03-18T15:28:37.109 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "facts_cache_timeout", 2024-03-18T15:28:37.109 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "secs", 2024-03-18T15:28:37.109 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.109 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.110 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "60", 2024-03-18T15:28:37.110 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.110 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.110 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.110 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "seconds to cache host facts data", 2024-03-18T15:28:37.110 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.110 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.110 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.110 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.110 INFO:teuthology.orchestra.run.smithi019.stdout: "host_check_interval": { 2024-03-18T15:28:37.110 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "host_check_interval", 2024-03-18T15:28:37.110 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "secs", 2024-03-18T15:28:37.110 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.110 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.111 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "600", 2024-03-18T15:28:37.111 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.111 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.111 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.111 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "how frequently to perform a host check", 2024-03-18T15:28:37.111 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.111 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.111 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.111 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.111 INFO:teuthology.orchestra.run.smithi019.stdout: "hw_monitoring": { 2024-03-18T15:28:37.111 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "hw_monitoring", 2024-03-18T15:28:37.111 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.111 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.111 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.111 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.112 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.112 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.112 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.112 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-18T15:28:37.112 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.112 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.112 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.112 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.112 INFO:teuthology.orchestra.run.smithi019.stdout: "inventory_list_all": { 2024-03-18T15:28:37.112 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "inventory_list_all", 2024-03-18T15:28:37.112 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.112 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.112 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.112 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.112 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.113 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.113 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.113 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-18T15:28:37.113 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.113 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.113 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.113 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.113 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:37.113 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:37.113 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.113 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.113 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.113 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.113 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.114 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.114 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.114 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.114 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.114 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.114 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.114 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.114 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.114 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.114 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.114 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.114 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.114 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.115 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.115 INFO:teuthology.orchestra.run.smithi019.stdout: "log_refresh_metadata": { 2024-03-18T15:28:37.115 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_refresh_metadata", 2024-03-18T15:28:37.115 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.115 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.115 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.115 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.115 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.115 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.115 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.115 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-18T15:28:37.115 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.115 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.115 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.115 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.116 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:37.116 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:37.116 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.116 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.116 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.116 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:37.116 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.116 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.116 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.116 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-18T15:28:37.116 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.116 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.116 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.116 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.116 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:37.117 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:37.117 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.117 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.117 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.117 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:37.117 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.117 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.117 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.117 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.117 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.117 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.117 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.117 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.117 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.118 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.118 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.118 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.118 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.118 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.118 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.118 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:37.118 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:37.118 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.118 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.118 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.118 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.118 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.119 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.119 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.119 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.119 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.119 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.119 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.119 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.119 INFO:teuthology.orchestra.run.smithi019.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-18T15:28:37.119 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-18T15:28:37.119 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.119 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.119 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.119 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.119 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.119 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.120 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.120 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-18T15:28:37.120 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.120 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.120 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.120 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.120 INFO:teuthology.orchestra.run.smithi019.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-18T15:28:37.120 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-18T15:28:37.120 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.120 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.120 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.120 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "*", 2024-03-18T15:28:37.120 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.120 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.121 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.121 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-18T15:28:37.121 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.121 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.121 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.121 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.121 INFO:teuthology.orchestra.run.smithi019.stdout: "max_count_per_host": { 2024-03-18T15:28:37.121 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "max_count_per_host", 2024-03-18T15:28:37.121 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:37.121 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.121 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.121 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "10", 2024-03-18T15:28:37.121 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.121 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.121 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.122 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "max number of daemons per service per host", 2024-03-18T15:28:37.122 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.122 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.122 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.122 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.122 INFO:teuthology.orchestra.run.smithi019.stdout: "max_osd_draining_count": { 2024-03-18T15:28:37.122 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "max_osd_draining_count", 2024-03-18T15:28:37.122 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:37.122 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.122 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.122 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "10", 2024-03-18T15:28:37.122 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.122 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.122 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.123 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-18T15:28:37.123 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.123 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.123 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.123 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.123 INFO:teuthology.orchestra.run.smithi019.stdout: "migration_current": { 2024-03-18T15:28:37.123 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "migration_current", 2024-03-18T15:28:37.123 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:37.123 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.123 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.123 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.123 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.123 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.123 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.124 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "internal - do not modify", 2024-03-18T15:28:37.124 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.124 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.124 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.124 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.124 INFO:teuthology.orchestra.run.smithi019.stdout: "mode": { 2024-03-18T15:28:37.124 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "mode", 2024-03-18T15:28:37.124 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.124 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.124 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.124 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "root", 2024-03-18T15:28:37.124 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.124 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.124 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.125 INFO:teuthology.orchestra.run.smithi019.stdout: "cephadm-package", 2024-03-18T15:28:37.125 INFO:teuthology.orchestra.run.smithi019.stdout: "root" 2024-03-18T15:28:37.125 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.125 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "mode for remote execution of cephadm", 2024-03-18T15:28:37.125 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.125 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.125 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.125 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.125 INFO:teuthology.orchestra.run.smithi019.stdout: "oob_default_addr": { 2024-03-18T15:28:37.125 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "oob_default_addr", 2024-03-18T15:28:37.125 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.125 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.125 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.125 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "169.254.1.1", 2024-03-18T15:28:37.125 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.126 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.126 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.126 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-18T15:28:37.126 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.126 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.126 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.126 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.126 INFO:teuthology.orchestra.run.smithi019.stdout: "prometheus_alerts_path": { 2024-03-18T15:28:37.126 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "prometheus_alerts_path", 2024-03-18T15:28:37.126 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.126 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.126 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.126 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-18T15:28:37.126 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.127 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.127 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.127 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-18T15:28:37.127 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.127 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.127 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.127 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.127 INFO:teuthology.orchestra.run.smithi019.stdout: "registry_insecure": { 2024-03-18T15:28:37.127 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "registry_insecure", 2024-03-18T15:28:37.127 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.127 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.127 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.127 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.127 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.128 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.128 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.128 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-18T15:28:37.128 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.128 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.128 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.128 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.128 INFO:teuthology.orchestra.run.smithi019.stdout: "registry_password": { 2024-03-18T15:28:37.128 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "registry_password", 2024-03-18T15:28:37.128 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.128 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.128 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.128 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.128 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.128 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.129 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.129 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-18T15:28:37.129 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.129 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.129 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.129 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.129 INFO:teuthology.orchestra.run.smithi019.stdout: "registry_url": { 2024-03-18T15:28:37.129 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "registry_url", 2024-03-18T15:28:37.129 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.129 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.129 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.129 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.129 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.129 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.129 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.130 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-18T15:28:37.130 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.130 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.130 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.130 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.130 INFO:teuthology.orchestra.run.smithi019.stdout: "registry_username": { 2024-03-18T15:28:37.130 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "registry_username", 2024-03-18T15:28:37.130 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.130 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.130 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.130 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.130 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.130 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.131 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.131 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-18T15:28:37.131 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.131 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.131 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.131 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.131 INFO:teuthology.orchestra.run.smithi019.stdout: "secure_monitoring_stack": { 2024-03-18T15:28:37.131 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "secure_monitoring_stack", 2024-03-18T15:28:37.131 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.131 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.131 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.131 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.131 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.131 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.132 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.132 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-18T15:28:37.132 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.132 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.132 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.132 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.132 INFO:teuthology.orchestra.run.smithi019.stdout: "service_discovery_port": { 2024-03-18T15:28:37.132 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "service_discovery_port", 2024-03-18T15:28:37.132 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:37.132 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.132 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.132 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "8765", 2024-03-18T15:28:37.132 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.132 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.133 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.133 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "cephadm service discovery port", 2024-03-18T15:28:37.133 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.133 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.133 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.133 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.133 INFO:teuthology.orchestra.run.smithi019.stdout: "ssh_config_file": { 2024-03-18T15:28:37.133 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "ssh_config_file", 2024-03-18T15:28:37.133 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.133 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.133 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.133 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.133 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.133 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.134 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.134 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-18T15:28:37.134 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.134 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.134 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.134 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.134 INFO:teuthology.orchestra.run.smithi019.stdout: "use_agent": { 2024-03-18T15:28:37.134 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "use_agent", 2024-03-18T15:28:37.134 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.134 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.134 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.134 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.134 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.134 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.135 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.135 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-18T15:28:37.135 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.135 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.135 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.135 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.135 INFO:teuthology.orchestra.run.smithi019.stdout: "use_repo_digest": { 2024-03-18T15:28:37.135 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "use_repo_digest", 2024-03-18T15:28:37.135 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.135 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.135 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.135 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:37.135 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.135 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.136 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.136 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-18T15:28:37.136 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.136 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.136 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.136 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.136 INFO:teuthology.orchestra.run.smithi019.stdout: "warn_on_failed_host_check": { 2024-03-18T15:28:37.136 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "warn_on_failed_host_check", 2024-03-18T15:28:37.136 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.136 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.136 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.136 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:37.136 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.136 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.137 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.137 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "raise a health warning if the host check fails", 2024-03-18T15:28:37.137 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.137 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.137 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.137 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.137 INFO:teuthology.orchestra.run.smithi019.stdout: "warn_on_stray_daemons": { 2024-03-18T15:28:37.137 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "warn_on_stray_daemons", 2024-03-18T15:28:37.137 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.137 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.137 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.137 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:37.137 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.137 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.138 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.138 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-18T15:28:37.138 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.138 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.138 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.138 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.138 INFO:teuthology.orchestra.run.smithi019.stdout: "warn_on_stray_hosts": { 2024-03-18T15:28:37.138 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "warn_on_stray_hosts", 2024-03-18T15:28:37.138 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.138 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.138 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.138 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:37.138 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.138 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.139 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.139 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-18T15:28:37.139 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.139 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.139 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.139 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.139 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.139 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.139 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:37.139 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "crash", 2024-03-18T15:28:37.139 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": true, 2024-03-18T15:28:37.139 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "", 2024-03-18T15:28:37.139 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:37.139 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:37.140 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:37.140 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.140 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.140 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.140 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.140 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.140 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.140 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.140 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.140 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.140 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.140 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.140 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.140 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.140 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.141 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.141 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.141 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.141 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.141 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.141 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:37.141 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:37.141 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.141 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.141 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.141 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.141 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.141 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.141 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.141 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.142 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.142 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.142 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.142 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.142 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:37.142 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:37.142 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.142 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.142 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.142 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:37.142 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.142 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.142 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.142 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.143 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.143 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.143 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.143 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.143 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.143 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.143 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.143 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.143 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.143 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.143 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.143 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:37.143 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:37.144 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.144 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.144 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.144 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.144 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.144 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.144 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.144 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.144 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.144 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.144 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.144 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.144 INFO:teuthology.orchestra.run.smithi019.stdout: "retain_interval": { 2024-03-18T15:28:37.144 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "retain_interval", 2024-03-18T15:28:37.145 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "secs", 2024-03-18T15:28:37.145 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.145 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.145 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "31536000", 2024-03-18T15:28:37.145 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.145 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.145 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.145 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-18T15:28:37.145 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.145 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.145 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.145 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.145 INFO:teuthology.orchestra.run.smithi019.stdout: "warn_recent_interval": { 2024-03-18T15:28:37.145 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "warn_recent_interval", 2024-03-18T15:28:37.145 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "secs", 2024-03-18T15:28:37.146 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.146 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.146 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "1209600", 2024-03-18T15:28:37.146 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.146 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.146 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.146 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-18T15:28:37.146 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.146 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.146 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.146 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.146 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.146 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.146 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:37.147 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "dashboard", 2024-03-18T15:28:37.147 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": true, 2024-03-18T15:28:37.147 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "", 2024-03-18T15:28:37.147 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:37.147 INFO:teuthology.orchestra.run.smithi019.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-18T15:28:37.147 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-18T15:28:37.147 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:37.147 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.147 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.147 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "10", 2024-03-18T15:28:37.147 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.147 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.147 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.147 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.147 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.148 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.148 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.148 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.148 INFO:teuthology.orchestra.run.smithi019.stdout: "ALERTMANAGER_API_HOST": { 2024-03-18T15:28:37.148 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-18T15:28:37.148 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.148 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.148 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.148 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.148 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.148 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.148 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.148 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.148 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.149 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.149 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.149 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.149 INFO:teuthology.orchestra.run.smithi019.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-18T15:28:37.149 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-18T15:28:37.149 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.149 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.149 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.149 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:37.149 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.149 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.149 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.149 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.149 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.149 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.150 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.150 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.150 INFO:teuthology.orchestra.run.smithi019.stdout: "AUDIT_API_ENABLED": { 2024-03-18T15:28:37.150 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "AUDIT_API_ENABLED", 2024-03-18T15:28:37.150 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.150 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.150 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.150 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.150 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.150 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.150 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.150 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.150 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.150 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.151 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.151 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.151 INFO:teuthology.orchestra.run.smithi019.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-18T15:28:37.151 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-18T15:28:37.151 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.151 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.151 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.151 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:37.151 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.151 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.151 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.151 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.151 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.151 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.151 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.152 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.152 INFO:teuthology.orchestra.run.smithi019.stdout: "ENABLE_BROWSABLE_API": { 2024-03-18T15:28:37.152 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-18T15:28:37.152 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.152 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.152 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.152 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:37.152 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.152 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.152 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.152 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.152 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.152 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.152 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.153 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.153 INFO:teuthology.orchestra.run.smithi019.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-18T15:28:37.153 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-18T15:28:37.153 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.153 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.153 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.153 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:37.153 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.153 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.153 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.153 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.153 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.153 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.153 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.154 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.154 INFO:teuthology.orchestra.run.smithi019.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-18T15:28:37.154 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-18T15:28:37.154 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.154 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.154 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.154 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:37.154 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.154 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.154 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.154 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.154 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.154 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.154 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.155 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.155 INFO:teuthology.orchestra.run.smithi019.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-18T15:28:37.155 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-18T15:28:37.155 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.155 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.155 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.155 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:37.155 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.155 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.155 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.155 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.155 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.155 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.155 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.155 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.156 INFO:teuthology.orchestra.run.smithi019.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-18T15:28:37.156 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-18T15:28:37.156 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.156 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.156 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.156 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:37.156 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.156 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.156 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.156 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.156 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.156 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.156 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.157 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.157 INFO:teuthology.orchestra.run.smithi019.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-18T15:28:37.157 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-18T15:28:37.157 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.157 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.157 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.157 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:37.157 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.157 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.157 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.157 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.157 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.157 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.157 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.158 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.158 INFO:teuthology.orchestra.run.smithi019.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-18T15:28:37.158 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-18T15:28:37.158 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.158 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.158 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.158 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:37.158 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.158 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.158 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.158 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.158 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.158 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.158 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.159 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.159 INFO:teuthology.orchestra.run.smithi019.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-18T15:28:37.159 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-18T15:28:37.159 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.159 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.159 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.159 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:37.159 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.159 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.159 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.159 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.159 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.159 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.159 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.160 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.160 INFO:teuthology.orchestra.run.smithi019.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-18T15:28:37.160 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-18T15:28:37.160 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.160 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.160 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.160 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.160 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.160 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.160 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.160 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.160 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.160 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.160 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.160 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.161 INFO:teuthology.orchestra.run.smithi019.stdout: "GRAFANA_API_PASSWORD": { 2024-03-18T15:28:37.161 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-18T15:28:37.161 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.161 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.161 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.161 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "admin", 2024-03-18T15:28:37.161 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.161 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.161 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.161 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.161 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.161 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.161 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.161 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.162 INFO:teuthology.orchestra.run.smithi019.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-18T15:28:37.162 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-18T15:28:37.162 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.162 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.162 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.162 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:37.162 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.162 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.162 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.162 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.162 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.162 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.162 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.163 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.163 INFO:teuthology.orchestra.run.smithi019.stdout: "GRAFANA_API_URL": { 2024-03-18T15:28:37.163 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "GRAFANA_API_URL", 2024-03-18T15:28:37.163 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.163 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.163 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.163 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.163 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.163 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.163 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.163 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.163 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.163 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.163 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.163 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.164 INFO:teuthology.orchestra.run.smithi019.stdout: "GRAFANA_API_USERNAME": { 2024-03-18T15:28:37.164 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-18T15:28:37.164 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.164 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.164 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.164 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "admin", 2024-03-18T15:28:37.164 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.164 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.164 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.164 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.164 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.164 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.164 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.164 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.165 INFO:teuthology.orchestra.run.smithi019.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-18T15:28:37.165 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-18T15:28:37.165 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.165 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.165 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.165 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.165 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.165 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.165 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.165 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.165 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.165 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.165 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.165 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.165 INFO:teuthology.orchestra.run.smithi019.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-18T15:28:37.166 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-18T15:28:37.166 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.166 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.166 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.166 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.166 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.166 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.166 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.166 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.166 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.166 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.166 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.166 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.166 INFO:teuthology.orchestra.run.smithi019.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-18T15:28:37.166 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-18T15:28:37.167 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.167 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.167 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.167 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:37.167 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.167 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.167 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.167 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.167 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.167 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.167 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.167 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.167 INFO:teuthology.orchestra.run.smithi019.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-18T15:28:37.167 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-18T15:28:37.168 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.168 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.168 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.168 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.168 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.168 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.168 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.168 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.168 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.168 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.168 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.168 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.168 INFO:teuthology.orchestra.run.smithi019.stdout: "MULTICLUSTER_CONFIG": { 2024-03-18T15:28:37.168 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-18T15:28:37.168 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.169 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.169 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.169 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "{}", 2024-03-18T15:28:37.169 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.169 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.169 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.169 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.169 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.169 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.169 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.169 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.169 INFO:teuthology.orchestra.run.smithi019.stdout: "PROMETHEUS_API_HOST": { 2024-03-18T15:28:37.170 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-18T15:28:37.170 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.170 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.170 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.170 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.170 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.170 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.170 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.170 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.170 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.170 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.170 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.170 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.171 INFO:teuthology.orchestra.run.smithi019.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-18T15:28:37.171 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-18T15:28:37.171 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.171 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.171 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.171 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:37.171 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.171 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.171 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.171 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.171 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.171 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.171 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.171 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.172 INFO:teuthology.orchestra.run.smithi019.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-18T15:28:37.172 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-18T15:28:37.172 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.172 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.172 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.172 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.172 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.172 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.172 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.172 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.172 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.172 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.172 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.172 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.172 INFO:teuthology.orchestra.run.smithi019.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-18T15:28:37.173 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-18T15:28:37.173 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.173 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.173 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.173 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.173 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.173 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.173 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.173 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.173 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.173 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.173 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.173 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.173 INFO:teuthology.orchestra.run.smithi019.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-18T15:28:37.174 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-18T15:28:37.174 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.174 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.174 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.174 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:37.174 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.174 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.174 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.174 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.174 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.174 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.174 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.174 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.175 INFO:teuthology.orchestra.run.smithi019.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-18T15:28:37.175 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-18T15:28:37.175 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.175 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.175 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.175 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:37.175 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.175 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.175 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.175 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.175 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.175 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.175 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.175 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.176 INFO:teuthology.orchestra.run.smithi019.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-18T15:28:37.176 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-18T15:28:37.176 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.176 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.176 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.176 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.176 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.176 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.176 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.176 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.176 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.176 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.176 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.176 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.176 INFO:teuthology.orchestra.run.smithi019.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-18T15:28:37.177 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-18T15:28:37.177 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.177 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.177 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.177 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.177 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.177 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.177 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.177 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.177 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.177 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.177 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.177 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.177 INFO:teuthology.orchestra.run.smithi019.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-18T15:28:37.177 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-18T15:28:37.178 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.178 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.178 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.178 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.178 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.178 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.178 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.178 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.178 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.178 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.178 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.178 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.178 INFO:teuthology.orchestra.run.smithi019.stdout: "PWD_POLICY_ENABLED": { 2024-03-18T15:28:37.178 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-18T15:28:37.179 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.179 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.179 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.179 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:37.179 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.179 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.179 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.179 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.179 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.179 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.179 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.179 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.179 INFO:teuthology.orchestra.run.smithi019.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-18T15:28:37.179 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-18T15:28:37.179 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.180 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.180 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.180 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-18T15:28:37.180 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.180 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.180 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.180 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.180 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.180 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.180 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.180 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.180 INFO:teuthology.orchestra.run.smithi019.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-18T15:28:37.180 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-18T15:28:37.180 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:37.180 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.181 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.181 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "10", 2024-03-18T15:28:37.181 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.181 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.181 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.181 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.181 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.181 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.181 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.181 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.181 INFO:teuthology.orchestra.run.smithi019.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-18T15:28:37.181 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-18T15:28:37.181 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:37.181 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.182 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.182 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "8", 2024-03-18T15:28:37.182 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.182 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.182 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.182 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.182 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.182 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.182 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.182 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.182 INFO:teuthology.orchestra.run.smithi019.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-18T15:28:37.182 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-18T15:28:37.182 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:37.183 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.183 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.183 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "45", 2024-03-18T15:28:37.183 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.183 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.183 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.183 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.183 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.183 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.183 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.183 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.183 INFO:teuthology.orchestra.run.smithi019.stdout: "RGW_API_ACCESS_KEY": { 2024-03-18T15:28:37.183 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-18T15:28:37.183 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.184 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.184 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.184 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.184 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.184 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.184 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.184 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.184 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.184 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.184 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.184 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.184 INFO:teuthology.orchestra.run.smithi019.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-18T15:28:37.184 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-18T15:28:37.184 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.185 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.185 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.185 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "admin", 2024-03-18T15:28:37.185 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.185 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.185 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.185 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.185 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.185 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.185 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.185 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.185 INFO:teuthology.orchestra.run.smithi019.stdout: "RGW_API_SECRET_KEY": { 2024-03-18T15:28:37.185 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-18T15:28:37.185 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.185 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.186 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.186 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.186 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.186 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.186 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.186 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.186 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.186 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.186 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.186 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.186 INFO:teuthology.orchestra.run.smithi019.stdout: "RGW_API_SSL_VERIFY": { 2024-03-18T15:28:37.186 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-18T15:28:37.186 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.187 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.187 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.187 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:37.187 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.187 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.187 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.187 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.187 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.187 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.187 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.187 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.187 INFO:teuthology.orchestra.run.smithi019.stdout: "UNSAFE_TLS_v1_2": { 2024-03-18T15:28:37.187 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-18T15:28:37.187 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.187 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.188 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.188 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.188 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.188 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.188 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.188 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.188 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.188 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.188 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.188 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.188 INFO:teuthology.orchestra.run.smithi019.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-18T15:28:37.188 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-18T15:28:37.188 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:37.188 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.189 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.189 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "0", 2024-03-18T15:28:37.189 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.189 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.189 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.189 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.189 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.189 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.189 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.189 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.189 INFO:teuthology.orchestra.run.smithi019.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-18T15:28:37.189 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-18T15:28:37.189 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:37.189 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.189 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.190 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "10", 2024-03-18T15:28:37.190 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.190 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.190 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.190 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.190 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.190 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.190 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.190 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.190 INFO:teuthology.orchestra.run.smithi019.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-18T15:28:37.190 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-18T15:28:37.190 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:37.190 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.190 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.191 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "5", 2024-03-18T15:28:37.191 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.191 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.191 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.191 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.191 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.191 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.191 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.191 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.191 INFO:teuthology.orchestra.run.smithi019.stdout: "cross_origin_url": { 2024-03-18T15:28:37.191 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "cross_origin_url", 2024-03-18T15:28:37.191 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.191 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.191 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.191 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.192 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.192 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.192 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.192 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.192 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.192 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.192 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.192 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.192 INFO:teuthology.orchestra.run.smithi019.stdout: "crt_file": { 2024-03-18T15:28:37.192 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "crt_file", 2024-03-18T15:28:37.192 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.192 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.192 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.192 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.192 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.193 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.193 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.193 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.193 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.193 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.193 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.193 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.193 INFO:teuthology.orchestra.run.smithi019.stdout: "debug": { 2024-03-18T15:28:37.193 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "debug", 2024-03-18T15:28:37.193 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.193 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.193 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.193 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.193 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.194 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.194 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.194 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Enable/disable debug options", 2024-03-18T15:28:37.194 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.194 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.194 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.194 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.194 INFO:teuthology.orchestra.run.smithi019.stdout: "jwt_token_ttl": { 2024-03-18T15:28:37.194 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "jwt_token_ttl", 2024-03-18T15:28:37.194 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:37.194 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.194 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.194 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "28800", 2024-03-18T15:28:37.194 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.195 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.195 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.195 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.195 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.195 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.195 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.195 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.195 INFO:teuthology.orchestra.run.smithi019.stdout: "key_file": { 2024-03-18T15:28:37.195 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "key_file", 2024-03-18T15:28:37.195 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.195 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.195 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.196 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.196 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.196 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.196 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.196 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.196 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.196 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.196 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.196 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.196 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:37.196 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:37.196 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.196 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.196 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.196 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.197 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.197 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.197 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.197 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.197 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.197 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.197 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.197 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.197 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.197 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.197 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.197 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.197 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.197 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.198 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.198 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:37.198 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:37.198 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.198 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.198 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.198 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.198 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.198 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.198 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.198 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.198 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.198 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.198 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.199 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.199 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:37.199 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:37.199 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.199 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.199 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.199 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:37.199 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.199 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.199 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.199 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.199 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.199 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.199 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.200 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.200 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.200 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.200 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.200 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.200 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.200 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.200 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.200 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:37.200 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:37.200 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.200 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.200 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.200 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.200 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.201 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.201 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.201 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.201 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.201 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.201 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.201 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.201 INFO:teuthology.orchestra.run.smithi019.stdout: "motd": { 2024-03-18T15:28:37.201 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "motd", 2024-03-18T15:28:37.201 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.201 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.201 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.201 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.201 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.202 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.202 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.202 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "The message of the day", 2024-03-18T15:28:37.202 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.202 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.202 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.202 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.202 INFO:teuthology.orchestra.run.smithi019.stdout: "redirect_resolve_ip_addr": { 2024-03-18T15:28:37.202 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "redirect_resolve_ip_addr", 2024-03-18T15:28:37.202 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.202 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.202 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.202 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.202 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.203 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.203 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.203 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.203 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.203 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.203 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.203 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.203 INFO:teuthology.orchestra.run.smithi019.stdout: "server_addr": { 2024-03-18T15:28:37.203 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "server_addr", 2024-03-18T15:28:37.203 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.203 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.203 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.203 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "::", 2024-03-18T15:28:37.203 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.204 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.204 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.204 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.204 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.204 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.204 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.204 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.204 INFO:teuthology.orchestra.run.smithi019.stdout: "server_port": { 2024-03-18T15:28:37.204 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "server_port", 2024-03-18T15:28:37.204 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:37.204 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.204 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.204 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "8080", 2024-03-18T15:28:37.204 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.205 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.205 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.205 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.205 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.205 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.205 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.205 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.205 INFO:teuthology.orchestra.run.smithi019.stdout: "ssl": { 2024-03-18T15:28:37.205 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "ssl", 2024-03-18T15:28:37.205 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.205 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.205 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.205 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:37.205 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.206 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.206 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.206 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.206 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.206 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.206 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.206 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.206 INFO:teuthology.orchestra.run.smithi019.stdout: "ssl_server_port": { 2024-03-18T15:28:37.206 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "ssl_server_port", 2024-03-18T15:28:37.206 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:37.206 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.206 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.206 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "8443", 2024-03-18T15:28:37.206 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.206 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.207 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.207 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.207 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.207 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.207 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.207 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.207 INFO:teuthology.orchestra.run.smithi019.stdout: "standby_behaviour": { 2024-03-18T15:28:37.207 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "standby_behaviour", 2024-03-18T15:28:37.207 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.207 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.207 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.207 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "redirect", 2024-03-18T15:28:37.207 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.207 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.208 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.208 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.208 INFO:teuthology.orchestra.run.smithi019.stdout: "redirect" 2024-03-18T15:28:37.208 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.208 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.208 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.208 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.208 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.208 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.208 INFO:teuthology.orchestra.run.smithi019.stdout: "standby_error_status_code": { 2024-03-18T15:28:37.208 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "standby_error_status_code", 2024-03-18T15:28:37.208 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:37.208 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.208 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.209 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "500", 2024-03-18T15:28:37.209 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "400", 2024-03-18T15:28:37.209 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "599", 2024-03-18T15:28:37.209 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.209 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.209 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.209 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.209 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.209 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.209 INFO:teuthology.orchestra.run.smithi019.stdout: "url_prefix": { 2024-03-18T15:28:37.209 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "url_prefix", 2024-03-18T15:28:37.209 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.209 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.209 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.209 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.210 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.210 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.210 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.210 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.210 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.210 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.210 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.210 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.210 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.210 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.210 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:37.210 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "devicehealth", 2024-03-18T15:28:37.210 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": true, 2024-03-18T15:28:37.211 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "", 2024-03-18T15:28:37.211 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:37.211 INFO:teuthology.orchestra.run.smithi019.stdout: "enable_monitoring": { 2024-03-18T15:28:37.211 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "enable_monitoring", 2024-03-18T15:28:37.211 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.211 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.211 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.211 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:37.211 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.211 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.211 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.211 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "monitor device health metrics", 2024-03-18T15:28:37.211 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.211 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.211 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.212 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.212 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:37.212 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:37.212 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.212 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.212 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.212 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.212 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.212 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.212 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.212 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.212 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.212 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.212 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.213 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.213 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.213 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.213 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.213 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.213 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.213 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.213 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.213 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:37.213 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:37.213 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.213 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.213 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.213 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.214 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.214 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.214 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.214 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.214 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.214 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.214 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.214 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.214 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:37.214 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:37.214 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.214 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.214 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.215 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:37.215 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.215 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.215 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.215 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.215 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.215 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.215 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.215 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.215 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.215 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.215 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.215 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.215 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.215 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.216 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.216 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:37.216 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:37.216 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.216 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.216 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.216 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.216 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.216 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.216 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.216 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.216 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.216 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.216 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.216 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.217 INFO:teuthology.orchestra.run.smithi019.stdout: "mark_out_threshold": { 2024-03-18T15:28:37.217 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "mark_out_threshold", 2024-03-18T15:28:37.217 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "secs", 2024-03-18T15:28:37.217 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.217 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.217 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "2419200", 2024-03-18T15:28:37.217 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.217 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.217 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.217 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-18T15:28:37.217 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.217 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.217 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.217 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.218 INFO:teuthology.orchestra.run.smithi019.stdout: "pool_name": { 2024-03-18T15:28:37.218 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "pool_name", 2024-03-18T15:28:37.218 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.218 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.218 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.218 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "device_health_metrics", 2024-03-18T15:28:37.218 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.218 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.218 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.218 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-18T15:28:37.218 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.218 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.218 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.218 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.218 INFO:teuthology.orchestra.run.smithi019.stdout: "retention_period": { 2024-03-18T15:28:37.219 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "retention_period", 2024-03-18T15:28:37.219 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "secs", 2024-03-18T15:28:37.219 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.219 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.219 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "15552000", 2024-03-18T15:28:37.219 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.219 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.219 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.219 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "how long to retain device health metrics", 2024-03-18T15:28:37.219 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.219 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.219 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.219 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.219 INFO:teuthology.orchestra.run.smithi019.stdout: "scrape_frequency": { 2024-03-18T15:28:37.220 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "scrape_frequency", 2024-03-18T15:28:37.220 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "secs", 2024-03-18T15:28:37.220 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.220 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.220 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "86400", 2024-03-18T15:28:37.220 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.220 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.220 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.220 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-18T15:28:37.220 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.220 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.220 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.220 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.220 INFO:teuthology.orchestra.run.smithi019.stdout: "self_heal": { 2024-03-18T15:28:37.220 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "self_heal", 2024-03-18T15:28:37.221 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.221 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.221 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.221 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:37.221 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.221 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.221 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.221 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-18T15:28:37.221 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.221 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.221 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.221 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.221 INFO:teuthology.orchestra.run.smithi019.stdout: "sleep_interval": { 2024-03-18T15:28:37.222 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "sleep_interval", 2024-03-18T15:28:37.222 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "secs", 2024-03-18T15:28:37.222 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.222 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.222 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "600", 2024-03-18T15:28:37.222 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.222 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.222 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.222 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "how frequently to wake up and check device health", 2024-03-18T15:28:37.222 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.222 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.222 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.222 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.222 INFO:teuthology.orchestra.run.smithi019.stdout: "warn_threshold": { 2024-03-18T15:28:37.223 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "warn_threshold", 2024-03-18T15:28:37.223 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "secs", 2024-03-18T15:28:37.223 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.223 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.223 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "7257600", 2024-03-18T15:28:37.223 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.223 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.223 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.223 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-18T15:28:37.223 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.223 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.223 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.223 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.223 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.223 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.224 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:37.224 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "diskprediction_local", 2024-03-18T15:28:37.224 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": true, 2024-03-18T15:28:37.224 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "", 2024-03-18T15:28:37.224 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:37.224 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:37.224 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:37.224 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.224 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.224 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.224 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.224 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.224 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.225 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.225 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.225 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.225 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.225 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.225 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.225 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.225 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.225 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.225 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.225 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.225 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.225 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.225 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:37.225 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:37.226 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.226 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.226 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.226 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.226 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.226 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.226 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.226 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.226 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.226 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.226 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.226 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.226 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:37.227 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:37.227 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.227 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.227 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.227 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:37.227 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.227 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.227 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.227 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.227 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.227 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.227 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.227 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.227 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.227 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.228 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.228 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.228 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.228 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.228 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.228 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:37.228 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:37.228 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.228 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.228 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.228 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.228 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.228 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.228 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.228 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.229 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.229 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.229 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.229 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.229 INFO:teuthology.orchestra.run.smithi019.stdout: "predict_interval": { 2024-03-18T15:28:37.229 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "predict_interval", 2024-03-18T15:28:37.229 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.229 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.229 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.229 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "86400", 2024-03-18T15:28:37.229 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.229 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.229 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.229 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.229 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.230 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.230 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.230 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.230 INFO:teuthology.orchestra.run.smithi019.stdout: "predictor_model": { 2024-03-18T15:28:37.230 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "predictor_model", 2024-03-18T15:28:37.230 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.230 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.230 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.230 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "prophetstor", 2024-03-18T15:28:37.230 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.230 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.230 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.230 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.231 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.231 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.231 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.231 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.231 INFO:teuthology.orchestra.run.smithi019.stdout: "sleep_interval": { 2024-03-18T15:28:37.231 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "sleep_interval", 2024-03-18T15:28:37.231 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.231 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.231 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.231 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "600", 2024-03-18T15:28:37.231 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.231 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.231 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.231 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.232 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.232 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.232 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.232 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.232 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.232 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.232 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:37.232 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "influx", 2024-03-18T15:28:37.232 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": false, 2024-03-18T15:28:37.232 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "influxdb python module not found", 2024-03-18T15:28:37.232 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:37.232 INFO:teuthology.orchestra.run.smithi019.stdout: "batch_size": { 2024-03-18T15:28:37.232 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "batch_size", 2024-03-18T15:28:37.232 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:37.232 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.233 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.233 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "5000", 2024-03-18T15:28:37.233 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.233 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.233 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.233 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-18T15:28:37.233 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.233 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.233 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.233 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.233 INFO:teuthology.orchestra.run.smithi019.stdout: "database": { 2024-03-18T15:28:37.233 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "database", 2024-03-18T15:28:37.233 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.233 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.233 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.234 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "ceph", 2024-03-18T15:28:37.234 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.234 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.234 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.234 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-18T15:28:37.234 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.234 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.234 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.234 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.234 INFO:teuthology.orchestra.run.smithi019.stdout: "hostname": { 2024-03-18T15:28:37.234 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "hostname", 2024-03-18T15:28:37.234 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.234 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.235 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.235 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.235 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.235 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.235 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.235 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "InfluxDB server hostname", 2024-03-18T15:28:37.235 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.235 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.235 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.235 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.235 INFO:teuthology.orchestra.run.smithi019.stdout: "interval": { 2024-03-18T15:28:37.235 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "interval", 2024-03-18T15:28:37.235 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "secs", 2024-03-18T15:28:37.235 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.236 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.236 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "30", 2024-03-18T15:28:37.236 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "5", 2024-03-18T15:28:37.236 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.236 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.236 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-18T15:28:37.236 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.236 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.236 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.236 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.236 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:37.236 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:37.236 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.236 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.237 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.237 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.237 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.237 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.237 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.237 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.237 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.237 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.237 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.237 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.237 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.237 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.237 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.237 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.237 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.238 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.238 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.238 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:37.238 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:37.238 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.238 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.238 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.238 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.238 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.238 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.238 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.238 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.238 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.238 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.239 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.239 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.239 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:37.239 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:37.239 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.239 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.239 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.239 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:37.239 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.239 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.239 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.239 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.239 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.239 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.239 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.240 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.240 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.240 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.240 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.240 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.240 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.240 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.240 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.240 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:37.240 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:37.240 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.240 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.240 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.240 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.241 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.241 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.241 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.241 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.241 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.241 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.241 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.241 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.241 INFO:teuthology.orchestra.run.smithi019.stdout: "password": { 2024-03-18T15:28:37.241 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "password", 2024-03-18T15:28:37.241 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.241 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.241 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.241 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.241 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.242 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.242 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.242 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "password of InfluxDB server user", 2024-03-18T15:28:37.242 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.242 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.242 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.242 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.242 INFO:teuthology.orchestra.run.smithi019.stdout: "port": { 2024-03-18T15:28:37.242 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "port", 2024-03-18T15:28:37.242 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:37.242 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.242 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.242 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "8086", 2024-03-18T15:28:37.242 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.243 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.243 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.243 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "InfluxDB server port", 2024-03-18T15:28:37.243 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.243 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.243 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.243 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.243 INFO:teuthology.orchestra.run.smithi019.stdout: "ssl": { 2024-03-18T15:28:37.243 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "ssl", 2024-03-18T15:28:37.243 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.243 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.243 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.243 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "false", 2024-03-18T15:28:37.243 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.244 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.244 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.244 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-18T15:28:37.244 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.244 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.244 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.244 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.244 INFO:teuthology.orchestra.run.smithi019.stdout: "threads": { 2024-03-18T15:28:37.244 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "threads", 2024-03-18T15:28:37.244 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:37.244 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.244 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.244 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "5", 2024-03-18T15:28:37.244 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "1", 2024-03-18T15:28:37.244 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "32", 2024-03-18T15:28:37.245 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.245 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-18T15:28:37.245 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.245 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.245 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.245 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.245 INFO:teuthology.orchestra.run.smithi019.stdout: "username": { 2024-03-18T15:28:37.245 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "username", 2024-03-18T15:28:37.245 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.245 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.245 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.245 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.245 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.245 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.245 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.246 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "username of InfluxDB server user", 2024-03-18T15:28:37.246 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.246 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.246 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.246 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.246 INFO:teuthology.orchestra.run.smithi019.stdout: "verify_ssl": { 2024-03-18T15:28:37.246 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "verify_ssl", 2024-03-18T15:28:37.246 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.246 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.246 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.246 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "true", 2024-03-18T15:28:37.246 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.246 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.246 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.247 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-18T15:28:37.247 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.247 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.247 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.247 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.247 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.247 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.247 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:37.247 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "insights", 2024-03-18T15:28:37.247 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": true, 2024-03-18T15:28:37.247 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "", 2024-03-18T15:28:37.247 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:37.247 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:37.247 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:37.248 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.248 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.248 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.248 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.248 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.248 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.248 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.248 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.248 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.248 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.248 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.248 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.248 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.248 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.249 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.249 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.249 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.249 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.249 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.249 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:37.249 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:37.249 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.249 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.249 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.249 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.249 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.249 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.249 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.250 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.250 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.250 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.250 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.250 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.250 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:37.250 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:37.250 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.250 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.250 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.250 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:37.250 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.250 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.250 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.251 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.251 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.251 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.251 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.251 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.251 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.251 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.251 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.251 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.251 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.251 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.251 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.251 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:37.251 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:37.251 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.252 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.252 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.252 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.252 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.252 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.252 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.252 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.252 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.252 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.252 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.252 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.252 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.252 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.252 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:37.253 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "iostat", 2024-03-18T15:28:37.253 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": true, 2024-03-18T15:28:37.253 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "", 2024-03-18T15:28:37.253 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:37.253 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:37.253 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:37.253 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.253 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.253 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.253 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.253 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.253 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.253 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.253 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.254 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.254 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.254 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.254 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.254 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.254 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.254 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.254 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.254 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.254 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.254 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.254 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:37.254 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:37.254 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.255 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.255 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.255 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.255 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.255 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.255 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.255 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.255 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.255 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.255 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.255 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.255 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:37.255 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:37.255 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.255 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.256 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.256 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:37.256 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.256 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.256 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.256 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.256 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.256 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.256 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.256 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.256 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.256 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.256 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.256 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.256 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.257 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.257 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.257 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:37.257 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:37.257 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.257 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.257 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.257 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.257 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.257 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.257 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.257 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.257 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.257 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.258 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.258 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.258 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.258 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.258 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:37.258 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "k8sevents", 2024-03-18T15:28:37.258 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": true, 2024-03-18T15:28:37.258 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "", 2024-03-18T15:28:37.258 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:37.258 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph_event_retention_days": { 2024-03-18T15:28:37.258 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "ceph_event_retention_days", 2024-03-18T15:28:37.258 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:37.258 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.258 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.258 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "7", 2024-03-18T15:28:37.259 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.259 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.259 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.259 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-18T15:28:37.259 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.259 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.259 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.259 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.259 INFO:teuthology.orchestra.run.smithi019.stdout: "config_check_secs": { 2024-03-18T15:28:37.259 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "config_check_secs", 2024-03-18T15:28:37.259 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:37.259 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.259 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.259 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "10", 2024-03-18T15:28:37.259 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "10", 2024-03-18T15:28:37.260 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.260 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.260 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-18T15:28:37.260 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.260 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.260 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.260 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.260 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:37.260 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:37.260 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.260 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.260 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.260 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.260 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.261 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.261 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.261 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.261 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.261 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.261 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.261 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.261 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.261 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.261 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.261 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.261 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.261 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.261 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.262 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:37.262 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:37.262 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.262 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.262 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.262 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.262 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.262 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.265 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.265 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.265 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.266 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.266 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.266 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.266 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:37.266 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:37.266 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.266 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.266 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.266 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:37.266 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.266 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.266 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.266 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.266 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.267 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.267 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.267 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.267 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.267 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.267 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.267 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.267 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.267 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.267 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.267 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:37.267 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:37.267 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.267 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.268 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.268 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.268 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.268 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.268 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.268 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.268 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.268 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.268 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.268 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.268 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.268 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.268 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:37.268 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "localpool", 2024-03-18T15:28:37.269 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": true, 2024-03-18T15:28:37.269 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "", 2024-03-18T15:28:37.269 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:37.269 INFO:teuthology.orchestra.run.smithi019.stdout: "failure_domain": { 2024-03-18T15:28:37.269 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "failure_domain", 2024-03-18T15:28:37.269 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.269 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.269 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.269 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "host", 2024-03-18T15:28:37.269 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.269 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.269 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.269 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "failure domain for any created local pool", 2024-03-18T15:28:37.269 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-18T15:28:37.270 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.270 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.270 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.270 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:37.270 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:37.270 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.270 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.270 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.270 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.270 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.270 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.270 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.270 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.270 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.271 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.271 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.271 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.271 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.271 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.271 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.271 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.271 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.271 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.271 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.271 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:37.271 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:37.271 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.271 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.272 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.272 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.272 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.272 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.272 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.272 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.272 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.272 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.272 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.272 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.272 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:37.272 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:37.272 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.272 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.272 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.273 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:37.273 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.273 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.273 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.273 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.273 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.273 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.273 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.273 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.273 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.273 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.273 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.273 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.273 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.273 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.274 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.274 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:37.274 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:37.274 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.274 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.274 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.274 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.274 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.274 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.274 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.274 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.274 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.274 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.274 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.275 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.275 INFO:teuthology.orchestra.run.smithi019.stdout: "min_size": { 2024-03-18T15:28:37.275 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "min_size", 2024-03-18T15:28:37.275 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:37.275 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.275 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.275 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.275 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.275 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.275 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.275 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "default min_size for any created local pool", 2024-03-18T15:28:37.275 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-18T15:28:37.275 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.275 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.276 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.276 INFO:teuthology.orchestra.run.smithi019.stdout: "num_rep": { 2024-03-18T15:28:37.276 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "num_rep", 2024-03-18T15:28:37.276 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:37.276 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.276 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.276 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "3", 2024-03-18T15:28:37.276 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.276 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.276 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.276 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "default replica count for any created local pool", 2024-03-18T15:28:37.276 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.276 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.276 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.276 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.277 INFO:teuthology.orchestra.run.smithi019.stdout: "pg_num": { 2024-03-18T15:28:37.277 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "pg_num", 2024-03-18T15:28:37.277 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:37.277 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.277 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.277 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "128", 2024-03-18T15:28:37.277 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.277 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.277 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.277 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "default pg_num for any created local pool", 2024-03-18T15:28:37.277 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.277 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.278 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.278 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.278 INFO:teuthology.orchestra.run.smithi019.stdout: "prefix": { 2024-03-18T15:28:37.278 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "prefix", 2024-03-18T15:28:37.278 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.278 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.278 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.278 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.278 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.278 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.278 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.278 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "name prefix for any created local pool", 2024-03-18T15:28:37.279 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.279 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.279 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.279 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.279 INFO:teuthology.orchestra.run.smithi019.stdout: "subtree": { 2024-03-18T15:28:37.279 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "subtree", 2024-03-18T15:28:37.279 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.279 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.279 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.279 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "rack", 2024-03-18T15:28:37.279 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.279 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.279 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.279 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-18T15:28:37.280 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-18T15:28:37.280 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.280 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.280 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.280 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.280 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.280 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:37.280 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "mds_autoscaler", 2024-03-18T15:28:37.280 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": true, 2024-03-18T15:28:37.280 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "", 2024-03-18T15:28:37.280 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:37.280 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:37.280 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:37.280 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.280 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.281 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.281 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.281 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.281 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.281 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.281 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.281 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.281 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.281 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.281 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.281 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.281 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.281 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.281 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.282 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.282 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.282 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.282 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:37.282 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:37.282 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.282 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.282 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.282 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.282 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.282 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.282 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.282 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.282 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.283 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.283 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.283 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.283 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:37.283 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:37.283 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.283 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.283 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.283 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:37.283 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.283 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.283 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.283 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.283 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.284 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.284 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.284 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.284 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.284 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.284 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.284 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.284 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.284 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.284 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.284 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:37.284 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:37.284 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.285 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.285 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.285 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.285 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.285 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.285 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.285 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.285 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.285 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.285 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.285 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.285 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.285 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.285 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:37.285 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "mirroring", 2024-03-18T15:28:37.286 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": true, 2024-03-18T15:28:37.286 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "", 2024-03-18T15:28:37.286 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:37.286 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:37.286 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:37.286 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.286 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.286 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.286 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.286 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.286 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.286 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.286 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.286 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.286 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.287 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.287 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.287 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.287 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.287 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.287 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.287 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.287 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.287 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.287 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:37.287 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:37.287 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.287 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.287 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.288 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.288 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.288 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.288 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.288 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.288 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.288 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.288 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.288 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.288 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:37.288 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:37.288 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.288 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.288 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.288 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:37.289 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.289 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.289 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.289 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.289 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.289 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.289 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.289 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.289 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.289 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.289 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.289 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.289 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.289 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.290 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.290 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:37.290 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:37.290 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.290 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.290 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.290 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.290 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.290 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.290 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.290 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.290 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.290 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.290 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.291 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.291 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.291 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.291 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:37.291 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "nfs", 2024-03-18T15:28:37.291 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": true, 2024-03-18T15:28:37.291 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "", 2024-03-18T15:28:37.291 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:37.291 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:37.291 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:37.291 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.291 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.291 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.291 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.292 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.292 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.292 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.292 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.292 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.292 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.292 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.292 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.292 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.292 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.292 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.292 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.292 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.292 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.293 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.293 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:37.293 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:37.293 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.293 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.293 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.293 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.293 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.293 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.293 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.293 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.293 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.293 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.293 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.294 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.294 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:37.294 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:37.294 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.294 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.294 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.294 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:37.294 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.294 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.294 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.294 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.294 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.294 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.294 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.295 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.295 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.295 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.295 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.295 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.295 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.295 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.295 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.295 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:37.295 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:37.295 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.295 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.295 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.295 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.295 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.296 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.296 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.296 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.296 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.296 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.296 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.296 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.296 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.296 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.296 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:37.296 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "orchestrator", 2024-03-18T15:28:37.296 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": true, 2024-03-18T15:28:37.296 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "", 2024-03-18T15:28:37.296 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:37.297 INFO:teuthology.orchestra.run.smithi019.stdout: "fail_fs": { 2024-03-18T15:28:37.297 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "fail_fs", 2024-03-18T15:28:37.297 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.297 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.297 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.297 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.297 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.297 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.297 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.297 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-18T15:28:37.297 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.297 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.297 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.297 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.298 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:37.298 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:37.298 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.298 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.298 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.298 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.298 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.298 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.298 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.298 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.298 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.298 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.298 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.298 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.299 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.299 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.299 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.299 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.299 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.299 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.299 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.299 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:37.299 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:37.299 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.299 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.299 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.299 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.299 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.299 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.300 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.300 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.300 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.300 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.300 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.300 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.300 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:37.300 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:37.300 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.300 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.300 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.300 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:37.300 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.300 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.301 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.301 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.301 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.301 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.301 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.301 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.301 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.301 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.301 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.301 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.301 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.301 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.301 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.301 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:37.301 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:37.302 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.302 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.302 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.302 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.302 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.302 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.302 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.302 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.302 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.302 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.302 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.302 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.302 INFO:teuthology.orchestra.run.smithi019.stdout: "orchestrator": { 2024-03-18T15:28:37.302 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "orchestrator", 2024-03-18T15:28:37.303 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.303 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.303 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.303 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.303 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.303 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.303 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.303 INFO:teuthology.orchestra.run.smithi019.stdout: "cephadm", 2024-03-18T15:28:37.303 INFO:teuthology.orchestra.run.smithi019.stdout: "rook", 2024-03-18T15:28:37.303 INFO:teuthology.orchestra.run.smithi019.stdout: "test_orchestrator" 2024-03-18T15:28:37.303 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.303 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Orchestrator backend", 2024-03-18T15:28:37.303 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.303 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.304 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.304 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.304 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.304 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.304 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:37.304 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "osd_perf_query", 2024-03-18T15:28:37.304 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": true, 2024-03-18T15:28:37.304 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "", 2024-03-18T15:28:37.304 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:37.304 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:37.304 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:37.304 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.304 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.304 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.305 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.305 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.305 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.305 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.305 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.305 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.305 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.305 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.305 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.305 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.305 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.305 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.305 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.305 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.306 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.306 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.306 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:37.306 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:37.306 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.306 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.306 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.306 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.306 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.306 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.306 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.306 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.306 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.306 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.307 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.307 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.307 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:37.307 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:37.307 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.307 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.307 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.307 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:37.307 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.307 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.307 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.307 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.307 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.307 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.308 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.308 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.308 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.308 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.308 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.308 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.308 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.308 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.308 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.308 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:37.308 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:37.308 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.308 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.308 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.308 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.309 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.309 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.309 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.309 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.309 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.309 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.309 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.309 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.309 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.309 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.309 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:37.309 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "osd_support", 2024-03-18T15:28:37.309 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": true, 2024-03-18T15:28:37.309 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "", 2024-03-18T15:28:37.310 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:37.310 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:37.310 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:37.310 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.310 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.310 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.310 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.310 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.310 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.310 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.310 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.310 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.310 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.310 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.311 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.311 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.311 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.311 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.311 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.311 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.311 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.311 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.311 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:37.311 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:37.311 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.311 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.311 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.311 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.312 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.312 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.312 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.312 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.312 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.312 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.312 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.312 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.312 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:37.312 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:37.312 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.312 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.312 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.312 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:37.312 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.313 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.313 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.313 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.313 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.313 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.313 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.313 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.313 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.313 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.313 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.313 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.313 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.313 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.313 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.314 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:37.314 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:37.314 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.314 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.314 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.314 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.314 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.314 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.314 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.314 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.314 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.314 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.314 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.314 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.314 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.315 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.315 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:37.315 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "pg_autoscaler", 2024-03-18T15:28:37.315 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": true, 2024-03-18T15:28:37.315 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "", 2024-03-18T15:28:37.315 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:37.315 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:37.315 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:37.315 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.315 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.315 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.315 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.315 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.315 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.315 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.316 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.316 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.316 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.316 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.316 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.316 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.316 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.316 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.316 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.316 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.316 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.316 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.316 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:37.316 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:37.316 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.317 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.317 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.317 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.317 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.317 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.317 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.317 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.317 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.317 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.317 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.317 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.317 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:37.317 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:37.317 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.317 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.318 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.318 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:37.318 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.318 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.318 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.318 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.318 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.318 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.318 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.318 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.318 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.318 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.318 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.318 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.319 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.319 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.319 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.319 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:37.319 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:37.319 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.319 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.319 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.319 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.319 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.319 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.319 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.319 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.320 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.320 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.320 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.320 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.320 INFO:teuthology.orchestra.run.smithi019.stdout: "sleep_interval": { 2024-03-18T15:28:37.320 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "sleep_interval", 2024-03-18T15:28:37.320 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "secs", 2024-03-18T15:28:37.320 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.320 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.320 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "60", 2024-03-18T15:28:37.320 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.320 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.320 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.320 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.321 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.321 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.321 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.321 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.321 INFO:teuthology.orchestra.run.smithi019.stdout: "threshold": { 2024-03-18T15:28:37.321 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "threshold", 2024-03-18T15:28:37.321 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "float", 2024-03-18T15:28:37.321 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.321 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.321 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "3.0", 2024-03-18T15:28:37.321 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "1.0", 2024-03-18T15:28:37.321 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.321 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.321 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "scaling threshold", 2024-03-18T15:28:37.321 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-18T15:28:37.322 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.322 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.322 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.322 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.322 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.322 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:37.322 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "progress", 2024-03-18T15:28:37.322 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": true, 2024-03-18T15:28:37.322 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "", 2024-03-18T15:28:37.322 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:37.322 INFO:teuthology.orchestra.run.smithi019.stdout: "allow_pg_recovery_event": { 2024-03-18T15:28:37.322 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "allow_pg_recovery_event", 2024-03-18T15:28:37.322 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.323 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.323 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.323 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.323 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.323 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.323 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.323 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-18T15:28:37.323 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.323 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.323 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.323 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.323 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled": { 2024-03-18T15:28:37.323 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "enabled", 2024-03-18T15:28:37.323 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.324 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.324 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.324 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:37.324 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.324 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.324 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.324 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.324 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.324 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.324 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.324 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.324 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:37.324 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:37.324 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.325 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.325 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.325 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.325 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.325 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.325 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.325 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.325 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.325 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.325 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.325 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.325 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.325 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.325 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.326 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.326 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.326 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.326 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.326 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:37.326 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:37.326 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.326 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.326 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.326 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.326 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.326 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.326 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.326 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.326 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.327 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.327 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.327 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.327 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:37.327 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:37.327 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.327 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.327 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.327 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:37.327 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.327 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.327 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.327 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.327 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.328 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.328 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.328 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.328 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.328 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.328 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.328 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.328 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.328 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.328 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.328 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:37.328 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:37.328 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.328 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.328 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.329 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.329 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.329 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.329 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.329 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.329 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.329 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.329 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.329 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.329 INFO:teuthology.orchestra.run.smithi019.stdout: "max_completed_events": { 2024-03-18T15:28:37.329 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "max_completed_events", 2024-03-18T15:28:37.329 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:37.329 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.329 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.329 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "50", 2024-03-18T15:28:37.330 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.330 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.330 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.330 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "number of past completed events to remember", 2024-03-18T15:28:37.330 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.330 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.330 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.330 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.330 INFO:teuthology.orchestra.run.smithi019.stdout: "sleep_interval": { 2024-03-18T15:28:37.330 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "sleep_interval", 2024-03-18T15:28:37.330 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "secs", 2024-03-18T15:28:37.330 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.330 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.330 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "5", 2024-03-18T15:28:37.331 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.331 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.331 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.331 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "how long the module is going to sleep", 2024-03-18T15:28:37.331 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.331 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.331 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.331 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.331 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.331 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.331 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:37.331 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "prometheus", 2024-03-18T15:28:37.331 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": true, 2024-03-18T15:28:37.331 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "", 2024-03-18T15:28:37.331 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:37.332 INFO:teuthology.orchestra.run.smithi019.stdout: "cache": { 2024-03-18T15:28:37.332 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "cache", 2024-03-18T15:28:37.332 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.332 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.332 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.332 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:37.332 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.332 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.332 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.332 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.332 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.332 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.332 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.333 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.333 INFO:teuthology.orchestra.run.smithi019.stdout: "exclude_perf_counters": { 2024-03-18T15:28:37.333 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "exclude_perf_counters", 2024-03-18T15:28:37.333 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.333 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.333 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.333 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:37.333 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.333 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.333 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.333 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-18T15:28:37.333 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-18T15:28:37.333 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.333 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.334 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.334 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:37.334 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:37.334 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.334 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.334 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.334 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.334 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.334 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.334 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.334 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.334 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.334 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.335 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.335 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.335 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.335 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.335 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.335 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.335 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.335 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.335 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.335 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:37.335 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:37.335 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.335 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.335 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.335 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.336 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.336 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.336 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.336 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.336 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.336 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.336 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.336 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.336 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:37.336 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:37.336 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.336 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.336 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.336 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:37.336 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.337 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.337 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.337 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.337 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.337 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.337 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.337 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.337 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.337 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.337 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.337 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.337 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.337 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.337 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.338 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:37.338 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:37.338 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.338 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.338 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.338 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.338 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.338 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.338 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.338 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.338 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.338 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.338 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.338 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.339 INFO:teuthology.orchestra.run.smithi019.stdout: "rbd_stats_pools": { 2024-03-18T15:28:37.339 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "rbd_stats_pools", 2024-03-18T15:28:37.339 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.339 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.339 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.339 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.339 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.339 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.339 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.339 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.339 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.339 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.339 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.339 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.340 INFO:teuthology.orchestra.run.smithi019.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-18T15:28:37.340 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-18T15:28:37.340 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:37.340 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.340 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.340 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "300", 2024-03-18T15:28:37.340 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.340 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.340 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.340 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.340 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.340 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.340 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.340 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.340 INFO:teuthology.orchestra.run.smithi019.stdout: "scrape_interval": { 2024-03-18T15:28:37.341 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "scrape_interval", 2024-03-18T15:28:37.341 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "float", 2024-03-18T15:28:37.341 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.341 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.341 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "15.0", 2024-03-18T15:28:37.341 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.341 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.341 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.341 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.341 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.341 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.341 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.341 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.341 INFO:teuthology.orchestra.run.smithi019.stdout: "server_addr": { 2024-03-18T15:28:37.342 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "server_addr", 2024-03-18T15:28:37.342 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.342 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.342 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.342 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "::", 2024-03-18T15:28:37.342 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.342 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.342 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.342 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-18T15:28:37.342 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.342 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.342 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.342 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.342 INFO:teuthology.orchestra.run.smithi019.stdout: "server_port": { 2024-03-18T15:28:37.343 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "server_port", 2024-03-18T15:28:37.343 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:37.343 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.343 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.343 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "9283", 2024-03-18T15:28:37.343 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.343 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.343 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.343 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-18T15:28:37.343 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.343 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.343 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.343 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.343 INFO:teuthology.orchestra.run.smithi019.stdout: "stale_cache_strategy": { 2024-03-18T15:28:37.343 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "stale_cache_strategy", 2024-03-18T15:28:37.344 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.344 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.344 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.344 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "log", 2024-03-18T15:28:37.344 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.344 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.344 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.344 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.344 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.344 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.344 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.344 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.345 INFO:teuthology.orchestra.run.smithi019.stdout: "standby_behaviour": { 2024-03-18T15:28:37.345 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "standby_behaviour", 2024-03-18T15:28:37.345 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.345 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.345 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.345 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "default", 2024-03-18T15:28:37.345 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.345 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.345 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.345 INFO:teuthology.orchestra.run.smithi019.stdout: "default", 2024-03-18T15:28:37.345 INFO:teuthology.orchestra.run.smithi019.stdout: "error" 2024-03-18T15:28:37.345 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.345 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.345 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.346 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.346 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.346 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.346 INFO:teuthology.orchestra.run.smithi019.stdout: "standby_error_status_code": { 2024-03-18T15:28:37.346 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "standby_error_status_code", 2024-03-18T15:28:37.346 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:37.346 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.346 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.346 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "500", 2024-03-18T15:28:37.346 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "400", 2024-03-18T15:28:37.346 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "599", 2024-03-18T15:28:37.346 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.346 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.346 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.347 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.347 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.347 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.347 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.347 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.347 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:37.347 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "rbd_support", 2024-03-18T15:28:37.347 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": true, 2024-03-18T15:28:37.347 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "", 2024-03-18T15:28:37.347 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:37.347 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:37.347 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:37.347 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.347 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.348 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.348 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.348 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.348 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.348 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.348 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.348 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.348 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.348 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.348 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.348 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.348 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.348 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.348 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.348 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.349 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.349 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.349 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:37.349 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:37.349 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.349 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.349 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.349 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.349 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.349 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.349 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.349 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.349 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.350 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.350 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.350 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.350 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:37.350 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:37.350 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.350 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.350 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.350 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:37.350 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.350 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.350 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.350 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.350 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.351 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.351 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.351 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.351 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.351 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.351 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.351 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.351 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.351 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.351 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.351 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:37.351 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:37.351 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.351 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.351 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.352 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.352 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.352 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.352 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.352 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.352 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.352 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.352 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.352 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.352 INFO:teuthology.orchestra.run.smithi019.stdout: "max_concurrent_snap_create": { 2024-03-18T15:28:37.352 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "max_concurrent_snap_create", 2024-03-18T15:28:37.352 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:37.352 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.352 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.353 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "10", 2024-03-18T15:28:37.353 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.353 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.353 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.353 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.353 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.353 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.353 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.353 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.353 INFO:teuthology.orchestra.run.smithi019.stdout: "mirror_snapshot_schedule": { 2024-03-18T15:28:37.353 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "mirror_snapshot_schedule", 2024-03-18T15:28:37.353 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.353 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.353 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.353 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.354 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.354 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.354 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.354 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.354 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.354 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.354 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.354 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.354 INFO:teuthology.orchestra.run.smithi019.stdout: "trash_purge_schedule": { 2024-03-18T15:28:37.354 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "trash_purge_schedule", 2024-03-18T15:28:37.354 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.354 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.354 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.355 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.355 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.355 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.355 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.355 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.355 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.355 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.355 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.355 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.355 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.355 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.355 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:37.355 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "restful", 2024-03-18T15:28:37.356 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": true, 2024-03-18T15:28:37.356 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "", 2024-03-18T15:28:37.356 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:37.356 INFO:teuthology.orchestra.run.smithi019.stdout: "enable_auth": { 2024-03-18T15:28:37.356 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "enable_auth", 2024-03-18T15:28:37.356 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.356 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.356 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.356 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:37.356 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.356 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.356 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.356 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.356 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.356 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.357 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.357 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.357 INFO:teuthology.orchestra.run.smithi019.stdout: "key_file": { 2024-03-18T15:28:37.357 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "key_file", 2024-03-18T15:28:37.357 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.357 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.357 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.357 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.357 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.357 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.357 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.357 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.357 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.357 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.357 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.358 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.358 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:37.358 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:37.358 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.358 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.358 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.358 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.358 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.358 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.358 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.358 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.358 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.358 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.358 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.359 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.359 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.359 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.359 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.359 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.359 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.359 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.359 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.359 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:37.359 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:37.359 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.359 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.359 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.359 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.360 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.360 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.360 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.360 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.360 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.360 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.360 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.360 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.360 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:37.360 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:37.360 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.360 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.360 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.360 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:37.361 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.361 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.361 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.361 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.361 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.361 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.361 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.361 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.361 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.361 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.361 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.361 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.361 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.361 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.361 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.362 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:37.362 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:37.362 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.362 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.362 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.362 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.362 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.362 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.362 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.362 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.362 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.362 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.362 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.363 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.363 INFO:teuthology.orchestra.run.smithi019.stdout: "server_addr": { 2024-03-18T15:28:37.363 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "server_addr", 2024-03-18T15:28:37.363 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.363 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.363 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.363 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.363 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.363 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.363 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.363 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.363 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.363 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.363 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.364 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.364 INFO:teuthology.orchestra.run.smithi019.stdout: "server_port": { 2024-03-18T15:28:37.364 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "server_port", 2024-03-18T15:28:37.364 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.364 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.364 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.364 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.364 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.364 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.364 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.364 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.364 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.364 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.364 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.364 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.365 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.365 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.365 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:37.365 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "rgw", 2024-03-18T15:28:37.365 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": true, 2024-03-18T15:28:37.365 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "", 2024-03-18T15:28:37.365 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:37.365 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:37.365 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:37.365 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.365 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.365 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.365 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.365 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.366 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.366 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.366 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.366 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.366 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.366 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.366 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.366 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.366 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.366 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.366 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.366 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.366 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.366 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.367 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:37.367 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:37.367 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.367 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.367 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.367 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.367 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.367 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.367 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.367 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.367 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.367 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.367 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.367 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.368 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:37.368 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:37.368 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.368 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.368 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.368 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:37.368 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.368 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.368 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.368 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.368 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.368 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.368 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.368 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.368 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.369 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.369 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.369 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.369 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.369 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.369 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.369 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:37.369 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:37.369 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.369 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.369 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.369 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.369 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.369 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.369 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.370 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.370 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.370 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.370 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.370 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.370 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.370 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.370 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:37.370 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "rook", 2024-03-18T15:28:37.370 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": true, 2024-03-18T15:28:37.370 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "", 2024-03-18T15:28:37.370 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:37.370 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:37.370 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:37.371 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.371 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.371 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.371 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.371 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.371 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.371 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.371 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.371 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.371 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.371 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.371 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.371 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.371 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.371 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.372 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.372 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.372 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.372 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.372 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:37.372 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:37.372 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.372 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.372 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.372 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.372 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.372 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.372 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.372 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.373 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.373 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.373 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.373 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.373 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:37.373 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:37.373 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.373 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.373 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.373 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:37.373 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.373 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.373 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.374 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.374 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.374 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.374 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.374 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.374 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.374 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.374 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.374 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.374 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.374 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.374 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.374 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:37.374 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:37.375 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.375 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.375 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.375 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.375 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.375 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.375 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.375 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.375 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.375 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.375 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.375 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.375 INFO:teuthology.orchestra.run.smithi019.stdout: "storage_class": { 2024-03-18T15:28:37.375 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "storage_class", 2024-03-18T15:28:37.375 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.376 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.376 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.376 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "local", 2024-03-18T15:28:37.376 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.376 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.376 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.376 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-18T15:28:37.376 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.376 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.376 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.376 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.376 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.376 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.376 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:37.377 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "selftest", 2024-03-18T15:28:37.377 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": true, 2024-03-18T15:28:37.377 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "", 2024-03-18T15:28:37.377 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:37.377 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:37.377 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:37.377 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.377 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.377 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.377 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.377 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.377 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.377 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.377 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.377 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.378 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.378 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.378 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.378 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.378 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.378 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.378 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.378 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.378 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.378 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.378 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:37.378 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:37.378 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.378 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.379 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.379 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.379 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.379 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.379 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.379 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.379 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.379 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.379 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.379 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.379 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:37.379 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:37.379 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.379 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.380 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.380 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:37.380 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.380 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.380 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.380 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.380 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.380 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.380 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.380 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.380 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.380 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.380 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.380 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.381 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.381 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.381 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.381 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:37.381 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:37.381 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.381 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.381 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.381 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.381 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.381 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.381 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.381 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.381 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.381 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.382 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.382 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.382 INFO:teuthology.orchestra.run.smithi019.stdout: "roption1": { 2024-03-18T15:28:37.382 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "roption1", 2024-03-18T15:28:37.382 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.382 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.382 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.382 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.382 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.382 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.382 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.382 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.382 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.382 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.383 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.383 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.383 INFO:teuthology.orchestra.run.smithi019.stdout: "roption2": { 2024-03-18T15:28:37.383 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "roption2", 2024-03-18T15:28:37.383 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.383 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.383 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.383 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "xyz", 2024-03-18T15:28:37.383 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.383 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.383 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.383 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.383 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.383 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.384 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.384 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.384 INFO:teuthology.orchestra.run.smithi019.stdout: "rwoption1": { 2024-03-18T15:28:37.384 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "rwoption1", 2024-03-18T15:28:37.384 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.384 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.384 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.384 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.384 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.384 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.384 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.384 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.384 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.384 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.384 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.385 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.385 INFO:teuthology.orchestra.run.smithi019.stdout: "rwoption2": { 2024-03-18T15:28:37.385 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "rwoption2", 2024-03-18T15:28:37.385 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:37.385 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.385 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.385 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.385 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.385 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.385 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.385 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.385 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.385 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.385 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.386 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.386 INFO:teuthology.orchestra.run.smithi019.stdout: "rwoption3": { 2024-03-18T15:28:37.386 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "rwoption3", 2024-03-18T15:28:37.386 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "float", 2024-03-18T15:28:37.386 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.386 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.386 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.386 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.386 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.386 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.386 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.386 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.386 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.387 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.387 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.387 INFO:teuthology.orchestra.run.smithi019.stdout: "rwoption4": { 2024-03-18T15:28:37.387 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "rwoption4", 2024-03-18T15:28:37.387 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.387 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.387 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.387 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.387 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.387 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.387 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.387 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.387 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.387 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.388 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.388 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.388 INFO:teuthology.orchestra.run.smithi019.stdout: "rwoption5": { 2024-03-18T15:28:37.388 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "rwoption5", 2024-03-18T15:28:37.388 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.388 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.388 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.388 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.388 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.388 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.388 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.388 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.388 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.388 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.389 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.389 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.389 INFO:teuthology.orchestra.run.smithi019.stdout: "rwoption6": { 2024-03-18T15:28:37.389 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "rwoption6", 2024-03-18T15:28:37.389 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.389 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.389 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.389 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:37.389 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.389 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.389 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.389 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.389 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.389 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.389 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.390 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.390 INFO:teuthology.orchestra.run.smithi019.stdout: "rwoption7": { 2024-03-18T15:28:37.390 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "rwoption7", 2024-03-18T15:28:37.390 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:37.390 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.390 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.390 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.390 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "1", 2024-03-18T15:28:37.390 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "42", 2024-03-18T15:28:37.390 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.390 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.390 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.390 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.390 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.391 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.391 INFO:teuthology.orchestra.run.smithi019.stdout: "testkey": { 2024-03-18T15:28:37.391 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "testkey", 2024-03-18T15:28:37.391 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.391 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.391 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.391 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.391 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.391 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.391 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.391 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.391 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.391 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.391 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.392 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.392 INFO:teuthology.orchestra.run.smithi019.stdout: "testlkey": { 2024-03-18T15:28:37.392 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "testlkey", 2024-03-18T15:28:37.392 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.392 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.392 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.392 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.392 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.392 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.392 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.392 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.392 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.392 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.392 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.392 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.393 INFO:teuthology.orchestra.run.smithi019.stdout: "testnewline": { 2024-03-18T15:28:37.393 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "testnewline", 2024-03-18T15:28:37.393 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.393 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.393 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.393 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.393 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.393 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.393 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.393 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.393 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.393 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.393 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.393 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.394 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.394 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.394 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:37.394 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "snap_schedule", 2024-03-18T15:28:37.394 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": true, 2024-03-18T15:28:37.394 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "", 2024-03-18T15:28:37.394 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:37.394 INFO:teuthology.orchestra.run.smithi019.stdout: "allow_m_granularity": { 2024-03-18T15:28:37.394 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "allow_m_granularity", 2024-03-18T15:28:37.394 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.394 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.394 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.394 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.394 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.395 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.395 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.395 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "allow minute scheduled snapshots", 2024-03-18T15:28:37.395 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.395 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.395 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.395 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.395 INFO:teuthology.orchestra.run.smithi019.stdout: "dump_on_update": { 2024-03-18T15:28:37.395 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "dump_on_update", 2024-03-18T15:28:37.395 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.395 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.395 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.395 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.395 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.396 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.396 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.396 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "dump database to debug log on update", 2024-03-18T15:28:37.396 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.396 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.396 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.396 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.396 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:37.396 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:37.396 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.396 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.396 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.396 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.396 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.397 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.397 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.397 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.397 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.397 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.397 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.397 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.397 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.397 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.397 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.397 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.397 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.397 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.397 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.397 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:37.398 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:37.398 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.398 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.398 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.398 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.398 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.398 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.398 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.398 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.398 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.398 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.398 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.398 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.398 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:37.399 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:37.399 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.399 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.399 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.399 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:37.399 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.399 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.399 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.399 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.399 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.399 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.399 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.399 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.399 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.399 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.400 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.400 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.400 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.400 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.400 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.400 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:37.400 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:37.400 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.400 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.400 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.400 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.400 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.400 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.401 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.401 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.401 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.401 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.401 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.401 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.401 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.401 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.401 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:37.401 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "stats", 2024-03-18T15:28:37.401 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": true, 2024-03-18T15:28:37.401 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "", 2024-03-18T15:28:37.401 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:37.401 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:37.402 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:37.402 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.402 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.402 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.402 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.402 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.402 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.402 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.402 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.402 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.402 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.402 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.402 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.402 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.403 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.403 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.403 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.403 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.403 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.403 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.403 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:37.403 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:37.403 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.403 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.403 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.403 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.403 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.403 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.404 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.404 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.404 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.404 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.404 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.404 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.404 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:37.404 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:37.404 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.404 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.404 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.404 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:37.404 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.404 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.405 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.405 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.405 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.405 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.405 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.405 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.405 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.405 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.405 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.405 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.405 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.405 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.405 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.405 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:37.406 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:37.406 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.406 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.406 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.406 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.406 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.406 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.406 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.406 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.406 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.406 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.406 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.407 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.407 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.407 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.407 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:37.407 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "status", 2024-03-18T15:28:37.407 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": true, 2024-03-18T15:28:37.407 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "", 2024-03-18T15:28:37.407 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:37.407 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:37.407 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:37.407 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.407 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.407 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.407 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.407 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.408 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.408 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.408 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.408 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.408 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.408 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.408 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.408 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.408 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.408 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.408 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.408 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.408 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.408 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.409 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:37.409 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:37.409 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.409 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.409 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.409 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.409 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.409 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.409 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.409 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.409 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.409 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.409 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.409 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.409 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:37.410 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:37.410 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.410 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.410 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.410 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:37.410 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.410 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.410 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.410 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.410 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.410 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.410 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.410 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.410 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.411 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.411 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.411 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.411 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.411 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.411 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.411 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:37.411 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:37.411 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.411 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.411 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.411 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.411 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.411 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.411 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.412 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.412 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.412 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.412 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.412 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.412 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.412 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.412 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:37.412 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "telegraf", 2024-03-18T15:28:37.412 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": true, 2024-03-18T15:28:37.412 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "", 2024-03-18T15:28:37.412 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:37.412 INFO:teuthology.orchestra.run.smithi019.stdout: "address": { 2024-03-18T15:28:37.412 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "address", 2024-03-18T15:28:37.413 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.413 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.413 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.413 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-18T15:28:37.413 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.413 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.413 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.413 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.413 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.413 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.413 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.413 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.413 INFO:teuthology.orchestra.run.smithi019.stdout: "interval": { 2024-03-18T15:28:37.414 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "interval", 2024-03-18T15:28:37.414 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "secs", 2024-03-18T15:28:37.414 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.414 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.414 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "15", 2024-03-18T15:28:37.414 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.414 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.414 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.414 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.414 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.414 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.414 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.414 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.415 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:37.415 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:37.415 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.415 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.415 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.415 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.415 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.415 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.415 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.415 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.415 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.415 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.415 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.415 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.415 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.416 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.416 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.416 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.416 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.416 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.416 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.416 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:37.416 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:37.416 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.416 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.416 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.416 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.416 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.416 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.417 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.417 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.417 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.417 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.417 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.417 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.417 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:37.417 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:37.417 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.417 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.417 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.417 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:37.417 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.417 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.418 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.418 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.418 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.418 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.418 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.418 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.418 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.418 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.418 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.418 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.418 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.418 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.418 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.418 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:37.419 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:37.419 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.419 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.419 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.419 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.419 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.419 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.419 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.419 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.419 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.419 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.419 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.419 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.419 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.419 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.420 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:37.420 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "telemetry", 2024-03-18T15:28:37.420 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": true, 2024-03-18T15:28:37.420 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "", 2024-03-18T15:28:37.420 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:37.420 INFO:teuthology.orchestra.run.smithi019.stdout: "channel_basic": { 2024-03-18T15:28:37.420 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "channel_basic", 2024-03-18T15:28:37.420 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.420 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.420 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.420 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:37.420 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.420 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.420 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.421 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-18T15:28:37.421 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.421 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.421 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.421 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.421 INFO:teuthology.orchestra.run.smithi019.stdout: "channel_crash": { 2024-03-18T15:28:37.421 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "channel_crash", 2024-03-18T15:28:37.421 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.421 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.421 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.421 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:37.421 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.421 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.421 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.422 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-18T15:28:37.422 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.422 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.422 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.422 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.422 INFO:teuthology.orchestra.run.smithi019.stdout: "channel_device": { 2024-03-18T15:28:37.422 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "channel_device", 2024-03-18T15:28:37.422 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.422 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.422 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.422 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:37.422 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.422 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.422 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.423 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-18T15:28:37.423 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.423 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.423 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.423 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.423 INFO:teuthology.orchestra.run.smithi019.stdout: "channel_ident": { 2024-03-18T15:28:37.423 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "channel_ident", 2024-03-18T15:28:37.423 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.423 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.423 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.423 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.423 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.423 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.423 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.424 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-18T15:28:37.424 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.424 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.424 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.424 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.424 INFO:teuthology.orchestra.run.smithi019.stdout: "channel_perf": { 2024-03-18T15:28:37.424 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "channel_perf", 2024-03-18T15:28:37.424 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.424 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.424 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.424 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.424 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.424 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.424 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.424 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-18T15:28:37.425 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.425 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.425 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.425 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.425 INFO:teuthology.orchestra.run.smithi019.stdout: "contact": { 2024-03-18T15:28:37.425 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "contact", 2024-03-18T15:28:37.425 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.425 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.425 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.425 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.425 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.425 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.425 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.425 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.425 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.426 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.426 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.426 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.426 INFO:teuthology.orchestra.run.smithi019.stdout: "description": { 2024-03-18T15:28:37.426 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "description", 2024-03-18T15:28:37.426 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.426 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.426 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.426 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.426 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.426 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.426 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.426 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.426 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.427 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.427 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.427 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.427 INFO:teuthology.orchestra.run.smithi019.stdout: "device_url": { 2024-03-18T15:28:37.427 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "device_url", 2024-03-18T15:28:37.427 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.427 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.427 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.427 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-18T15:28:37.427 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.427 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.427 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.427 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.427 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.428 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.428 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.428 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.428 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled": { 2024-03-18T15:28:37.428 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "enabled", 2024-03-18T15:28:37.428 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.428 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.428 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.428 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.428 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.428 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.428 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.428 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.428 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.429 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.429 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.429 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.429 INFO:teuthology.orchestra.run.smithi019.stdout: "interval": { 2024-03-18T15:28:37.429 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "interval", 2024-03-18T15:28:37.429 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:37.429 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.429 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.429 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "24", 2024-03-18T15:28:37.429 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "8", 2024-03-18T15:28:37.429 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.429 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.429 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.429 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.430 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.430 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.430 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.430 INFO:teuthology.orchestra.run.smithi019.stdout: "last_opt_revision": { 2024-03-18T15:28:37.430 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "last_opt_revision", 2024-03-18T15:28:37.430 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:37.430 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.430 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.430 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "1", 2024-03-18T15:28:37.430 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.430 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.430 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.430 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.430 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.431 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.431 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.431 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.431 INFO:teuthology.orchestra.run.smithi019.stdout: "leaderboard": { 2024-03-18T15:28:37.431 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "leaderboard", 2024-03-18T15:28:37.431 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.431 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.431 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.431 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.431 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.431 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.431 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.431 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.431 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.431 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.432 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.432 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.432 INFO:teuthology.orchestra.run.smithi019.stdout: "leaderboard_description": { 2024-03-18T15:28:37.432 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "leaderboard_description", 2024-03-18T15:28:37.432 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.432 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.432 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.432 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.432 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.432 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.432 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.432 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.432 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.432 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.433 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.433 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.433 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:37.433 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:37.433 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.433 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.433 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.433 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.433 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.433 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.433 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.433 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.433 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.433 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.434 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.434 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.434 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.434 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.434 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.434 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.434 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.434 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.434 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.434 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:37.434 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:37.434 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.434 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.434 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.434 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.435 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.435 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.435 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.435 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.435 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.435 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.435 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.435 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.435 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:37.435 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:37.435 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.435 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.435 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.435 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:37.436 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.436 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.436 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.436 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.436 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.436 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.436 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.436 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.436 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.436 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.436 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.436 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.436 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.436 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.436 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.437 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:37.437 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:37.437 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.437 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.437 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.437 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.437 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.437 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.437 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.437 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.437 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.437 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.437 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.437 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.437 INFO:teuthology.orchestra.run.smithi019.stdout: "organization": { 2024-03-18T15:28:37.438 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "organization", 2024-03-18T15:28:37.438 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.438 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.438 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.438 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.438 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.438 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.438 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.438 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.438 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.438 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.438 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.438 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.438 INFO:teuthology.orchestra.run.smithi019.stdout: "proxy": { 2024-03-18T15:28:37.439 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "proxy", 2024-03-18T15:28:37.439 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.439 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.439 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.439 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.439 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.439 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.439 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.439 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.439 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.439 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.439 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.439 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.439 INFO:teuthology.orchestra.run.smithi019.stdout: "url": { 2024-03-18T15:28:37.439 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "url", 2024-03-18T15:28:37.440 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.440 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.440 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.440 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-18T15:28:37.440 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.440 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.440 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.440 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.440 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.440 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.440 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.440 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.440 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.440 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.440 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:37.441 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "test_orchestrator", 2024-03-18T15:28:37.441 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": true, 2024-03-18T15:28:37.441 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "", 2024-03-18T15:28:37.441 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:37.441 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:37.441 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:37.441 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.441 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.441 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.441 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.441 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.441 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.441 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.442 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.442 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.442 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.442 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.442 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.442 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.442 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.442 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.442 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.442 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.442 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.442 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.442 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:37.442 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:37.443 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.443 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.443 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.443 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.443 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.443 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.443 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.443 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.443 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.443 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.443 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.443 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.443 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:37.443 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:37.444 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.444 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.444 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.444 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:37.444 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.444 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.444 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.444 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.444 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.444 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.444 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.444 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.444 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.444 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.445 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.445 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.445 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.445 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.445 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.445 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:37.445 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:37.445 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.445 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.445 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.445 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.445 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.445 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.445 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.446 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.446 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.446 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.446 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.446 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.446 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.446 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.446 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:37.446 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "volumes", 2024-03-18T15:28:37.446 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": true, 2024-03-18T15:28:37.446 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "", 2024-03-18T15:28:37.446 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:37.446 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:37.446 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:37.447 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.447 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.447 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.447 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.447 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.447 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.447 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.447 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.447 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.447 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.447 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.447 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.447 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.447 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.448 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.448 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.448 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.448 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.448 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.448 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:37.448 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:37.448 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.448 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.448 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.448 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.448 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.448 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.448 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.448 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.449 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.449 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.449 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.449 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.449 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:37.449 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:37.449 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.449 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.449 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.449 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:37.449 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.449 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.449 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.449 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.450 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.450 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.450 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.450 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.450 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.450 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.450 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.450 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.450 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.450 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.450 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.450 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:37.450 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:37.450 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.450 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.451 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.451 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.451 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.451 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.451 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.451 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.451 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.451 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.451 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.451 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.451 INFO:teuthology.orchestra.run.smithi019.stdout: "max_concurrent_clones": { 2024-03-18T15:28:37.451 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "max_concurrent_clones", 2024-03-18T15:28:37.451 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:37.452 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.452 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.452 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "4", 2024-03-18T15:28:37.452 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.452 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.452 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.452 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-18T15:28:37.452 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.452 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.452 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.452 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.452 INFO:teuthology.orchestra.run.smithi019.stdout: "periodic_async_work": { 2024-03-18T15:28:37.452 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "periodic_async_work", 2024-03-18T15:28:37.452 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.452 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.453 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.453 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.453 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.453 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.453 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.453 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Periodically check for async work", 2024-03-18T15:28:37.453 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.453 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.453 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.453 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.453 INFO:teuthology.orchestra.run.smithi019.stdout: "snapshot_clone_delay": { 2024-03-18T15:28:37.453 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "snapshot_clone_delay", 2024-03-18T15:28:37.453 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:37.453 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.454 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.454 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "0", 2024-03-18T15:28:37.454 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.454 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.454 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.454 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-18T15:28:37.454 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.454 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.454 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.454 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.454 INFO:teuthology.orchestra.run.smithi019.stdout: "snapshot_clone_no_wait": { 2024-03-18T15:28:37.454 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "snapshot_clone_no_wait", 2024-03-18T15:28:37.454 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.455 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.455 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.455 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:37.455 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.455 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.455 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.455 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-18T15:28:37.455 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.455 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.455 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.455 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.455 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.455 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.455 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:37.456 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "zabbix", 2024-03-18T15:28:37.456 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": true, 2024-03-18T15:28:37.456 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "", 2024-03-18T15:28:37.456 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:37.456 INFO:teuthology.orchestra.run.smithi019.stdout: "discovery_interval": { 2024-03-18T15:28:37.456 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "discovery_interval", 2024-03-18T15:28:37.456 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "uint", 2024-03-18T15:28:37.456 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.456 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.456 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "100", 2024-03-18T15:28:37.456 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.456 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.456 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.456 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.456 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.457 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.457 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.457 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.457 INFO:teuthology.orchestra.run.smithi019.stdout: "identifier": { 2024-03-18T15:28:37.457 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "identifier", 2024-03-18T15:28:37.457 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.457 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.457 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.457 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.457 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.457 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.457 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.457 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.457 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.458 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.458 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.458 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.458 INFO:teuthology.orchestra.run.smithi019.stdout: "interval": { 2024-03-18T15:28:37.458 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "interval", 2024-03-18T15:28:37.458 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "secs", 2024-03-18T15:28:37.458 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.458 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.458 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "60", 2024-03-18T15:28:37.458 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.458 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.458 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.458 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.458 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.459 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.459 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.459 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.459 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:37.459 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:37.459 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.459 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.459 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.459 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.459 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.459 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.459 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.459 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.459 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.460 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.460 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.460 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.460 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.460 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.460 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.460 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.460 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.460 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.460 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.460 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:37.460 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:37.460 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.460 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.460 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.461 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.461 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.461 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.461 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.461 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.461 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.461 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.461 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.461 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.461 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:37.461 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:37.461 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.461 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.461 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.461 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:37.462 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.462 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.462 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.462 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.462 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.462 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.462 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.462 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.462 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.462 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.462 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.462 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.462 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.463 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.463 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.463 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:37.463 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:37.463 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.463 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.463 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.463 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.463 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.463 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.463 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.463 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.463 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.463 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.464 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.464 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.464 INFO:teuthology.orchestra.run.smithi019.stdout: "zabbix_host": { 2024-03-18T15:28:37.464 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "zabbix_host", 2024-03-18T15:28:37.464 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.464 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.464 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.464 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.464 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.464 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.464 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.464 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.464 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.464 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.465 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.465 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.465 INFO:teuthology.orchestra.run.smithi019.stdout: "zabbix_port": { 2024-03-18T15:28:37.465 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "zabbix_port", 2024-03-18T15:28:37.465 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:37.465 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.465 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.465 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "10051", 2024-03-18T15:28:37.465 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.465 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.465 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.465 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.465 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.465 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.466 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.466 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.466 INFO:teuthology.orchestra.run.smithi019.stdout: "zabbix_sender": { 2024-03-18T15:28:37.466 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "zabbix_sender", 2024-03-18T15:28:37.466 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.466 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.466 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.466 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-18T15:28:37.466 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.466 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.466 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.466 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.466 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.466 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.467 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.467 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.467 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.467 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.467 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-03-18T15:28:37.467 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.467 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.467 INFO:teuthology.orchestra.run.smithi019.stdout: "modules": [ 2024-03-18T15:28:37.467 INFO:teuthology.orchestra.run.smithi019.stdout: "cephadm", 2024-03-18T15:28:37.467 INFO:teuthology.orchestra.run.smithi019.stdout: "dashboard", 2024-03-18T15:28:37.467 INFO:teuthology.orchestra.run.smithi019.stdout: "iostat", 2024-03-18T15:28:37.467 INFO:teuthology.orchestra.run.smithi019.stdout: "nfs", 2024-03-18T15:28:37.467 INFO:teuthology.orchestra.run.smithi019.stdout: "restful", 2024-03-18T15:28:37.467 INFO:teuthology.orchestra.run.smithi019.stdout: "snap_schedule" 2024-03-18T15:28:37.467 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.468 INFO:teuthology.orchestra.run.smithi019.stdout: "available_modules": [ 2024-03-18T15:28:37.468 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:37.468 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "alerts", 2024-03-18T15:28:37.468 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": true, 2024-03-18T15:28:37.468 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "", 2024-03-18T15:28:37.468 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:37.468 INFO:teuthology.orchestra.run.smithi019.stdout: "interval": { 2024-03-18T15:28:37.468 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "interval", 2024-03-18T15:28:37.468 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "secs", 2024-03-18T15:28:37.468 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.468 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.468 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "60", 2024-03-18T15:28:37.468 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.468 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.468 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.469 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "How frequently to reexamine health status", 2024-03-18T15:28:37.469 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.469 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.469 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.469 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.469 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:37.469 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:37.469 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.469 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.469 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.469 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.469 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.470 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.470 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.470 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.470 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.470 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.470 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.470 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.470 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.470 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.470 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.470 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.470 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.470 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.470 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.471 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:37.471 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:37.471 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.471 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.471 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.471 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.471 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.471 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.471 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.471 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.471 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.471 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.471 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.471 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.472 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:37.472 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:37.472 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.472 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.472 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.472 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:37.472 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.472 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.472 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.472 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.472 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.472 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.472 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.472 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.473 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.473 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.473 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.473 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.473 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.473 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.473 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.473 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:37.473 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:37.473 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.473 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.473 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.473 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.473 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.473 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.474 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.474 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.474 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.474 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.474 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.474 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.474 INFO:teuthology.orchestra.run.smithi019.stdout: "smtp_destination": { 2024-03-18T15:28:37.474 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "smtp_destination", 2024-03-18T15:28:37.474 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.474 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.474 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.474 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.474 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.474 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.475 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.475 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Email address to send alerts to", 2024-03-18T15:28:37.475 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.475 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.475 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.475 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.475 INFO:teuthology.orchestra.run.smithi019.stdout: "smtp_from_name": { 2024-03-18T15:28:37.475 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "smtp_from_name", 2024-03-18T15:28:37.475 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.475 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.475 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.475 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "Ceph", 2024-03-18T15:28:37.475 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.475 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.476 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.476 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Email From: name", 2024-03-18T15:28:37.476 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.476 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.476 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.476 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.476 INFO:teuthology.orchestra.run.smithi019.stdout: "smtp_host": { 2024-03-18T15:28:37.476 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "smtp_host", 2024-03-18T15:28:37.476 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.476 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.476 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.476 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.476 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.476 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.477 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.477 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "SMTP server", 2024-03-18T15:28:37.477 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.477 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.477 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.477 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.477 INFO:teuthology.orchestra.run.smithi019.stdout: "smtp_password": { 2024-03-18T15:28:37.477 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "smtp_password", 2024-03-18T15:28:37.477 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.477 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.477 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.477 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.477 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.477 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.477 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.478 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Password to authenticate with", 2024-03-18T15:28:37.478 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.478 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.478 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.478 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.478 INFO:teuthology.orchestra.run.smithi019.stdout: "smtp_port": { 2024-03-18T15:28:37.478 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "smtp_port", 2024-03-18T15:28:37.478 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:37.478 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.478 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.478 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "465", 2024-03-18T15:28:37.478 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.478 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.479 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.479 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "SMTP port", 2024-03-18T15:28:37.479 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.479 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.479 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.479 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.479 INFO:teuthology.orchestra.run.smithi019.stdout: "smtp_sender": { 2024-03-18T15:28:37.479 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "smtp_sender", 2024-03-18T15:28:37.479 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.479 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.479 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.479 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.479 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.479 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.479 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.480 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "SMTP envelope sender", 2024-03-18T15:28:37.480 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.480 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.480 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.480 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.480 INFO:teuthology.orchestra.run.smithi019.stdout: "smtp_ssl": { 2024-03-18T15:28:37.480 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "smtp_ssl", 2024-03-18T15:28:37.480 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.480 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.480 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.480 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:37.480 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.480 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.480 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.480 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-18T15:28:37.481 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.481 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.481 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.481 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.481 INFO:teuthology.orchestra.run.smithi019.stdout: "smtp_user": { 2024-03-18T15:28:37.481 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "smtp_user", 2024-03-18T15:28:37.481 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.481 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.481 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.481 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.481 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.481 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.481 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.481 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "User to authenticate as", 2024-03-18T15:28:37.482 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.482 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.482 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.482 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.482 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.482 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.482 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:37.482 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "balancer", 2024-03-18T15:28:37.482 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": true, 2024-03-18T15:28:37.482 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "", 2024-03-18T15:28:37.482 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:37.482 INFO:teuthology.orchestra.run.smithi019.stdout: "active": { 2024-03-18T15:28:37.482 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "active", 2024-03-18T15:28:37.483 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.483 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.483 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.483 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:37.483 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.483 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.483 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.483 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "automatically balance PGs across cluster", 2024-03-18T15:28:37.483 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.483 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.484 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.484 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.484 INFO:teuthology.orchestra.run.smithi019.stdout: "begin_time": { 2024-03-18T15:28:37.484 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "begin_time", 2024-03-18T15:28:37.484 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.484 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.484 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.484 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "0000", 2024-03-18T15:28:37.484 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.484 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.484 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.484 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "beginning time of day to automatically balance", 2024-03-18T15:28:37.485 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-18T15:28:37.485 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.485 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.485 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.485 INFO:teuthology.orchestra.run.smithi019.stdout: "begin_weekday": { 2024-03-18T15:28:37.485 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "begin_weekday", 2024-03-18T15:28:37.485 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "uint", 2024-03-18T15:28:37.485 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.485 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.485 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "0", 2024-03-18T15:28:37.485 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "0", 2024-03-18T15:28:37.485 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "6", 2024-03-18T15:28:37.486 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.486 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-18T15:28:37.486 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-18T15:28:37.486 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.486 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.486 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.486 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_compat_max_iterations": { 2024-03-18T15:28:37.486 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "crush_compat_max_iterations", 2024-03-18T15:28:37.486 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "uint", 2024-03-18T15:28:37.486 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.486 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.487 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "25", 2024-03-18T15:28:37.487 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "1", 2024-03-18T15:28:37.487 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "250", 2024-03-18T15:28:37.487 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.487 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-18T15:28:37.487 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.487 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.487 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.487 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.487 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_compat_metrics": { 2024-03-18T15:28:37.487 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "crush_compat_metrics", 2024-03-18T15:28:37.487 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.488 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.488 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.488 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "pgs,objects,bytes", 2024-03-18T15:28:37.488 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.488 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.488 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.488 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-18T15:28:37.488 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-18T15:28:37.488 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.488 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.488 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.488 INFO:teuthology.orchestra.run.smithi019.stdout: "crush_compat_step": { 2024-03-18T15:28:37.489 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "crush_compat_step", 2024-03-18T15:28:37.489 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "float", 2024-03-18T15:28:37.489 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.489 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.489 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "0.5", 2024-03-18T15:28:37.489 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "0.001", 2024-03-18T15:28:37.489 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "0.999", 2024-03-18T15:28:37.489 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.489 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "aggressiveness of optimization", 2024-03-18T15:28:37.489 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-18T15:28:37.489 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.489 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.490 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.490 INFO:teuthology.orchestra.run.smithi019.stdout: "end_time": { 2024-03-18T15:28:37.490 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "end_time", 2024-03-18T15:28:37.490 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.490 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.490 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.490 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "2359", 2024-03-18T15:28:37.490 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.490 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.490 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.490 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "ending time of day to automatically balance", 2024-03-18T15:28:37.490 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-18T15:28:37.491 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.491 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.491 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.491 INFO:teuthology.orchestra.run.smithi019.stdout: "end_weekday": { 2024-03-18T15:28:37.491 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "end_weekday", 2024-03-18T15:28:37.491 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "uint", 2024-03-18T15:28:37.491 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.491 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.491 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "0", 2024-03-18T15:28:37.491 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "0", 2024-03-18T15:28:37.491 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "6", 2024-03-18T15:28:37.491 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.491 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-18T15:28:37.492 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-18T15:28:37.492 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.492 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.492 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.492 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:37.492 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:37.492 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.492 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.492 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.492 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.492 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.492 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.493 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.493 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.493 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.493 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.493 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.493 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.493 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.493 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.493 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.493 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.493 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.493 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.494 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.494 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:37.494 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:37.494 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.494 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.494 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.494 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.494 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.494 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.494 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.494 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.494 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.495 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.495 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.495 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.495 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:37.495 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:37.495 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.495 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.495 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.495 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:37.495 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.495 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.495 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.496 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.496 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.496 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.496 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.496 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.496 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.496 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.496 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.496 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.496 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.496 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.496 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.496 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:37.497 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:37.497 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.497 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.497 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.497 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.497 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.497 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.497 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.497 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.497 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.497 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.497 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.498 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.498 INFO:teuthology.orchestra.run.smithi019.stdout: "min_score": { 2024-03-18T15:28:37.498 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "min_score", 2024-03-18T15:28:37.498 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "float", 2024-03-18T15:28:37.498 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.498 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.498 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "0", 2024-03-18T15:28:37.498 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.498 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.498 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.498 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-18T15:28:37.498 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.499 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.499 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.499 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.499 INFO:teuthology.orchestra.run.smithi019.stdout: "mode": { 2024-03-18T15:28:37.499 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "mode", 2024-03-18T15:28:37.499 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.499 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.499 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.499 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "upmap", 2024-03-18T15:28:37.499 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.499 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.499 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.499 INFO:teuthology.orchestra.run.smithi019.stdout: "crush-compat", 2024-03-18T15:28:37.500 INFO:teuthology.orchestra.run.smithi019.stdout: "none", 2024-03-18T15:28:37.500 INFO:teuthology.orchestra.run.smithi019.stdout: "read", 2024-03-18T15:28:37.500 INFO:teuthology.orchestra.run.smithi019.stdout: "upmap", 2024-03-18T15:28:37.500 INFO:teuthology.orchestra.run.smithi019.stdout: "upmap-read" 2024-03-18T15:28:37.500 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.500 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Balancer mode", 2024-03-18T15:28:37.500 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.500 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.500 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.500 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.500 INFO:teuthology.orchestra.run.smithi019.stdout: "pool_ids": { 2024-03-18T15:28:37.500 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "pool_ids", 2024-03-18T15:28:37.501 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.501 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.501 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.501 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.501 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.501 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.501 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.501 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-18T15:28:37.501 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.501 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.501 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.501 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.501 INFO:teuthology.orchestra.run.smithi019.stdout: "sleep_interval": { 2024-03-18T15:28:37.502 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "sleep_interval", 2024-03-18T15:28:37.502 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "secs", 2024-03-18T15:28:37.502 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.502 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.502 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "60", 2024-03-18T15:28:37.502 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.502 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.502 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.502 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-18T15:28:37.502 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.502 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.502 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.503 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.503 INFO:teuthology.orchestra.run.smithi019.stdout: "upmap_max_deviation": { 2024-03-18T15:28:37.503 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "upmap_max_deviation", 2024-03-18T15:28:37.503 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:37.503 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.503 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.503 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "5", 2024-03-18T15:28:37.503 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "1", 2024-03-18T15:28:37.503 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.503 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.503 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-18T15:28:37.503 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-18T15:28:37.504 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.504 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.504 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.504 INFO:teuthology.orchestra.run.smithi019.stdout: "upmap_max_optimizations": { 2024-03-18T15:28:37.504 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "upmap_max_optimizations", 2024-03-18T15:28:37.504 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "uint", 2024-03-18T15:28:37.504 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.504 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.504 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "10", 2024-03-18T15:28:37.504 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.504 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.504 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.505 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-18T15:28:37.505 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.505 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.505 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.505 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.505 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.505 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.505 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:37.505 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "cephadm", 2024-03-18T15:28:37.505 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": true, 2024-03-18T15:28:37.505 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "", 2024-03-18T15:28:37.505 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:37.505 INFO:teuthology.orchestra.run.smithi019.stdout: "agent_down_multiplier": { 2024-03-18T15:28:37.506 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "agent_down_multiplier", 2024-03-18T15:28:37.506 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "float", 2024-03-18T15:28:37.506 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.506 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.506 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "3.0", 2024-03-18T15:28:37.506 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.506 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.506 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.506 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-18T15:28:37.506 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.506 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.507 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.507 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.507 INFO:teuthology.orchestra.run.smithi019.stdout: "agent_refresh_rate": { 2024-03-18T15:28:37.507 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "agent_refresh_rate", 2024-03-18T15:28:37.507 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "secs", 2024-03-18T15:28:37.507 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.507 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.507 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "20", 2024-03-18T15:28:37.507 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.507 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.507 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.508 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-18T15:28:37.508 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.508 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.508 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.508 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.508 INFO:teuthology.orchestra.run.smithi019.stdout: "agent_starting_port": { 2024-03-18T15:28:37.508 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "agent_starting_port", 2024-03-18T15:28:37.508 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:37.508 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.508 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.508 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "4721", 2024-03-18T15:28:37.508 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.509 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.509 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.509 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-18T15:28:37.509 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.509 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.509 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.509 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.509 INFO:teuthology.orchestra.run.smithi019.stdout: "allow_ptrace": { 2024-03-18T15:28:37.509 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "allow_ptrace", 2024-03-18T15:28:37.509 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.509 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.509 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.510 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.510 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.510 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.510 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.510 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-18T15:28:37.510 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-18T15:28:37.510 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.510 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.510 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.510 INFO:teuthology.orchestra.run.smithi019.stdout: "autotune_interval": { 2024-03-18T15:28:37.510 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "autotune_interval", 2024-03-18T15:28:37.510 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "secs", 2024-03-18T15:28:37.511 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.511 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.511 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "600", 2024-03-18T15:28:37.511 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.511 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.511 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.511 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-18T15:28:37.511 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.511 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.511 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.511 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.511 INFO:teuthology.orchestra.run.smithi019.stdout: "autotune_memory_target_ratio": { 2024-03-18T15:28:37.511 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "autotune_memory_target_ratio", 2024-03-18T15:28:37.512 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "float", 2024-03-18T15:28:37.512 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.512 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.512 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "0.7", 2024-03-18T15:28:37.512 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.512 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.512 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.512 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-18T15:28:37.512 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.512 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.512 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.512 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.513 INFO:teuthology.orchestra.run.smithi019.stdout: "cephadm_log_destination": { 2024-03-18T15:28:37.513 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "cephadm_log_destination", 2024-03-18T15:28:37.513 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.513 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.513 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.513 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.513 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.513 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.513 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.513 INFO:teuthology.orchestra.run.smithi019.stdout: "file", 2024-03-18T15:28:37.513 INFO:teuthology.orchestra.run.smithi019.stdout: "file,syslog", 2024-03-18T15:28:37.513 INFO:teuthology.orchestra.run.smithi019.stdout: "syslog" 2024-03-18T15:28:37.513 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.514 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-18T15:28:37.514 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.514 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.514 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.514 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.514 INFO:teuthology.orchestra.run.smithi019.stdout: "cgroups_split": { 2024-03-18T15:28:37.514 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "cgroups_split", 2024-03-18T15:28:37.514 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.514 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.514 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.514 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:37.514 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.515 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.515 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.515 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-18T15:28:37.515 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.515 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.515 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.515 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.515 INFO:teuthology.orchestra.run.smithi019.stdout: "config_checks_enabled": { 2024-03-18T15:28:37.515 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "config_checks_enabled", 2024-03-18T15:28:37.515 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.515 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.515 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.515 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.516 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.516 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.516 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.516 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-18T15:28:37.516 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.516 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.516 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.516 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.516 INFO:teuthology.orchestra.run.smithi019.stdout: "config_dashboard": { 2024-03-18T15:28:37.516 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "config_dashboard", 2024-03-18T15:28:37.516 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.516 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.516 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.517 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:37.517 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.517 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.517 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.517 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-18T15:28:37.517 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.517 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.517 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.517 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.517 INFO:teuthology.orchestra.run.smithi019.stdout: "container_image_alertmanager": { 2024-03-18T15:28:37.517 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "container_image_alertmanager", 2024-03-18T15:28:37.517 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.518 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.518 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.518 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-18T15:28:37.518 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.518 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.518 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.518 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Prometheus container image", 2024-03-18T15:28:37.518 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.518 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.518 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.518 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.518 INFO:teuthology.orchestra.run.smithi019.stdout: "container_image_base": { 2024-03-18T15:28:37.519 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "container_image_base", 2024-03-18T15:28:37.519 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.519 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.519 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.519 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-18T15:28:37.519 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.519 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.519 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.519 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Container image name, without the tag", 2024-03-18T15:28:37.519 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.519 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.519 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.519 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.520 INFO:teuthology.orchestra.run.smithi019.stdout: "container_image_elasticsearch": { 2024-03-18T15:28:37.520 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "container_image_elasticsearch", 2024-03-18T15:28:37.520 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.520 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.520 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.520 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-18T15:28:37.520 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.520 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.520 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.520 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "elasticsearch container image", 2024-03-18T15:28:37.520 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.520 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.521 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.521 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.521 INFO:teuthology.orchestra.run.smithi019.stdout: "container_image_grafana": { 2024-03-18T15:28:37.521 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "container_image_grafana", 2024-03-18T15:28:37.521 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.521 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.521 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.521 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-18T15:28:37.521 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.521 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.521 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.521 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Prometheus container image", 2024-03-18T15:28:37.522 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.522 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.522 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.522 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.522 INFO:teuthology.orchestra.run.smithi019.stdout: "container_image_haproxy": { 2024-03-18T15:28:37.522 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "container_image_haproxy", 2024-03-18T15:28:37.522 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.522 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.522 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.522 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-18T15:28:37.522 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.522 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.523 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.523 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "HAproxy container image", 2024-03-18T15:28:37.523 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.523 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.523 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.523 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.523 INFO:teuthology.orchestra.run.smithi019.stdout: "container_image_jaeger_agent": { 2024-03-18T15:28:37.523 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "container_image_jaeger_agent", 2024-03-18T15:28:37.523 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.523 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.523 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.523 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-18T15:28:37.524 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.524 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.524 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.524 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Jaeger agent container image", 2024-03-18T15:28:37.524 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.524 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.524 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.524 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.524 INFO:teuthology.orchestra.run.smithi019.stdout: "container_image_jaeger_collector": { 2024-03-18T15:28:37.524 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "container_image_jaeger_collector", 2024-03-18T15:28:37.524 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.524 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.525 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.525 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-18T15:28:37.525 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.525 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.525 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.525 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Jaeger collector container image", 2024-03-18T15:28:37.525 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.525 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.525 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.525 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.525 INFO:teuthology.orchestra.run.smithi019.stdout: "container_image_jaeger_query": { 2024-03-18T15:28:37.525 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "container_image_jaeger_query", 2024-03-18T15:28:37.526 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.526 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.526 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.526 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-18T15:28:37.526 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.526 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.526 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.526 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Jaeger query container image", 2024-03-18T15:28:37.526 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.526 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.526 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.527 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.527 INFO:teuthology.orchestra.run.smithi019.stdout: "container_image_keepalived": { 2024-03-18T15:28:37.527 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "container_image_keepalived", 2024-03-18T15:28:37.527 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.527 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.527 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.527 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-18T15:28:37.527 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.527 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.527 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.527 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Keepalived container image", 2024-03-18T15:28:37.527 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.528 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.528 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.528 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.528 INFO:teuthology.orchestra.run.smithi019.stdout: "container_image_loki": { 2024-03-18T15:28:37.528 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "container_image_loki", 2024-03-18T15:28:37.528 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.528 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.528 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.528 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-18T15:28:37.528 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.528 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.528 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.528 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Loki container image", 2024-03-18T15:28:37.529 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.529 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.529 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.529 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.529 INFO:teuthology.orchestra.run.smithi019.stdout: "container_image_node_exporter": { 2024-03-18T15:28:37.529 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "container_image_node_exporter", 2024-03-18T15:28:37.529 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.529 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.529 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.529 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-18T15:28:37.529 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.529 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.530 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.530 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Prometheus container image", 2024-03-18T15:28:37.530 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.530 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.530 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.530 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.530 INFO:teuthology.orchestra.run.smithi019.stdout: "container_image_nvmeof": { 2024-03-18T15:28:37.530 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "container_image_nvmeof", 2024-03-18T15:28:37.530 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.530 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.530 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.530 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-18T15:28:37.530 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.531 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.531 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.531 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Nvme-of container image", 2024-03-18T15:28:37.531 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.531 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.531 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.531 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.531 INFO:teuthology.orchestra.run.smithi019.stdout: "container_image_prometheus": { 2024-03-18T15:28:37.531 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "container_image_prometheus", 2024-03-18T15:28:37.531 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.531 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.531 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.532 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-18T15:28:37.532 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.532 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.532 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.532 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Prometheus container image", 2024-03-18T15:28:37.532 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.532 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.532 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.532 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.532 INFO:teuthology.orchestra.run.smithi019.stdout: "container_image_promtail": { 2024-03-18T15:28:37.532 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "container_image_promtail", 2024-03-18T15:28:37.532 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.532 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.533 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.533 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-18T15:28:37.533 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.533 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.533 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.533 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Promtail container image", 2024-03-18T15:28:37.533 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.533 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.533 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.533 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.533 INFO:teuthology.orchestra.run.smithi019.stdout: "container_image_snmp_gateway": { 2024-03-18T15:28:37.533 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "container_image_snmp_gateway", 2024-03-18T15:28:37.534 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.534 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.534 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.534 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-18T15:28:37.534 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.534 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.534 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.534 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "SNMP Gateway container image", 2024-03-18T15:28:37.534 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.534 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.534 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.534 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.535 INFO:teuthology.orchestra.run.smithi019.stdout: "container_init": { 2024-03-18T15:28:37.535 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "container_init", 2024-03-18T15:28:37.535 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.535 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.535 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.535 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:37.535 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.535 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.535 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.535 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Run podman/docker with `--init`", 2024-03-18T15:28:37.535 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.535 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.536 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.536 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.536 INFO:teuthology.orchestra.run.smithi019.stdout: "daemon_cache_timeout": { 2024-03-18T15:28:37.536 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "daemon_cache_timeout", 2024-03-18T15:28:37.536 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "secs", 2024-03-18T15:28:37.536 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.536 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.536 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "600", 2024-03-18T15:28:37.536 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.536 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.536 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.537 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-18T15:28:37.537 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.537 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.537 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.537 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.537 INFO:teuthology.orchestra.run.smithi019.stdout: "default_cephadm_command_timeout": { 2024-03-18T15:28:37.537 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "default_cephadm_command_timeout", 2024-03-18T15:28:37.537 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:37.537 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.537 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.537 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "900", 2024-03-18T15:28:37.537 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.538 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.538 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.538 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-18T15:28:37.538 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.538 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.538 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.538 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.538 INFO:teuthology.orchestra.run.smithi019.stdout: "default_registry": { 2024-03-18T15:28:37.538 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "default_registry", 2024-03-18T15:28:37.538 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.538 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.538 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.539 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "docker.io", 2024-03-18T15:28:37.539 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.539 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.539 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.539 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-18T15:28:37.539 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.539 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.539 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.539 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.539 INFO:teuthology.orchestra.run.smithi019.stdout: "device_cache_timeout": { 2024-03-18T15:28:37.539 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "device_cache_timeout", 2024-03-18T15:28:37.539 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "secs", 2024-03-18T15:28:37.540 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.540 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.540 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "1800", 2024-03-18T15:28:37.540 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.540 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.540 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.540 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "seconds to cache device inventory", 2024-03-18T15:28:37.540 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.540 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.540 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.540 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.540 INFO:teuthology.orchestra.run.smithi019.stdout: "device_enhanced_scan": { 2024-03-18T15:28:37.541 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "device_enhanced_scan", 2024-03-18T15:28:37.541 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.541 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.541 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.541 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.541 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.541 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.541 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.541 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-18T15:28:37.541 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.541 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.541 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.542 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.542 INFO:teuthology.orchestra.run.smithi019.stdout: "facts_cache_timeout": { 2024-03-18T15:28:37.542 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "facts_cache_timeout", 2024-03-18T15:28:37.542 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "secs", 2024-03-18T15:28:37.542 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.542 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.542 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "60", 2024-03-18T15:28:37.542 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.542 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.542 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.542 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "seconds to cache host facts data", 2024-03-18T15:28:37.542 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.543 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.543 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.543 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.543 INFO:teuthology.orchestra.run.smithi019.stdout: "host_check_interval": { 2024-03-18T15:28:37.543 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "host_check_interval", 2024-03-18T15:28:37.543 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "secs", 2024-03-18T15:28:37.543 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.544 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.544 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "600", 2024-03-18T15:28:37.544 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.544 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.544 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.544 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "how frequently to perform a host check", 2024-03-18T15:28:37.544 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.545 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.545 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.545 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.545 INFO:teuthology.orchestra.run.smithi019.stdout: "hw_monitoring": { 2024-03-18T15:28:37.545 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "hw_monitoring", 2024-03-18T15:28:37.546 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.546 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.546 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.546 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.546 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.546 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.547 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.547 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-18T15:28:37.547 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.547 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.547 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.547 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.548 INFO:teuthology.orchestra.run.smithi019.stdout: "inventory_list_all": { 2024-03-18T15:28:37.548 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "inventory_list_all", 2024-03-18T15:28:37.548 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.548 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.548 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.548 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.548 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.549 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.549 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.549 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-18T15:28:37.549 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.549 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.549 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.550 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.550 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:37.550 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:37.550 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.550 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.550 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.551 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.551 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.551 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.551 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.551 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.551 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.552 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.552 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.552 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.552 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.552 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.552 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.553 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.553 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.553 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.553 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.553 INFO:teuthology.orchestra.run.smithi019.stdout: "log_refresh_metadata": { 2024-03-18T15:28:37.553 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_refresh_metadata", 2024-03-18T15:28:37.553 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.553 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.553 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.553 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.554 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.554 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.554 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.554 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-18T15:28:37.554 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.554 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.554 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.554 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.554 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:37.554 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:37.554 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.554 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.555 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.555 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:37.555 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.555 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.555 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.555 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-18T15:28:37.555 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.555 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.555 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.555 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.555 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:37.555 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:37.555 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.556 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.556 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.556 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:37.556 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.556 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.556 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.556 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.556 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.556 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.556 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.556 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.556 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.556 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.557 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.557 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.557 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.557 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.557 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.557 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:37.557 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:37.557 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.557 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.557 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.557 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.557 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.557 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.558 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.558 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.558 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.558 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.558 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.558 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.558 INFO:teuthology.orchestra.run.smithi019.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-18T15:28:37.558 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-18T15:28:37.558 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.558 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.558 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.559 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.559 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.559 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.559 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.559 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-18T15:28:37.559 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.559 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.559 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.559 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.559 INFO:teuthology.orchestra.run.smithi019.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-18T15:28:37.559 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-18T15:28:37.559 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.560 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.560 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.560 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "*", 2024-03-18T15:28:37.560 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.560 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.560 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.560 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-18T15:28:37.560 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.560 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.560 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.560 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.560 INFO:teuthology.orchestra.run.smithi019.stdout: "max_count_per_host": { 2024-03-18T15:28:37.561 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "max_count_per_host", 2024-03-18T15:28:37.561 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:37.561 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.561 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.561 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "10", 2024-03-18T15:28:37.561 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.561 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.561 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.561 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "max number of daemons per service per host", 2024-03-18T15:28:37.561 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.561 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.561 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.562 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.562 INFO:teuthology.orchestra.run.smithi019.stdout: "max_osd_draining_count": { 2024-03-18T15:28:37.562 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "max_osd_draining_count", 2024-03-18T15:28:37.562 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:37.562 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.562 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.562 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "10", 2024-03-18T15:28:37.562 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.562 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.562 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.562 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-18T15:28:37.562 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.563 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.563 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.563 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.563 INFO:teuthology.orchestra.run.smithi019.stdout: "migration_current": { 2024-03-18T15:28:37.563 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "migration_current", 2024-03-18T15:28:37.563 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:37.563 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.563 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.563 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.563 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.563 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.563 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.563 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "internal - do not modify", 2024-03-18T15:28:37.564 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.564 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.564 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.564 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.564 INFO:teuthology.orchestra.run.smithi019.stdout: "mode": { 2024-03-18T15:28:37.564 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "mode", 2024-03-18T15:28:37.564 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.564 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.564 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.564 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "root", 2024-03-18T15:28:37.564 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.564 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.564 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.565 INFO:teuthology.orchestra.run.smithi019.stdout: "cephadm-package", 2024-03-18T15:28:37.565 INFO:teuthology.orchestra.run.smithi019.stdout: "root" 2024-03-18T15:28:37.565 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.565 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "mode for remote execution of cephadm", 2024-03-18T15:28:37.565 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.565 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.565 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.565 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.565 INFO:teuthology.orchestra.run.smithi019.stdout: "oob_default_addr": { 2024-03-18T15:28:37.565 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "oob_default_addr", 2024-03-18T15:28:37.565 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.565 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.566 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.566 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "169.254.1.1", 2024-03-18T15:28:37.566 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.566 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.566 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.566 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-18T15:28:37.566 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.566 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.566 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.566 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.566 INFO:teuthology.orchestra.run.smithi019.stdout: "prometheus_alerts_path": { 2024-03-18T15:28:37.567 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "prometheus_alerts_path", 2024-03-18T15:28:37.567 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.567 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.567 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.567 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-18T15:28:37.567 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.567 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.567 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.567 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-18T15:28:37.567 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.567 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.567 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.567 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.568 INFO:teuthology.orchestra.run.smithi019.stdout: "registry_insecure": { 2024-03-18T15:28:37.568 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "registry_insecure", 2024-03-18T15:28:37.568 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.568 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.568 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.568 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.568 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.568 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.568 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.568 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-18T15:28:37.568 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.568 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.568 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.569 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.569 INFO:teuthology.orchestra.run.smithi019.stdout: "registry_password": { 2024-03-18T15:28:37.569 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "registry_password", 2024-03-18T15:28:37.569 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.569 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.569 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.569 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.569 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.569 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.569 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.569 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-18T15:28:37.569 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.570 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.570 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.570 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.570 INFO:teuthology.orchestra.run.smithi019.stdout: "registry_url": { 2024-03-18T15:28:37.570 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "registry_url", 2024-03-18T15:28:37.570 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.570 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.570 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.570 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.570 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.570 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.570 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.570 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-18T15:28:37.571 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.571 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.571 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.571 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.571 INFO:teuthology.orchestra.run.smithi019.stdout: "registry_username": { 2024-03-18T15:28:37.571 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "registry_username", 2024-03-18T15:28:37.571 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.571 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.571 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.571 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.571 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.571 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.571 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.572 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-18T15:28:37.572 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.572 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.572 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.572 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.572 INFO:teuthology.orchestra.run.smithi019.stdout: "secure_monitoring_stack": { 2024-03-18T15:28:37.572 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "secure_monitoring_stack", 2024-03-18T15:28:37.572 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.572 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.572 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.572 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.572 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.572 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.573 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.573 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-18T15:28:37.573 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.573 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.573 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.573 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.573 INFO:teuthology.orchestra.run.smithi019.stdout: "service_discovery_port": { 2024-03-18T15:28:37.573 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "service_discovery_port", 2024-03-18T15:28:37.573 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:37.573 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.573 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.573 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "8765", 2024-03-18T15:28:37.573 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.574 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.574 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.574 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "cephadm service discovery port", 2024-03-18T15:28:37.574 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.574 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.574 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.574 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.574 INFO:teuthology.orchestra.run.smithi019.stdout: "ssh_config_file": { 2024-03-18T15:28:37.574 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "ssh_config_file", 2024-03-18T15:28:37.574 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.574 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.574 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.575 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.575 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.575 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.575 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.575 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-18T15:28:37.575 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.575 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.575 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.575 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.575 INFO:teuthology.orchestra.run.smithi019.stdout: "use_agent": { 2024-03-18T15:28:37.575 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "use_agent", 2024-03-18T15:28:37.575 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.575 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.575 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.576 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.576 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.576 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.576 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.576 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-18T15:28:37.576 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.576 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.576 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.576 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.576 INFO:teuthology.orchestra.run.smithi019.stdout: "use_repo_digest": { 2024-03-18T15:28:37.576 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "use_repo_digest", 2024-03-18T15:28:37.577 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.577 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.577 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.577 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:37.577 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.577 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.577 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.577 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-18T15:28:37.577 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.577 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.577 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.577 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.577 INFO:teuthology.orchestra.run.smithi019.stdout: "warn_on_failed_host_check": { 2024-03-18T15:28:37.578 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "warn_on_failed_host_check", 2024-03-18T15:28:37.578 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.578 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.578 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.578 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:37.578 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.578 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.578 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.578 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "raise a health warning if the host check fails", 2024-03-18T15:28:37.578 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.578 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.578 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.579 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.579 INFO:teuthology.orchestra.run.smithi019.stdout: "warn_on_stray_daemons": { 2024-03-18T15:28:37.579 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "warn_on_stray_daemons", 2024-03-18T15:28:37.579 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.579 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.579 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.579 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:37.579 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.579 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.579 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.579 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-18T15:28:37.579 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.580 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.580 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.580 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.580 INFO:teuthology.orchestra.run.smithi019.stdout: "warn_on_stray_hosts": { 2024-03-18T15:28:37.580 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "warn_on_stray_hosts", 2024-03-18T15:28:37.580 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.580 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.580 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.580 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:37.580 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.580 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.580 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.581 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-18T15:28:37.581 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.581 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.581 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.581 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.581 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.581 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.581 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:37.581 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "crash", 2024-03-18T15:28:37.581 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": true, 2024-03-18T15:28:37.581 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "", 2024-03-18T15:28:37.581 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:37.581 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:37.582 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:37.582 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.582 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.582 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.582 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.582 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.582 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.582 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.582 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.582 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.582 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.582 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.583 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.583 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.583 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.583 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.583 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.583 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.583 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.583 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.583 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:37.583 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:37.583 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.583 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.584 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.584 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.584 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.584 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.584 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.584 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.584 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.584 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.584 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.584 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.584 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:37.584 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:37.584 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.585 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.585 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.585 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:37.585 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.585 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.585 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.585 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.585 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.585 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.585 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.585 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.585 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.585 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.586 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.586 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.586 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.586 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.586 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.586 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:37.586 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:37.586 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.586 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.586 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.586 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.586 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.587 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.587 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.587 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.587 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.587 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.587 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.587 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.587 INFO:teuthology.orchestra.run.smithi019.stdout: "retain_interval": { 2024-03-18T15:28:37.587 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "retain_interval", 2024-03-18T15:28:37.587 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "secs", 2024-03-18T15:28:37.587 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.587 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.587 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "31536000", 2024-03-18T15:28:37.588 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.588 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.588 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.588 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-18T15:28:37.588 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.588 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.588 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.588 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.588 INFO:teuthology.orchestra.run.smithi019.stdout: "warn_recent_interval": { 2024-03-18T15:28:37.588 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "warn_recent_interval", 2024-03-18T15:28:37.588 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "secs", 2024-03-18T15:28:37.588 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.588 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.589 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "1209600", 2024-03-18T15:28:37.589 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.589 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.589 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.589 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-18T15:28:37.589 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.589 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.589 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.589 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.589 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.589 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.589 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:37.589 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "dashboard", 2024-03-18T15:28:37.590 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": true, 2024-03-18T15:28:37.590 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "", 2024-03-18T15:28:37.590 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:37.590 INFO:teuthology.orchestra.run.smithi019.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-18T15:28:37.590 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-18T15:28:37.590 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:37.590 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.590 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.590 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "10", 2024-03-18T15:28:37.590 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.590 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.590 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.591 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.591 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.591 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.591 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.591 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.591 INFO:teuthology.orchestra.run.smithi019.stdout: "ALERTMANAGER_API_HOST": { 2024-03-18T15:28:37.591 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-18T15:28:37.591 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.591 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.591 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.591 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.591 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.591 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.592 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.592 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.592 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.592 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.592 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.592 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.592 INFO:teuthology.orchestra.run.smithi019.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-18T15:28:37.592 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-18T15:28:37.592 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.592 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.592 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.592 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:37.592 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.593 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.593 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.593 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.593 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.593 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.593 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.593 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.593 INFO:teuthology.orchestra.run.smithi019.stdout: "AUDIT_API_ENABLED": { 2024-03-18T15:28:37.593 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "AUDIT_API_ENABLED", 2024-03-18T15:28:37.593 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.593 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.593 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.594 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.594 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.594 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.594 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.594 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.594 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.594 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.594 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.594 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.594 INFO:teuthology.orchestra.run.smithi019.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-18T15:28:37.594 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-18T15:28:37.595 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.595 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.595 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.595 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:37.595 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.595 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.595 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.595 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.595 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.595 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.595 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.595 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.595 INFO:teuthology.orchestra.run.smithi019.stdout: "ENABLE_BROWSABLE_API": { 2024-03-18T15:28:37.596 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-18T15:28:37.596 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.596 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.596 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.596 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:37.596 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.596 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.596 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.596 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.596 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.596 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.596 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.597 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.597 INFO:teuthology.orchestra.run.smithi019.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-18T15:28:37.597 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-18T15:28:37.597 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.597 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.597 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.597 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:37.597 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.597 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.597 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.597 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.597 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.597 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.598 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.598 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.598 INFO:teuthology.orchestra.run.smithi019.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-18T15:28:37.598 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-18T15:28:37.598 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.598 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.598 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.598 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:37.598 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.598 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.598 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.598 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.599 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.599 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.599 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.599 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.599 INFO:teuthology.orchestra.run.smithi019.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-18T15:28:37.599 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-18T15:28:37.599 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.599 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.599 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.599 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:37.599 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.599 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.599 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.600 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.600 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.600 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.600 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.600 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.600 INFO:teuthology.orchestra.run.smithi019.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-18T15:28:37.600 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-18T15:28:37.600 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.600 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.600 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.600 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:37.600 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.601 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.601 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.601 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.601 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.601 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.601 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.601 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.601 INFO:teuthology.orchestra.run.smithi019.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-18T15:28:37.601 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-18T15:28:37.601 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.601 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.601 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.601 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:37.602 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.602 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.602 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.602 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.602 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.602 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.602 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.602 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.602 INFO:teuthology.orchestra.run.smithi019.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-18T15:28:37.602 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-18T15:28:37.602 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.602 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.603 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.603 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:37.603 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.603 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.603 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.603 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.603 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.603 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.603 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.603 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.603 INFO:teuthology.orchestra.run.smithi019.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-18T15:28:37.603 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-18T15:28:37.604 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.604 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.604 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.604 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:37.604 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.604 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.604 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.604 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.604 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.604 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.604 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.604 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.604 INFO:teuthology.orchestra.run.smithi019.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-18T15:28:37.605 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-18T15:28:37.605 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.605 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.605 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.605 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.605 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.605 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.605 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.605 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.605 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.605 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.605 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.605 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.606 INFO:teuthology.orchestra.run.smithi019.stdout: "GRAFANA_API_PASSWORD": { 2024-03-18T15:28:37.606 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-18T15:28:37.606 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.606 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.606 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.606 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "admin", 2024-03-18T15:28:37.606 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.606 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.606 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.606 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.606 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.606 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.606 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.607 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.607 INFO:teuthology.orchestra.run.smithi019.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-18T15:28:37.607 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-18T15:28:37.607 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.607 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.607 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.607 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:37.607 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.607 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.607 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.607 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.607 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.608 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.608 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.608 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.608 INFO:teuthology.orchestra.run.smithi019.stdout: "GRAFANA_API_URL": { 2024-03-18T15:28:37.608 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "GRAFANA_API_URL", 2024-03-18T15:28:37.608 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.608 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.608 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.608 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.608 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.608 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.608 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.608 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.609 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.609 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.609 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.609 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.609 INFO:teuthology.orchestra.run.smithi019.stdout: "GRAFANA_API_USERNAME": { 2024-03-18T15:28:37.609 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-18T15:28:37.609 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.609 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.609 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.609 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "admin", 2024-03-18T15:28:37.609 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.609 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.610 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.610 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.610 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.610 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.610 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.610 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.610 INFO:teuthology.orchestra.run.smithi019.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-18T15:28:37.610 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-18T15:28:37.610 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.610 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.610 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.610 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.611 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.611 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.611 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.611 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.611 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.611 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.611 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.611 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.611 INFO:teuthology.orchestra.run.smithi019.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-18T15:28:37.611 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-18T15:28:37.611 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.611 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.611 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.612 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.612 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.612 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.612 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.612 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.612 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.612 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.612 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.612 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.612 INFO:teuthology.orchestra.run.smithi019.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-18T15:28:37.612 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-18T15:28:37.612 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.613 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.613 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.613 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:37.613 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.613 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.613 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.613 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.613 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.613 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.613 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.613 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.613 INFO:teuthology.orchestra.run.smithi019.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-18T15:28:37.614 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-18T15:28:37.614 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.614 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.614 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.614 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.614 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.614 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.614 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.614 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.614 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.614 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.614 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.615 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.615 INFO:teuthology.orchestra.run.smithi019.stdout: "MULTICLUSTER_CONFIG": { 2024-03-18T15:28:37.615 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-18T15:28:37.615 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.615 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.615 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.615 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "{}", 2024-03-18T15:28:37.615 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.615 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.615 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.615 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.615 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.615 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.616 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.616 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.616 INFO:teuthology.orchestra.run.smithi019.stdout: "PROMETHEUS_API_HOST": { 2024-03-18T15:28:37.616 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-18T15:28:37.616 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.616 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.616 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.616 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.616 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.616 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.616 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.616 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.617 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.617 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.617 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.617 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.617 INFO:teuthology.orchestra.run.smithi019.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-18T15:28:37.617 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-18T15:28:37.617 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.617 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.617 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.617 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:37.617 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.617 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.617 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.618 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.618 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.618 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.618 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.618 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.618 INFO:teuthology.orchestra.run.smithi019.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-18T15:28:37.618 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-18T15:28:37.618 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.618 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.618 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.618 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.618 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.619 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.619 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.619 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.619 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.619 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.619 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.619 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.619 INFO:teuthology.orchestra.run.smithi019.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-18T15:28:37.619 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-18T15:28:37.619 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.619 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.619 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.620 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.620 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.620 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.620 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.620 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.620 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.620 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.620 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.620 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.620 INFO:teuthology.orchestra.run.smithi019.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-18T15:28:37.620 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-18T15:28:37.620 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.621 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.621 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.621 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:37.621 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.621 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.621 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.621 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.621 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.621 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.621 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.621 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.621 INFO:teuthology.orchestra.run.smithi019.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-18T15:28:37.621 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-18T15:28:37.622 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.622 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.622 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.622 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:37.622 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.622 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.622 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.622 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.622 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.622 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.622 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.622 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.623 INFO:teuthology.orchestra.run.smithi019.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-18T15:28:37.623 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-18T15:28:37.623 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.623 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.623 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.623 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.623 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.623 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.623 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.623 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.623 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.623 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.624 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.624 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.624 INFO:teuthology.orchestra.run.smithi019.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-18T15:28:37.624 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-18T15:28:37.624 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.624 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.624 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.624 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.624 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.624 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.624 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.624 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.625 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.625 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.625 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.625 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.625 INFO:teuthology.orchestra.run.smithi019.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-18T15:28:37.625 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-18T15:28:37.625 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.625 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.625 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.625 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.626 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.626 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.626 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.626 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.626 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.626 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.626 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.626 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.626 INFO:teuthology.orchestra.run.smithi019.stdout: "PWD_POLICY_ENABLED": { 2024-03-18T15:28:37.626 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-18T15:28:37.626 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.627 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.627 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.627 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:37.627 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.627 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.627 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.627 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.627 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.627 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.627 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.627 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.627 INFO:teuthology.orchestra.run.smithi019.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-18T15:28:37.628 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-18T15:28:37.628 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.628 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.628 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.628 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-18T15:28:37.628 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.628 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.628 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.628 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.628 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.628 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.628 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.628 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.629 INFO:teuthology.orchestra.run.smithi019.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-18T15:28:37.629 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-18T15:28:37.629 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:37.629 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.629 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.629 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "10", 2024-03-18T15:28:37.629 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.629 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.629 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.629 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.629 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.629 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.630 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.630 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.630 INFO:teuthology.orchestra.run.smithi019.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-18T15:28:37.630 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-18T15:28:37.630 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:37.630 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.630 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.630 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "8", 2024-03-18T15:28:37.630 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.630 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.630 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.630 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.631 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.631 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.631 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.631 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.631 INFO:teuthology.orchestra.run.smithi019.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-18T15:28:37.631 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-18T15:28:37.631 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:37.631 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.631 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.631 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "45", 2024-03-18T15:28:37.631 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.631 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.632 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.632 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.632 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.632 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.632 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.632 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.632 INFO:teuthology.orchestra.run.smithi019.stdout: "RGW_API_ACCESS_KEY": { 2024-03-18T15:28:37.632 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-18T15:28:37.632 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.632 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.632 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.632 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.633 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.633 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.633 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.633 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.633 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.633 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.633 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.633 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.633 INFO:teuthology.orchestra.run.smithi019.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-18T15:28:37.633 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-18T15:28:37.633 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.634 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.634 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.634 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "admin", 2024-03-18T15:28:37.634 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.634 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.634 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.634 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.634 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.634 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.634 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.634 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.635 INFO:teuthology.orchestra.run.smithi019.stdout: "RGW_API_SECRET_KEY": { 2024-03-18T15:28:37.635 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-18T15:28:37.635 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.635 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.635 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.635 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.635 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.635 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.635 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.635 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.635 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.635 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.635 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.636 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.636 INFO:teuthology.orchestra.run.smithi019.stdout: "RGW_API_SSL_VERIFY": { 2024-03-18T15:28:37.636 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-18T15:28:37.636 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.636 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.636 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.636 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:37.636 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.636 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.636 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.636 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.636 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.637 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.637 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.637 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.637 INFO:teuthology.orchestra.run.smithi019.stdout: "UNSAFE_TLS_v1_2": { 2024-03-18T15:28:37.637 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-18T15:28:37.637 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.637 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.637 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.637 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.637 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.637 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.637 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.637 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.638 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.638 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.638 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.638 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.638 INFO:teuthology.orchestra.run.smithi019.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-18T15:28:37.638 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-18T15:28:37.638 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:37.638 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.638 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.638 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "0", 2024-03-18T15:28:37.638 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.638 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.639 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.639 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.639 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.639 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.639 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.639 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.639 INFO:teuthology.orchestra.run.smithi019.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-18T15:28:37.639 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-18T15:28:37.639 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:37.639 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.639 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.639 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "10", 2024-03-18T15:28:37.639 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.640 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.640 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.640 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.640 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.640 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.640 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.640 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.640 INFO:teuthology.orchestra.run.smithi019.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-18T15:28:37.640 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-18T15:28:37.640 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:37.640 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.640 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.640 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "5", 2024-03-18T15:28:37.641 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.641 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.641 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.641 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.641 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.641 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.641 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.641 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.641 INFO:teuthology.orchestra.run.smithi019.stdout: "cross_origin_url": { 2024-03-18T15:28:37.641 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "cross_origin_url", 2024-03-18T15:28:37.641 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.641 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.642 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.642 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.642 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.642 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.642 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.642 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.642 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.642 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.642 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.642 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.642 INFO:teuthology.orchestra.run.smithi019.stdout: "crt_file": { 2024-03-18T15:28:37.642 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "crt_file", 2024-03-18T15:28:37.642 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.643 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.643 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.643 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.643 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.643 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.643 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.643 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.643 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.643 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.643 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.643 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.643 INFO:teuthology.orchestra.run.smithi019.stdout: "debug": { 2024-03-18T15:28:37.643 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "debug", 2024-03-18T15:28:37.644 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.644 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.644 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.644 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.644 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.644 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.644 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.644 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Enable/disable debug options", 2024-03-18T15:28:37.644 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.644 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.644 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.644 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.645 INFO:teuthology.orchestra.run.smithi019.stdout: "jwt_token_ttl": { 2024-03-18T15:28:37.645 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "jwt_token_ttl", 2024-03-18T15:28:37.645 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:37.645 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.645 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.645 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "28800", 2024-03-18T15:28:37.645 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.645 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.645 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.645 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.645 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.645 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.645 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.646 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.646 INFO:teuthology.orchestra.run.smithi019.stdout: "key_file": { 2024-03-18T15:28:37.646 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "key_file", 2024-03-18T15:28:37.646 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.646 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.646 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.646 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.646 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.646 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.646 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.646 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.646 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.646 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.647 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.647 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.647 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:37.647 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:37.647 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.647 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.647 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.647 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.647 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.647 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.647 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.647 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.647 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.648 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.648 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.648 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.648 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.648 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.648 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.648 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.648 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.648 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.648 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.648 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:37.648 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:37.649 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.649 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.649 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.649 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.649 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.649 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.649 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.649 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.649 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.649 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.649 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.649 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.650 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:37.650 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:37.650 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.650 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.650 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.650 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:37.650 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.650 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.650 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.650 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.650 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.650 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.651 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.651 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.651 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.651 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.651 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.651 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.651 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.651 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.651 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.651 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:37.651 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:37.651 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.651 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.652 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.652 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.652 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.652 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.652 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.652 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.652 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.652 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.652 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.652 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.652 INFO:teuthology.orchestra.run.smithi019.stdout: "motd": { 2024-03-18T15:28:37.652 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "motd", 2024-03-18T15:28:37.653 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.653 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.653 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.653 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.653 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.653 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.653 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.653 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "The message of the day", 2024-03-18T15:28:37.653 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.653 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.653 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.653 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.653 INFO:teuthology.orchestra.run.smithi019.stdout: "redirect_resolve_ip_addr": { 2024-03-18T15:28:37.654 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "redirect_resolve_ip_addr", 2024-03-18T15:28:37.654 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.654 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.654 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.654 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.654 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.654 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.654 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.654 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.654 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.654 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.654 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.655 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.655 INFO:teuthology.orchestra.run.smithi019.stdout: "server_addr": { 2024-03-18T15:28:37.655 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "server_addr", 2024-03-18T15:28:37.655 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.655 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.655 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.655 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "::", 2024-03-18T15:28:37.655 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.655 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.655 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.655 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.655 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.655 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.656 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.656 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.656 INFO:teuthology.orchestra.run.smithi019.stdout: "server_port": { 2024-03-18T15:28:37.656 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "server_port", 2024-03-18T15:28:37.656 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:37.656 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.656 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.656 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "8080", 2024-03-18T15:28:37.656 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.656 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.656 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.656 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.657 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.657 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.657 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.657 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.657 INFO:teuthology.orchestra.run.smithi019.stdout: "ssl": { 2024-03-18T15:28:37.657 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "ssl", 2024-03-18T15:28:37.657 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.657 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.657 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.657 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:37.657 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.657 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.657 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.658 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.658 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.658 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.658 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.658 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.658 INFO:teuthology.orchestra.run.smithi019.stdout: "ssl_server_port": { 2024-03-18T15:28:37.658 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "ssl_server_port", 2024-03-18T15:28:37.658 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:37.658 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.658 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.658 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "8443", 2024-03-18T15:28:37.658 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.659 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.659 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.659 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.659 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.659 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.659 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.659 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.659 INFO:teuthology.orchestra.run.smithi019.stdout: "standby_behaviour": { 2024-03-18T15:28:37.659 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "standby_behaviour", 2024-03-18T15:28:37.659 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.659 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.659 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.660 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "redirect", 2024-03-18T15:28:37.660 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.660 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.660 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.660 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.660 INFO:teuthology.orchestra.run.smithi019.stdout: "redirect" 2024-03-18T15:28:37.660 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.660 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.660 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.660 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.660 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.660 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.660 INFO:teuthology.orchestra.run.smithi019.stdout: "standby_error_status_code": { 2024-03-18T15:28:37.661 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "standby_error_status_code", 2024-03-18T15:28:37.661 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:37.661 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.661 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.661 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "500", 2024-03-18T15:28:37.661 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "400", 2024-03-18T15:28:37.661 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "599", 2024-03-18T15:28:37.661 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.661 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.661 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.661 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.661 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.662 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.662 INFO:teuthology.orchestra.run.smithi019.stdout: "url_prefix": { 2024-03-18T15:28:37.662 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "url_prefix", 2024-03-18T15:28:37.662 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.662 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.662 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.662 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.662 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.662 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.662 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.662 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.662 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.663 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.663 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.663 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.663 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.663 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.663 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:37.663 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "devicehealth", 2024-03-18T15:28:37.663 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": true, 2024-03-18T15:28:37.663 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "", 2024-03-18T15:28:37.663 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:37.663 INFO:teuthology.orchestra.run.smithi019.stdout: "enable_monitoring": { 2024-03-18T15:28:37.663 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "enable_monitoring", 2024-03-18T15:28:37.663 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.664 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.664 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.664 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:37.664 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.664 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.664 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.664 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "monitor device health metrics", 2024-03-18T15:28:37.664 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.664 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.664 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.664 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.664 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:37.664 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:37.665 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.665 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.665 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.665 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.665 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.665 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.665 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.665 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.665 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.665 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.665 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.665 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.666 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.666 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.666 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.666 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.666 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.666 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.666 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.666 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:37.666 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:37.666 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.666 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.666 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.667 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.667 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.667 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.667 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.667 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.667 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.667 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.667 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.667 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.667 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:37.667 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:37.667 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.668 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.668 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.668 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:37.668 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.668 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.668 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.668 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.668 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.668 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.668 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.668 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.668 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.668 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.669 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.669 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.669 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.669 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.669 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.669 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:37.669 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:37.669 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.669 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.669 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.669 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.670 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.670 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.670 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.670 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.670 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.670 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.670 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.670 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.670 INFO:teuthology.orchestra.run.smithi019.stdout: "mark_out_threshold": { 2024-03-18T15:28:37.670 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "mark_out_threshold", 2024-03-18T15:28:37.670 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "secs", 2024-03-18T15:28:37.670 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.670 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.671 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "2419200", 2024-03-18T15:28:37.671 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.671 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.671 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.671 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-18T15:28:37.671 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.671 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.671 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.671 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.671 INFO:teuthology.orchestra.run.smithi019.stdout: "pool_name": { 2024-03-18T15:28:37.671 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "pool_name", 2024-03-18T15:28:37.671 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.672 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.672 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.672 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "device_health_metrics", 2024-03-18T15:28:37.672 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.672 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.672 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.672 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-18T15:28:37.672 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.672 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.672 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.672 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.672 INFO:teuthology.orchestra.run.smithi019.stdout: "retention_period": { 2024-03-18T15:28:37.673 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "retention_period", 2024-03-18T15:28:37.673 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "secs", 2024-03-18T15:28:37.673 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.673 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.673 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "15552000", 2024-03-18T15:28:37.673 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.673 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.673 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.673 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "how long to retain device health metrics", 2024-03-18T15:28:37.673 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.673 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.673 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.673 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.674 INFO:teuthology.orchestra.run.smithi019.stdout: "scrape_frequency": { 2024-03-18T15:28:37.674 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "scrape_frequency", 2024-03-18T15:28:37.674 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "secs", 2024-03-18T15:28:37.674 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.674 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.674 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "86400", 2024-03-18T15:28:37.674 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.674 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.674 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.674 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-18T15:28:37.674 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.674 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.675 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.675 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.675 INFO:teuthology.orchestra.run.smithi019.stdout: "self_heal": { 2024-03-18T15:28:37.675 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "self_heal", 2024-03-18T15:28:37.675 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.675 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.675 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.675 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:37.675 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.675 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.675 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.675 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-18T15:28:37.676 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.676 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.676 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.676 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.676 INFO:teuthology.orchestra.run.smithi019.stdout: "sleep_interval": { 2024-03-18T15:28:37.676 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "sleep_interval", 2024-03-18T15:28:37.676 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "secs", 2024-03-18T15:28:37.676 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.676 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.676 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "600", 2024-03-18T15:28:37.676 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.676 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.676 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.677 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "how frequently to wake up and check device health", 2024-03-18T15:28:37.677 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.677 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.677 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.677 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.677 INFO:teuthology.orchestra.run.smithi019.stdout: "warn_threshold": { 2024-03-18T15:28:37.677 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "warn_threshold", 2024-03-18T15:28:37.677 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "secs", 2024-03-18T15:28:37.677 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.677 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.677 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "7257600", 2024-03-18T15:28:37.677 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.678 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.678 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.678 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-18T15:28:37.678 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.678 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.678 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.678 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.678 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.678 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.678 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:37.678 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "diskprediction_local", 2024-03-18T15:28:37.678 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": true, 2024-03-18T15:28:37.679 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "", 2024-03-18T15:28:37.679 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:37.679 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:37.679 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:37.679 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.679 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.679 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.679 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.679 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.679 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.679 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.679 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.679 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.680 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.680 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.680 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.680 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.680 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.680 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.680 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.680 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.680 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.680 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.680 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:37.680 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:37.680 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.681 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.681 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.681 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.681 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.681 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.681 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.681 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.681 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.681 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.681 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.681 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.681 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:37.682 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:37.682 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.682 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.682 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.682 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:37.682 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.682 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.682 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.682 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.682 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.682 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.682 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.682 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.683 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.683 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.683 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.683 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.683 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.683 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.683 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.683 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:37.683 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:37.683 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.683 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.683 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.684 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.684 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.684 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.684 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.684 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.684 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.684 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.684 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.684 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.684 INFO:teuthology.orchestra.run.smithi019.stdout: "predict_interval": { 2024-03-18T15:28:37.684 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "predict_interval", 2024-03-18T15:28:37.684 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.684 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.685 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.685 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "86400", 2024-03-18T15:28:37.685 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.685 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.685 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.685 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.685 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.685 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.685 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.685 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.685 INFO:teuthology.orchestra.run.smithi019.stdout: "predictor_model": { 2024-03-18T15:28:37.685 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "predictor_model", 2024-03-18T15:28:37.686 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.686 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.686 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.686 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "prophetstor", 2024-03-18T15:28:37.686 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.686 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.686 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.686 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.686 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.687 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.687 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.687 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.687 INFO:teuthology.orchestra.run.smithi019.stdout: "sleep_interval": { 2024-03-18T15:28:37.687 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "sleep_interval", 2024-03-18T15:28:37.687 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.687 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.687 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.687 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "600", 2024-03-18T15:28:37.687 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.687 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.687 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.688 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.688 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.688 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.688 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.688 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.688 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.688 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.688 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:37.688 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "influx", 2024-03-18T15:28:37.688 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": false, 2024-03-18T15:28:37.688 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "influxdb python module not found", 2024-03-18T15:28:37.688 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:37.689 INFO:teuthology.orchestra.run.smithi019.stdout: "batch_size": { 2024-03-18T15:28:37.689 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "batch_size", 2024-03-18T15:28:37.689 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:37.689 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.689 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.689 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "5000", 2024-03-18T15:28:37.689 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.689 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.689 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.689 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-18T15:28:37.689 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.689 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.689 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.690 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.690 INFO:teuthology.orchestra.run.smithi019.stdout: "database": { 2024-03-18T15:28:37.690 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "database", 2024-03-18T15:28:37.690 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.690 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.690 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.690 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "ceph", 2024-03-18T15:28:37.690 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.690 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.690 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.690 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-18T15:28:37.691 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.691 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.691 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.691 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.691 INFO:teuthology.orchestra.run.smithi019.stdout: "hostname": { 2024-03-18T15:28:37.691 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "hostname", 2024-03-18T15:28:37.691 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.691 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.691 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.691 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.691 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.692 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.692 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.692 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "InfluxDB server hostname", 2024-03-18T15:28:37.692 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.692 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.692 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.692 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.692 INFO:teuthology.orchestra.run.smithi019.stdout: "interval": { 2024-03-18T15:28:37.692 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "interval", 2024-03-18T15:28:37.692 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "secs", 2024-03-18T15:28:37.692 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.692 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.693 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "30", 2024-03-18T15:28:37.693 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "5", 2024-03-18T15:28:37.693 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.693 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.693 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-18T15:28:37.693 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.693 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.693 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.693 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.693 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:37.693 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:37.693 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.694 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.694 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.694 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.694 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.694 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.694 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.694 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.694 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.694 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.694 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.694 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.694 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.695 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.695 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.695 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.695 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.695 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.695 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.695 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:37.695 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:37.695 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.695 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.695 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.696 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.696 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.696 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.696 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.696 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.696 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.696 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.696 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.696 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.696 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:37.696 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:37.697 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.697 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.697 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.697 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:37.697 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.697 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.697 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.697 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.697 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.697 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.697 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.697 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.698 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.698 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.698 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.698 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.698 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.698 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.698 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.698 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:37.698 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:37.698 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.698 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.698 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.699 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.699 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.699 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.699 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.699 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.699 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.699 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.699 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.699 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.699 INFO:teuthology.orchestra.run.smithi019.stdout: "password": { 2024-03-18T15:28:37.699 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "password", 2024-03-18T15:28:37.699 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.700 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.700 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.700 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.700 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.700 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.700 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.700 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "password of InfluxDB server user", 2024-03-18T15:28:37.700 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.700 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.700 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.700 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.700 INFO:teuthology.orchestra.run.smithi019.stdout: "port": { 2024-03-18T15:28:37.701 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "port", 2024-03-18T15:28:37.701 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:37.701 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.701 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.701 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "8086", 2024-03-18T15:28:37.701 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.701 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.701 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.701 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "InfluxDB server port", 2024-03-18T15:28:37.701 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.701 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.702 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.702 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.702 INFO:teuthology.orchestra.run.smithi019.stdout: "ssl": { 2024-03-18T15:28:37.702 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "ssl", 2024-03-18T15:28:37.702 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.702 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.702 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.702 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "false", 2024-03-18T15:28:37.702 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.702 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.702 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.702 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-18T15:28:37.703 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.703 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.703 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.703 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.703 INFO:teuthology.orchestra.run.smithi019.stdout: "threads": { 2024-03-18T15:28:37.703 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "threads", 2024-03-18T15:28:37.703 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:37.703 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.703 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.703 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "5", 2024-03-18T15:28:37.703 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "1", 2024-03-18T15:28:37.703 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "32", 2024-03-18T15:28:37.704 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.704 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-18T15:28:37.704 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.704 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.704 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.704 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.704 INFO:teuthology.orchestra.run.smithi019.stdout: "username": { 2024-03-18T15:28:37.704 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "username", 2024-03-18T15:28:37.704 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.704 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.704 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.704 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.705 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.705 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.705 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.705 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "username of InfluxDB server user", 2024-03-18T15:28:37.705 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.705 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.705 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.705 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.705 INFO:teuthology.orchestra.run.smithi019.stdout: "verify_ssl": { 2024-03-18T15:28:37.705 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "verify_ssl", 2024-03-18T15:28:37.705 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.706 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.706 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.706 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "true", 2024-03-18T15:28:37.706 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.706 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.706 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.706 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-18T15:28:37.706 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.706 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.706 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.706 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.707 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.707 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.707 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:37.707 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "insights", 2024-03-18T15:28:37.707 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": true, 2024-03-18T15:28:37.707 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "", 2024-03-18T15:28:37.707 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:37.707 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:37.707 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:37.707 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.707 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.707 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.708 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.708 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.708 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.708 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.708 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.708 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.708 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.708 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.708 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.708 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.708 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.708 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.708 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.709 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.709 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.709 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.709 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:37.709 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:37.709 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.709 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.709 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.709 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.709 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.709 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.709 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.710 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.710 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.710 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.710 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.710 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.710 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:37.710 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:37.710 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.710 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.710 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.710 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:37.710 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.711 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.711 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.711 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.711 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.711 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.711 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.711 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.711 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.711 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.711 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.711 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.711 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.711 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.712 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.712 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:37.712 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:37.712 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.712 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.712 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.712 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.712 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.712 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.712 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.712 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.712 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.713 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.713 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.713 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.713 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.713 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.713 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:37.713 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "iostat", 2024-03-18T15:28:37.713 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": true, 2024-03-18T15:28:37.713 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "", 2024-03-18T15:28:37.713 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:37.713 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:37.713 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:37.713 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.714 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.714 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.714 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.714 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.714 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.714 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.714 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.714 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.714 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.714 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.714 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.714 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.715 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.715 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.715 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.715 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.715 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.715 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.715 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:37.715 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:37.715 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.715 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.715 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.715 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.715 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.716 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.716 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.716 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.716 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.716 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.716 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.716 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.716 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:37.716 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:37.716 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.716 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.716 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.717 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:37.717 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.717 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.717 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.717 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.717 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.717 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.717 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.717 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.717 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.717 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.717 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.718 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.718 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.718 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.718 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.718 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:37.718 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:37.718 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.718 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.718 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.718 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.718 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.718 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.718 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.719 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.719 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.719 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.719 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.719 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.719 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.719 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.719 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:37.719 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "k8sevents", 2024-03-18T15:28:37.719 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": true, 2024-03-18T15:28:37.719 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "", 2024-03-18T15:28:37.719 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:37.720 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph_event_retention_days": { 2024-03-18T15:28:37.720 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "ceph_event_retention_days", 2024-03-18T15:28:37.720 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:37.720 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.720 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.720 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "7", 2024-03-18T15:28:37.720 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.720 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.720 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.720 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-18T15:28:37.720 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.720 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.721 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.721 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.721 INFO:teuthology.orchestra.run.smithi019.stdout: "config_check_secs": { 2024-03-18T15:28:37.721 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "config_check_secs", 2024-03-18T15:28:37.721 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:37.721 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.721 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.721 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "10", 2024-03-18T15:28:37.721 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "10", 2024-03-18T15:28:37.721 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.721 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.721 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-18T15:28:37.721 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.722 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.722 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.722 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.722 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:37.722 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:37.722 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.722 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.722 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.722 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.722 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.722 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.723 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.723 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.723 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.723 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.723 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.723 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.723 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.723 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.723 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.723 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.723 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.723 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.723 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.724 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:37.724 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:37.724 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.724 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.724 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.724 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.724 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.724 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.724 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.724 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.724 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.724 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.724 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.725 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.725 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:37.725 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:37.725 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.725 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.725 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.725 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:37.725 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.725 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.725 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.725 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.725 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.726 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.726 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.726 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.726 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.726 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.726 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.726 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.726 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.726 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.726 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.726 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:37.726 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:37.727 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.727 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.727 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.727 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.727 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.727 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.727 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.727 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.727 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.727 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.727 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.727 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.727 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.728 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.728 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:37.728 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "localpool", 2024-03-18T15:28:37.728 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": true, 2024-03-18T15:28:37.728 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "", 2024-03-18T15:28:37.728 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:37.728 INFO:teuthology.orchestra.run.smithi019.stdout: "failure_domain": { 2024-03-18T15:28:37.728 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "failure_domain", 2024-03-18T15:28:37.728 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.728 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.728 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.728 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "host", 2024-03-18T15:28:37.729 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.729 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.729 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.729 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "failure domain for any created local pool", 2024-03-18T15:28:37.729 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-18T15:28:37.729 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.729 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.729 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.729 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:37.729 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:37.729 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.729 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.729 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.730 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.730 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.730 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.730 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.730 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.730 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.730 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.730 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.730 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.730 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.730 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.731 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.731 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.731 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.731 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.731 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.731 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:37.731 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:37.731 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.731 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.731 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.731 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.731 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.732 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.732 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.732 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.732 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.732 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.732 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.732 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.732 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:37.732 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:37.732 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.732 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.732 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.732 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:37.733 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.733 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.733 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.733 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.733 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.733 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.733 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.733 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.733 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.733 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.733 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.733 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.734 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.734 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.734 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.734 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:37.734 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:37.734 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.734 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.734 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.734 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.734 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.734 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.734 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.735 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.735 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.735 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.735 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.735 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.735 INFO:teuthology.orchestra.run.smithi019.stdout: "min_size": { 2024-03-18T15:28:37.735 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "min_size", 2024-03-18T15:28:37.735 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:37.735 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.735 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.735 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.735 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.735 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.736 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.736 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "default min_size for any created local pool", 2024-03-18T15:28:37.736 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-18T15:28:37.736 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.736 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.736 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.736 INFO:teuthology.orchestra.run.smithi019.stdout: "num_rep": { 2024-03-18T15:28:37.736 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "num_rep", 2024-03-18T15:28:37.736 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:37.736 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.736 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.736 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "3", 2024-03-18T15:28:37.736 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.737 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.737 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.737 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "default replica count for any created local pool", 2024-03-18T15:28:37.737 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.737 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.737 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.737 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.737 INFO:teuthology.orchestra.run.smithi019.stdout: "pg_num": { 2024-03-18T15:28:37.737 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "pg_num", 2024-03-18T15:28:37.737 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:37.737 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.737 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.738 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "128", 2024-03-18T15:28:37.738 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.738 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.738 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.738 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "default pg_num for any created local pool", 2024-03-18T15:28:37.738 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.738 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.738 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.738 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.738 INFO:teuthology.orchestra.run.smithi019.stdout: "prefix": { 2024-03-18T15:28:37.738 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "prefix", 2024-03-18T15:28:37.738 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.739 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.739 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.739 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.739 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.739 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.739 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.739 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "name prefix for any created local pool", 2024-03-18T15:28:37.739 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.739 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.739 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.739 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.739 INFO:teuthology.orchestra.run.smithi019.stdout: "subtree": { 2024-03-18T15:28:37.739 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "subtree", 2024-03-18T15:28:37.740 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.740 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.740 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.740 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "rack", 2024-03-18T15:28:37.740 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.740 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.740 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.740 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-18T15:28:37.740 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-18T15:28:37.740 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.740 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.740 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.740 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.741 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.741 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:37.741 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "mds_autoscaler", 2024-03-18T15:28:37.741 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": true, 2024-03-18T15:28:37.741 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "", 2024-03-18T15:28:37.741 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:37.741 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:37.741 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:37.741 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.741 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.741 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.741 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.741 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.742 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.742 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.742 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.742 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.742 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.742 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.742 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.742 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.742 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.742 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.742 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.742 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.743 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.743 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.743 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:37.743 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:37.743 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.743 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.743 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.743 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.743 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.743 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.743 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.743 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.744 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.744 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.744 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.744 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.744 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:37.744 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:37.744 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.744 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.744 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.744 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:37.744 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.744 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.745 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.745 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.745 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.745 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.745 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.745 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.745 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.745 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.745 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.745 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.745 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.745 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.745 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.746 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:37.746 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:37.746 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.746 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.746 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.746 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.746 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.746 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.746 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.746 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.746 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.746 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.747 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.747 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.747 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.747 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.747 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:37.747 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "mirroring", 2024-03-18T15:28:37.747 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": true, 2024-03-18T15:28:37.747 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "", 2024-03-18T15:28:37.747 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:37.747 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:37.747 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:37.747 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.748 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.748 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.748 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.748 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.748 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.748 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.748 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.748 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.748 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.748 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.748 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.748 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.748 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.749 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.749 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.749 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.749 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.749 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.749 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:37.749 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:37.749 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.749 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.749 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.749 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.749 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.750 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.750 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.750 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.750 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.750 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.750 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.750 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.750 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:37.750 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:37.750 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.750 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.750 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.751 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:37.751 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.751 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.751 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.751 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.751 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.751 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.751 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.751 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.751 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.751 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.751 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.751 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.752 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.752 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.752 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.752 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:37.752 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:37.752 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.752 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.752 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.752 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.752 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.752 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.752 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.753 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.753 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.753 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.753 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.753 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.753 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.753 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.753 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:37.753 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "nfs", 2024-03-18T15:28:37.753 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": true, 2024-03-18T15:28:37.753 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "", 2024-03-18T15:28:37.753 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:37.754 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:37.754 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:37.754 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.754 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.754 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.754 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.754 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.754 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.754 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.754 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.754 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.754 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.755 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.755 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.755 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.755 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.755 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.755 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.755 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.755 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.755 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.755 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:37.755 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:37.755 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.755 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.756 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.756 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.756 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.756 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.756 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.756 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.756 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.756 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.756 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.756 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.756 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:37.756 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:37.757 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.757 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.757 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.757 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:37.757 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.757 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.757 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.757 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.757 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.757 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.757 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.757 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.757 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.758 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.758 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.758 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.758 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.758 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.758 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.758 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:37.758 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:37.758 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.758 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.758 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.758 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.759 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.759 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.759 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.759 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.759 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.759 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.759 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.759 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.759 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.759 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.759 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:37.759 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "orchestrator", 2024-03-18T15:28:37.759 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": true, 2024-03-18T15:28:37.760 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "", 2024-03-18T15:28:37.760 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:37.760 INFO:teuthology.orchestra.run.smithi019.stdout: "fail_fs": { 2024-03-18T15:28:37.760 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "fail_fs", 2024-03-18T15:28:37.760 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.760 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.760 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.760 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.760 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.760 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.760 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.760 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-18T15:28:37.761 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.761 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.761 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.761 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.761 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:37.761 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:37.761 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.761 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.761 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.761 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.761 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.761 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.762 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.762 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.762 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.762 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.762 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.762 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.762 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.762 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.762 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.762 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.762 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.762 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.763 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.763 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:37.763 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:37.763 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.763 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.763 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.763 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.763 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.763 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.763 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.763 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.764 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.764 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.764 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.764 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.764 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:37.764 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:37.764 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.764 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.764 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.764 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:37.764 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.764 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.765 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.765 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.765 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.765 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.765 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.765 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.765 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.765 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.765 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.765 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.766 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.766 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.766 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.766 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:37.766 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:37.766 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.766 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.766 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.766 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.766 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.766 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.766 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.767 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.767 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.767 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.767 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.767 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.767 INFO:teuthology.orchestra.run.smithi019.stdout: "orchestrator": { 2024-03-18T15:28:37.767 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "orchestrator", 2024-03-18T15:28:37.767 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.767 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.767 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.767 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.767 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.768 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.768 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.768 INFO:teuthology.orchestra.run.smithi019.stdout: "cephadm", 2024-03-18T15:28:37.768 INFO:teuthology.orchestra.run.smithi019.stdout: "rook", 2024-03-18T15:28:37.768 INFO:teuthology.orchestra.run.smithi019.stdout: "test_orchestrator" 2024-03-18T15:28:37.768 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.768 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Orchestrator backend", 2024-03-18T15:28:37.768 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.768 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.768 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.769 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.769 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.769 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.769 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:37.769 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "osd_perf_query", 2024-03-18T15:28:37.769 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": true, 2024-03-18T15:28:37.769 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "", 2024-03-18T15:28:37.769 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:37.769 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:37.769 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:37.769 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.769 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.770 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.770 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.770 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.770 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.770 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.770 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.770 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.770 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.770 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.770 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.770 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.770 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.771 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.771 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.771 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.771 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.771 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.771 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:37.771 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:37.771 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.771 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.771 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.771 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.771 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.772 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.772 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.772 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.772 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.772 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.772 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.772 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.772 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:37.772 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:37.772 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.772 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.772 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.773 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:37.773 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.773 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.773 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.773 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.773 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.773 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.773 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.773 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.773 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.773 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.773 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.773 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.774 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.774 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.774 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.774 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:37.774 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:37.774 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.774 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.774 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.774 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.774 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.774 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.774 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.775 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.775 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.775 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.775 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.775 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.775 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.775 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.775 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:37.775 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "osd_support", 2024-03-18T15:28:37.775 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": true, 2024-03-18T15:28:37.775 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "", 2024-03-18T15:28:37.775 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:37.775 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:37.776 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:37.776 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.776 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.776 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.776 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.776 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.776 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.776 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.776 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.776 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.776 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.776 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.776 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.777 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.777 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.777 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.777 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.777 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.777 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.777 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.777 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:37.777 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:37.777 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.777 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.777 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.777 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.778 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.778 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.778 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.778 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.778 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.778 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.778 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.778 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.778 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:37.778 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:37.778 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.778 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.779 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.779 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:37.779 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.779 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.779 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.779 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.779 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.779 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.779 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.779 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.779 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.779 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.779 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.780 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.780 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.780 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.780 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.780 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:37.780 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:37.780 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.780 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.780 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.780 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.780 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.780 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.781 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.781 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.781 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.781 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.781 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.781 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.781 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.781 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.781 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:37.781 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "pg_autoscaler", 2024-03-18T15:28:37.781 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": true, 2024-03-18T15:28:37.781 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "", 2024-03-18T15:28:37.781 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:37.782 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:37.782 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:37.782 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.782 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.782 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.782 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.782 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.782 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.782 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.782 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.782 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.782 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.783 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.783 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.783 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.783 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.783 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.783 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.783 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.783 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.783 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.783 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:37.783 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:37.783 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.783 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.784 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.784 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.784 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.784 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.784 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.784 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.784 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.784 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.784 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.784 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.784 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:37.785 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:37.785 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.785 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.785 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.785 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:37.785 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.785 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.785 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.785 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.785 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.785 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.785 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.786 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.786 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.786 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.786 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.786 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.786 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.786 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.786 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.786 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:37.786 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:37.786 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.786 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.787 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.787 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.787 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.787 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.787 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.787 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.787 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.787 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.787 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.787 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.787 INFO:teuthology.orchestra.run.smithi019.stdout: "sleep_interval": { 2024-03-18T15:28:37.787 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "sleep_interval", 2024-03-18T15:28:37.787 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "secs", 2024-03-18T15:28:37.788 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.788 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.788 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "60", 2024-03-18T15:28:37.788 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.788 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.788 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.788 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.788 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.788 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.788 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.788 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.788 INFO:teuthology.orchestra.run.smithi019.stdout: "threshold": { 2024-03-18T15:28:37.789 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "threshold", 2024-03-18T15:28:37.789 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "float", 2024-03-18T15:28:37.789 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.789 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.789 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "3.0", 2024-03-18T15:28:37.789 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "1.0", 2024-03-18T15:28:37.789 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.789 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.789 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "scaling threshold", 2024-03-18T15:28:37.789 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-18T15:28:37.789 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.789 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.789 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.790 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.790 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.790 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:37.790 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "progress", 2024-03-18T15:28:37.790 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": true, 2024-03-18T15:28:37.790 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "", 2024-03-18T15:28:37.790 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:37.790 INFO:teuthology.orchestra.run.smithi019.stdout: "allow_pg_recovery_event": { 2024-03-18T15:28:37.790 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "allow_pg_recovery_event", 2024-03-18T15:28:37.790 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.790 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.790 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.791 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.791 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.791 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.791 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.791 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-18T15:28:37.791 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.791 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.791 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.791 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.791 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled": { 2024-03-18T15:28:37.791 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "enabled", 2024-03-18T15:28:37.791 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.792 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.792 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.792 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:37.792 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.792 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.792 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.792 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.792 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.792 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.792 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.792 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.792 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:37.792 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:37.793 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.793 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.793 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.793 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.793 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.793 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.793 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.793 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.793 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.793 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.793 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.793 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.794 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.794 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.794 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.794 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.794 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.794 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.794 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.794 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:37.794 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:37.794 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.794 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.794 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.794 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.795 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.795 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.795 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.795 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.795 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.795 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.795 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.795 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.795 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:37.795 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:37.795 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.795 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.796 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.796 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:37.796 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.796 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.796 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.796 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.796 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.796 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.796 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.796 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.796 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.796 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.796 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.797 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.797 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.797 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.797 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.797 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:37.797 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:37.797 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.797 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.797 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.797 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.797 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.797 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.797 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.798 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.798 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.798 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.798 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.798 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.798 INFO:teuthology.orchestra.run.smithi019.stdout: "max_completed_events": { 2024-03-18T15:28:37.798 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "max_completed_events", 2024-03-18T15:28:37.798 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:37.798 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.798 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.798 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "50", 2024-03-18T15:28:37.798 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.799 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.799 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.799 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "number of past completed events to remember", 2024-03-18T15:28:37.799 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.799 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.799 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.799 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.799 INFO:teuthology.orchestra.run.smithi019.stdout: "sleep_interval": { 2024-03-18T15:28:37.799 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "sleep_interval", 2024-03-18T15:28:37.799 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "secs", 2024-03-18T15:28:37.799 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.799 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.799 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "5", 2024-03-18T15:28:37.800 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.800 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.800 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.800 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "how long the module is going to sleep", 2024-03-18T15:28:37.800 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.800 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.800 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.800 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.800 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.800 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.800 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:37.800 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "prometheus", 2024-03-18T15:28:37.800 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": true, 2024-03-18T15:28:37.801 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "", 2024-03-18T15:28:37.801 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:37.801 INFO:teuthology.orchestra.run.smithi019.stdout: "cache": { 2024-03-18T15:28:37.801 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "cache", 2024-03-18T15:28:37.801 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.801 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.801 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.801 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:37.801 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.801 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.801 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.801 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.801 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.802 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.802 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.802 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.802 INFO:teuthology.orchestra.run.smithi019.stdout: "exclude_perf_counters": { 2024-03-18T15:28:37.802 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "exclude_perf_counters", 2024-03-18T15:28:37.802 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.802 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.802 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.802 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:37.802 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.802 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.802 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.803 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-18T15:28:37.803 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-18T15:28:37.803 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.803 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.803 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.803 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:37.803 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:37.803 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.803 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.803 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.803 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.803 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.804 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.804 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.804 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.804 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.804 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.804 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.804 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.804 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.804 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.804 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.804 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.804 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.804 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.805 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.805 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:37.805 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:37.805 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.805 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.805 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.805 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.805 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.805 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.805 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.805 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.805 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.806 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.806 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.806 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.806 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:37.806 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:37.806 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.806 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.806 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.806 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:37.806 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.806 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.806 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.807 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.807 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.807 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.807 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.807 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.807 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.807 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.807 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.807 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.807 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.807 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.807 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.808 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:37.808 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:37.808 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.808 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.808 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.808 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.808 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.808 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.808 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.808 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.808 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.808 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.809 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.809 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.809 INFO:teuthology.orchestra.run.smithi019.stdout: "rbd_stats_pools": { 2024-03-18T15:28:37.809 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "rbd_stats_pools", 2024-03-18T15:28:37.809 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.809 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.809 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.809 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.809 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.809 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.809 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.809 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.810 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.810 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.810 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.810 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.810 INFO:teuthology.orchestra.run.smithi019.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-18T15:28:37.810 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-18T15:28:37.810 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:37.810 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.810 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.810 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "300", 2024-03-18T15:28:37.810 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.810 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.811 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.811 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.811 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.811 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.811 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.811 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.811 INFO:teuthology.orchestra.run.smithi019.stdout: "scrape_interval": { 2024-03-18T15:28:37.811 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "scrape_interval", 2024-03-18T15:28:37.811 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "float", 2024-03-18T15:28:37.811 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.811 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.811 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "15.0", 2024-03-18T15:28:37.811 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.812 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.812 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.812 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.812 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.812 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.812 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.812 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.812 INFO:teuthology.orchestra.run.smithi019.stdout: "server_addr": { 2024-03-18T15:28:37.812 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "server_addr", 2024-03-18T15:28:37.812 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.812 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.812 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.812 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "::", 2024-03-18T15:28:37.813 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.813 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.813 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.813 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-18T15:28:37.813 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.813 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.813 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.813 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.813 INFO:teuthology.orchestra.run.smithi019.stdout: "server_port": { 2024-03-18T15:28:37.813 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "server_port", 2024-03-18T15:28:37.813 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:37.813 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.814 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.814 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "9283", 2024-03-18T15:28:37.814 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.814 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.814 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.814 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-18T15:28:37.814 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.814 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.814 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.814 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.814 INFO:teuthology.orchestra.run.smithi019.stdout: "stale_cache_strategy": { 2024-03-18T15:28:37.814 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "stale_cache_strategy", 2024-03-18T15:28:37.815 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.815 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.815 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.815 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "log", 2024-03-18T15:28:37.815 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.815 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.815 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.815 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.815 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.815 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.815 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.815 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.815 INFO:teuthology.orchestra.run.smithi019.stdout: "standby_behaviour": { 2024-03-18T15:28:37.816 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "standby_behaviour", 2024-03-18T15:28:37.816 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.816 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.816 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.816 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "default", 2024-03-18T15:28:37.816 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.816 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.816 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.816 INFO:teuthology.orchestra.run.smithi019.stdout: "default", 2024-03-18T15:28:37.816 INFO:teuthology.orchestra.run.smithi019.stdout: "error" 2024-03-18T15:28:37.816 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.816 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.817 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.817 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.817 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.817 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.817 INFO:teuthology.orchestra.run.smithi019.stdout: "standby_error_status_code": { 2024-03-18T15:28:37.817 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "standby_error_status_code", 2024-03-18T15:28:37.817 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:37.817 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.817 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.817 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "500", 2024-03-18T15:28:37.817 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "400", 2024-03-18T15:28:37.817 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "599", 2024-03-18T15:28:37.817 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.818 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.818 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.818 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.818 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.818 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.818 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.818 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.818 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:37.818 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "rbd_support", 2024-03-18T15:28:37.818 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": true, 2024-03-18T15:28:37.818 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "", 2024-03-18T15:28:37.818 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:37.818 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:37.819 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:37.819 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.819 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.819 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.819 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.819 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.819 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.819 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.819 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.819 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.819 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.819 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.820 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.820 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.820 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.820 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.820 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.820 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.820 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.820 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.820 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:37.820 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:37.820 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.820 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.821 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.821 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.821 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.821 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.821 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.821 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.821 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.821 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.821 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.821 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.821 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:37.821 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:37.822 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.822 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.822 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.822 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:37.822 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.822 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.822 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.822 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.822 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.822 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.822 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.822 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.823 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.823 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.823 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.823 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.823 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.823 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.823 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.823 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:37.823 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:37.823 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.823 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.823 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.823 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.824 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.824 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.824 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.824 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.824 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.824 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.824 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.824 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.824 INFO:teuthology.orchestra.run.smithi019.stdout: "max_concurrent_snap_create": { 2024-03-18T15:28:37.824 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "max_concurrent_snap_create", 2024-03-18T15:28:37.824 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:37.824 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.825 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.825 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "10", 2024-03-18T15:28:37.825 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.825 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.825 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.825 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.825 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.825 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.825 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.825 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.825 INFO:teuthology.orchestra.run.smithi019.stdout: "mirror_snapshot_schedule": { 2024-03-18T15:28:37.825 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "mirror_snapshot_schedule", 2024-03-18T15:28:37.826 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.826 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.826 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.826 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.826 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.826 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.826 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.826 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.826 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.826 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.826 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.826 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.827 INFO:teuthology.orchestra.run.smithi019.stdout: "trash_purge_schedule": { 2024-03-18T15:28:37.827 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "trash_purge_schedule", 2024-03-18T15:28:37.827 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.827 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.827 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.827 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.827 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.827 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.827 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.827 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.827 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.827 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.827 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.828 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.828 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.828 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.828 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:37.828 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "restful", 2024-03-18T15:28:37.828 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": true, 2024-03-18T15:28:37.828 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "", 2024-03-18T15:28:37.828 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:37.828 INFO:teuthology.orchestra.run.smithi019.stdout: "enable_auth": { 2024-03-18T15:28:37.828 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "enable_auth", 2024-03-18T15:28:37.828 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.828 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.829 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.829 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:37.829 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.829 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.829 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.829 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.829 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.829 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.829 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.829 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.829 INFO:teuthology.orchestra.run.smithi019.stdout: "key_file": { 2024-03-18T15:28:37.829 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "key_file", 2024-03-18T15:28:37.829 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.830 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.830 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.830 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.830 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.830 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.830 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.830 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.830 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.830 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.830 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.830 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.830 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:37.831 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:37.831 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.831 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.831 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.831 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.831 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.831 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.831 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.831 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.831 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.831 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.831 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.831 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.832 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.832 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.832 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.832 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.832 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.832 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.832 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.832 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:37.832 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:37.832 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.832 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.832 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.832 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.833 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.833 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.833 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.833 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.833 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.833 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.833 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.833 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.833 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:37.833 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:37.833 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.833 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.834 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.834 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:37.834 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.834 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.834 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.834 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.834 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.834 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.834 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.834 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.834 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.834 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.835 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.835 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.835 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.835 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.835 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.835 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:37.835 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:37.835 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.835 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.835 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.835 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.835 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.835 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.836 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.836 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.836 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.836 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.836 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.836 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.836 INFO:teuthology.orchestra.run.smithi019.stdout: "server_addr": { 2024-03-18T15:28:37.836 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "server_addr", 2024-03-18T15:28:37.836 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.836 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.836 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.836 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.837 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.837 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.837 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.837 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.837 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.837 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.837 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.837 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.837 INFO:teuthology.orchestra.run.smithi019.stdout: "server_port": { 2024-03-18T15:28:37.837 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "server_port", 2024-03-18T15:28:37.837 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.837 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.837 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.838 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.838 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.838 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.838 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.838 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.838 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.838 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.838 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.838 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.838 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.838 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.838 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:37.839 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "rgw", 2024-03-18T15:28:37.839 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": true, 2024-03-18T15:28:37.839 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "", 2024-03-18T15:28:37.839 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:37.839 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:37.839 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:37.839 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.839 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.839 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.839 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.839 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.839 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.840 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.840 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.840 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.840 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.840 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.840 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.840 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.840 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.840 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.840 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.840 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.840 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.841 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.841 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:37.841 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:37.841 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.841 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.841 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.841 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.841 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.841 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.841 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.841 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.841 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.842 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.842 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.842 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.842 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:37.842 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:37.842 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.842 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.842 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.842 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:37.842 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.842 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.842 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.843 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.843 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.843 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.843 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.843 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.843 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.843 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.843 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.843 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.843 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.843 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.843 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.844 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:37.844 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:37.844 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.844 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.844 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.844 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.844 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.844 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.844 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.844 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.844 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.844 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.844 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.845 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.845 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.845 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.845 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:37.845 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "rook", 2024-03-18T15:28:37.845 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": true, 2024-03-18T15:28:37.845 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "", 2024-03-18T15:28:37.845 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:37.845 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:37.845 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:37.845 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.845 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.845 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.846 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.846 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.846 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.846 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.846 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.846 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.846 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.846 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.846 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.846 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.846 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.846 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.847 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.847 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.847 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.847 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.847 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:37.847 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:37.847 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.847 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.847 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.847 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.847 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.847 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.848 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.848 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.848 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.848 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.848 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.848 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.848 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:37.848 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:37.848 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.848 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.848 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.848 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:37.849 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.849 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.849 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.849 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.849 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.849 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.849 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.849 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.849 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.849 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.849 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.849 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.850 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.850 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.850 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.850 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:37.850 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:37.850 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.850 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.850 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.850 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.850 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.850 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.850 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.851 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.851 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.851 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.851 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.851 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.851 INFO:teuthology.orchestra.run.smithi019.stdout: "storage_class": { 2024-03-18T15:28:37.851 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "storage_class", 2024-03-18T15:28:37.851 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.851 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.851 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.851 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "local", 2024-03-18T15:28:37.851 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.851 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.852 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.852 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-18T15:28:37.852 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.852 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.852 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.852 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.852 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.852 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.852 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:37.852 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "selftest", 2024-03-18T15:28:37.852 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": true, 2024-03-18T15:28:37.852 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "", 2024-03-18T15:28:37.853 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:37.853 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:37.853 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:37.853 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.853 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.853 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.853 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.853 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.853 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.853 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.853 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.853 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.853 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.854 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.854 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.854 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.854 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.854 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.854 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.854 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.854 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.854 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.854 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:37.854 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:37.854 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.855 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.855 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.855 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.855 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.855 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.855 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.855 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.855 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.855 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.855 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.855 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.855 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:37.855 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:37.856 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.856 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.856 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.856 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:37.856 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.856 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.856 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.856 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.856 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.856 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.856 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.856 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.857 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.857 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.857 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.857 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.857 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.857 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.857 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.857 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:37.857 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:37.857 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.857 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.857 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.858 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.858 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.858 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.858 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.858 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.858 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.858 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.858 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.858 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.858 INFO:teuthology.orchestra.run.smithi019.stdout: "roption1": { 2024-03-18T15:28:37.858 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "roption1", 2024-03-18T15:28:37.858 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.859 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.859 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.859 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.859 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.859 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.859 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.859 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.859 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.859 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.859 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.859 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.859 INFO:teuthology.orchestra.run.smithi019.stdout: "roption2": { 2024-03-18T15:28:37.859 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "roption2", 2024-03-18T15:28:37.860 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.860 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.860 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.860 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "xyz", 2024-03-18T15:28:37.860 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.860 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.860 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.860 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.860 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.860 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.860 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.861 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.861 INFO:teuthology.orchestra.run.smithi019.stdout: "rwoption1": { 2024-03-18T15:28:37.861 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "rwoption1", 2024-03-18T15:28:37.861 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.861 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.861 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.861 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.861 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.861 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.861 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.861 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.861 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.861 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.862 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.862 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.862 INFO:teuthology.orchestra.run.smithi019.stdout: "rwoption2": { 2024-03-18T15:28:37.862 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "rwoption2", 2024-03-18T15:28:37.862 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:37.862 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.862 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.862 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.862 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.862 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.862 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.862 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.863 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.863 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.863 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.863 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.863 INFO:teuthology.orchestra.run.smithi019.stdout: "rwoption3": { 2024-03-18T15:28:37.863 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "rwoption3", 2024-03-18T15:28:37.863 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "float", 2024-03-18T15:28:37.863 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.863 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.863 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.863 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.863 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.864 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.864 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.864 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.864 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.864 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.864 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.864 INFO:teuthology.orchestra.run.smithi019.stdout: "rwoption4": { 2024-03-18T15:28:37.864 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "rwoption4", 2024-03-18T15:28:37.864 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.864 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.864 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.864 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.864 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.865 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.865 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.865 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.865 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.865 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.865 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.865 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.865 INFO:teuthology.orchestra.run.smithi019.stdout: "rwoption5": { 2024-03-18T15:28:37.865 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "rwoption5", 2024-03-18T15:28:37.865 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.865 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.865 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.866 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.866 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.866 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.866 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.866 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.866 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.866 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.866 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.866 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.866 INFO:teuthology.orchestra.run.smithi019.stdout: "rwoption6": { 2024-03-18T15:28:37.866 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "rwoption6", 2024-03-18T15:28:37.866 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.867 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.867 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.867 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:37.867 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.867 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.867 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.867 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.867 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.867 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.867 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.867 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.867 INFO:teuthology.orchestra.run.smithi019.stdout: "rwoption7": { 2024-03-18T15:28:37.868 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "rwoption7", 2024-03-18T15:28:37.868 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:37.868 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.868 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.868 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.868 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "1", 2024-03-18T15:28:37.868 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "42", 2024-03-18T15:28:37.868 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.868 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.868 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.868 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.868 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.868 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.869 INFO:teuthology.orchestra.run.smithi019.stdout: "testkey": { 2024-03-18T15:28:37.869 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "testkey", 2024-03-18T15:28:37.869 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.869 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.869 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.869 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.869 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.869 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.869 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.869 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.869 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.869 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.870 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.870 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.870 INFO:teuthology.orchestra.run.smithi019.stdout: "testlkey": { 2024-03-18T15:28:37.870 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "testlkey", 2024-03-18T15:28:37.870 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.870 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.870 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.870 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.870 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.870 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.870 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.870 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.871 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.871 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.871 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.871 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.871 INFO:teuthology.orchestra.run.smithi019.stdout: "testnewline": { 2024-03-18T15:28:37.871 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "testnewline", 2024-03-18T15:28:37.871 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.871 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.871 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.871 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.871 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.871 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.871 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.872 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.872 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.872 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.872 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.872 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.872 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.872 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.872 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:37.872 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "snap_schedule", 2024-03-18T15:28:37.872 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": true, 2024-03-18T15:28:37.872 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "", 2024-03-18T15:28:37.872 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:37.872 INFO:teuthology.orchestra.run.smithi019.stdout: "allow_m_granularity": { 2024-03-18T15:28:37.873 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "allow_m_granularity", 2024-03-18T15:28:37.873 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.873 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.873 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.873 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.873 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.873 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.873 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.873 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "allow minute scheduled snapshots", 2024-03-18T15:28:37.873 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.873 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.873 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.874 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.874 INFO:teuthology.orchestra.run.smithi019.stdout: "dump_on_update": { 2024-03-18T15:28:37.874 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "dump_on_update", 2024-03-18T15:28:37.874 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.874 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.874 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.874 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.874 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.874 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.874 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.874 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "dump database to debug log on update", 2024-03-18T15:28:37.874 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.875 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.875 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.875 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.875 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:37.875 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:37.875 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.875 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.875 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.875 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.875 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.875 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.875 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.876 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.876 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.876 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.876 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.876 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.876 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.876 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.876 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.876 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.876 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.876 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.876 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.876 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:37.877 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:37.877 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.877 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.877 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.877 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.877 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.877 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.877 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.877 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.877 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.877 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.877 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.878 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.878 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:37.878 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:37.878 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.878 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.878 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.878 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:37.878 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.878 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.882 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.882 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.882 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.882 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.882 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.882 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.883 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.883 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.883 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.883 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.883 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.883 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.883 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.883 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:37.883 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:37.883 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.883 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.883 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.884 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.884 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.884 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.884 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.884 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.884 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.884 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.884 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.884 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.884 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.884 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.884 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:37.885 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "stats", 2024-03-18T15:28:37.885 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": true, 2024-03-18T15:28:37.885 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "", 2024-03-18T15:28:37.885 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:37.885 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:37.885 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:37.885 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.885 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.885 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.885 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.885 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.886 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.886 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.886 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.886 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.886 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.886 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.886 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.886 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.886 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.886 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.886 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.886 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.887 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.887 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.887 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:37.887 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:37.887 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.887 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.887 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.887 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.887 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.887 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.887 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.887 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.888 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.888 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.888 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.888 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.888 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:37.888 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:37.888 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.888 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.888 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.888 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:37.888 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.889 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.889 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.889 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.889 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.889 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.889 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.889 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.889 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.889 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.889 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.889 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.889 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.890 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.890 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.890 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:37.890 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:37.890 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.890 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.890 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.890 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.890 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.890 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.890 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.890 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.891 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.891 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.891 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.891 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.891 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.891 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.891 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:37.891 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "status", 2024-03-18T15:28:37.891 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": true, 2024-03-18T15:28:37.891 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "", 2024-03-18T15:28:37.891 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:37.892 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:37.892 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:37.892 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.892 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.892 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.892 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.892 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.892 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.892 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.892 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.892 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.892 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.893 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.893 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.893 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.893 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.893 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.893 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.893 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.893 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.893 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.893 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:37.893 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:37.893 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.894 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.894 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.894 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.894 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.894 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.894 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.894 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.894 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.894 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.894 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.894 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.895 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:37.895 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:37.895 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.895 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.895 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.895 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:37.895 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.895 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.895 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.895 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.895 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.896 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.896 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.896 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.896 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.896 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.896 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.896 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.896 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.896 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.896 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.896 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:37.896 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:37.897 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.897 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.897 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.897 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.897 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.897 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.897 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.897 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.897 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.897 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.897 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.897 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.898 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.898 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.898 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:37.898 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "telegraf", 2024-03-18T15:28:37.898 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": true, 2024-03-18T15:28:37.898 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "", 2024-03-18T15:28:37.898 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:37.898 INFO:teuthology.orchestra.run.smithi019.stdout: "address": { 2024-03-18T15:28:37.898 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "address", 2024-03-18T15:28:37.898 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.898 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.899 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.899 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-18T15:28:37.899 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.899 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.899 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.899 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.899 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.899 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.899 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.899 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.899 INFO:teuthology.orchestra.run.smithi019.stdout: "interval": { 2024-03-18T15:28:37.899 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "interval", 2024-03-18T15:28:37.900 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "secs", 2024-03-18T15:28:37.900 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.900 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.900 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "15", 2024-03-18T15:28:37.900 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.900 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.900 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.900 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.900 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.900 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.900 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.900 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.901 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:37.901 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:37.901 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.901 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.901 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.901 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.901 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.901 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.901 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.901 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.901 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.901 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.902 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.902 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.902 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.902 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.902 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.902 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.902 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.902 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.902 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.902 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:37.902 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:37.902 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.903 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.903 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.903 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.903 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.903 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.903 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.903 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.903 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.903 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.903 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.903 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.903 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:37.904 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:37.904 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.904 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.904 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.904 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:37.904 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.904 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.904 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.904 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.904 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.904 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.904 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.905 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.905 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.905 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.905 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.905 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.905 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.905 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.905 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.905 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:37.905 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:37.905 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.906 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.906 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.906 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.906 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.906 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.906 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.906 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.906 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.906 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.906 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.907 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.907 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.907 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.907 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:37.907 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "telemetry", 2024-03-18T15:28:37.907 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": true, 2024-03-18T15:28:37.907 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "", 2024-03-18T15:28:37.907 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:37.907 INFO:teuthology.orchestra.run.smithi019.stdout: "channel_basic": { 2024-03-18T15:28:37.907 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "channel_basic", 2024-03-18T15:28:37.907 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.907 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.907 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.908 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:37.908 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.908 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.908 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.908 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-18T15:28:37.908 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.908 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.908 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.908 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.908 INFO:teuthology.orchestra.run.smithi019.stdout: "channel_crash": { 2024-03-18T15:28:37.908 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "channel_crash", 2024-03-18T15:28:37.909 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.909 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.909 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.909 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:37.909 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.909 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.909 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.909 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-18T15:28:37.909 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.909 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.909 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.909 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.910 INFO:teuthology.orchestra.run.smithi019.stdout: "channel_device": { 2024-03-18T15:28:37.910 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "channel_device", 2024-03-18T15:28:37.910 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.910 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.910 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.910 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:37.910 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.910 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.910 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.910 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-18T15:28:37.910 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.910 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.911 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.911 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.911 INFO:teuthology.orchestra.run.smithi019.stdout: "channel_ident": { 2024-03-18T15:28:37.911 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "channel_ident", 2024-03-18T15:28:37.911 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.911 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.911 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.911 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.911 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.911 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.911 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.911 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-18T15:28:37.912 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.912 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.912 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.912 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.912 INFO:teuthology.orchestra.run.smithi019.stdout: "channel_perf": { 2024-03-18T15:28:37.912 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "channel_perf", 2024-03-18T15:28:37.912 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.912 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.912 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.912 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.912 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.912 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.913 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.913 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-18T15:28:37.913 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.913 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.913 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.913 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.913 INFO:teuthology.orchestra.run.smithi019.stdout: "contact": { 2024-03-18T15:28:37.913 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "contact", 2024-03-18T15:28:37.913 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.913 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.913 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.913 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.914 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.914 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.914 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.914 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.914 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.914 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.914 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.914 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.914 INFO:teuthology.orchestra.run.smithi019.stdout: "description": { 2024-03-18T15:28:37.914 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "description", 2024-03-18T15:28:37.914 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.914 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.915 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.915 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.915 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.915 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.915 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.915 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.915 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.915 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.915 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.915 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.915 INFO:teuthology.orchestra.run.smithi019.stdout: "device_url": { 2024-03-18T15:28:37.916 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "device_url", 2024-03-18T15:28:37.916 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.916 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.916 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.916 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-18T15:28:37.916 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.916 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.916 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.916 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.916 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.916 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.917 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.917 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.917 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled": { 2024-03-18T15:28:37.917 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "enabled", 2024-03-18T15:28:37.917 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.917 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.917 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.917 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.917 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.917 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.917 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.917 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.918 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.918 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.918 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.918 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.918 INFO:teuthology.orchestra.run.smithi019.stdout: "interval": { 2024-03-18T15:28:37.918 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "interval", 2024-03-18T15:28:37.918 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:37.918 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.918 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.918 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "24", 2024-03-18T15:28:37.918 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "8", 2024-03-18T15:28:37.919 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.919 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.919 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.919 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.919 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.919 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.919 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.919 INFO:teuthology.orchestra.run.smithi019.stdout: "last_opt_revision": { 2024-03-18T15:28:37.919 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "last_opt_revision", 2024-03-18T15:28:37.919 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:37.919 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.919 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.919 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "1", 2024-03-18T15:28:37.920 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.920 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.920 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.920 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.920 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.920 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.920 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.920 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.920 INFO:teuthology.orchestra.run.smithi019.stdout: "leaderboard": { 2024-03-18T15:28:37.920 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "leaderboard", 2024-03-18T15:28:37.920 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.920 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.921 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.921 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.921 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.921 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.921 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.921 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.921 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.921 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.921 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.921 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.921 INFO:teuthology.orchestra.run.smithi019.stdout: "leaderboard_description": { 2024-03-18T15:28:37.921 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "leaderboard_description", 2024-03-18T15:28:37.922 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.922 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.922 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.922 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.922 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.922 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.922 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.922 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.922 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.922 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.922 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.922 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.922 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:37.923 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:37.923 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.923 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.923 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.923 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.923 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.923 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.923 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.923 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.923 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.923 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.923 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.924 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.924 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.924 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.924 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.924 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.924 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.924 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.924 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.924 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:37.924 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:37.924 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.924 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.925 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.925 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.925 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.925 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.925 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.925 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.925 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.925 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.925 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.925 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.925 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:37.925 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:37.925 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.926 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.926 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.926 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:37.926 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.926 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.926 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.926 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.926 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.926 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.926 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.926 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.926 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.927 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.927 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.927 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.927 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.927 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.927 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.927 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:37.927 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:37.927 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.927 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.927 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.927 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.928 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.928 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.928 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.928 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.928 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.928 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.928 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.928 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.928 INFO:teuthology.orchestra.run.smithi019.stdout: "organization": { 2024-03-18T15:28:37.928 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "organization", 2024-03-18T15:28:37.928 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.928 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.929 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.929 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.929 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.929 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.929 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.929 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.929 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.929 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.929 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.929 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.929 INFO:teuthology.orchestra.run.smithi019.stdout: "proxy": { 2024-03-18T15:28:37.929 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "proxy", 2024-03-18T15:28:37.930 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.930 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.930 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.930 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.930 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.930 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.930 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.930 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.930 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.930 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.930 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.931 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.931 INFO:teuthology.orchestra.run.smithi019.stdout: "url": { 2024-03-18T15:28:37.931 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "url", 2024-03-18T15:28:37.931 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.931 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.931 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.931 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-18T15:28:37.931 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.931 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.931 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.931 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.931 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.932 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.932 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.932 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.932 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.932 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.932 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:37.932 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "test_orchestrator", 2024-03-18T15:28:37.932 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": true, 2024-03-18T15:28:37.932 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "", 2024-03-18T15:28:37.932 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:37.932 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:37.932 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:37.932 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.933 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.933 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.933 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.933 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.933 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.933 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.933 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.933 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.933 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.933 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.933 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.933 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.934 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.934 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.934 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.934 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.934 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.934 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.934 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:37.934 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:37.934 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.934 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.934 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.934 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.935 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.935 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.935 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.935 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.935 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.935 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.935 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.935 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.935 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:37.935 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:37.935 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.935 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.935 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.936 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:37.936 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.936 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.936 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.936 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.936 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.936 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.936 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.936 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.936 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.936 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.936 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.937 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.937 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.937 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.937 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.937 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:37.937 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:37.937 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.937 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.937 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.937 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.937 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.937 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.937 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.938 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.938 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.938 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.938 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.938 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.938 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.938 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.938 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:37.938 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "volumes", 2024-03-18T15:28:37.938 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": true, 2024-03-18T15:28:37.938 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "", 2024-03-18T15:28:37.938 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:37.939 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:37.939 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:37.939 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.939 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.939 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.939 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.939 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.939 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.939 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.939 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.939 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.939 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.940 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.940 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.940 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.940 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.940 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.940 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.940 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.940 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.940 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.940 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:37.940 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:37.940 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.941 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.941 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.941 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.941 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.941 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.941 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.941 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.941 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.941 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.941 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.941 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.941 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:37.942 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:37.942 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.942 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.942 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.942 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:37.942 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.942 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.942 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.942 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.942 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.942 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.942 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.943 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.943 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.943 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.943 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.943 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.943 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.943 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.943 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.943 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:37.943 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:37.943 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.943 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.944 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.944 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.944 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.944 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.944 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.944 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.944 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.944 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.944 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.944 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.944 INFO:teuthology.orchestra.run.smithi019.stdout: "max_concurrent_clones": { 2024-03-18T15:28:37.944 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "max_concurrent_clones", 2024-03-18T15:28:37.945 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:37.945 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.945 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.945 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "4", 2024-03-18T15:28:37.945 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.945 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.945 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.945 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-18T15:28:37.945 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.945 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.945 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.945 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.945 INFO:teuthology.orchestra.run.smithi019.stdout: "periodic_async_work": { 2024-03-18T15:28:37.946 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "periodic_async_work", 2024-03-18T15:28:37.946 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.946 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.946 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.946 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.946 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.946 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.946 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.946 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Periodically check for async work", 2024-03-18T15:28:37.946 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.946 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.946 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.947 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.947 INFO:teuthology.orchestra.run.smithi019.stdout: "snapshot_clone_delay": { 2024-03-18T15:28:37.947 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "snapshot_clone_delay", 2024-03-18T15:28:37.947 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:37.947 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.947 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.947 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "0", 2024-03-18T15:28:37.947 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.947 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.947 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.947 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-18T15:28:37.947 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.948 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.948 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.948 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.948 INFO:teuthology.orchestra.run.smithi019.stdout: "snapshot_clone_no_wait": { 2024-03-18T15:28:37.948 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "snapshot_clone_no_wait", 2024-03-18T15:28:37.948 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.948 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.948 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.948 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "True", 2024-03-18T15:28:37.948 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.948 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.948 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.949 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-18T15:28:37.949 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.949 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.949 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.949 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.949 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.949 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.949 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:37.949 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "zabbix", 2024-03-18T15:28:37.949 INFO:teuthology.orchestra.run.smithi019.stdout: "can_run": true, 2024-03-18T15:28:37.949 INFO:teuthology.orchestra.run.smithi019.stdout: "error_string": "", 2024-03-18T15:28:37.949 INFO:teuthology.orchestra.run.smithi019.stdout: "module_options": { 2024-03-18T15:28:37.949 INFO:teuthology.orchestra.run.smithi019.stdout: "discovery_interval": { 2024-03-18T15:28:37.950 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "discovery_interval", 2024-03-18T15:28:37.950 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "uint", 2024-03-18T15:28:37.950 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.950 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.950 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "100", 2024-03-18T15:28:37.950 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.950 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.950 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.950 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.950 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.950 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.950 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.951 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.951 INFO:teuthology.orchestra.run.smithi019.stdout: "identifier": { 2024-03-18T15:28:37.951 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "identifier", 2024-03-18T15:28:37.951 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.951 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.951 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.951 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.951 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.951 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.951 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.951 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.951 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.952 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.952 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.952 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.952 INFO:teuthology.orchestra.run.smithi019.stdout: "interval": { 2024-03-18T15:28:37.952 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "interval", 2024-03-18T15:28:37.952 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "secs", 2024-03-18T15:28:37.952 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.952 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.952 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "60", 2024-03-18T15:28:37.952 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.952 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.952 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.952 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.953 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.953 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.953 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.953 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.953 INFO:teuthology.orchestra.run.smithi019.stdout: "log_level": { 2024-03-18T15:28:37.953 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_level", 2024-03-18T15:28:37.953 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.953 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.953 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.953 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.953 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.953 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.953 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.954 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.954 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.954 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.954 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.954 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.954 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.954 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.954 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.954 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.954 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.954 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.954 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.955 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster": { 2024-03-18T15:28:37.955 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster", 2024-03-18T15:28:37.955 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.955 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.955 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.955 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.955 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.955 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.955 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.955 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.955 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.955 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.955 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.956 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.956 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_cluster_level": { 2024-03-18T15:28:37.956 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_cluster_level", 2024-03-18T15:28:37.956 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.956 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.956 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.956 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "info", 2024-03-18T15:28:37.956 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.956 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.956 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [ 2024-03-18T15:28:37.956 INFO:teuthology.orchestra.run.smithi019.stdout: "", 2024-03-18T15:28:37.956 INFO:teuthology.orchestra.run.smithi019.stdout: "critical", 2024-03-18T15:28:37.957 INFO:teuthology.orchestra.run.smithi019.stdout: "debug", 2024-03-18T15:28:37.957 INFO:teuthology.orchestra.run.smithi019.stdout: "error", 2024-03-18T15:28:37.957 INFO:teuthology.orchestra.run.smithi019.stdout: "info", 2024-03-18T15:28:37.957 INFO:teuthology.orchestra.run.smithi019.stdout: "warning" 2024-03-18T15:28:37.957 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.957 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.957 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.957 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.957 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.957 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.957 INFO:teuthology.orchestra.run.smithi019.stdout: "log_to_file": { 2024-03-18T15:28:37.957 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "log_to_file", 2024-03-18T15:28:37.957 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "bool", 2024-03-18T15:28:37.958 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.958 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 1, 2024-03-18T15:28:37.958 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "False", 2024-03-18T15:28:37.958 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.958 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.958 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.958 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.958 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.958 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.958 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.958 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.958 INFO:teuthology.orchestra.run.smithi019.stdout: "zabbix_host": { 2024-03-18T15:28:37.959 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "zabbix_host", 2024-03-18T15:28:37.959 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.959 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.959 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.959 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "", 2024-03-18T15:28:37.959 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.959 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.959 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.959 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.959 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.959 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.959 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.959 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.960 INFO:teuthology.orchestra.run.smithi019.stdout: "zabbix_port": { 2024-03-18T15:28:37.960 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "zabbix_port", 2024-03-18T15:28:37.960 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "int", 2024-03-18T15:28:37.960 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.960 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.960 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "10051", 2024-03-18T15:28:37.960 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.960 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.960 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.960 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.960 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.960 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.961 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.961 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.961 INFO:teuthology.orchestra.run.smithi019.stdout: "zabbix_sender": { 2024-03-18T15:28:37.961 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "zabbix_sender", 2024-03-18T15:28:37.961 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "str", 2024-03-18T15:28:37.961 INFO:teuthology.orchestra.run.smithi019.stdout: "level": "advanced", 2024-03-18T15:28:37.961 INFO:teuthology.orchestra.run.smithi019.stdout: "flags": 0, 2024-03-18T15:28:37.961 INFO:teuthology.orchestra.run.smithi019.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-18T15:28:37.961 INFO:teuthology.orchestra.run.smithi019.stdout: "min": "", 2024-03-18T15:28:37.961 INFO:teuthology.orchestra.run.smithi019.stdout: "max": "", 2024-03-18T15:28:37.961 INFO:teuthology.orchestra.run.smithi019.stdout: "enum_allowed": [], 2024-03-18T15:28:37.961 INFO:teuthology.orchestra.run.smithi019.stdout: "desc": "", 2024-03-18T15:28:37.962 INFO:teuthology.orchestra.run.smithi019.stdout: "long_desc": "", 2024-03-18T15:28:37.962 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-03-18T15:28:37.962 INFO:teuthology.orchestra.run.smithi019.stdout: "see_also": [] 2024-03-18T15:28:37.962 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.962 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.962 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.962 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.962 INFO:teuthology.orchestra.run.smithi019.stdout: "services": { 2024-03-18T15:28:37.962 INFO:teuthology.orchestra.run.smithi019.stdout: "dashboard": "https://172.21.15.19:8443/" 2024-03-18T15:28:37.962 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.962 INFO:teuthology.orchestra.run.smithi019.stdout: "always_on_modules": { 2024-03-18T15:28:37.962 INFO:teuthology.orchestra.run.smithi019.stdout: "octopus": [ 2024-03-18T15:28:37.963 INFO:teuthology.orchestra.run.smithi019.stdout: "balancer", 2024-03-18T15:28:37.963 INFO:teuthology.orchestra.run.smithi019.stdout: "crash", 2024-03-18T15:28:37.963 INFO:teuthology.orchestra.run.smithi019.stdout: "devicehealth", 2024-03-18T15:28:37.963 INFO:teuthology.orchestra.run.smithi019.stdout: "orchestrator", 2024-03-18T15:28:37.963 INFO:teuthology.orchestra.run.smithi019.stdout: "pg_autoscaler", 2024-03-18T15:28:37.963 INFO:teuthology.orchestra.run.smithi019.stdout: "progress", 2024-03-18T15:28:37.963 INFO:teuthology.orchestra.run.smithi019.stdout: "rbd_support", 2024-03-18T15:28:37.963 INFO:teuthology.orchestra.run.smithi019.stdout: "status", 2024-03-18T15:28:37.963 INFO:teuthology.orchestra.run.smithi019.stdout: "telemetry", 2024-03-18T15:28:37.963 INFO:teuthology.orchestra.run.smithi019.stdout: "volumes" 2024-03-18T15:28:37.963 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.963 INFO:teuthology.orchestra.run.smithi019.stdout: "pacific": [ 2024-03-18T15:28:37.964 INFO:teuthology.orchestra.run.smithi019.stdout: "balancer", 2024-03-18T15:28:37.964 INFO:teuthology.orchestra.run.smithi019.stdout: "crash", 2024-03-18T15:28:37.964 INFO:teuthology.orchestra.run.smithi019.stdout: "devicehealth", 2024-03-18T15:28:37.964 INFO:teuthology.orchestra.run.smithi019.stdout: "orchestrator", 2024-03-18T15:28:37.964 INFO:teuthology.orchestra.run.smithi019.stdout: "pg_autoscaler", 2024-03-18T15:28:37.964 INFO:teuthology.orchestra.run.smithi019.stdout: "progress", 2024-03-18T15:28:37.964 INFO:teuthology.orchestra.run.smithi019.stdout: "rbd_support", 2024-03-18T15:28:37.964 INFO:teuthology.orchestra.run.smithi019.stdout: "status", 2024-03-18T15:28:37.964 INFO:teuthology.orchestra.run.smithi019.stdout: "telemetry", 2024-03-18T15:28:37.964 INFO:teuthology.orchestra.run.smithi019.stdout: "volumes" 2024-03-18T15:28:37.964 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.964 INFO:teuthology.orchestra.run.smithi019.stdout: "quincy": [ 2024-03-18T15:28:37.965 INFO:teuthology.orchestra.run.smithi019.stdout: "balancer", 2024-03-18T15:28:37.965 INFO:teuthology.orchestra.run.smithi019.stdout: "crash", 2024-03-18T15:28:37.965 INFO:teuthology.orchestra.run.smithi019.stdout: "devicehealth", 2024-03-18T15:28:37.965 INFO:teuthology.orchestra.run.smithi019.stdout: "orchestrator", 2024-03-18T15:28:37.965 INFO:teuthology.orchestra.run.smithi019.stdout: "pg_autoscaler", 2024-03-18T15:28:37.965 INFO:teuthology.orchestra.run.smithi019.stdout: "progress", 2024-03-18T15:28:37.965 INFO:teuthology.orchestra.run.smithi019.stdout: "rbd_support", 2024-03-18T15:28:37.965 INFO:teuthology.orchestra.run.smithi019.stdout: "status", 2024-03-18T15:28:37.965 INFO:teuthology.orchestra.run.smithi019.stdout: "telemetry", 2024-03-18T15:28:37.965 INFO:teuthology.orchestra.run.smithi019.stdout: "volumes" 2024-03-18T15:28:37.966 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.966 INFO:teuthology.orchestra.run.smithi019.stdout: "reef": [ 2024-03-18T15:28:37.966 INFO:teuthology.orchestra.run.smithi019.stdout: "balancer", 2024-03-18T15:28:37.966 INFO:teuthology.orchestra.run.smithi019.stdout: "crash", 2024-03-18T15:28:37.966 INFO:teuthology.orchestra.run.smithi019.stdout: "devicehealth", 2024-03-18T15:28:37.966 INFO:teuthology.orchestra.run.smithi019.stdout: "orchestrator", 2024-03-18T15:28:37.966 INFO:teuthology.orchestra.run.smithi019.stdout: "pg_autoscaler", 2024-03-18T15:28:37.966 INFO:teuthology.orchestra.run.smithi019.stdout: "progress", 2024-03-18T15:28:37.966 INFO:teuthology.orchestra.run.smithi019.stdout: "rbd_support", 2024-03-18T15:28:37.966 INFO:teuthology.orchestra.run.smithi019.stdout: "status", 2024-03-18T15:28:37.966 INFO:teuthology.orchestra.run.smithi019.stdout: "telemetry", 2024-03-18T15:28:37.966 INFO:teuthology.orchestra.run.smithi019.stdout: "volumes" 2024-03-18T15:28:37.967 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-18T15:28:37.967 INFO:teuthology.orchestra.run.smithi019.stdout: "squid": [ 2024-03-18T15:28:37.967 INFO:teuthology.orchestra.run.smithi019.stdout: "balancer", 2024-03-18T15:28:37.967 INFO:teuthology.orchestra.run.smithi019.stdout: "crash", 2024-03-18T15:28:37.967 INFO:teuthology.orchestra.run.smithi019.stdout: "devicehealth", 2024-03-18T15:28:37.967 INFO:teuthology.orchestra.run.smithi019.stdout: "orchestrator", 2024-03-18T15:28:37.967 INFO:teuthology.orchestra.run.smithi019.stdout: "pg_autoscaler", 2024-03-18T15:28:37.967 INFO:teuthology.orchestra.run.smithi019.stdout: "progress", 2024-03-18T15:28:37.967 INFO:teuthology.orchestra.run.smithi019.stdout: "rbd_support", 2024-03-18T15:28:37.967 INFO:teuthology.orchestra.run.smithi019.stdout: "status", 2024-03-18T15:28:37.967 INFO:teuthology.orchestra.run.smithi019.stdout: "telemetry", 2024-03-18T15:28:37.967 INFO:teuthology.orchestra.run.smithi019.stdout: "volumes" 2024-03-18T15:28:37.968 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-03-18T15:28:37.968 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.968 INFO:teuthology.orchestra.run.smithi019.stdout: "last_failure_osd_epoch": 94, 2024-03-18T15:28:37.968 INFO:teuthology.orchestra.run.smithi019.stdout: "active_clients": [ 2024-03-18T15:28:37.968 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:37.968 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "devicehealth", 2024-03-18T15:28:37.968 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-03-18T15:28:37.968 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:37.968 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-03-18T15:28:37.968 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:0", 2024-03-18T15:28:37.968 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 3784419500 2024-03-18T15:28:37.968 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.969 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-03-18T15:28:37.969 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.969 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:37.969 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "libcephsqlite", 2024-03-18T15:28:37.969 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-03-18T15:28:37.969 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:37.969 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-03-18T15:28:37.969 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:0", 2024-03-18T15:28:37.969 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 1191755641 2024-03-18T15:28:37.969 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.969 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-03-18T15:28:37.969 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.970 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:37.970 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "rbd_support", 2024-03-18T15:28:37.970 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-03-18T15:28:37.970 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:37.970 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-03-18T15:28:37.970 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:0", 2024-03-18T15:28:37.970 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 303904666 2024-03-18T15:28:37.970 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.970 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-03-18T15:28:37.970 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.970 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:37.971 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "snap_schedule", 2024-03-18T15:28:37.971 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-03-18T15:28:37.971 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:37.971 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-03-18T15:28:37.971 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:0", 2024-03-18T15:28:37.971 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 694155752 2024-03-18T15:28:37.971 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.971 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-03-18T15:28:37.971 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.971 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:37.971 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "snap_schedule", 2024-03-18T15:28:37.971 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-03-18T15:28:37.972 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:37.972 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-03-18T15:28:37.972 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:0", 2024-03-18T15:28:37.972 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 2896728700 2024-03-18T15:28:37.972 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.972 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-03-18T15:28:37.972 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:37.972 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:37.972 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "volumes", 2024-03-18T15:28:37.972 INFO:teuthology.orchestra.run.smithi019.stdout: "addrvec": [ 2024-03-18T15:28:37.972 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-18T15:28:37.972 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v2", 2024-03-18T15:28:37.973 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "172.21.15.19:0", 2024-03-18T15:28:37.973 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 469770015 2024-03-18T15:28:37.973 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.973 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-03-18T15:28:37.973 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:37.973 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-03-18T15:28:37.973 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-03-18T15:28:37.975 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:28:36 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/81568607' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T15:28:37.975 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:28:36 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/1438829839' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T15:28:37.975 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:28:37 smithi104 ceph-mon[36070]: pgmap v315: 129 pgs: 129 active+clean; 1.7 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 0 B/s wr, 14 op/s 2024-03-18T15:28:37.975 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:28:37 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/2368939074' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-18T15:28:37.976 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:28:36 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/81568607' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T15:28:37.976 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:28:36 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/1438829839' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T15:28:37.976 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:28:37 smithi033 ceph-mon[36648]: pgmap v315: 129 pgs: 129 active+clean; 1.7 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 0 B/s wr, 14 op/s 2024-03-18T15:28:37.976 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:28:37 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/2368939074' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-18T15:28:37.976 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:36.599+0000 7f9c25000640 1 -- 172.21.15.19:0/2368939074 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f9c040a2970 msgr2=0x7f9c040a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:28:37.977 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:36.599+0000 7f9c25000640 1 --2- 172.21.15.19:0/2368939074 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f9c040a2970 0x7f9c040a4e30 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f9c20069430 tx=0x7f9c0c005f90 comp rx=0 tx=0).stop 2024-03-18T15:28:37.977 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:36.599+0000 7f9c25000640 1 -- 172.21.15.19:0/2368939074 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c20103990 msgr2=0x7f9c2019a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:28:37.977 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:36.599+0000 7f9c25000640 1 --2- 172.21.15.19:0/2368939074 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c20103990 0x7f9c2019a240 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f9c1000ebc0 tx=0x7f9c1000c6a0 comp rx=0 tx=0).stop 2024-03-18T15:28:37.977 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:36.599+0000 7f9c25000640 1 -- 172.21.15.19:0/2368939074 shutdown_connections 2024-03-18T15:28:37.977 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:36.599+0000 7f9c25000640 1 --2- 172.21.15.19:0/2368939074 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f9c040a2970 0x7f9c040a4e30 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:28:37.977 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:36.599+0000 7f9c25000640 1 --2- 172.21.15.19:0/2368939074 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9c201099a0 0x7f9c2019eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:28:37.977 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:36.599+0000 7f9c25000640 1 --2- 172.21.15.19:0/2368939074 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9c20104350 0x7f9c2019a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:28:37.977 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:36.599+0000 7f9c25000640 1 --2- 172.21.15.19:0/2368939074 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c20103990 0x7f9c2019a240 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:28:37.977 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:36.599+0000 7f9c25000640 1 -- 172.21.15.19:0/2368939074 >> 172.21.15.19:0/2368939074 conn(0x7f9c200ff460 msgr2=0x7f9c200ff9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:28:37.978 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:36.599+0000 7f9c25000640 1 -- 172.21.15.19:0/2368939074 shutdown_connections 2024-03-18T15:28:37.978 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:28:36.599+0000 7f9c25000640 1 -- 172.21.15.19:0/2368939074 wait complete. 2024-03-18T15:28:37.984 DEBUG:tasks.check_counter:Getting stats from a 2024-03-18T15:28:37.985 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 -- ceph daemon mds.a perf dump 2024-03-18T15:28:37.985 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:28:36 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/81568607' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T15:28:37.985 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:28:36 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/1438829839' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T15:28:37.985 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:28:37 smithi019 ceph-mon[31522]: pgmap v315: 129 pgs: 129 active+clean; 1.7 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 0 B/s wr, 14 op/s 2024-03-18T15:28:37.985 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:28:37 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/2368939074' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-18T15:28:38.506 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:28:39.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:28:39 smithi019 ceph-mon[31522]: pgmap v316: 129 pgs: 129 active+clean; 1.7 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-18T15:28:39.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:28:39 smithi033 ceph-mon[36648]: pgmap v316: 129 pgs: 129 active+clean; 1.7 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-18T15:28:39.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:28:39 smithi104 ceph-mon[36070]: pgmap v316: 129 pgs: 129 active+clean; 1.7 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-18T15:28:40.876 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-03-18T15:28:40.877 INFO:teuthology.orchestra.run.smithi019.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T15:28:40.877 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_recv_messages": 1031, 2024-03-18T15:28:40.877 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_send_messages": 1012, 2024-03-18T15:28:40.877 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_recv_bytes": 2037456, 2024-03-18T15:28:40.877 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_send_bytes": 384184, 2024-03-18T15:28:40.877 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_created_connections": 5, 2024-03-18T15:28:40.877 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_active_connections": 1, 2024-03-18T15:28:40.877 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_running_total_time": 0.308011449, 2024-03-18T15:28:40.877 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_running_send_time": 0.124937734, 2024-03-18T15:28:40.877 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_running_recv_time": 0.131437950, 2024-03-18T15:28:40.877 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_running_fast_dispatch_time": 0.056928099, 2024-03-18T15:28:40.877 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T15:28:40.878 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 1012, 2024-03-18T15:28:40.878 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.101691851, 2024-03-18T15:28:40.878 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000100486 2024-03-18T15:28:40.878 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:40.878 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_handle_ack_lat": { 2024-03-18T15:28:40.878 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 0, 2024-03-18T15:28:40.878 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.000000000, 2024-03-18T15:28:40.878 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000000000 2024-03-18T15:28:40.878 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:40.878 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_recv_encrypted_bytes": 1778064, 2024-03-18T15:28:40.878 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_send_encrypted_bytes": 138688 2024-03-18T15:28:40.878 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:40.878 INFO:teuthology.orchestra.run.smithi019.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T15:28:40.879 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_recv_messages": 1401, 2024-03-18T15:28:40.879 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_send_messages": 1567, 2024-03-18T15:28:40.879 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_recv_bytes": 358852, 2024-03-18T15:28:40.879 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_send_bytes": 897627, 2024-03-18T15:28:40.879 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_created_connections": 7, 2024-03-18T15:28:40.879 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_active_connections": 5, 2024-03-18T15:28:40.879 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_running_total_time": 0.273323507, 2024-03-18T15:28:40.879 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_running_send_time": 0.107675078, 2024-03-18T15:28:40.879 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_running_recv_time": 0.065135556, 2024-03-18T15:28:40.879 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_running_fast_dispatch_time": 0.075048852, 2024-03-18T15:28:40.879 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T15:28:40.879 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 1564, 2024-03-18T15:28:40.879 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.150421942, 2024-03-18T15:28:40.879 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000096177 2024-03-18T15:28:40.880 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:40.880 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_handle_ack_lat": { 2024-03-18T15:28:40.880 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 0, 2024-03-18T15:28:40.880 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.000000000, 2024-03-18T15:28:40.880 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000000000 2024-03-18T15:28:40.880 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:40.880 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-18T15:28:40.880 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_send_encrypted_bytes": 452688 2024-03-18T15:28:40.880 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:40.880 INFO:teuthology.orchestra.run.smithi019.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T15:28:40.880 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_recv_messages": 4, 2024-03-18T15:28:40.880 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_send_messages": 6, 2024-03-18T15:28:40.880 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_recv_bytes": 1629, 2024-03-18T15:28:40.880 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_send_bytes": 1284, 2024-03-18T15:28:40.880 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_created_connections": 5, 2024-03-18T15:28:40.881 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_active_connections": 4, 2024-03-18T15:28:40.881 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_running_total_time": 0.003425461, 2024-03-18T15:28:40.881 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_running_send_time": 0.000173835, 2024-03-18T15:28:40.881 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_running_recv_time": 0.002950451, 2024-03-18T15:28:40.881 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_running_fast_dispatch_time": 0.000097705, 2024-03-18T15:28:40.881 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T15:28:40.881 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 4, 2024-03-18T15:28:40.881 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.004451568, 2024-03-18T15:28:40.881 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.001112892 2024-03-18T15:28:40.881 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:40.881 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_handle_ack_lat": { 2024-03-18T15:28:40.881 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 0, 2024-03-18T15:28:40.881 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.000000000, 2024-03-18T15:28:40.881 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000000000 2024-03-18T15:28:40.881 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:40.882 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-18T15:28:40.882 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_send_encrypted_bytes": 0 2024-03-18T15:28:40.882 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:40.882 INFO:teuthology.orchestra.run.smithi019.stdout: "cct": { 2024-03-18T15:28:40.882 INFO:teuthology.orchestra.run.smithi019.stdout: "total_workers": 1, 2024-03-18T15:28:40.882 INFO:teuthology.orchestra.run.smithi019.stdout: "unhealthy_workers": 0 2024-03-18T15:28:40.882 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:40.882 INFO:teuthology.orchestra.run.smithi019.stdout: "finisher-MDSRank": { 2024-03-18T15:28:40.882 INFO:teuthology.orchestra.run.smithi019.stdout: "queue_len": 0, 2024-03-18T15:28:40.882 INFO:teuthology.orchestra.run.smithi019.stdout: "complete_latency": { 2024-03-18T15:28:40.882 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 3717, 2024-03-18T15:28:40.882 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.195347400, 2024-03-18T15:28:40.882 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000052555 2024-03-18T15:28:40.882 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:40.883 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:40.883 INFO:teuthology.orchestra.run.smithi019.stdout: "finisher-PurgeQueue": { 2024-03-18T15:28:40.883 INFO:teuthology.orchestra.run.smithi019.stdout: "queue_len": 0, 2024-03-18T15:28:40.883 INFO:teuthology.orchestra.run.smithi019.stdout: "complete_latency": { 2024-03-18T15:28:40.883 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 0, 2024-03-18T15:28:40.883 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.000000000, 2024-03-18T15:28:40.883 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000000000 2024-03-18T15:28:40.883 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:40.883 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:40.883 INFO:teuthology.orchestra.run.smithi019.stdout: "mds": { 2024-03-18T15:28:40.883 INFO:teuthology.orchestra.run.smithi019.stdout: "request": 0, 2024-03-18T15:28:40.883 INFO:teuthology.orchestra.run.smithi019.stdout: "reply": 0, 2024-03-18T15:28:40.883 INFO:teuthology.orchestra.run.smithi019.stdout: "reply_latency": { 2024-03-18T15:28:40.883 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 0, 2024-03-18T15:28:40.884 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.000000000, 2024-03-18T15:28:40.884 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000000000 2024-03-18T15:28:40.884 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:40.884 INFO:teuthology.orchestra.run.smithi019.stdout: "slow_reply": 0, 2024-03-18T15:28:40.884 INFO:teuthology.orchestra.run.smithi019.stdout: "forward": 0, 2024-03-18T15:28:40.884 INFO:teuthology.orchestra.run.smithi019.stdout: "dir_fetch_complete": 0, 2024-03-18T15:28:40.884 INFO:teuthology.orchestra.run.smithi019.stdout: "dir_fetch_keys": 0, 2024-03-18T15:28:40.884 INFO:teuthology.orchestra.run.smithi019.stdout: "dir_commit": 0, 2024-03-18T15:28:40.884 INFO:teuthology.orchestra.run.smithi019.stdout: "dir_split": 0, 2024-03-18T15:28:40.884 INFO:teuthology.orchestra.run.smithi019.stdout: "dir_merge": 0, 2024-03-18T15:28:40.884 INFO:teuthology.orchestra.run.smithi019.stdout: "inodes": 0, 2024-03-18T15:28:40.884 INFO:teuthology.orchestra.run.smithi019.stdout: "inodes_top": 0, 2024-03-18T15:28:40.884 INFO:teuthology.orchestra.run.smithi019.stdout: "inodes_bottom": 0, 2024-03-18T15:28:40.884 INFO:teuthology.orchestra.run.smithi019.stdout: "inodes_pin_tail": 0, 2024-03-18T15:28:40.884 INFO:teuthology.orchestra.run.smithi019.stdout: "inodes_pinned": 0, 2024-03-18T15:28:40.885 INFO:teuthology.orchestra.run.smithi019.stdout: "inodes_expired": 0, 2024-03-18T15:28:40.885 INFO:teuthology.orchestra.run.smithi019.stdout: "inodes_with_caps": 0, 2024-03-18T15:28:40.885 INFO:teuthology.orchestra.run.smithi019.stdout: "caps": 0, 2024-03-18T15:28:40.885 INFO:teuthology.orchestra.run.smithi019.stdout: "subtrees": 1, 2024-03-18T15:28:40.885 INFO:teuthology.orchestra.run.smithi019.stdout: "traverse": 0, 2024-03-18T15:28:40.885 INFO:teuthology.orchestra.run.smithi019.stdout: "traverse_hit": 0, 2024-03-18T15:28:40.885 INFO:teuthology.orchestra.run.smithi019.stdout: "traverse_forward": 0, 2024-03-18T15:28:40.885 INFO:teuthology.orchestra.run.smithi019.stdout: "traverse_discover": 0, 2024-03-18T15:28:40.885 INFO:teuthology.orchestra.run.smithi019.stdout: "traverse_dir_fetch": 0, 2024-03-18T15:28:40.885 INFO:teuthology.orchestra.run.smithi019.stdout: "traverse_remote_ino": 0, 2024-03-18T15:28:40.885 INFO:teuthology.orchestra.run.smithi019.stdout: "traverse_lock": 0, 2024-03-18T15:28:40.885 INFO:teuthology.orchestra.run.smithi019.stdout: "load_cent": 0, 2024-03-18T15:28:40.885 INFO:teuthology.orchestra.run.smithi019.stdout: "q": 0, 2024-03-18T15:28:40.885 INFO:teuthology.orchestra.run.smithi019.stdout: "exported": 0, 2024-03-18T15:28:40.886 INFO:teuthology.orchestra.run.smithi019.stdout: "exported_inodes": 0, 2024-03-18T15:28:40.886 INFO:teuthology.orchestra.run.smithi019.stdout: "imported": 0, 2024-03-18T15:28:40.886 INFO:teuthology.orchestra.run.smithi019.stdout: "imported_inodes": 0, 2024-03-18T15:28:40.886 INFO:teuthology.orchestra.run.smithi019.stdout: "openino_dir_fetch": 0, 2024-03-18T15:28:40.886 INFO:teuthology.orchestra.run.smithi019.stdout: "openino_backtrace_fetch": 0, 2024-03-18T15:28:40.886 INFO:teuthology.orchestra.run.smithi019.stdout: "openino_peer_discover": 0, 2024-03-18T15:28:40.886 INFO:teuthology.orchestra.run.smithi019.stdout: "root_rfiles": 0, 2024-03-18T15:28:40.886 INFO:teuthology.orchestra.run.smithi019.stdout: "root_rbytes": 0, 2024-03-18T15:28:40.886 INFO:teuthology.orchestra.run.smithi019.stdout: "root_rsnaps": 0, 2024-03-18T15:28:40.886 INFO:teuthology.orchestra.run.smithi019.stdout: "scrub_backtrace_fetch": 0, 2024-03-18T15:28:40.886 INFO:teuthology.orchestra.run.smithi019.stdout: "scrub_set_tag": 0, 2024-03-18T15:28:40.886 INFO:teuthology.orchestra.run.smithi019.stdout: "scrub_backtrace_repaired": 0, 2024-03-18T15:28:40.886 INFO:teuthology.orchestra.run.smithi019.stdout: "scrub_inotable_repaired": 0, 2024-03-18T15:28:40.886 INFO:teuthology.orchestra.run.smithi019.stdout: "scrub_dir_inodes": 0, 2024-03-18T15:28:40.886 INFO:teuthology.orchestra.run.smithi019.stdout: "scrub_dir_base_inodes": 0, 2024-03-18T15:28:40.887 INFO:teuthology.orchestra.run.smithi019.stdout: "scrub_dirfrag_rstats": 0, 2024-03-18T15:28:40.887 INFO:teuthology.orchestra.run.smithi019.stdout: "scrub_file_inodes": 0, 2024-03-18T15:28:40.887 INFO:teuthology.orchestra.run.smithi019.stdout: "handle_inode_file_caps": 0, 2024-03-18T15:28:40.887 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph_cap_op_revoke": 0, 2024-03-18T15:28:40.887 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph_cap_op_grant": 0, 2024-03-18T15:28:40.887 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph_cap_op_trunc": 0, 2024-03-18T15:28:40.887 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-18T15:28:40.887 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-18T15:28:40.887 INFO:teuthology.orchestra.run.smithi019.stdout: "handle_client_caps": 0, 2024-03-18T15:28:40.887 INFO:teuthology.orchestra.run.smithi019.stdout: "handle_client_caps_dirty": 0, 2024-03-18T15:28:40.887 INFO:teuthology.orchestra.run.smithi019.stdout: "handle_client_cap_release": 0, 2024-03-18T15:28:40.887 INFO:teuthology.orchestra.run.smithi019.stdout: "process_request_cap_release": 0 2024-03-18T15:28:40.887 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:40.887 INFO:teuthology.orchestra.run.smithi019.stdout: "mds_cache": { 2024-03-18T15:28:40.887 INFO:teuthology.orchestra.run.smithi019.stdout: "dir_update": 0, 2024-03-18T15:28:40.888 INFO:teuthology.orchestra.run.smithi019.stdout: "dir_update_receipt": 0, 2024-03-18T15:28:40.888 INFO:teuthology.orchestra.run.smithi019.stdout: "dir_try_discover": 0, 2024-03-18T15:28:40.888 INFO:teuthology.orchestra.run.smithi019.stdout: "dir_send_discover": 0, 2024-03-18T15:28:40.888 INFO:teuthology.orchestra.run.smithi019.stdout: "dir_handle_discover": 0, 2024-03-18T15:28:40.888 INFO:teuthology.orchestra.run.smithi019.stdout: "num_strays": 0, 2024-03-18T15:28:40.888 INFO:teuthology.orchestra.run.smithi019.stdout: "num_strays_delayed": 0, 2024-03-18T15:28:40.888 INFO:teuthology.orchestra.run.smithi019.stdout: "num_strays_enqueuing": 0, 2024-03-18T15:28:40.888 INFO:teuthology.orchestra.run.smithi019.stdout: "strays_created": 0, 2024-03-18T15:28:40.888 INFO:teuthology.orchestra.run.smithi019.stdout: "strays_enqueued": 0, 2024-03-18T15:28:40.888 INFO:teuthology.orchestra.run.smithi019.stdout: "strays_reintegrated": 0, 2024-03-18T15:28:40.888 INFO:teuthology.orchestra.run.smithi019.stdout: "strays_migrated": 0, 2024-03-18T15:28:40.888 INFO:teuthology.orchestra.run.smithi019.stdout: "num_recovering_processing": 0, 2024-03-18T15:28:40.888 INFO:teuthology.orchestra.run.smithi019.stdout: "num_recovering_enqueued": 0, 2024-03-18T15:28:40.888 INFO:teuthology.orchestra.run.smithi019.stdout: "num_recovering_prioritized": 0, 2024-03-18T15:28:40.888 INFO:teuthology.orchestra.run.smithi019.stdout: "recovery_started": 0, 2024-03-18T15:28:40.888 INFO:teuthology.orchestra.run.smithi019.stdout: "recovery_completed": 0, 2024-03-18T15:28:40.889 INFO:teuthology.orchestra.run.smithi019.stdout: "ireq_quiesce_path": 0, 2024-03-18T15:28:40.889 INFO:teuthology.orchestra.run.smithi019.stdout: "ireq_quiesce_inode": 0, 2024-03-18T15:28:40.889 INFO:teuthology.orchestra.run.smithi019.stdout: "ireq_enqueue_scrub": 0, 2024-03-18T15:28:40.889 INFO:teuthology.orchestra.run.smithi019.stdout: "ireq_exportdir": 0, 2024-03-18T15:28:40.889 INFO:teuthology.orchestra.run.smithi019.stdout: "ireq_flush": 0, 2024-03-18T15:28:40.889 INFO:teuthology.orchestra.run.smithi019.stdout: "ireq_fragmentdir": 0, 2024-03-18T15:28:40.889 INFO:teuthology.orchestra.run.smithi019.stdout: "ireq_fragstats": 0, 2024-03-18T15:28:40.889 INFO:teuthology.orchestra.run.smithi019.stdout: "ireq_inodestats": 0 2024-03-18T15:28:40.889 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:40.889 INFO:teuthology.orchestra.run.smithi019.stdout: "mds_log": { 2024-03-18T15:28:40.889 INFO:teuthology.orchestra.run.smithi019.stdout: "evlrg": 0, 2024-03-18T15:28:40.889 INFO:teuthology.orchestra.run.smithi019.stdout: "evadd": 0, 2024-03-18T15:28:40.889 INFO:teuthology.orchestra.run.smithi019.stdout: "evex": 0, 2024-03-18T15:28:40.889 INFO:teuthology.orchestra.run.smithi019.stdout: "evtrm": 0, 2024-03-18T15:28:40.889 INFO:teuthology.orchestra.run.smithi019.stdout: "ev": 7, 2024-03-18T15:28:40.890 INFO:teuthology.orchestra.run.smithi019.stdout: "evexg": 0, 2024-03-18T15:28:40.890 INFO:teuthology.orchestra.run.smithi019.stdout: "evexd": 0, 2024-03-18T15:28:40.890 INFO:teuthology.orchestra.run.smithi019.stdout: "segadd": 0, 2024-03-18T15:28:40.890 INFO:teuthology.orchestra.run.smithi019.stdout: "segex": 0, 2024-03-18T15:28:40.890 INFO:teuthology.orchestra.run.smithi019.stdout: "segtrm": 0, 2024-03-18T15:28:40.890 INFO:teuthology.orchestra.run.smithi019.stdout: "seg": 2, 2024-03-18T15:28:40.890 INFO:teuthology.orchestra.run.smithi019.stdout: "segmjr": 2, 2024-03-18T15:28:40.890 INFO:teuthology.orchestra.run.smithi019.stdout: "segexg": 0, 2024-03-18T15:28:40.890 INFO:teuthology.orchestra.run.smithi019.stdout: "segexd": 0, 2024-03-18T15:28:40.890 INFO:teuthology.orchestra.run.smithi019.stdout: "expos": 4194304, 2024-03-18T15:28:40.890 INFO:teuthology.orchestra.run.smithi019.stdout: "wrpos": 4196266, 2024-03-18T15:28:40.890 INFO:teuthology.orchestra.run.smithi019.stdout: "rdpos": 4196125, 2024-03-18T15:28:40.890 INFO:teuthology.orchestra.run.smithi019.stdout: "jlat": { 2024-03-18T15:28:40.890 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 0, 2024-03-18T15:28:40.890 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.000000000, 2024-03-18T15:28:40.891 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000000000 2024-03-18T15:28:40.891 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:40.891 INFO:teuthology.orchestra.run.smithi019.stdout: "replayed": 7 2024-03-18T15:28:40.891 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:40.891 INFO:teuthology.orchestra.run.smithi019.stdout: "mds_mem": { 2024-03-18T15:28:40.891 INFO:teuthology.orchestra.run.smithi019.stdout: "ino": 3, 2024-03-18T15:28:40.891 INFO:teuthology.orchestra.run.smithi019.stdout: "ino+": 3, 2024-03-18T15:28:40.891 INFO:teuthology.orchestra.run.smithi019.stdout: "ino-": 0, 2024-03-18T15:28:40.891 INFO:teuthology.orchestra.run.smithi019.stdout: "dir": 1, 2024-03-18T15:28:40.891 INFO:teuthology.orchestra.run.smithi019.stdout: "dir+": 1, 2024-03-18T15:28:40.891 INFO:teuthology.orchestra.run.smithi019.stdout: "dir-": 0, 2024-03-18T15:28:40.891 INFO:teuthology.orchestra.run.smithi019.stdout: "dn": 0, 2024-03-18T15:28:40.891 INFO:teuthology.orchestra.run.smithi019.stdout: "dn+": 0, 2024-03-18T15:28:40.891 INFO:teuthology.orchestra.run.smithi019.stdout: "dn-": 0, 2024-03-18T15:28:40.891 INFO:teuthology.orchestra.run.smithi019.stdout: "cap": 0, 2024-03-18T15:28:40.891 INFO:teuthology.orchestra.run.smithi019.stdout: "cap+": 0, 2024-03-18T15:28:40.892 INFO:teuthology.orchestra.run.smithi019.stdout: "cap-": 0, 2024-03-18T15:28:40.892 INFO:teuthology.orchestra.run.smithi019.stdout: "rss": 48868, 2024-03-18T15:28:40.892 INFO:teuthology.orchestra.run.smithi019.stdout: "heap": 215420 2024-03-18T15:28:40.892 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:40.892 INFO:teuthology.orchestra.run.smithi019.stdout: "mds_server": { 2024-03-18T15:28:40.892 INFO:teuthology.orchestra.run.smithi019.stdout: "dispatch_client_request": 0, 2024-03-18T15:28:40.892 INFO:teuthology.orchestra.run.smithi019.stdout: "dispatch_server_request": 0, 2024-03-18T15:28:40.892 INFO:teuthology.orchestra.run.smithi019.stdout: "handle_client_request": 0, 2024-03-18T15:28:40.892 INFO:teuthology.orchestra.run.smithi019.stdout: "handle_client_session": 0, 2024-03-18T15:28:40.892 INFO:teuthology.orchestra.run.smithi019.stdout: "handle_peer_request": 0, 2024-03-18T15:28:40.892 INFO:teuthology.orchestra.run.smithi019.stdout: "req_create_latency": { 2024-03-18T15:28:40.892 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 0, 2024-03-18T15:28:40.892 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.000000000, 2024-03-18T15:28:40.892 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000000000 2024-03-18T15:28:40.892 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:40.893 INFO:teuthology.orchestra.run.smithi019.stdout: "req_getattr_latency": { 2024-03-18T15:28:40.893 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 0, 2024-03-18T15:28:40.893 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.000000000, 2024-03-18T15:28:40.893 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000000000 2024-03-18T15:28:40.893 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:40.893 INFO:teuthology.orchestra.run.smithi019.stdout: "req_getfilelock_latency": { 2024-03-18T15:28:40.893 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 0, 2024-03-18T15:28:40.893 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.000000000, 2024-03-18T15:28:40.893 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000000000 2024-03-18T15:28:40.893 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:40.893 INFO:teuthology.orchestra.run.smithi019.stdout: "req_link_latency": { 2024-03-18T15:28:40.893 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 0, 2024-03-18T15:28:40.893 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.000000000, 2024-03-18T15:28:40.893 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000000000 2024-03-18T15:28:40.893 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:40.894 INFO:teuthology.orchestra.run.smithi019.stdout: "req_lookup_latency": { 2024-03-18T15:28:40.894 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 0, 2024-03-18T15:28:40.894 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.000000000, 2024-03-18T15:28:40.894 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000000000 2024-03-18T15:28:40.894 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:40.894 INFO:teuthology.orchestra.run.smithi019.stdout: "req_lookuphash_latency": { 2024-03-18T15:28:40.894 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 0, 2024-03-18T15:28:40.894 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.000000000, 2024-03-18T15:28:40.894 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000000000 2024-03-18T15:28:40.894 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:40.894 INFO:teuthology.orchestra.run.smithi019.stdout: "req_lookupino_latency": { 2024-03-18T15:28:40.894 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 0, 2024-03-18T15:28:40.894 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.000000000, 2024-03-18T15:28:40.894 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000000000 2024-03-18T15:28:40.895 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:40.895 INFO:teuthology.orchestra.run.smithi019.stdout: "req_lookupname_latency": { 2024-03-18T15:28:40.895 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 0, 2024-03-18T15:28:40.895 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.000000000, 2024-03-18T15:28:40.895 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000000000 2024-03-18T15:28:40.895 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:40.895 INFO:teuthology.orchestra.run.smithi019.stdout: "req_lookupparent_latency": { 2024-03-18T15:28:40.895 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 0, 2024-03-18T15:28:40.895 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.000000000, 2024-03-18T15:28:40.895 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000000000 2024-03-18T15:28:40.895 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:40.895 INFO:teuthology.orchestra.run.smithi019.stdout: "req_lookupsnap_latency": { 2024-03-18T15:28:40.895 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 0, 2024-03-18T15:28:40.895 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.000000000, 2024-03-18T15:28:40.895 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000000000 2024-03-18T15:28:40.896 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:40.896 INFO:teuthology.orchestra.run.smithi019.stdout: "req_lssnap_latency": { 2024-03-18T15:28:40.896 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 0, 2024-03-18T15:28:40.896 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.000000000, 2024-03-18T15:28:40.896 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000000000 2024-03-18T15:28:40.896 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:40.896 INFO:teuthology.orchestra.run.smithi019.stdout: "req_mkdir_latency": { 2024-03-18T15:28:40.896 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 0, 2024-03-18T15:28:40.896 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.000000000, 2024-03-18T15:28:40.896 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000000000 2024-03-18T15:28:40.896 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:40.896 INFO:teuthology.orchestra.run.smithi019.stdout: "req_mknod_latency": { 2024-03-18T15:28:40.896 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 0, 2024-03-18T15:28:40.896 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.000000000, 2024-03-18T15:28:40.896 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000000000 2024-03-18T15:28:40.897 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:40.897 INFO:teuthology.orchestra.run.smithi019.stdout: "req_mksnap_latency": { 2024-03-18T15:28:40.897 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 0, 2024-03-18T15:28:40.897 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.000000000, 2024-03-18T15:28:40.897 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000000000 2024-03-18T15:28:40.897 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:40.897 INFO:teuthology.orchestra.run.smithi019.stdout: "req_open_latency": { 2024-03-18T15:28:40.897 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 0, 2024-03-18T15:28:40.897 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.000000000, 2024-03-18T15:28:40.897 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000000000 2024-03-18T15:28:40.897 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:40.897 INFO:teuthology.orchestra.run.smithi019.stdout: "req_readdir_latency": { 2024-03-18T15:28:40.897 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 0, 2024-03-18T15:28:40.897 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.000000000, 2024-03-18T15:28:40.897 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000000000 2024-03-18T15:28:40.898 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:40.898 INFO:teuthology.orchestra.run.smithi019.stdout: "req_rename_latency": { 2024-03-18T15:28:40.898 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 0, 2024-03-18T15:28:40.898 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.000000000, 2024-03-18T15:28:40.898 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000000000 2024-03-18T15:28:40.898 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:40.898 INFO:teuthology.orchestra.run.smithi019.stdout: "req_renamesnap_latency": { 2024-03-18T15:28:40.898 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 0, 2024-03-18T15:28:40.898 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.000000000, 2024-03-18T15:28:40.898 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000000000 2024-03-18T15:28:40.898 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:40.898 INFO:teuthology.orchestra.run.smithi019.stdout: "req_snapdiff_latency": { 2024-03-18T15:28:40.898 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 0, 2024-03-18T15:28:40.898 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.000000000, 2024-03-18T15:28:40.899 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000000000 2024-03-18T15:28:40.899 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:40.899 INFO:teuthology.orchestra.run.smithi019.stdout: "req_rmdir_latency": { 2024-03-18T15:28:40.899 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 0, 2024-03-18T15:28:40.899 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.000000000, 2024-03-18T15:28:40.899 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000000000 2024-03-18T15:28:40.899 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:40.899 INFO:teuthology.orchestra.run.smithi019.stdout: "req_rmsnap_latency": { 2024-03-18T15:28:40.899 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 0, 2024-03-18T15:28:40.899 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.000000000, 2024-03-18T15:28:40.899 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000000000 2024-03-18T15:28:40.899 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:40.899 INFO:teuthology.orchestra.run.smithi019.stdout: "req_rmxattr_latency": { 2024-03-18T15:28:40.899 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 0, 2024-03-18T15:28:40.899 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.000000000, 2024-03-18T15:28:40.900 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000000000 2024-03-18T15:28:40.900 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:40.900 INFO:teuthology.orchestra.run.smithi019.stdout: "req_setattr_latency": { 2024-03-18T15:28:40.900 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 0, 2024-03-18T15:28:40.900 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.000000000, 2024-03-18T15:28:40.900 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000000000 2024-03-18T15:28:40.900 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:40.900 INFO:teuthology.orchestra.run.smithi019.stdout: "req_setdirlayout_latency": { 2024-03-18T15:28:40.900 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 0, 2024-03-18T15:28:40.900 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.000000000, 2024-03-18T15:28:40.900 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000000000 2024-03-18T15:28:40.900 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:40.900 INFO:teuthology.orchestra.run.smithi019.stdout: "req_setfilelock_latency": { 2024-03-18T15:28:40.900 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 0, 2024-03-18T15:28:40.900 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.000000000, 2024-03-18T15:28:40.901 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000000000 2024-03-18T15:28:40.901 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:40.901 INFO:teuthology.orchestra.run.smithi019.stdout: "req_setlayout_latency": { 2024-03-18T15:28:40.901 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 0, 2024-03-18T15:28:40.901 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.000000000, 2024-03-18T15:28:40.901 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000000000 2024-03-18T15:28:40.901 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:40.901 INFO:teuthology.orchestra.run.smithi019.stdout: "req_setxattr_latency": { 2024-03-18T15:28:40.901 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 0, 2024-03-18T15:28:40.901 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.000000000, 2024-03-18T15:28:40.901 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000000000 2024-03-18T15:28:40.901 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:40.901 INFO:teuthology.orchestra.run.smithi019.stdout: "req_symlink_latency": { 2024-03-18T15:28:40.901 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 0, 2024-03-18T15:28:40.901 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.000000000, 2024-03-18T15:28:40.902 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000000000 2024-03-18T15:28:40.902 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:40.902 INFO:teuthology.orchestra.run.smithi019.stdout: "req_unlink_latency": { 2024-03-18T15:28:40.902 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 0, 2024-03-18T15:28:40.902 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.000000000, 2024-03-18T15:28:40.902 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000000000 2024-03-18T15:28:40.902 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:40.902 INFO:teuthology.orchestra.run.smithi019.stdout: "cap_revoke_eviction": 0, 2024-03-18T15:28:40.902 INFO:teuthology.orchestra.run.smithi019.stdout: "cap_acquisition_throttle": 0, 2024-03-18T15:28:40.902 INFO:teuthology.orchestra.run.smithi019.stdout: "req_getvxattr_latency": { 2024-03-18T15:28:40.902 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 0, 2024-03-18T15:28:40.902 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.000000000, 2024-03-18T15:28:40.902 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000000000 2024-03-18T15:28:40.902 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:40.903 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:40.903 INFO:teuthology.orchestra.run.smithi019.stdout: "mds_sessions": { 2024-03-18T15:28:40.903 INFO:teuthology.orchestra.run.smithi019.stdout: "session_count": 1, 2024-03-18T15:28:40.903 INFO:teuthology.orchestra.run.smithi019.stdout: "session_add": 3, 2024-03-18T15:28:40.903 INFO:teuthology.orchestra.run.smithi019.stdout: "session_remove": 2, 2024-03-18T15:28:40.903 INFO:teuthology.orchestra.run.smithi019.stdout: "sessions_open": 3, 2024-03-18T15:28:40.903 INFO:teuthology.orchestra.run.smithi019.stdout: "sessions_stale": 0, 2024-03-18T15:28:40.903 INFO:teuthology.orchestra.run.smithi019.stdout: "total_load": 0, 2024-03-18T15:28:40.903 INFO:teuthology.orchestra.run.smithi019.stdout: "average_load": 0, 2024-03-18T15:28:40.903 INFO:teuthology.orchestra.run.smithi019.stdout: "avg_session_uptime": 4799, 2024-03-18T15:28:40.903 INFO:teuthology.orchestra.run.smithi019.stdout: "mdthresh_evicted": 0 2024-03-18T15:28:40.903 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:40.903 INFO:teuthology.orchestra.run.smithi019.stdout: "mempool": { 2024-03-18T15:28:40.903 INFO:teuthology.orchestra.run.smithi019.stdout: "bloom_filter_bytes": 0, 2024-03-18T15:28:40.904 INFO:teuthology.orchestra.run.smithi019.stdout: "bloom_filter_items": 0, 2024-03-18T15:28:40.904 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T15:28:40.904 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_alloc_items": 0, 2024-03-18T15:28:40.904 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T15:28:40.904 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_cache_data_items": 0, 2024-03-18T15:28:40.904 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T15:28:40.904 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T15:28:40.904 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T15:28:40.904 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T15:28:40.904 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T15:28:40.904 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_cache_other_items": 0, 2024-03-18T15:28:40.904 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T15:28:40.904 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T15:28:40.904 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_extent_bytes": 0, 2024-03-18T15:28:40.904 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_extent_items": 0, 2024-03-18T15:28:40.904 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_blob_bytes": 0, 2024-03-18T15:28:40.905 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_blob_items": 0, 2024-03-18T15:28:40.905 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T15:28:40.905 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T15:28:40.905 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T15:28:40.905 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T15:28:40.905 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T15:28:40.905 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_fsck_items": 0, 2024-03-18T15:28:40.905 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_txc_bytes": 0, 2024-03-18T15:28:40.905 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_txc_items": 0, 2024-03-18T15:28:40.905 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T15:28:40.905 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T15:28:40.905 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_writing_bytes": 0, 2024-03-18T15:28:40.905 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_writing_items": 0, 2024-03-18T15:28:40.905 INFO:teuthology.orchestra.run.smithi019.stdout: "bluefs_bytes": 0, 2024-03-18T15:28:40.905 INFO:teuthology.orchestra.run.smithi019.stdout: "bluefs_items": 0, 2024-03-18T15:28:40.906 INFO:teuthology.orchestra.run.smithi019.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T15:28:40.906 INFO:teuthology.orchestra.run.smithi019.stdout: "bluefs_file_reader_items": 0, 2024-03-18T15:28:40.906 INFO:teuthology.orchestra.run.smithi019.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T15:28:40.906 INFO:teuthology.orchestra.run.smithi019.stdout: "bluefs_file_writer_items": 0, 2024-03-18T15:28:40.906 INFO:teuthology.orchestra.run.smithi019.stdout: "buffer_anon_bytes": 199272, 2024-03-18T15:28:40.906 INFO:teuthology.orchestra.run.smithi019.stdout: "buffer_anon_items": 57, 2024-03-18T15:28:40.906 INFO:teuthology.orchestra.run.smithi019.stdout: "buffer_meta_bytes": 0, 2024-03-18T15:28:40.906 INFO:teuthology.orchestra.run.smithi019.stdout: "buffer_meta_items": 0, 2024-03-18T15:28:40.906 INFO:teuthology.orchestra.run.smithi019.stdout: "osd_bytes": 0, 2024-03-18T15:28:40.906 INFO:teuthology.orchestra.run.smithi019.stdout: "osd_items": 0, 2024-03-18T15:28:40.906 INFO:teuthology.orchestra.run.smithi019.stdout: "osd_mapbl_bytes": 0, 2024-03-18T15:28:40.906 INFO:teuthology.orchestra.run.smithi019.stdout: "osd_mapbl_items": 0, 2024-03-18T15:28:40.906 INFO:teuthology.orchestra.run.smithi019.stdout: "osd_pglog_bytes": 0, 2024-03-18T15:28:40.906 INFO:teuthology.orchestra.run.smithi019.stdout: "osd_pglog_items": 0, 2024-03-18T15:28:40.907 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap_bytes": 9928, 2024-03-18T15:28:40.907 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap_items": 207, 2024-03-18T15:28:40.907 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T15:28:40.907 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap_mapping_items": 0, 2024-03-18T15:28:40.907 INFO:teuthology.orchestra.run.smithi019.stdout: "pgmap_bytes": 0, 2024-03-18T15:28:40.907 INFO:teuthology.orchestra.run.smithi019.stdout: "pgmap_items": 0, 2024-03-18T15:28:40.907 INFO:teuthology.orchestra.run.smithi019.stdout: "mds_co_bytes": 8464, 2024-03-18T15:28:40.907 INFO:teuthology.orchestra.run.smithi019.stdout: "mds_co_items": 13, 2024-03-18T15:28:40.907 INFO:teuthology.orchestra.run.smithi019.stdout: "unittest_1_bytes": 0, 2024-03-18T15:28:40.907 INFO:teuthology.orchestra.run.smithi019.stdout: "unittest_1_items": 0, 2024-03-18T15:28:40.907 INFO:teuthology.orchestra.run.smithi019.stdout: "unittest_2_bytes": 0, 2024-03-18T15:28:40.907 INFO:teuthology.orchestra.run.smithi019.stdout: "unittest_2_items": 0 2024-03-18T15:28:40.907 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:40.907 INFO:teuthology.orchestra.run.smithi019.stdout: "objecter": { 2024-03-18T15:28:40.907 INFO:teuthology.orchestra.run.smithi019.stdout: "op_active": 0, 2024-03-18T15:28:40.908 INFO:teuthology.orchestra.run.smithi019.stdout: "op_laggy": 0, 2024-03-18T15:28:40.908 INFO:teuthology.orchestra.run.smithi019.stdout: "op_send": 2175, 2024-03-18T15:28:40.908 INFO:teuthology.orchestra.run.smithi019.stdout: "op_send_bytes": 21, 2024-03-18T15:28:40.908 INFO:teuthology.orchestra.run.smithi019.stdout: "op_resend": 3, 2024-03-18T15:28:40.908 INFO:teuthology.orchestra.run.smithi019.stdout: "op_reply": 2175, 2024-03-18T15:28:40.908 INFO:teuthology.orchestra.run.smithi019.stdout: "op_latency": { 2024-03-18T15:28:40.908 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 2175, 2024-03-18T15:28:40.908 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 1.935978606, 2024-03-18T15:28:40.908 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000890105 2024-03-18T15:28:40.908 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:40.908 INFO:teuthology.orchestra.run.smithi019.stdout: "op_inflight": 0, 2024-03-18T15:28:40.908 INFO:teuthology.orchestra.run.smithi019.stdout: "oplen_avg": { 2024-03-18T15:28:40.908 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 2175, 2024-03-18T15:28:40.908 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 2176 2024-03-18T15:28:40.909 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:40.909 INFO:teuthology.orchestra.run.smithi019.stdout: "op": 2175, 2024-03-18T15:28:40.909 INFO:teuthology.orchestra.run.smithi019.stdout: "op_r": 2175, 2024-03-18T15:28:40.909 INFO:teuthology.orchestra.run.smithi019.stdout: "op_w": 0, 2024-03-18T15:28:40.909 INFO:teuthology.orchestra.run.smithi019.stdout: "op_rmw": 0, 2024-03-18T15:28:40.909 INFO:teuthology.orchestra.run.smithi019.stdout: "op_pg": 0, 2024-03-18T15:28:40.909 INFO:teuthology.orchestra.run.smithi019.stdout: "osdop_stat": 1392, 2024-03-18T15:28:40.909 INFO:teuthology.orchestra.run.smithi019.stdout: "osdop_create": 0, 2024-03-18T15:28:40.909 INFO:teuthology.orchestra.run.smithi019.stdout: "osdop_read": 781, 2024-03-18T15:28:40.909 INFO:teuthology.orchestra.run.smithi019.stdout: "osdop_write": 0, 2024-03-18T15:28:40.909 INFO:teuthology.orchestra.run.smithi019.stdout: "osdop_writefull": 0, 2024-03-18T15:28:40.909 INFO:teuthology.orchestra.run.smithi019.stdout: "osdop_writesame": 0, 2024-03-18T15:28:40.909 INFO:teuthology.orchestra.run.smithi019.stdout: "osdop_append": 0, 2024-03-18T15:28:40.909 INFO:teuthology.orchestra.run.smithi019.stdout: "osdop_zero": 0, 2024-03-18T15:28:40.909 INFO:teuthology.orchestra.run.smithi019.stdout: "osdop_truncate": 0, 2024-03-18T15:28:40.909 INFO:teuthology.orchestra.run.smithi019.stdout: "osdop_delete": 0, 2024-03-18T15:28:40.910 INFO:teuthology.orchestra.run.smithi019.stdout: "osdop_mapext": 0, 2024-03-18T15:28:40.910 INFO:teuthology.orchestra.run.smithi019.stdout: "osdop_sparse_read": 0, 2024-03-18T15:28:40.910 INFO:teuthology.orchestra.run.smithi019.stdout: "osdop_clonerange": 0, 2024-03-18T15:28:40.910 INFO:teuthology.orchestra.run.smithi019.stdout: "osdop_getxattr": 1, 2024-03-18T15:28:40.910 INFO:teuthology.orchestra.run.smithi019.stdout: "osdop_setxattr": 0, 2024-03-18T15:28:40.910 INFO:teuthology.orchestra.run.smithi019.stdout: "osdop_cmpxattr": 0, 2024-03-18T15:28:40.910 INFO:teuthology.orchestra.run.smithi019.stdout: "osdop_rmxattr": 0, 2024-03-18T15:28:40.910 INFO:teuthology.orchestra.run.smithi019.stdout: "osdop_resetxattrs": 0, 2024-03-18T15:28:40.910 INFO:teuthology.orchestra.run.smithi019.stdout: "osdop_call": 0, 2024-03-18T15:28:40.910 INFO:teuthology.orchestra.run.smithi019.stdout: "osdop_watch": 0, 2024-03-18T15:28:40.910 INFO:teuthology.orchestra.run.smithi019.stdout: "osdop_notify": 0, 2024-03-18T15:28:40.910 INFO:teuthology.orchestra.run.smithi019.stdout: "osdop_src_cmpxattr": 0, 2024-03-18T15:28:40.910 INFO:teuthology.orchestra.run.smithi019.stdout: "osdop_pgls": 0, 2024-03-18T15:28:40.910 INFO:teuthology.orchestra.run.smithi019.stdout: "osdop_pgls_filter": 0, 2024-03-18T15:28:40.911 INFO:teuthology.orchestra.run.smithi019.stdout: "osdop_other": 0, 2024-03-18T15:28:40.911 INFO:teuthology.orchestra.run.smithi019.stdout: "linger_active": 0, 2024-03-18T15:28:40.911 INFO:teuthology.orchestra.run.smithi019.stdout: "linger_send": 0, 2024-03-18T15:28:40.911 INFO:teuthology.orchestra.run.smithi019.stdout: "linger_resend": 0, 2024-03-18T15:28:40.911 INFO:teuthology.orchestra.run.smithi019.stdout: "linger_ping": 0, 2024-03-18T15:28:40.911 INFO:teuthology.orchestra.run.smithi019.stdout: "poolop_active": 0, 2024-03-18T15:28:40.911 INFO:teuthology.orchestra.run.smithi019.stdout: "poolop_send": 0, 2024-03-18T15:28:40.911 INFO:teuthology.orchestra.run.smithi019.stdout: "poolop_resend": 0, 2024-03-18T15:28:40.911 INFO:teuthology.orchestra.run.smithi019.stdout: "poolstat_active": 0, 2024-03-18T15:28:40.911 INFO:teuthology.orchestra.run.smithi019.stdout: "poolstat_send": 0, 2024-03-18T15:28:40.911 INFO:teuthology.orchestra.run.smithi019.stdout: "poolstat_resend": 0, 2024-03-18T15:28:40.911 INFO:teuthology.orchestra.run.smithi019.stdout: "statfs_active": 0, 2024-03-18T15:28:40.911 INFO:teuthology.orchestra.run.smithi019.stdout: "statfs_send": 0, 2024-03-18T15:28:40.911 INFO:teuthology.orchestra.run.smithi019.stdout: "statfs_resend": 0, 2024-03-18T15:28:40.911 INFO:teuthology.orchestra.run.smithi019.stdout: "command_active": 0, 2024-03-18T15:28:40.912 INFO:teuthology.orchestra.run.smithi019.stdout: "command_send": 0, 2024-03-18T15:28:40.912 INFO:teuthology.orchestra.run.smithi019.stdout: "command_resend": 0, 2024-03-18T15:28:40.912 INFO:teuthology.orchestra.run.smithi019.stdout: "map_epoch": 103, 2024-03-18T15:28:40.912 INFO:teuthology.orchestra.run.smithi019.stdout: "map_full": 0, 2024-03-18T15:28:40.912 INFO:teuthology.orchestra.run.smithi019.stdout: "map_inc": 14, 2024-03-18T15:28:40.912 INFO:teuthology.orchestra.run.smithi019.stdout: "osd_sessions": 6, 2024-03-18T15:28:40.912 INFO:teuthology.orchestra.run.smithi019.stdout: "osd_session_open": 6, 2024-03-18T15:28:40.912 INFO:teuthology.orchestra.run.smithi019.stdout: "osd_session_close": 0, 2024-03-18T15:28:40.912 INFO:teuthology.orchestra.run.smithi019.stdout: "osd_laggy": 0, 2024-03-18T15:28:40.912 INFO:teuthology.orchestra.run.smithi019.stdout: "omap_wr": 0, 2024-03-18T15:28:40.912 INFO:teuthology.orchestra.run.smithi019.stdout: "omap_rd": 2, 2024-03-18T15:28:40.912 INFO:teuthology.orchestra.run.smithi019.stdout: "omap_del": 0 2024-03-18T15:28:40.912 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:40.912 INFO:teuthology.orchestra.run.smithi019.stdout: "oft": { 2024-03-18T15:28:40.912 INFO:teuthology.orchestra.run.smithi019.stdout: "omap_total_objs": 0, 2024-03-18T15:28:40.913 INFO:teuthology.orchestra.run.smithi019.stdout: "omap_total_kv_pairs": 0, 2024-03-18T15:28:40.913 INFO:teuthology.orchestra.run.smithi019.stdout: "omap_total_updates": 0, 2024-03-18T15:28:40.913 INFO:teuthology.orchestra.run.smithi019.stdout: "omap_total_removes": 0 2024-03-18T15:28:40.913 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:40.913 INFO:teuthology.orchestra.run.smithi019.stdout: "osd-slow-ops": { 2024-03-18T15:28:40.913 INFO:teuthology.orchestra.run.smithi019.stdout: "slow_ops_count": 0 2024-03-18T15:28:40.913 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:40.913 INFO:teuthology.orchestra.run.smithi019.stdout: "purge_queue": { 2024-03-18T15:28:40.913 INFO:teuthology.orchestra.run.smithi019.stdout: "pq_executing_ops": 0, 2024-03-18T15:28:40.913 INFO:teuthology.orchestra.run.smithi019.stdout: "pq_executing_ops_high_water": 0, 2024-03-18T15:28:40.913 INFO:teuthology.orchestra.run.smithi019.stdout: "pq_executing": 0, 2024-03-18T15:28:40.913 INFO:teuthology.orchestra.run.smithi019.stdout: "pq_executing_high_water": 0, 2024-03-18T15:28:40.913 INFO:teuthology.orchestra.run.smithi019.stdout: "pq_executed_ops": 0, 2024-03-18T15:28:40.913 INFO:teuthology.orchestra.run.smithi019.stdout: "pq_executed": 0, 2024-03-18T15:28:40.913 INFO:teuthology.orchestra.run.smithi019.stdout: "pq_item_in_journal": 0 2024-03-18T15:28:40.914 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:40.914 INFO:teuthology.orchestra.run.smithi019.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T15:28:40.914 INFO:teuthology.orchestra.run.smithi019.stdout: "val": 0, 2024-03-18T15:28:40.914 INFO:teuthology.orchestra.run.smithi019.stdout: "max": 104857600, 2024-03-18T15:28:40.914 INFO:teuthology.orchestra.run.smithi019.stdout: "get_started": 0, 2024-03-18T15:28:40.914 INFO:teuthology.orchestra.run.smithi019.stdout: "get": 2432, 2024-03-18T15:28:40.914 INFO:teuthology.orchestra.run.smithi019.stdout: "get_sum": 2162950, 2024-03-18T15:28:40.914 INFO:teuthology.orchestra.run.smithi019.stdout: "get_or_fail_fail": 0, 2024-03-18T15:28:40.914 INFO:teuthology.orchestra.run.smithi019.stdout: "get_or_fail_success": 2432, 2024-03-18T15:28:40.914 INFO:teuthology.orchestra.run.smithi019.stdout: "take": 0, 2024-03-18T15:28:40.914 INFO:teuthology.orchestra.run.smithi019.stdout: "take_sum": 0, 2024-03-18T15:28:40.915 INFO:teuthology.orchestra.run.smithi019.stdout: "put": 2432, 2024-03-18T15:28:40.915 INFO:teuthology.orchestra.run.smithi019.stdout: "put_sum": 2162950, 2024-03-18T15:28:40.915 INFO:teuthology.orchestra.run.smithi019.stdout: "wait": { 2024-03-18T15:28:40.915 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 0, 2024-03-18T15:28:40.915 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.000000000, 2024-03-18T15:28:40.915 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000000000 2024-03-18T15:28:40.915 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:40.915 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:40.915 INFO:teuthology.orchestra.run.smithi019.stdout: "throttle-objecter_bytes": { 2024-03-18T15:28:40.915 INFO:teuthology.orchestra.run.smithi019.stdout: "val": 0, 2024-03-18T15:28:40.915 INFO:teuthology.orchestra.run.smithi019.stdout: "max": 104857600, 2024-03-18T15:28:40.915 INFO:teuthology.orchestra.run.smithi019.stdout: "get_started": 0, 2024-03-18T15:28:40.915 INFO:teuthology.orchestra.run.smithi019.stdout: "get": 0, 2024-03-18T15:28:40.915 INFO:teuthology.orchestra.run.smithi019.stdout: "get_sum": 0, 2024-03-18T15:28:40.915 INFO:teuthology.orchestra.run.smithi019.stdout: "get_or_fail_fail": 0, 2024-03-18T15:28:40.916 INFO:teuthology.orchestra.run.smithi019.stdout: "get_or_fail_success": 0, 2024-03-18T15:28:40.916 INFO:teuthology.orchestra.run.smithi019.stdout: "take": 2175, 2024-03-18T15:28:40.916 INFO:teuthology.orchestra.run.smithi019.stdout: "take_sum": 1967, 2024-03-18T15:28:40.916 INFO:teuthology.orchestra.run.smithi019.stdout: "put": 2175, 2024-03-18T15:28:40.916 INFO:teuthology.orchestra.run.smithi019.stdout: "put_sum": 1967, 2024-03-18T15:28:40.916 INFO:teuthology.orchestra.run.smithi019.stdout: "wait": { 2024-03-18T15:28:40.916 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 0, 2024-03-18T15:28:40.916 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.000000000, 2024-03-18T15:28:40.916 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000000000 2024-03-18T15:28:40.916 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:40.916 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:40.916 INFO:teuthology.orchestra.run.smithi019.stdout: "throttle-objecter_ops": { 2024-03-18T15:28:40.916 INFO:teuthology.orchestra.run.smithi019.stdout: "val": 0, 2024-03-18T15:28:40.916 INFO:teuthology.orchestra.run.smithi019.stdout: "max": 1024, 2024-03-18T15:28:40.916 INFO:teuthology.orchestra.run.smithi019.stdout: "get_started": 0, 2024-03-18T15:28:40.917 INFO:teuthology.orchestra.run.smithi019.stdout: "get": 0, 2024-03-18T15:28:40.917 INFO:teuthology.orchestra.run.smithi019.stdout: "get_sum": 0, 2024-03-18T15:28:40.917 INFO:teuthology.orchestra.run.smithi019.stdout: "get_or_fail_fail": 0, 2024-03-18T15:28:40.917 INFO:teuthology.orchestra.run.smithi019.stdout: "get_or_fail_success": 0, 2024-03-18T15:28:40.917 INFO:teuthology.orchestra.run.smithi019.stdout: "take": 2175, 2024-03-18T15:28:40.917 INFO:teuthology.orchestra.run.smithi019.stdout: "take_sum": 2175, 2024-03-18T15:28:40.917 INFO:teuthology.orchestra.run.smithi019.stdout: "put": 2175, 2024-03-18T15:28:40.917 INFO:teuthology.orchestra.run.smithi019.stdout: "put_sum": 2175, 2024-03-18T15:28:40.917 INFO:teuthology.orchestra.run.smithi019.stdout: "wait": { 2024-03-18T15:28:40.917 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 0, 2024-03-18T15:28:40.917 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.000000000, 2024-03-18T15:28:40.917 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000000000 2024-03-18T15:28:40.917 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:40.917 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:40.917 INFO:teuthology.orchestra.run.smithi019.stdout: "throttle-write_buf_throttle": { 2024-03-18T15:28:40.918 INFO:teuthology.orchestra.run.smithi019.stdout: "val": 0, 2024-03-18T15:28:40.918 INFO:teuthology.orchestra.run.smithi019.stdout: "max": 3758096384, 2024-03-18T15:28:40.918 INFO:teuthology.orchestra.run.smithi019.stdout: "get_started": 0, 2024-03-18T15:28:40.918 INFO:teuthology.orchestra.run.smithi019.stdout: "get": 0, 2024-03-18T15:28:40.918 INFO:teuthology.orchestra.run.smithi019.stdout: "get_sum": 0, 2024-03-18T15:28:40.918 INFO:teuthology.orchestra.run.smithi019.stdout: "get_or_fail_fail": 0, 2024-03-18T15:28:40.918 INFO:teuthology.orchestra.run.smithi019.stdout: "get_or_fail_success": 0, 2024-03-18T15:28:40.918 INFO:teuthology.orchestra.run.smithi019.stdout: "take": 0, 2024-03-18T15:28:40.918 INFO:teuthology.orchestra.run.smithi019.stdout: "take_sum": 0, 2024-03-18T15:28:40.918 INFO:teuthology.orchestra.run.smithi019.stdout: "put": 0, 2024-03-18T15:28:40.918 INFO:teuthology.orchestra.run.smithi019.stdout: "put_sum": 0, 2024-03-18T15:28:40.918 INFO:teuthology.orchestra.run.smithi019.stdout: "wait": { 2024-03-18T15:28:40.918 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 0, 2024-03-18T15:28:40.918 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.000000000, 2024-03-18T15:28:40.919 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000000000 2024-03-18T15:28:40.919 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:40.919 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:40.919 INFO:teuthology.orchestra.run.smithi019.stdout: "throttle-write_buf_throttle-0x555a31f92780": { 2024-03-18T15:28:40.919 INFO:teuthology.orchestra.run.smithi019.stdout: "val": 0, 2024-03-18T15:28:40.919 INFO:teuthology.orchestra.run.smithi019.stdout: "max": 3758096384, 2024-03-18T15:28:40.919 INFO:teuthology.orchestra.run.smithi019.stdout: "get_started": 0, 2024-03-18T15:28:40.919 INFO:teuthology.orchestra.run.smithi019.stdout: "get": 0, 2024-03-18T15:28:40.919 INFO:teuthology.orchestra.run.smithi019.stdout: "get_sum": 0, 2024-03-18T15:28:40.919 INFO:teuthology.orchestra.run.smithi019.stdout: "get_or_fail_fail": 0, 2024-03-18T15:28:40.919 INFO:teuthology.orchestra.run.smithi019.stdout: "get_or_fail_success": 0, 2024-03-18T15:28:40.919 INFO:teuthology.orchestra.run.smithi019.stdout: "take": 0, 2024-03-18T15:28:40.919 INFO:teuthology.orchestra.run.smithi019.stdout: "take_sum": 0, 2024-03-18T15:28:40.919 INFO:teuthology.orchestra.run.smithi019.stdout: "put": 0, 2024-03-18T15:28:40.919 INFO:teuthology.orchestra.run.smithi019.stdout: "put_sum": 0, 2024-03-18T15:28:40.920 INFO:teuthology.orchestra.run.smithi019.stdout: "wait": { 2024-03-18T15:28:40.920 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 0, 2024-03-18T15:28:40.920 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.000000000, 2024-03-18T15:28:40.920 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000000000 2024-03-18T15:28:40.920 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:40.920 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:40.920 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-03-18T15:28:41.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:28:41 smithi019 ceph-mon[31522]: pgmap v317: 129 pgs: 129 active+clean; 1.7 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-18T15:28:41.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:28:41 smithi033 ceph-mon[36648]: pgmap v317: 129 pgs: 129 active+clean; 1.7 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-18T15:28:41.831 INFO:tasks.check_counter:Daemon mds.a mds.exported=0 2024-03-18T15:28:41.831 INFO:tasks.check_counter:Daemon mds.a mds.imported=0 2024-03-18T15:28:41.831 INFO:tasks.check_counter:Daemon mds.a mds_cache.dir_update=0 2024-03-18T15:28:41.832 INFO:tasks.check_counter:Daemon mds.a mds_cache.dir_update_receipt=0 2024-03-18T15:28:41.832 INFO:tasks.check_counter:Daemon mds.a mds.root_rsnaps=0 2024-03-18T15:28:41.832 INFO:tasks.check_counter:Daemon mds.a mds_server.req_mksnap_latency.avgcount=0 2024-03-18T15:28:41.832 INFO:tasks.check_counter:Daemon mds.a mds_server.req_rmsnap_latency.avgcount=0 2024-03-18T15:28:41.832 DEBUG:tasks.check_counter:Getting stats from d 2024-03-18T15:28:41.832 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 -- ceph daemon mds.d perf dump 2024-03-18T15:28:41.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:28:41 smithi104 ceph-mon[36070]: pgmap v317: 129 pgs: 129 active+clean; 1.7 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-18T15:28:42.320 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:28:43.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:28:43 smithi033 ceph-mon[36648]: pgmap v318: 129 pgs: 129 active+clean; 1.7 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 0 B/s wr, 14 op/s 2024-03-18T15:28:43.802 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:28:43 smithi019 ceph-mon[31522]: pgmap v318: 129 pgs: 129 active+clean; 1.7 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 0 B/s wr, 14 op/s 2024-03-18T15:28:43.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:28:43 smithi104 ceph-mon[36070]: pgmap v318: 129 pgs: 129 active+clean; 1.7 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 0 B/s wr, 14 op/s 2024-03-18T15:28:44.736 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-03-18T15:28:44.736 INFO:teuthology.orchestra.run.smithi019.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T15:28:44.736 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_recv_messages": 2185, 2024-03-18T15:28:44.736 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_send_messages": 2186, 2024-03-18T15:28:44.736 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_recv_bytes": 621866, 2024-03-18T15:28:44.736 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_send_bytes": 694519, 2024-03-18T15:28:44.736 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_created_connections": 6, 2024-03-18T15:28:44.736 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_active_connections": 2, 2024-03-18T15:28:44.737 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_running_total_time": 0.388865237, 2024-03-18T15:28:44.737 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_running_send_time": 0.128965775, 2024-03-18T15:28:44.737 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_running_recv_time": 0.101765356, 2024-03-18T15:28:44.737 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_running_fast_dispatch_time": 0.122993496, 2024-03-18T15:28:44.737 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T15:28:44.737 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 2184, 2024-03-18T15:28:44.737 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.168798192, 2024-03-18T15:28:44.737 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000077288 2024-03-18T15:28:44.737 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:44.737 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_handle_ack_lat": { 2024-03-18T15:28:44.737 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 0, 2024-03-18T15:28:44.737 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.000000000, 2024-03-18T15:28:44.737 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000000000 2024-03-18T15:28:44.738 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:44.738 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-18T15:28:44.738 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_send_encrypted_bytes": 0 2024-03-18T15:28:44.738 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:44.738 INFO:teuthology.orchestra.run.smithi019.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T15:28:44.738 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_recv_messages": 264, 2024-03-18T15:28:44.738 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_send_messages": 247, 2024-03-18T15:28:44.738 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_recv_bytes": 1777452, 2024-03-18T15:28:44.738 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_send_bytes": 143418, 2024-03-18T15:28:44.738 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_created_connections": 3, 2024-03-18T15:28:44.738 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_active_connections": 2, 2024-03-18T15:28:44.738 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_running_total_time": 0.104041905, 2024-03-18T15:28:44.738 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_running_send_time": 0.026913019, 2024-03-18T15:28:44.738 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_running_recv_time": 0.095046120, 2024-03-18T15:28:44.739 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_running_fast_dispatch_time": 0.010874083, 2024-03-18T15:28:44.739 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T15:28:44.739 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 246, 2024-03-18T15:28:44.739 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.024352358, 2024-03-18T15:28:44.739 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000098993 2024-03-18T15:28:44.739 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:44.739 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_handle_ack_lat": { 2024-03-18T15:28:44.739 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 0, 2024-03-18T15:28:44.739 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.000000000, 2024-03-18T15:28:44.739 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000000000 2024-03-18T15:28:44.739 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:44.739 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_recv_encrypted_bytes": 1776896, 2024-03-18T15:28:44.739 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_send_encrypted_bytes": 142720 2024-03-18T15:28:44.739 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:44.740 INFO:teuthology.orchestra.run.smithi019.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T15:28:44.740 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_recv_messages": 7, 2024-03-18T15:28:44.740 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_send_messages": 176, 2024-03-18T15:28:44.740 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_recv_bytes": 3660, 2024-03-18T15:28:44.740 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_send_bytes": 454334, 2024-03-18T15:28:44.740 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_created_connections": 6, 2024-03-18T15:28:44.740 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_active_connections": 4, 2024-03-18T15:28:44.740 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_running_total_time": 0.036203479, 2024-03-18T15:28:44.740 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_running_send_time": 0.025288741, 2024-03-18T15:28:44.740 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_running_recv_time": 0.004844117, 2024-03-18T15:28:44.740 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_running_fast_dispatch_time": 0.000017526, 2024-03-18T15:28:44.740 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T15:28:44.740 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 174, 2024-03-18T15:28:44.740 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.026634861, 2024-03-18T15:28:44.741 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000153073 2024-03-18T15:28:44.741 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:44.741 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_handle_ack_lat": { 2024-03-18T15:28:44.741 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 0, 2024-03-18T15:28:44.741 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.000000000, 2024-03-18T15:28:44.741 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000000000 2024-03-18T15:28:44.741 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:44.741 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_recv_encrypted_bytes": 3392, 2024-03-18T15:28:44.741 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_send_encrypted_bytes": 454016 2024-03-18T15:28:44.741 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:44.741 INFO:teuthology.orchestra.run.smithi019.stdout: "cct": { 2024-03-18T15:28:44.741 INFO:teuthology.orchestra.run.smithi019.stdout: "total_workers": 1, 2024-03-18T15:28:44.742 INFO:teuthology.orchestra.run.smithi019.stdout: "unhealthy_workers": 0 2024-03-18T15:28:44.742 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:44.742 INFO:teuthology.orchestra.run.smithi019.stdout: "finisher-MDSRank": { 2024-03-18T15:28:44.742 INFO:teuthology.orchestra.run.smithi019.stdout: "queue_len": 0, 2024-03-18T15:28:44.742 INFO:teuthology.orchestra.run.smithi019.stdout: "complete_latency": { 2024-03-18T15:28:44.742 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 3705, 2024-03-18T15:28:44.742 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.190264260, 2024-03-18T15:28:44.742 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000051353 2024-03-18T15:28:44.742 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:44.742 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:44.742 INFO:teuthology.orchestra.run.smithi019.stdout: "finisher-PurgeQueue": { 2024-03-18T15:28:44.742 INFO:teuthology.orchestra.run.smithi019.stdout: "queue_len": 0, 2024-03-18T15:28:44.743 INFO:teuthology.orchestra.run.smithi019.stdout: "complete_latency": { 2024-03-18T15:28:44.743 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 0, 2024-03-18T15:28:44.743 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.000000000, 2024-03-18T15:28:44.743 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000000000 2024-03-18T15:28:44.743 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:44.743 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:44.743 INFO:teuthology.orchestra.run.smithi019.stdout: "mds": { 2024-03-18T15:28:44.743 INFO:teuthology.orchestra.run.smithi019.stdout: "request": 0, 2024-03-18T15:28:44.743 INFO:teuthology.orchestra.run.smithi019.stdout: "reply": 0, 2024-03-18T15:28:44.743 INFO:teuthology.orchestra.run.smithi019.stdout: "reply_latency": { 2024-03-18T15:28:44.743 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 0, 2024-03-18T15:28:44.743 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.000000000, 2024-03-18T15:28:44.743 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000000000 2024-03-18T15:28:44.744 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:44.744 INFO:teuthology.orchestra.run.smithi019.stdout: "slow_reply": 0, 2024-03-18T15:28:44.744 INFO:teuthology.orchestra.run.smithi019.stdout: "forward": 0, 2024-03-18T15:28:44.744 INFO:teuthology.orchestra.run.smithi019.stdout: "dir_fetch_complete": 0, 2024-03-18T15:28:44.744 INFO:teuthology.orchestra.run.smithi019.stdout: "dir_fetch_keys": 0, 2024-03-18T15:28:44.744 INFO:teuthology.orchestra.run.smithi019.stdout: "dir_commit": 0, 2024-03-18T15:28:44.744 INFO:teuthology.orchestra.run.smithi019.stdout: "dir_split": 0, 2024-03-18T15:28:44.744 INFO:teuthology.orchestra.run.smithi019.stdout: "dir_merge": 0, 2024-03-18T15:28:44.744 INFO:teuthology.orchestra.run.smithi019.stdout: "inodes": 0, 2024-03-18T15:28:44.744 INFO:teuthology.orchestra.run.smithi019.stdout: "inodes_top": 0, 2024-03-18T15:28:44.744 INFO:teuthology.orchestra.run.smithi019.stdout: "inodes_bottom": 0, 2024-03-18T15:28:44.744 INFO:teuthology.orchestra.run.smithi019.stdout: "inodes_pin_tail": 0, 2024-03-18T15:28:44.744 INFO:teuthology.orchestra.run.smithi019.stdout: "inodes_pinned": 0, 2024-03-18T15:28:44.744 INFO:teuthology.orchestra.run.smithi019.stdout: "inodes_expired": 0, 2024-03-18T15:28:44.745 INFO:teuthology.orchestra.run.smithi019.stdout: "inodes_with_caps": 0, 2024-03-18T15:28:44.745 INFO:teuthology.orchestra.run.smithi019.stdout: "caps": 0, 2024-03-18T15:28:44.745 INFO:teuthology.orchestra.run.smithi019.stdout: "subtrees": 1, 2024-03-18T15:28:44.745 INFO:teuthology.orchestra.run.smithi019.stdout: "traverse": 0, 2024-03-18T15:28:44.745 INFO:teuthology.orchestra.run.smithi019.stdout: "traverse_hit": 0, 2024-03-18T15:28:44.745 INFO:teuthology.orchestra.run.smithi019.stdout: "traverse_forward": 0, 2024-03-18T15:28:44.745 INFO:teuthology.orchestra.run.smithi019.stdout: "traverse_discover": 0, 2024-03-18T15:28:44.745 INFO:teuthology.orchestra.run.smithi019.stdout: "traverse_dir_fetch": 0, 2024-03-18T15:28:44.745 INFO:teuthology.orchestra.run.smithi019.stdout: "traverse_remote_ino": 0, 2024-03-18T15:28:44.745 INFO:teuthology.orchestra.run.smithi019.stdout: "traverse_lock": 0, 2024-03-18T15:28:44.745 INFO:teuthology.orchestra.run.smithi019.stdout: "load_cent": 0, 2024-03-18T15:28:44.745 INFO:teuthology.orchestra.run.smithi019.stdout: "q": 0, 2024-03-18T15:28:44.745 INFO:teuthology.orchestra.run.smithi019.stdout: "exported": 0, 2024-03-18T15:28:44.745 INFO:teuthology.orchestra.run.smithi019.stdout: "exported_inodes": 0, 2024-03-18T15:28:44.746 INFO:teuthology.orchestra.run.smithi019.stdout: "imported": 0, 2024-03-18T15:28:44.746 INFO:teuthology.orchestra.run.smithi019.stdout: "imported_inodes": 0, 2024-03-18T15:28:44.746 INFO:teuthology.orchestra.run.smithi019.stdout: "openino_dir_fetch": 0, 2024-03-18T15:28:44.746 INFO:teuthology.orchestra.run.smithi019.stdout: "openino_backtrace_fetch": 0, 2024-03-18T15:28:44.746 INFO:teuthology.orchestra.run.smithi019.stdout: "openino_peer_discover": 0, 2024-03-18T15:28:44.746 INFO:teuthology.orchestra.run.smithi019.stdout: "root_rfiles": 0, 2024-03-18T15:28:44.746 INFO:teuthology.orchestra.run.smithi019.stdout: "root_rbytes": 0, 2024-03-18T15:28:44.746 INFO:teuthology.orchestra.run.smithi019.stdout: "root_rsnaps": 0, 2024-03-18T15:28:44.746 INFO:teuthology.orchestra.run.smithi019.stdout: "scrub_backtrace_fetch": 0, 2024-03-18T15:28:44.746 INFO:teuthology.orchestra.run.smithi019.stdout: "scrub_set_tag": 0, 2024-03-18T15:28:44.746 INFO:teuthology.orchestra.run.smithi019.stdout: "scrub_backtrace_repaired": 0, 2024-03-18T15:28:44.746 INFO:teuthology.orchestra.run.smithi019.stdout: "scrub_inotable_repaired": 0, 2024-03-18T15:28:44.746 INFO:teuthology.orchestra.run.smithi019.stdout: "scrub_dir_inodes": 0, 2024-03-18T15:28:44.746 INFO:teuthology.orchestra.run.smithi019.stdout: "scrub_dir_base_inodes": 0, 2024-03-18T15:28:44.747 INFO:teuthology.orchestra.run.smithi019.stdout: "scrub_dirfrag_rstats": 0, 2024-03-18T15:28:44.747 INFO:teuthology.orchestra.run.smithi019.stdout: "scrub_file_inodes": 0, 2024-03-18T15:28:44.747 INFO:teuthology.orchestra.run.smithi019.stdout: "handle_inode_file_caps": 0, 2024-03-18T15:28:44.747 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph_cap_op_revoke": 0, 2024-03-18T15:28:44.747 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph_cap_op_grant": 0, 2024-03-18T15:28:44.747 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph_cap_op_trunc": 0, 2024-03-18T15:28:44.747 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-18T15:28:44.747 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-18T15:28:44.747 INFO:teuthology.orchestra.run.smithi019.stdout: "handle_client_caps": 0, 2024-03-18T15:28:44.747 INFO:teuthology.orchestra.run.smithi019.stdout: "handle_client_caps_dirty": 0, 2024-03-18T15:28:44.747 INFO:teuthology.orchestra.run.smithi019.stdout: "handle_client_cap_release": 0, 2024-03-18T15:28:44.747 INFO:teuthology.orchestra.run.smithi019.stdout: "process_request_cap_release": 0 2024-03-18T15:28:44.747 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:44.747 INFO:teuthology.orchestra.run.smithi019.stdout: "mds_cache": { 2024-03-18T15:28:44.748 INFO:teuthology.orchestra.run.smithi019.stdout: "dir_update": 0, 2024-03-18T15:28:44.748 INFO:teuthology.orchestra.run.smithi019.stdout: "dir_update_receipt": 0, 2024-03-18T15:28:44.748 INFO:teuthology.orchestra.run.smithi019.stdout: "dir_try_discover": 0, 2024-03-18T15:28:44.748 INFO:teuthology.orchestra.run.smithi019.stdout: "dir_send_discover": 0, 2024-03-18T15:28:44.748 INFO:teuthology.orchestra.run.smithi019.stdout: "dir_handle_discover": 0, 2024-03-18T15:28:44.748 INFO:teuthology.orchestra.run.smithi019.stdout: "num_strays": 0, 2024-03-18T15:28:44.748 INFO:teuthology.orchestra.run.smithi019.stdout: "num_strays_delayed": 0, 2024-03-18T15:28:44.748 INFO:teuthology.orchestra.run.smithi019.stdout: "num_strays_enqueuing": 0, 2024-03-18T15:28:44.748 INFO:teuthology.orchestra.run.smithi019.stdout: "strays_created": 0, 2024-03-18T15:28:44.748 INFO:teuthology.orchestra.run.smithi019.stdout: "strays_enqueued": 0, 2024-03-18T15:28:44.748 INFO:teuthology.orchestra.run.smithi019.stdout: "strays_reintegrated": 0, 2024-03-18T15:28:44.748 INFO:teuthology.orchestra.run.smithi019.stdout: "strays_migrated": 0, 2024-03-18T15:28:44.748 INFO:teuthology.orchestra.run.smithi019.stdout: "num_recovering_processing": 0, 2024-03-18T15:28:44.748 INFO:teuthology.orchestra.run.smithi019.stdout: "num_recovering_enqueued": 0, 2024-03-18T15:28:44.748 INFO:teuthology.orchestra.run.smithi019.stdout: "num_recovering_prioritized": 0, 2024-03-18T15:28:44.749 INFO:teuthology.orchestra.run.smithi019.stdout: "recovery_started": 0, 2024-03-18T15:28:44.749 INFO:teuthology.orchestra.run.smithi019.stdout: "recovery_completed": 0, 2024-03-18T15:28:44.749 INFO:teuthology.orchestra.run.smithi019.stdout: "ireq_quiesce_path": 0, 2024-03-18T15:28:44.749 INFO:teuthology.orchestra.run.smithi019.stdout: "ireq_quiesce_inode": 0, 2024-03-18T15:28:44.749 INFO:teuthology.orchestra.run.smithi019.stdout: "ireq_enqueue_scrub": 0, 2024-03-18T15:28:44.749 INFO:teuthology.orchestra.run.smithi019.stdout: "ireq_exportdir": 0, 2024-03-18T15:28:44.749 INFO:teuthology.orchestra.run.smithi019.stdout: "ireq_flush": 0, 2024-03-18T15:28:44.749 INFO:teuthology.orchestra.run.smithi019.stdout: "ireq_fragmentdir": 0, 2024-03-18T15:28:44.749 INFO:teuthology.orchestra.run.smithi019.stdout: "ireq_fragstats": 0, 2024-03-18T15:28:44.749 INFO:teuthology.orchestra.run.smithi019.stdout: "ireq_inodestats": 0 2024-03-18T15:28:44.749 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:44.749 INFO:teuthology.orchestra.run.smithi019.stdout: "mds_log": { 2024-03-18T15:28:44.749 INFO:teuthology.orchestra.run.smithi019.stdout: "evlrg": 0, 2024-03-18T15:28:44.749 INFO:teuthology.orchestra.run.smithi019.stdout: "evadd": 0, 2024-03-18T15:28:44.750 INFO:teuthology.orchestra.run.smithi019.stdout: "evex": 0, 2024-03-18T15:28:44.750 INFO:teuthology.orchestra.run.smithi019.stdout: "evtrm": 0, 2024-03-18T15:28:44.750 INFO:teuthology.orchestra.run.smithi019.stdout: "ev": 4, 2024-03-18T15:28:44.750 INFO:teuthology.orchestra.run.smithi019.stdout: "evexg": 0, 2024-03-18T15:28:44.750 INFO:teuthology.orchestra.run.smithi019.stdout: "evexd": 0, 2024-03-18T15:28:44.750 INFO:teuthology.orchestra.run.smithi019.stdout: "segadd": 0, 2024-03-18T15:28:44.750 INFO:teuthology.orchestra.run.smithi019.stdout: "segex": 0, 2024-03-18T15:28:44.750 INFO:teuthology.orchestra.run.smithi019.stdout: "segtrm": 0, 2024-03-18T15:28:44.750 INFO:teuthology.orchestra.run.smithi019.stdout: "seg": 2, 2024-03-18T15:28:44.750 INFO:teuthology.orchestra.run.smithi019.stdout: "segmjr": 2, 2024-03-18T15:28:44.750 INFO:teuthology.orchestra.run.smithi019.stdout: "segexg": 0, 2024-03-18T15:28:44.750 INFO:teuthology.orchestra.run.smithi019.stdout: "segexd": 0, 2024-03-18T15:28:44.750 INFO:teuthology.orchestra.run.smithi019.stdout: "expos": 4194304, 2024-03-18T15:28:44.750 INFO:teuthology.orchestra.run.smithi019.stdout: "wrpos": 4195593, 2024-03-18T15:28:44.751 INFO:teuthology.orchestra.run.smithi019.stdout: "rdpos": 4195202, 2024-03-18T15:28:44.751 INFO:teuthology.orchestra.run.smithi019.stdout: "jlat": { 2024-03-18T15:28:44.751 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 0, 2024-03-18T15:28:44.751 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.000000000, 2024-03-18T15:28:44.751 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000000000 2024-03-18T15:28:44.751 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:44.751 INFO:teuthology.orchestra.run.smithi019.stdout: "replayed": 4 2024-03-18T15:28:44.752 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:44.752 INFO:teuthology.orchestra.run.smithi019.stdout: "mds_mem": { 2024-03-18T15:28:44.752 INFO:teuthology.orchestra.run.smithi019.stdout: "ino": 3, 2024-03-18T15:28:44.752 INFO:teuthology.orchestra.run.smithi019.stdout: "ino+": 3, 2024-03-18T15:28:44.752 INFO:teuthology.orchestra.run.smithi019.stdout: "ino-": 0, 2024-03-18T15:28:44.752 INFO:teuthology.orchestra.run.smithi019.stdout: "dir": 1, 2024-03-18T15:28:44.752 INFO:teuthology.orchestra.run.smithi019.stdout: "dir+": 1, 2024-03-18T15:28:44.752 INFO:teuthology.orchestra.run.smithi019.stdout: "dir-": 0, 2024-03-18T15:28:44.752 INFO:teuthology.orchestra.run.smithi019.stdout: "dn": 0, 2024-03-18T15:28:44.752 INFO:teuthology.orchestra.run.smithi019.stdout: "dn+": 0, 2024-03-18T15:28:44.752 INFO:teuthology.orchestra.run.smithi019.stdout: "dn-": 0, 2024-03-18T15:28:44.752 INFO:teuthology.orchestra.run.smithi019.stdout: "cap": 0, 2024-03-18T15:28:44.753 INFO:teuthology.orchestra.run.smithi019.stdout: "cap+": 0, 2024-03-18T15:28:44.753 INFO:teuthology.orchestra.run.smithi019.stdout: "cap-": 0, 2024-03-18T15:28:44.753 INFO:teuthology.orchestra.run.smithi019.stdout: "rss": 49016, 2024-03-18T15:28:44.753 INFO:teuthology.orchestra.run.smithi019.stdout: "heap": 215420 2024-03-18T15:28:44.753 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:44.753 INFO:teuthology.orchestra.run.smithi019.stdout: "mds_server": { 2024-03-18T15:28:44.753 INFO:teuthology.orchestra.run.smithi019.stdout: "dispatch_client_request": 0, 2024-03-18T15:28:44.753 INFO:teuthology.orchestra.run.smithi019.stdout: "dispatch_server_request": 0, 2024-03-18T15:28:44.753 INFO:teuthology.orchestra.run.smithi019.stdout: "handle_client_request": 0, 2024-03-18T15:28:44.753 INFO:teuthology.orchestra.run.smithi019.stdout: "handle_client_session": 0, 2024-03-18T15:28:44.753 INFO:teuthology.orchestra.run.smithi019.stdout: "handle_peer_request": 0, 2024-03-18T15:28:44.753 INFO:teuthology.orchestra.run.smithi019.stdout: "req_create_latency": { 2024-03-18T15:28:44.753 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 0, 2024-03-18T15:28:44.753 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.000000000, 2024-03-18T15:28:44.753 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000000000 2024-03-18T15:28:44.754 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:44.754 INFO:teuthology.orchestra.run.smithi019.stdout: "req_getattr_latency": { 2024-03-18T15:28:44.754 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 0, 2024-03-18T15:28:44.754 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.000000000, 2024-03-18T15:28:44.754 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000000000 2024-03-18T15:28:44.754 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:44.754 INFO:teuthology.orchestra.run.smithi019.stdout: "req_getfilelock_latency": { 2024-03-18T15:28:44.754 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 0, 2024-03-18T15:28:44.754 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.000000000, 2024-03-18T15:28:44.754 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000000000 2024-03-18T15:28:44.754 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:44.754 INFO:teuthology.orchestra.run.smithi019.stdout: "req_link_latency": { 2024-03-18T15:28:44.754 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 0, 2024-03-18T15:28:44.754 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.000000000, 2024-03-18T15:28:44.755 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000000000 2024-03-18T15:28:44.755 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:44.755 INFO:teuthology.orchestra.run.smithi019.stdout: "req_lookup_latency": { 2024-03-18T15:28:44.755 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 0, 2024-03-18T15:28:44.755 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.000000000, 2024-03-18T15:28:44.755 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000000000 2024-03-18T15:28:44.755 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:44.755 INFO:teuthology.orchestra.run.smithi019.stdout: "req_lookuphash_latency": { 2024-03-18T15:28:44.755 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 0, 2024-03-18T15:28:44.755 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.000000000, 2024-03-18T15:28:44.755 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000000000 2024-03-18T15:28:44.755 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:44.755 INFO:teuthology.orchestra.run.smithi019.stdout: "req_lookupino_latency": { 2024-03-18T15:28:44.755 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 0, 2024-03-18T15:28:44.756 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.000000000, 2024-03-18T15:28:44.756 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000000000 2024-03-18T15:28:44.756 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:44.756 INFO:teuthology.orchestra.run.smithi019.stdout: "req_lookupname_latency": { 2024-03-18T15:28:44.756 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 0, 2024-03-18T15:28:44.756 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.000000000, 2024-03-18T15:28:44.756 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000000000 2024-03-18T15:28:44.756 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:44.756 INFO:teuthology.orchestra.run.smithi019.stdout: "req_lookupparent_latency": { 2024-03-18T15:28:44.756 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 0, 2024-03-18T15:28:44.756 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.000000000, 2024-03-18T15:28:44.756 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000000000 2024-03-18T15:28:44.756 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:44.756 INFO:teuthology.orchestra.run.smithi019.stdout: "req_lookupsnap_latency": { 2024-03-18T15:28:44.756 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 0, 2024-03-18T15:28:44.757 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.000000000, 2024-03-18T15:28:44.757 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000000000 2024-03-18T15:28:44.757 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:44.757 INFO:teuthology.orchestra.run.smithi019.stdout: "req_lssnap_latency": { 2024-03-18T15:28:44.757 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 0, 2024-03-18T15:28:44.757 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.000000000, 2024-03-18T15:28:44.757 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000000000 2024-03-18T15:28:44.757 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:44.757 INFO:teuthology.orchestra.run.smithi019.stdout: "req_mkdir_latency": { 2024-03-18T15:28:44.757 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 0, 2024-03-18T15:28:44.759 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.000000000, 2024-03-18T15:28:44.759 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000000000 2024-03-18T15:28:44.759 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:44.759 INFO:teuthology.orchestra.run.smithi019.stdout: "req_mknod_latency": { 2024-03-18T15:28:44.759 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 0, 2024-03-18T15:28:44.759 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.000000000, 2024-03-18T15:28:44.759 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000000000 2024-03-18T15:28:44.759 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:44.759 INFO:teuthology.orchestra.run.smithi019.stdout: "req_mksnap_latency": { 2024-03-18T15:28:44.759 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 0, 2024-03-18T15:28:44.759 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.000000000, 2024-03-18T15:28:44.759 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000000000 2024-03-18T15:28:44.760 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:44.760 INFO:teuthology.orchestra.run.smithi019.stdout: "req_open_latency": { 2024-03-18T15:28:44.760 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 0, 2024-03-18T15:28:44.760 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.000000000, 2024-03-18T15:28:44.760 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000000000 2024-03-18T15:28:44.760 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:44.760 INFO:teuthology.orchestra.run.smithi019.stdout: "req_readdir_latency": { 2024-03-18T15:28:44.760 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 0, 2024-03-18T15:28:44.760 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.000000000, 2024-03-18T15:28:44.760 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000000000 2024-03-18T15:28:44.760 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:44.760 INFO:teuthology.orchestra.run.smithi019.stdout: "req_rename_latency": { 2024-03-18T15:28:44.760 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 0, 2024-03-18T15:28:44.760 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.000000000, 2024-03-18T15:28:44.761 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000000000 2024-03-18T15:28:44.761 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:44.761 INFO:teuthology.orchestra.run.smithi019.stdout: "req_renamesnap_latency": { 2024-03-18T15:28:44.761 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 0, 2024-03-18T15:28:44.761 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.000000000, 2024-03-18T15:28:44.761 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000000000 2024-03-18T15:28:44.761 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:44.761 INFO:teuthology.orchestra.run.smithi019.stdout: "req_snapdiff_latency": { 2024-03-18T15:28:44.761 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 0, 2024-03-18T15:28:44.761 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.000000000, 2024-03-18T15:28:44.761 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000000000 2024-03-18T15:28:44.761 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:44.761 INFO:teuthology.orchestra.run.smithi019.stdout: "req_rmdir_latency": { 2024-03-18T15:28:44.761 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 0, 2024-03-18T15:28:44.761 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.000000000, 2024-03-18T15:28:44.762 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000000000 2024-03-18T15:28:44.762 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:44.762 INFO:teuthology.orchestra.run.smithi019.stdout: "req_rmsnap_latency": { 2024-03-18T15:28:44.762 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 0, 2024-03-18T15:28:44.762 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.000000000, 2024-03-18T15:28:44.762 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000000000 2024-03-18T15:28:44.762 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:44.762 INFO:teuthology.orchestra.run.smithi019.stdout: "req_rmxattr_latency": { 2024-03-18T15:28:44.762 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 0, 2024-03-18T15:28:44.762 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.000000000, 2024-03-18T15:28:44.763 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000000000 2024-03-18T15:28:44.763 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:44.763 INFO:teuthology.orchestra.run.smithi019.stdout: "req_setattr_latency": { 2024-03-18T15:28:44.763 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 0, 2024-03-18T15:28:44.763 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.000000000, 2024-03-18T15:28:44.763 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000000000 2024-03-18T15:28:44.763 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:44.763 INFO:teuthology.orchestra.run.smithi019.stdout: "req_setdirlayout_latency": { 2024-03-18T15:28:44.763 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 0, 2024-03-18T15:28:44.763 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.000000000, 2024-03-18T15:28:44.763 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000000000 2024-03-18T15:28:44.763 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:44.763 INFO:teuthology.orchestra.run.smithi019.stdout: "req_setfilelock_latency": { 2024-03-18T15:28:44.763 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 0, 2024-03-18T15:28:44.764 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.000000000, 2024-03-18T15:28:44.764 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000000000 2024-03-18T15:28:44.764 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:44.764 INFO:teuthology.orchestra.run.smithi019.stdout: "req_setlayout_latency": { 2024-03-18T15:28:44.764 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 0, 2024-03-18T15:28:44.764 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.000000000, 2024-03-18T15:28:44.764 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000000000 2024-03-18T15:28:44.764 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:44.764 INFO:teuthology.orchestra.run.smithi019.stdout: "req_setxattr_latency": { 2024-03-18T15:28:44.764 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 0, 2024-03-18T15:28:44.764 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.000000000, 2024-03-18T15:28:44.764 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000000000 2024-03-18T15:28:44.764 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:44.764 INFO:teuthology.orchestra.run.smithi019.stdout: "req_symlink_latency": { 2024-03-18T15:28:44.764 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 0, 2024-03-18T15:28:44.765 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.000000000, 2024-03-18T15:28:44.765 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000000000 2024-03-18T15:28:44.765 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:44.765 INFO:teuthology.orchestra.run.smithi019.stdout: "req_unlink_latency": { 2024-03-18T15:28:44.765 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 0, 2024-03-18T15:28:44.765 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.000000000, 2024-03-18T15:28:44.765 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000000000 2024-03-18T15:28:44.765 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:44.765 INFO:teuthology.orchestra.run.smithi019.stdout: "cap_revoke_eviction": 0, 2024-03-18T15:28:44.765 INFO:teuthology.orchestra.run.smithi019.stdout: "cap_acquisition_throttle": 0, 2024-03-18T15:28:44.766 INFO:teuthology.orchestra.run.smithi019.stdout: "req_getvxattr_latency": { 2024-03-18T15:28:44.766 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 0, 2024-03-18T15:28:44.766 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.000000000, 2024-03-18T15:28:44.766 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000000000 2024-03-18T15:28:44.766 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:44.766 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:44.766 INFO:teuthology.orchestra.run.smithi019.stdout: "mds_sessions": { 2024-03-18T15:28:44.766 INFO:teuthology.orchestra.run.smithi019.stdout: "session_count": 2, 2024-03-18T15:28:44.766 INFO:teuthology.orchestra.run.smithi019.stdout: "session_add": 2, 2024-03-18T15:28:44.766 INFO:teuthology.orchestra.run.smithi019.stdout: "session_remove": 0, 2024-03-18T15:28:44.767 INFO:teuthology.orchestra.run.smithi019.stdout: "sessions_open": 2, 2024-03-18T15:28:44.767 INFO:teuthology.orchestra.run.smithi019.stdout: "sessions_stale": 0, 2024-03-18T15:28:44.767 INFO:teuthology.orchestra.run.smithi019.stdout: "total_load": 0, 2024-03-18T15:28:44.767 INFO:teuthology.orchestra.run.smithi019.stdout: "average_load": 0, 2024-03-18T15:28:44.767 INFO:teuthology.orchestra.run.smithi019.stdout: "avg_session_uptime": 2014, 2024-03-18T15:28:44.767 INFO:teuthology.orchestra.run.smithi019.stdout: "mdthresh_evicted": 0 2024-03-18T15:28:44.767 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:44.767 INFO:teuthology.orchestra.run.smithi019.stdout: "mempool": { 2024-03-18T15:28:44.767 INFO:teuthology.orchestra.run.smithi019.stdout: "bloom_filter_bytes": 0, 2024-03-18T15:28:44.767 INFO:teuthology.orchestra.run.smithi019.stdout: "bloom_filter_items": 0, 2024-03-18T15:28:44.767 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T15:28:44.767 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_alloc_items": 0, 2024-03-18T15:28:44.767 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T15:28:44.767 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_cache_data_items": 0, 2024-03-18T15:28:44.768 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T15:28:44.768 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T15:28:44.768 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T15:28:44.768 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T15:28:44.768 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T15:28:44.768 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_cache_other_items": 0, 2024-03-18T15:28:44.768 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T15:28:44.768 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T15:28:44.768 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_extent_bytes": 0, 2024-03-18T15:28:44.768 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_extent_items": 0, 2024-03-18T15:28:44.768 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_blob_bytes": 0, 2024-03-18T15:28:44.768 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_blob_items": 0, 2024-03-18T15:28:44.769 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T15:28:44.769 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T15:28:44.769 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T15:28:44.769 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T15:28:44.769 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T15:28:44.769 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_fsck_items": 0, 2024-03-18T15:28:44.769 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_txc_bytes": 0, 2024-03-18T15:28:44.769 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_txc_items": 0, 2024-03-18T15:28:44.769 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T15:28:44.769 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T15:28:44.769 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_writing_bytes": 0, 2024-03-18T15:28:44.769 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_writing_items": 0, 2024-03-18T15:28:44.769 INFO:teuthology.orchestra.run.smithi019.stdout: "bluefs_bytes": 0, 2024-03-18T15:28:44.769 INFO:teuthology.orchestra.run.smithi019.stdout: "bluefs_items": 0, 2024-03-18T15:28:44.770 INFO:teuthology.orchestra.run.smithi019.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T15:28:44.770 INFO:teuthology.orchestra.run.smithi019.stdout: "bluefs_file_reader_items": 0, 2024-03-18T15:28:44.770 INFO:teuthology.orchestra.run.smithi019.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T15:28:44.770 INFO:teuthology.orchestra.run.smithi019.stdout: "bluefs_file_writer_items": 0, 2024-03-18T15:28:44.770 INFO:teuthology.orchestra.run.smithi019.stdout: "buffer_anon_bytes": 183112, 2024-03-18T15:28:44.770 INFO:teuthology.orchestra.run.smithi019.stdout: "buffer_anon_items": 49, 2024-03-18T15:28:44.770 INFO:teuthology.orchestra.run.smithi019.stdout: "buffer_meta_bytes": 0, 2024-03-18T15:28:44.770 INFO:teuthology.orchestra.run.smithi019.stdout: "buffer_meta_items": 0, 2024-03-18T15:28:44.770 INFO:teuthology.orchestra.run.smithi019.stdout: "osd_bytes": 0, 2024-03-18T15:28:44.770 INFO:teuthology.orchestra.run.smithi019.stdout: "osd_items": 0, 2024-03-18T15:28:44.770 INFO:teuthology.orchestra.run.smithi019.stdout: "osd_mapbl_bytes": 0, 2024-03-18T15:28:44.770 INFO:teuthology.orchestra.run.smithi019.stdout: "osd_mapbl_items": 0, 2024-03-18T15:28:44.770 INFO:teuthology.orchestra.run.smithi019.stdout: "osd_pglog_bytes": 0, 2024-03-18T15:28:44.770 INFO:teuthology.orchestra.run.smithi019.stdout: "osd_pglog_items": 0, 2024-03-18T15:28:44.771 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap_bytes": 9928, 2024-03-18T15:28:44.771 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap_items": 207, 2024-03-18T15:28:44.771 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T15:28:44.771 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap_mapping_items": 0, 2024-03-18T15:28:44.771 INFO:teuthology.orchestra.run.smithi019.stdout: "pgmap_bytes": 0, 2024-03-18T15:28:44.771 INFO:teuthology.orchestra.run.smithi019.stdout: "pgmap_items": 0, 2024-03-18T15:28:44.771 INFO:teuthology.orchestra.run.smithi019.stdout: "mds_co_bytes": 8464, 2024-03-18T15:28:44.771 INFO:teuthology.orchestra.run.smithi019.stdout: "mds_co_items": 13, 2024-03-18T15:28:44.771 INFO:teuthology.orchestra.run.smithi019.stdout: "unittest_1_bytes": 0, 2024-03-18T15:28:44.771 INFO:teuthology.orchestra.run.smithi019.stdout: "unittest_1_items": 0, 2024-03-18T15:28:44.771 INFO:teuthology.orchestra.run.smithi019.stdout: "unittest_2_bytes": 0, 2024-03-18T15:28:44.771 INFO:teuthology.orchestra.run.smithi019.stdout: "unittest_2_items": 0 2024-03-18T15:28:44.771 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:44.771 INFO:teuthology.orchestra.run.smithi019.stdout: "objecter": { 2024-03-18T15:28:44.771 INFO:teuthology.orchestra.run.smithi019.stdout: "op_active": 0, 2024-03-18T15:28:44.772 INFO:teuthology.orchestra.run.smithi019.stdout: "op_laggy": 0, 2024-03-18T15:28:44.772 INFO:teuthology.orchestra.run.smithi019.stdout: "op_send": 2187, 2024-03-18T15:28:44.772 INFO:teuthology.orchestra.run.smithi019.stdout: "op_send_bytes": 21, 2024-03-18T15:28:44.772 INFO:teuthology.orchestra.run.smithi019.stdout: "op_resend": 3, 2024-03-18T15:28:44.772 INFO:teuthology.orchestra.run.smithi019.stdout: "op_reply": 2187, 2024-03-18T15:28:44.772 INFO:teuthology.orchestra.run.smithi019.stdout: "op_latency": { 2024-03-18T15:28:44.772 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 2187, 2024-03-18T15:28:44.772 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 2.055977453, 2024-03-18T15:28:44.772 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000940090 2024-03-18T15:28:44.772 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:44.772 INFO:teuthology.orchestra.run.smithi019.stdout: "op_inflight": 0, 2024-03-18T15:28:44.772 INFO:teuthology.orchestra.run.smithi019.stdout: "oplen_avg": { 2024-03-18T15:28:44.772 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 2187, 2024-03-18T15:28:44.772 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 2188 2024-03-18T15:28:44.772 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:44.773 INFO:teuthology.orchestra.run.smithi019.stdout: "op": 2187, 2024-03-18T15:28:44.773 INFO:teuthology.orchestra.run.smithi019.stdout: "op_r": 2187, 2024-03-18T15:28:44.773 INFO:teuthology.orchestra.run.smithi019.stdout: "op_w": 0, 2024-03-18T15:28:44.773 INFO:teuthology.orchestra.run.smithi019.stdout: "op_rmw": 0, 2024-03-18T15:28:44.773 INFO:teuthology.orchestra.run.smithi019.stdout: "op_pg": 0, 2024-03-18T15:28:44.773 INFO:teuthology.orchestra.run.smithi019.stdout: "osdop_stat": 1403, 2024-03-18T15:28:44.773 INFO:teuthology.orchestra.run.smithi019.stdout: "osdop_create": 0, 2024-03-18T15:28:44.773 INFO:teuthology.orchestra.run.smithi019.stdout: "osdop_read": 782, 2024-03-18T15:28:44.773 INFO:teuthology.orchestra.run.smithi019.stdout: "osdop_write": 0, 2024-03-18T15:28:44.773 INFO:teuthology.orchestra.run.smithi019.stdout: "osdop_writefull": 0, 2024-03-18T15:28:44.773 INFO:teuthology.orchestra.run.smithi019.stdout: "osdop_writesame": 0, 2024-03-18T15:28:44.773 INFO:teuthology.orchestra.run.smithi019.stdout: "osdop_append": 0, 2024-03-18T15:28:44.773 INFO:teuthology.orchestra.run.smithi019.stdout: "osdop_zero": 0, 2024-03-18T15:28:44.773 INFO:teuthology.orchestra.run.smithi019.stdout: "osdop_truncate": 0, 2024-03-18T15:28:44.773 INFO:teuthology.orchestra.run.smithi019.stdout: "osdop_delete": 0, 2024-03-18T15:28:44.774 INFO:teuthology.orchestra.run.smithi019.stdout: "osdop_mapext": 0, 2024-03-18T15:28:44.774 INFO:teuthology.orchestra.run.smithi019.stdout: "osdop_sparse_read": 0, 2024-03-18T15:28:44.774 INFO:teuthology.orchestra.run.smithi019.stdout: "osdop_clonerange": 0, 2024-03-18T15:28:44.774 INFO:teuthology.orchestra.run.smithi019.stdout: "osdop_getxattr": 1, 2024-03-18T15:28:44.774 INFO:teuthology.orchestra.run.smithi019.stdout: "osdop_setxattr": 0, 2024-03-18T15:28:44.774 INFO:teuthology.orchestra.run.smithi019.stdout: "osdop_cmpxattr": 0, 2024-03-18T15:28:44.774 INFO:teuthology.orchestra.run.smithi019.stdout: "osdop_rmxattr": 0, 2024-03-18T15:28:44.774 INFO:teuthology.orchestra.run.smithi019.stdout: "osdop_resetxattrs": 0, 2024-03-18T15:28:44.774 INFO:teuthology.orchestra.run.smithi019.stdout: "osdop_call": 0, 2024-03-18T15:28:44.774 INFO:teuthology.orchestra.run.smithi019.stdout: "osdop_watch": 0, 2024-03-18T15:28:44.774 INFO:teuthology.orchestra.run.smithi019.stdout: "osdop_notify": 0, 2024-03-18T15:28:44.774 INFO:teuthology.orchestra.run.smithi019.stdout: "osdop_src_cmpxattr": 0, 2024-03-18T15:28:44.774 INFO:teuthology.orchestra.run.smithi019.stdout: "osdop_pgls": 0, 2024-03-18T15:28:44.774 INFO:teuthology.orchestra.run.smithi019.stdout: "osdop_pgls_filter": 0, 2024-03-18T15:28:44.774 INFO:teuthology.orchestra.run.smithi019.stdout: "osdop_other": 0, 2024-03-18T15:28:44.775 INFO:teuthology.orchestra.run.smithi019.stdout: "linger_active": 0, 2024-03-18T15:28:44.775 INFO:teuthology.orchestra.run.smithi019.stdout: "linger_send": 0, 2024-03-18T15:28:44.775 INFO:teuthology.orchestra.run.smithi019.stdout: "linger_resend": 0, 2024-03-18T15:28:44.775 INFO:teuthology.orchestra.run.smithi019.stdout: "linger_ping": 0, 2024-03-18T15:28:44.775 INFO:teuthology.orchestra.run.smithi019.stdout: "poolop_active": 0, 2024-03-18T15:28:44.775 INFO:teuthology.orchestra.run.smithi019.stdout: "poolop_send": 0, 2024-03-18T15:28:44.775 INFO:teuthology.orchestra.run.smithi019.stdout: "poolop_resend": 0, 2024-03-18T15:28:44.775 INFO:teuthology.orchestra.run.smithi019.stdout: "poolstat_active": 0, 2024-03-18T15:28:44.775 INFO:teuthology.orchestra.run.smithi019.stdout: "poolstat_send": 0, 2024-03-18T15:28:44.775 INFO:teuthology.orchestra.run.smithi019.stdout: "poolstat_resend": 0, 2024-03-18T15:28:44.775 INFO:teuthology.orchestra.run.smithi019.stdout: "statfs_active": 0, 2024-03-18T15:28:44.775 INFO:teuthology.orchestra.run.smithi019.stdout: "statfs_send": 0, 2024-03-18T15:28:44.775 INFO:teuthology.orchestra.run.smithi019.stdout: "statfs_resend": 0, 2024-03-18T15:28:44.775 INFO:teuthology.orchestra.run.smithi019.stdout: "command_active": 0, 2024-03-18T15:28:44.775 INFO:teuthology.orchestra.run.smithi019.stdout: "command_send": 0, 2024-03-18T15:28:44.776 INFO:teuthology.orchestra.run.smithi019.stdout: "command_resend": 0, 2024-03-18T15:28:44.776 INFO:teuthology.orchestra.run.smithi019.stdout: "map_epoch": 103, 2024-03-18T15:28:44.776 INFO:teuthology.orchestra.run.smithi019.stdout: "map_full": 0, 2024-03-18T15:28:44.776 INFO:teuthology.orchestra.run.smithi019.stdout: "map_inc": 14, 2024-03-18T15:28:44.776 INFO:teuthology.orchestra.run.smithi019.stdout: "osd_sessions": 4, 2024-03-18T15:28:44.776 INFO:teuthology.orchestra.run.smithi019.stdout: "osd_session_open": 4, 2024-03-18T15:28:44.776 INFO:teuthology.orchestra.run.smithi019.stdout: "osd_session_close": 0, 2024-03-18T15:28:44.776 INFO:teuthology.orchestra.run.smithi019.stdout: "osd_laggy": 0, 2024-03-18T15:28:44.776 INFO:teuthology.orchestra.run.smithi019.stdout: "omap_wr": 0, 2024-03-18T15:28:44.776 INFO:teuthology.orchestra.run.smithi019.stdout: "omap_rd": 2, 2024-03-18T15:28:44.776 INFO:teuthology.orchestra.run.smithi019.stdout: "omap_del": 0 2024-03-18T15:28:44.776 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:44.776 INFO:teuthology.orchestra.run.smithi019.stdout: "oft": { 2024-03-18T15:28:44.776 INFO:teuthology.orchestra.run.smithi019.stdout: "omap_total_objs": 0, 2024-03-18T15:28:44.776 INFO:teuthology.orchestra.run.smithi019.stdout: "omap_total_kv_pairs": 0, 2024-03-18T15:28:44.777 INFO:teuthology.orchestra.run.smithi019.stdout: "omap_total_updates": 0, 2024-03-18T15:28:44.777 INFO:teuthology.orchestra.run.smithi019.stdout: "omap_total_removes": 0 2024-03-18T15:28:44.777 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:44.777 INFO:teuthology.orchestra.run.smithi019.stdout: "osd-slow-ops": { 2024-03-18T15:28:44.777 INFO:teuthology.orchestra.run.smithi019.stdout: "slow_ops_count": 0 2024-03-18T15:28:44.777 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:44.777 INFO:teuthology.orchestra.run.smithi019.stdout: "purge_queue": { 2024-03-18T15:28:44.777 INFO:teuthology.orchestra.run.smithi019.stdout: "pq_executing_ops": 0, 2024-03-18T15:28:44.777 INFO:teuthology.orchestra.run.smithi019.stdout: "pq_executing_ops_high_water": 0, 2024-03-18T15:28:44.777 INFO:teuthology.orchestra.run.smithi019.stdout: "pq_executing": 0, 2024-03-18T15:28:44.777 INFO:teuthology.orchestra.run.smithi019.stdout: "pq_executing_high_water": 0, 2024-03-18T15:28:44.777 INFO:teuthology.orchestra.run.smithi019.stdout: "pq_executed_ops": 0, 2024-03-18T15:28:44.777 INFO:teuthology.orchestra.run.smithi019.stdout: "pq_executed": 0, 2024-03-18T15:28:44.777 INFO:teuthology.orchestra.run.smithi019.stdout: "pq_item_in_journal": 0 2024-03-18T15:28:44.777 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:44.777 INFO:teuthology.orchestra.run.smithi019.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T15:28:44.778 INFO:teuthology.orchestra.run.smithi019.stdout: "val": 0, 2024-03-18T15:28:44.778 INFO:teuthology.orchestra.run.smithi019.stdout: "max": 104857600, 2024-03-18T15:28:44.778 INFO:teuthology.orchestra.run.smithi019.stdout: "get_started": 0, 2024-03-18T15:28:44.778 INFO:teuthology.orchestra.run.smithi019.stdout: "get": 2452, 2024-03-18T15:28:44.778 INFO:teuthology.orchestra.run.smithi019.stdout: "get_sum": 2165827, 2024-03-18T15:28:44.778 INFO:teuthology.orchestra.run.smithi019.stdout: "get_or_fail_fail": 0, 2024-03-18T15:28:44.778 INFO:teuthology.orchestra.run.smithi019.stdout: "get_or_fail_success": 2452, 2024-03-18T15:28:44.778 INFO:teuthology.orchestra.run.smithi019.stdout: "take": 0, 2024-03-18T15:28:44.778 INFO:teuthology.orchestra.run.smithi019.stdout: "take_sum": 0, 2024-03-18T15:28:44.778 INFO:teuthology.orchestra.run.smithi019.stdout: "put": 2452, 2024-03-18T15:28:44.778 INFO:teuthology.orchestra.run.smithi019.stdout: "put_sum": 2165827, 2024-03-18T15:28:44.778 INFO:teuthology.orchestra.run.smithi019.stdout: "wait": { 2024-03-18T15:28:44.778 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 0, 2024-03-18T15:28:44.778 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.000000000, 2024-03-18T15:28:44.779 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000000000 2024-03-18T15:28:44.779 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:44.779 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:44.779 INFO:teuthology.orchestra.run.smithi019.stdout: "throttle-objecter_bytes": { 2024-03-18T15:28:44.779 INFO:teuthology.orchestra.run.smithi019.stdout: "val": 0, 2024-03-18T15:28:44.779 INFO:teuthology.orchestra.run.smithi019.stdout: "max": 104857600, 2024-03-18T15:28:44.779 INFO:teuthology.orchestra.run.smithi019.stdout: "get_started": 0, 2024-03-18T15:28:44.779 INFO:teuthology.orchestra.run.smithi019.stdout: "get": 0, 2024-03-18T15:28:44.779 INFO:teuthology.orchestra.run.smithi019.stdout: "get_sum": 0, 2024-03-18T15:28:44.779 INFO:teuthology.orchestra.run.smithi019.stdout: "get_or_fail_fail": 0, 2024-03-18T15:28:44.779 INFO:teuthology.orchestra.run.smithi019.stdout: "get_or_fail_success": 0, 2024-03-18T15:28:44.779 INFO:teuthology.orchestra.run.smithi019.stdout: "take": 2187, 2024-03-18T15:28:44.779 INFO:teuthology.orchestra.run.smithi019.stdout: "take_sum": 1294, 2024-03-18T15:28:44.779 INFO:teuthology.orchestra.run.smithi019.stdout: "put": 2187, 2024-03-18T15:28:44.779 INFO:teuthology.orchestra.run.smithi019.stdout: "put_sum": 1294, 2024-03-18T15:28:44.779 INFO:teuthology.orchestra.run.smithi019.stdout: "wait": { 2024-03-18T15:28:44.780 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 0, 2024-03-18T15:28:44.780 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.000000000, 2024-03-18T15:28:44.780 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000000000 2024-03-18T15:28:44.780 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:44.780 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:44.780 INFO:teuthology.orchestra.run.smithi019.stdout: "throttle-objecter_ops": { 2024-03-18T15:28:44.780 INFO:teuthology.orchestra.run.smithi019.stdout: "val": 0, 2024-03-18T15:28:44.780 INFO:teuthology.orchestra.run.smithi019.stdout: "max": 1024, 2024-03-18T15:28:44.780 INFO:teuthology.orchestra.run.smithi019.stdout: "get_started": 0, 2024-03-18T15:28:44.780 INFO:teuthology.orchestra.run.smithi019.stdout: "get": 0, 2024-03-18T15:28:44.780 INFO:teuthology.orchestra.run.smithi019.stdout: "get_sum": 0, 2024-03-18T15:28:44.780 INFO:teuthology.orchestra.run.smithi019.stdout: "get_or_fail_fail": 0, 2024-03-18T15:28:44.780 INFO:teuthology.orchestra.run.smithi019.stdout: "get_or_fail_success": 0, 2024-03-18T15:28:44.780 INFO:teuthology.orchestra.run.smithi019.stdout: "take": 2187, 2024-03-18T15:28:44.780 INFO:teuthology.orchestra.run.smithi019.stdout: "take_sum": 2187, 2024-03-18T15:28:44.781 INFO:teuthology.orchestra.run.smithi019.stdout: "put": 2187, 2024-03-18T15:28:44.781 INFO:teuthology.orchestra.run.smithi019.stdout: "put_sum": 2187, 2024-03-18T15:28:44.781 INFO:teuthology.orchestra.run.smithi019.stdout: "wait": { 2024-03-18T15:28:44.781 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 0, 2024-03-18T15:28:44.781 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.000000000, 2024-03-18T15:28:44.781 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000000000 2024-03-18T15:28:44.781 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:44.781 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:44.781 INFO:teuthology.orchestra.run.smithi019.stdout: "throttle-write_buf_throttle": { 2024-03-18T15:28:44.781 INFO:teuthology.orchestra.run.smithi019.stdout: "val": 0, 2024-03-18T15:28:44.781 INFO:teuthology.orchestra.run.smithi019.stdout: "max": 3758096384, 2024-03-18T15:28:44.781 INFO:teuthology.orchestra.run.smithi019.stdout: "get_started": 0, 2024-03-18T15:28:44.781 INFO:teuthology.orchestra.run.smithi019.stdout: "get": 0, 2024-03-18T15:28:44.781 INFO:teuthology.orchestra.run.smithi019.stdout: "get_sum": 0, 2024-03-18T15:28:44.782 INFO:teuthology.orchestra.run.smithi019.stdout: "get_or_fail_fail": 0, 2024-03-18T15:28:44.782 INFO:teuthology.orchestra.run.smithi019.stdout: "get_or_fail_success": 0, 2024-03-18T15:28:44.782 INFO:teuthology.orchestra.run.smithi019.stdout: "take": 0, 2024-03-18T15:28:44.783 INFO:teuthology.orchestra.run.smithi019.stdout: "take_sum": 0, 2024-03-18T15:28:44.783 INFO:teuthology.orchestra.run.smithi019.stdout: "put": 0, 2024-03-18T15:28:44.783 INFO:teuthology.orchestra.run.smithi019.stdout: "put_sum": 0, 2024-03-18T15:28:44.783 INFO:teuthology.orchestra.run.smithi019.stdout: "wait": { 2024-03-18T15:28:44.783 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 0, 2024-03-18T15:28:44.783 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.000000000, 2024-03-18T15:28:44.783 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000000000 2024-03-18T15:28:44.783 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:44.783 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:44.783 INFO:teuthology.orchestra.run.smithi019.stdout: "throttle-write_buf_throttle-0x556abbbc4780": { 2024-03-18T15:28:44.783 INFO:teuthology.orchestra.run.smithi019.stdout: "val": 0, 2024-03-18T15:28:44.783 INFO:teuthology.orchestra.run.smithi019.stdout: "max": 3758096384, 2024-03-18T15:28:44.783 INFO:teuthology.orchestra.run.smithi019.stdout: "get_started": 0, 2024-03-18T15:28:44.783 INFO:teuthology.orchestra.run.smithi019.stdout: "get": 0, 2024-03-18T15:28:44.784 INFO:teuthology.orchestra.run.smithi019.stdout: "get_sum": 0, 2024-03-18T15:28:44.784 INFO:teuthology.orchestra.run.smithi019.stdout: "get_or_fail_fail": 0, 2024-03-18T15:28:44.784 INFO:teuthology.orchestra.run.smithi019.stdout: "get_or_fail_success": 0, 2024-03-18T15:28:44.784 INFO:teuthology.orchestra.run.smithi019.stdout: "take": 0, 2024-03-18T15:28:44.784 INFO:teuthology.orchestra.run.smithi019.stdout: "take_sum": 0, 2024-03-18T15:28:44.784 INFO:teuthology.orchestra.run.smithi019.stdout: "put": 0, 2024-03-18T15:28:44.784 INFO:teuthology.orchestra.run.smithi019.stdout: "put_sum": 0, 2024-03-18T15:28:44.784 INFO:teuthology.orchestra.run.smithi019.stdout: "wait": { 2024-03-18T15:28:44.784 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 0, 2024-03-18T15:28:44.784 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.000000000, 2024-03-18T15:28:44.784 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000000000 2024-03-18T15:28:44.784 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:44.784 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:44.784 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-03-18T15:28:45.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:28:45 smithi019 ceph-mon[31522]: pgmap v319: 129 pgs: 129 active+clean; 1.7 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-18T15:28:45.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:28:45 smithi033 ceph-mon[36648]: pgmap v319: 129 pgs: 129 active+clean; 1.7 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-18T15:28:45.885 INFO:tasks.check_counter:Daemon mds.d mds.exported=0 2024-03-18T15:28:45.885 INFO:tasks.check_counter:Daemon mds.d mds.imported=0 2024-03-18T15:28:45.885 INFO:tasks.check_counter:Daemon mds.d mds_cache.dir_update=0 2024-03-18T15:28:45.885 INFO:tasks.check_counter:Daemon mds.d mds_cache.dir_update_receipt=0 2024-03-18T15:28:45.885 INFO:tasks.check_counter:Daemon mds.d mds.root_rsnaps=0 2024-03-18T15:28:45.885 INFO:tasks.check_counter:Daemon mds.d mds_server.req_mksnap_latency.avgcount=0 2024-03-18T15:28:45.885 INFO:tasks.check_counter:Daemon mds.d mds_server.req_rmsnap_latency.avgcount=0 2024-03-18T15:28:45.885 DEBUG:tasks.check_counter:Getting stats from g 2024-03-18T15:28:45.886 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 -- ceph daemon mds.g perf dump 2024-03-18T15:28:45.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:28:45 smithi104 ceph-mon[36070]: pgmap v319: 129 pgs: 129 active+clean; 1.7 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-18T15:28:46.376 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:28:47.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:28:47 smithi019 ceph-mon[31522]: pgmap v320: 129 pgs: 129 active+clean; 1.7 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 0 B/s wr, 14 op/s 2024-03-18T15:28:47.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:28:47 smithi033 ceph-mon[36648]: pgmap v320: 129 pgs: 129 active+clean; 1.7 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 0 B/s wr, 14 op/s 2024-03-18T15:28:47.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:28:47 smithi104 ceph-mon[36070]: pgmap v320: 129 pgs: 129 active+clean; 1.7 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 0 B/s wr, 14 op/s 2024-03-18T15:28:48.786 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-03-18T15:28:48.786 INFO:teuthology.orchestra.run.smithi019.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T15:28:48.786 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_recv_messages": 0, 2024-03-18T15:28:48.786 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_send_messages": 2, 2024-03-18T15:28:48.786 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_recv_bytes": 0, 2024-03-18T15:28:48.786 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_send_bytes": 0, 2024-03-18T15:28:48.786 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_created_connections": 4, 2024-03-18T15:28:48.786 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_active_connections": 0, 2024-03-18T15:28:48.786 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_running_total_time": 0.004332537, 2024-03-18T15:28:48.786 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_running_send_time": 0.000000000, 2024-03-18T15:28:48.786 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_running_recv_time": 0.000761463, 2024-03-18T15:28:48.787 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-18T15:28:48.787 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T15:28:48.787 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 0, 2024-03-18T15:28:48.787 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.000000000, 2024-03-18T15:28:48.787 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000000000 2024-03-18T15:28:48.787 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:48.787 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_handle_ack_lat": { 2024-03-18T15:28:48.787 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 0, 2024-03-18T15:28:48.787 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.000000000, 2024-03-18T15:28:48.787 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000000000 2024-03-18T15:28:48.787 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:48.787 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-18T15:28:48.787 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_send_encrypted_bytes": 0 2024-03-18T15:28:48.787 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:48.788 INFO:teuthology.orchestra.run.smithi019.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T15:28:48.788 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_recv_messages": 234, 2024-03-18T15:28:48.788 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_send_messages": 222, 2024-03-18T15:28:48.788 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_recv_bytes": 1715280, 2024-03-18T15:28:48.788 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_send_bytes": 136864, 2024-03-18T15:28:48.788 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_created_connections": 2, 2024-03-18T15:28:48.788 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_active_connections": 1, 2024-03-18T15:28:48.788 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_running_total_time": 0.099834844, 2024-03-18T15:28:48.788 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_running_send_time": 0.034950327, 2024-03-18T15:28:48.788 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_running_recv_time": 0.072942860, 2024-03-18T15:28:48.788 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_running_fast_dispatch_time": 0.010293859, 2024-03-18T15:28:48.788 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T15:28:48.788 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 221, 2024-03-18T15:28:48.788 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.020746790, 2024-03-18T15:28:48.788 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000093876 2024-03-18T15:28:48.789 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:48.789 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_handle_ack_lat": { 2024-03-18T15:28:48.789 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 0, 2024-03-18T15:28:48.789 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.000000000, 2024-03-18T15:28:48.789 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000000000 2024-03-18T15:28:48.789 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:48.789 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_recv_encrypted_bytes": 1715280, 2024-03-18T15:28:48.789 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_send_encrypted_bytes": 136864 2024-03-18T15:28:48.789 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:48.789 INFO:teuthology.orchestra.run.smithi019.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T15:28:48.789 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_recv_messages": 6, 2024-03-18T15:28:48.789 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_send_messages": 172, 2024-03-18T15:28:48.789 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_recv_bytes": 3392, 2024-03-18T15:28:48.789 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_send_bytes": 186064, 2024-03-18T15:28:48.789 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_created_connections": 5, 2024-03-18T15:28:48.790 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_active_connections": 3, 2024-03-18T15:28:48.790 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_running_total_time": 0.036865315, 2024-03-18T15:28:48.790 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_running_send_time": 0.026100751, 2024-03-18T15:28:48.790 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_running_recv_time": 0.004764348, 2024-03-18T15:28:48.790 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-18T15:28:48.790 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T15:28:48.790 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 170, 2024-03-18T15:28:48.790 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.029551533, 2024-03-18T15:28:48.790 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000173832 2024-03-18T15:28:48.790 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:48.790 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_handle_ack_lat": { 2024-03-18T15:28:48.790 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 0, 2024-03-18T15:28:48.790 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.000000000, 2024-03-18T15:28:48.790 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000000000 2024-03-18T15:28:48.791 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:48.791 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_recv_encrypted_bytes": 3392, 2024-03-18T15:28:48.791 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_send_encrypted_bytes": 186064 2024-03-18T15:28:48.791 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:48.791 INFO:teuthology.orchestra.run.smithi019.stdout: "cct": { 2024-03-18T15:28:48.791 INFO:teuthology.orchestra.run.smithi019.stdout: "total_workers": 0, 2024-03-18T15:28:48.791 INFO:teuthology.orchestra.run.smithi019.stdout: "unhealthy_workers": 0 2024-03-18T15:28:48.791 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:48.791 INFO:teuthology.orchestra.run.smithi019.stdout: "mempool": { 2024-03-18T15:28:48.791 INFO:teuthology.orchestra.run.smithi019.stdout: "bloom_filter_bytes": 0, 2024-03-18T15:28:48.791 INFO:teuthology.orchestra.run.smithi019.stdout: "bloom_filter_items": 0, 2024-03-18T15:28:48.791 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T15:28:48.791 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_alloc_items": 0, 2024-03-18T15:28:48.791 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T15:28:48.791 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_cache_data_items": 0, 2024-03-18T15:28:48.792 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T15:28:48.792 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T15:28:48.792 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T15:28:48.792 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T15:28:48.792 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T15:28:48.792 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_cache_other_items": 0, 2024-03-18T15:28:48.792 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T15:28:48.792 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T15:28:48.792 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_extent_bytes": 0, 2024-03-18T15:28:48.792 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_extent_items": 0, 2024-03-18T15:28:48.792 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_blob_bytes": 0, 2024-03-18T15:28:48.792 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_blob_items": 0, 2024-03-18T15:28:48.792 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T15:28:48.792 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T15:28:48.792 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T15:28:48.793 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T15:28:48.793 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T15:28:48.793 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_fsck_items": 0, 2024-03-18T15:28:48.793 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_txc_bytes": 0, 2024-03-18T15:28:48.793 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_txc_items": 0, 2024-03-18T15:28:48.793 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T15:28:48.793 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T15:28:48.793 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_writing_bytes": 0, 2024-03-18T15:28:48.793 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_writing_items": 0, 2024-03-18T15:28:48.793 INFO:teuthology.orchestra.run.smithi019.stdout: "bluefs_bytes": 0, 2024-03-18T15:28:48.793 INFO:teuthology.orchestra.run.smithi019.stdout: "bluefs_items": 0, 2024-03-18T15:28:48.793 INFO:teuthology.orchestra.run.smithi019.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T15:28:48.793 INFO:teuthology.orchestra.run.smithi019.stdout: "bluefs_file_reader_items": 0, 2024-03-18T15:28:48.793 INFO:teuthology.orchestra.run.smithi019.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T15:28:48.794 INFO:teuthology.orchestra.run.smithi019.stdout: "bluefs_file_writer_items": 0, 2024-03-18T15:28:48.794 INFO:teuthology.orchestra.run.smithi019.stdout: "buffer_anon_bytes": 149762, 2024-03-18T15:28:48.794 INFO:teuthology.orchestra.run.smithi019.stdout: "buffer_anon_items": 32, 2024-03-18T15:28:48.794 INFO:teuthology.orchestra.run.smithi019.stdout: "buffer_meta_bytes": 0, 2024-03-18T15:28:48.794 INFO:teuthology.orchestra.run.smithi019.stdout: "buffer_meta_items": 0, 2024-03-18T15:28:48.794 INFO:teuthology.orchestra.run.smithi019.stdout: "osd_bytes": 0, 2024-03-18T15:28:48.794 INFO:teuthology.orchestra.run.smithi019.stdout: "osd_items": 0, 2024-03-18T15:28:48.794 INFO:teuthology.orchestra.run.smithi019.stdout: "osd_mapbl_bytes": 0, 2024-03-18T15:28:48.794 INFO:teuthology.orchestra.run.smithi019.stdout: "osd_mapbl_items": 0, 2024-03-18T15:28:48.794 INFO:teuthology.orchestra.run.smithi019.stdout: "osd_pglog_bytes": 0, 2024-03-18T15:28:48.794 INFO:teuthology.orchestra.run.smithi019.stdout: "osd_pglog_items": 0, 2024-03-18T15:28:48.794 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap_bytes": 0, 2024-03-18T15:28:48.794 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap_items": 0, 2024-03-18T15:28:48.794 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T15:28:48.794 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap_mapping_items": 0, 2024-03-18T15:28:48.795 INFO:teuthology.orchestra.run.smithi019.stdout: "pgmap_bytes": 0, 2024-03-18T15:28:48.795 INFO:teuthology.orchestra.run.smithi019.stdout: "pgmap_items": 0, 2024-03-18T15:28:48.795 INFO:teuthology.orchestra.run.smithi019.stdout: "mds_co_bytes": 928, 2024-03-18T15:28:48.795 INFO:teuthology.orchestra.run.smithi019.stdout: "mds_co_items": 2, 2024-03-18T15:28:48.795 INFO:teuthology.orchestra.run.smithi019.stdout: "unittest_1_bytes": 0, 2024-03-18T15:28:48.795 INFO:teuthology.orchestra.run.smithi019.stdout: "unittest_1_items": 0, 2024-03-18T15:28:48.795 INFO:teuthology.orchestra.run.smithi019.stdout: "unittest_2_bytes": 0, 2024-03-18T15:28:48.795 INFO:teuthology.orchestra.run.smithi019.stdout: "unittest_2_items": 0 2024-03-18T15:28:48.795 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:48.795 INFO:teuthology.orchestra.run.smithi019.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T15:28:48.795 INFO:teuthology.orchestra.run.smithi019.stdout: "val": 0, 2024-03-18T15:28:48.795 INFO:teuthology.orchestra.run.smithi019.stdout: "max": 104857600, 2024-03-18T15:28:48.795 INFO:teuthology.orchestra.run.smithi019.stdout: "get_started": 0, 2024-03-18T15:28:48.795 INFO:teuthology.orchestra.run.smithi019.stdout: "get": 236, 2024-03-18T15:28:48.795 INFO:teuthology.orchestra.run.smithi019.stdout: "get_sum": 1682310, 2024-03-18T15:28:48.796 INFO:teuthology.orchestra.run.smithi019.stdout: "get_or_fail_fail": 0, 2024-03-18T15:28:48.796 INFO:teuthology.orchestra.run.smithi019.stdout: "get_or_fail_success": 236, 2024-03-18T15:28:48.796 INFO:teuthology.orchestra.run.smithi019.stdout: "take": 0, 2024-03-18T15:28:48.796 INFO:teuthology.orchestra.run.smithi019.stdout: "take_sum": 0, 2024-03-18T15:28:48.796 INFO:teuthology.orchestra.run.smithi019.stdout: "put": 236, 2024-03-18T15:28:48.796 INFO:teuthology.orchestra.run.smithi019.stdout: "put_sum": 1682310, 2024-03-18T15:28:48.796 INFO:teuthology.orchestra.run.smithi019.stdout: "wait": { 2024-03-18T15:28:48.796 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 0, 2024-03-18T15:28:48.796 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.000000000, 2024-03-18T15:28:48.796 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000000000 2024-03-18T15:28:48.796 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:48.797 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:48.797 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-03-18T15:28:49.601 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.g 2024-03-18T15:28:49.601 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.g 2024-03-18T15:28:49.601 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.g 2024-03-18T15:28:49.601 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.g 2024-03-18T15:28:49.601 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.g 2024-03-18T15:28:49.601 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.g 2024-03-18T15:28:49.601 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.g 2024-03-18T15:28:49.601 DEBUG:tasks.check_counter:Getting stats from j 2024-03-18T15:28:49.602 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 -- ceph daemon mds.j perf dump 2024-03-18T15:28:49.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:28:49 smithi033 ceph-mon[36648]: pgmap v321: 129 pgs: 129 active+clean; 1.7 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:28:49.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:28:49 smithi104 ceph-mon[36070]: pgmap v321: 129 pgs: 129 active+clean; 1.7 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:28:49.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:28:49 smithi019 ceph-mon[31522]: pgmap v321: 129 pgs: 129 active+clean; 1.7 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:28:50.059 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:28:51.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:28:51 smithi019 ceph-mon[31522]: pgmap v322: 129 pgs: 129 active+clean; 1.7 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:28:51.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:28:51 smithi033 ceph-mon[36648]: pgmap v322: 129 pgs: 129 active+clean; 1.7 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:28:51.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:28:51 smithi104 ceph-mon[36070]: pgmap v322: 129 pgs: 129 active+clean; 1.7 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:28:52.338 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-03-18T15:28:52.338 INFO:teuthology.orchestra.run.smithi019.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T15:28:52.339 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_recv_messages": 245, 2024-03-18T15:28:52.339 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_send_messages": 231, 2024-03-18T15:28:52.339 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_recv_bytes": 1720256, 2024-03-18T15:28:52.339 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_send_bytes": 141792, 2024-03-18T15:28:52.339 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_created_connections": 4, 2024-03-18T15:28:52.339 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_active_connections": 0, 2024-03-18T15:28:52.339 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_running_total_time": 0.119875283, 2024-03-18T15:28:52.339 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_running_send_time": 0.035905402, 2024-03-18T15:28:52.339 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_running_recv_time": 0.104576277, 2024-03-18T15:28:52.339 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_running_fast_dispatch_time": 0.010680815, 2024-03-18T15:28:52.339 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T15:28:52.339 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 231, 2024-03-18T15:28:52.339 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.023595209, 2024-03-18T15:28:52.340 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000102143 2024-03-18T15:28:52.340 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:52.340 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_handle_ack_lat": { 2024-03-18T15:28:52.340 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 0, 2024-03-18T15:28:52.340 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.000000000, 2024-03-18T15:28:52.340 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000000000 2024-03-18T15:28:52.340 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:52.340 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_recv_encrypted_bytes": 1720256, 2024-03-18T15:28:52.340 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_send_encrypted_bytes": 141792 2024-03-18T15:28:52.340 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:52.340 INFO:teuthology.orchestra.run.smithi019.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T15:28:52.340 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_recv_messages": 2, 2024-03-18T15:28:52.340 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_send_messages": 176, 2024-03-18T15:28:52.340 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_recv_bytes": 320, 2024-03-18T15:28:52.340 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_send_bytes": 186672, 2024-03-18T15:28:52.341 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_created_connections": 5, 2024-03-18T15:28:52.341 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_active_connections": 3, 2024-03-18T15:28:52.341 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_running_total_time": 0.034943625, 2024-03-18T15:28:52.341 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_running_send_time": 0.025704888, 2024-03-18T15:28:52.341 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_running_recv_time": 0.003262761, 2024-03-18T15:28:52.341 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-18T15:28:52.341 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T15:28:52.341 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 173, 2024-03-18T15:28:52.341 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.026874068, 2024-03-18T15:28:52.341 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000155341 2024-03-18T15:28:52.341 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:52.341 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_handle_ack_lat": { 2024-03-18T15:28:52.341 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 0, 2024-03-18T15:28:52.341 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.000000000, 2024-03-18T15:28:52.342 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000000000 2024-03-18T15:28:52.342 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:52.342 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-18T15:28:52.342 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_send_encrypted_bytes": 186672 2024-03-18T15:28:52.342 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:52.342 INFO:teuthology.orchestra.run.smithi019.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T15:28:52.342 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_recv_messages": 0, 2024-03-18T15:28:52.342 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_send_messages": 2, 2024-03-18T15:28:52.342 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_recv_bytes": 0, 2024-03-18T15:28:52.342 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_send_bytes": 0, 2024-03-18T15:28:52.342 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_created_connections": 2, 2024-03-18T15:28:52.342 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_active_connections": 1, 2024-03-18T15:28:52.343 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_running_total_time": 0.001224966, 2024-03-18T15:28:52.343 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_running_send_time": 0.000000000, 2024-03-18T15:28:52.343 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_running_recv_time": 0.001053257, 2024-03-18T15:28:52.343 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-18T15:28:52.343 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T15:28:52.343 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 0, 2024-03-18T15:28:52.343 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.000000000, 2024-03-18T15:28:52.343 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000000000 2024-03-18T15:28:52.343 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:52.343 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_handle_ack_lat": { 2024-03-18T15:28:52.343 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 0, 2024-03-18T15:28:52.344 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.000000000, 2024-03-18T15:28:52.344 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000000000 2024-03-18T15:28:52.344 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:52.344 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-18T15:28:52.344 INFO:teuthology.orchestra.run.smithi019.stdout: "msgr_send_encrypted_bytes": 0 2024-03-18T15:28:52.344 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:52.344 INFO:teuthology.orchestra.run.smithi019.stdout: "cct": { 2024-03-18T15:28:52.344 INFO:teuthology.orchestra.run.smithi019.stdout: "total_workers": 0, 2024-03-18T15:28:52.344 INFO:teuthology.orchestra.run.smithi019.stdout: "unhealthy_workers": 0 2024-03-18T15:28:52.344 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:52.344 INFO:teuthology.orchestra.run.smithi019.stdout: "mempool": { 2024-03-18T15:28:52.344 INFO:teuthology.orchestra.run.smithi019.stdout: "bloom_filter_bytes": 0, 2024-03-18T15:28:52.344 INFO:teuthology.orchestra.run.smithi019.stdout: "bloom_filter_items": 0, 2024-03-18T15:28:52.344 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T15:28:52.345 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_alloc_items": 0, 2024-03-18T15:28:52.345 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T15:28:52.345 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_cache_data_items": 0, 2024-03-18T15:28:52.345 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T15:28:52.345 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T15:28:52.345 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T15:28:52.345 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T15:28:52.345 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T15:28:52.345 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_cache_other_items": 0, 2024-03-18T15:28:52.345 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T15:28:52.345 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T15:28:52.345 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_extent_bytes": 0, 2024-03-18T15:28:52.345 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_extent_items": 0, 2024-03-18T15:28:52.345 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_blob_bytes": 0, 2024-03-18T15:28:52.346 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_blob_items": 0, 2024-03-18T15:28:52.346 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T15:28:52.346 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T15:28:52.346 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T15:28:52.346 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T15:28:52.346 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T15:28:52.346 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_fsck_items": 0, 2024-03-18T15:28:52.346 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_txc_bytes": 0, 2024-03-18T15:28:52.346 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_txc_items": 0, 2024-03-18T15:28:52.346 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T15:28:52.346 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T15:28:52.346 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_writing_bytes": 0, 2024-03-18T15:28:52.346 INFO:teuthology.orchestra.run.smithi019.stdout: "bluestore_writing_items": 0, 2024-03-18T15:28:52.346 INFO:teuthology.orchestra.run.smithi019.stdout: "bluefs_bytes": 0, 2024-03-18T15:28:52.346 INFO:teuthology.orchestra.run.smithi019.stdout: "bluefs_items": 0, 2024-03-18T15:28:52.347 INFO:teuthology.orchestra.run.smithi019.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T15:28:52.347 INFO:teuthology.orchestra.run.smithi019.stdout: "bluefs_file_reader_items": 0, 2024-03-18T15:28:52.347 INFO:teuthology.orchestra.run.smithi019.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T15:28:52.347 INFO:teuthology.orchestra.run.smithi019.stdout: "bluefs_file_writer_items": 0, 2024-03-18T15:28:52.347 INFO:teuthology.orchestra.run.smithi019.stdout: "buffer_anon_bytes": 149745, 2024-03-18T15:28:52.347 INFO:teuthology.orchestra.run.smithi019.stdout: "buffer_anon_items": 31, 2024-03-18T15:28:52.347 INFO:teuthology.orchestra.run.smithi019.stdout: "buffer_meta_bytes": 0, 2024-03-18T15:28:52.347 INFO:teuthology.orchestra.run.smithi019.stdout: "buffer_meta_items": 0, 2024-03-18T15:28:52.347 INFO:teuthology.orchestra.run.smithi019.stdout: "osd_bytes": 0, 2024-03-18T15:28:52.347 INFO:teuthology.orchestra.run.smithi019.stdout: "osd_items": 0, 2024-03-18T15:28:52.347 INFO:teuthology.orchestra.run.smithi019.stdout: "osd_mapbl_bytes": 0, 2024-03-18T15:28:52.347 INFO:teuthology.orchestra.run.smithi019.stdout: "osd_mapbl_items": 0, 2024-03-18T15:28:52.347 INFO:teuthology.orchestra.run.smithi019.stdout: "osd_pglog_bytes": 0, 2024-03-18T15:28:52.347 INFO:teuthology.orchestra.run.smithi019.stdout: "osd_pglog_items": 0, 2024-03-18T15:28:52.348 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap_bytes": 0, 2024-03-18T15:28:52.348 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap_items": 0, 2024-03-18T15:28:52.348 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T15:28:52.348 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap_mapping_items": 0, 2024-03-18T15:28:52.348 INFO:teuthology.orchestra.run.smithi019.stdout: "pgmap_bytes": 0, 2024-03-18T15:28:52.348 INFO:teuthology.orchestra.run.smithi019.stdout: "pgmap_items": 0, 2024-03-18T15:28:52.348 INFO:teuthology.orchestra.run.smithi019.stdout: "mds_co_bytes": 928, 2024-03-18T15:28:52.348 INFO:teuthology.orchestra.run.smithi019.stdout: "mds_co_items": 2, 2024-03-18T15:28:52.348 INFO:teuthology.orchestra.run.smithi019.stdout: "unittest_1_bytes": 0, 2024-03-18T15:28:52.348 INFO:teuthology.orchestra.run.smithi019.stdout: "unittest_1_items": 0, 2024-03-18T15:28:52.348 INFO:teuthology.orchestra.run.smithi019.stdout: "unittest_2_bytes": 0, 2024-03-18T15:28:52.348 INFO:teuthology.orchestra.run.smithi019.stdout: "unittest_2_items": 0 2024-03-18T15:28:52.348 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-18T15:28:52.348 INFO:teuthology.orchestra.run.smithi019.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T15:28:52.348 INFO:teuthology.orchestra.run.smithi019.stdout: "val": 0, 2024-03-18T15:28:52.349 INFO:teuthology.orchestra.run.smithi019.stdout: "max": 104857600, 2024-03-18T15:28:52.349 INFO:teuthology.orchestra.run.smithi019.stdout: "get_started": 0, 2024-03-18T15:28:52.349 INFO:teuthology.orchestra.run.smithi019.stdout: "get": 243, 2024-03-18T15:28:52.349 INFO:teuthology.orchestra.run.smithi019.stdout: "get_sum": 1683220, 2024-03-18T15:28:52.349 INFO:teuthology.orchestra.run.smithi019.stdout: "get_or_fail_fail": 0, 2024-03-18T15:28:52.349 INFO:teuthology.orchestra.run.smithi019.stdout: "get_or_fail_success": 243, 2024-03-18T15:28:52.349 INFO:teuthology.orchestra.run.smithi019.stdout: "take": 0, 2024-03-18T15:28:52.349 INFO:teuthology.orchestra.run.smithi019.stdout: "take_sum": 0, 2024-03-18T15:28:52.349 INFO:teuthology.orchestra.run.smithi019.stdout: "put": 243, 2024-03-18T15:28:52.349 INFO:teuthology.orchestra.run.smithi019.stdout: "put_sum": 1683220, 2024-03-18T15:28:52.349 INFO:teuthology.orchestra.run.smithi019.stdout: "wait": { 2024-03-18T15:28:52.349 INFO:teuthology.orchestra.run.smithi019.stdout: "avgcount": 0, 2024-03-18T15:28:52.349 INFO:teuthology.orchestra.run.smithi019.stdout: "sum": 0.000000000, 2024-03-18T15:28:52.349 INFO:teuthology.orchestra.run.smithi019.stdout: "avgtime": 0.000000000 2024-03-18T15:28:52.350 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:52.350 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-18T15:28:52.350 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-03-18T15:28:53.409 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-03-18T15:28:53.409 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-03-18T15:28:53.409 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.j 2024-03-18T15:28:53.410 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.j 2024-03-18T15:28:53.410 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.j 2024-03-18T15:28:53.410 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.j 2024-03-18T15:28:53.410 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.j 2024-03-18T15:28:53.410 DEBUG:tasks.check_counter:Getting stats from b 2024-03-18T15:28:53.410 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 -- ceph daemon mds.b perf dump 2024-03-18T15:28:53.682 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T15:28:53.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:28:53 smithi019 ceph-mon[31522]: pgmap v323: 129 pgs: 129 active+clean; 1.7 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:28:53.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:28:53 smithi033 ceph-mon[36648]: pgmap v323: 129 pgs: 129 active+clean; 1.7 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:28:53.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:28:53 smithi104 ceph-mon[36070]: pgmap v323: 129 pgs: 129 active+clean; 1.7 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:28:54.845 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:28:54 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:28:54.845 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:28:54 smithi033 ceph-mon[36648]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:28:54.846 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:28:54 smithi033 ceph-mon[36648]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:28:54.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:28:54 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:28:54.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:28:54 smithi104 ceph-mon[36070]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:28:54.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:28:54 smithi104 ceph-mon[36070]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:28:54.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:28:54 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:28:54.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:28:54 smithi019 ceph-mon[31522]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:28:54.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:28:54 smithi019 ceph-mon[31522]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:28:55.651 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-03-18T15:28:55.651 INFO:teuthology.orchestra.run.smithi033.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T15:28:55.651 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_messages": 1864, 2024-03-18T15:28:55.651 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_messages": 910, 2024-03-18T15:28:55.651 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_bytes": 2501998, 2024-03-18T15:28:55.651 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_bytes": 427902, 2024-03-18T15:28:55.651 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_created_connections": 17, 2024-03-18T15:28:55.651 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_active_connections": 6, 2024-03-18T15:28:55.651 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_total_time": 0.156848835, 2024-03-18T15:28:55.651 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_send_time": 0.035397028, 2024-03-18T15:28:55.651 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_recv_time": 0.086220619, 2024-03-18T15:28:55.652 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_fast_dispatch_time": 0.047504176, 2024-03-18T15:28:55.652 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T15:28:55.652 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 907, 2024-03-18T15:28:55.652 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.460623000, 2024-03-18T15:28:55.652 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000507853 2024-03-18T15:28:55.652 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:55.652 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_handle_ack_lat": { 2024-03-18T15:28:55.652 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 1817, 2024-03-18T15:28:55.652 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000274435, 2024-03-18T15:28:55.652 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000151 2024-03-18T15:28:55.652 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:55.652 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_encrypted_bytes": 1915456, 2024-03-18T15:28:55.653 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_encrypted_bytes": 180288 2024-03-18T15:28:55.653 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:55.653 INFO:teuthology.orchestra.run.smithi033.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T15:28:55.653 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_messages": 888, 2024-03-18T15:28:55.653 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_messages": 763, 2024-03-18T15:28:55.653 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_bytes": 337679, 2024-03-18T15:28:55.653 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_bytes": 689881, 2024-03-18T15:28:55.653 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_created_connections": 13, 2024-03-18T15:28:55.653 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_active_connections": 5, 2024-03-18T15:28:55.653 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_total_time": 0.062838617, 2024-03-18T15:28:55.653 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_send_time": 0.019073526, 2024-03-18T15:28:55.653 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_recv_time": 0.024445517, 2024-03-18T15:28:55.653 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_fast_dispatch_time": 0.017101964, 2024-03-18T15:28:55.654 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T15:28:55.654 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 763, 2024-03-18T15:28:55.654 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.429563598, 2024-03-18T15:28:55.654 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000562992 2024-03-18T15:28:55.654 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:55.654 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_handle_ack_lat": { 2024-03-18T15:28:55.654 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 965, 2024-03-18T15:28:55.654 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000161876, 2024-03-18T15:28:55.654 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000167 2024-03-18T15:28:55.654 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:55.654 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_encrypted_bytes": 42544, 2024-03-18T15:28:55.654 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_encrypted_bytes": 471200 2024-03-18T15:28:55.654 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:55.654 INFO:teuthology.orchestra.run.smithi033.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T15:28:55.655 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_messages": 2034, 2024-03-18T15:28:55.655 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_messages": 1098, 2024-03-18T15:28:55.655 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_bytes": 978813, 2024-03-18T15:28:55.655 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_bytes": 741488, 2024-03-18T15:28:55.655 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_created_connections": 13, 2024-03-18T15:28:55.655 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_active_connections": 8, 2024-03-18T15:28:55.655 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_total_time": 0.150685670, 2024-03-18T15:28:55.655 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_send_time": 0.042530367, 2024-03-18T15:28:55.655 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_recv_time": 0.056997451, 2024-03-18T15:28:55.655 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_fast_dispatch_time": 0.049363914, 2024-03-18T15:28:55.655 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T15:28:55.655 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 1086, 2024-03-18T15:28:55.656 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.212600460, 2024-03-18T15:28:55.656 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000195764 2024-03-18T15:28:55.656 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:55.656 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_handle_ack_lat": { 2024-03-18T15:28:55.656 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 2477, 2024-03-18T15:28:55.656 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000372211, 2024-03-18T15:28:55.656 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000150 2024-03-18T15:28:55.656 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:55.656 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_encrypted_bytes": 312992, 2024-03-18T15:28:55.656 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_encrypted_bytes": 86896 2024-03-18T15:28:55.656 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:55.656 INFO:teuthology.orchestra.run.smithi033.stdout: "cct": { 2024-03-18T15:28:55.656 INFO:teuthology.orchestra.run.smithi033.stdout: "total_workers": 1, 2024-03-18T15:28:55.656 INFO:teuthology.orchestra.run.smithi033.stdout: "unhealthy_workers": 0 2024-03-18T15:28:55.657 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:55.657 INFO:teuthology.orchestra.run.smithi033.stdout: "finisher-MDSRank": { 2024-03-18T15:28:55.657 INFO:teuthology.orchestra.run.smithi033.stdout: "queue_len": 0, 2024-03-18T15:28:55.657 INFO:teuthology.orchestra.run.smithi033.stdout: "complete_latency": { 2024-03-18T15:28:55.657 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 308, 2024-03-18T15:28:55.657 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.068769014, 2024-03-18T15:28:55.657 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000223276 2024-03-18T15:28:55.657 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-18T15:28:55.657 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:55.657 INFO:teuthology.orchestra.run.smithi033.stdout: "finisher-PurgeQueue": { 2024-03-18T15:28:55.657 INFO:teuthology.orchestra.run.smithi033.stdout: "queue_len": 0, 2024-03-18T15:28:55.657 INFO:teuthology.orchestra.run.smithi033.stdout: "complete_latency": { 2024-03-18T15:28:55.657 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 2, 2024-03-18T15:28:55.657 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000012215, 2024-03-18T15:28:55.658 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000006107 2024-03-18T15:28:55.658 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-18T15:28:55.658 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:55.658 INFO:teuthology.orchestra.run.smithi033.stdout: "mds": { 2024-03-18T15:28:55.658 INFO:teuthology.orchestra.run.smithi033.stdout: "request": 167, 2024-03-18T15:28:55.658 INFO:teuthology.orchestra.run.smithi033.stdout: "reply": 167, 2024-03-18T15:28:55.658 INFO:teuthology.orchestra.run.smithi033.stdout: "reply_latency": { 2024-03-18T15:28:55.658 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 167, 2024-03-18T15:28:55.658 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.160912722, 2024-03-18T15:28:55.658 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000963549 2024-03-18T15:28:55.658 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:55.658 INFO:teuthology.orchestra.run.smithi033.stdout: "slow_reply": 0, 2024-03-18T15:28:55.658 INFO:teuthology.orchestra.run.smithi033.stdout: "forward": 0, 2024-03-18T15:28:55.659 INFO:teuthology.orchestra.run.smithi033.stdout: "dir_fetch_complete": 0, 2024-03-18T15:28:55.659 INFO:teuthology.orchestra.run.smithi033.stdout: "dir_fetch_keys": 0, 2024-03-18T15:28:55.659 INFO:teuthology.orchestra.run.smithi033.stdout: "dir_commit": 12, 2024-03-18T15:28:55.659 INFO:teuthology.orchestra.run.smithi033.stdout: "dir_split": 0, 2024-03-18T15:28:55.659 INFO:teuthology.orchestra.run.smithi033.stdout: "dir_merge": 0, 2024-03-18T15:28:55.659 INFO:teuthology.orchestra.run.smithi033.stdout: "inodes": 31, 2024-03-18T15:28:55.659 INFO:teuthology.orchestra.run.smithi033.stdout: "inodes_top": 3, 2024-03-18T15:28:55.659 INFO:teuthology.orchestra.run.smithi033.stdout: "inodes_bottom": 28, 2024-03-18T15:28:55.659 INFO:teuthology.orchestra.run.smithi033.stdout: "inodes_pin_tail": 0, 2024-03-18T15:28:55.659 INFO:teuthology.orchestra.run.smithi033.stdout: "inodes_pinned": 27, 2024-03-18T15:28:55.659 INFO:teuthology.orchestra.run.smithi033.stdout: "inodes_expired": 7, 2024-03-18T15:28:55.660 INFO:teuthology.orchestra.run.smithi033.stdout: "inodes_with_caps": 2, 2024-03-18T15:28:55.660 INFO:teuthology.orchestra.run.smithi033.stdout: "caps": 2, 2024-03-18T15:28:55.660 INFO:teuthology.orchestra.run.smithi033.stdout: "subtrees": 3, 2024-03-18T15:28:55.660 INFO:teuthology.orchestra.run.smithi033.stdout: "traverse": 216, 2024-03-18T15:28:55.660 INFO:teuthology.orchestra.run.smithi033.stdout: "traverse_hit": 208, 2024-03-18T15:28:55.660 INFO:teuthology.orchestra.run.smithi033.stdout: "traverse_forward": 0, 2024-03-18T15:28:55.660 INFO:teuthology.orchestra.run.smithi033.stdout: "traverse_discover": 1, 2024-03-18T15:28:55.660 INFO:teuthology.orchestra.run.smithi033.stdout: "traverse_dir_fetch": 0, 2024-03-18T15:28:55.660 INFO:teuthology.orchestra.run.smithi033.stdout: "traverse_remote_ino": 0, 2024-03-18T15:28:55.660 INFO:teuthology.orchestra.run.smithi033.stdout: "traverse_lock": 0, 2024-03-18T15:28:55.660 INFO:teuthology.orchestra.run.smithi033.stdout: "load_cent": 0, 2024-03-18T15:28:55.662 INFO:teuthology.orchestra.run.smithi033.stdout: "q": 0, 2024-03-18T15:28:55.662 INFO:teuthology.orchestra.run.smithi033.stdout: "exported": 1, 2024-03-18T15:28:55.662 INFO:teuthology.orchestra.run.smithi033.stdout: "exported_inodes": 5, 2024-03-18T15:28:55.662 INFO:teuthology.orchestra.run.smithi033.stdout: "imported": 1, 2024-03-18T15:28:55.662 INFO:teuthology.orchestra.run.smithi033.stdout: "imported_inodes": 2, 2024-03-18T15:28:55.662 INFO:teuthology.orchestra.run.smithi033.stdout: "openino_dir_fetch": 0, 2024-03-18T15:28:55.662 INFO:teuthology.orchestra.run.smithi033.stdout: "openino_backtrace_fetch": 0, 2024-03-18T15:28:55.662 INFO:teuthology.orchestra.run.smithi033.stdout: "openino_peer_discover": 0, 2024-03-18T15:28:55.662 INFO:teuthology.orchestra.run.smithi033.stdout: "root_rfiles": 4, 2024-03-18T15:28:55.662 INFO:teuthology.orchestra.run.smithi033.stdout: "root_rbytes": 234, 2024-03-18T15:28:55.662 INFO:teuthology.orchestra.run.smithi033.stdout: "root_rsnaps": 6, 2024-03-18T15:28:55.663 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_backtrace_fetch": 0, 2024-03-18T15:28:55.663 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_set_tag": 0, 2024-03-18T15:28:55.663 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_backtrace_repaired": 0, 2024-03-18T15:28:55.663 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_inotable_repaired": 0, 2024-03-18T15:28:55.663 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_dir_inodes": 0, 2024-03-18T15:28:55.663 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_dir_base_inodes": 0, 2024-03-18T15:28:55.663 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_dirfrag_rstats": 0, 2024-03-18T15:28:55.663 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_file_inodes": 0, 2024-03-18T15:28:55.663 INFO:teuthology.orchestra.run.smithi033.stdout: "handle_inode_file_caps": 2, 2024-03-18T15:28:55.663 INFO:teuthology.orchestra.run.smithi033.stdout: "ceph_cap_op_revoke": 11, 2024-03-18T15:28:55.663 INFO:teuthology.orchestra.run.smithi033.stdout: "ceph_cap_op_grant": 20, 2024-03-18T15:28:55.663 INFO:teuthology.orchestra.run.smithi033.stdout: "ceph_cap_op_trunc": 0, 2024-03-18T15:28:55.663 INFO:teuthology.orchestra.run.smithi033.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-18T15:28:55.663 INFO:teuthology.orchestra.run.smithi033.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-18T15:28:55.663 INFO:teuthology.orchestra.run.smithi033.stdout: "handle_client_caps": 31, 2024-03-18T15:28:55.664 INFO:teuthology.orchestra.run.smithi033.stdout: "handle_client_caps_dirty": 17, 2024-03-18T15:28:55.664 INFO:teuthology.orchestra.run.smithi033.stdout: "handle_client_cap_release": 1, 2024-03-18T15:28:55.664 INFO:teuthology.orchestra.run.smithi033.stdout: "process_request_cap_release": 20 2024-03-18T15:28:55.664 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:55.664 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_cache": { 2024-03-18T15:28:55.664 INFO:teuthology.orchestra.run.smithi033.stdout: "dir_update": 7, 2024-03-18T15:28:55.664 INFO:teuthology.orchestra.run.smithi033.stdout: "dir_update_receipt": 3, 2024-03-18T15:28:55.664 INFO:teuthology.orchestra.run.smithi033.stdout: "dir_try_discover": 2, 2024-03-18T15:28:55.664 INFO:teuthology.orchestra.run.smithi033.stdout: "dir_send_discover": 2, 2024-03-18T15:28:55.664 INFO:teuthology.orchestra.run.smithi033.stdout: "dir_handle_discover": 26, 2024-03-18T15:28:55.664 INFO:teuthology.orchestra.run.smithi033.stdout: "num_strays": 1, 2024-03-18T15:28:55.664 INFO:teuthology.orchestra.run.smithi033.stdout: "num_strays_delayed": 0, 2024-03-18T15:28:55.665 INFO:teuthology.orchestra.run.smithi033.stdout: "num_strays_enqueuing": 0, 2024-03-18T15:28:55.665 INFO:teuthology.orchestra.run.smithi033.stdout: "strays_created": 1, 2024-03-18T15:28:55.665 INFO:teuthology.orchestra.run.smithi033.stdout: "strays_enqueued": 0, 2024-03-18T15:28:55.665 INFO:teuthology.orchestra.run.smithi033.stdout: "strays_reintegrated": 0, 2024-03-18T15:28:55.665 INFO:teuthology.orchestra.run.smithi033.stdout: "strays_migrated": 0, 2024-03-18T15:28:55.665 INFO:teuthology.orchestra.run.smithi033.stdout: "num_recovering_processing": 0, 2024-03-18T15:28:55.665 INFO:teuthology.orchestra.run.smithi033.stdout: "num_recovering_enqueued": 0, 2024-03-18T15:28:55.665 INFO:teuthology.orchestra.run.smithi033.stdout: "num_recovering_prioritized": 0, 2024-03-18T15:28:55.665 INFO:teuthology.orchestra.run.smithi033.stdout: "recovery_started": 0, 2024-03-18T15:28:55.665 INFO:teuthology.orchestra.run.smithi033.stdout: "recovery_completed": 0, 2024-03-18T15:28:55.665 INFO:teuthology.orchestra.run.smithi033.stdout: "ireq_quiesce_path": 0, 2024-03-18T15:28:55.665 INFO:teuthology.orchestra.run.smithi033.stdout: "ireq_quiesce_inode": 0, 2024-03-18T15:28:55.665 INFO:teuthology.orchestra.run.smithi033.stdout: "ireq_enqueue_scrub": 0, 2024-03-18T15:28:55.665 INFO:teuthology.orchestra.run.smithi033.stdout: "ireq_exportdir": 1, 2024-03-18T15:28:55.665 INFO:teuthology.orchestra.run.smithi033.stdout: "ireq_flush": 0, 2024-03-18T15:28:55.666 INFO:teuthology.orchestra.run.smithi033.stdout: "ireq_fragmentdir": 0, 2024-03-18T15:28:55.666 INFO:teuthology.orchestra.run.smithi033.stdout: "ireq_fragstats": 0, 2024-03-18T15:28:55.666 INFO:teuthology.orchestra.run.smithi033.stdout: "ireq_inodestats": 0 2024-03-18T15:28:55.666 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:55.666 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_log": { 2024-03-18T15:28:55.666 INFO:teuthology.orchestra.run.smithi033.stdout: "evlrg": 0, 2024-03-18T15:28:55.666 INFO:teuthology.orchestra.run.smithi033.stdout: "evadd": 143, 2024-03-18T15:28:55.666 INFO:teuthology.orchestra.run.smithi033.stdout: "evex": 0, 2024-03-18T15:28:55.666 INFO:teuthology.orchestra.run.smithi033.stdout: "evtrm": 0, 2024-03-18T15:28:55.666 INFO:teuthology.orchestra.run.smithi033.stdout: "ev": 143, 2024-03-18T15:28:55.666 INFO:teuthology.orchestra.run.smithi033.stdout: "evexg": 0, 2024-03-18T15:28:55.666 INFO:teuthology.orchestra.run.smithi033.stdout: "evexd": 0, 2024-03-18T15:28:55.666 INFO:teuthology.orchestra.run.smithi033.stdout: "segadd": 2, 2024-03-18T15:28:55.667 INFO:teuthology.orchestra.run.smithi033.stdout: "segex": 0, 2024-03-18T15:28:55.667 INFO:teuthology.orchestra.run.smithi033.stdout: "segtrm": 0, 2024-03-18T15:28:55.667 INFO:teuthology.orchestra.run.smithi033.stdout: "seg": 2, 2024-03-18T15:28:55.667 INFO:teuthology.orchestra.run.smithi033.stdout: "segmjr": 2, 2024-03-18T15:28:55.667 INFO:teuthology.orchestra.run.smithi033.stdout: "segexg": 0, 2024-03-18T15:28:55.667 INFO:teuthology.orchestra.run.smithi033.stdout: "segexd": 0, 2024-03-18T15:28:55.667 INFO:teuthology.orchestra.run.smithi033.stdout: "expos": 4194304, 2024-03-18T15:28:55.667 INFO:teuthology.orchestra.run.smithi033.stdout: "wrpos": 4474023, 2024-03-18T15:28:55.667 INFO:teuthology.orchestra.run.smithi033.stdout: "rdpos": 0, 2024-03-18T15:28:55.667 INFO:teuthology.orchestra.run.smithi033.stdout: "jlat": { 2024-03-18T15:28:55.667 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 115, 2024-03-18T15:28:55.667 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.183679165, 2024-03-18T15:28:55.667 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.001597210 2024-03-18T15:28:55.667 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:55.668 INFO:teuthology.orchestra.run.smithi033.stdout: "replayed": 0 2024-03-18T15:28:55.668 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:55.668 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_mem": { 2024-03-18T15:28:55.668 INFO:teuthology.orchestra.run.smithi033.stdout: "ino": 26, 2024-03-18T15:28:55.668 INFO:teuthology.orchestra.run.smithi033.stdout: "ino+": 32, 2024-03-18T15:28:55.668 INFO:teuthology.orchestra.run.smithi033.stdout: "ino-": 6, 2024-03-18T15:28:55.668 INFO:teuthology.orchestra.run.smithi033.stdout: "dir": 21, 2024-03-18T15:28:55.668 INFO:teuthology.orchestra.run.smithi033.stdout: "dir+": 23, 2024-03-18T15:28:55.668 INFO:teuthology.orchestra.run.smithi033.stdout: "dir-": 2, 2024-03-18T15:28:55.668 INFO:teuthology.orchestra.run.smithi033.stdout: "dn": 31, 2024-03-18T15:28:55.668 INFO:teuthology.orchestra.run.smithi033.stdout: "dn+": 38, 2024-03-18T15:28:55.668 INFO:teuthology.orchestra.run.smithi033.stdout: "dn-": 7, 2024-03-18T15:28:55.668 INFO:teuthology.orchestra.run.smithi033.stdout: "cap": 2, 2024-03-18T15:28:55.668 INFO:teuthology.orchestra.run.smithi033.stdout: "cap+": 21, 2024-03-18T15:28:55.668 INFO:teuthology.orchestra.run.smithi033.stdout: "cap-": 19, 2024-03-18T15:28:55.669 INFO:teuthology.orchestra.run.smithi033.stdout: "rss": 51988, 2024-03-18T15:28:55.669 INFO:teuthology.orchestra.run.smithi033.stdout: "heap": 231804 2024-03-18T15:28:55.669 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:55.669 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_server": { 2024-03-18T15:28:55.669 INFO:teuthology.orchestra.run.smithi033.stdout: "dispatch_client_request": 212, 2024-03-18T15:28:55.669 INFO:teuthology.orchestra.run.smithi033.stdout: "dispatch_server_request": 3, 2024-03-18T15:28:55.669 INFO:teuthology.orchestra.run.smithi033.stdout: "handle_client_request": 167, 2024-03-18T15:28:55.670 INFO:teuthology.orchestra.run.smithi033.stdout: "handle_client_session": 40, 2024-03-18T15:28:55.670 INFO:teuthology.orchestra.run.smithi033.stdout: "handle_peer_request": 14, 2024-03-18T15:28:55.670 INFO:teuthology.orchestra.run.smithi033.stdout: "req_create_latency": { 2024-03-18T15:28:55.670 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 8, 2024-03-18T15:28:55.670 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.006939390, 2024-03-18T15:28:55.670 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000867423 2024-03-18T15:28:55.670 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:55.670 INFO:teuthology.orchestra.run.smithi033.stdout: "req_getattr_latency": { 2024-03-18T15:28:55.670 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 112, 2024-03-18T15:28:55.670 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.037015039, 2024-03-18T15:28:55.670 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000330491 2024-03-18T15:28:55.670 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:55.670 INFO:teuthology.orchestra.run.smithi033.stdout: "req_getfilelock_latency": { 2024-03-18T15:28:55.670 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:28:55.671 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:28:55.671 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:28:55.671 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:55.671 INFO:teuthology.orchestra.run.smithi033.stdout: "req_link_latency": { 2024-03-18T15:28:55.671 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:28:55.671 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:28:55.671 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:28:55.671 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:55.671 INFO:teuthology.orchestra.run.smithi033.stdout: "req_lookup_latency": { 2024-03-18T15:28:55.671 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:28:55.671 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:28:55.671 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:28:55.671 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:55.673 INFO:teuthology.orchestra.run.smithi033.stdout: "req_lookuphash_latency": { 2024-03-18T15:28:55.673 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:28:55.673 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:28:55.673 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:28:55.673 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:55.673 INFO:teuthology.orchestra.run.smithi033.stdout: "req_lookupino_latency": { 2024-03-18T15:28:55.673 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:28:55.673 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:28:55.673 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:28:55.673 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:55.673 INFO:teuthology.orchestra.run.smithi033.stdout: "req_lookupname_latency": { 2024-03-18T15:28:55.673 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:28:55.674 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:28:55.674 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:28:55.674 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:55.674 INFO:teuthology.orchestra.run.smithi033.stdout: "req_lookupparent_latency": { 2024-03-18T15:28:55.674 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:28:55.674 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:28:55.674 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:28:55.674 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:55.674 INFO:teuthology.orchestra.run.smithi033.stdout: "req_lookupsnap_latency": { 2024-03-18T15:28:55.674 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:28:55.675 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:28:55.675 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:28:55.675 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:55.675 INFO:teuthology.orchestra.run.smithi033.stdout: "req_lssnap_latency": { 2024-03-18T15:28:55.675 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 10, 2024-03-18T15:28:55.675 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.001787135, 2024-03-18T15:28:55.675 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000178713 2024-03-18T15:28:55.675 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:55.676 INFO:teuthology.orchestra.run.smithi033.stdout: "req_mkdir_latency": { 2024-03-18T15:28:55.676 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 8, 2024-03-18T15:28:55.677 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.012371236, 2024-03-18T15:28:55.677 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.001546404 2024-03-18T15:28:55.677 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:55.677 INFO:teuthology.orchestra.run.smithi033.stdout: "req_mknod_latency": { 2024-03-18T15:28:55.677 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:28:55.677 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:28:55.677 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:28:55.677 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:55.677 INFO:teuthology.orchestra.run.smithi033.stdout: "req_mksnap_latency": { 2024-03-18T15:28:55.677 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 10, 2024-03-18T15:28:55.677 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.045096149, 2024-03-18T15:28:55.678 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.004509614 2024-03-18T15:28:55.678 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:55.678 INFO:teuthology.orchestra.run.smithi033.stdout: "req_open_latency": { 2024-03-18T15:28:55.678 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 1, 2024-03-18T15:28:55.678 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000149040, 2024-03-18T15:28:55.678 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000149040 2024-03-18T15:28:55.678 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:55.679 INFO:teuthology.orchestra.run.smithi033.stdout: "req_readdir_latency": { 2024-03-18T15:28:55.679 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:28:55.679 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:28:55.679 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:28:55.679 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:55.679 INFO:teuthology.orchestra.run.smithi033.stdout: "req_rename_latency": { 2024-03-18T15:28:55.679 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 2, 2024-03-18T15:28:55.679 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.003292216, 2024-03-18T15:28:55.679 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.001646108 2024-03-18T15:28:55.679 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:55.679 INFO:teuthology.orchestra.run.smithi033.stdout: "req_renamesnap_latency": { 2024-03-18T15:28:55.679 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:28:55.679 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:28:55.680 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:28:55.680 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:55.680 INFO:teuthology.orchestra.run.smithi033.stdout: "req_snapdiff_latency": { 2024-03-18T15:28:55.680 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:28:55.680 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:28:55.680 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:28:55.680 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:55.680 INFO:teuthology.orchestra.run.smithi033.stdout: "req_rmdir_latency": { 2024-03-18T15:28:55.680 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 1, 2024-03-18T15:28:55.681 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.014394098, 2024-03-18T15:28:55.681 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.014394098 2024-03-18T15:28:55.681 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:55.681 INFO:teuthology.orchestra.run.smithi033.stdout: "req_rmsnap_latency": { 2024-03-18T15:28:55.681 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 4, 2024-03-18T15:28:55.681 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.016672611, 2024-03-18T15:28:55.681 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.004168152 2024-03-18T15:28:55.681 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:55.681 INFO:teuthology.orchestra.run.smithi033.stdout: "req_rmxattr_latency": { 2024-03-18T15:28:55.681 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:28:55.682 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:28:55.682 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:28:55.682 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:55.682 INFO:teuthology.orchestra.run.smithi033.stdout: "req_setattr_latency": { 2024-03-18T15:28:55.682 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 2, 2024-03-18T15:28:55.682 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.001294659, 2024-03-18T15:28:55.682 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000647329 2024-03-18T15:28:55.682 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:55.682 INFO:teuthology.orchestra.run.smithi033.stdout: "req_setdirlayout_latency": { 2024-03-18T15:28:55.682 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:28:55.682 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:28:55.682 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:28:55.683 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:55.683 INFO:teuthology.orchestra.run.smithi033.stdout: "req_setfilelock_latency": { 2024-03-18T15:28:55.683 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:28:55.684 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:28:55.684 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:28:55.684 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:55.684 INFO:teuthology.orchestra.run.smithi033.stdout: "req_setlayout_latency": { 2024-03-18T15:28:55.684 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:28:55.684 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:28:55.684 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:28:55.684 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:55.684 INFO:teuthology.orchestra.run.smithi033.stdout: "req_setxattr_latency": { 2024-03-18T15:28:55.684 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 9, 2024-03-18T15:28:55.684 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.021901149, 2024-03-18T15:28:55.685 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.002433461 2024-03-18T15:28:55.685 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:55.685 INFO:teuthology.orchestra.run.smithi033.stdout: "req_symlink_latency": { 2024-03-18T15:28:55.685 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:28:55.685 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:28:55.685 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:28:55.685 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:55.685 INFO:teuthology.orchestra.run.smithi033.stdout: "req_unlink_latency": { 2024-03-18T15:28:55.685 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:28:55.685 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:28:55.685 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:28:55.685 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:55.685 INFO:teuthology.orchestra.run.smithi033.stdout: "cap_revoke_eviction": 0, 2024-03-18T15:28:55.685 INFO:teuthology.orchestra.run.smithi033.stdout: "cap_acquisition_throttle": 0, 2024-03-18T15:28:55.686 INFO:teuthology.orchestra.run.smithi033.stdout: "req_getvxattr_latency": { 2024-03-18T15:28:55.686 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:28:55.686 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:28:55.686 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:28:55.686 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-18T15:28:55.686 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:55.686 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_sessions": { 2024-03-18T15:28:55.686 INFO:teuthology.orchestra.run.smithi033.stdout: "session_count": 2, 2024-03-18T15:28:55.686 INFO:teuthology.orchestra.run.smithi033.stdout: "session_add": 7, 2024-03-18T15:28:55.686 INFO:teuthology.orchestra.run.smithi033.stdout: "session_remove": 5, 2024-03-18T15:28:55.686 INFO:teuthology.orchestra.run.smithi033.stdout: "sessions_open": 2, 2024-03-18T15:28:55.686 INFO:teuthology.orchestra.run.smithi033.stdout: "sessions_stale": 0, 2024-03-18T15:28:55.686 INFO:teuthology.orchestra.run.smithi033.stdout: "total_load": 27, 2024-03-18T15:28:55.687 INFO:teuthology.orchestra.run.smithi033.stdout: "average_load": 13, 2024-03-18T15:28:55.687 INFO:teuthology.orchestra.run.smithi033.stdout: "avg_session_uptime": 615, 2024-03-18T15:28:55.687 INFO:teuthology.orchestra.run.smithi033.stdout: "mdthresh_evicted": 0 2024-03-18T15:28:55.687 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:55.687 INFO:teuthology.orchestra.run.smithi033.stdout: "mempool": { 2024-03-18T15:28:55.687 INFO:teuthology.orchestra.run.smithi033.stdout: "bloom_filter_bytes": 0, 2024-03-18T15:28:55.687 INFO:teuthology.orchestra.run.smithi033.stdout: "bloom_filter_items": 0, 2024-03-18T15:28:55.687 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T15:28:55.687 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_alloc_items": 0, 2024-03-18T15:28:55.687 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T15:28:55.687 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_data_items": 0, 2024-03-18T15:28:55.687 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T15:28:55.687 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T15:28:55.687 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T15:28:55.688 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T15:28:55.688 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T15:28:55.688 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_other_items": 0, 2024-03-18T15:28:55.688 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T15:28:55.688 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T15:28:55.688 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_extent_bytes": 0, 2024-03-18T15:28:55.688 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_extent_items": 0, 2024-03-18T15:28:55.688 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_blob_bytes": 0, 2024-03-18T15:28:55.689 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_blob_items": 0, 2024-03-18T15:28:55.689 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T15:28:55.689 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T15:28:55.689 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T15:28:55.689 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T15:28:55.689 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T15:28:55.689 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_fsck_items": 0, 2024-03-18T15:28:55.689 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_txc_bytes": 0, 2024-03-18T15:28:55.689 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_txc_items": 0, 2024-03-18T15:28:55.689 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T15:28:55.689 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T15:28:55.689 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_writing_bytes": 0, 2024-03-18T15:28:55.689 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_writing_items": 0, 2024-03-18T15:28:55.690 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_bytes": 0, 2024-03-18T15:28:55.690 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_items": 0, 2024-03-18T15:28:55.690 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T15:28:55.690 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_file_reader_items": 0, 2024-03-18T15:28:55.690 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T15:28:55.690 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_file_writer_items": 0, 2024-03-18T15:28:55.690 INFO:teuthology.orchestra.run.smithi033.stdout: "buffer_anon_bytes": 364044, 2024-03-18T15:28:55.690 INFO:teuthology.orchestra.run.smithi033.stdout: "buffer_anon_items": 151, 2024-03-18T15:28:55.690 INFO:teuthology.orchestra.run.smithi033.stdout: "buffer_meta_bytes": 792, 2024-03-18T15:28:55.690 INFO:teuthology.orchestra.run.smithi033.stdout: "buffer_meta_items": 9, 2024-03-18T15:28:55.690 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_bytes": 0, 2024-03-18T15:28:55.690 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_items": 0, 2024-03-18T15:28:55.690 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mapbl_bytes": 0, 2024-03-18T15:28:55.691 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mapbl_items": 0, 2024-03-18T15:28:55.691 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pglog_bytes": 0, 2024-03-18T15:28:55.691 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pglog_items": 0, 2024-03-18T15:28:55.691 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_bytes": 9928, 2024-03-18T15:28:55.691 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_items": 207, 2024-03-18T15:28:55.691 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T15:28:55.691 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_mapping_items": 0, 2024-03-18T15:28:55.691 INFO:teuthology.orchestra.run.smithi033.stdout: "pgmap_bytes": 0, 2024-03-18T15:28:55.691 INFO:teuthology.orchestra.run.smithi033.stdout: "pgmap_items": 0, 2024-03-18T15:28:55.691 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_co_bytes": 123092, 2024-03-18T15:28:55.691 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_co_items": 883, 2024-03-18T15:28:55.691 INFO:teuthology.orchestra.run.smithi033.stdout: "unittest_1_bytes": 0, 2024-03-18T15:28:55.691 INFO:teuthology.orchestra.run.smithi033.stdout: "unittest_1_items": 0, 2024-03-18T15:28:55.692 INFO:teuthology.orchestra.run.smithi033.stdout: "unittest_2_bytes": 0, 2024-03-18T15:28:55.692 INFO:teuthology.orchestra.run.smithi033.stdout: "unittest_2_items": 0 2024-03-18T15:28:55.692 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:55.692 INFO:teuthology.orchestra.run.smithi033.stdout: "objecter": { 2024-03-18T15:28:55.692 INFO:teuthology.orchestra.run.smithi033.stdout: "op_active": 0, 2024-03-18T15:28:55.692 INFO:teuthology.orchestra.run.smithi033.stdout: "op_laggy": 0, 2024-03-18T15:28:55.692 INFO:teuthology.orchestra.run.smithi033.stdout: "op_send": 691, 2024-03-18T15:28:55.692 INFO:teuthology.orchestra.run.smithi033.stdout: "op_send_bytes": 295957, 2024-03-18T15:28:55.692 INFO:teuthology.orchestra.run.smithi033.stdout: "op_resend": 33, 2024-03-18T15:28:55.692 INFO:teuthology.orchestra.run.smithi033.stdout: "op_reply": 691, 2024-03-18T15:28:55.692 INFO:teuthology.orchestra.run.smithi033.stdout: "op_latency": { 2024-03-18T15:28:55.692 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 691, 2024-03-18T15:28:55.692 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 21.482631161, 2024-03-18T15:28:55.692 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.031089191 2024-03-18T15:28:55.693 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:55.693 INFO:teuthology.orchestra.run.smithi033.stdout: "op_inflight": 0, 2024-03-18T15:28:55.693 INFO:teuthology.orchestra.run.smithi033.stdout: "oplen_avg": { 2024-03-18T15:28:55.693 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 691, 2024-03-18T15:28:55.693 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 734 2024-03-18T15:28:55.693 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:55.693 INFO:teuthology.orchestra.run.smithi033.stdout: "op": 691, 2024-03-18T15:28:55.693 INFO:teuthology.orchestra.run.smithi033.stdout: "op_r": 0, 2024-03-18T15:28:55.693 INFO:teuthology.orchestra.run.smithi033.stdout: "op_w": 691, 2024-03-18T15:28:55.693 INFO:teuthology.orchestra.run.smithi033.stdout: "op_rmw": 0, 2024-03-18T15:28:55.693 INFO:teuthology.orchestra.run.smithi033.stdout: "op_pg": 0, 2024-03-18T15:28:55.693 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_stat": 0, 2024-03-18T15:28:55.693 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_create": 11, 2024-03-18T15:28:55.693 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_read": 0, 2024-03-18T15:28:55.694 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_write": 115, 2024-03-18T15:28:55.694 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_writefull": 28, 2024-03-18T15:28:55.694 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_writesame": 0, 2024-03-18T15:28:55.694 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_append": 0, 2024-03-18T15:28:55.694 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_zero": 0, 2024-03-18T15:28:55.694 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_truncate": 0, 2024-03-18T15:28:55.694 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_delete": 506, 2024-03-18T15:28:55.694 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_mapext": 0, 2024-03-18T15:28:55.694 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_sparse_read": 0, 2024-03-18T15:28:55.694 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_clonerange": 0, 2024-03-18T15:28:55.694 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_getxattr": 0, 2024-03-18T15:28:55.694 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_setxattr": 22, 2024-03-18T15:28:55.694 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_cmpxattr": 0, 2024-03-18T15:28:55.695 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_rmxattr": 0, 2024-03-18T15:28:55.695 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_resetxattrs": 0, 2024-03-18T15:28:55.695 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_call": 0, 2024-03-18T15:28:55.695 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_watch": 0, 2024-03-18T15:28:55.695 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_notify": 0, 2024-03-18T15:28:55.695 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_src_cmpxattr": 0, 2024-03-18T15:28:55.695 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_pgls": 0, 2024-03-18T15:28:55.695 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_pgls_filter": 0, 2024-03-18T15:28:55.695 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_other": 0, 2024-03-18T15:28:55.695 INFO:teuthology.orchestra.run.smithi033.stdout: "linger_active": 0, 2024-03-18T15:28:55.695 INFO:teuthology.orchestra.run.smithi033.stdout: "linger_send": 0, 2024-03-18T15:28:55.695 INFO:teuthology.orchestra.run.smithi033.stdout: "linger_resend": 0, 2024-03-18T15:28:55.695 INFO:teuthology.orchestra.run.smithi033.stdout: "linger_ping": 0, 2024-03-18T15:28:55.695 INFO:teuthology.orchestra.run.smithi033.stdout: "poolop_active": 0, 2024-03-18T15:28:55.696 INFO:teuthology.orchestra.run.smithi033.stdout: "poolop_send": 0, 2024-03-18T15:28:55.696 INFO:teuthology.orchestra.run.smithi033.stdout: "poolop_resend": 0, 2024-03-18T15:28:55.696 INFO:teuthology.orchestra.run.smithi033.stdout: "poolstat_active": 0, 2024-03-18T15:28:55.696 INFO:teuthology.orchestra.run.smithi033.stdout: "poolstat_send": 0, 2024-03-18T15:28:55.696 INFO:teuthology.orchestra.run.smithi033.stdout: "poolstat_resend": 0, 2024-03-18T15:28:55.696 INFO:teuthology.orchestra.run.smithi033.stdout: "statfs_active": 0, 2024-03-18T15:28:55.696 INFO:teuthology.orchestra.run.smithi033.stdout: "statfs_send": 0, 2024-03-18T15:28:55.696 INFO:teuthology.orchestra.run.smithi033.stdout: "statfs_resend": 0, 2024-03-18T15:28:55.696 INFO:teuthology.orchestra.run.smithi033.stdout: "command_active": 0, 2024-03-18T15:28:55.696 INFO:teuthology.orchestra.run.smithi033.stdout: "command_send": 0, 2024-03-18T15:28:55.696 INFO:teuthology.orchestra.run.smithi033.stdout: "command_resend": 0, 2024-03-18T15:28:55.696 INFO:teuthology.orchestra.run.smithi033.stdout: "map_epoch": 103, 2024-03-18T15:28:55.696 INFO:teuthology.orchestra.run.smithi033.stdout: "map_full": 0, 2024-03-18T15:28:55.696 INFO:teuthology.orchestra.run.smithi033.stdout: "map_inc": 19, 2024-03-18T15:28:55.696 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_sessions": 12, 2024-03-18T15:28:55.697 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_session_open": 12, 2024-03-18T15:28:55.697 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_session_close": 0, 2024-03-18T15:28:55.697 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_laggy": 0, 2024-03-18T15:28:55.697 INFO:teuthology.orchestra.run.smithi033.stdout: "omap_wr": 44, 2024-03-18T15:28:55.697 INFO:teuthology.orchestra.run.smithi033.stdout: "omap_rd": 0, 2024-03-18T15:28:55.697 INFO:teuthology.orchestra.run.smithi033.stdout: "omap_del": 8 2024-03-18T15:28:55.697 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:55.697 INFO:teuthology.orchestra.run.smithi033.stdout: "oft": { 2024-03-18T15:28:55.697 INFO:teuthology.orchestra.run.smithi033.stdout: "omap_total_objs": 1, 2024-03-18T15:28:55.697 INFO:teuthology.orchestra.run.smithi033.stdout: "omap_total_kv_pairs": 0, 2024-03-18T15:28:55.697 INFO:teuthology.orchestra.run.smithi033.stdout: "omap_total_updates": 0, 2024-03-18T15:28:55.697 INFO:teuthology.orchestra.run.smithi033.stdout: "omap_total_removes": 0 2024-03-18T15:28:55.697 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:55.697 INFO:teuthology.orchestra.run.smithi033.stdout: "osd-slow-ops": { 2024-03-18T15:28:55.697 INFO:teuthology.orchestra.run.smithi033.stdout: "slow_ops_count": 0 2024-03-18T15:28:55.698 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:55.698 INFO:teuthology.orchestra.run.smithi033.stdout: "purge_queue": { 2024-03-18T15:28:55.698 INFO:teuthology.orchestra.run.smithi033.stdout: "pq_executing_ops": 0, 2024-03-18T15:28:55.698 INFO:teuthology.orchestra.run.smithi033.stdout: "pq_executing_ops_high_water": 0, 2024-03-18T15:28:55.698 INFO:teuthology.orchestra.run.smithi033.stdout: "pq_executing": 0, 2024-03-18T15:28:55.698 INFO:teuthology.orchestra.run.smithi033.stdout: "pq_executing_high_water": 0, 2024-03-18T15:28:55.698 INFO:teuthology.orchestra.run.smithi033.stdout: "pq_executed_ops": 0, 2024-03-18T15:28:55.698 INFO:teuthology.orchestra.run.smithi033.stdout: "pq_executed": 0, 2024-03-18T15:28:55.698 INFO:teuthology.orchestra.run.smithi033.stdout: "pq_item_in_journal": 0 2024-03-18T15:28:55.698 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:55.698 INFO:teuthology.orchestra.run.smithi033.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T15:28:55.698 INFO:teuthology.orchestra.run.smithi033.stdout: "val": 0, 2024-03-18T15:28:55.698 INFO:teuthology.orchestra.run.smithi033.stdout: "max": 104857600, 2024-03-18T15:28:55.699 INFO:teuthology.orchestra.run.smithi033.stdout: "get_started": 0, 2024-03-18T15:28:55.699 INFO:teuthology.orchestra.run.smithi033.stdout: "get": 4782, 2024-03-18T15:28:55.699 INFO:teuthology.orchestra.run.smithi033.stdout: "get_sum": 3321847, 2024-03-18T15:28:55.699 INFO:teuthology.orchestra.run.smithi033.stdout: "get_or_fail_fail": 0, 2024-03-18T15:28:55.699 INFO:teuthology.orchestra.run.smithi033.stdout: "get_or_fail_success": 4782, 2024-03-18T15:28:55.699 INFO:teuthology.orchestra.run.smithi033.stdout: "take": 0, 2024-03-18T15:28:55.699 INFO:teuthology.orchestra.run.smithi033.stdout: "take_sum": 0, 2024-03-18T15:28:55.699 INFO:teuthology.orchestra.run.smithi033.stdout: "put": 4782, 2024-03-18T15:28:55.701 INFO:teuthology.orchestra.run.smithi033.stdout: "put_sum": 3321847, 2024-03-18T15:28:55.701 INFO:teuthology.orchestra.run.smithi033.stdout: "wait": { 2024-03-18T15:28:55.701 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:28:55.701 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:28:55.701 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:28:55.701 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-18T15:28:55.701 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:55.701 INFO:teuthology.orchestra.run.smithi033.stdout: "throttle-objecter_bytes": { 2024-03-18T15:28:55.701 INFO:teuthology.orchestra.run.smithi033.stdout: "val": 0, 2024-03-18T15:28:55.702 INFO:teuthology.orchestra.run.smithi033.stdout: "max": 104857600, 2024-03-18T15:28:55.702 INFO:teuthology.orchestra.run.smithi033.stdout: "get_started": 0, 2024-03-18T15:28:55.702 INFO:teuthology.orchestra.run.smithi033.stdout: "get": 0, 2024-03-18T15:28:55.702 INFO:teuthology.orchestra.run.smithi033.stdout: "get_sum": 0, 2024-03-18T15:28:55.702 INFO:teuthology.orchestra.run.smithi033.stdout: "get_or_fail_fail": 0, 2024-03-18T15:28:55.702 INFO:teuthology.orchestra.run.smithi033.stdout: "get_or_fail_success": 0, 2024-03-18T15:28:55.702 INFO:teuthology.orchestra.run.smithi033.stdout: "take": 691, 2024-03-18T15:28:55.702 INFO:teuthology.orchestra.run.smithi033.stdout: "take_sum": 295957, 2024-03-18T15:28:55.702 INFO:teuthology.orchestra.run.smithi033.stdout: "put": 691, 2024-03-18T15:28:55.702 INFO:teuthology.orchestra.run.smithi033.stdout: "put_sum": 295957, 2024-03-18T15:28:55.702 INFO:teuthology.orchestra.run.smithi033.stdout: "wait": { 2024-03-18T15:28:55.702 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:28:55.702 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:28:55.702 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:28:55.703 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-18T15:28:55.703 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:55.703 INFO:teuthology.orchestra.run.smithi033.stdout: "throttle-objecter_ops": { 2024-03-18T15:28:55.703 INFO:teuthology.orchestra.run.smithi033.stdout: "val": 0, 2024-03-18T15:28:55.703 INFO:teuthology.orchestra.run.smithi033.stdout: "max": 1024, 2024-03-18T15:28:55.703 INFO:teuthology.orchestra.run.smithi033.stdout: "get_started": 0, 2024-03-18T15:28:55.703 INFO:teuthology.orchestra.run.smithi033.stdout: "get": 0, 2024-03-18T15:28:55.703 INFO:teuthology.orchestra.run.smithi033.stdout: "get_sum": 0, 2024-03-18T15:28:55.703 INFO:teuthology.orchestra.run.smithi033.stdout: "get_or_fail_fail": 0, 2024-03-18T15:28:55.703 INFO:teuthology.orchestra.run.smithi033.stdout: "get_or_fail_success": 0, 2024-03-18T15:28:55.703 INFO:teuthology.orchestra.run.smithi033.stdout: "take": 691, 2024-03-18T15:28:55.703 INFO:teuthology.orchestra.run.smithi033.stdout: "take_sum": 691, 2024-03-18T15:28:55.703 INFO:teuthology.orchestra.run.smithi033.stdout: "put": 691, 2024-03-18T15:28:55.703 INFO:teuthology.orchestra.run.smithi033.stdout: "put_sum": 691, 2024-03-18T15:28:55.703 INFO:teuthology.orchestra.run.smithi033.stdout: "wait": { 2024-03-18T15:28:55.704 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:28:55.704 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:28:55.704 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:28:55.704 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-18T15:28:55.704 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:55.704 INFO:teuthology.orchestra.run.smithi033.stdout: "throttle-write_buf_throttle": { 2024-03-18T15:28:55.704 INFO:teuthology.orchestra.run.smithi033.stdout: "val": 0, 2024-03-18T15:28:55.704 INFO:teuthology.orchestra.run.smithi033.stdout: "max": 3758096384, 2024-03-18T15:28:55.704 INFO:teuthology.orchestra.run.smithi033.stdout: "get_started": 0, 2024-03-18T15:28:55.704 INFO:teuthology.orchestra.run.smithi033.stdout: "get": 0, 2024-03-18T15:28:55.704 INFO:teuthology.orchestra.run.smithi033.stdout: "get_sum": 0, 2024-03-18T15:28:55.704 INFO:teuthology.orchestra.run.smithi033.stdout: "get_or_fail_fail": 0, 2024-03-18T15:28:55.704 INFO:teuthology.orchestra.run.smithi033.stdout: "get_or_fail_success": 0, 2024-03-18T15:28:55.704 INFO:teuthology.orchestra.run.smithi033.stdout: "take": 0, 2024-03-18T15:28:55.705 INFO:teuthology.orchestra.run.smithi033.stdout: "take_sum": 0, 2024-03-18T15:28:55.705 INFO:teuthology.orchestra.run.smithi033.stdout: "put": 0, 2024-03-18T15:28:55.705 INFO:teuthology.orchestra.run.smithi033.stdout: "put_sum": 0, 2024-03-18T15:28:55.705 INFO:teuthology.orchestra.run.smithi033.stdout: "wait": { 2024-03-18T15:28:55.705 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:28:55.705 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:28:55.705 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:28:55.705 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-18T15:28:55.705 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:55.705 INFO:teuthology.orchestra.run.smithi033.stdout: "throttle-write_buf_throttle-0x55e308bea500": { 2024-03-18T15:28:55.705 INFO:teuthology.orchestra.run.smithi033.stdout: "val": 0, 2024-03-18T15:28:55.705 INFO:teuthology.orchestra.run.smithi033.stdout: "max": 3758096384, 2024-03-18T15:28:55.705 INFO:teuthology.orchestra.run.smithi033.stdout: "get_started": 0, 2024-03-18T15:28:55.705 INFO:teuthology.orchestra.run.smithi033.stdout: "get": 143, 2024-03-18T15:28:55.706 INFO:teuthology.orchestra.run.smithi033.stdout: "get_sum": 279719, 2024-03-18T15:28:55.706 INFO:teuthology.orchestra.run.smithi033.stdout: "get_or_fail_fail": 0, 2024-03-18T15:28:55.706 INFO:teuthology.orchestra.run.smithi033.stdout: "get_or_fail_success": 143, 2024-03-18T15:28:55.706 INFO:teuthology.orchestra.run.smithi033.stdout: "take": 0, 2024-03-18T15:28:55.706 INFO:teuthology.orchestra.run.smithi033.stdout: "take_sum": 0, 2024-03-18T15:28:55.706 INFO:teuthology.orchestra.run.smithi033.stdout: "put": 115, 2024-03-18T15:28:55.706 INFO:teuthology.orchestra.run.smithi033.stdout: "put_sum": 279719, 2024-03-18T15:28:55.706 INFO:teuthology.orchestra.run.smithi033.stdout: "wait": { 2024-03-18T15:28:55.706 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:28:55.706 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:28:55.706 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:28:55.706 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-18T15:28:55.706 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-18T15:28:55.707 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-03-18T15:28:55.707 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:28:55 smithi104 ceph-mon[36070]: pgmap v324: 129 pgs: 129 active+clean; 1.7 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:28:55.707 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:28:55 smithi104 ceph-mon[36070]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:28:55.707 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:28:55 smithi104 ceph-mon[36070]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:28:55.707 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:28:55 smithi104 ceph-mon[36070]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:28:55.707 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:28:55 smithi104 ceph-mon[36070]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:28:55.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:28:55 smithi033 ceph-mon[36648]: pgmap v324: 129 pgs: 129 active+clean; 1.7 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:28:55.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:28:55 smithi033 ceph-mon[36648]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:28:55.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:28:55 smithi033 ceph-mon[36648]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:28:55.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:28:55 smithi033 ceph-mon[36648]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:28:55.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:28:55 smithi033 ceph-mon[36648]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:28:55.862 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:28:55 smithi019 ceph-mon[31522]: pgmap v324: 129 pgs: 129 active+clean; 1.7 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:28:55.862 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:28:55 smithi019 ceph-mon[31522]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:28:55.862 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:28:55 smithi019 ceph-mon[31522]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:28:55.862 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:28:55 smithi019 ceph-mon[31522]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:28:55.862 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:28:55 smithi019 ceph-mon[31522]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:28:56.585 INFO:tasks.check_counter:Daemon mds.b mds.exported=1 2024-03-18T15:28:56.586 INFO:tasks.check_counter:Daemon mds.b mds.imported=1 2024-03-18T15:28:56.586 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update=7 2024-03-18T15:28:56.586 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update_receipt=3 2024-03-18T15:28:56.586 INFO:tasks.check_counter:Daemon mds.b mds.root_rsnaps=6 2024-03-18T15:28:56.586 INFO:tasks.check_counter:Daemon mds.b mds_server.req_mksnap_latency.avgcount=10 2024-03-18T15:28:56.586 INFO:tasks.check_counter:Daemon mds.b mds_server.req_rmsnap_latency.avgcount=4 2024-03-18T15:28:56.586 DEBUG:tasks.check_counter:Getting stats from e 2024-03-18T15:28:56.586 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 -- ceph daemon mds.e perf dump 2024-03-18T15:28:57.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:28:57 smithi033 ceph-mon[36648]: pgmap v325: 129 pgs: 129 active+clean; 1.7 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:28:57.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:28:57 smithi033 ceph-mon[36648]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:28:57.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:28:57 smithi033 ceph-mon[36648]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:28:57.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:28:57 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:28:57.507 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:28:57 smithi033 ceph-mon[36648]: from='mgr.25466 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:28:57.508 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:28:57 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:28:57.508 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:28:57 smithi033 ceph-mon[36648]: from='mgr.25466 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:28:57.508 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:28:57 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:28:57.508 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:28:57 smithi033 ceph-mon[36648]: from='mgr.25466 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:28:57.508 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:28:57 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:28:57.508 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:28:57 smithi033 ceph-mon[36648]: from='mgr.25466 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:28:57.551 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T15:28:57.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:28:57 smithi104 ceph-mon[36070]: pgmap v325: 129 pgs: 129 active+clean; 1.7 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:28:57.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:28:57 smithi104 ceph-mon[36070]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:28:57.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:28:57 smithi104 ceph-mon[36070]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:28:57.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:28:57 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:28:57.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:28:57 smithi104 ceph-mon[36070]: from='mgr.25466 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:28:57.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:28:57 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:28:57.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:28:57 smithi104 ceph-mon[36070]: from='mgr.25466 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:28:57.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:28:57 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:28:57.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:28:57 smithi104 ceph-mon[36070]: from='mgr.25466 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:28:57.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:28:57 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:28:57.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:28:57 smithi104 ceph-mon[36070]: from='mgr.25466 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:28:57.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:28:57 smithi019 ceph-mon[31522]: pgmap v325: 129 pgs: 129 active+clean; 1.7 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-18T15:28:57.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:28:57 smithi019 ceph-mon[31522]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:28:57.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:28:57 smithi019 ceph-mon[31522]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:28:57.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:28:57 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:28:57.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:28:57 smithi019 ceph-mon[31522]: from='mgr.25466 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:28:57.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:28:57 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:28:57.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:28:57 smithi019 ceph-mon[31522]: from='mgr.25466 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:28:57.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:28:57 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:28:57.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:28:57 smithi019 ceph-mon[31522]: from='mgr.25466 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:28:57.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:28:57 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:28:57.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:28:57 smithi019 ceph-mon[31522]: from='mgr.25466 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:28:58.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:28:58 smithi033 ceph-mon[36648]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-03-18T15:28:58.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:28:58 smithi033 ceph-mon[36648]: Unable to set osd_memory_target on smithi104 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-18T15:28:58.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:28:58 smithi033 ceph-mon[36648]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:28:58.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:28:58 smithi033 ceph-mon[36648]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:28:58.822 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:28:58 smithi019 ceph-mon[31522]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-03-18T15:28:58.822 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:28:58 smithi019 ceph-mon[31522]: Unable to set osd_memory_target on smithi104 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-18T15:28:58.822 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:28:58 smithi019 ceph-mon[31522]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:28:58.822 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:28:58 smithi019 ceph-mon[31522]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:28:58.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:28:58 smithi104 ceph-mon[36070]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-03-18T15:28:58.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:28:58 smithi104 ceph-mon[36070]: Unable to set osd_memory_target on smithi104 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-18T15:28:58.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:28:58 smithi104 ceph-mon[36070]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:28:58.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:28:58 smithi104 ceph-mon[36070]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:28:59.633 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-03-18T15:28:59.633 INFO:teuthology.orchestra.run.smithi033.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T15:28:59.633 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_messages": 1645, 2024-03-18T15:28:59.633 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_messages": 1647, 2024-03-18T15:28:59.633 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_bytes": 768856, 2024-03-18T15:28:59.633 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_bytes": 523120, 2024-03-18T15:28:59.633 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_created_connections": 6, 2024-03-18T15:28:59.633 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_active_connections": 2, 2024-03-18T15:28:59.633 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_total_time": 0.108255500, 2024-03-18T15:28:59.633 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_send_time": 0.038101952, 2024-03-18T15:28:59.633 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_recv_time": 0.026908094, 2024-03-18T15:28:59.633 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_fast_dispatch_time": 0.033465854, 2024-03-18T15:28:59.634 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T15:28:59.634 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 1645, 2024-03-18T15:28:59.634 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.035908698, 2024-03-18T15:28:59.634 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000021828 2024-03-18T15:28:59.634 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:59.634 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_handle_ack_lat": { 2024-03-18T15:28:59.634 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:28:59.634 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:28:59.634 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:28:59.634 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:59.634 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-18T15:28:59.635 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_encrypted_bytes": 0 2024-03-18T15:28:59.635 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:59.635 INFO:teuthology.orchestra.run.smithi033.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T15:28:59.635 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_messages": 11, 2024-03-18T15:28:59.635 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_messages": 183, 2024-03-18T15:28:59.635 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_bytes": 5208, 2024-03-18T15:28:59.635 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_bytes": 538842, 2024-03-18T15:28:59.635 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_created_connections": 8, 2024-03-18T15:28:59.635 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_active_connections": 5, 2024-03-18T15:28:59.635 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_total_time": 0.009480993, 2024-03-18T15:28:59.635 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_send_time": 0.005390353, 2024-03-18T15:28:59.635 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_recv_time": 0.002509167, 2024-03-18T15:28:59.635 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_fast_dispatch_time": 0.000095443, 2024-03-18T15:28:59.636 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T15:28:59.636 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 180, 2024-03-18T15:28:59.636 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.038098616, 2024-03-18T15:28:59.636 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000211658 2024-03-18T15:28:59.636 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:59.636 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_handle_ack_lat": { 2024-03-18T15:28:59.636 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:28:59.636 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:28:59.636 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:28:59.636 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:59.636 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_encrypted_bytes": 3552, 2024-03-18T15:28:59.636 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_encrypted_bytes": 537504 2024-03-18T15:28:59.636 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:59.636 INFO:teuthology.orchestra.run.smithi033.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T15:28:59.636 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_messages": 1059, 2024-03-18T15:28:59.637 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_messages": 1041, 2024-03-18T15:28:59.637 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_bytes": 1973209, 2024-03-18T15:28:59.637 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_bytes": 396657, 2024-03-18T15:28:59.637 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_created_connections": 3, 2024-03-18T15:28:59.637 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_active_connections": 2, 2024-03-18T15:28:59.637 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_total_time": 0.091635528, 2024-03-18T15:28:59.637 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_send_time": 0.023743583, 2024-03-18T15:28:59.637 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_recv_time": 0.052249401, 2024-03-18T15:28:59.637 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_fast_dispatch_time": 0.024779437, 2024-03-18T15:28:59.638 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T15:28:59.638 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 1040, 2024-03-18T15:28:59.638 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.023282177, 2024-03-18T15:28:59.638 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000022386 2024-03-18T15:28:59.638 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:59.639 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_handle_ack_lat": { 2024-03-18T15:28:59.639 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:28:59.639 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:28:59.639 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:28:59.639 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:59.639 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_encrypted_bytes": 1777872, 2024-03-18T15:28:59.639 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_encrypted_bytes": 144800 2024-03-18T15:28:59.639 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:59.640 INFO:teuthology.orchestra.run.smithi033.stdout: "cct": { 2024-03-18T15:28:59.640 INFO:teuthology.orchestra.run.smithi033.stdout: "total_workers": 1, 2024-03-18T15:28:59.640 INFO:teuthology.orchestra.run.smithi033.stdout: "unhealthy_workers": 0 2024-03-18T15:28:59.640 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:59.640 INFO:teuthology.orchestra.run.smithi033.stdout: "finisher-MDSRank": { 2024-03-18T15:28:59.640 INFO:teuthology.orchestra.run.smithi033.stdout: "queue_len": 0, 2024-03-18T15:28:59.640 INFO:teuthology.orchestra.run.smithi033.stdout: "complete_latency": { 2024-03-18T15:28:59.640 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 4044, 2024-03-18T15:28:59.640 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.081918254, 2024-03-18T15:28:59.640 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000020256 2024-03-18T15:28:59.640 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-18T15:28:59.640 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:59.640 INFO:teuthology.orchestra.run.smithi033.stdout: "finisher-PurgeQueue": { 2024-03-18T15:28:59.640 INFO:teuthology.orchestra.run.smithi033.stdout: "queue_len": 0, 2024-03-18T15:28:59.641 INFO:teuthology.orchestra.run.smithi033.stdout: "complete_latency": { 2024-03-18T15:28:59.641 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:28:59.641 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:28:59.641 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:28:59.641 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-18T15:28:59.641 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:59.641 INFO:teuthology.orchestra.run.smithi033.stdout: "mds": { 2024-03-18T15:28:59.641 INFO:teuthology.orchestra.run.smithi033.stdout: "request": 0, 2024-03-18T15:28:59.641 INFO:teuthology.orchestra.run.smithi033.stdout: "reply": 0, 2024-03-18T15:28:59.641 INFO:teuthology.orchestra.run.smithi033.stdout: "reply_latency": { 2024-03-18T15:28:59.642 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:28:59.642 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:28:59.642 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:28:59.642 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:59.642 INFO:teuthology.orchestra.run.smithi033.stdout: "slow_reply": 0, 2024-03-18T15:28:59.642 INFO:teuthology.orchestra.run.smithi033.stdout: "forward": 0, 2024-03-18T15:28:59.642 INFO:teuthology.orchestra.run.smithi033.stdout: "dir_fetch_complete": 0, 2024-03-18T15:28:59.642 INFO:teuthology.orchestra.run.smithi033.stdout: "dir_fetch_keys": 0, 2024-03-18T15:28:59.642 INFO:teuthology.orchestra.run.smithi033.stdout: "dir_commit": 0, 2024-03-18T15:28:59.642 INFO:teuthology.orchestra.run.smithi033.stdout: "dir_split": 0, 2024-03-18T15:28:59.642 INFO:teuthology.orchestra.run.smithi033.stdout: "dir_merge": 0, 2024-03-18T15:28:59.642 INFO:teuthology.orchestra.run.smithi033.stdout: "inodes": 18, 2024-03-18T15:28:59.642 INFO:teuthology.orchestra.run.smithi033.stdout: "inodes_top": 0, 2024-03-18T15:28:59.642 INFO:teuthology.orchestra.run.smithi033.stdout: "inodes_bottom": 18, 2024-03-18T15:28:59.643 INFO:teuthology.orchestra.run.smithi033.stdout: "inodes_pin_tail": 0, 2024-03-18T15:28:59.643 INFO:teuthology.orchestra.run.smithi033.stdout: "inodes_pinned": 18, 2024-03-18T15:28:59.643 INFO:teuthology.orchestra.run.smithi033.stdout: "inodes_expired": 0, 2024-03-18T15:28:59.643 INFO:teuthology.orchestra.run.smithi033.stdout: "inodes_with_caps": 0, 2024-03-18T15:28:59.643 INFO:teuthology.orchestra.run.smithi033.stdout: "caps": 0, 2024-03-18T15:28:59.643 INFO:teuthology.orchestra.run.smithi033.stdout: "subtrees": 2, 2024-03-18T15:28:59.643 INFO:teuthology.orchestra.run.smithi033.stdout: "traverse": 0, 2024-03-18T15:28:59.643 INFO:teuthology.orchestra.run.smithi033.stdout: "traverse_hit": 0, 2024-03-18T15:28:59.643 INFO:teuthology.orchestra.run.smithi033.stdout: "traverse_forward": 0, 2024-03-18T15:28:59.643 INFO:teuthology.orchestra.run.smithi033.stdout: "traverse_discover": 0, 2024-03-18T15:28:59.643 INFO:teuthology.orchestra.run.smithi033.stdout: "traverse_dir_fetch": 0, 2024-03-18T15:28:59.643 INFO:teuthology.orchestra.run.smithi033.stdout: "traverse_remote_ino": 0, 2024-03-18T15:28:59.644 INFO:teuthology.orchestra.run.smithi033.stdout: "traverse_lock": 0, 2024-03-18T15:28:59.644 INFO:teuthology.orchestra.run.smithi033.stdout: "load_cent": 0, 2024-03-18T15:28:59.644 INFO:teuthology.orchestra.run.smithi033.stdout: "q": 0, 2024-03-18T15:28:59.644 INFO:teuthology.orchestra.run.smithi033.stdout: "exported": 0, 2024-03-18T15:28:59.644 INFO:teuthology.orchestra.run.smithi033.stdout: "exported_inodes": 0, 2024-03-18T15:28:59.644 INFO:teuthology.orchestra.run.smithi033.stdout: "imported": 0, 2024-03-18T15:28:59.644 INFO:teuthology.orchestra.run.smithi033.stdout: "imported_inodes": 0, 2024-03-18T15:28:59.644 INFO:teuthology.orchestra.run.smithi033.stdout: "openino_dir_fetch": 0, 2024-03-18T15:28:59.644 INFO:teuthology.orchestra.run.smithi033.stdout: "openino_backtrace_fetch": 0, 2024-03-18T15:28:59.644 INFO:teuthology.orchestra.run.smithi033.stdout: "openino_peer_discover": 0, 2024-03-18T15:28:59.644 INFO:teuthology.orchestra.run.smithi033.stdout: "root_rfiles": 4, 2024-03-18T15:28:59.644 INFO:teuthology.orchestra.run.smithi033.stdout: "root_rbytes": 234, 2024-03-18T15:28:59.644 INFO:teuthology.orchestra.run.smithi033.stdout: "root_rsnaps": 6, 2024-03-18T15:28:59.644 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_backtrace_fetch": 0, 2024-03-18T15:28:59.644 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_set_tag": 0, 2024-03-18T15:28:59.645 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_backtrace_repaired": 0, 2024-03-18T15:28:59.645 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_inotable_repaired": 0, 2024-03-18T15:28:59.645 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_dir_inodes": 0, 2024-03-18T15:28:59.645 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_dir_base_inodes": 0, 2024-03-18T15:28:59.645 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_dirfrag_rstats": 0, 2024-03-18T15:28:59.645 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_file_inodes": 0, 2024-03-18T15:28:59.645 INFO:teuthology.orchestra.run.smithi033.stdout: "handle_inode_file_caps": 0, 2024-03-18T15:28:59.645 INFO:teuthology.orchestra.run.smithi033.stdout: "ceph_cap_op_revoke": 0, 2024-03-18T15:28:59.645 INFO:teuthology.orchestra.run.smithi033.stdout: "ceph_cap_op_grant": 0, 2024-03-18T15:28:59.645 INFO:teuthology.orchestra.run.smithi033.stdout: "ceph_cap_op_trunc": 0, 2024-03-18T15:28:59.645 INFO:teuthology.orchestra.run.smithi033.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-18T15:28:59.645 INFO:teuthology.orchestra.run.smithi033.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-18T15:28:59.645 INFO:teuthology.orchestra.run.smithi033.stdout: "handle_client_caps": 0, 2024-03-18T15:28:59.645 INFO:teuthology.orchestra.run.smithi033.stdout: "handle_client_caps_dirty": 0, 2024-03-18T15:28:59.646 INFO:teuthology.orchestra.run.smithi033.stdout: "handle_client_cap_release": 0, 2024-03-18T15:28:59.646 INFO:teuthology.orchestra.run.smithi033.stdout: "process_request_cap_release": 0 2024-03-18T15:28:59.646 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:59.646 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_cache": { 2024-03-18T15:28:59.646 INFO:teuthology.orchestra.run.smithi033.stdout: "dir_update": 0, 2024-03-18T15:28:59.646 INFO:teuthology.orchestra.run.smithi033.stdout: "dir_update_receipt": 0, 2024-03-18T15:28:59.646 INFO:teuthology.orchestra.run.smithi033.stdout: "dir_try_discover": 0, 2024-03-18T15:28:59.646 INFO:teuthology.orchestra.run.smithi033.stdout: "dir_send_discover": 0, 2024-03-18T15:28:59.646 INFO:teuthology.orchestra.run.smithi033.stdout: "dir_handle_discover": 0, 2024-03-18T15:28:59.646 INFO:teuthology.orchestra.run.smithi033.stdout: "num_strays": 0, 2024-03-18T15:28:59.646 INFO:teuthology.orchestra.run.smithi033.stdout: "num_strays_delayed": 0, 2024-03-18T15:28:59.646 INFO:teuthology.orchestra.run.smithi033.stdout: "num_strays_enqueuing": 0, 2024-03-18T15:28:59.646 INFO:teuthology.orchestra.run.smithi033.stdout: "strays_created": 0, 2024-03-18T15:28:59.646 INFO:teuthology.orchestra.run.smithi033.stdout: "strays_enqueued": 0, 2024-03-18T15:28:59.646 INFO:teuthology.orchestra.run.smithi033.stdout: "strays_reintegrated": 0, 2024-03-18T15:28:59.647 INFO:teuthology.orchestra.run.smithi033.stdout: "strays_migrated": 0, 2024-03-18T15:28:59.647 INFO:teuthology.orchestra.run.smithi033.stdout: "num_recovering_processing": 0, 2024-03-18T15:28:59.647 INFO:teuthology.orchestra.run.smithi033.stdout: "num_recovering_enqueued": 0, 2024-03-18T15:28:59.647 INFO:teuthology.orchestra.run.smithi033.stdout: "num_recovering_prioritized": 0, 2024-03-18T15:28:59.647 INFO:teuthology.orchestra.run.smithi033.stdout: "recovery_started": 0, 2024-03-18T15:28:59.647 INFO:teuthology.orchestra.run.smithi033.stdout: "recovery_completed": 0, 2024-03-18T15:28:59.647 INFO:teuthology.orchestra.run.smithi033.stdout: "ireq_quiesce_path": 0, 2024-03-18T15:28:59.647 INFO:teuthology.orchestra.run.smithi033.stdout: "ireq_quiesce_inode": 0, 2024-03-18T15:28:59.647 INFO:teuthology.orchestra.run.smithi033.stdout: "ireq_enqueue_scrub": 0, 2024-03-18T15:28:59.647 INFO:teuthology.orchestra.run.smithi033.stdout: "ireq_exportdir": 0, 2024-03-18T15:28:59.647 INFO:teuthology.orchestra.run.smithi033.stdout: "ireq_flush": 0, 2024-03-18T15:28:59.647 INFO:teuthology.orchestra.run.smithi033.stdout: "ireq_fragmentdir": 0, 2024-03-18T15:28:59.647 INFO:teuthology.orchestra.run.smithi033.stdout: "ireq_fragstats": 0, 2024-03-18T15:28:59.647 INFO:teuthology.orchestra.run.smithi033.stdout: "ireq_inodestats": 0 2024-03-18T15:28:59.648 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:59.648 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_log": { 2024-03-18T15:28:59.648 INFO:teuthology.orchestra.run.smithi033.stdout: "evlrg": 0, 2024-03-18T15:28:59.648 INFO:teuthology.orchestra.run.smithi033.stdout: "evadd": 0, 2024-03-18T15:28:59.648 INFO:teuthology.orchestra.run.smithi033.stdout: "evex": 0, 2024-03-18T15:28:59.648 INFO:teuthology.orchestra.run.smithi033.stdout: "evtrm": 0, 2024-03-18T15:28:59.648 INFO:teuthology.orchestra.run.smithi033.stdout: "ev": 143, 2024-03-18T15:28:59.648 INFO:teuthology.orchestra.run.smithi033.stdout: "evexg": 0, 2024-03-18T15:28:59.648 INFO:teuthology.orchestra.run.smithi033.stdout: "evexd": 0, 2024-03-18T15:28:59.648 INFO:teuthology.orchestra.run.smithi033.stdout: "segadd": 0, 2024-03-18T15:28:59.648 INFO:teuthology.orchestra.run.smithi033.stdout: "segex": 0, 2024-03-18T15:28:59.648 INFO:teuthology.orchestra.run.smithi033.stdout: "segtrm": 0, 2024-03-18T15:28:59.648 INFO:teuthology.orchestra.run.smithi033.stdout: "seg": 2, 2024-03-18T15:28:59.648 INFO:teuthology.orchestra.run.smithi033.stdout: "segmjr": 2, 2024-03-18T15:28:59.648 INFO:teuthology.orchestra.run.smithi033.stdout: "segexg": 0, 2024-03-18T15:28:59.649 INFO:teuthology.orchestra.run.smithi033.stdout: "segexd": 0, 2024-03-18T15:28:59.649 INFO:teuthology.orchestra.run.smithi033.stdout: "expos": 4194304, 2024-03-18T15:28:59.649 INFO:teuthology.orchestra.run.smithi033.stdout: "wrpos": 4474023, 2024-03-18T15:28:59.649 INFO:teuthology.orchestra.run.smithi033.stdout: "rdpos": 4473887, 2024-03-18T15:28:59.649 INFO:teuthology.orchestra.run.smithi033.stdout: "jlat": { 2024-03-18T15:28:59.649 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:28:59.649 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:28:59.649 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:28:59.649 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:59.649 INFO:teuthology.orchestra.run.smithi033.stdout: "replayed": 143 2024-03-18T15:28:59.649 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:59.649 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_mem": { 2024-03-18T15:28:59.649 INFO:teuthology.orchestra.run.smithi033.stdout: "ino": 15, 2024-03-18T15:28:59.649 INFO:teuthology.orchestra.run.smithi033.stdout: "ino+": 20, 2024-03-18T15:28:59.650 INFO:teuthology.orchestra.run.smithi033.stdout: "ino-": 5, 2024-03-18T15:28:59.650 INFO:teuthology.orchestra.run.smithi033.stdout: "dir": 10, 2024-03-18T15:28:59.650 INFO:teuthology.orchestra.run.smithi033.stdout: "dir+": 11, 2024-03-18T15:28:59.650 INFO:teuthology.orchestra.run.smithi033.stdout: "dir-": 1, 2024-03-18T15:28:59.650 INFO:teuthology.orchestra.run.smithi033.stdout: "dn": 18, 2024-03-18T15:28:59.650 INFO:teuthology.orchestra.run.smithi033.stdout: "dn+": 23, 2024-03-18T15:28:59.650 INFO:teuthology.orchestra.run.smithi033.stdout: "dn-": 5, 2024-03-18T15:28:59.650 INFO:teuthology.orchestra.run.smithi033.stdout: "cap": 0, 2024-03-18T15:28:59.650 INFO:teuthology.orchestra.run.smithi033.stdout: "cap+": 0, 2024-03-18T15:28:59.650 INFO:teuthology.orchestra.run.smithi033.stdout: "cap-": 0, 2024-03-18T15:28:59.651 INFO:teuthology.orchestra.run.smithi033.stdout: "rss": 49456, 2024-03-18T15:28:59.651 INFO:teuthology.orchestra.run.smithi033.stdout: "heap": 215420 2024-03-18T15:28:59.651 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:59.651 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_server": { 2024-03-18T15:28:59.651 INFO:teuthology.orchestra.run.smithi033.stdout: "dispatch_client_request": 0, 2024-03-18T15:28:59.651 INFO:teuthology.orchestra.run.smithi033.stdout: "dispatch_server_request": 0, 2024-03-18T15:28:59.651 INFO:teuthology.orchestra.run.smithi033.stdout: "handle_client_request": 0, 2024-03-18T15:28:59.651 INFO:teuthology.orchestra.run.smithi033.stdout: "handle_client_session": 0, 2024-03-18T15:28:59.651 INFO:teuthology.orchestra.run.smithi033.stdout: "handle_peer_request": 0, 2024-03-18T15:28:59.651 INFO:teuthology.orchestra.run.smithi033.stdout: "req_create_latency": { 2024-03-18T15:28:59.651 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:28:59.651 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:28:59.651 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:28:59.652 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:59.652 INFO:teuthology.orchestra.run.smithi033.stdout: "req_getattr_latency": { 2024-03-18T15:28:59.652 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:28:59.652 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:28:59.652 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:28:59.652 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:59.652 INFO:teuthology.orchestra.run.smithi033.stdout: "req_getfilelock_latency": { 2024-03-18T15:28:59.652 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:28:59.652 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:28:59.652 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:28:59.652 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:59.652 INFO:teuthology.orchestra.run.smithi033.stdout: "req_link_latency": { 2024-03-18T15:28:59.652 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:28:59.652 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:28:59.653 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:28:59.653 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:59.653 INFO:teuthology.orchestra.run.smithi033.stdout: "req_lookup_latency": { 2024-03-18T15:28:59.653 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:28:59.653 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:28:59.653 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:28:59.653 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:59.653 INFO:teuthology.orchestra.run.smithi033.stdout: "req_lookuphash_latency": { 2024-03-18T15:28:59.653 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:28:59.653 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:28:59.653 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:28:59.653 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:59.653 INFO:teuthology.orchestra.run.smithi033.stdout: "req_lookupino_latency": { 2024-03-18T15:28:59.653 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:28:59.654 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:28:59.654 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:28:59.654 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:59.654 INFO:teuthology.orchestra.run.smithi033.stdout: "req_lookupname_latency": { 2024-03-18T15:28:59.654 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:28:59.654 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:28:59.654 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:28:59.654 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:59.654 INFO:teuthology.orchestra.run.smithi033.stdout: "req_lookupparent_latency": { 2024-03-18T15:28:59.654 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:28:59.654 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:28:59.654 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:28:59.655 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:59.655 INFO:teuthology.orchestra.run.smithi033.stdout: "req_lookupsnap_latency": { 2024-03-18T15:28:59.655 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:28:59.655 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:28:59.655 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:28:59.655 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:59.655 INFO:teuthology.orchestra.run.smithi033.stdout: "req_lssnap_latency": { 2024-03-18T15:28:59.655 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:28:59.655 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:28:59.655 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:28:59.655 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:59.655 INFO:teuthology.orchestra.run.smithi033.stdout: "req_mkdir_latency": { 2024-03-18T15:28:59.655 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:28:59.656 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:28:59.656 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:28:59.656 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:59.656 INFO:teuthology.orchestra.run.smithi033.stdout: "req_mknod_latency": { 2024-03-18T15:28:59.656 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:28:59.656 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:28:59.656 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:28:59.656 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:59.656 INFO:teuthology.orchestra.run.smithi033.stdout: "req_mksnap_latency": { 2024-03-18T15:28:59.656 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:28:59.656 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:28:59.656 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:28:59.657 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:59.657 INFO:teuthology.orchestra.run.smithi033.stdout: "req_open_latency": { 2024-03-18T15:28:59.657 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:28:59.657 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:28:59.657 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:28:59.657 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:59.657 INFO:teuthology.orchestra.run.smithi033.stdout: "req_readdir_latency": { 2024-03-18T15:28:59.657 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:28:59.657 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:28:59.657 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:28:59.657 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:59.657 INFO:teuthology.orchestra.run.smithi033.stdout: "req_rename_latency": { 2024-03-18T15:28:59.657 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:28:59.657 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:28:59.658 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:28:59.658 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:59.658 INFO:teuthology.orchestra.run.smithi033.stdout: "req_renamesnap_latency": { 2024-03-18T15:28:59.658 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:28:59.658 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:28:59.658 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:28:59.658 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:59.658 INFO:teuthology.orchestra.run.smithi033.stdout: "req_snapdiff_latency": { 2024-03-18T15:28:59.658 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:28:59.658 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:28:59.658 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:28:59.658 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:59.658 INFO:teuthology.orchestra.run.smithi033.stdout: "req_rmdir_latency": { 2024-03-18T15:28:59.658 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:28:59.659 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:28:59.659 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:28:59.659 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:59.659 INFO:teuthology.orchestra.run.smithi033.stdout: "req_rmsnap_latency": { 2024-03-18T15:28:59.659 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:28:59.659 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:28:59.659 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:28:59.659 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:59.659 INFO:teuthology.orchestra.run.smithi033.stdout: "req_rmxattr_latency": { 2024-03-18T15:28:59.659 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:28:59.659 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:28:59.659 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:28:59.659 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:59.659 INFO:teuthology.orchestra.run.smithi033.stdout: "req_setattr_latency": { 2024-03-18T15:28:59.660 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:28:59.660 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:28:59.660 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:28:59.660 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:59.660 INFO:teuthology.orchestra.run.smithi033.stdout: "req_setdirlayout_latency": { 2024-03-18T15:28:59.660 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:28:59.660 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:28:59.660 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:28:59.660 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:59.660 INFO:teuthology.orchestra.run.smithi033.stdout: "req_setfilelock_latency": { 2024-03-18T15:28:59.660 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:28:59.660 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:28:59.660 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:28:59.661 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:59.661 INFO:teuthology.orchestra.run.smithi033.stdout: "req_setlayout_latency": { 2024-03-18T15:28:59.661 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:28:59.661 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:28:59.661 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:28:59.661 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:59.661 INFO:teuthology.orchestra.run.smithi033.stdout: "req_setxattr_latency": { 2024-03-18T15:28:59.661 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:28:59.661 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:28:59.661 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:28:59.661 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:59.661 INFO:teuthology.orchestra.run.smithi033.stdout: "req_symlink_latency": { 2024-03-18T15:28:59.661 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:28:59.661 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:28:59.662 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:28:59.662 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:59.662 INFO:teuthology.orchestra.run.smithi033.stdout: "req_unlink_latency": { 2024-03-18T15:28:59.662 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:28:59.662 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:28:59.662 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:28:59.662 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:59.662 INFO:teuthology.orchestra.run.smithi033.stdout: "cap_revoke_eviction": 0, 2024-03-18T15:28:59.662 INFO:teuthology.orchestra.run.smithi033.stdout: "cap_acquisition_throttle": 0, 2024-03-18T15:28:59.662 INFO:teuthology.orchestra.run.smithi033.stdout: "req_getvxattr_latency": { 2024-03-18T15:28:59.662 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:28:59.662 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:28:59.662 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:28:59.663 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-18T15:28:59.663 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:59.663 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_sessions": { 2024-03-18T15:28:59.663 INFO:teuthology.orchestra.run.smithi033.stdout: "session_count": 2, 2024-03-18T15:28:59.663 INFO:teuthology.orchestra.run.smithi033.stdout: "session_add": 7, 2024-03-18T15:28:59.663 INFO:teuthology.orchestra.run.smithi033.stdout: "session_remove": 5, 2024-03-18T15:28:59.663 INFO:teuthology.orchestra.run.smithi033.stdout: "sessions_open": 5, 2024-03-18T15:28:59.663 INFO:teuthology.orchestra.run.smithi033.stdout: "sessions_stale": 0, 2024-03-18T15:28:59.663 INFO:teuthology.orchestra.run.smithi033.stdout: "total_load": 0, 2024-03-18T15:28:59.663 INFO:teuthology.orchestra.run.smithi033.stdout: "average_load": 0, 2024-03-18T15:28:59.663 INFO:teuthology.orchestra.run.smithi033.stdout: "avg_session_uptime": 4802, 2024-03-18T15:28:59.663 INFO:teuthology.orchestra.run.smithi033.stdout: "mdthresh_evicted": 0 2024-03-18T15:28:59.663 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:59.664 INFO:teuthology.orchestra.run.smithi033.stdout: "mempool": { 2024-03-18T15:28:59.664 INFO:teuthology.orchestra.run.smithi033.stdout: "bloom_filter_bytes": 0, 2024-03-18T15:28:59.664 INFO:teuthology.orchestra.run.smithi033.stdout: "bloom_filter_items": 0, 2024-03-18T15:28:59.664 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T15:28:59.664 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_alloc_items": 0, 2024-03-18T15:28:59.664 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T15:28:59.664 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_data_items": 0, 2024-03-18T15:28:59.664 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T15:28:59.664 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T15:28:59.664 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T15:28:59.664 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T15:28:59.664 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T15:28:59.665 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_other_items": 0, 2024-03-18T15:28:59.665 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T15:28:59.665 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T15:28:59.665 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_extent_bytes": 0, 2024-03-18T15:28:59.665 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_extent_items": 0, 2024-03-18T15:28:59.665 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_blob_bytes": 0, 2024-03-18T15:28:59.665 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_blob_items": 0, 2024-03-18T15:28:59.665 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T15:28:59.665 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T15:28:59.665 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T15:28:59.665 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T15:28:59.665 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T15:28:59.666 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_fsck_items": 0, 2024-03-18T15:28:59.666 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_txc_bytes": 0, 2024-03-18T15:28:59.666 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_txc_items": 0, 2024-03-18T15:28:59.666 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T15:28:59.666 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T15:28:59.666 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_writing_bytes": 0, 2024-03-18T15:28:59.666 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_writing_items": 0, 2024-03-18T15:28:59.666 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_bytes": 0, 2024-03-18T15:28:59.666 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_items": 0, 2024-03-18T15:28:59.666 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T15:28:59.666 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_file_reader_items": 0, 2024-03-18T15:28:59.666 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T15:28:59.666 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_file_writer_items": 0, 2024-03-18T15:28:59.667 INFO:teuthology.orchestra.run.smithi033.stdout: "buffer_anon_bytes": 190312, 2024-03-18T15:28:59.667 INFO:teuthology.orchestra.run.smithi033.stdout: "buffer_anon_items": 56, 2024-03-18T15:28:59.667 INFO:teuthology.orchestra.run.smithi033.stdout: "buffer_meta_bytes": 0, 2024-03-18T15:28:59.667 INFO:teuthology.orchestra.run.smithi033.stdout: "buffer_meta_items": 0, 2024-03-18T15:28:59.667 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_bytes": 0, 2024-03-18T15:28:59.667 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_items": 0, 2024-03-18T15:28:59.667 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mapbl_bytes": 0, 2024-03-18T15:28:59.667 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mapbl_items": 0, 2024-03-18T15:28:59.667 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pglog_bytes": 0, 2024-03-18T15:28:59.667 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pglog_items": 0, 2024-03-18T15:28:59.667 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_bytes": 9928, 2024-03-18T15:28:59.667 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_items": 207, 2024-03-18T15:28:59.667 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T15:28:59.667 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_mapping_items": 0, 2024-03-18T15:28:59.668 INFO:teuthology.orchestra.run.smithi033.stdout: "pgmap_bytes": 0, 2024-03-18T15:28:59.668 INFO:teuthology.orchestra.run.smithi033.stdout: "pgmap_items": 0, 2024-03-18T15:28:59.668 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_co_bytes": 69004, 2024-03-18T15:28:59.668 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_co_items": 408, 2024-03-18T15:28:59.668 INFO:teuthology.orchestra.run.smithi033.stdout: "unittest_1_bytes": 0, 2024-03-18T15:28:59.668 INFO:teuthology.orchestra.run.smithi033.stdout: "unittest_1_items": 0, 2024-03-18T15:28:59.668 INFO:teuthology.orchestra.run.smithi033.stdout: "unittest_2_bytes": 0, 2024-03-18T15:28:59.668 INFO:teuthology.orchestra.run.smithi033.stdout: "unittest_2_items": 0 2024-03-18T15:28:59.668 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:59.668 INFO:teuthology.orchestra.run.smithi033.stdout: "objecter": { 2024-03-18T15:28:59.668 INFO:teuthology.orchestra.run.smithi033.stdout: "op_active": 0, 2024-03-18T15:28:59.668 INFO:teuthology.orchestra.run.smithi033.stdout: "op_laggy": 0, 2024-03-18T15:28:59.668 INFO:teuthology.orchestra.run.smithi033.stdout: "op_send": 2441, 2024-03-18T15:28:59.669 INFO:teuthology.orchestra.run.smithi033.stdout: "op_send_bytes": 26, 2024-03-18T15:28:59.669 INFO:teuthology.orchestra.run.smithi033.stdout: "op_resend": 4, 2024-03-18T15:28:59.669 INFO:teuthology.orchestra.run.smithi033.stdout: "op_reply": 2441, 2024-03-18T15:28:59.669 INFO:teuthology.orchestra.run.smithi033.stdout: "op_latency": { 2024-03-18T15:28:59.669 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 2441, 2024-03-18T15:28:59.669 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 1.628972204, 2024-03-18T15:28:59.669 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000667338 2024-03-18T15:28:59.669 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:59.669 INFO:teuthology.orchestra.run.smithi033.stdout: "op_inflight": 0, 2024-03-18T15:28:59.669 INFO:teuthology.orchestra.run.smithi033.stdout: "oplen_avg": { 2024-03-18T15:28:59.669 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 2441, 2024-03-18T15:28:59.669 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 2442 2024-03-18T15:28:59.669 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:59.669 INFO:teuthology.orchestra.run.smithi033.stdout: "op": 2441, 2024-03-18T15:28:59.670 INFO:teuthology.orchestra.run.smithi033.stdout: "op_r": 2441, 2024-03-18T15:28:59.670 INFO:teuthology.orchestra.run.smithi033.stdout: "op_w": 0, 2024-03-18T15:28:59.670 INFO:teuthology.orchestra.run.smithi033.stdout: "op_rmw": 0, 2024-03-18T15:28:59.670 INFO:teuthology.orchestra.run.smithi033.stdout: "op_pg": 0, 2024-03-18T15:28:59.670 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_stat": 1582, 2024-03-18T15:28:59.670 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_create": 0, 2024-03-18T15:28:59.670 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_read": 856, 2024-03-18T15:28:59.670 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_write": 0, 2024-03-18T15:28:59.670 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_writefull": 0, 2024-03-18T15:28:59.670 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_writesame": 0, 2024-03-18T15:28:59.670 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_append": 0, 2024-03-18T15:28:59.670 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_zero": 0, 2024-03-18T15:28:59.671 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_truncate": 0, 2024-03-18T15:28:59.671 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_delete": 0, 2024-03-18T15:28:59.671 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_mapext": 0, 2024-03-18T15:28:59.671 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_sparse_read": 0, 2024-03-18T15:28:59.671 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_clonerange": 0, 2024-03-18T15:28:59.671 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_getxattr": 2, 2024-03-18T15:28:59.671 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_setxattr": 0, 2024-03-18T15:28:59.671 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_cmpxattr": 0, 2024-03-18T15:28:59.671 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_rmxattr": 0, 2024-03-18T15:28:59.671 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_resetxattrs": 0, 2024-03-18T15:28:59.672 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_call": 0, 2024-03-18T15:28:59.672 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_watch": 0, 2024-03-18T15:28:59.672 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_notify": 0, 2024-03-18T15:28:59.672 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_src_cmpxattr": 0, 2024-03-18T15:28:59.672 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_pgls": 0, 2024-03-18T15:28:59.672 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_pgls_filter": 0, 2024-03-18T15:28:59.672 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_other": 0, 2024-03-18T15:28:59.672 INFO:teuthology.orchestra.run.smithi033.stdout: "linger_active": 0, 2024-03-18T15:28:59.672 INFO:teuthology.orchestra.run.smithi033.stdout: "linger_send": 0, 2024-03-18T15:28:59.672 INFO:teuthology.orchestra.run.smithi033.stdout: "linger_resend": 0, 2024-03-18T15:28:59.672 INFO:teuthology.orchestra.run.smithi033.stdout: "linger_ping": 0, 2024-03-18T15:28:59.672 INFO:teuthology.orchestra.run.smithi033.stdout: "poolop_active": 0, 2024-03-18T15:28:59.672 INFO:teuthology.orchestra.run.smithi033.stdout: "poolop_send": 0, 2024-03-18T15:28:59.672 INFO:teuthology.orchestra.run.smithi033.stdout: "poolop_resend": 0, 2024-03-18T15:28:59.672 INFO:teuthology.orchestra.run.smithi033.stdout: "poolstat_active": 0, 2024-03-18T15:28:59.673 INFO:teuthology.orchestra.run.smithi033.stdout: "poolstat_send": 0, 2024-03-18T15:28:59.673 INFO:teuthology.orchestra.run.smithi033.stdout: "poolstat_resend": 0, 2024-03-18T15:28:59.673 INFO:teuthology.orchestra.run.smithi033.stdout: "statfs_active": 0, 2024-03-18T15:28:59.673 INFO:teuthology.orchestra.run.smithi033.stdout: "statfs_send": 0, 2024-03-18T15:28:59.673 INFO:teuthology.orchestra.run.smithi033.stdout: "statfs_resend": 0, 2024-03-18T15:28:59.673 INFO:teuthology.orchestra.run.smithi033.stdout: "command_active": 0, 2024-03-18T15:28:59.673 INFO:teuthology.orchestra.run.smithi033.stdout: "command_send": 0, 2024-03-18T15:28:59.673 INFO:teuthology.orchestra.run.smithi033.stdout: "command_resend": 0, 2024-03-18T15:28:59.673 INFO:teuthology.orchestra.run.smithi033.stdout: "map_epoch": 103, 2024-03-18T15:28:59.673 INFO:teuthology.orchestra.run.smithi033.stdout: "map_full": 0, 2024-03-18T15:28:59.673 INFO:teuthology.orchestra.run.smithi033.stdout: "map_inc": 14, 2024-03-18T15:28:59.673 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_sessions": 5, 2024-03-18T15:28:59.673 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_session_open": 5, 2024-03-18T15:28:59.673 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_session_close": 0, 2024-03-18T15:28:59.673 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_laggy": 0, 2024-03-18T15:28:59.674 INFO:teuthology.orchestra.run.smithi033.stdout: "omap_wr": 0, 2024-03-18T15:28:59.674 INFO:teuthology.orchestra.run.smithi033.stdout: "omap_rd": 2, 2024-03-18T15:28:59.674 INFO:teuthology.orchestra.run.smithi033.stdout: "omap_del": 0 2024-03-18T15:28:59.674 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:59.674 INFO:teuthology.orchestra.run.smithi033.stdout: "oft": { 2024-03-18T15:28:59.674 INFO:teuthology.orchestra.run.smithi033.stdout: "omap_total_objs": 0, 2024-03-18T15:28:59.674 INFO:teuthology.orchestra.run.smithi033.stdout: "omap_total_kv_pairs": 0, 2024-03-18T15:28:59.674 INFO:teuthology.orchestra.run.smithi033.stdout: "omap_total_updates": 0, 2024-03-18T15:28:59.674 INFO:teuthology.orchestra.run.smithi033.stdout: "omap_total_removes": 0 2024-03-18T15:28:59.674 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:59.674 INFO:teuthology.orchestra.run.smithi033.stdout: "osd-slow-ops": { 2024-03-18T15:28:59.674 INFO:teuthology.orchestra.run.smithi033.stdout: "slow_ops_count": 0 2024-03-18T15:28:59.674 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:59.674 INFO:teuthology.orchestra.run.smithi033.stdout: "purge_queue": { 2024-03-18T15:28:59.675 INFO:teuthology.orchestra.run.smithi033.stdout: "pq_executing_ops": 0, 2024-03-18T15:28:59.675 INFO:teuthology.orchestra.run.smithi033.stdout: "pq_executing_ops_high_water": 0, 2024-03-18T15:28:59.675 INFO:teuthology.orchestra.run.smithi033.stdout: "pq_executing": 0, 2024-03-18T15:28:59.675 INFO:teuthology.orchestra.run.smithi033.stdout: "pq_executing_high_water": 0, 2024-03-18T15:28:59.675 INFO:teuthology.orchestra.run.smithi033.stdout: "pq_executed_ops": 0, 2024-03-18T15:28:59.675 INFO:teuthology.orchestra.run.smithi033.stdout: "pq_executed": 0, 2024-03-18T15:28:59.675 INFO:teuthology.orchestra.run.smithi033.stdout: "pq_item_in_journal": 0 2024-03-18T15:28:59.675 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:59.675 INFO:teuthology.orchestra.run.smithi033.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T15:28:59.675 INFO:teuthology.orchestra.run.smithi033.stdout: "val": 0, 2024-03-18T15:28:59.676 INFO:teuthology.orchestra.run.smithi033.stdout: "max": 104857600, 2024-03-18T15:28:59.676 INFO:teuthology.orchestra.run.smithi033.stdout: "get_started": 0, 2024-03-18T15:28:59.676 INFO:teuthology.orchestra.run.smithi033.stdout: "get": 2711, 2024-03-18T15:28:59.676 INFO:teuthology.orchestra.run.smithi033.stdout: "get_sum": 2486567, 2024-03-18T15:28:59.676 INFO:teuthology.orchestra.run.smithi033.stdout: "get_or_fail_fail": 0, 2024-03-18T15:28:59.676 INFO:teuthology.orchestra.run.smithi033.stdout: "get_or_fail_success": 2711, 2024-03-18T15:28:59.676 INFO:teuthology.orchestra.run.smithi033.stdout: "take": 0, 2024-03-18T15:28:59.676 INFO:teuthology.orchestra.run.smithi033.stdout: "take_sum": 0, 2024-03-18T15:28:59.676 INFO:teuthology.orchestra.run.smithi033.stdout: "put": 2711, 2024-03-18T15:28:59.676 INFO:teuthology.orchestra.run.smithi033.stdout: "put_sum": 2486567, 2024-03-18T15:28:59.676 INFO:teuthology.orchestra.run.smithi033.stdout: "wait": { 2024-03-18T15:28:59.676 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:28:59.676 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:28:59.676 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:28:59.677 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-18T15:28:59.677 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:59.677 INFO:teuthology.orchestra.run.smithi033.stdout: "throttle-objecter_bytes": { 2024-03-18T15:28:59.677 INFO:teuthology.orchestra.run.smithi033.stdout: "val": 0, 2024-03-18T15:28:59.677 INFO:teuthology.orchestra.run.smithi033.stdout: "max": 104857600, 2024-03-18T15:28:59.677 INFO:teuthology.orchestra.run.smithi033.stdout: "get_started": 0, 2024-03-18T15:28:59.677 INFO:teuthology.orchestra.run.smithi033.stdout: "get": 0, 2024-03-18T15:28:59.677 INFO:teuthology.orchestra.run.smithi033.stdout: "get_sum": 0, 2024-03-18T15:28:59.677 INFO:teuthology.orchestra.run.smithi033.stdout: "get_or_fail_fail": 0, 2024-03-18T15:28:59.677 INFO:teuthology.orchestra.run.smithi033.stdout: "get_or_fail_success": 0, 2024-03-18T15:28:59.677 INFO:teuthology.orchestra.run.smithi033.stdout: "take": 2441, 2024-03-18T15:28:59.677 INFO:teuthology.orchestra.run.smithi033.stdout: "take_sum": 279729, 2024-03-18T15:28:59.677 INFO:teuthology.orchestra.run.smithi033.stdout: "put": 2441, 2024-03-18T15:28:59.677 INFO:teuthology.orchestra.run.smithi033.stdout: "put_sum": 279729, 2024-03-18T15:28:59.678 INFO:teuthology.orchestra.run.smithi033.stdout: "wait": { 2024-03-18T15:28:59.678 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:28:59.678 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:28:59.678 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:28:59.678 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-18T15:28:59.678 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:59.678 INFO:teuthology.orchestra.run.smithi033.stdout: "throttle-objecter_ops": { 2024-03-18T15:28:59.678 INFO:teuthology.orchestra.run.smithi033.stdout: "val": 0, 2024-03-18T15:28:59.678 INFO:teuthology.orchestra.run.smithi033.stdout: "max": 1024, 2024-03-18T15:28:59.678 INFO:teuthology.orchestra.run.smithi033.stdout: "get_started": 0, 2024-03-18T15:28:59.678 INFO:teuthology.orchestra.run.smithi033.stdout: "get": 0, 2024-03-18T15:28:59.678 INFO:teuthology.orchestra.run.smithi033.stdout: "get_sum": 0, 2024-03-18T15:28:59.678 INFO:teuthology.orchestra.run.smithi033.stdout: "get_or_fail_fail": 0, 2024-03-18T15:28:59.678 INFO:teuthology.orchestra.run.smithi033.stdout: "get_or_fail_success": 0, 2024-03-18T15:28:59.679 INFO:teuthology.orchestra.run.smithi033.stdout: "take": 2441, 2024-03-18T15:28:59.679 INFO:teuthology.orchestra.run.smithi033.stdout: "take_sum": 2441, 2024-03-18T15:28:59.679 INFO:teuthology.orchestra.run.smithi033.stdout: "put": 2441, 2024-03-18T15:28:59.679 INFO:teuthology.orchestra.run.smithi033.stdout: "put_sum": 2441, 2024-03-18T15:28:59.679 INFO:teuthology.orchestra.run.smithi033.stdout: "wait": { 2024-03-18T15:28:59.679 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:28:59.679 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:28:59.679 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:28:59.679 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-18T15:28:59.679 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:59.679 INFO:teuthology.orchestra.run.smithi033.stdout: "throttle-write_buf_throttle": { 2024-03-18T15:28:59.679 INFO:teuthology.orchestra.run.smithi033.stdout: "val": 0, 2024-03-18T15:28:59.679 INFO:teuthology.orchestra.run.smithi033.stdout: "max": 3758096384, 2024-03-18T15:28:59.679 INFO:teuthology.orchestra.run.smithi033.stdout: "get_started": 0, 2024-03-18T15:28:59.680 INFO:teuthology.orchestra.run.smithi033.stdout: "get": 0, 2024-03-18T15:28:59.680 INFO:teuthology.orchestra.run.smithi033.stdout: "get_sum": 0, 2024-03-18T15:28:59.680 INFO:teuthology.orchestra.run.smithi033.stdout: "get_or_fail_fail": 0, 2024-03-18T15:28:59.680 INFO:teuthology.orchestra.run.smithi033.stdout: "get_or_fail_success": 0, 2024-03-18T15:28:59.680 INFO:teuthology.orchestra.run.smithi033.stdout: "take": 0, 2024-03-18T15:28:59.680 INFO:teuthology.orchestra.run.smithi033.stdout: "take_sum": 0, 2024-03-18T15:28:59.680 INFO:teuthology.orchestra.run.smithi033.stdout: "put": 0, 2024-03-18T15:28:59.680 INFO:teuthology.orchestra.run.smithi033.stdout: "put_sum": 0, 2024-03-18T15:28:59.680 INFO:teuthology.orchestra.run.smithi033.stdout: "wait": { 2024-03-18T15:28:59.680 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:28:59.680 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:28:59.680 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:28:59.680 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-18T15:28:59.680 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:28:59.680 INFO:teuthology.orchestra.run.smithi033.stdout: "throttle-write_buf_throttle-0x558fa9e2e6e0": { 2024-03-18T15:28:59.681 INFO:teuthology.orchestra.run.smithi033.stdout: "val": 0, 2024-03-18T15:28:59.681 INFO:teuthology.orchestra.run.smithi033.stdout: "max": 3758096384, 2024-03-18T15:28:59.681 INFO:teuthology.orchestra.run.smithi033.stdout: "get_started": 0, 2024-03-18T15:28:59.681 INFO:teuthology.orchestra.run.smithi033.stdout: "get": 0, 2024-03-18T15:28:59.681 INFO:teuthology.orchestra.run.smithi033.stdout: "get_sum": 0, 2024-03-18T15:28:59.681 INFO:teuthology.orchestra.run.smithi033.stdout: "get_or_fail_fail": 0, 2024-03-18T15:28:59.681 INFO:teuthology.orchestra.run.smithi033.stdout: "get_or_fail_success": 0, 2024-03-18T15:28:59.681 INFO:teuthology.orchestra.run.smithi033.stdout: "take": 0, 2024-03-18T15:28:59.681 INFO:teuthology.orchestra.run.smithi033.stdout: "take_sum": 0, 2024-03-18T15:28:59.681 INFO:teuthology.orchestra.run.smithi033.stdout: "put": 0, 2024-03-18T15:28:59.681 INFO:teuthology.orchestra.run.smithi033.stdout: "put_sum": 0, 2024-03-18T15:28:59.682 INFO:teuthology.orchestra.run.smithi033.stdout: "wait": { 2024-03-18T15:28:59.682 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:28:59.682 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:28:59.682 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:28:59.682 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-18T15:28:59.682 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-18T15:28:59.682 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-03-18T15:28:59.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:28:59 smithi033 ceph-mon[36648]: pgmap v326: 129 pgs: 129 active+clean; 1.7 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:28:59.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:28:59 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:28:59.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:28:59 smithi033 ceph-mon[36648]: from='mgr.25466 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:28:59.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:28:59 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:28:59.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:28:59 smithi033 ceph-mon[36648]: from='mgr.25466 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:28:59.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:28:59 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:28:59.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:28:59 smithi033 ceph-mon[36648]: from='mgr.25466 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:28:59.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:28:59 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:28:59.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:28:59 smithi033 ceph-mon[36648]: from='mgr.25466 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:28:59.845 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:28:59 smithi104 ceph-mon[36070]: pgmap v326: 129 pgs: 129 active+clean; 1.7 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:28:59.845 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:28:59 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:28:59.846 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:28:59 smithi104 ceph-mon[36070]: from='mgr.25466 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:28:59.846 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:28:59 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:28:59.846 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:28:59 smithi104 ceph-mon[36070]: from='mgr.25466 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:28:59.846 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:28:59 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:28:59.846 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:28:59 smithi104 ceph-mon[36070]: from='mgr.25466 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:28:59.846 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:28:59 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:28:59.846 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:28:59 smithi104 ceph-mon[36070]: from='mgr.25466 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:28:59.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:28:59 smithi019 ceph-mon[31522]: pgmap v326: 129 pgs: 129 active+clean; 1.7 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:28:59.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:28:59 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:28:59.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:28:59 smithi019 ceph-mon[31522]: from='mgr.25466 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:28:59.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:28:59 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:28:59.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:28:59 smithi019 ceph-mon[31522]: from='mgr.25466 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:28:59.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:28:59 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:28:59.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:28:59 smithi019 ceph-mon[31522]: from='mgr.25466 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:28:59.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:28:59 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:28:59.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:28:59 smithi019 ceph-mon[31522]: from='mgr.25466 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:29:00.355 INFO:tasks.check_counter:Daemon mds.e mds.exported=0 2024-03-18T15:29:00.355 INFO:tasks.check_counter:Daemon mds.e mds.imported=0 2024-03-18T15:29:00.355 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update=0 2024-03-18T15:29:00.356 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update_receipt=0 2024-03-18T15:29:00.356 INFO:tasks.check_counter:Daemon mds.e mds.root_rsnaps=6 2024-03-18T15:29:00.356 INFO:tasks.check_counter:Daemon mds.e mds_server.req_mksnap_latency.avgcount=0 2024-03-18T15:29:00.356 INFO:tasks.check_counter:Daemon mds.e mds_server.req_rmsnap_latency.avgcount=0 2024-03-18T15:29:00.356 DEBUG:tasks.check_counter:Getting stats from h 2024-03-18T15:29:00.356 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 -- ceph daemon mds.h perf dump 2024-03-18T15:29:00.676 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T15:29:00.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:29:00 smithi033 ceph-mon[36648]: Adjusting osd_memory_target on smithi019 to 177.8M 2024-03-18T15:29:00.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:29:00 smithi033 ceph-mon[36648]: Unable to set osd_memory_target on smithi019 to 186497740: error parsing value: Value '186497740' is below minimum 939524096 2024-03-18T15:29:00.863 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:29:00 smithi104 ceph-mon[36070]: Adjusting osd_memory_target on smithi019 to 177.8M 2024-03-18T15:29:00.864 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:29:00 smithi104 ceph-mon[36070]: Unable to set osd_memory_target on smithi019 to 186497740: error parsing value: Value '186497740' is below minimum 939524096 2024-03-18T15:29:00.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:29:00 smithi019 ceph-mon[31522]: Adjusting osd_memory_target on smithi019 to 177.8M 2024-03-18T15:29:00.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:29:00 smithi019 ceph-mon[31522]: Unable to set osd_memory_target on smithi019 to 186497740: error parsing value: Value '186497740' is below minimum 939524096 2024-03-18T15:29:01.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:29:01 smithi104 ceph-mon[36070]: pgmap v327: 129 pgs: 129 active+clean; 1.7 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:29:01.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:29:01 smithi104 ceph-mon[36070]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:29:01.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:29:01 smithi104 ceph-mon[36070]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:29:01.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:29:01 smithi019 ceph-mon[31522]: pgmap v327: 129 pgs: 129 active+clean; 1.7 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:29:01.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:29:01 smithi019 ceph-mon[31522]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:29:01.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:29:01 smithi019 ceph-mon[31522]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:29:01.979 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:29:01 smithi033 ceph-mon[36648]: pgmap v327: 129 pgs: 129 active+clean; 1.7 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T15:29:01.979 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:29:01 smithi033 ceph-mon[36648]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:29:01.979 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:29:01 smithi033 ceph-mon[36648]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:29:02.503 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-03-18T15:29:02.503 INFO:teuthology.orchestra.run.smithi033.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T15:29:02.504 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_messages": 367, 2024-03-18T15:29:02.504 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_messages": 537, 2024-03-18T15:29:02.504 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_bytes": 152840, 2024-03-18T15:29:02.504 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_bytes": 301416, 2024-03-18T15:29:02.504 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_created_connections": 9, 2024-03-18T15:29:02.504 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_active_connections": 3, 2024-03-18T15:29:02.504 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_total_time": 0.041979630, 2024-03-18T15:29:02.504 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_send_time": 0.018166648, 2024-03-18T15:29:02.504 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_recv_time": 0.018239889, 2024-03-18T15:29:02.504 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_fast_dispatch_time": 0.003298378, 2024-03-18T15:29:02.504 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T15:29:02.504 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 539, 2024-03-18T15:29:02.505 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.032919948, 2024-03-18T15:29:02.505 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000061075 2024-03-18T15:29:02.505 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:02.505 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_handle_ack_lat": { 2024-03-18T15:29:02.505 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 829, 2024-03-18T15:29:02.505 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000156215, 2024-03-18T15:29:02.505 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000188 2024-03-18T15:29:02.505 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:02.505 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-18T15:29:02.505 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_encrypted_bytes": 0 2024-03-18T15:29:02.505 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:02.505 INFO:teuthology.orchestra.run.smithi033.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T15:29:02.505 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_messages": 313, 2024-03-18T15:29:02.505 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_messages": 420, 2024-03-18T15:29:02.506 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_bytes": 1796603, 2024-03-18T15:29:02.506 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_bytes": 470631, 2024-03-18T15:29:02.506 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_created_connections": 6, 2024-03-18T15:29:02.506 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_active_connections": 5, 2024-03-18T15:29:02.506 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_total_time": 0.047232509, 2024-03-18T15:29:02.506 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_send_time": 0.012925250, 2024-03-18T15:29:02.506 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_recv_time": 0.046262194, 2024-03-18T15:29:02.506 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_fast_dispatch_time": 0.004270633, 2024-03-18T15:29:02.506 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T15:29:02.506 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 419, 2024-03-18T15:29:02.506 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.041102750, 2024-03-18T15:29:02.507 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000098097 2024-03-18T15:29:02.507 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:02.507 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_handle_ack_lat": { 2024-03-18T15:29:02.507 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:29:02.507 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:29:02.507 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:29:02.507 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:02.507 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_encrypted_bytes": 1793312, 2024-03-18T15:29:02.507 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_encrypted_bytes": 464960 2024-03-18T15:29:02.507 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:02.507 INFO:teuthology.orchestra.run.smithi033.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T15:29:02.507 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_messages": 36, 2024-03-18T15:29:02.507 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_messages": 123, 2024-03-18T15:29:02.508 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_bytes": 11511, 2024-03-18T15:29:02.508 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_bytes": 291165, 2024-03-18T15:29:02.508 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_created_connections": 15, 2024-03-18T15:29:02.508 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_active_connections": 4, 2024-03-18T15:29:02.508 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_total_time": 0.012574780, 2024-03-18T15:29:02.508 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_send_time": 0.003688175, 2024-03-18T15:29:02.508 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_recv_time": 0.007634619, 2024-03-18T15:29:02.508 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_fast_dispatch_time": 0.000119777, 2024-03-18T15:29:02.508 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T15:29:02.508 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 112, 2024-03-18T15:29:02.508 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.025085332, 2024-03-18T15:29:02.508 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000223976 2024-03-18T15:29:02.508 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:02.508 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_handle_ack_lat": { 2024-03-18T15:29:02.508 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 62, 2024-03-18T15:29:02.509 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000012529, 2024-03-18T15:29:02.509 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000202 2024-03-18T15:29:02.509 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:02.509 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_encrypted_bytes": 6176, 2024-03-18T15:29:02.509 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_encrypted_bytes": 227392 2024-03-18T15:29:02.509 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:02.509 INFO:teuthology.orchestra.run.smithi033.stdout: "cct": { 2024-03-18T15:29:02.509 INFO:teuthology.orchestra.run.smithi033.stdout: "total_workers": 1, 2024-03-18T15:29:02.509 INFO:teuthology.orchestra.run.smithi033.stdout: "unhealthy_workers": 0 2024-03-18T15:29:02.509 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:02.509 INFO:teuthology.orchestra.run.smithi033.stdout: "finisher-MDSRank": { 2024-03-18T15:29:02.509 INFO:teuthology.orchestra.run.smithi033.stdout: "queue_len": 0, 2024-03-18T15:29:02.509 INFO:teuthology.orchestra.run.smithi033.stdout: "complete_latency": { 2024-03-18T15:29:02.510 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 64, 2024-03-18T15:29:02.510 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.001690104, 2024-03-18T15:29:02.510 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000026407 2024-03-18T15:29:02.510 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-18T15:29:02.510 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:02.511 INFO:teuthology.orchestra.run.smithi033.stdout: "finisher-PurgeQueue": { 2024-03-18T15:29:02.511 INFO:teuthology.orchestra.run.smithi033.stdout: "queue_len": 0, 2024-03-18T15:29:02.511 INFO:teuthology.orchestra.run.smithi033.stdout: "complete_latency": { 2024-03-18T15:29:02.511 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 2, 2024-03-18T15:29:02.511 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000005678, 2024-03-18T15:29:02.511 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000002839 2024-03-18T15:29:02.511 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-18T15:29:02.512 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:02.512 INFO:teuthology.orchestra.run.smithi033.stdout: "mds": { 2024-03-18T15:29:02.512 INFO:teuthology.orchestra.run.smithi033.stdout: "request": 5, 2024-03-18T15:29:02.512 INFO:teuthology.orchestra.run.smithi033.stdout: "reply": 4, 2024-03-18T15:29:02.512 INFO:teuthology.orchestra.run.smithi033.stdout: "reply_latency": { 2024-03-18T15:29:02.512 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 4, 2024-03-18T15:29:02.512 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.001376391, 2024-03-18T15:29:02.512 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000344097 2024-03-18T15:29:02.512 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:02.512 INFO:teuthology.orchestra.run.smithi033.stdout: "slow_reply": 0, 2024-03-18T15:29:02.512 INFO:teuthology.orchestra.run.smithi033.stdout: "forward": 1, 2024-03-18T15:29:02.512 INFO:teuthology.orchestra.run.smithi033.stdout: "dir_fetch_complete": 0, 2024-03-18T15:29:02.512 INFO:teuthology.orchestra.run.smithi033.stdout: "dir_fetch_keys": 0, 2024-03-18T15:29:02.512 INFO:teuthology.orchestra.run.smithi033.stdout: "dir_commit": 11, 2024-03-18T15:29:02.513 INFO:teuthology.orchestra.run.smithi033.stdout: "dir_split": 0, 2024-03-18T15:29:02.513 INFO:teuthology.orchestra.run.smithi033.stdout: "dir_merge": 0, 2024-03-18T15:29:02.513 INFO:teuthology.orchestra.run.smithi033.stdout: "inodes": 16, 2024-03-18T15:29:02.513 INFO:teuthology.orchestra.run.smithi033.stdout: "inodes_top": 0, 2024-03-18T15:29:02.513 INFO:teuthology.orchestra.run.smithi033.stdout: "inodes_bottom": 16, 2024-03-18T15:29:02.513 INFO:teuthology.orchestra.run.smithi033.stdout: "inodes_pin_tail": 0, 2024-03-18T15:29:02.513 INFO:teuthology.orchestra.run.smithi033.stdout: "inodes_pinned": 13, 2024-03-18T15:29:02.513 INFO:teuthology.orchestra.run.smithi033.stdout: "inodes_expired": 5, 2024-03-18T15:29:02.513 INFO:teuthology.orchestra.run.smithi033.stdout: "inodes_with_caps": 1, 2024-03-18T15:29:02.513 INFO:teuthology.orchestra.run.smithi033.stdout: "caps": 1, 2024-03-18T15:29:02.513 INFO:teuthology.orchestra.run.smithi033.stdout: "subtrees": 4, 2024-03-18T15:29:02.513 INFO:teuthology.orchestra.run.smithi033.stdout: "traverse": 15, 2024-03-18T15:29:02.514 INFO:teuthology.orchestra.run.smithi033.stdout: "traverse_hit": 7, 2024-03-18T15:29:02.514 INFO:teuthology.orchestra.run.smithi033.stdout: "traverse_forward": 1, 2024-03-18T15:29:02.514 INFO:teuthology.orchestra.run.smithi033.stdout: "traverse_discover": 5, 2024-03-18T15:29:02.514 INFO:teuthology.orchestra.run.smithi033.stdout: "traverse_dir_fetch": 0, 2024-03-18T15:29:02.514 INFO:teuthology.orchestra.run.smithi033.stdout: "traverse_remote_ino": 0, 2024-03-18T15:29:02.514 INFO:teuthology.orchestra.run.smithi033.stdout: "traverse_lock": 0, 2024-03-18T15:29:02.514 INFO:teuthology.orchestra.run.smithi033.stdout: "load_cent": 0, 2024-03-18T15:29:02.514 INFO:teuthology.orchestra.run.smithi033.stdout: "q": 0, 2024-03-18T15:29:02.514 INFO:teuthology.orchestra.run.smithi033.stdout: "exported": 0, 2024-03-18T15:29:02.514 INFO:teuthology.orchestra.run.smithi033.stdout: "exported_inodes": 0, 2024-03-18T15:29:02.514 INFO:teuthology.orchestra.run.smithi033.stdout: "imported": 0, 2024-03-18T15:29:02.515 INFO:teuthology.orchestra.run.smithi033.stdout: "imported_inodes": 0, 2024-03-18T15:29:02.515 INFO:teuthology.orchestra.run.smithi033.stdout: "openino_dir_fetch": 0, 2024-03-18T15:29:02.515 INFO:teuthology.orchestra.run.smithi033.stdout: "openino_backtrace_fetch": 0, 2024-03-18T15:29:02.515 INFO:teuthology.orchestra.run.smithi033.stdout: "openino_peer_discover": 0, 2024-03-18T15:29:02.515 INFO:teuthology.orchestra.run.smithi033.stdout: "root_rfiles": 4, 2024-03-18T15:29:02.515 INFO:teuthology.orchestra.run.smithi033.stdout: "root_rbytes": 234, 2024-03-18T15:29:02.515 INFO:teuthology.orchestra.run.smithi033.stdout: "root_rsnaps": 0, 2024-03-18T15:29:02.515 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_backtrace_fetch": 0, 2024-03-18T15:29:02.515 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_set_tag": 0, 2024-03-18T15:29:02.515 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_backtrace_repaired": 0, 2024-03-18T15:29:02.515 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_inotable_repaired": 0, 2024-03-18T15:29:02.515 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_dir_inodes": 0, 2024-03-18T15:29:02.516 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_dir_base_inodes": 0, 2024-03-18T15:29:02.516 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_dirfrag_rstats": 0, 2024-03-18T15:29:02.516 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_file_inodes": 0, 2024-03-18T15:29:02.516 INFO:teuthology.orchestra.run.smithi033.stdout: "handle_inode_file_caps": 0, 2024-03-18T15:29:02.516 INFO:teuthology.orchestra.run.smithi033.stdout: "ceph_cap_op_revoke": 0, 2024-03-18T15:29:02.517 INFO:teuthology.orchestra.run.smithi033.stdout: "ceph_cap_op_grant": 0, 2024-03-18T15:29:02.517 INFO:teuthology.orchestra.run.smithi033.stdout: "ceph_cap_op_trunc": 0, 2024-03-18T15:29:02.517 INFO:teuthology.orchestra.run.smithi033.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-18T15:29:02.517 INFO:teuthology.orchestra.run.smithi033.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-18T15:29:02.517 INFO:teuthology.orchestra.run.smithi033.stdout: "handle_client_caps": 0, 2024-03-18T15:29:02.517 INFO:teuthology.orchestra.run.smithi033.stdout: "handle_client_caps_dirty": 0, 2024-03-18T15:29:02.517 INFO:teuthology.orchestra.run.smithi033.stdout: "handle_client_cap_release": 0, 2024-03-18T15:29:02.517 INFO:teuthology.orchestra.run.smithi033.stdout: "process_request_cap_release": 0 2024-03-18T15:29:02.518 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:02.518 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_cache": { 2024-03-18T15:29:02.518 INFO:teuthology.orchestra.run.smithi033.stdout: "dir_update": 0, 2024-03-18T15:29:02.518 INFO:teuthology.orchestra.run.smithi033.stdout: "dir_update_receipt": 8, 2024-03-18T15:29:02.518 INFO:teuthology.orchestra.run.smithi033.stdout: "dir_try_discover": 6, 2024-03-18T15:29:02.518 INFO:teuthology.orchestra.run.smithi033.stdout: "dir_send_discover": 9, 2024-03-18T15:29:02.518 INFO:teuthology.orchestra.run.smithi033.stdout: "dir_handle_discover": 0, 2024-03-18T15:29:02.518 INFO:teuthology.orchestra.run.smithi033.stdout: "num_strays": 0, 2024-03-18T15:29:02.519 INFO:teuthology.orchestra.run.smithi033.stdout: "num_strays_delayed": 0, 2024-03-18T15:29:02.519 INFO:teuthology.orchestra.run.smithi033.stdout: "num_strays_enqueuing": 0, 2024-03-18T15:29:02.519 INFO:teuthology.orchestra.run.smithi033.stdout: "strays_created": 0, 2024-03-18T15:29:02.519 INFO:teuthology.orchestra.run.smithi033.stdout: "strays_enqueued": 0, 2024-03-18T15:29:02.519 INFO:teuthology.orchestra.run.smithi033.stdout: "strays_reintegrated": 0, 2024-03-18T15:29:02.519 INFO:teuthology.orchestra.run.smithi033.stdout: "strays_migrated": 0, 2024-03-18T15:29:02.519 INFO:teuthology.orchestra.run.smithi033.stdout: "num_recovering_processing": 0, 2024-03-18T15:29:02.519 INFO:teuthology.orchestra.run.smithi033.stdout: "num_recovering_enqueued": 0, 2024-03-18T15:29:02.519 INFO:teuthology.orchestra.run.smithi033.stdout: "num_recovering_prioritized": 0, 2024-03-18T15:29:02.519 INFO:teuthology.orchestra.run.smithi033.stdout: "recovery_started": 0, 2024-03-18T15:29:02.519 INFO:teuthology.orchestra.run.smithi033.stdout: "recovery_completed": 0, 2024-03-18T15:29:02.519 INFO:teuthology.orchestra.run.smithi033.stdout: "ireq_quiesce_path": 0, 2024-03-18T15:29:02.519 INFO:teuthology.orchestra.run.smithi033.stdout: "ireq_quiesce_inode": 0, 2024-03-18T15:29:02.520 INFO:teuthology.orchestra.run.smithi033.stdout: "ireq_enqueue_scrub": 0, 2024-03-18T15:29:02.520 INFO:teuthology.orchestra.run.smithi033.stdout: "ireq_exportdir": 0, 2024-03-18T15:29:02.520 INFO:teuthology.orchestra.run.smithi033.stdout: "ireq_flush": 0, 2024-03-18T15:29:02.520 INFO:teuthology.orchestra.run.smithi033.stdout: "ireq_fragmentdir": 0, 2024-03-18T15:29:02.520 INFO:teuthology.orchestra.run.smithi033.stdout: "ireq_fragstats": 0, 2024-03-18T15:29:02.520 INFO:teuthology.orchestra.run.smithi033.stdout: "ireq_inodestats": 0 2024-03-18T15:29:02.520 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:02.520 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_log": { 2024-03-18T15:29:02.520 INFO:teuthology.orchestra.run.smithi033.stdout: "evlrg": 0, 2024-03-18T15:29:02.520 INFO:teuthology.orchestra.run.smithi033.stdout: "evadd": 7, 2024-03-18T15:29:02.520 INFO:teuthology.orchestra.run.smithi033.stdout: "evex": 0, 2024-03-18T15:29:02.520 INFO:teuthology.orchestra.run.smithi033.stdout: "evtrm": 0, 2024-03-18T15:29:02.521 INFO:teuthology.orchestra.run.smithi033.stdout: "ev": 7, 2024-03-18T15:29:02.521 INFO:teuthology.orchestra.run.smithi033.stdout: "evexg": 0, 2024-03-18T15:29:02.521 INFO:teuthology.orchestra.run.smithi033.stdout: "evexd": 0, 2024-03-18T15:29:02.521 INFO:teuthology.orchestra.run.smithi033.stdout: "segadd": 2, 2024-03-18T15:29:02.521 INFO:teuthology.orchestra.run.smithi033.stdout: "segex": 0, 2024-03-18T15:29:02.521 INFO:teuthology.orchestra.run.smithi033.stdout: "segtrm": 0, 2024-03-18T15:29:02.521 INFO:teuthology.orchestra.run.smithi033.stdout: "seg": 2, 2024-03-18T15:29:02.521 INFO:teuthology.orchestra.run.smithi033.stdout: "segmjr": 2, 2024-03-18T15:29:02.521 INFO:teuthology.orchestra.run.smithi033.stdout: "segexg": 0, 2024-03-18T15:29:02.521 INFO:teuthology.orchestra.run.smithi033.stdout: "segexd": 0, 2024-03-18T15:29:02.521 INFO:teuthology.orchestra.run.smithi033.stdout: "expos": 4194304, 2024-03-18T15:29:02.521 INFO:teuthology.orchestra.run.smithi033.stdout: "wrpos": 4196266, 2024-03-18T15:29:02.521 INFO:teuthology.orchestra.run.smithi033.stdout: "rdpos": 0, 2024-03-18T15:29:02.522 INFO:teuthology.orchestra.run.smithi033.stdout: "jlat": { 2024-03-18T15:29:02.522 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 6, 2024-03-18T15:29:02.522 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.020937984, 2024-03-18T15:29:02.522 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.003489664 2024-03-18T15:29:02.522 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:02.522 INFO:teuthology.orchestra.run.smithi033.stdout: "replayed": 0 2024-03-18T15:29:02.522 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:02.522 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_mem": { 2024-03-18T15:29:02.523 INFO:teuthology.orchestra.run.smithi033.stdout: "ino": 21, 2024-03-18T15:29:02.523 INFO:teuthology.orchestra.run.smithi033.stdout: "ino+": 23, 2024-03-18T15:29:02.523 INFO:teuthology.orchestra.run.smithi033.stdout: "ino-": 2, 2024-03-18T15:29:02.523 INFO:teuthology.orchestra.run.smithi033.stdout: "dir": 20, 2024-03-18T15:29:02.523 INFO:teuthology.orchestra.run.smithi033.stdout: "dir+": 23, 2024-03-18T15:29:02.523 INFO:teuthology.orchestra.run.smithi033.stdout: "dir-": 3, 2024-03-18T15:29:02.523 INFO:teuthology.orchestra.run.smithi033.stdout: "dn": 16, 2024-03-18T15:29:02.523 INFO:teuthology.orchestra.run.smithi033.stdout: "dn+": 21, 2024-03-18T15:29:02.523 INFO:teuthology.orchestra.run.smithi033.stdout: "dn-": 5, 2024-03-18T15:29:02.523 INFO:teuthology.orchestra.run.smithi033.stdout: "cap": 1, 2024-03-18T15:29:02.523 INFO:teuthology.orchestra.run.smithi033.stdout: "cap+": 3, 2024-03-18T15:29:02.523 INFO:teuthology.orchestra.run.smithi033.stdout: "cap-": 2, 2024-03-18T15:29:02.524 INFO:teuthology.orchestra.run.smithi033.stdout: "rss": 49240, 2024-03-18T15:29:02.524 INFO:teuthology.orchestra.run.smithi033.stdout: "heap": 223612 2024-03-18T15:29:02.524 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:02.524 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_server": { 2024-03-18T15:29:02.524 INFO:teuthology.orchestra.run.smithi033.stdout: "dispatch_client_request": 6, 2024-03-18T15:29:02.524 INFO:teuthology.orchestra.run.smithi033.stdout: "dispatch_server_request": 1, 2024-03-18T15:29:02.524 INFO:teuthology.orchestra.run.smithi033.stdout: "handle_client_request": 5, 2024-03-18T15:29:02.524 INFO:teuthology.orchestra.run.smithi033.stdout: "handle_client_session": 16, 2024-03-18T15:29:02.524 INFO:teuthology.orchestra.run.smithi033.stdout: "handle_peer_request": 2, 2024-03-18T15:29:02.524 INFO:teuthology.orchestra.run.smithi033.stdout: "req_create_latency": { 2024-03-18T15:29:02.524 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:29:02.524 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:29:02.525 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:29:02.525 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:02.525 INFO:teuthology.orchestra.run.smithi033.stdout: "req_getattr_latency": { 2024-03-18T15:29:02.525 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 1, 2024-03-18T15:29:02.525 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000408682, 2024-03-18T15:29:02.525 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000408682 2024-03-18T15:29:02.525 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:02.525 INFO:teuthology.orchestra.run.smithi033.stdout: "req_getfilelock_latency": { 2024-03-18T15:29:02.525 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:29:02.525 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:29:02.525 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:29:02.525 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:02.525 INFO:teuthology.orchestra.run.smithi033.stdout: "req_link_latency": { 2024-03-18T15:29:02.526 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:29:02.526 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:29:02.526 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:29:02.526 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:02.526 INFO:teuthology.orchestra.run.smithi033.stdout: "req_lookup_latency": { 2024-03-18T15:29:02.526 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 2, 2024-03-18T15:29:02.526 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000861354, 2024-03-18T15:29:02.526 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000430677 2024-03-18T15:29:02.526 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:02.526 INFO:teuthology.orchestra.run.smithi033.stdout: "req_lookuphash_latency": { 2024-03-18T15:29:02.527 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:29:02.527 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:29:02.527 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:29:02.527 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:02.527 INFO:teuthology.orchestra.run.smithi033.stdout: "req_lookupino_latency": { 2024-03-18T15:29:02.527 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 1, 2024-03-18T15:29:02.528 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000106355, 2024-03-18T15:29:02.528 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000106355 2024-03-18T15:29:02.528 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:02.528 INFO:teuthology.orchestra.run.smithi033.stdout: "req_lookupname_latency": { 2024-03-18T15:29:02.529 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:29:02.529 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:29:02.529 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:29:02.529 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:02.529 INFO:teuthology.orchestra.run.smithi033.stdout: "req_lookupparent_latency": { 2024-03-18T15:29:02.529 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:29:02.529 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:29:02.529 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:29:02.529 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:02.529 INFO:teuthology.orchestra.run.smithi033.stdout: "req_lookupsnap_latency": { 2024-03-18T15:29:02.529 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:29:02.529 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:29:02.529 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:29:02.530 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:02.530 INFO:teuthology.orchestra.run.smithi033.stdout: "req_lssnap_latency": { 2024-03-18T15:29:02.530 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:29:02.530 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:29:02.530 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:29:02.531 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:02.531 INFO:teuthology.orchestra.run.smithi033.stdout: "req_mkdir_latency": { 2024-03-18T15:29:02.531 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:29:02.531 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:29:02.531 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:29:02.531 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:02.531 INFO:teuthology.orchestra.run.smithi033.stdout: "req_mknod_latency": { 2024-03-18T15:29:02.531 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:29:02.531 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:29:02.531 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:29:02.531 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:02.532 INFO:teuthology.orchestra.run.smithi033.stdout: "req_mksnap_latency": { 2024-03-18T15:29:02.532 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:29:02.532 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:29:02.532 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:29:02.532 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:02.532 INFO:teuthology.orchestra.run.smithi033.stdout: "req_open_latency": { 2024-03-18T15:29:02.532 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:29:02.532 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:29:02.532 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:29:02.532 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:02.532 INFO:teuthology.orchestra.run.smithi033.stdout: "req_readdir_latency": { 2024-03-18T15:29:02.532 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:29:02.532 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:29:02.532 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:29:02.532 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:02.533 INFO:teuthology.orchestra.run.smithi033.stdout: "req_rename_latency": { 2024-03-18T15:29:02.533 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:29:02.533 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:29:02.533 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:29:02.533 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:02.533 INFO:teuthology.orchestra.run.smithi033.stdout: "req_renamesnap_latency": { 2024-03-18T15:29:02.533 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:29:02.533 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:29:02.533 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:29:02.533 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:02.533 INFO:teuthology.orchestra.run.smithi033.stdout: "req_snapdiff_latency": { 2024-03-18T15:29:02.533 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:29:02.533 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:29:02.533 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:29:02.534 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:02.534 INFO:teuthology.orchestra.run.smithi033.stdout: "req_rmdir_latency": { 2024-03-18T15:29:02.534 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:29:02.534 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:29:02.534 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:29:02.534 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:02.534 INFO:teuthology.orchestra.run.smithi033.stdout: "req_rmsnap_latency": { 2024-03-18T15:29:02.534 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:29:02.534 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:29:02.534 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:29:02.534 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:02.534 INFO:teuthology.orchestra.run.smithi033.stdout: "req_rmxattr_latency": { 2024-03-18T15:29:02.534 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:29:02.534 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:29:02.535 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:29:02.535 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:02.535 INFO:teuthology.orchestra.run.smithi033.stdout: "req_setattr_latency": { 2024-03-18T15:29:02.535 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:29:02.535 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:29:02.535 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:29:02.535 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:02.535 INFO:teuthology.orchestra.run.smithi033.stdout: "req_setdirlayout_latency": { 2024-03-18T15:29:02.535 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:29:02.535 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:29:02.535 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:29:02.535 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:02.535 INFO:teuthology.orchestra.run.smithi033.stdout: "req_setfilelock_latency": { 2024-03-18T15:29:02.535 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:29:02.536 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:29:02.536 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:29:02.536 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:02.536 INFO:teuthology.orchestra.run.smithi033.stdout: "req_setlayout_latency": { 2024-03-18T15:29:02.536 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:29:02.536 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:29:02.536 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:29:02.536 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:02.536 INFO:teuthology.orchestra.run.smithi033.stdout: "req_setxattr_latency": { 2024-03-18T15:29:02.536 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:29:02.536 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:29:02.536 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:29:02.536 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:02.536 INFO:teuthology.orchestra.run.smithi033.stdout: "req_symlink_latency": { 2024-03-18T15:29:02.536 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:29:02.537 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:29:02.537 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:29:02.537 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:02.537 INFO:teuthology.orchestra.run.smithi033.stdout: "req_unlink_latency": { 2024-03-18T15:29:02.537 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:29:02.537 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:29:02.537 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:29:02.537 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:02.537 INFO:teuthology.orchestra.run.smithi033.stdout: "cap_revoke_eviction": 0, 2024-03-18T15:29:02.537 INFO:teuthology.orchestra.run.smithi033.stdout: "cap_acquisition_throttle": 0, 2024-03-18T15:29:02.537 INFO:teuthology.orchestra.run.smithi033.stdout: "req_getvxattr_latency": { 2024-03-18T15:29:02.537 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:29:02.537 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:29:02.537 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:29:02.537 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-18T15:29:02.538 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:02.538 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_sessions": { 2024-03-18T15:29:02.538 INFO:teuthology.orchestra.run.smithi033.stdout: "session_count": 1, 2024-03-18T15:29:02.538 INFO:teuthology.orchestra.run.smithi033.stdout: "session_add": 3, 2024-03-18T15:29:02.538 INFO:teuthology.orchestra.run.smithi033.stdout: "session_remove": 2, 2024-03-18T15:29:02.538 INFO:teuthology.orchestra.run.smithi033.stdout: "sessions_open": 1, 2024-03-18T15:29:02.538 INFO:teuthology.orchestra.run.smithi033.stdout: "sessions_stale": 0, 2024-03-18T15:29:02.538 INFO:teuthology.orchestra.run.smithi033.stdout: "total_load": 3, 2024-03-18T15:29:02.538 INFO:teuthology.orchestra.run.smithi033.stdout: "average_load": 1, 2024-03-18T15:29:02.538 INFO:teuthology.orchestra.run.smithi033.stdout: "avg_session_uptime": 627, 2024-03-18T15:29:02.538 INFO:teuthology.orchestra.run.smithi033.stdout: "mdthresh_evicted": 0 2024-03-18T15:29:02.538 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:02.538 INFO:teuthology.orchestra.run.smithi033.stdout: "mempool": { 2024-03-18T15:29:02.538 INFO:teuthology.orchestra.run.smithi033.stdout: "bloom_filter_bytes": 0, 2024-03-18T15:29:02.539 INFO:teuthology.orchestra.run.smithi033.stdout: "bloom_filter_items": 0, 2024-03-18T15:29:02.539 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T15:29:02.539 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_alloc_items": 0, 2024-03-18T15:29:02.539 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T15:29:02.539 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_data_items": 0, 2024-03-18T15:29:02.539 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T15:29:02.539 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T15:29:02.539 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T15:29:02.539 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T15:29:02.539 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T15:29:02.539 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_other_items": 0, 2024-03-18T15:29:02.539 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T15:29:02.539 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T15:29:02.539 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_extent_bytes": 0, 2024-03-18T15:29:02.540 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_extent_items": 0, 2024-03-18T15:29:02.540 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_blob_bytes": 0, 2024-03-18T15:29:02.540 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_blob_items": 0, 2024-03-18T15:29:02.540 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T15:29:02.540 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T15:29:02.540 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T15:29:02.540 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T15:29:02.540 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T15:29:02.540 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_fsck_items": 0, 2024-03-18T15:29:02.540 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_txc_bytes": 0, 2024-03-18T15:29:02.540 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_txc_items": 0, 2024-03-18T15:29:02.540 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T15:29:02.540 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T15:29:02.540 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_writing_bytes": 0, 2024-03-18T15:29:02.541 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_writing_items": 0, 2024-03-18T15:29:02.541 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_bytes": 0, 2024-03-18T15:29:02.541 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_items": 0, 2024-03-18T15:29:02.541 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T15:29:02.542 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_file_reader_items": 0, 2024-03-18T15:29:02.542 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T15:29:02.542 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_file_writer_items": 0, 2024-03-18T15:29:02.542 INFO:teuthology.orchestra.run.smithi033.stdout: "buffer_anon_bytes": 263895, 2024-03-18T15:29:02.542 INFO:teuthology.orchestra.run.smithi033.stdout: "buffer_anon_items": 87, 2024-03-18T15:29:02.542 INFO:teuthology.orchestra.run.smithi033.stdout: "buffer_meta_bytes": 0, 2024-03-18T15:29:02.542 INFO:teuthology.orchestra.run.smithi033.stdout: "buffer_meta_items": 0, 2024-03-18T15:29:02.542 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_bytes": 0, 2024-03-18T15:29:02.542 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_items": 0, 2024-03-18T15:29:02.542 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mapbl_bytes": 0, 2024-03-18T15:29:02.542 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mapbl_items": 0, 2024-03-18T15:29:02.542 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pglog_bytes": 0, 2024-03-18T15:29:02.542 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pglog_items": 0, 2024-03-18T15:29:02.542 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_bytes": 9928, 2024-03-18T15:29:02.542 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_items": 207, 2024-03-18T15:29:02.543 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T15:29:02.543 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_mapping_items": 0, 2024-03-18T15:29:02.543 INFO:teuthology.orchestra.run.smithi033.stdout: "pgmap_bytes": 0, 2024-03-18T15:29:02.543 INFO:teuthology.orchestra.run.smithi033.stdout: "pgmap_items": 0, 2024-03-18T15:29:02.543 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_co_bytes": 86165, 2024-03-18T15:29:02.543 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_co_items": 342, 2024-03-18T15:29:02.543 INFO:teuthology.orchestra.run.smithi033.stdout: "unittest_1_bytes": 0, 2024-03-18T15:29:02.543 INFO:teuthology.orchestra.run.smithi033.stdout: "unittest_1_items": 0, 2024-03-18T15:29:02.543 INFO:teuthology.orchestra.run.smithi033.stdout: "unittest_2_bytes": 0, 2024-03-18T15:29:02.543 INFO:teuthology.orchestra.run.smithi033.stdout: "unittest_2_items": 0 2024-03-18T15:29:02.543 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:02.543 INFO:teuthology.orchestra.run.smithi033.stdout: "objecter": { 2024-03-18T15:29:02.544 INFO:teuthology.orchestra.run.smithi033.stdout: "op_active": 0, 2024-03-18T15:29:02.544 INFO:teuthology.orchestra.run.smithi033.stdout: "op_laggy": 0, 2024-03-18T15:29:02.544 INFO:teuthology.orchestra.run.smithi033.stdout: "op_send": 46, 2024-03-18T15:29:02.544 INFO:teuthology.orchestra.run.smithi033.stdout: "op_send_bytes": 12705, 2024-03-18T15:29:02.544 INFO:teuthology.orchestra.run.smithi033.stdout: "op_resend": 33, 2024-03-18T15:29:02.544 INFO:teuthology.orchestra.run.smithi033.stdout: "op_reply": 46, 2024-03-18T15:29:02.544 INFO:teuthology.orchestra.run.smithi033.stdout: "op_latency": { 2024-03-18T15:29:02.544 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 46, 2024-03-18T15:29:02.544 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.363993756, 2024-03-18T15:29:02.544 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.007912907 2024-03-18T15:29:02.545 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:02.545 INFO:teuthology.orchestra.run.smithi033.stdout: "op_inflight": 0, 2024-03-18T15:29:02.545 INFO:teuthology.orchestra.run.smithi033.stdout: "oplen_avg": { 2024-03-18T15:29:02.545 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 46, 2024-03-18T15:29:02.545 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 71 2024-03-18T15:29:02.545 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:02.545 INFO:teuthology.orchestra.run.smithi033.stdout: "op": 46, 2024-03-18T15:29:02.545 INFO:teuthology.orchestra.run.smithi033.stdout: "op_r": 0, 2024-03-18T15:29:02.545 INFO:teuthology.orchestra.run.smithi033.stdout: "op_w": 46, 2024-03-18T15:29:02.545 INFO:teuthology.orchestra.run.smithi033.stdout: "op_rmw": 0, 2024-03-18T15:29:02.545 INFO:teuthology.orchestra.run.smithi033.stdout: "op_pg": 0, 2024-03-18T15:29:02.545 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_stat": 0, 2024-03-18T15:29:02.545 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_create": 10, 2024-03-18T15:29:02.545 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_read": 0, 2024-03-18T15:29:02.545 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_write": 6, 2024-03-18T15:29:02.546 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_writefull": 8, 2024-03-18T15:29:02.546 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_writesame": 0, 2024-03-18T15:29:02.546 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_append": 0, 2024-03-18T15:29:02.546 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_zero": 0, 2024-03-18T15:29:02.546 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_truncate": 0, 2024-03-18T15:29:02.546 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_delete": 6, 2024-03-18T15:29:02.546 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_mapext": 0, 2024-03-18T15:29:02.546 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_sparse_read": 0, 2024-03-18T15:29:02.546 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_clonerange": 0, 2024-03-18T15:29:02.546 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_getxattr": 0, 2024-03-18T15:29:02.546 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_setxattr": 20, 2024-03-18T15:29:02.546 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_cmpxattr": 0, 2024-03-18T15:29:02.546 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_rmxattr": 0, 2024-03-18T15:29:02.546 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_resetxattrs": 0, 2024-03-18T15:29:02.546 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_call": 0, 2024-03-18T15:29:02.547 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_watch": 0, 2024-03-18T15:29:02.547 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_notify": 0, 2024-03-18T15:29:02.547 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_src_cmpxattr": 0, 2024-03-18T15:29:02.547 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_pgls": 0, 2024-03-18T15:29:02.547 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_pgls_filter": 0, 2024-03-18T15:29:02.547 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_other": 0, 2024-03-18T15:29:02.547 INFO:teuthology.orchestra.run.smithi033.stdout: "linger_active": 0, 2024-03-18T15:29:02.547 INFO:teuthology.orchestra.run.smithi033.stdout: "linger_send": 0, 2024-03-18T15:29:02.547 INFO:teuthology.orchestra.run.smithi033.stdout: "linger_resend": 0, 2024-03-18T15:29:02.547 INFO:teuthology.orchestra.run.smithi033.stdout: "linger_ping": 0, 2024-03-18T15:29:02.547 INFO:teuthology.orchestra.run.smithi033.stdout: "poolop_active": 0, 2024-03-18T15:29:02.547 INFO:teuthology.orchestra.run.smithi033.stdout: "poolop_send": 0, 2024-03-18T15:29:02.547 INFO:teuthology.orchestra.run.smithi033.stdout: "poolop_resend": 0, 2024-03-18T15:29:02.547 INFO:teuthology.orchestra.run.smithi033.stdout: "poolstat_active": 0, 2024-03-18T15:29:02.547 INFO:teuthology.orchestra.run.smithi033.stdout: "poolstat_send": 0, 2024-03-18T15:29:02.548 INFO:teuthology.orchestra.run.smithi033.stdout: "poolstat_resend": 0, 2024-03-18T15:29:02.548 INFO:teuthology.orchestra.run.smithi033.stdout: "statfs_active": 0, 2024-03-18T15:29:02.548 INFO:teuthology.orchestra.run.smithi033.stdout: "statfs_send": 0, 2024-03-18T15:29:02.548 INFO:teuthology.orchestra.run.smithi033.stdout: "statfs_resend": 0, 2024-03-18T15:29:02.548 INFO:teuthology.orchestra.run.smithi033.stdout: "command_active": 0, 2024-03-18T15:29:02.548 INFO:teuthology.orchestra.run.smithi033.stdout: "command_send": 0, 2024-03-18T15:29:02.548 INFO:teuthology.orchestra.run.smithi033.stdout: "command_resend": 0, 2024-03-18T15:29:02.548 INFO:teuthology.orchestra.run.smithi033.stdout: "map_epoch": 103, 2024-03-18T15:29:02.548 INFO:teuthology.orchestra.run.smithi033.stdout: "map_full": 0, 2024-03-18T15:29:02.548 INFO:teuthology.orchestra.run.smithi033.stdout: "map_inc": 14, 2024-03-18T15:29:02.548 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_sessions": 8, 2024-03-18T15:29:02.548 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_session_open": 8, 2024-03-18T15:29:02.548 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_session_close": 0, 2024-03-18T15:29:02.548 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_laggy": 0, 2024-03-18T15:29:02.548 INFO:teuthology.orchestra.run.smithi033.stdout: "omap_wr": 20, 2024-03-18T15:29:02.549 INFO:teuthology.orchestra.run.smithi033.stdout: "omap_rd": 0, 2024-03-18T15:29:02.549 INFO:teuthology.orchestra.run.smithi033.stdout: "omap_del": 1 2024-03-18T15:29:02.549 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:02.549 INFO:teuthology.orchestra.run.smithi033.stdout: "oft": { 2024-03-18T15:29:02.549 INFO:teuthology.orchestra.run.smithi033.stdout: "omap_total_objs": 1, 2024-03-18T15:29:02.549 INFO:teuthology.orchestra.run.smithi033.stdout: "omap_total_kv_pairs": 0, 2024-03-18T15:29:02.549 INFO:teuthology.orchestra.run.smithi033.stdout: "omap_total_updates": 0, 2024-03-18T15:29:02.549 INFO:teuthology.orchestra.run.smithi033.stdout: "omap_total_removes": 0 2024-03-18T15:29:02.549 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:02.549 INFO:teuthology.orchestra.run.smithi033.stdout: "osd-slow-ops": { 2024-03-18T15:29:02.549 INFO:teuthology.orchestra.run.smithi033.stdout: "slow_ops_count": 0 2024-03-18T15:29:02.549 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:02.549 INFO:teuthology.orchestra.run.smithi033.stdout: "purge_queue": { 2024-03-18T15:29:02.549 INFO:teuthology.orchestra.run.smithi033.stdout: "pq_executing_ops": 0, 2024-03-18T15:29:02.549 INFO:teuthology.orchestra.run.smithi033.stdout: "pq_executing_ops_high_water": 0, 2024-03-18T15:29:02.550 INFO:teuthology.orchestra.run.smithi033.stdout: "pq_executing": 0, 2024-03-18T15:29:02.550 INFO:teuthology.orchestra.run.smithi033.stdout: "pq_executing_high_water": 0, 2024-03-18T15:29:02.550 INFO:teuthology.orchestra.run.smithi033.stdout: "pq_executed_ops": 0, 2024-03-18T15:29:02.550 INFO:teuthology.orchestra.run.smithi033.stdout: "pq_executed": 0, 2024-03-18T15:29:02.550 INFO:teuthology.orchestra.run.smithi033.stdout: "pq_item_in_journal": 0 2024-03-18T15:29:02.550 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:02.550 INFO:teuthology.orchestra.run.smithi033.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T15:29:02.550 INFO:teuthology.orchestra.run.smithi033.stdout: "val": 0, 2024-03-18T15:29:02.550 INFO:teuthology.orchestra.run.smithi033.stdout: "max": 104857600, 2024-03-18T15:29:02.550 INFO:teuthology.orchestra.run.smithi033.stdout: "get_started": 0, 2024-03-18T15:29:02.550 INFO:teuthology.orchestra.run.smithi033.stdout: "get": 712, 2024-03-18T15:29:02.550 INFO:teuthology.orchestra.run.smithi033.stdout: "get_sum": 1878167, 2024-03-18T15:29:02.550 INFO:teuthology.orchestra.run.smithi033.stdout: "get_or_fail_fail": 0, 2024-03-18T15:29:02.550 INFO:teuthology.orchestra.run.smithi033.stdout: "get_or_fail_success": 712, 2024-03-18T15:29:02.550 INFO:teuthology.orchestra.run.smithi033.stdout: "take": 0, 2024-03-18T15:29:02.551 INFO:teuthology.orchestra.run.smithi033.stdout: "take_sum": 0, 2024-03-18T15:29:02.551 INFO:teuthology.orchestra.run.smithi033.stdout: "put": 712, 2024-03-18T15:29:02.551 INFO:teuthology.orchestra.run.smithi033.stdout: "put_sum": 1878167, 2024-03-18T15:29:02.551 INFO:teuthology.orchestra.run.smithi033.stdout: "wait": { 2024-03-18T15:29:02.551 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:29:02.551 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:29:02.551 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:29:02.551 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-18T15:29:02.551 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:02.551 INFO:teuthology.orchestra.run.smithi033.stdout: "throttle-objecter_bytes": { 2024-03-18T15:29:02.551 INFO:teuthology.orchestra.run.smithi033.stdout: "val": 0, 2024-03-18T15:29:02.551 INFO:teuthology.orchestra.run.smithi033.stdout: "max": 104857600, 2024-03-18T15:29:02.551 INFO:teuthology.orchestra.run.smithi033.stdout: "get_started": 0, 2024-03-18T15:29:02.551 INFO:teuthology.orchestra.run.smithi033.stdout: "get": 0, 2024-03-18T15:29:02.551 INFO:teuthology.orchestra.run.smithi033.stdout: "get_sum": 0, 2024-03-18T15:29:02.552 INFO:teuthology.orchestra.run.smithi033.stdout: "get_or_fail_fail": 0, 2024-03-18T15:29:02.552 INFO:teuthology.orchestra.run.smithi033.stdout: "get_or_fail_success": 0, 2024-03-18T15:29:02.552 INFO:teuthology.orchestra.run.smithi033.stdout: "take": 46, 2024-03-18T15:29:02.552 INFO:teuthology.orchestra.run.smithi033.stdout: "take_sum": 12705, 2024-03-18T15:29:02.552 INFO:teuthology.orchestra.run.smithi033.stdout: "put": 46, 2024-03-18T15:29:02.552 INFO:teuthology.orchestra.run.smithi033.stdout: "put_sum": 12705, 2024-03-18T15:29:02.552 INFO:teuthology.orchestra.run.smithi033.stdout: "wait": { 2024-03-18T15:29:02.552 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:29:02.552 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:29:02.552 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:29:02.552 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-18T15:29:02.552 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:02.552 INFO:teuthology.orchestra.run.smithi033.stdout: "throttle-objecter_ops": { 2024-03-18T15:29:02.552 INFO:teuthology.orchestra.run.smithi033.stdout: "val": 0, 2024-03-18T15:29:02.553 INFO:teuthology.orchestra.run.smithi033.stdout: "max": 1024, 2024-03-18T15:29:02.553 INFO:teuthology.orchestra.run.smithi033.stdout: "get_started": 0, 2024-03-18T15:29:02.553 INFO:teuthology.orchestra.run.smithi033.stdout: "get": 0, 2024-03-18T15:29:02.553 INFO:teuthology.orchestra.run.smithi033.stdout: "get_sum": 0, 2024-03-18T15:29:02.553 INFO:teuthology.orchestra.run.smithi033.stdout: "get_or_fail_fail": 0, 2024-03-18T15:29:02.553 INFO:teuthology.orchestra.run.smithi033.stdout: "get_or_fail_success": 0, 2024-03-18T15:29:02.553 INFO:teuthology.orchestra.run.smithi033.stdout: "take": 46, 2024-03-18T15:29:02.553 INFO:teuthology.orchestra.run.smithi033.stdout: "take_sum": 46, 2024-03-18T15:29:02.553 INFO:teuthology.orchestra.run.smithi033.stdout: "put": 46, 2024-03-18T15:29:02.553 INFO:teuthology.orchestra.run.smithi033.stdout: "put_sum": 46, 2024-03-18T15:29:02.553 INFO:teuthology.orchestra.run.smithi033.stdout: "wait": { 2024-03-18T15:29:02.553 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:29:02.553 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:29:02.553 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:29:02.553 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-18T15:29:02.554 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:02.554 INFO:teuthology.orchestra.run.smithi033.stdout: "throttle-write_buf_throttle": { 2024-03-18T15:29:02.554 INFO:teuthology.orchestra.run.smithi033.stdout: "val": 0, 2024-03-18T15:29:02.554 INFO:teuthology.orchestra.run.smithi033.stdout: "max": 3758096384, 2024-03-18T15:29:02.554 INFO:teuthology.orchestra.run.smithi033.stdout: "get_started": 0, 2024-03-18T15:29:02.554 INFO:teuthology.orchestra.run.smithi033.stdout: "get": 0, 2024-03-18T15:29:02.554 INFO:teuthology.orchestra.run.smithi033.stdout: "get_sum": 0, 2024-03-18T15:29:02.554 INFO:teuthology.orchestra.run.smithi033.stdout: "get_or_fail_fail": 0, 2024-03-18T15:29:02.554 INFO:teuthology.orchestra.run.smithi033.stdout: "get_or_fail_success": 0, 2024-03-18T15:29:02.554 INFO:teuthology.orchestra.run.smithi033.stdout: "take": 0, 2024-03-18T15:29:02.554 INFO:teuthology.orchestra.run.smithi033.stdout: "take_sum": 0, 2024-03-18T15:29:02.554 INFO:teuthology.orchestra.run.smithi033.stdout: "put": 0, 2024-03-18T15:29:02.554 INFO:teuthology.orchestra.run.smithi033.stdout: "put_sum": 0, 2024-03-18T15:29:02.554 INFO:teuthology.orchestra.run.smithi033.stdout: "wait": { 2024-03-18T15:29:02.555 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:29:02.555 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:29:02.555 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:29:02.555 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-18T15:29:02.555 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:02.555 INFO:teuthology.orchestra.run.smithi033.stdout: "throttle-write_buf_throttle-0x55f1c0642500": { 2024-03-18T15:29:02.555 INFO:teuthology.orchestra.run.smithi033.stdout: "val": 0, 2024-03-18T15:29:02.555 INFO:teuthology.orchestra.run.smithi033.stdout: "max": 3758096384, 2024-03-18T15:29:02.555 INFO:teuthology.orchestra.run.smithi033.stdout: "get_started": 0, 2024-03-18T15:29:02.555 INFO:teuthology.orchestra.run.smithi033.stdout: "get": 7, 2024-03-18T15:29:02.555 INFO:teuthology.orchestra.run.smithi033.stdout: "get_sum": 1962, 2024-03-18T15:29:02.555 INFO:teuthology.orchestra.run.smithi033.stdout: "get_or_fail_fail": 0, 2024-03-18T15:29:02.555 INFO:teuthology.orchestra.run.smithi033.stdout: "get_or_fail_success": 7, 2024-03-18T15:29:02.555 INFO:teuthology.orchestra.run.smithi033.stdout: "take": 0, 2024-03-18T15:29:02.555 INFO:teuthology.orchestra.run.smithi033.stdout: "take_sum": 0, 2024-03-18T15:29:02.556 INFO:teuthology.orchestra.run.smithi033.stdout: "put": 6, 2024-03-18T15:29:02.556 INFO:teuthology.orchestra.run.smithi033.stdout: "put_sum": 1962, 2024-03-18T15:29:02.556 INFO:teuthology.orchestra.run.smithi033.stdout: "wait": { 2024-03-18T15:29:02.556 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:29:02.556 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:29:02.556 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:29:02.556 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-18T15:29:02.556 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-18T15:29:02.556 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-03-18T15:29:02.602 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:29:02 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:29:02.603 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:29:02 smithi033 ceph-mon[36648]: from='mgr.25466 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:29:02.603 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:29:02 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:29:02.603 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:29:02 smithi033 ceph-mon[36648]: from='mgr.25466 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:29:02.603 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:29:02 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:29:02.603 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:29:02 smithi033 ceph-mon[36648]: from='mgr.25466 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:29:02.758 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:29:02 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:29:02.758 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:29:02 smithi033 ceph-mon[36648]: from='mgr.25466 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:29:02.758 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:29:02 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:29:02.758 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:29:02 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:29:02.758 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:29:02 smithi033 ceph-mon[36648]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:29:02.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:29:02 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:29:02.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:29:02 smithi104 ceph-mon[36070]: from='mgr.25466 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:29:02.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:29:02 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:29:02.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:29:02 smithi104 ceph-mon[36070]: from='mgr.25466 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:29:02.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:29:02 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:29:02.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:29:02 smithi104 ceph-mon[36070]: from='mgr.25466 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:29:02.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:29:02 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:29:02.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:29:02 smithi104 ceph-mon[36070]: from='mgr.25466 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:29:02.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:29:02 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:29:02.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:29:02 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:29:02.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:29:02 smithi104 ceph-mon[36070]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:29:02.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:29:02 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:29:02.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:29:02 smithi019 ceph-mon[31522]: from='mgr.25466 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:29:02.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:29:02 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:29:02.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:29:02 smithi019 ceph-mon[31522]: from='mgr.25466 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:29:02.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:29:02 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:29:02.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:29:02 smithi019 ceph-mon[31522]: from='mgr.25466 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:29:02.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:29:02 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:29:02.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:29:02 smithi019 ceph-mon[31522]: from='mgr.25466 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:29:02.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:29:02 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:29:02.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:29:02 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:29:02.967 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:29:02 smithi019 ceph-mon[31522]: from='mgr.25466 ' entity='mgr.x' 2024-03-18T15:29:03.294 INFO:tasks.check_counter:Daemon mds.h mds.exported=0 2024-03-18T15:29:03.294 INFO:tasks.check_counter:Daemon mds.h mds.imported=0 2024-03-18T15:29:03.294 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update=0 2024-03-18T15:29:03.294 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update_receipt=8 2024-03-18T15:29:03.294 INFO:tasks.check_counter:Daemon mds.h mds.root_rsnaps=0 2024-03-18T15:29:03.294 INFO:tasks.check_counter:Daemon mds.h mds_server.req_mksnap_latency.avgcount=0 2024-03-18T15:29:03.294 INFO:tasks.check_counter:Daemon mds.h mds_server.req_rmsnap_latency.avgcount=0 2024-03-18T15:29:03.295 DEBUG:tasks.check_counter:Getting stats from k 2024-03-18T15:29:03.295 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 -- ceph daemon mds.k perf dump 2024-03-18T15:29:03.555 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T15:29:03.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:29:03 smithi104 ceph-mon[36070]: pgmap v328: 129 pgs: 129 active+clean; 1.8 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-18T15:29:03.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:29:03 smithi104 ceph-mon[36070]: Adjusting osd_memory_target on smithi033 to 175.2M 2024-03-18T15:29:03.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:29:03 smithi104 ceph-mon[36070]: Unable to set osd_memory_target on smithi033 to 183744512: error parsing value: Value '183744512' is below minimum 939524096 2024-03-18T15:29:03.954 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:29:03 smithi033 ceph-mon[36648]: pgmap v328: 129 pgs: 129 active+clean; 1.8 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-18T15:29:03.954 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:29:03 smithi033 ceph-mon[36648]: Adjusting osd_memory_target on smithi033 to 175.2M 2024-03-18T15:29:03.954 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:29:03 smithi033 ceph-mon[36648]: Unable to set osd_memory_target on smithi033 to 183744512: error parsing value: Value '183744512' is below minimum 939524096 2024-03-18T15:29:03.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:29:03 smithi019 ceph-mon[31522]: pgmap v328: 129 pgs: 129 active+clean; 1.8 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-18T15:29:03.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:29:03 smithi019 ceph-mon[31522]: Adjusting osd_memory_target on smithi033 to 175.2M 2024-03-18T15:29:03.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:29:03 smithi019 ceph-mon[31522]: Unable to set osd_memory_target on smithi033 to 183744512: error parsing value: Value '183744512' is below minimum 939524096 2024-03-18T15:29:05.401 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-03-18T15:29:05.401 INFO:teuthology.orchestra.run.smithi033.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T15:29:05.402 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_messages": 863, 2024-03-18T15:29:05.402 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_messages": 865, 2024-03-18T15:29:05.402 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_bytes": 458548, 2024-03-18T15:29:05.402 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_bytes": 274498, 2024-03-18T15:29:05.402 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_created_connections": 6, 2024-03-18T15:29:05.402 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_active_connections": 2, 2024-03-18T15:29:05.402 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_total_time": 0.056623816, 2024-03-18T15:29:05.402 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_send_time": 0.020661262, 2024-03-18T15:29:05.402 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_recv_time": 0.014114745, 2024-03-18T15:29:05.402 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_fast_dispatch_time": 0.016019523, 2024-03-18T15:29:05.402 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T15:29:05.403 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 863, 2024-03-18T15:29:05.403 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.020176981, 2024-03-18T15:29:05.403 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000023380 2024-03-18T15:29:05.403 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:05.403 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_handle_ack_lat": { 2024-03-18T15:29:05.403 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:29:05.403 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:29:05.403 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:29:05.403 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:05.403 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-18T15:29:05.403 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_encrypted_bytes": 0 2024-03-18T15:29:05.404 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:05.404 INFO:teuthology.orchestra.run.smithi033.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T15:29:05.404 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_messages": 272, 2024-03-18T15:29:05.404 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_messages": 382, 2024-03-18T15:29:05.404 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_bytes": 1779474, 2024-03-18T15:29:05.404 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_bytes": 462993, 2024-03-18T15:29:05.404 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_created_connections": 4, 2024-03-18T15:29:05.404 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_active_connections": 3, 2024-03-18T15:29:05.404 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_total_time": 0.046132210, 2024-03-18T15:29:05.404 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_send_time": 0.013382408, 2024-03-18T15:29:05.404 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_recv_time": 0.040803978, 2024-03-18T15:29:05.405 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_fast_dispatch_time": 0.004325949, 2024-03-18T15:29:05.405 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T15:29:05.405 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 381, 2024-03-18T15:29:05.405 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.023923598, 2024-03-18T15:29:05.405 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000062791 2024-03-18T15:29:05.405 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:05.405 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_handle_ack_lat": { 2024-03-18T15:29:05.405 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:29:05.405 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:29:05.405 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:29:05.406 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:05.406 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_encrypted_bytes": 1778960, 2024-03-18T15:29:05.406 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_encrypted_bytes": 462352 2024-03-18T15:29:05.406 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:05.406 INFO:teuthology.orchestra.run.smithi033.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T15:29:05.406 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_messages": 1436, 2024-03-18T15:29:05.406 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_messages": 1483, 2024-03-18T15:29:05.406 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_bytes": 426988, 2024-03-18T15:29:05.406 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_bytes": 599650, 2024-03-18T15:29:05.406 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_created_connections": 6, 2024-03-18T15:29:05.406 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_active_connections": 4, 2024-03-18T15:29:05.406 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_total_time": 0.098761257, 2024-03-18T15:29:05.407 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_send_time": 0.029405965, 2024-03-18T15:29:05.407 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_recv_time": 0.024392693, 2024-03-18T15:29:05.407 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_fast_dispatch_time": 0.035531041, 2024-03-18T15:29:05.407 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T15:29:05.407 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 1481, 2024-03-18T15:29:05.407 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.036502466, 2024-03-18T15:29:05.407 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000024647 2024-03-18T15:29:05.407 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:05.407 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_handle_ack_lat": { 2024-03-18T15:29:05.407 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:29:05.407 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:29:05.408 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:29:05.408 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:05.408 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_encrypted_bytes": 3232, 2024-03-18T15:29:05.408 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_encrypted_bytes": 144592 2024-03-18T15:29:05.408 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:05.408 INFO:teuthology.orchestra.run.smithi033.stdout: "cct": { 2024-03-18T15:29:05.408 INFO:teuthology.orchestra.run.smithi033.stdout: "total_workers": 1, 2024-03-18T15:29:05.408 INFO:teuthology.orchestra.run.smithi033.stdout: "unhealthy_workers": 0 2024-03-18T15:29:05.408 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:05.408 INFO:teuthology.orchestra.run.smithi033.stdout: "finisher-MDSRank": { 2024-03-18T15:29:05.408 INFO:teuthology.orchestra.run.smithi033.stdout: "queue_len": 0, 2024-03-18T15:29:05.409 INFO:teuthology.orchestra.run.smithi033.stdout: "complete_latency": { 2024-03-18T15:29:05.409 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 3950, 2024-03-18T15:29:05.409 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.079895052, 2024-03-18T15:29:05.409 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000020226 2024-03-18T15:29:05.409 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-18T15:29:05.409 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:05.409 INFO:teuthology.orchestra.run.smithi033.stdout: "finisher-PurgeQueue": { 2024-03-18T15:29:05.409 INFO:teuthology.orchestra.run.smithi033.stdout: "queue_len": 0, 2024-03-18T15:29:05.409 INFO:teuthology.orchestra.run.smithi033.stdout: "complete_latency": { 2024-03-18T15:29:05.409 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:29:05.409 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:29:05.409 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:29:05.410 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-18T15:29:05.410 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:05.410 INFO:teuthology.orchestra.run.smithi033.stdout: "mds": { 2024-03-18T15:29:05.410 INFO:teuthology.orchestra.run.smithi033.stdout: "request": 0, 2024-03-18T15:29:05.410 INFO:teuthology.orchestra.run.smithi033.stdout: "reply": 0, 2024-03-18T15:29:05.410 INFO:teuthology.orchestra.run.smithi033.stdout: "reply_latency": { 2024-03-18T15:29:05.410 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:29:05.410 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:29:05.410 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:29:05.410 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:05.410 INFO:teuthology.orchestra.run.smithi033.stdout: "slow_reply": 0, 2024-03-18T15:29:05.410 INFO:teuthology.orchestra.run.smithi033.stdout: "forward": 0, 2024-03-18T15:29:05.410 INFO:teuthology.orchestra.run.smithi033.stdout: "dir_fetch_complete": 0, 2024-03-18T15:29:05.411 INFO:teuthology.orchestra.run.smithi033.stdout: "dir_fetch_keys": 0, 2024-03-18T15:29:05.411 INFO:teuthology.orchestra.run.smithi033.stdout: "dir_commit": 0, 2024-03-18T15:29:05.411 INFO:teuthology.orchestra.run.smithi033.stdout: "dir_split": 0, 2024-03-18T15:29:05.411 INFO:teuthology.orchestra.run.smithi033.stdout: "dir_merge": 0, 2024-03-18T15:29:05.411 INFO:teuthology.orchestra.run.smithi033.stdout: "inodes": 14, 2024-03-18T15:29:05.411 INFO:teuthology.orchestra.run.smithi033.stdout: "inodes_top": 0, 2024-03-18T15:29:05.411 INFO:teuthology.orchestra.run.smithi033.stdout: "inodes_bottom": 14, 2024-03-18T15:29:05.411 INFO:teuthology.orchestra.run.smithi033.stdout: "inodes_pin_tail": 0, 2024-03-18T15:29:05.411 INFO:teuthology.orchestra.run.smithi033.stdout: "inodes_pinned": 14, 2024-03-18T15:29:05.411 INFO:teuthology.orchestra.run.smithi033.stdout: "inodes_expired": 0, 2024-03-18T15:29:05.411 INFO:teuthology.orchestra.run.smithi033.stdout: "inodes_with_caps": 0, 2024-03-18T15:29:05.411 INFO:teuthology.orchestra.run.smithi033.stdout: "caps": 0, 2024-03-18T15:29:05.411 INFO:teuthology.orchestra.run.smithi033.stdout: "subtrees": 1, 2024-03-18T15:29:05.411 INFO:teuthology.orchestra.run.smithi033.stdout: "traverse": 0, 2024-03-18T15:29:05.412 INFO:teuthology.orchestra.run.smithi033.stdout: "traverse_hit": 0, 2024-03-18T15:29:05.412 INFO:teuthology.orchestra.run.smithi033.stdout: "traverse_forward": 0, 2024-03-18T15:29:05.412 INFO:teuthology.orchestra.run.smithi033.stdout: "traverse_discover": 0, 2024-03-18T15:29:05.412 INFO:teuthology.orchestra.run.smithi033.stdout: "traverse_dir_fetch": 0, 2024-03-18T15:29:05.412 INFO:teuthology.orchestra.run.smithi033.stdout: "traverse_remote_ino": 0, 2024-03-18T15:29:05.412 INFO:teuthology.orchestra.run.smithi033.stdout: "traverse_lock": 0, 2024-03-18T15:29:05.412 INFO:teuthology.orchestra.run.smithi033.stdout: "load_cent": 0, 2024-03-18T15:29:05.412 INFO:teuthology.orchestra.run.smithi033.stdout: "q": 0, 2024-03-18T15:29:05.412 INFO:teuthology.orchestra.run.smithi033.stdout: "exported": 0, 2024-03-18T15:29:05.412 INFO:teuthology.orchestra.run.smithi033.stdout: "exported_inodes": 0, 2024-03-18T15:29:05.412 INFO:teuthology.orchestra.run.smithi033.stdout: "imported": 0, 2024-03-18T15:29:05.412 INFO:teuthology.orchestra.run.smithi033.stdout: "imported_inodes": 0, 2024-03-18T15:29:05.412 INFO:teuthology.orchestra.run.smithi033.stdout: "openino_dir_fetch": 0, 2024-03-18T15:29:05.412 INFO:teuthology.orchestra.run.smithi033.stdout: "openino_backtrace_fetch": 0, 2024-03-18T15:29:05.413 INFO:teuthology.orchestra.run.smithi033.stdout: "openino_peer_discover": 0, 2024-03-18T15:29:05.413 INFO:teuthology.orchestra.run.smithi033.stdout: "root_rfiles": 0, 2024-03-18T15:29:05.413 INFO:teuthology.orchestra.run.smithi033.stdout: "root_rbytes": 0, 2024-03-18T15:29:05.413 INFO:teuthology.orchestra.run.smithi033.stdout: "root_rsnaps": 0, 2024-03-18T15:29:05.413 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_backtrace_fetch": 0, 2024-03-18T15:29:05.413 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_set_tag": 0, 2024-03-18T15:29:05.413 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_backtrace_repaired": 0, 2024-03-18T15:29:05.413 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_inotable_repaired": 0, 2024-03-18T15:29:05.413 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_dir_inodes": 0, 2024-03-18T15:29:05.413 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_dir_base_inodes": 0, 2024-03-18T15:29:05.413 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_dirfrag_rstats": 0, 2024-03-18T15:29:05.413 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_file_inodes": 0, 2024-03-18T15:29:05.413 INFO:teuthology.orchestra.run.smithi033.stdout: "handle_inode_file_caps": 0, 2024-03-18T15:29:05.413 INFO:teuthology.orchestra.run.smithi033.stdout: "ceph_cap_op_revoke": 0, 2024-03-18T15:29:05.413 INFO:teuthology.orchestra.run.smithi033.stdout: "ceph_cap_op_grant": 0, 2024-03-18T15:29:05.414 INFO:teuthology.orchestra.run.smithi033.stdout: "ceph_cap_op_trunc": 0, 2024-03-18T15:29:05.414 INFO:teuthology.orchestra.run.smithi033.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-18T15:29:05.414 INFO:teuthology.orchestra.run.smithi033.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-18T15:29:05.414 INFO:teuthology.orchestra.run.smithi033.stdout: "handle_client_caps": 0, 2024-03-18T15:29:05.414 INFO:teuthology.orchestra.run.smithi033.stdout: "handle_client_caps_dirty": 0, 2024-03-18T15:29:05.414 INFO:teuthology.orchestra.run.smithi033.stdout: "handle_client_cap_release": 0, 2024-03-18T15:29:05.414 INFO:teuthology.orchestra.run.smithi033.stdout: "process_request_cap_release": 0 2024-03-18T15:29:05.414 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:05.414 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_cache": { 2024-03-18T15:29:05.414 INFO:teuthology.orchestra.run.smithi033.stdout: "dir_update": 0, 2024-03-18T15:29:05.414 INFO:teuthology.orchestra.run.smithi033.stdout: "dir_update_receipt": 0, 2024-03-18T15:29:05.414 INFO:teuthology.orchestra.run.smithi033.stdout: "dir_try_discover": 0, 2024-03-18T15:29:05.414 INFO:teuthology.orchestra.run.smithi033.stdout: "dir_send_discover": 0, 2024-03-18T15:29:05.414 INFO:teuthology.orchestra.run.smithi033.stdout: "dir_handle_discover": 0, 2024-03-18T15:29:05.415 INFO:teuthology.orchestra.run.smithi033.stdout: "num_strays": 0, 2024-03-18T15:29:05.415 INFO:teuthology.orchestra.run.smithi033.stdout: "num_strays_delayed": 0, 2024-03-18T15:29:05.415 INFO:teuthology.orchestra.run.smithi033.stdout: "num_strays_enqueuing": 0, 2024-03-18T15:29:05.415 INFO:teuthology.orchestra.run.smithi033.stdout: "strays_created": 0, 2024-03-18T15:29:05.415 INFO:teuthology.orchestra.run.smithi033.stdout: "strays_enqueued": 0, 2024-03-18T15:29:05.415 INFO:teuthology.orchestra.run.smithi033.stdout: "strays_reintegrated": 0, 2024-03-18T15:29:05.415 INFO:teuthology.orchestra.run.smithi033.stdout: "strays_migrated": 0, 2024-03-18T15:29:05.415 INFO:teuthology.orchestra.run.smithi033.stdout: "num_recovering_processing": 0, 2024-03-18T15:29:05.415 INFO:teuthology.orchestra.run.smithi033.stdout: "num_recovering_enqueued": 0, 2024-03-18T15:29:05.415 INFO:teuthology.orchestra.run.smithi033.stdout: "num_recovering_prioritized": 0, 2024-03-18T15:29:05.415 INFO:teuthology.orchestra.run.smithi033.stdout: "recovery_started": 0, 2024-03-18T15:29:05.415 INFO:teuthology.orchestra.run.smithi033.stdout: "recovery_completed": 0, 2024-03-18T15:29:05.415 INFO:teuthology.orchestra.run.smithi033.stdout: "ireq_quiesce_path": 0, 2024-03-18T15:29:05.415 INFO:teuthology.orchestra.run.smithi033.stdout: "ireq_quiesce_inode": 0, 2024-03-18T15:29:05.416 INFO:teuthology.orchestra.run.smithi033.stdout: "ireq_enqueue_scrub": 0, 2024-03-18T15:29:05.416 INFO:teuthology.orchestra.run.smithi033.stdout: "ireq_exportdir": 0, 2024-03-18T15:29:05.416 INFO:teuthology.orchestra.run.smithi033.stdout: "ireq_flush": 0, 2024-03-18T15:29:05.416 INFO:teuthology.orchestra.run.smithi033.stdout: "ireq_fragmentdir": 0, 2024-03-18T15:29:05.416 INFO:teuthology.orchestra.run.smithi033.stdout: "ireq_fragstats": 0, 2024-03-18T15:29:05.416 INFO:teuthology.orchestra.run.smithi033.stdout: "ireq_inodestats": 0 2024-03-18T15:29:05.416 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:05.416 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_log": { 2024-03-18T15:29:05.416 INFO:teuthology.orchestra.run.smithi033.stdout: "evlrg": 0, 2024-03-18T15:29:05.416 INFO:teuthology.orchestra.run.smithi033.stdout: "evadd": 0, 2024-03-18T15:29:05.416 INFO:teuthology.orchestra.run.smithi033.stdout: "evex": 0, 2024-03-18T15:29:05.416 INFO:teuthology.orchestra.run.smithi033.stdout: "evtrm": 0, 2024-03-18T15:29:05.416 INFO:teuthology.orchestra.run.smithi033.stdout: "ev": 106, 2024-03-18T15:29:05.416 INFO:teuthology.orchestra.run.smithi033.stdout: "evexg": 0, 2024-03-18T15:29:05.416 INFO:teuthology.orchestra.run.smithi033.stdout: "evexd": 0, 2024-03-18T15:29:05.417 INFO:teuthology.orchestra.run.smithi033.stdout: "segadd": 0, 2024-03-18T15:29:05.417 INFO:teuthology.orchestra.run.smithi033.stdout: "segex": 0, 2024-03-18T15:29:05.417 INFO:teuthology.orchestra.run.smithi033.stdout: "segtrm": 0, 2024-03-18T15:29:05.417 INFO:teuthology.orchestra.run.smithi033.stdout: "seg": 2, 2024-03-18T15:29:05.417 INFO:teuthology.orchestra.run.smithi033.stdout: "segmjr": 2, 2024-03-18T15:29:05.417 INFO:teuthology.orchestra.run.smithi033.stdout: "segexg": 0, 2024-03-18T15:29:05.417 INFO:teuthology.orchestra.run.smithi033.stdout: "segexd": 0, 2024-03-18T15:29:05.417 INFO:teuthology.orchestra.run.smithi033.stdout: "expos": 4194304, 2024-03-18T15:29:05.417 INFO:teuthology.orchestra.run.smithi033.stdout: "wrpos": 4427115, 2024-03-18T15:29:05.417 INFO:teuthology.orchestra.run.smithi033.stdout: "rdpos": 4426516, 2024-03-18T15:29:05.417 INFO:teuthology.orchestra.run.smithi033.stdout: "jlat": { 2024-03-18T15:29:05.417 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:29:05.417 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:29:05.418 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:29:05.418 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:05.418 INFO:teuthology.orchestra.run.smithi033.stdout: "replayed": 106 2024-03-18T15:29:05.418 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:05.418 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_mem": { 2024-03-18T15:29:05.418 INFO:teuthology.orchestra.run.smithi033.stdout: "ino": 9, 2024-03-18T15:29:05.418 INFO:teuthology.orchestra.run.smithi033.stdout: "ino+": 20, 2024-03-18T15:29:05.418 INFO:teuthology.orchestra.run.smithi033.stdout: "ino-": 11, 2024-03-18T15:29:05.418 INFO:teuthology.orchestra.run.smithi033.stdout: "dir": 3, 2024-03-18T15:29:05.418 INFO:teuthology.orchestra.run.smithi033.stdout: "dir+": 11, 2024-03-18T15:29:05.418 INFO:teuthology.orchestra.run.smithi033.stdout: "dir-": 8, 2024-03-18T15:29:05.418 INFO:teuthology.orchestra.run.smithi033.stdout: "dn": 14, 2024-03-18T15:29:05.418 INFO:teuthology.orchestra.run.smithi033.stdout: "dn+": 23, 2024-03-18T15:29:05.418 INFO:teuthology.orchestra.run.smithi033.stdout: "dn-": 9, 2024-03-18T15:29:05.418 INFO:teuthology.orchestra.run.smithi033.stdout: "cap": 0, 2024-03-18T15:29:05.419 INFO:teuthology.orchestra.run.smithi033.stdout: "cap+": 0, 2024-03-18T15:29:05.419 INFO:teuthology.orchestra.run.smithi033.stdout: "cap-": 0, 2024-03-18T15:29:05.419 INFO:teuthology.orchestra.run.smithi033.stdout: "rss": 46508, 2024-03-18T15:29:05.419 INFO:teuthology.orchestra.run.smithi033.stdout: "heap": 215420 2024-03-18T15:29:05.419 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:05.419 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_server": { 2024-03-18T15:29:05.419 INFO:teuthology.orchestra.run.smithi033.stdout: "dispatch_client_request": 0, 2024-03-18T15:29:05.419 INFO:teuthology.orchestra.run.smithi033.stdout: "dispatch_server_request": 0, 2024-03-18T15:29:05.419 INFO:teuthology.orchestra.run.smithi033.stdout: "handle_client_request": 0, 2024-03-18T15:29:05.419 INFO:teuthology.orchestra.run.smithi033.stdout: "handle_client_session": 0, 2024-03-18T15:29:05.419 INFO:teuthology.orchestra.run.smithi033.stdout: "handle_peer_request": 0, 2024-03-18T15:29:05.419 INFO:teuthology.orchestra.run.smithi033.stdout: "req_create_latency": { 2024-03-18T15:29:05.419 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:29:05.419 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:29:05.420 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:29:05.420 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:05.420 INFO:teuthology.orchestra.run.smithi033.stdout: "req_getattr_latency": { 2024-03-18T15:29:05.420 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:29:05.420 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:29:05.420 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:29:05.420 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:05.420 INFO:teuthology.orchestra.run.smithi033.stdout: "req_getfilelock_latency": { 2024-03-18T15:29:05.420 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:29:05.420 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:29:05.420 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:29:05.420 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:05.420 INFO:teuthology.orchestra.run.smithi033.stdout: "req_link_latency": { 2024-03-18T15:29:05.420 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:29:05.421 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:29:05.421 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:29:05.421 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:05.421 INFO:teuthology.orchestra.run.smithi033.stdout: "req_lookup_latency": { 2024-03-18T15:29:05.421 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:29:05.421 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:29:05.421 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:29:05.421 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:05.421 INFO:teuthology.orchestra.run.smithi033.stdout: "req_lookuphash_latency": { 2024-03-18T15:29:05.421 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:29:05.421 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:29:05.421 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:29:05.421 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:05.422 INFO:teuthology.orchestra.run.smithi033.stdout: "req_lookupino_latency": { 2024-03-18T15:29:05.422 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:29:05.422 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:29:05.422 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:29:05.422 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:05.422 INFO:teuthology.orchestra.run.smithi033.stdout: "req_lookupname_latency": { 2024-03-18T15:29:05.422 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:29:05.422 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:29:05.422 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:29:05.422 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:05.422 INFO:teuthology.orchestra.run.smithi033.stdout: "req_lookupparent_latency": { 2024-03-18T15:29:05.422 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:29:05.423 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:29:05.423 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:29:05.423 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:05.423 INFO:teuthology.orchestra.run.smithi033.stdout: "req_lookupsnap_latency": { 2024-03-18T15:29:05.423 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:29:05.423 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:29:05.423 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:29:05.423 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:05.423 INFO:teuthology.orchestra.run.smithi033.stdout: "req_lssnap_latency": { 2024-03-18T15:29:05.423 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:29:05.423 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:29:05.423 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:29:05.423 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:05.424 INFO:teuthology.orchestra.run.smithi033.stdout: "req_mkdir_latency": { 2024-03-18T15:29:05.424 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:29:05.424 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:29:05.425 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:29:05.425 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:05.425 INFO:teuthology.orchestra.run.smithi033.stdout: "req_mknod_latency": { 2024-03-18T15:29:05.425 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:29:05.425 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:29:05.425 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:29:05.425 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:05.425 INFO:teuthology.orchestra.run.smithi033.stdout: "req_mksnap_latency": { 2024-03-18T15:29:05.425 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:29:05.425 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:29:05.425 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:29:05.426 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:05.426 INFO:teuthology.orchestra.run.smithi033.stdout: "req_open_latency": { 2024-03-18T15:29:05.426 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:29:05.426 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:29:05.426 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:29:05.426 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:05.426 INFO:teuthology.orchestra.run.smithi033.stdout: "req_readdir_latency": { 2024-03-18T15:29:05.426 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:29:05.426 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:29:05.426 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:29:05.426 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:05.427 INFO:teuthology.orchestra.run.smithi033.stdout: "req_rename_latency": { 2024-03-18T15:29:05.427 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:29:05.427 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:29:05.427 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:29:05.427 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:05.427 INFO:teuthology.orchestra.run.smithi033.stdout: "req_renamesnap_latency": { 2024-03-18T15:29:05.427 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:29:05.427 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:29:05.427 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:29:05.427 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:05.427 INFO:teuthology.orchestra.run.smithi033.stdout: "req_snapdiff_latency": { 2024-03-18T15:29:05.427 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:29:05.428 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:29:05.428 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:29:05.428 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:05.428 INFO:teuthology.orchestra.run.smithi033.stdout: "req_rmdir_latency": { 2024-03-18T15:29:05.428 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:29:05.428 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:29:05.428 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:29:05.428 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:05.428 INFO:teuthology.orchestra.run.smithi033.stdout: "req_rmsnap_latency": { 2024-03-18T15:29:05.428 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:29:05.428 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:29:05.428 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:29:05.428 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:05.429 INFO:teuthology.orchestra.run.smithi033.stdout: "req_rmxattr_latency": { 2024-03-18T15:29:05.429 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:29:05.429 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:29:05.429 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:29:05.429 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:05.429 INFO:teuthology.orchestra.run.smithi033.stdout: "req_setattr_latency": { 2024-03-18T15:29:05.429 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:29:05.429 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:29:05.429 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:29:05.429 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:05.429 INFO:teuthology.orchestra.run.smithi033.stdout: "req_setdirlayout_latency": { 2024-03-18T15:29:05.429 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:29:05.429 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:29:05.429 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:29:05.430 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:05.430 INFO:teuthology.orchestra.run.smithi033.stdout: "req_setfilelock_latency": { 2024-03-18T15:29:05.430 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:29:05.430 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:29:05.430 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:29:05.430 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:05.430 INFO:teuthology.orchestra.run.smithi033.stdout: "req_setlayout_latency": { 2024-03-18T15:29:05.430 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:29:05.430 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:29:05.430 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:29:05.430 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:05.430 INFO:teuthology.orchestra.run.smithi033.stdout: "req_setxattr_latency": { 2024-03-18T15:29:05.430 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:29:05.431 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:29:05.431 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:29:05.431 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:05.431 INFO:teuthology.orchestra.run.smithi033.stdout: "req_symlink_latency": { 2024-03-18T15:29:05.431 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:29:05.431 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:29:05.431 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:29:05.431 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:05.431 INFO:teuthology.orchestra.run.smithi033.stdout: "req_unlink_latency": { 2024-03-18T15:29:05.431 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:29:05.431 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:29:05.432 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:29:05.432 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:05.432 INFO:teuthology.orchestra.run.smithi033.stdout: "cap_revoke_eviction": 0, 2024-03-18T15:29:05.432 INFO:teuthology.orchestra.run.smithi033.stdout: "cap_acquisition_throttle": 0, 2024-03-18T15:29:05.432 INFO:teuthology.orchestra.run.smithi033.stdout: "req_getvxattr_latency": { 2024-03-18T15:29:05.432 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:29:05.432 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:29:05.432 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:29:05.432 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-18T15:29:05.432 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:05.432 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_sessions": { 2024-03-18T15:29:05.432 INFO:teuthology.orchestra.run.smithi033.stdout: "session_count": 1, 2024-03-18T15:29:05.433 INFO:teuthology.orchestra.run.smithi033.stdout: "session_add": 2, 2024-03-18T15:29:05.433 INFO:teuthology.orchestra.run.smithi033.stdout: "session_remove": 1, 2024-03-18T15:29:05.433 INFO:teuthology.orchestra.run.smithi033.stdout: "sessions_open": 2, 2024-03-18T15:29:05.433 INFO:teuthology.orchestra.run.smithi033.stdout: "sessions_stale": 0, 2024-03-18T15:29:05.433 INFO:teuthology.orchestra.run.smithi033.stdout: "total_load": 0, 2024-03-18T15:29:05.433 INFO:teuthology.orchestra.run.smithi033.stdout: "average_load": 0, 2024-03-18T15:29:05.433 INFO:teuthology.orchestra.run.smithi033.stdout: "avg_session_uptime": 3975, 2024-03-18T15:29:05.433 INFO:teuthology.orchestra.run.smithi033.stdout: "mdthresh_evicted": 0 2024-03-18T15:29:05.433 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:05.433 INFO:teuthology.orchestra.run.smithi033.stdout: "mempool": { 2024-03-18T15:29:05.433 INFO:teuthology.orchestra.run.smithi033.stdout: "bloom_filter_bytes": 0, 2024-03-18T15:29:05.433 INFO:teuthology.orchestra.run.smithi033.stdout: "bloom_filter_items": 0, 2024-03-18T15:29:05.434 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T15:29:05.434 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_alloc_items": 0, 2024-03-18T15:29:05.434 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T15:29:05.434 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_data_items": 0, 2024-03-18T15:29:05.434 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T15:29:05.434 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T15:29:05.434 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T15:29:05.434 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T15:29:05.434 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T15:29:05.434 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_other_items": 0, 2024-03-18T15:29:05.434 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T15:29:05.435 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T15:29:05.435 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_extent_bytes": 0, 2024-03-18T15:29:05.435 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_extent_items": 0, 2024-03-18T15:29:05.435 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_blob_bytes": 0, 2024-03-18T15:29:05.435 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_blob_items": 0, 2024-03-18T15:29:05.435 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T15:29:05.435 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T15:29:05.435 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T15:29:05.435 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T15:29:05.435 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T15:29:05.435 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_fsck_items": 0, 2024-03-18T15:29:05.435 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_txc_bytes": 0, 2024-03-18T15:29:05.436 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_txc_items": 0, 2024-03-18T15:29:05.436 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T15:29:05.436 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T15:29:05.436 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_writing_bytes": 0, 2024-03-18T15:29:05.436 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_writing_items": 0, 2024-03-18T15:29:05.436 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_bytes": 0, 2024-03-18T15:29:05.436 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_items": 0, 2024-03-18T15:29:05.436 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T15:29:05.436 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_file_reader_items": 0, 2024-03-18T15:29:05.436 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T15:29:05.436 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_file_writer_items": 0, 2024-03-18T15:29:05.437 INFO:teuthology.orchestra.run.smithi033.stdout: "buffer_anon_bytes": 190417, 2024-03-18T15:29:05.437 INFO:teuthology.orchestra.run.smithi033.stdout: "buffer_anon_items": 60, 2024-03-18T15:29:05.437 INFO:teuthology.orchestra.run.smithi033.stdout: "buffer_meta_bytes": 0, 2024-03-18T15:29:05.437 INFO:teuthology.orchestra.run.smithi033.stdout: "buffer_meta_items": 0, 2024-03-18T15:29:05.437 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_bytes": 0, 2024-03-18T15:29:05.437 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_items": 0, 2024-03-18T15:29:05.437 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mapbl_bytes": 0, 2024-03-18T15:29:05.437 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mapbl_items": 0, 2024-03-18T15:29:05.437 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pglog_bytes": 0, 2024-03-18T15:29:05.437 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pglog_items": 0, 2024-03-18T15:29:05.437 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_bytes": 9928, 2024-03-18T15:29:05.437 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_items": 207, 2024-03-18T15:29:05.438 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T15:29:05.438 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_mapping_items": 0, 2024-03-18T15:29:05.438 INFO:teuthology.orchestra.run.smithi033.stdout: "pgmap_bytes": 0, 2024-03-18T15:29:05.438 INFO:teuthology.orchestra.run.smithi033.stdout: "pgmap_items": 0, 2024-03-18T15:29:05.438 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_co_bytes": 37569, 2024-03-18T15:29:05.438 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_co_items": 385, 2024-03-18T15:29:05.438 INFO:teuthology.orchestra.run.smithi033.stdout: "unittest_1_bytes": 0, 2024-03-18T15:29:05.438 INFO:teuthology.orchestra.run.smithi033.stdout: "unittest_1_items": 0, 2024-03-18T15:29:05.438 INFO:teuthology.orchestra.run.smithi033.stdout: "unittest_2_bytes": 0, 2024-03-18T15:29:05.438 INFO:teuthology.orchestra.run.smithi033.stdout: "unittest_2_items": 0 2024-03-18T15:29:05.438 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:05.439 INFO:teuthology.orchestra.run.smithi033.stdout: "objecter": { 2024-03-18T15:29:05.439 INFO:teuthology.orchestra.run.smithi033.stdout: "op_active": 0, 2024-03-18T15:29:05.439 INFO:teuthology.orchestra.run.smithi033.stdout: "op_laggy": 0, 2024-03-18T15:29:05.439 INFO:teuthology.orchestra.run.smithi033.stdout: "op_send": 2296, 2024-03-18T15:29:05.439 INFO:teuthology.orchestra.run.smithi033.stdout: "op_send_bytes": 21, 2024-03-18T15:29:05.439 INFO:teuthology.orchestra.run.smithi033.stdout: "op_resend": 3, 2024-03-18T15:29:05.439 INFO:teuthology.orchestra.run.smithi033.stdout: "op_reply": 2296, 2024-03-18T15:29:05.439 INFO:teuthology.orchestra.run.smithi033.stdout: "op_latency": { 2024-03-18T15:29:05.439 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 2296, 2024-03-18T15:29:05.439 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 1.648971890, 2024-03-18T15:29:05.439 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000718193 2024-03-18T15:29:05.440 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:05.440 INFO:teuthology.orchestra.run.smithi033.stdout: "op_inflight": 0, 2024-03-18T15:29:05.440 INFO:teuthology.orchestra.run.smithi033.stdout: "oplen_avg": { 2024-03-18T15:29:05.440 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 2296, 2024-03-18T15:29:05.440 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 2297 2024-03-18T15:29:05.440 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:05.440 INFO:teuthology.orchestra.run.smithi033.stdout: "op": 2296, 2024-03-18T15:29:05.440 INFO:teuthology.orchestra.run.smithi033.stdout: "op_r": 2296, 2024-03-18T15:29:05.440 INFO:teuthology.orchestra.run.smithi033.stdout: "op_w": 0, 2024-03-18T15:29:05.441 INFO:teuthology.orchestra.run.smithi033.stdout: "op_rmw": 0, 2024-03-18T15:29:05.441 INFO:teuthology.orchestra.run.smithi033.stdout: "op_pg": 0, 2024-03-18T15:29:05.441 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_stat": 1431, 2024-03-18T15:29:05.441 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_create": 0, 2024-03-18T15:29:05.441 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_read": 863, 2024-03-18T15:29:05.441 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_write": 0, 2024-03-18T15:29:05.441 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_writefull": 0, 2024-03-18T15:29:05.441 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_writesame": 0, 2024-03-18T15:29:05.441 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_append": 0, 2024-03-18T15:29:05.441 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_zero": 0, 2024-03-18T15:29:05.441 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_truncate": 0, 2024-03-18T15:29:05.441 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_delete": 0, 2024-03-18T15:29:05.442 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_mapext": 0, 2024-03-18T15:29:05.442 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_sparse_read": 0, 2024-03-18T15:29:05.443 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_clonerange": 0, 2024-03-18T15:29:05.443 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_getxattr": 1, 2024-03-18T15:29:05.443 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_setxattr": 0, 2024-03-18T15:29:05.443 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_cmpxattr": 0, 2024-03-18T15:29:05.443 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_rmxattr": 0, 2024-03-18T15:29:05.444 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_resetxattrs": 0, 2024-03-18T15:29:05.444 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_call": 0, 2024-03-18T15:29:05.444 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_watch": 0, 2024-03-18T15:29:05.444 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_notify": 0, 2024-03-18T15:29:05.444 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_src_cmpxattr": 0, 2024-03-18T15:29:05.444 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_pgls": 0, 2024-03-18T15:29:05.444 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_pgls_filter": 0, 2024-03-18T15:29:05.444 INFO:teuthology.orchestra.run.smithi033.stdout: "osdop_other": 0, 2024-03-18T15:29:05.444 INFO:teuthology.orchestra.run.smithi033.stdout: "linger_active": 0, 2024-03-18T15:29:05.444 INFO:teuthology.orchestra.run.smithi033.stdout: "linger_send": 0, 2024-03-18T15:29:05.444 INFO:teuthology.orchestra.run.smithi033.stdout: "linger_resend": 0, 2024-03-18T15:29:05.444 INFO:teuthology.orchestra.run.smithi033.stdout: "linger_ping": 0, 2024-03-18T15:29:05.444 INFO:teuthology.orchestra.run.smithi033.stdout: "poolop_active": 0, 2024-03-18T15:29:05.444 INFO:teuthology.orchestra.run.smithi033.stdout: "poolop_send": 0, 2024-03-18T15:29:05.445 INFO:teuthology.orchestra.run.smithi033.stdout: "poolop_resend": 0, 2024-03-18T15:29:05.445 INFO:teuthology.orchestra.run.smithi033.stdout: "poolstat_active": 0, 2024-03-18T15:29:05.445 INFO:teuthology.orchestra.run.smithi033.stdout: "poolstat_send": 0, 2024-03-18T15:29:05.445 INFO:teuthology.orchestra.run.smithi033.stdout: "poolstat_resend": 0, 2024-03-18T15:29:05.445 INFO:teuthology.orchestra.run.smithi033.stdout: "statfs_active": 0, 2024-03-18T15:29:05.445 INFO:teuthology.orchestra.run.smithi033.stdout: "statfs_send": 0, 2024-03-18T15:29:05.445 INFO:teuthology.orchestra.run.smithi033.stdout: "statfs_resend": 0, 2024-03-18T15:29:05.445 INFO:teuthology.orchestra.run.smithi033.stdout: "command_active": 0, 2024-03-18T15:29:05.445 INFO:teuthology.orchestra.run.smithi033.stdout: "command_send": 0, 2024-03-18T15:29:05.445 INFO:teuthology.orchestra.run.smithi033.stdout: "command_resend": 0, 2024-03-18T15:29:05.445 INFO:teuthology.orchestra.run.smithi033.stdout: "map_epoch": 103, 2024-03-18T15:29:05.445 INFO:teuthology.orchestra.run.smithi033.stdout: "map_full": 0, 2024-03-18T15:29:05.445 INFO:teuthology.orchestra.run.smithi033.stdout: "map_inc": 14, 2024-03-18T15:29:05.445 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_sessions": 5, 2024-03-18T15:29:05.446 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_session_open": 5, 2024-03-18T15:29:05.446 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_session_close": 0, 2024-03-18T15:29:05.446 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_laggy": 0, 2024-03-18T15:29:05.446 INFO:teuthology.orchestra.run.smithi033.stdout: "omap_wr": 0, 2024-03-18T15:29:05.446 INFO:teuthology.orchestra.run.smithi033.stdout: "omap_rd": 2, 2024-03-18T15:29:05.446 INFO:teuthology.orchestra.run.smithi033.stdout: "omap_del": 0 2024-03-18T15:29:05.446 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:05.446 INFO:teuthology.orchestra.run.smithi033.stdout: "oft": { 2024-03-18T15:29:05.446 INFO:teuthology.orchestra.run.smithi033.stdout: "omap_total_objs": 0, 2024-03-18T15:29:05.446 INFO:teuthology.orchestra.run.smithi033.stdout: "omap_total_kv_pairs": 0, 2024-03-18T15:29:05.446 INFO:teuthology.orchestra.run.smithi033.stdout: "omap_total_updates": 0, 2024-03-18T15:29:05.446 INFO:teuthology.orchestra.run.smithi033.stdout: "omap_total_removes": 0 2024-03-18T15:29:05.446 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:05.446 INFO:teuthology.orchestra.run.smithi033.stdout: "osd-slow-ops": { 2024-03-18T15:29:05.447 INFO:teuthology.orchestra.run.smithi033.stdout: "slow_ops_count": 0 2024-03-18T15:29:05.447 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:05.447 INFO:teuthology.orchestra.run.smithi033.stdout: "purge_queue": { 2024-03-18T15:29:05.447 INFO:teuthology.orchestra.run.smithi033.stdout: "pq_executing_ops": 0, 2024-03-18T15:29:05.447 INFO:teuthology.orchestra.run.smithi033.stdout: "pq_executing_ops_high_water": 0, 2024-03-18T15:29:05.447 INFO:teuthology.orchestra.run.smithi033.stdout: "pq_executing": 0, 2024-03-18T15:29:05.447 INFO:teuthology.orchestra.run.smithi033.stdout: "pq_executing_high_water": 0, 2024-03-18T15:29:05.448 INFO:teuthology.orchestra.run.smithi033.stdout: "pq_executed_ops": 0, 2024-03-18T15:29:05.448 INFO:teuthology.orchestra.run.smithi033.stdout: "pq_executed": 0, 2024-03-18T15:29:05.448 INFO:teuthology.orchestra.run.smithi033.stdout: "pq_item_in_journal": 0 2024-03-18T15:29:05.448 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:05.448 INFO:teuthology.orchestra.run.smithi033.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T15:29:05.448 INFO:teuthology.orchestra.run.smithi033.stdout: "val": 0, 2024-03-18T15:29:05.449 INFO:teuthology.orchestra.run.smithi033.stdout: "max": 104857600, 2024-03-18T15:29:05.449 INFO:teuthology.orchestra.run.smithi033.stdout: "get_started": 0, 2024-03-18T15:29:05.449 INFO:teuthology.orchestra.run.smithi033.stdout: "get": 2567, 2024-03-18T15:29:05.449 INFO:teuthology.orchestra.run.smithi033.stdout: "get_sum": 2417145, 2024-03-18T15:29:05.449 INFO:teuthology.orchestra.run.smithi033.stdout: "get_or_fail_fail": 0, 2024-03-18T15:29:05.449 INFO:teuthology.orchestra.run.smithi033.stdout: "get_or_fail_success": 2567, 2024-03-18T15:29:05.449 INFO:teuthology.orchestra.run.smithi033.stdout: "take": 0, 2024-03-18T15:29:05.449 INFO:teuthology.orchestra.run.smithi033.stdout: "take_sum": 0, 2024-03-18T15:29:05.449 INFO:teuthology.orchestra.run.smithi033.stdout: "put": 2567, 2024-03-18T15:29:05.449 INFO:teuthology.orchestra.run.smithi033.stdout: "put_sum": 2417145, 2024-03-18T15:29:05.449 INFO:teuthology.orchestra.run.smithi033.stdout: "wait": { 2024-03-18T15:29:05.449 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:29:05.449 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:29:05.449 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:29:05.450 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-18T15:29:05.450 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:05.450 INFO:teuthology.orchestra.run.smithi033.stdout: "throttle-objecter_bytes": { 2024-03-18T15:29:05.450 INFO:teuthology.orchestra.run.smithi033.stdout: "val": 0, 2024-03-18T15:29:05.451 INFO:teuthology.orchestra.run.smithi033.stdout: "max": 104857600, 2024-03-18T15:29:05.451 INFO:teuthology.orchestra.run.smithi033.stdout: "get_started": 0, 2024-03-18T15:29:05.451 INFO:teuthology.orchestra.run.smithi033.stdout: "get": 0, 2024-03-18T15:29:05.451 INFO:teuthology.orchestra.run.smithi033.stdout: "get_sum": 0, 2024-03-18T15:29:05.451 INFO:teuthology.orchestra.run.smithi033.stdout: "get_or_fail_fail": 0, 2024-03-18T15:29:05.451 INFO:teuthology.orchestra.run.smithi033.stdout: "get_or_fail_success": 0, 2024-03-18T15:29:05.451 INFO:teuthology.orchestra.run.smithi033.stdout: "take": 2296, 2024-03-18T15:29:05.451 INFO:teuthology.orchestra.run.smithi033.stdout: "take_sum": 232816, 2024-03-18T15:29:05.451 INFO:teuthology.orchestra.run.smithi033.stdout: "put": 2296, 2024-03-18T15:29:05.451 INFO:teuthology.orchestra.run.smithi033.stdout: "put_sum": 232816, 2024-03-18T15:29:05.451 INFO:teuthology.orchestra.run.smithi033.stdout: "wait": { 2024-03-18T15:29:05.451 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:29:05.451 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:29:05.451 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:29:05.452 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-18T15:29:05.452 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:05.452 INFO:teuthology.orchestra.run.smithi033.stdout: "throttle-objecter_ops": { 2024-03-18T15:29:05.452 INFO:teuthology.orchestra.run.smithi033.stdout: "val": 0, 2024-03-18T15:29:05.452 INFO:teuthology.orchestra.run.smithi033.stdout: "max": 1024, 2024-03-18T15:29:05.452 INFO:teuthology.orchestra.run.smithi033.stdout: "get_started": 0, 2024-03-18T15:29:05.452 INFO:teuthology.orchestra.run.smithi033.stdout: "get": 0, 2024-03-18T15:29:05.452 INFO:teuthology.orchestra.run.smithi033.stdout: "get_sum": 0, 2024-03-18T15:29:05.452 INFO:teuthology.orchestra.run.smithi033.stdout: "get_or_fail_fail": 0, 2024-03-18T15:29:05.452 INFO:teuthology.orchestra.run.smithi033.stdout: "get_or_fail_success": 0, 2024-03-18T15:29:05.452 INFO:teuthology.orchestra.run.smithi033.stdout: "take": 2296, 2024-03-18T15:29:05.452 INFO:teuthology.orchestra.run.smithi033.stdout: "take_sum": 2296, 2024-03-18T15:29:05.452 INFO:teuthology.orchestra.run.smithi033.stdout: "put": 2296, 2024-03-18T15:29:05.452 INFO:teuthology.orchestra.run.smithi033.stdout: "put_sum": 2296, 2024-03-18T15:29:05.453 INFO:teuthology.orchestra.run.smithi033.stdout: "wait": { 2024-03-18T15:29:05.453 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:29:05.453 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:29:05.453 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:29:05.453 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-18T15:29:05.453 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:05.453 INFO:teuthology.orchestra.run.smithi033.stdout: "throttle-write_buf_throttle": { 2024-03-18T15:29:05.453 INFO:teuthology.orchestra.run.smithi033.stdout: "val": 0, 2024-03-18T15:29:05.453 INFO:teuthology.orchestra.run.smithi033.stdout: "max": 3758096384, 2024-03-18T15:29:05.453 INFO:teuthology.orchestra.run.smithi033.stdout: "get_started": 0, 2024-03-18T15:29:05.453 INFO:teuthology.orchestra.run.smithi033.stdout: "get": 0, 2024-03-18T15:29:05.453 INFO:teuthology.orchestra.run.smithi033.stdout: "get_sum": 0, 2024-03-18T15:29:05.453 INFO:teuthology.orchestra.run.smithi033.stdout: "get_or_fail_fail": 0, 2024-03-18T15:29:05.454 INFO:teuthology.orchestra.run.smithi033.stdout: "get_or_fail_success": 0, 2024-03-18T15:29:05.454 INFO:teuthology.orchestra.run.smithi033.stdout: "take": 0, 2024-03-18T15:29:05.454 INFO:teuthology.orchestra.run.smithi033.stdout: "take_sum": 0, 2024-03-18T15:29:05.454 INFO:teuthology.orchestra.run.smithi033.stdout: "put": 0, 2024-03-18T15:29:05.454 INFO:teuthology.orchestra.run.smithi033.stdout: "put_sum": 0, 2024-03-18T15:29:05.454 INFO:teuthology.orchestra.run.smithi033.stdout: "wait": { 2024-03-18T15:29:05.454 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:29:05.454 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:29:05.454 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:29:05.454 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-18T15:29:05.454 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-18T15:29:05.454 INFO:teuthology.orchestra.run.smithi033.stdout: "throttle-write_buf_throttle-0x55b66cec26e0": { 2024-03-18T15:29:05.454 INFO:teuthology.orchestra.run.smithi033.stdout: "val": 0, 2024-03-18T15:29:05.455 INFO:teuthology.orchestra.run.smithi033.stdout: "max": 3758096384, 2024-03-18T15:29:05.455 INFO:teuthology.orchestra.run.smithi033.stdout: "get_started": 0, 2024-03-18T15:29:05.455 INFO:teuthology.orchestra.run.smithi033.stdout: "get": 0, 2024-03-18T15:29:05.455 INFO:teuthology.orchestra.run.smithi033.stdout: "get_sum": 0, 2024-03-18T15:29:05.455 INFO:teuthology.orchestra.run.smithi033.stdout: "get_or_fail_fail": 0, 2024-03-18T15:29:05.455 INFO:teuthology.orchestra.run.smithi033.stdout: "get_or_fail_success": 0, 2024-03-18T15:29:05.455 INFO:teuthology.orchestra.run.smithi033.stdout: "take": 0, 2024-03-18T15:29:05.455 INFO:teuthology.orchestra.run.smithi033.stdout: "take_sum": 0, 2024-03-18T15:29:05.455 INFO:teuthology.orchestra.run.smithi033.stdout: "put": 0, 2024-03-18T15:29:05.455 INFO:teuthology.orchestra.run.smithi033.stdout: "put_sum": 0, 2024-03-18T15:29:05.455 INFO:teuthology.orchestra.run.smithi033.stdout: "wait": { 2024-03-18T15:29:05.456 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-18T15:29:05.456 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-18T15:29:05.456 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-18T15:29:05.456 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-18T15:29:05.456 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-18T15:29:05.456 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-03-18T15:29:05.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:29:05 smithi104 ceph-mon[36070]: pgmap v329: 129 pgs: 129 active+clean; 1.8 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-18T15:29:05.957 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:29:05 smithi033 ceph-mon[36648]: pgmap v329: 129 pgs: 129 active+clean; 1.8 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-18T15:29:05.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:29:05 smithi019 ceph-mon[31522]: pgmap v329: 129 pgs: 129 active+clean; 1.8 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-18T15:29:06.148 INFO:tasks.check_counter:Daemon mds.k mds.exported=0 2024-03-18T15:29:06.149 INFO:tasks.check_counter:Daemon mds.k mds.imported=0 2024-03-18T15:29:06.149 INFO:tasks.check_counter:Daemon mds.k mds_cache.dir_update=0 2024-03-18T15:29:06.149 INFO:tasks.check_counter:Daemon mds.k mds_cache.dir_update_receipt=0 2024-03-18T15:29:06.149 INFO:tasks.check_counter:Daemon mds.k mds.root_rsnaps=0 2024-03-18T15:29:06.149 INFO:tasks.check_counter:Daemon mds.k mds_server.req_mksnap_latency.avgcount=0 2024-03-18T15:29:06.149 INFO:tasks.check_counter:Daemon mds.k mds_server.req_rmsnap_latency.avgcount=0 2024-03-18T15:29:06.149 DEBUG:tasks.check_counter:Getting stats from c 2024-03-18T15:29:06.150 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 -- ceph daemon mds.c perf dump 2024-03-18T15:29:06.482 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T15:29:07.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:29:07 smithi104 ceph-mon[36070]: pgmap v330: 129 pgs: 129 active+clean; 1.8 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-18T15:29:07.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:29:07 smithi019 ceph-mon[31522]: pgmap v330: 129 pgs: 129 active+clean; 1.8 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-18T15:29:08.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:29:07 smithi033 ceph-mon[36648]: pgmap v330: 129 pgs: 129 active+clean; 1.8 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-18T15:29:08.438 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-03-18T15:29:08.438 INFO:teuthology.orchestra.run.smithi104.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T15:29:08.438 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_recv_messages": 151, 2024-03-18T15:29:08.438 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_send_messages": 293, 2024-03-18T15:29:08.439 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_recv_bytes": 49262, 2024-03-18T15:29:08.439 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_send_bytes": 666218, 2024-03-18T15:29:08.439 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_created_connections": 15, 2024-03-18T15:29:08.439 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_active_connections": 6, 2024-03-18T15:29:08.439 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_running_total_time": 0.022074918, 2024-03-18T15:29:08.439 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_running_send_time": 0.008755690, 2024-03-18T15:29:08.439 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_running_recv_time": 0.007257162, 2024-03-18T15:29:08.439 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_running_fast_dispatch_time": 0.003452325, 2024-03-18T15:29:08.439 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T15:29:08.439 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 291, 2024-03-18T15:29:08.439 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.052908051, 2024-03-18T15:29:08.439 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000181814 2024-03-18T15:29:08.439 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:08.440 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_handle_ack_lat": { 2024-03-18T15:29:08.440 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 36, 2024-03-18T15:29:08.440 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000014687, 2024-03-18T15:29:08.440 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000407 2024-03-18T15:29:08.440 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:08.440 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-18T15:29:08.440 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_send_encrypted_bytes": 316944 2024-03-18T15:29:08.440 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:08.440 INFO:teuthology.orchestra.run.smithi104.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T15:29:08.440 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_recv_messages": 471, 2024-03-18T15:29:08.440 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_send_messages": 499, 2024-03-18T15:29:08.440 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_recv_bytes": 1863450, 2024-03-18T15:29:08.440 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_send_bytes": 250216, 2024-03-18T15:29:08.440 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_created_connections": 8, 2024-03-18T15:29:08.441 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_active_connections": 7, 2024-03-18T15:29:08.441 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_running_total_time": 0.056712400, 2024-03-18T15:29:08.441 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_running_send_time": 0.015425816, 2024-03-18T15:29:08.441 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_running_recv_time": 0.081789775, 2024-03-18T15:29:08.441 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_running_fast_dispatch_time": 0.005613245, 2024-03-18T15:29:08.441 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T15:29:08.441 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 498, 2024-03-18T15:29:08.441 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.032246343, 2024-03-18T15:29:08.441 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000064751 2024-03-18T15:29:08.441 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:08.441 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_handle_ack_lat": { 2024-03-18T15:29:08.441 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 251, 2024-03-18T15:29:08.441 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000049245, 2024-03-18T15:29:08.441 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000196 2024-03-18T15:29:08.442 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:08.442 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_recv_encrypted_bytes": 1798256, 2024-03-18T15:29:08.442 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_send_encrypted_bytes": 150416 2024-03-18T15:29:08.442 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:08.442 INFO:teuthology.orchestra.run.smithi104.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T15:29:08.442 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_recv_messages": 403, 2024-03-18T15:29:08.442 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_send_messages": 661, 2024-03-18T15:29:08.442 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_recv_bytes": 166626, 2024-03-18T15:29:08.442 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_send_bytes": 415634, 2024-03-18T15:29:08.442 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_created_connections": 11, 2024-03-18T15:29:08.442 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_active_connections": 8, 2024-03-18T15:29:08.443 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_running_total_time": 0.041006301, 2024-03-18T15:29:08.443 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_running_send_time": 0.016325017, 2024-03-18T15:29:08.443 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_running_recv_time": 0.019556231, 2024-03-18T15:29:08.443 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_running_fast_dispatch_time": 0.003629294, 2024-03-18T15:29:08.443 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T15:29:08.443 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 659, 2024-03-18T15:29:08.443 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.039460433, 2024-03-18T15:29:08.444 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000059879 2024-03-18T15:29:08.444 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:08.444 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_handle_ack_lat": { 2024-03-18T15:29:08.444 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 920, 2024-03-18T15:29:08.444 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000144175, 2024-03-18T15:29:08.444 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000156 2024-03-18T15:29:08.444 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:08.444 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_recv_encrypted_bytes": 4160, 2024-03-18T15:29:08.444 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_send_encrypted_bytes": 146656 2024-03-18T15:29:08.444 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:08.444 INFO:teuthology.orchestra.run.smithi104.stdout: "cct": { 2024-03-18T15:29:08.444 INFO:teuthology.orchestra.run.smithi104.stdout: "total_workers": 1, 2024-03-18T15:29:08.444 INFO:teuthology.orchestra.run.smithi104.stdout: "unhealthy_workers": 0 2024-03-18T15:29:08.445 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:08.445 INFO:teuthology.orchestra.run.smithi104.stdout: "finisher-MDSRank": { 2024-03-18T15:29:08.445 INFO:teuthology.orchestra.run.smithi104.stdout: "queue_len": 0, 2024-03-18T15:29:08.445 INFO:teuthology.orchestra.run.smithi104.stdout: "complete_latency": { 2024-03-18T15:29:08.445 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 223, 2024-03-18T15:29:08.445 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.036449896, 2024-03-18T15:29:08.445 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000163452 2024-03-18T15:29:08.445 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-03-18T15:29:08.445 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:08.445 INFO:teuthology.orchestra.run.smithi104.stdout: "finisher-PurgeQueue": { 2024-03-18T15:29:08.445 INFO:teuthology.orchestra.run.smithi104.stdout: "queue_len": 0, 2024-03-18T15:29:08.446 INFO:teuthology.orchestra.run.smithi104.stdout: "complete_latency": { 2024-03-18T15:29:08.446 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 86, 2024-03-18T15:29:08.446 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.002726490, 2024-03-18T15:29:08.446 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000031703 2024-03-18T15:29:08.446 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-03-18T15:29:08.446 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:08.446 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": { 2024-03-18T15:29:08.446 INFO:teuthology.orchestra.run.smithi104.stdout: "request": 8, 2024-03-18T15:29:08.446 INFO:teuthology.orchestra.run.smithi104.stdout: "reply": 8, 2024-03-18T15:29:08.446 INFO:teuthology.orchestra.run.smithi104.stdout: "reply_latency": { 2024-03-18T15:29:08.447 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 8, 2024-03-18T15:29:08.447 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.021126927, 2024-03-18T15:29:08.447 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.002640865 2024-03-18T15:29:08.447 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:08.447 INFO:teuthology.orchestra.run.smithi104.stdout: "slow_reply": 0, 2024-03-18T15:29:08.447 INFO:teuthology.orchestra.run.smithi104.stdout: "forward": 0, 2024-03-18T15:29:08.447 INFO:teuthology.orchestra.run.smithi104.stdout: "dir_fetch_complete": 0, 2024-03-18T15:29:08.447 INFO:teuthology.orchestra.run.smithi104.stdout: "dir_fetch_keys": 0, 2024-03-18T15:29:08.447 INFO:teuthology.orchestra.run.smithi104.stdout: "dir_commit": 11, 2024-03-18T15:29:08.447 INFO:teuthology.orchestra.run.smithi104.stdout: "dir_split": 0, 2024-03-18T15:29:08.447 INFO:teuthology.orchestra.run.smithi104.stdout: "dir_merge": 0, 2024-03-18T15:29:08.447 INFO:teuthology.orchestra.run.smithi104.stdout: "inodes": 24, 2024-03-18T15:29:08.447 INFO:teuthology.orchestra.run.smithi104.stdout: "inodes_top": 0, 2024-03-18T15:29:08.447 INFO:teuthology.orchestra.run.smithi104.stdout: "inodes_bottom": 13, 2024-03-18T15:29:08.448 INFO:teuthology.orchestra.run.smithi104.stdout: "inodes_pin_tail": 11, 2024-03-18T15:29:08.448 INFO:teuthology.orchestra.run.smithi104.stdout: "inodes_pinned": 22, 2024-03-18T15:29:08.448 INFO:teuthology.orchestra.run.smithi104.stdout: "inodes_expired": 5, 2024-03-18T15:29:08.448 INFO:teuthology.orchestra.run.smithi104.stdout: "inodes_with_caps": 0, 2024-03-18T15:29:08.448 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": 0, 2024-03-18T15:29:08.448 INFO:teuthology.orchestra.run.smithi104.stdout: "subtrees": 3, 2024-03-18T15:29:08.448 INFO:teuthology.orchestra.run.smithi104.stdout: "traverse": 18, 2024-03-18T15:29:08.448 INFO:teuthology.orchestra.run.smithi104.stdout: "traverse_hit": 13, 2024-03-18T15:29:08.448 INFO:teuthology.orchestra.run.smithi104.stdout: "traverse_forward": 0, 2024-03-18T15:29:08.448 INFO:teuthology.orchestra.run.smithi104.stdout: "traverse_discover": 3, 2024-03-18T15:29:08.448 INFO:teuthology.orchestra.run.smithi104.stdout: "traverse_dir_fetch": 0, 2024-03-18T15:29:08.448 INFO:teuthology.orchestra.run.smithi104.stdout: "traverse_remote_ino": 0, 2024-03-18T15:29:08.448 INFO:teuthology.orchestra.run.smithi104.stdout: "traverse_lock": 0, 2024-03-18T15:29:08.449 INFO:teuthology.orchestra.run.smithi104.stdout: "load_cent": 0, 2024-03-18T15:29:08.449 INFO:teuthology.orchestra.run.smithi104.stdout: "q": 0, 2024-03-18T15:29:08.449 INFO:teuthology.orchestra.run.smithi104.stdout: "exported": 1, 2024-03-18T15:29:08.449 INFO:teuthology.orchestra.run.smithi104.stdout: "exported_inodes": 2, 2024-03-18T15:29:08.449 INFO:teuthology.orchestra.run.smithi104.stdout: "imported": 1, 2024-03-18T15:29:08.449 INFO:teuthology.orchestra.run.smithi104.stdout: "imported_inodes": 5, 2024-03-18T15:29:08.449 INFO:teuthology.orchestra.run.smithi104.stdout: "openino_dir_fetch": 0, 2024-03-18T15:29:08.449 INFO:teuthology.orchestra.run.smithi104.stdout: "openino_backtrace_fetch": 0, 2024-03-18T15:29:08.449 INFO:teuthology.orchestra.run.smithi104.stdout: "openino_peer_discover": 0, 2024-03-18T15:29:08.449 INFO:teuthology.orchestra.run.smithi104.stdout: "root_rfiles": 4, 2024-03-18T15:29:08.449 INFO:teuthology.orchestra.run.smithi104.stdout: "root_rbytes": 234, 2024-03-18T15:29:08.449 INFO:teuthology.orchestra.run.smithi104.stdout: "root_rsnaps": 0, 2024-03-18T15:29:08.449 INFO:teuthology.orchestra.run.smithi104.stdout: "scrub_backtrace_fetch": 0, 2024-03-18T15:29:08.449 INFO:teuthology.orchestra.run.smithi104.stdout: "scrub_set_tag": 0, 2024-03-18T15:29:08.450 INFO:teuthology.orchestra.run.smithi104.stdout: "scrub_backtrace_repaired": 0, 2024-03-18T15:29:08.450 INFO:teuthology.orchestra.run.smithi104.stdout: "scrub_inotable_repaired": 0, 2024-03-18T15:29:08.450 INFO:teuthology.orchestra.run.smithi104.stdout: "scrub_dir_inodes": 0, 2024-03-18T15:29:08.450 INFO:teuthology.orchestra.run.smithi104.stdout: "scrub_dir_base_inodes": 0, 2024-03-18T15:29:08.450 INFO:teuthology.orchestra.run.smithi104.stdout: "scrub_dirfrag_rstats": 0, 2024-03-18T15:29:08.450 INFO:teuthology.orchestra.run.smithi104.stdout: "scrub_file_inodes": 0, 2024-03-18T15:29:08.450 INFO:teuthology.orchestra.run.smithi104.stdout: "handle_inode_file_caps": 0, 2024-03-18T15:29:08.450 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph_cap_op_revoke": 8, 2024-03-18T15:29:08.450 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph_cap_op_grant": 13, 2024-03-18T15:29:08.450 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph_cap_op_trunc": 0, 2024-03-18T15:29:08.450 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-18T15:29:08.450 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-18T15:29:08.450 INFO:teuthology.orchestra.run.smithi104.stdout: "handle_client_caps": 85, 2024-03-18T15:29:08.451 INFO:teuthology.orchestra.run.smithi104.stdout: "handle_client_caps_dirty": 76, 2024-03-18T15:29:08.451 INFO:teuthology.orchestra.run.smithi104.stdout: "handle_client_cap_release": 3, 2024-03-18T15:29:08.451 INFO:teuthology.orchestra.run.smithi104.stdout: "process_request_cap_release": 7 2024-03-18T15:29:08.451 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:08.451 INFO:teuthology.orchestra.run.smithi104.stdout: "mds_cache": { 2024-03-18T15:29:08.451 INFO:teuthology.orchestra.run.smithi104.stdout: "dir_update": 4, 2024-03-18T15:29:08.451 INFO:teuthology.orchestra.run.smithi104.stdout: "dir_update_receipt": 6, 2024-03-18T15:29:08.451 INFO:teuthology.orchestra.run.smithi104.stdout: "dir_try_discover": 4, 2024-03-18T15:29:08.451 INFO:teuthology.orchestra.run.smithi104.stdout: "dir_send_discover": 6, 2024-03-18T15:29:08.451 INFO:teuthology.orchestra.run.smithi104.stdout: "dir_handle_discover": 11, 2024-03-18T15:29:08.451 INFO:teuthology.orchestra.run.smithi104.stdout: "num_strays": 1, 2024-03-18T15:29:08.451 INFO:teuthology.orchestra.run.smithi104.stdout: "num_strays_delayed": 0, 2024-03-18T15:29:08.451 INFO:teuthology.orchestra.run.smithi104.stdout: "num_strays_enqueuing": 0, 2024-03-18T15:29:08.451 INFO:teuthology.orchestra.run.smithi104.stdout: "strays_created": 5, 2024-03-18T15:29:08.452 INFO:teuthology.orchestra.run.smithi104.stdout: "strays_enqueued": 4, 2024-03-18T15:29:08.452 INFO:teuthology.orchestra.run.smithi104.stdout: "strays_reintegrated": 0, 2024-03-18T15:29:08.452 INFO:teuthology.orchestra.run.smithi104.stdout: "strays_migrated": 0, 2024-03-18T15:29:08.452 INFO:teuthology.orchestra.run.smithi104.stdout: "num_recovering_processing": 0, 2024-03-18T15:29:08.452 INFO:teuthology.orchestra.run.smithi104.stdout: "num_recovering_enqueued": 0, 2024-03-18T15:29:08.452 INFO:teuthology.orchestra.run.smithi104.stdout: "num_recovering_prioritized": 0, 2024-03-18T15:29:08.452 INFO:teuthology.orchestra.run.smithi104.stdout: "recovery_started": 0, 2024-03-18T15:29:08.452 INFO:teuthology.orchestra.run.smithi104.stdout: "recovery_completed": 0, 2024-03-18T15:29:08.452 INFO:teuthology.orchestra.run.smithi104.stdout: "ireq_quiesce_path": 0, 2024-03-18T15:29:08.452 INFO:teuthology.orchestra.run.smithi104.stdout: "ireq_quiesce_inode": 0, 2024-03-18T15:29:08.452 INFO:teuthology.orchestra.run.smithi104.stdout: "ireq_enqueue_scrub": 0, 2024-03-18T15:29:08.452 INFO:teuthology.orchestra.run.smithi104.stdout: "ireq_exportdir": 1, 2024-03-18T15:29:08.452 INFO:teuthology.orchestra.run.smithi104.stdout: "ireq_flush": 0, 2024-03-18T15:29:08.452 INFO:teuthology.orchestra.run.smithi104.stdout: "ireq_fragmentdir": 0, 2024-03-18T15:29:08.453 INFO:teuthology.orchestra.run.smithi104.stdout: "ireq_fragstats": 0, 2024-03-18T15:29:08.453 INFO:teuthology.orchestra.run.smithi104.stdout: "ireq_inodestats": 0 2024-03-18T15:29:08.453 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:08.453 INFO:teuthology.orchestra.run.smithi104.stdout: "mds_log": { 2024-03-18T15:29:08.453 INFO:teuthology.orchestra.run.smithi104.stdout: "evlrg": 0, 2024-03-18T15:29:08.453 INFO:teuthology.orchestra.run.smithi104.stdout: "evadd": 106, 2024-03-18T15:29:08.453 INFO:teuthology.orchestra.run.smithi104.stdout: "evex": 0, 2024-03-18T15:29:08.453 INFO:teuthology.orchestra.run.smithi104.stdout: "evtrm": 0, 2024-03-18T15:29:08.453 INFO:teuthology.orchestra.run.smithi104.stdout: "ev": 106, 2024-03-18T15:29:08.453 INFO:teuthology.orchestra.run.smithi104.stdout: "evexg": 0, 2024-03-18T15:29:08.453 INFO:teuthology.orchestra.run.smithi104.stdout: "evexd": 0, 2024-03-18T15:29:08.453 INFO:teuthology.orchestra.run.smithi104.stdout: "segadd": 2, 2024-03-18T15:29:08.453 INFO:teuthology.orchestra.run.smithi104.stdout: "segex": 0, 2024-03-18T15:29:08.453 INFO:teuthology.orchestra.run.smithi104.stdout: "segtrm": 0, 2024-03-18T15:29:08.454 INFO:teuthology.orchestra.run.smithi104.stdout: "seg": 2, 2024-03-18T15:29:08.454 INFO:teuthology.orchestra.run.smithi104.stdout: "segmjr": 2, 2024-03-18T15:29:08.454 INFO:teuthology.orchestra.run.smithi104.stdout: "segexg": 0, 2024-03-18T15:29:08.454 INFO:teuthology.orchestra.run.smithi104.stdout: "segexd": 0, 2024-03-18T15:29:08.454 INFO:teuthology.orchestra.run.smithi104.stdout: "expos": 4194304, 2024-03-18T15:29:08.454 INFO:teuthology.orchestra.run.smithi104.stdout: "wrpos": 4427115, 2024-03-18T15:29:08.454 INFO:teuthology.orchestra.run.smithi104.stdout: "rdpos": 0, 2024-03-18T15:29:08.454 INFO:teuthology.orchestra.run.smithi104.stdout: "jlat": { 2024-03-18T15:29:08.454 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 83, 2024-03-18T15:29:08.454 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.150884401, 2024-03-18T15:29:08.454 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.001817884 2024-03-18T15:29:08.454 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:08.454 INFO:teuthology.orchestra.run.smithi104.stdout: "replayed": 0 2024-03-18T15:29:08.455 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:08.455 INFO:teuthology.orchestra.run.smithi104.stdout: "mds_mem": { 2024-03-18T15:29:08.455 INFO:teuthology.orchestra.run.smithi104.stdout: "ino": 24, 2024-03-18T15:29:08.455 INFO:teuthology.orchestra.run.smithi104.stdout: "ino+": 30, 2024-03-18T15:29:08.455 INFO:teuthology.orchestra.run.smithi104.stdout: "ino-": 6, 2024-03-18T15:29:08.455 INFO:teuthology.orchestra.run.smithi104.stdout: "dir": 19, 2024-03-18T15:29:08.455 INFO:teuthology.orchestra.run.smithi104.stdout: "dir+": 22, 2024-03-18T15:29:08.455 INFO:teuthology.orchestra.run.smithi104.stdout: "dir-": 3, 2024-03-18T15:29:08.455 INFO:teuthology.orchestra.run.smithi104.stdout: "dn": 24, 2024-03-18T15:29:08.455 INFO:teuthology.orchestra.run.smithi104.stdout: "dn+": 33, 2024-03-18T15:29:08.455 INFO:teuthology.orchestra.run.smithi104.stdout: "dn-": 9, 2024-03-18T15:29:08.455 INFO:teuthology.orchestra.run.smithi104.stdout: "cap": 0, 2024-03-18T15:29:08.455 INFO:teuthology.orchestra.run.smithi104.stdout: "cap+": 6, 2024-03-18T15:29:08.455 INFO:teuthology.orchestra.run.smithi104.stdout: "cap-": 6, 2024-03-18T15:29:08.456 INFO:teuthology.orchestra.run.smithi104.stdout: "rss": 52000, 2024-03-18T15:29:08.456 INFO:teuthology.orchestra.run.smithi104.stdout: "heap": 223612 2024-03-18T15:29:08.456 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:08.456 INFO:teuthology.orchestra.run.smithi104.stdout: "mds_server": { 2024-03-18T15:29:08.456 INFO:teuthology.orchestra.run.smithi104.stdout: "dispatch_client_request": 14, 2024-03-18T15:29:08.456 INFO:teuthology.orchestra.run.smithi104.stdout: "dispatch_server_request": 1, 2024-03-18T15:29:08.456 INFO:teuthology.orchestra.run.smithi104.stdout: "handle_client_request": 8, 2024-03-18T15:29:08.456 INFO:teuthology.orchestra.run.smithi104.stdout: "handle_client_session": 13, 2024-03-18T15:29:08.456 INFO:teuthology.orchestra.run.smithi104.stdout: "handle_peer_request": 4, 2024-03-18T15:29:08.456 INFO:teuthology.orchestra.run.smithi104.stdout: "req_create_latency": { 2024-03-18T15:29:08.456 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:08.456 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:08.456 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:08.457 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:08.457 INFO:teuthology.orchestra.run.smithi104.stdout: "req_getattr_latency": { 2024-03-18T15:29:08.457 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 1, 2024-03-18T15:29:08.457 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000805209, 2024-03-18T15:29:08.457 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000805209 2024-03-18T15:29:08.457 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:08.457 INFO:teuthology.orchestra.run.smithi104.stdout: "req_getfilelock_latency": { 2024-03-18T15:29:08.457 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:08.457 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:08.457 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:08.457 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:08.457 INFO:teuthology.orchestra.run.smithi104.stdout: "req_link_latency": { 2024-03-18T15:29:08.457 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:08.457 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:08.458 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:08.458 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:08.458 INFO:teuthology.orchestra.run.smithi104.stdout: "req_lookup_latency": { 2024-03-18T15:29:08.458 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 1, 2024-03-18T15:29:08.458 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000373382, 2024-03-18T15:29:08.458 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000373382 2024-03-18T15:29:08.458 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:08.458 INFO:teuthology.orchestra.run.smithi104.stdout: "req_lookuphash_latency": { 2024-03-18T15:29:08.458 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:08.458 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:08.458 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:08.458 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:08.458 INFO:teuthology.orchestra.run.smithi104.stdout: "req_lookupino_latency": { 2024-03-18T15:29:08.458 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:08.459 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:08.459 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:08.459 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:08.459 INFO:teuthology.orchestra.run.smithi104.stdout: "req_lookupname_latency": { 2024-03-18T15:29:08.459 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:08.459 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:08.459 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:08.459 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:08.459 INFO:teuthology.orchestra.run.smithi104.stdout: "req_lookupparent_latency": { 2024-03-18T15:29:08.459 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:08.459 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:08.459 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:08.459 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:08.459 INFO:teuthology.orchestra.run.smithi104.stdout: "req_lookupsnap_latency": { 2024-03-18T15:29:08.460 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:08.460 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:08.460 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:08.460 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:08.460 INFO:teuthology.orchestra.run.smithi104.stdout: "req_lssnap_latency": { 2024-03-18T15:29:08.460 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:08.460 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:08.460 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:08.460 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:08.460 INFO:teuthology.orchestra.run.smithi104.stdout: "req_mkdir_latency": { 2024-03-18T15:29:08.461 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:08.461 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:08.461 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:08.461 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:08.461 INFO:teuthology.orchestra.run.smithi104.stdout: "req_mknod_latency": { 2024-03-18T15:29:08.461 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:08.461 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:08.461 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:08.462 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:08.462 INFO:teuthology.orchestra.run.smithi104.stdout: "req_mksnap_latency": { 2024-03-18T15:29:08.462 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:08.462 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:08.462 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:08.462 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:08.462 INFO:teuthology.orchestra.run.smithi104.stdout: "req_open_latency": { 2024-03-18T15:29:08.462 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:08.462 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:08.462 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:08.462 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:08.462 INFO:teuthology.orchestra.run.smithi104.stdout: "req_readdir_latency": { 2024-03-18T15:29:08.462 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 1, 2024-03-18T15:29:08.462 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000579290, 2024-03-18T15:29:08.463 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000579290 2024-03-18T15:29:08.463 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:08.463 INFO:teuthology.orchestra.run.smithi104.stdout: "req_rename_latency": { 2024-03-18T15:29:08.463 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:08.463 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:08.463 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:08.463 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:08.463 INFO:teuthology.orchestra.run.smithi104.stdout: "req_renamesnap_latency": { 2024-03-18T15:29:08.463 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:08.463 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:08.463 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:08.463 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:08.463 INFO:teuthology.orchestra.run.smithi104.stdout: "req_snapdiff_latency": { 2024-03-18T15:29:08.464 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:08.464 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:08.464 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:08.464 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:08.464 INFO:teuthology.orchestra.run.smithi104.stdout: "req_rmdir_latency": { 2024-03-18T15:29:08.464 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 1, 2024-03-18T15:29:08.464 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.011155193, 2024-03-18T15:29:08.464 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.011155193 2024-03-18T15:29:08.464 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:08.464 INFO:teuthology.orchestra.run.smithi104.stdout: "req_rmsnap_latency": { 2024-03-18T15:29:08.464 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:08.464 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:08.464 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:08.465 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:08.465 INFO:teuthology.orchestra.run.smithi104.stdout: "req_rmxattr_latency": { 2024-03-18T15:29:08.465 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:08.465 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:08.465 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:08.465 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:08.465 INFO:teuthology.orchestra.run.smithi104.stdout: "req_setattr_latency": { 2024-03-18T15:29:08.465 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:08.465 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:08.465 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:08.465 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:08.465 INFO:teuthology.orchestra.run.smithi104.stdout: "req_setdirlayout_latency": { 2024-03-18T15:29:08.465 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:08.465 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:08.466 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:08.466 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:08.466 INFO:teuthology.orchestra.run.smithi104.stdout: "req_setfilelock_latency": { 2024-03-18T15:29:08.466 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:08.466 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:08.466 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:08.466 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:08.466 INFO:teuthology.orchestra.run.smithi104.stdout: "req_setlayout_latency": { 2024-03-18T15:29:08.467 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:08.467 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:08.467 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:08.467 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:08.467 INFO:teuthology.orchestra.run.smithi104.stdout: "req_setxattr_latency": { 2024-03-18T15:29:08.467 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:08.467 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:08.467 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:08.467 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:08.467 INFO:teuthology.orchestra.run.smithi104.stdout: "req_symlink_latency": { 2024-03-18T15:29:08.467 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:08.467 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:08.467 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:08.468 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:08.468 INFO:teuthology.orchestra.run.smithi104.stdout: "req_unlink_latency": { 2024-03-18T15:29:08.468 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 4, 2024-03-18T15:29:08.468 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.008213853, 2024-03-18T15:29:08.468 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.002053463 2024-03-18T15:29:08.468 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:08.468 INFO:teuthology.orchestra.run.smithi104.stdout: "cap_revoke_eviction": 0, 2024-03-18T15:29:08.468 INFO:teuthology.orchestra.run.smithi104.stdout: "cap_acquisition_throttle": 0, 2024-03-18T15:29:08.468 INFO:teuthology.orchestra.run.smithi104.stdout: "req_getvxattr_latency": { 2024-03-18T15:29:08.468 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:08.468 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:08.468 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:08.468 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-03-18T15:29:08.468 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:08.468 INFO:teuthology.orchestra.run.smithi104.stdout: "mds_sessions": { 2024-03-18T15:29:08.469 INFO:teuthology.orchestra.run.smithi104.stdout: "session_count": 1, 2024-03-18T15:29:08.469 INFO:teuthology.orchestra.run.smithi104.stdout: "session_add": 2, 2024-03-18T15:29:08.469 INFO:teuthology.orchestra.run.smithi104.stdout: "session_remove": 1, 2024-03-18T15:29:08.469 INFO:teuthology.orchestra.run.smithi104.stdout: "sessions_open": 1, 2024-03-18T15:29:08.469 INFO:teuthology.orchestra.run.smithi104.stdout: "sessions_stale": 0, 2024-03-18T15:29:08.469 INFO:teuthology.orchestra.run.smithi104.stdout: "total_load": 7, 2024-03-18T15:29:08.469 INFO:teuthology.orchestra.run.smithi104.stdout: "average_load": 7, 2024-03-18T15:29:08.469 INFO:teuthology.orchestra.run.smithi104.stdout: "avg_session_uptime": 637, 2024-03-18T15:29:08.469 INFO:teuthology.orchestra.run.smithi104.stdout: "mdthresh_evicted": 0 2024-03-18T15:29:08.469 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:08.469 INFO:teuthology.orchestra.run.smithi104.stdout: "mempool": { 2024-03-18T15:29:08.469 INFO:teuthology.orchestra.run.smithi104.stdout: "bloom_filter_bytes": 0, 2024-03-18T15:29:08.469 INFO:teuthology.orchestra.run.smithi104.stdout: "bloom_filter_items": 0, 2024-03-18T15:29:08.469 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T15:29:08.470 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_alloc_items": 0, 2024-03-18T15:29:08.470 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T15:29:08.470 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_cache_data_items": 0, 2024-03-18T15:29:08.470 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T15:29:08.470 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T15:29:08.470 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T15:29:08.470 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T15:29:08.470 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T15:29:08.470 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_cache_other_items": 0, 2024-03-18T15:29:08.470 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T15:29:08.470 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T15:29:08.470 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_extent_bytes": 0, 2024-03-18T15:29:08.470 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_extent_items": 0, 2024-03-18T15:29:08.471 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_blob_bytes": 0, 2024-03-18T15:29:08.471 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_blob_items": 0, 2024-03-18T15:29:08.471 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T15:29:08.471 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T15:29:08.471 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T15:29:08.471 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T15:29:08.471 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T15:29:08.471 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_fsck_items": 0, 2024-03-18T15:29:08.471 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_txc_bytes": 0, 2024-03-18T15:29:08.471 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_txc_items": 0, 2024-03-18T15:29:08.471 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T15:29:08.471 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T15:29:08.471 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_writing_bytes": 0, 2024-03-18T15:29:08.471 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_writing_items": 0, 2024-03-18T15:29:08.471 INFO:teuthology.orchestra.run.smithi104.stdout: "bluefs_bytes": 0, 2024-03-18T15:29:08.472 INFO:teuthology.orchestra.run.smithi104.stdout: "bluefs_items": 0, 2024-03-18T15:29:08.472 INFO:teuthology.orchestra.run.smithi104.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T15:29:08.472 INFO:teuthology.orchestra.run.smithi104.stdout: "bluefs_file_reader_items": 0, 2024-03-18T15:29:08.472 INFO:teuthology.orchestra.run.smithi104.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T15:29:08.472 INFO:teuthology.orchestra.run.smithi104.stdout: "bluefs_file_writer_items": 0, 2024-03-18T15:29:08.472 INFO:teuthology.orchestra.run.smithi104.stdout: "buffer_anon_bytes": 329779, 2024-03-18T15:29:08.472 INFO:teuthology.orchestra.run.smithi104.stdout: "buffer_anon_items": 131, 2024-03-18T15:29:08.472 INFO:teuthology.orchestra.run.smithi104.stdout: "buffer_meta_bytes": 0, 2024-03-18T15:29:08.472 INFO:teuthology.orchestra.run.smithi104.stdout: "buffer_meta_items": 0, 2024-03-18T15:29:08.472 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_bytes": 0, 2024-03-18T15:29:08.472 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_items": 0, 2024-03-18T15:29:08.472 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_mapbl_bytes": 0, 2024-03-18T15:29:08.472 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_mapbl_items": 0, 2024-03-18T15:29:08.472 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_pglog_bytes": 0, 2024-03-18T15:29:08.473 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_pglog_items": 0, 2024-03-18T15:29:08.473 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap_bytes": 9928, 2024-03-18T15:29:08.473 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap_items": 207, 2024-03-18T15:29:08.473 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T15:29:08.473 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap_mapping_items": 0, 2024-03-18T15:29:08.473 INFO:teuthology.orchestra.run.smithi104.stdout: "pgmap_bytes": 0, 2024-03-18T15:29:08.473 INFO:teuthology.orchestra.run.smithi104.stdout: "pgmap_items": 0, 2024-03-18T15:29:08.473 INFO:teuthology.orchestra.run.smithi104.stdout: "mds_co_bytes": 100718, 2024-03-18T15:29:08.473 INFO:teuthology.orchestra.run.smithi104.stdout: "mds_co_items": 737, 2024-03-18T15:29:08.473 INFO:teuthology.orchestra.run.smithi104.stdout: "unittest_1_bytes": 0, 2024-03-18T15:29:08.473 INFO:teuthology.orchestra.run.smithi104.stdout: "unittest_1_items": 0, 2024-03-18T15:29:08.473 INFO:teuthology.orchestra.run.smithi104.stdout: "unittest_2_bytes": 0, 2024-03-18T15:29:08.473 INFO:teuthology.orchestra.run.smithi104.stdout: "unittest_2_items": 0 2024-03-18T15:29:08.473 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:08.474 INFO:teuthology.orchestra.run.smithi104.stdout: "objecter": { 2024-03-18T15:29:08.474 INFO:teuthology.orchestra.run.smithi104.stdout: "op_active": 0, 2024-03-18T15:29:08.474 INFO:teuthology.orchestra.run.smithi104.stdout: "op_laggy": 0, 2024-03-18T15:29:08.474 INFO:teuthology.orchestra.run.smithi104.stdout: "op_send": 213, 2024-03-18T15:29:08.474 INFO:teuthology.orchestra.run.smithi104.stdout: "op_send_bytes": 246074, 2024-03-18T15:29:08.474 INFO:teuthology.orchestra.run.smithi104.stdout: "op_resend": 33, 2024-03-18T15:29:08.474 INFO:teuthology.orchestra.run.smithi104.stdout: "op_reply": 213, 2024-03-18T15:29:08.474 INFO:teuthology.orchestra.run.smithi104.stdout: "op_latency": { 2024-03-18T15:29:08.474 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 213, 2024-03-18T15:29:08.474 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 1.514980875, 2024-03-18T15:29:08.474 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.007112586 2024-03-18T15:29:08.474 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:08.474 INFO:teuthology.orchestra.run.smithi104.stdout: "op_inflight": 0, 2024-03-18T15:29:08.474 INFO:teuthology.orchestra.run.smithi104.stdout: "oplen_avg": { 2024-03-18T15:29:08.474 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 213, 2024-03-18T15:29:08.475 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 242 2024-03-18T15:29:08.475 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:08.475 INFO:teuthology.orchestra.run.smithi104.stdout: "op": 213, 2024-03-18T15:29:08.475 INFO:teuthology.orchestra.run.smithi104.stdout: "op_r": 2, 2024-03-18T15:29:08.475 INFO:teuthology.orchestra.run.smithi104.stdout: "op_w": 211, 2024-03-18T15:29:08.475 INFO:teuthology.orchestra.run.smithi104.stdout: "op_rmw": 0, 2024-03-18T15:29:08.475 INFO:teuthology.orchestra.run.smithi104.stdout: "op_pg": 0, 2024-03-18T15:29:08.475 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_stat": 0, 2024-03-18T15:29:08.475 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_create": 10, 2024-03-18T15:29:08.475 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_read": 2, 2024-03-18T15:29:08.475 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_write": 85, 2024-03-18T15:29:08.475 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_writefull": 14, 2024-03-18T15:29:08.475 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_writesame": 0, 2024-03-18T15:29:08.476 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_append": 0, 2024-03-18T15:29:08.476 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_zero": 0, 2024-03-18T15:29:08.476 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_truncate": 0, 2024-03-18T15:29:08.476 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_delete": 83, 2024-03-18T15:29:08.476 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_mapext": 0, 2024-03-18T15:29:08.476 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_sparse_read": 0, 2024-03-18T15:29:08.476 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_clonerange": 0, 2024-03-18T15:29:08.476 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_getxattr": 0, 2024-03-18T15:29:08.476 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_setxattr": 20, 2024-03-18T15:29:08.476 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_cmpxattr": 0, 2024-03-18T15:29:08.476 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_rmxattr": 0, 2024-03-18T15:29:08.476 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_resetxattrs": 0, 2024-03-18T15:29:08.476 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_call": 0, 2024-03-18T15:29:08.476 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_watch": 0, 2024-03-18T15:29:08.477 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_notify": 0, 2024-03-18T15:29:08.477 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_src_cmpxattr": 0, 2024-03-18T15:29:08.477 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_pgls": 0, 2024-03-18T15:29:08.477 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_pgls_filter": 0, 2024-03-18T15:29:08.477 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_other": 0, 2024-03-18T15:29:08.477 INFO:teuthology.orchestra.run.smithi104.stdout: "linger_active": 0, 2024-03-18T15:29:08.477 INFO:teuthology.orchestra.run.smithi104.stdout: "linger_send": 0, 2024-03-18T15:29:08.477 INFO:teuthology.orchestra.run.smithi104.stdout: "linger_resend": 0, 2024-03-18T15:29:08.477 INFO:teuthology.orchestra.run.smithi104.stdout: "linger_ping": 0, 2024-03-18T15:29:08.477 INFO:teuthology.orchestra.run.smithi104.stdout: "poolop_active": 0, 2024-03-18T15:29:08.477 INFO:teuthology.orchestra.run.smithi104.stdout: "poolop_send": 0, 2024-03-18T15:29:08.477 INFO:teuthology.orchestra.run.smithi104.stdout: "poolop_resend": 0, 2024-03-18T15:29:08.477 INFO:teuthology.orchestra.run.smithi104.stdout: "poolstat_active": 0, 2024-03-18T15:29:08.477 INFO:teuthology.orchestra.run.smithi104.stdout: "poolstat_send": 0, 2024-03-18T15:29:08.477 INFO:teuthology.orchestra.run.smithi104.stdout: "poolstat_resend": 0, 2024-03-18T15:29:08.478 INFO:teuthology.orchestra.run.smithi104.stdout: "statfs_active": 0, 2024-03-18T15:29:08.478 INFO:teuthology.orchestra.run.smithi104.stdout: "statfs_send": 0, 2024-03-18T15:29:08.478 INFO:teuthology.orchestra.run.smithi104.stdout: "statfs_resend": 0, 2024-03-18T15:29:08.478 INFO:teuthology.orchestra.run.smithi104.stdout: "command_active": 0, 2024-03-18T15:29:08.478 INFO:teuthology.orchestra.run.smithi104.stdout: "command_send": 0, 2024-03-18T15:29:08.478 INFO:teuthology.orchestra.run.smithi104.stdout: "command_resend": 0, 2024-03-18T15:29:08.478 INFO:teuthology.orchestra.run.smithi104.stdout: "map_epoch": 103, 2024-03-18T15:29:08.478 INFO:teuthology.orchestra.run.smithi104.stdout: "map_full": 0, 2024-03-18T15:29:08.478 INFO:teuthology.orchestra.run.smithi104.stdout: "map_inc": 14, 2024-03-18T15:29:08.478 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_sessions": 12, 2024-03-18T15:29:08.478 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_session_open": 12, 2024-03-18T15:29:08.478 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_session_close": 0, 2024-03-18T15:29:08.478 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_laggy": 0, 2024-03-18T15:29:08.478 INFO:teuthology.orchestra.run.smithi104.stdout: "omap_wr": 23, 2024-03-18T15:29:08.479 INFO:teuthology.orchestra.run.smithi104.stdout: "omap_rd": 0, 2024-03-18T15:29:08.479 INFO:teuthology.orchestra.run.smithi104.stdout: "omap_del": 5 2024-03-18T15:29:08.479 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:08.479 INFO:teuthology.orchestra.run.smithi104.stdout: "oft": { 2024-03-18T15:29:08.479 INFO:teuthology.orchestra.run.smithi104.stdout: "omap_total_objs": 1, 2024-03-18T15:29:08.479 INFO:teuthology.orchestra.run.smithi104.stdout: "omap_total_kv_pairs": 0, 2024-03-18T15:29:08.479 INFO:teuthology.orchestra.run.smithi104.stdout: "omap_total_updates": 0, 2024-03-18T15:29:08.479 INFO:teuthology.orchestra.run.smithi104.stdout: "omap_total_removes": 0 2024-03-18T15:29:08.479 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:08.479 INFO:teuthology.orchestra.run.smithi104.stdout: "osd-slow-ops": { 2024-03-18T15:29:08.479 INFO:teuthology.orchestra.run.smithi104.stdout: "slow_ops_count": 0 2024-03-18T15:29:08.479 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:08.479 INFO:teuthology.orchestra.run.smithi104.stdout: "purge_queue": { 2024-03-18T15:29:08.479 INFO:teuthology.orchestra.run.smithi104.stdout: "pq_executing_ops": 0, 2024-03-18T15:29:08.479 INFO:teuthology.orchestra.run.smithi104.stdout: "pq_executing_ops_high_water": 65, 2024-03-18T15:29:08.480 INFO:teuthology.orchestra.run.smithi104.stdout: "pq_executing": 0, 2024-03-18T15:29:08.480 INFO:teuthology.orchestra.run.smithi104.stdout: "pq_executing_high_water": 3, 2024-03-18T15:29:08.480 INFO:teuthology.orchestra.run.smithi104.stdout: "pq_executed_ops": 71, 2024-03-18T15:29:08.480 INFO:teuthology.orchestra.run.smithi104.stdout: "pq_executed": 4, 2024-03-18T15:29:08.480 INFO:teuthology.orchestra.run.smithi104.stdout: "pq_item_in_journal": 0 2024-03-18T15:29:08.480 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:08.480 INFO:teuthology.orchestra.run.smithi104.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T15:29:08.480 INFO:teuthology.orchestra.run.smithi104.stdout: "val": 0, 2024-03-18T15:29:08.480 INFO:teuthology.orchestra.run.smithi104.stdout: "max": 104857600, 2024-03-18T15:29:08.480 INFO:teuthology.orchestra.run.smithi104.stdout: "get_started": 0, 2024-03-18T15:29:08.480 INFO:teuthology.orchestra.run.smithi104.stdout: "get": 1021, 2024-03-18T15:29:08.480 INFO:teuthology.orchestra.run.smithi104.stdout: "get_sum": 1968993, 2024-03-18T15:29:08.480 INFO:teuthology.orchestra.run.smithi104.stdout: "get_or_fail_fail": 0, 2024-03-18T15:29:08.480 INFO:teuthology.orchestra.run.smithi104.stdout: "get_or_fail_success": 1021, 2024-03-18T15:29:08.480 INFO:teuthology.orchestra.run.smithi104.stdout: "take": 0, 2024-03-18T15:29:08.480 INFO:teuthology.orchestra.run.smithi104.stdout: "take_sum": 0, 2024-03-18T15:29:08.481 INFO:teuthology.orchestra.run.smithi104.stdout: "put": 1021, 2024-03-18T15:29:08.481 INFO:teuthology.orchestra.run.smithi104.stdout: "put_sum": 1968993, 2024-03-18T15:29:08.481 INFO:teuthology.orchestra.run.smithi104.stdout: "wait": { 2024-03-18T15:29:08.481 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:08.481 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:08.481 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:08.481 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-03-18T15:29:08.481 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:08.481 INFO:teuthology.orchestra.run.smithi104.stdout: "throttle-objecter_bytes": { 2024-03-18T15:29:08.481 INFO:teuthology.orchestra.run.smithi104.stdout: "val": 0, 2024-03-18T15:29:08.481 INFO:teuthology.orchestra.run.smithi104.stdout: "max": 104857600, 2024-03-18T15:29:08.481 INFO:teuthology.orchestra.run.smithi104.stdout: "get_started": 0, 2024-03-18T15:29:08.481 INFO:teuthology.orchestra.run.smithi104.stdout: "get": 0, 2024-03-18T15:29:08.481 INFO:teuthology.orchestra.run.smithi104.stdout: "get_sum": 0, 2024-03-18T15:29:08.481 INFO:teuthology.orchestra.run.smithi104.stdout: "get_or_fail_fail": 0, 2024-03-18T15:29:08.482 INFO:teuthology.orchestra.run.smithi104.stdout: "get_or_fail_success": 0, 2024-03-18T15:29:08.482 INFO:teuthology.orchestra.run.smithi104.stdout: "take": 213, 2024-03-18T15:29:08.482 INFO:teuthology.orchestra.run.smithi104.stdout: "take_sum": 246542, 2024-03-18T15:29:08.482 INFO:teuthology.orchestra.run.smithi104.stdout: "put": 213, 2024-03-18T15:29:08.482 INFO:teuthology.orchestra.run.smithi104.stdout: "put_sum": 246542, 2024-03-18T15:29:08.482 INFO:teuthology.orchestra.run.smithi104.stdout: "wait": { 2024-03-18T15:29:08.482 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:08.482 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:08.482 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:08.482 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-03-18T15:29:08.482 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:08.483 INFO:teuthology.orchestra.run.smithi104.stdout: "throttle-objecter_ops": { 2024-03-18T15:29:08.483 INFO:teuthology.orchestra.run.smithi104.stdout: "val": 0, 2024-03-18T15:29:08.483 INFO:teuthology.orchestra.run.smithi104.stdout: "max": 1024, 2024-03-18T15:29:08.483 INFO:teuthology.orchestra.run.smithi104.stdout: "get_started": 0, 2024-03-18T15:29:08.483 INFO:teuthology.orchestra.run.smithi104.stdout: "get": 0, 2024-03-18T15:29:08.483 INFO:teuthology.orchestra.run.smithi104.stdout: "get_sum": 0, 2024-03-18T15:29:08.483 INFO:teuthology.orchestra.run.smithi104.stdout: "get_or_fail_fail": 0, 2024-03-18T15:29:08.483 INFO:teuthology.orchestra.run.smithi104.stdout: "get_or_fail_success": 0, 2024-03-18T15:29:08.483 INFO:teuthology.orchestra.run.smithi104.stdout: "take": 213, 2024-03-18T15:29:08.483 INFO:teuthology.orchestra.run.smithi104.stdout: "take_sum": 213, 2024-03-18T15:29:08.483 INFO:teuthology.orchestra.run.smithi104.stdout: "put": 213, 2024-03-18T15:29:08.483 INFO:teuthology.orchestra.run.smithi104.stdout: "put_sum": 213, 2024-03-18T15:29:08.483 INFO:teuthology.orchestra.run.smithi104.stdout: "wait": { 2024-03-18T15:29:08.483 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:08.484 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:08.484 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:08.484 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-03-18T15:29:08.484 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:08.484 INFO:teuthology.orchestra.run.smithi104.stdout: "throttle-write_buf_throttle": { 2024-03-18T15:29:08.484 INFO:teuthology.orchestra.run.smithi104.stdout: "val": 0, 2024-03-18T15:29:08.484 INFO:teuthology.orchestra.run.smithi104.stdout: "max": 3758096384, 2024-03-18T15:29:08.484 INFO:teuthology.orchestra.run.smithi104.stdout: "get_started": 0, 2024-03-18T15:29:08.484 INFO:teuthology.orchestra.run.smithi104.stdout: "get": 4, 2024-03-18T15:29:08.484 INFO:teuthology.orchestra.run.smithi104.stdout: "get_sum": 468, 2024-03-18T15:29:08.484 INFO:teuthology.orchestra.run.smithi104.stdout: "get_or_fail_fail": 0, 2024-03-18T15:29:08.484 INFO:teuthology.orchestra.run.smithi104.stdout: "get_or_fail_success": 4, 2024-03-18T15:29:08.484 INFO:teuthology.orchestra.run.smithi104.stdout: "take": 0, 2024-03-18T15:29:08.484 INFO:teuthology.orchestra.run.smithi104.stdout: "take_sum": 0, 2024-03-18T15:29:08.484 INFO:teuthology.orchestra.run.smithi104.stdout: "put": 2, 2024-03-18T15:29:08.485 INFO:teuthology.orchestra.run.smithi104.stdout: "put_sum": 468, 2024-03-18T15:29:08.485 INFO:teuthology.orchestra.run.smithi104.stdout: "wait": { 2024-03-18T15:29:08.485 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:08.485 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:08.485 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:08.485 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-03-18T15:29:08.485 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:08.485 INFO:teuthology.orchestra.run.smithi104.stdout: "throttle-write_buf_throttle-0x5572792c6500": { 2024-03-18T15:29:08.485 INFO:teuthology.orchestra.run.smithi104.stdout: "val": 0, 2024-03-18T15:29:08.485 INFO:teuthology.orchestra.run.smithi104.stdout: "max": 3758096384, 2024-03-18T15:29:08.485 INFO:teuthology.orchestra.run.smithi104.stdout: "get_started": 0, 2024-03-18T15:29:08.485 INFO:teuthology.orchestra.run.smithi104.stdout: "get": 106, 2024-03-18T15:29:08.485 INFO:teuthology.orchestra.run.smithi104.stdout: "get_sum": 232811, 2024-03-18T15:29:08.485 INFO:teuthology.orchestra.run.smithi104.stdout: "get_or_fail_fail": 0, 2024-03-18T15:29:08.485 INFO:teuthology.orchestra.run.smithi104.stdout: "get_or_fail_success": 106, 2024-03-18T15:29:08.486 INFO:teuthology.orchestra.run.smithi104.stdout: "take": 0, 2024-03-18T15:29:08.486 INFO:teuthology.orchestra.run.smithi104.stdout: "take_sum": 0, 2024-03-18T15:29:08.486 INFO:teuthology.orchestra.run.smithi104.stdout: "put": 83, 2024-03-18T15:29:08.486 INFO:teuthology.orchestra.run.smithi104.stdout: "put_sum": 232811, 2024-03-18T15:29:08.486 INFO:teuthology.orchestra.run.smithi104.stdout: "wait": { 2024-03-18T15:29:08.486 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:08.486 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:08.486 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:08.486 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-03-18T15:29:08.486 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-03-18T15:29:08.486 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-03-18T15:29:09.256 INFO:tasks.check_counter:Daemon mds.c mds.exported=1 2024-03-18T15:29:09.256 INFO:tasks.check_counter:Daemon mds.c mds.imported=1 2024-03-18T15:29:09.256 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update=4 2024-03-18T15:29:09.256 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update_receipt=6 2024-03-18T15:29:09.256 INFO:tasks.check_counter:Daemon mds.c mds.root_rsnaps=0 2024-03-18T15:29:09.256 INFO:tasks.check_counter:Daemon mds.c mds_server.req_mksnap_latency.avgcount=0 2024-03-18T15:29:09.256 INFO:tasks.check_counter:Daemon mds.c mds_server.req_rmsnap_latency.avgcount=0 2024-03-18T15:29:09.256 DEBUG:tasks.check_counter:Getting stats from f 2024-03-18T15:29:09.257 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 -- ceph daemon mds.f perf dump 2024-03-18T15:29:09.521 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T15:29:09.884 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:29:09 smithi104 ceph-mon[36070]: pgmap v331: 129 pgs: 129 active+clean; 1.8 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-18T15:29:09.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:29:09 smithi019 ceph-mon[31522]: pgmap v331: 129 pgs: 129 active+clean; 1.8 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-18T15:29:10.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:29:09 smithi033 ceph-mon[36648]: pgmap v331: 129 pgs: 129 active+clean; 1.8 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-18T15:29:10.923 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:29:10 smithi104 ceph-mon[36070]: osdmap e104: 12 total, 12 up, 12 in 2024-03-18T15:29:10.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:29:10 smithi019 ceph-mon[31522]: osdmap e104: 12 total, 12 up, 12 in 2024-03-18T15:29:11.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:29:10 smithi033 ceph-mon[36648]: osdmap e104: 12 total, 12 up, 12 in 2024-03-18T15:29:11.310 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-03-18T15:29:11.311 INFO:teuthology.orchestra.run.smithi104.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T15:29:11.311 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_recv_messages": 31, 2024-03-18T15:29:11.311 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_send_messages": 176, 2024-03-18T15:29:11.311 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_recv_bytes": 7674, 2024-03-18T15:29:11.311 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_send_bytes": 378602, 2024-03-18T15:29:11.311 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_created_connections": 14, 2024-03-18T15:29:11.311 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_active_connections": 4, 2024-03-18T15:29:11.311 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_running_total_time": 0.010076753, 2024-03-18T15:29:11.311 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_running_send_time": 0.004014616, 2024-03-18T15:29:11.311 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_running_recv_time": 0.003436169, 2024-03-18T15:29:11.311 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_running_fast_dispatch_time": 0.000380114, 2024-03-18T15:29:11.311 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T15:29:11.312 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 160, 2024-03-18T15:29:11.312 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.024350142, 2024-03-18T15:29:11.312 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000152188 2024-03-18T15:29:11.312 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:11.312 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_handle_ack_lat": { 2024-03-18T15:29:11.312 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 18, 2024-03-18T15:29:11.312 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000002116, 2024-03-18T15:29:11.312 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000117 2024-03-18T15:29:11.312 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:11.312 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_recv_encrypted_bytes": 2176, 2024-03-18T15:29:11.312 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_send_encrypted_bytes": 321904 2024-03-18T15:29:11.312 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:11.313 INFO:teuthology.orchestra.run.smithi104.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T15:29:11.313 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_recv_messages": 632, 2024-03-18T15:29:11.313 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_send_messages": 826, 2024-03-18T15:29:11.313 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_recv_bytes": 1912292, 2024-03-18T15:29:11.313 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_send_bytes": 440629, 2024-03-18T15:29:11.313 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_created_connections": 7, 2024-03-18T15:29:11.313 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_active_connections": 5, 2024-03-18T15:29:11.313 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_running_total_time": 0.075217483, 2024-03-18T15:29:11.313 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_running_send_time": 0.023676237, 2024-03-18T15:29:11.313 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_running_recv_time": 0.065617034, 2024-03-18T15:29:11.313 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_running_fast_dispatch_time": 0.007058491, 2024-03-18T15:29:11.314 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T15:29:11.314 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 829, 2024-03-18T15:29:11.314 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.044560122, 2024-03-18T15:29:11.314 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000053751 2024-03-18T15:29:11.314 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:11.314 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_handle_ack_lat": { 2024-03-18T15:29:11.314 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 832, 2024-03-18T15:29:11.314 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000136399, 2024-03-18T15:29:11.314 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000163 2024-03-18T15:29:11.314 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:11.314 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_recv_encrypted_bytes": 1788464, 2024-03-18T15:29:11.314 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_send_encrypted_bytes": 151536 2024-03-18T15:29:11.314 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:11.315 INFO:teuthology.orchestra.run.smithi104.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T15:29:11.315 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_recv_messages": 56, 2024-03-18T15:29:11.315 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_send_messages": 91, 2024-03-18T15:29:11.315 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_recv_bytes": 34013, 2024-03-18T15:29:11.315 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_send_bytes": 248858, 2024-03-18T15:29:11.315 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_created_connections": 11, 2024-03-18T15:29:11.315 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_active_connections": 5, 2024-03-18T15:29:11.315 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_running_total_time": 0.008989566, 2024-03-18T15:29:11.315 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_running_send_time": 0.002798166, 2024-03-18T15:29:11.315 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_running_recv_time": 0.005406996, 2024-03-18T15:29:11.315 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_running_fast_dispatch_time": 0.000361460, 2024-03-18T15:29:11.315 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T15:29:11.315 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 88, 2024-03-18T15:29:11.316 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.018208171, 2024-03-18T15:29:11.316 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000206911 2024-03-18T15:29:11.316 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:11.316 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_handle_ack_lat": { 2024-03-18T15:29:11.316 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 52, 2024-03-18T15:29:11.316 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000007104, 2024-03-18T15:29:11.316 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000136 2024-03-18T15:29:11.316 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:11.316 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_recv_encrypted_bytes": 5536, 2024-03-18T15:29:11.316 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_send_encrypted_bytes": 225376 2024-03-18T15:29:11.316 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:11.316 INFO:teuthology.orchestra.run.smithi104.stdout: "cct": { 2024-03-18T15:29:11.316 INFO:teuthology.orchestra.run.smithi104.stdout: "total_workers": 1, 2024-03-18T15:29:11.316 INFO:teuthology.orchestra.run.smithi104.stdout: "unhealthy_workers": 0 2024-03-18T15:29:11.317 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:11.317 INFO:teuthology.orchestra.run.smithi104.stdout: "finisher-MDSRank": { 2024-03-18T15:29:11.317 INFO:teuthology.orchestra.run.smithi104.stdout: "queue_len": 0, 2024-03-18T15:29:11.317 INFO:teuthology.orchestra.run.smithi104.stdout: "complete_latency": { 2024-03-18T15:29:11.317 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 62, 2024-03-18T15:29:11.317 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.001458088, 2024-03-18T15:29:11.317 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000023517 2024-03-18T15:29:11.317 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-03-18T15:29:11.317 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:11.317 INFO:teuthology.orchestra.run.smithi104.stdout: "finisher-PurgeQueue": { 2024-03-18T15:29:11.317 INFO:teuthology.orchestra.run.smithi104.stdout: "queue_len": 0, 2024-03-18T15:29:11.317 INFO:teuthology.orchestra.run.smithi104.stdout: "complete_latency": { 2024-03-18T15:29:11.317 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 2, 2024-03-18T15:29:11.317 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000007014, 2024-03-18T15:29:11.318 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000003507 2024-03-18T15:29:11.318 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-03-18T15:29:11.318 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:11.318 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": { 2024-03-18T15:29:11.318 INFO:teuthology.orchestra.run.smithi104.stdout: "request": 9, 2024-03-18T15:29:11.318 INFO:teuthology.orchestra.run.smithi104.stdout: "reply": 7, 2024-03-18T15:29:11.318 INFO:teuthology.orchestra.run.smithi104.stdout: "reply_latency": { 2024-03-18T15:29:11.318 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 7, 2024-03-18T15:29:11.318 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.003263543, 2024-03-18T15:29:11.318 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000466220 2024-03-18T15:29:11.318 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:11.318 INFO:teuthology.orchestra.run.smithi104.stdout: "slow_reply": 0, 2024-03-18T15:29:11.318 INFO:teuthology.orchestra.run.smithi104.stdout: "forward": 2, 2024-03-18T15:29:11.318 INFO:teuthology.orchestra.run.smithi104.stdout: "dir_fetch_complete": 0, 2024-03-18T15:29:11.319 INFO:teuthology.orchestra.run.smithi104.stdout: "dir_fetch_keys": 0, 2024-03-18T15:29:11.319 INFO:teuthology.orchestra.run.smithi104.stdout: "dir_commit": 11, 2024-03-18T15:29:11.319 INFO:teuthology.orchestra.run.smithi104.stdout: "dir_split": 0, 2024-03-18T15:29:11.319 INFO:teuthology.orchestra.run.smithi104.stdout: "dir_merge": 0, 2024-03-18T15:29:11.319 INFO:teuthology.orchestra.run.smithi104.stdout: "inodes": 16, 2024-03-18T15:29:11.319 INFO:teuthology.orchestra.run.smithi104.stdout: "inodes_top": 0, 2024-03-18T15:29:11.319 INFO:teuthology.orchestra.run.smithi104.stdout: "inodes_bottom": 16, 2024-03-18T15:29:11.319 INFO:teuthology.orchestra.run.smithi104.stdout: "inodes_pin_tail": 0, 2024-03-18T15:29:11.319 INFO:teuthology.orchestra.run.smithi104.stdout: "inodes_pinned": 13, 2024-03-18T15:29:11.319 INFO:teuthology.orchestra.run.smithi104.stdout: "inodes_expired": 5, 2024-03-18T15:29:11.319 INFO:teuthology.orchestra.run.smithi104.stdout: "inodes_with_caps": 1, 2024-03-18T15:29:11.319 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": 1, 2024-03-18T15:29:11.319 INFO:teuthology.orchestra.run.smithi104.stdout: "subtrees": 4, 2024-03-18T15:29:11.319 INFO:teuthology.orchestra.run.smithi104.stdout: "traverse": 21, 2024-03-18T15:29:11.320 INFO:teuthology.orchestra.run.smithi104.stdout: "traverse_hit": 13, 2024-03-18T15:29:11.320 INFO:teuthology.orchestra.run.smithi104.stdout: "traverse_forward": 1, 2024-03-18T15:29:11.320 INFO:teuthology.orchestra.run.smithi104.stdout: "traverse_discover": 5, 2024-03-18T15:29:11.320 INFO:teuthology.orchestra.run.smithi104.stdout: "traverse_dir_fetch": 0, 2024-03-18T15:29:11.320 INFO:teuthology.orchestra.run.smithi104.stdout: "traverse_remote_ino": 0, 2024-03-18T15:29:11.320 INFO:teuthology.orchestra.run.smithi104.stdout: "traverse_lock": 0, 2024-03-18T15:29:11.320 INFO:teuthology.orchestra.run.smithi104.stdout: "load_cent": 0, 2024-03-18T15:29:11.320 INFO:teuthology.orchestra.run.smithi104.stdout: "q": 0, 2024-03-18T15:29:11.320 INFO:teuthology.orchestra.run.smithi104.stdout: "exported": 0, 2024-03-18T15:29:11.320 INFO:teuthology.orchestra.run.smithi104.stdout: "exported_inodes": 0, 2024-03-18T15:29:11.320 INFO:teuthology.orchestra.run.smithi104.stdout: "imported": 0, 2024-03-18T15:29:11.320 INFO:teuthology.orchestra.run.smithi104.stdout: "imported_inodes": 0, 2024-03-18T15:29:11.320 INFO:teuthology.orchestra.run.smithi104.stdout: "openino_dir_fetch": 0, 2024-03-18T15:29:11.320 INFO:teuthology.orchestra.run.smithi104.stdout: "openino_backtrace_fetch": 0, 2024-03-18T15:29:11.320 INFO:teuthology.orchestra.run.smithi104.stdout: "openino_peer_discover": 0, 2024-03-18T15:29:11.321 INFO:teuthology.orchestra.run.smithi104.stdout: "root_rfiles": 4, 2024-03-18T15:29:11.321 INFO:teuthology.orchestra.run.smithi104.stdout: "root_rbytes": 234, 2024-03-18T15:29:11.321 INFO:teuthology.orchestra.run.smithi104.stdout: "root_rsnaps": 0, 2024-03-18T15:29:11.321 INFO:teuthology.orchestra.run.smithi104.stdout: "scrub_backtrace_fetch": 0, 2024-03-18T15:29:11.321 INFO:teuthology.orchestra.run.smithi104.stdout: "scrub_set_tag": 0, 2024-03-18T15:29:11.321 INFO:teuthology.orchestra.run.smithi104.stdout: "scrub_backtrace_repaired": 0, 2024-03-18T15:29:11.321 INFO:teuthology.orchestra.run.smithi104.stdout: "scrub_inotable_repaired": 0, 2024-03-18T15:29:11.321 INFO:teuthology.orchestra.run.smithi104.stdout: "scrub_dir_inodes": 0, 2024-03-18T15:29:11.321 INFO:teuthology.orchestra.run.smithi104.stdout: "scrub_dir_base_inodes": 0, 2024-03-18T15:29:11.321 INFO:teuthology.orchestra.run.smithi104.stdout: "scrub_dirfrag_rstats": 0, 2024-03-18T15:29:11.321 INFO:teuthology.orchestra.run.smithi104.stdout: "scrub_file_inodes": 0, 2024-03-18T15:29:11.321 INFO:teuthology.orchestra.run.smithi104.stdout: "handle_inode_file_caps": 0, 2024-03-18T15:29:11.321 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph_cap_op_revoke": 0, 2024-03-18T15:29:11.321 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph_cap_op_grant": 0, 2024-03-18T15:29:11.322 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph_cap_op_trunc": 0, 2024-03-18T15:29:11.322 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-18T15:29:11.322 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-18T15:29:11.322 INFO:teuthology.orchestra.run.smithi104.stdout: "handle_client_caps": 0, 2024-03-18T15:29:11.322 INFO:teuthology.orchestra.run.smithi104.stdout: "handle_client_caps_dirty": 0, 2024-03-18T15:29:11.322 INFO:teuthology.orchestra.run.smithi104.stdout: "handle_client_cap_release": 0, 2024-03-18T15:29:11.322 INFO:teuthology.orchestra.run.smithi104.stdout: "process_request_cap_release": 0 2024-03-18T15:29:11.322 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:11.322 INFO:teuthology.orchestra.run.smithi104.stdout: "mds_cache": { 2024-03-18T15:29:11.322 INFO:teuthology.orchestra.run.smithi104.stdout: "dir_update": 0, 2024-03-18T15:29:11.322 INFO:teuthology.orchestra.run.smithi104.stdout: "dir_update_receipt": 8, 2024-03-18T15:29:11.322 INFO:teuthology.orchestra.run.smithi104.stdout: "dir_try_discover": 6, 2024-03-18T15:29:11.322 INFO:teuthology.orchestra.run.smithi104.stdout: "dir_send_discover": 9, 2024-03-18T15:29:11.323 INFO:teuthology.orchestra.run.smithi104.stdout: "dir_handle_discover": 0, 2024-03-18T15:29:11.323 INFO:teuthology.orchestra.run.smithi104.stdout: "num_strays": 0, 2024-03-18T15:29:11.323 INFO:teuthology.orchestra.run.smithi104.stdout: "num_strays_delayed": 0, 2024-03-18T15:29:11.323 INFO:teuthology.orchestra.run.smithi104.stdout: "num_strays_enqueuing": 0, 2024-03-18T15:29:11.323 INFO:teuthology.orchestra.run.smithi104.stdout: "strays_created": 0, 2024-03-18T15:29:11.323 INFO:teuthology.orchestra.run.smithi104.stdout: "strays_enqueued": 0, 2024-03-18T15:29:11.323 INFO:teuthology.orchestra.run.smithi104.stdout: "strays_reintegrated": 0, 2024-03-18T15:29:11.323 INFO:teuthology.orchestra.run.smithi104.stdout: "strays_migrated": 0, 2024-03-18T15:29:11.323 INFO:teuthology.orchestra.run.smithi104.stdout: "num_recovering_processing": 0, 2024-03-18T15:29:11.323 INFO:teuthology.orchestra.run.smithi104.stdout: "num_recovering_enqueued": 0, 2024-03-18T15:29:11.323 INFO:teuthology.orchestra.run.smithi104.stdout: "num_recovering_prioritized": 0, 2024-03-18T15:29:11.323 INFO:teuthology.orchestra.run.smithi104.stdout: "recovery_started": 0, 2024-03-18T15:29:11.323 INFO:teuthology.orchestra.run.smithi104.stdout: "recovery_completed": 0, 2024-03-18T15:29:11.324 INFO:teuthology.orchestra.run.smithi104.stdout: "ireq_quiesce_path": 0, 2024-03-18T15:29:11.324 INFO:teuthology.orchestra.run.smithi104.stdout: "ireq_quiesce_inode": 0, 2024-03-18T15:29:11.324 INFO:teuthology.orchestra.run.smithi104.stdout: "ireq_enqueue_scrub": 0, 2024-03-18T15:29:11.324 INFO:teuthology.orchestra.run.smithi104.stdout: "ireq_exportdir": 0, 2024-03-18T15:29:11.324 INFO:teuthology.orchestra.run.smithi104.stdout: "ireq_flush": 0, 2024-03-18T15:29:11.324 INFO:teuthology.orchestra.run.smithi104.stdout: "ireq_fragmentdir": 0, 2024-03-18T15:29:11.324 INFO:teuthology.orchestra.run.smithi104.stdout: "ireq_fragstats": 0, 2024-03-18T15:29:11.324 INFO:teuthology.orchestra.run.smithi104.stdout: "ireq_inodestats": 0 2024-03-18T15:29:11.324 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:11.324 INFO:teuthology.orchestra.run.smithi104.stdout: "mds_log": { 2024-03-18T15:29:11.324 INFO:teuthology.orchestra.run.smithi104.stdout: "evlrg": 0, 2024-03-18T15:29:11.324 INFO:teuthology.orchestra.run.smithi104.stdout: "evadd": 7, 2024-03-18T15:29:11.324 INFO:teuthology.orchestra.run.smithi104.stdout: "evex": 0, 2024-03-18T15:29:11.325 INFO:teuthology.orchestra.run.smithi104.stdout: "evtrm": 0, 2024-03-18T15:29:11.325 INFO:teuthology.orchestra.run.smithi104.stdout: "ev": 7, 2024-03-18T15:29:11.325 INFO:teuthology.orchestra.run.smithi104.stdout: "evexg": 0, 2024-03-18T15:29:11.325 INFO:teuthology.orchestra.run.smithi104.stdout: "evexd": 0, 2024-03-18T15:29:11.325 INFO:teuthology.orchestra.run.smithi104.stdout: "segadd": 2, 2024-03-18T15:29:11.325 INFO:teuthology.orchestra.run.smithi104.stdout: "segex": 0, 2024-03-18T15:29:11.325 INFO:teuthology.orchestra.run.smithi104.stdout: "segtrm": 0, 2024-03-18T15:29:11.325 INFO:teuthology.orchestra.run.smithi104.stdout: "seg": 2, 2024-03-18T15:29:11.325 INFO:teuthology.orchestra.run.smithi104.stdout: "segmjr": 2, 2024-03-18T15:29:11.325 INFO:teuthology.orchestra.run.smithi104.stdout: "segexg": 0, 2024-03-18T15:29:11.325 INFO:teuthology.orchestra.run.smithi104.stdout: "segexd": 0, 2024-03-18T15:29:11.325 INFO:teuthology.orchestra.run.smithi104.stdout: "expos": 4194304, 2024-03-18T15:29:11.325 INFO:teuthology.orchestra.run.smithi104.stdout: "wrpos": 4196266, 2024-03-18T15:29:11.325 INFO:teuthology.orchestra.run.smithi104.stdout: "rdpos": 0, 2024-03-18T15:29:11.325 INFO:teuthology.orchestra.run.smithi104.stdout: "jlat": { 2024-03-18T15:29:11.326 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 6, 2024-03-18T15:29:11.326 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.027355036, 2024-03-18T15:29:11.326 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.004559172 2024-03-18T15:29:11.326 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:11.326 INFO:teuthology.orchestra.run.smithi104.stdout: "replayed": 0 2024-03-18T15:29:11.326 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:11.326 INFO:teuthology.orchestra.run.smithi104.stdout: "mds_mem": { 2024-03-18T15:29:11.326 INFO:teuthology.orchestra.run.smithi104.stdout: "ino": 21, 2024-03-18T15:29:11.326 INFO:teuthology.orchestra.run.smithi104.stdout: "ino+": 23, 2024-03-18T15:29:11.326 INFO:teuthology.orchestra.run.smithi104.stdout: "ino-": 2, 2024-03-18T15:29:11.326 INFO:teuthology.orchestra.run.smithi104.stdout: "dir": 20, 2024-03-18T15:29:11.326 INFO:teuthology.orchestra.run.smithi104.stdout: "dir+": 22, 2024-03-18T15:29:11.327 INFO:teuthology.orchestra.run.smithi104.stdout: "dir-": 2, 2024-03-18T15:29:11.327 INFO:teuthology.orchestra.run.smithi104.stdout: "dn": 16, 2024-03-18T15:29:11.327 INFO:teuthology.orchestra.run.smithi104.stdout: "dn+": 21, 2024-03-18T15:29:11.327 INFO:teuthology.orchestra.run.smithi104.stdout: "dn-": 5, 2024-03-18T15:29:11.327 INFO:teuthology.orchestra.run.smithi104.stdout: "cap": 1, 2024-03-18T15:29:11.327 INFO:teuthology.orchestra.run.smithi104.stdout: "cap+": 6, 2024-03-18T15:29:11.327 INFO:teuthology.orchestra.run.smithi104.stdout: "cap-": 5, 2024-03-18T15:29:11.327 INFO:teuthology.orchestra.run.smithi104.stdout: "rss": 48952, 2024-03-18T15:29:11.327 INFO:teuthology.orchestra.run.smithi104.stdout: "heap": 223612 2024-03-18T15:29:11.327 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:11.327 INFO:teuthology.orchestra.run.smithi104.stdout: "mds_server": { 2024-03-18T15:29:11.327 INFO:teuthology.orchestra.run.smithi104.stdout: "dispatch_client_request": 13, 2024-03-18T15:29:11.327 INFO:teuthology.orchestra.run.smithi104.stdout: "dispatch_server_request": 1, 2024-03-18T15:29:11.328 INFO:teuthology.orchestra.run.smithi104.stdout: "handle_client_request": 9, 2024-03-18T15:29:11.328 INFO:teuthology.orchestra.run.smithi104.stdout: "handle_client_session": 16, 2024-03-18T15:29:11.328 INFO:teuthology.orchestra.run.smithi104.stdout: "handle_peer_request": 3, 2024-03-18T15:29:11.328 INFO:teuthology.orchestra.run.smithi104.stdout: "req_create_latency": { 2024-03-18T15:29:11.328 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:11.328 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:11.328 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:11.328 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:11.328 INFO:teuthology.orchestra.run.smithi104.stdout: "req_getattr_latency": { 2024-03-18T15:29:11.328 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 2, 2024-03-18T15:29:11.328 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.002126558, 2024-03-18T15:29:11.328 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.001063279 2024-03-18T15:29:11.328 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:11.328 INFO:teuthology.orchestra.run.smithi104.stdout: "req_getfilelock_latency": { 2024-03-18T15:29:11.329 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:11.329 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:11.329 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:11.329 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:11.329 INFO:teuthology.orchestra.run.smithi104.stdout: "req_link_latency": { 2024-03-18T15:29:11.329 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:11.329 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:11.329 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:11.329 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:11.329 INFO:teuthology.orchestra.run.smithi104.stdout: "req_lookup_latency": { 2024-03-18T15:29:11.329 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 4, 2024-03-18T15:29:11.329 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.001041591, 2024-03-18T15:29:11.329 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000260397 2024-03-18T15:29:11.329 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:11.330 INFO:teuthology.orchestra.run.smithi104.stdout: "req_lookuphash_latency": { 2024-03-18T15:29:11.330 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:11.330 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:11.330 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:11.330 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:11.330 INFO:teuthology.orchestra.run.smithi104.stdout: "req_lookupino_latency": { 2024-03-18T15:29:11.330 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 1, 2024-03-18T15:29:11.330 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000095394, 2024-03-18T15:29:11.330 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000095394 2024-03-18T15:29:11.330 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:11.330 INFO:teuthology.orchestra.run.smithi104.stdout: "req_lookupname_latency": { 2024-03-18T15:29:11.330 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:11.331 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:11.331 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:11.331 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:11.331 INFO:teuthology.orchestra.run.smithi104.stdout: "req_lookupparent_latency": { 2024-03-18T15:29:11.331 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:11.331 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:11.331 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:11.331 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:11.331 INFO:teuthology.orchestra.run.smithi104.stdout: "req_lookupsnap_latency": { 2024-03-18T15:29:11.331 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:11.331 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:11.332 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:11.332 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:11.332 INFO:teuthology.orchestra.run.smithi104.stdout: "req_lssnap_latency": { 2024-03-18T15:29:11.332 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:11.332 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:11.332 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:11.332 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:11.332 INFO:teuthology.orchestra.run.smithi104.stdout: "req_mkdir_latency": { 2024-03-18T15:29:11.332 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:11.333 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:11.333 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:11.333 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:11.333 INFO:teuthology.orchestra.run.smithi104.stdout: "req_mknod_latency": { 2024-03-18T15:29:11.333 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:11.333 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:11.333 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:11.334 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:11.334 INFO:teuthology.orchestra.run.smithi104.stdout: "req_mksnap_latency": { 2024-03-18T15:29:11.334 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:11.334 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:11.334 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:11.334 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:11.334 INFO:teuthology.orchestra.run.smithi104.stdout: "req_open_latency": { 2024-03-18T15:29:11.334 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:11.334 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:11.334 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:11.335 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:11.335 INFO:teuthology.orchestra.run.smithi104.stdout: "req_readdir_latency": { 2024-03-18T15:29:11.335 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:11.335 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:11.335 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:11.335 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:11.335 INFO:teuthology.orchestra.run.smithi104.stdout: "req_rename_latency": { 2024-03-18T15:29:11.335 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:11.335 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:11.335 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:11.335 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:11.335 INFO:teuthology.orchestra.run.smithi104.stdout: "req_renamesnap_latency": { 2024-03-18T15:29:11.335 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:11.336 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:11.336 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:11.336 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:11.336 INFO:teuthology.orchestra.run.smithi104.stdout: "req_snapdiff_latency": { 2024-03-18T15:29:11.336 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:11.336 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:11.336 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:11.336 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:11.336 INFO:teuthology.orchestra.run.smithi104.stdout: "req_rmdir_latency": { 2024-03-18T15:29:11.336 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:11.336 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:11.336 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:11.336 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:11.336 INFO:teuthology.orchestra.run.smithi104.stdout: "req_rmsnap_latency": { 2024-03-18T15:29:11.337 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:11.337 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:11.337 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:11.337 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:11.337 INFO:teuthology.orchestra.run.smithi104.stdout: "req_rmxattr_latency": { 2024-03-18T15:29:11.337 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:11.337 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:11.337 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:11.337 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:11.337 INFO:teuthology.orchestra.run.smithi104.stdout: "req_setattr_latency": { 2024-03-18T15:29:11.337 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:11.337 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:11.337 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:11.337 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:11.338 INFO:teuthology.orchestra.run.smithi104.stdout: "req_setdirlayout_latency": { 2024-03-18T15:29:11.338 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:11.338 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:11.338 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:11.338 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:11.338 INFO:teuthology.orchestra.run.smithi104.stdout: "req_setfilelock_latency": { 2024-03-18T15:29:11.338 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:11.338 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:11.338 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:11.338 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:11.338 INFO:teuthology.orchestra.run.smithi104.stdout: "req_setlayout_latency": { 2024-03-18T15:29:11.338 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:11.338 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:11.339 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:11.339 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:11.339 INFO:teuthology.orchestra.run.smithi104.stdout: "req_setxattr_latency": { 2024-03-18T15:29:11.339 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:11.339 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:11.339 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:11.339 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:11.339 INFO:teuthology.orchestra.run.smithi104.stdout: "req_symlink_latency": { 2024-03-18T15:29:11.339 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:11.339 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:11.341 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:11.341 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:11.341 INFO:teuthology.orchestra.run.smithi104.stdout: "req_unlink_latency": { 2024-03-18T15:29:11.341 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:11.341 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:11.341 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:11.341 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:11.341 INFO:teuthology.orchestra.run.smithi104.stdout: "cap_revoke_eviction": 0, 2024-03-18T15:29:11.342 INFO:teuthology.orchestra.run.smithi104.stdout: "cap_acquisition_throttle": 0, 2024-03-18T15:29:11.342 INFO:teuthology.orchestra.run.smithi104.stdout: "req_getvxattr_latency": { 2024-03-18T15:29:11.342 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:11.343 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:11.343 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:11.343 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-03-18T15:29:11.344 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:11.344 INFO:teuthology.orchestra.run.smithi104.stdout: "mds_sessions": { 2024-03-18T15:29:11.345 INFO:teuthology.orchestra.run.smithi104.stdout: "session_count": 1, 2024-03-18T15:29:11.345 INFO:teuthology.orchestra.run.smithi104.stdout: "session_add": 3, 2024-03-18T15:29:11.345 INFO:teuthology.orchestra.run.smithi104.stdout: "session_remove": 2, 2024-03-18T15:29:11.345 INFO:teuthology.orchestra.run.smithi104.stdout: "sessions_open": 1, 2024-03-18T15:29:11.345 INFO:teuthology.orchestra.run.smithi104.stdout: "sessions_stale": 0, 2024-03-18T15:29:11.346 INFO:teuthology.orchestra.run.smithi104.stdout: "total_load": 6, 2024-03-18T15:29:11.346 INFO:teuthology.orchestra.run.smithi104.stdout: "average_load": 2, 2024-03-18T15:29:11.346 INFO:teuthology.orchestra.run.smithi104.stdout: "avg_session_uptime": 636, 2024-03-18T15:29:11.346 INFO:teuthology.orchestra.run.smithi104.stdout: "mdthresh_evicted": 0 2024-03-18T15:29:11.347 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:11.347 INFO:teuthology.orchestra.run.smithi104.stdout: "mempool": { 2024-03-18T15:29:11.347 INFO:teuthology.orchestra.run.smithi104.stdout: "bloom_filter_bytes": 0, 2024-03-18T15:29:11.347 INFO:teuthology.orchestra.run.smithi104.stdout: "bloom_filter_items": 0, 2024-03-18T15:29:11.347 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T15:29:11.348 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_alloc_items": 0, 2024-03-18T15:29:11.348 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T15:29:11.348 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_cache_data_items": 0, 2024-03-18T15:29:11.348 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T15:29:11.348 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T15:29:11.348 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T15:29:11.349 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T15:29:11.349 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T15:29:11.349 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_cache_other_items": 0, 2024-03-18T15:29:11.349 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T15:29:11.349 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T15:29:11.349 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_extent_bytes": 0, 2024-03-18T15:29:11.350 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_extent_items": 0, 2024-03-18T15:29:11.350 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_blob_bytes": 0, 2024-03-18T15:29:11.350 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_blob_items": 0, 2024-03-18T15:29:11.350 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T15:29:11.351 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T15:29:11.351 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T15:29:11.351 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T15:29:11.351 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T15:29:11.352 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_fsck_items": 0, 2024-03-18T15:29:11.352 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_txc_bytes": 0, 2024-03-18T15:29:11.352 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_txc_items": 0, 2024-03-18T15:29:11.352 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T15:29:11.353 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T15:29:11.353 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_writing_bytes": 0, 2024-03-18T15:29:11.353 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_writing_items": 0, 2024-03-18T15:29:11.353 INFO:teuthology.orchestra.run.smithi104.stdout: "bluefs_bytes": 0, 2024-03-18T15:29:11.353 INFO:teuthology.orchestra.run.smithi104.stdout: "bluefs_items": 0, 2024-03-18T15:29:11.354 INFO:teuthology.orchestra.run.smithi104.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T15:29:11.354 INFO:teuthology.orchestra.run.smithi104.stdout: "bluefs_file_reader_items": 0, 2024-03-18T15:29:11.354 INFO:teuthology.orchestra.run.smithi104.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T15:29:11.354 INFO:teuthology.orchestra.run.smithi104.stdout: "bluefs_file_writer_items": 0, 2024-03-18T15:29:11.354 INFO:teuthology.orchestra.run.smithi104.stdout: "buffer_anon_bytes": 264951, 2024-03-18T15:29:11.354 INFO:teuthology.orchestra.run.smithi104.stdout: "buffer_anon_items": 99, 2024-03-18T15:29:11.354 INFO:teuthology.orchestra.run.smithi104.stdout: "buffer_meta_bytes": 88, 2024-03-18T15:29:11.354 INFO:teuthology.orchestra.run.smithi104.stdout: "buffer_meta_items": 1, 2024-03-18T15:29:11.354 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_bytes": 0, 2024-03-18T15:29:11.355 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_items": 0, 2024-03-18T15:29:11.355 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_mapbl_bytes": 0, 2024-03-18T15:29:11.355 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_mapbl_items": 0, 2024-03-18T15:29:11.355 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_pglog_bytes": 0, 2024-03-18T15:29:11.355 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_pglog_items": 0, 2024-03-18T15:29:11.355 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap_bytes": 9928, 2024-03-18T15:29:11.355 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap_items": 207, 2024-03-18T15:29:11.356 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T15:29:11.356 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap_mapping_items": 0, 2024-03-18T15:29:11.356 INFO:teuthology.orchestra.run.smithi104.stdout: "pgmap_bytes": 0, 2024-03-18T15:29:11.356 INFO:teuthology.orchestra.run.smithi104.stdout: "pgmap_items": 0, 2024-03-18T15:29:11.356 INFO:teuthology.orchestra.run.smithi104.stdout: "mds_co_bytes": 86189, 2024-03-18T15:29:11.356 INFO:teuthology.orchestra.run.smithi104.stdout: "mds_co_items": 345, 2024-03-18T15:29:11.356 INFO:teuthology.orchestra.run.smithi104.stdout: "unittest_1_bytes": 0, 2024-03-18T15:29:11.356 INFO:teuthology.orchestra.run.smithi104.stdout: "unittest_1_items": 0, 2024-03-18T15:29:11.357 INFO:teuthology.orchestra.run.smithi104.stdout: "unittest_2_bytes": 0, 2024-03-18T15:29:11.357 INFO:teuthology.orchestra.run.smithi104.stdout: "unittest_2_items": 0 2024-03-18T15:29:11.357 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:11.357 INFO:teuthology.orchestra.run.smithi104.stdout: "objecter": { 2024-03-18T15:29:11.357 INFO:teuthology.orchestra.run.smithi104.stdout: "op_active": 0, 2024-03-18T15:29:11.357 INFO:teuthology.orchestra.run.smithi104.stdout: "op_laggy": 0, 2024-03-18T15:29:11.358 INFO:teuthology.orchestra.run.smithi104.stdout: "op_send": 45, 2024-03-18T15:29:11.358 INFO:teuthology.orchestra.run.smithi104.stdout: "op_send_bytes": 12579, 2024-03-18T15:29:11.362 INFO:teuthology.orchestra.run.smithi104.stdout: "op_resend": 33, 2024-03-18T15:29:11.362 INFO:teuthology.orchestra.run.smithi104.stdout: "op_reply": 45, 2024-03-18T15:29:11.362 INFO:teuthology.orchestra.run.smithi104.stdout: "op_latency": { 2024-03-18T15:29:11.362 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 45, 2024-03-18T15:29:11.362 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.304996249, 2024-03-18T15:29:11.362 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.006777694 2024-03-18T15:29:11.362 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:11.362 INFO:teuthology.orchestra.run.smithi104.stdout: "op_inflight": 0, 2024-03-18T15:29:11.362 INFO:teuthology.orchestra.run.smithi104.stdout: "oplen_avg": { 2024-03-18T15:29:11.362 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 45, 2024-03-18T15:29:11.362 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 69 2024-03-18T15:29:11.362 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:11.362 INFO:teuthology.orchestra.run.smithi104.stdout: "op": 45, 2024-03-18T15:29:11.363 INFO:teuthology.orchestra.run.smithi104.stdout: "op_r": 0, 2024-03-18T15:29:11.363 INFO:teuthology.orchestra.run.smithi104.stdout: "op_w": 45, 2024-03-18T15:29:11.363 INFO:teuthology.orchestra.run.smithi104.stdout: "op_rmw": 0, 2024-03-18T15:29:11.363 INFO:teuthology.orchestra.run.smithi104.stdout: "op_pg": 0, 2024-03-18T15:29:11.363 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_stat": 0, 2024-03-18T15:29:11.363 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_create": 10, 2024-03-18T15:29:11.363 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_read": 0, 2024-03-18T15:29:11.363 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_write": 6, 2024-03-18T15:29:11.363 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_writefull": 8, 2024-03-18T15:29:11.363 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_writesame": 0, 2024-03-18T15:29:11.363 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_append": 0, 2024-03-18T15:29:11.363 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_zero": 0, 2024-03-18T15:29:11.363 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_truncate": 0, 2024-03-18T15:29:11.363 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_delete": 6, 2024-03-18T15:29:11.364 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_mapext": 0, 2024-03-18T15:29:11.364 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_sparse_read": 0, 2024-03-18T15:29:11.364 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_clonerange": 0, 2024-03-18T15:29:11.364 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_getxattr": 0, 2024-03-18T15:29:11.364 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_setxattr": 20, 2024-03-18T15:29:11.364 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_cmpxattr": 0, 2024-03-18T15:29:11.364 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_rmxattr": 0, 2024-03-18T15:29:11.364 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_resetxattrs": 0, 2024-03-18T15:29:11.364 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_call": 0, 2024-03-18T15:29:11.364 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_watch": 0, 2024-03-18T15:29:11.364 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_notify": 0, 2024-03-18T15:29:11.364 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_src_cmpxattr": 0, 2024-03-18T15:29:11.364 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_pgls": 0, 2024-03-18T15:29:11.364 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_pgls_filter": 0, 2024-03-18T15:29:11.365 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_other": 0, 2024-03-18T15:29:11.365 INFO:teuthology.orchestra.run.smithi104.stdout: "linger_active": 0, 2024-03-18T15:29:11.365 INFO:teuthology.orchestra.run.smithi104.stdout: "linger_send": 0, 2024-03-18T15:29:11.365 INFO:teuthology.orchestra.run.smithi104.stdout: "linger_resend": 0, 2024-03-18T15:29:11.365 INFO:teuthology.orchestra.run.smithi104.stdout: "linger_ping": 0, 2024-03-18T15:29:11.365 INFO:teuthology.orchestra.run.smithi104.stdout: "poolop_active": 0, 2024-03-18T15:29:11.365 INFO:teuthology.orchestra.run.smithi104.stdout: "poolop_send": 0, 2024-03-18T15:29:11.365 INFO:teuthology.orchestra.run.smithi104.stdout: "poolop_resend": 0, 2024-03-18T15:29:11.365 INFO:teuthology.orchestra.run.smithi104.stdout: "poolstat_active": 0, 2024-03-18T15:29:11.365 INFO:teuthology.orchestra.run.smithi104.stdout: "poolstat_send": 0, 2024-03-18T15:29:11.365 INFO:teuthology.orchestra.run.smithi104.stdout: "poolstat_resend": 0, 2024-03-18T15:29:11.365 INFO:teuthology.orchestra.run.smithi104.stdout: "statfs_active": 0, 2024-03-18T15:29:11.365 INFO:teuthology.orchestra.run.smithi104.stdout: "statfs_send": 0, 2024-03-18T15:29:11.365 INFO:teuthology.orchestra.run.smithi104.stdout: "statfs_resend": 0, 2024-03-18T15:29:11.365 INFO:teuthology.orchestra.run.smithi104.stdout: "command_active": 0, 2024-03-18T15:29:11.366 INFO:teuthology.orchestra.run.smithi104.stdout: "command_send": 0, 2024-03-18T15:29:11.366 INFO:teuthology.orchestra.run.smithi104.stdout: "command_resend": 0, 2024-03-18T15:29:11.366 INFO:teuthology.orchestra.run.smithi104.stdout: "map_epoch": 104, 2024-03-18T15:29:11.366 INFO:teuthology.orchestra.run.smithi104.stdout: "map_full": 0, 2024-03-18T15:29:11.366 INFO:teuthology.orchestra.run.smithi104.stdout: "map_inc": 15, 2024-03-18T15:29:11.366 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_sessions": 10, 2024-03-18T15:29:11.366 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_session_open": 10, 2024-03-18T15:29:11.366 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_session_close": 0, 2024-03-18T15:29:11.366 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_laggy": 0, 2024-03-18T15:29:11.366 INFO:teuthology.orchestra.run.smithi104.stdout: "omap_wr": 17, 2024-03-18T15:29:11.366 INFO:teuthology.orchestra.run.smithi104.stdout: "omap_rd": 0, 2024-03-18T15:29:11.366 INFO:teuthology.orchestra.run.smithi104.stdout: "omap_del": 2 2024-03-18T15:29:11.366 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:11.366 INFO:teuthology.orchestra.run.smithi104.stdout: "oft": { 2024-03-18T15:29:11.367 INFO:teuthology.orchestra.run.smithi104.stdout: "omap_total_objs": 1, 2024-03-18T15:29:11.367 INFO:teuthology.orchestra.run.smithi104.stdout: "omap_total_kv_pairs": 0, 2024-03-18T15:29:11.367 INFO:teuthology.orchestra.run.smithi104.stdout: "omap_total_updates": 0, 2024-03-18T15:29:11.367 INFO:teuthology.orchestra.run.smithi104.stdout: "omap_total_removes": 0 2024-03-18T15:29:11.367 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:11.367 INFO:teuthology.orchestra.run.smithi104.stdout: "osd-slow-ops": { 2024-03-18T15:29:11.367 INFO:teuthology.orchestra.run.smithi104.stdout: "slow_ops_count": 0 2024-03-18T15:29:11.367 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:11.367 INFO:teuthology.orchestra.run.smithi104.stdout: "purge_queue": { 2024-03-18T15:29:11.367 INFO:teuthology.orchestra.run.smithi104.stdout: "pq_executing_ops": 0, 2024-03-18T15:29:11.367 INFO:teuthology.orchestra.run.smithi104.stdout: "pq_executing_ops_high_water": 0, 2024-03-18T15:29:11.367 INFO:teuthology.orchestra.run.smithi104.stdout: "pq_executing": 0, 2024-03-18T15:29:11.367 INFO:teuthology.orchestra.run.smithi104.stdout: "pq_executing_high_water": 0, 2024-03-18T15:29:11.367 INFO:teuthology.orchestra.run.smithi104.stdout: "pq_executed_ops": 0, 2024-03-18T15:29:11.367 INFO:teuthology.orchestra.run.smithi104.stdout: "pq_executed": 0, 2024-03-18T15:29:11.368 INFO:teuthology.orchestra.run.smithi104.stdout: "pq_item_in_journal": 0 2024-03-18T15:29:11.368 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:11.368 INFO:teuthology.orchestra.run.smithi104.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T15:29:11.368 INFO:teuthology.orchestra.run.smithi104.stdout: "val": 0, 2024-03-18T15:29:11.368 INFO:teuthology.orchestra.run.smithi104.stdout: "max": 104857600, 2024-03-18T15:29:11.368 INFO:teuthology.orchestra.run.smithi104.stdout: "get_started": 0, 2024-03-18T15:29:11.368 INFO:teuthology.orchestra.run.smithi104.stdout: "get": 715, 2024-03-18T15:29:11.368 INFO:teuthology.orchestra.run.smithi104.stdout: "get_sum": 1870679, 2024-03-18T15:29:11.368 INFO:teuthology.orchestra.run.smithi104.stdout: "get_or_fail_fail": 0, 2024-03-18T15:29:11.368 INFO:teuthology.orchestra.run.smithi104.stdout: "get_or_fail_success": 715, 2024-03-18T15:29:11.368 INFO:teuthology.orchestra.run.smithi104.stdout: "take": 0, 2024-03-18T15:29:11.368 INFO:teuthology.orchestra.run.smithi104.stdout: "take_sum": 0, 2024-03-18T15:29:11.368 INFO:teuthology.orchestra.run.smithi104.stdout: "put": 715, 2024-03-18T15:29:11.368 INFO:teuthology.orchestra.run.smithi104.stdout: "put_sum": 1870679, 2024-03-18T15:29:11.368 INFO:teuthology.orchestra.run.smithi104.stdout: "wait": { 2024-03-18T15:29:11.369 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:11.369 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:11.369 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:11.369 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-03-18T15:29:11.369 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:11.369 INFO:teuthology.orchestra.run.smithi104.stdout: "throttle-objecter_bytes": { 2024-03-18T15:29:11.369 INFO:teuthology.orchestra.run.smithi104.stdout: "val": 0, 2024-03-18T15:29:11.369 INFO:teuthology.orchestra.run.smithi104.stdout: "max": 104857600, 2024-03-18T15:29:11.369 INFO:teuthology.orchestra.run.smithi104.stdout: "get_started": 0, 2024-03-18T15:29:11.369 INFO:teuthology.orchestra.run.smithi104.stdout: "get": 0, 2024-03-18T15:29:11.369 INFO:teuthology.orchestra.run.smithi104.stdout: "get_sum": 0, 2024-03-18T15:29:11.369 INFO:teuthology.orchestra.run.smithi104.stdout: "get_or_fail_fail": 0, 2024-03-18T15:29:11.369 INFO:teuthology.orchestra.run.smithi104.stdout: "get_or_fail_success": 0, 2024-03-18T15:29:11.369 INFO:teuthology.orchestra.run.smithi104.stdout: "take": 45, 2024-03-18T15:29:11.370 INFO:teuthology.orchestra.run.smithi104.stdout: "take_sum": 12579, 2024-03-18T15:29:11.370 INFO:teuthology.orchestra.run.smithi104.stdout: "put": 45, 2024-03-18T15:29:11.370 INFO:teuthology.orchestra.run.smithi104.stdout: "put_sum": 12579, 2024-03-18T15:29:11.370 INFO:teuthology.orchestra.run.smithi104.stdout: "wait": { 2024-03-18T15:29:11.370 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:11.370 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:11.370 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:11.370 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-03-18T15:29:11.370 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:11.370 INFO:teuthology.orchestra.run.smithi104.stdout: "throttle-objecter_ops": { 2024-03-18T15:29:11.370 INFO:teuthology.orchestra.run.smithi104.stdout: "val": 0, 2024-03-18T15:29:11.370 INFO:teuthology.orchestra.run.smithi104.stdout: "max": 1024, 2024-03-18T15:29:11.370 INFO:teuthology.orchestra.run.smithi104.stdout: "get_started": 0, 2024-03-18T15:29:11.370 INFO:teuthology.orchestra.run.smithi104.stdout: "get": 0, 2024-03-18T15:29:11.371 INFO:teuthology.orchestra.run.smithi104.stdout: "get_sum": 0, 2024-03-18T15:29:11.371 INFO:teuthology.orchestra.run.smithi104.stdout: "get_or_fail_fail": 0, 2024-03-18T15:29:11.371 INFO:teuthology.orchestra.run.smithi104.stdout: "get_or_fail_success": 0, 2024-03-18T15:29:11.371 INFO:teuthology.orchestra.run.smithi104.stdout: "take": 45, 2024-03-18T15:29:11.371 INFO:teuthology.orchestra.run.smithi104.stdout: "take_sum": 45, 2024-03-18T15:29:11.371 INFO:teuthology.orchestra.run.smithi104.stdout: "put": 45, 2024-03-18T15:29:11.371 INFO:teuthology.orchestra.run.smithi104.stdout: "put_sum": 45, 2024-03-18T15:29:11.371 INFO:teuthology.orchestra.run.smithi104.stdout: "wait": { 2024-03-18T15:29:11.371 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:11.371 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:11.371 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:11.371 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-03-18T15:29:11.371 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:11.371 INFO:teuthology.orchestra.run.smithi104.stdout: "throttle-write_buf_throttle": { 2024-03-18T15:29:11.371 INFO:teuthology.orchestra.run.smithi104.stdout: "val": 0, 2024-03-18T15:29:11.372 INFO:teuthology.orchestra.run.smithi104.stdout: "max": 3758096384, 2024-03-18T15:29:11.372 INFO:teuthology.orchestra.run.smithi104.stdout: "get_started": 0, 2024-03-18T15:29:11.372 INFO:teuthology.orchestra.run.smithi104.stdout: "get": 0, 2024-03-18T15:29:11.372 INFO:teuthology.orchestra.run.smithi104.stdout: "get_sum": 0, 2024-03-18T15:29:11.372 INFO:teuthology.orchestra.run.smithi104.stdout: "get_or_fail_fail": 0, 2024-03-18T15:29:11.372 INFO:teuthology.orchestra.run.smithi104.stdout: "get_or_fail_success": 0, 2024-03-18T15:29:11.372 INFO:teuthology.orchestra.run.smithi104.stdout: "take": 0, 2024-03-18T15:29:11.372 INFO:teuthology.orchestra.run.smithi104.stdout: "take_sum": 0, 2024-03-18T15:29:11.372 INFO:teuthology.orchestra.run.smithi104.stdout: "put": 0, 2024-03-18T15:29:11.372 INFO:teuthology.orchestra.run.smithi104.stdout: "put_sum": 0, 2024-03-18T15:29:11.372 INFO:teuthology.orchestra.run.smithi104.stdout: "wait": { 2024-03-18T15:29:11.372 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:11.372 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:11.372 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:11.372 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-03-18T15:29:11.373 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:11.373 INFO:teuthology.orchestra.run.smithi104.stdout: "throttle-write_buf_throttle-0x5586725e8500": { 2024-03-18T15:29:11.373 INFO:teuthology.orchestra.run.smithi104.stdout: "val": 0, 2024-03-18T15:29:11.373 INFO:teuthology.orchestra.run.smithi104.stdout: "max": 3758096384, 2024-03-18T15:29:11.373 INFO:teuthology.orchestra.run.smithi104.stdout: "get_started": 0, 2024-03-18T15:29:11.373 INFO:teuthology.orchestra.run.smithi104.stdout: "get": 7, 2024-03-18T15:29:11.373 INFO:teuthology.orchestra.run.smithi104.stdout: "get_sum": 1962, 2024-03-18T15:29:11.373 INFO:teuthology.orchestra.run.smithi104.stdout: "get_or_fail_fail": 0, 2024-03-18T15:29:11.373 INFO:teuthology.orchestra.run.smithi104.stdout: "get_or_fail_success": 7, 2024-03-18T15:29:11.373 INFO:teuthology.orchestra.run.smithi104.stdout: "take": 0, 2024-03-18T15:29:11.373 INFO:teuthology.orchestra.run.smithi104.stdout: "take_sum": 0, 2024-03-18T15:29:11.373 INFO:teuthology.orchestra.run.smithi104.stdout: "put": 6, 2024-03-18T15:29:11.374 INFO:teuthology.orchestra.run.smithi104.stdout: "put_sum": 1962, 2024-03-18T15:29:11.374 INFO:teuthology.orchestra.run.smithi104.stdout: "wait": { 2024-03-18T15:29:11.374 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:11.374 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:11.374 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:11.374 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-03-18T15:29:11.374 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-03-18T15:29:11.374 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-03-18T15:29:11.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:29:11 smithi104 ceph-mon[36070]: pgmap v333: 129 pgs: 129 active+clean; 1.8 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-18T15:29:11.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:29:11 smithi019 ceph-mon[31522]: pgmap v333: 129 pgs: 129 active+clean; 1.8 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-18T15:29:12.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:29:11 smithi033 ceph-mon[36648]: pgmap v333: 129 pgs: 129 active+clean; 1.8 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-18T15:29:12.200 INFO:tasks.check_counter:Daemon mds.f mds.exported=0 2024-03-18T15:29:12.200 INFO:tasks.check_counter:Daemon mds.f mds.imported=0 2024-03-18T15:29:12.200 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update=0 2024-03-18T15:29:12.200 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update_receipt=8 2024-03-18T15:29:12.200 INFO:tasks.check_counter:Daemon mds.f mds.root_rsnaps=0 2024-03-18T15:29:12.201 INFO:tasks.check_counter:Daemon mds.f mds_server.req_mksnap_latency.avgcount=0 2024-03-18T15:29:12.201 INFO:tasks.check_counter:Daemon mds.f mds_server.req_rmsnap_latency.avgcount=0 2024-03-18T15:29:12.201 DEBUG:tasks.check_counter:Getting stats from i 2024-03-18T15:29:12.201 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 -- ceph daemon mds.i perf dump 2024-03-18T15:29:12.477 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T15:29:13.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:29:13 smithi104 ceph-mon[36070]: pgmap v334: 129 pgs: 129 active+clean; 1.8 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-18T15:29:13.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:29:13 smithi019 ceph-mon[31522]: pgmap v334: 129 pgs: 129 active+clean; 1.8 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-18T15:29:14.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:29:13 smithi033 ceph-mon[36648]: pgmap v334: 129 pgs: 129 active+clean; 1.8 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-18T15:29:14.226 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-03-18T15:29:14.227 INFO:teuthology.orchestra.run.smithi104.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T15:29:14.227 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_recv_messages": 378, 2024-03-18T15:29:14.227 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_send_messages": 556, 2024-03-18T15:29:14.227 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_recv_bytes": 157879, 2024-03-18T15:29:14.227 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_send_bytes": 345828, 2024-03-18T15:29:14.227 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_created_connections": 9, 2024-03-18T15:29:14.227 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_active_connections": 4, 2024-03-18T15:29:14.227 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_running_total_time": 0.037831066, 2024-03-18T15:29:14.227 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_running_send_time": 0.014216073, 2024-03-18T15:29:14.227 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_running_recv_time": 0.018464879, 2024-03-18T15:29:14.227 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_running_fast_dispatch_time": 0.003355725, 2024-03-18T15:29:14.227 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T15:29:14.227 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 554, 2024-03-18T15:29:14.228 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.029883754, 2024-03-18T15:29:14.228 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000053941 2024-03-18T15:29:14.228 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:14.228 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_handle_ack_lat": { 2024-03-18T15:29:14.228 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 867, 2024-03-18T15:29:14.228 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000149187, 2024-03-18T15:29:14.228 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000172 2024-03-18T15:29:14.228 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:14.228 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-18T15:29:14.228 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_send_encrypted_bytes": 0 2024-03-18T15:29:14.228 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:14.228 INFO:teuthology.orchestra.run.smithi104.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T15:29:14.228 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_recv_messages": 26, 2024-03-18T15:29:14.228 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_send_messages": 216, 2024-03-18T15:29:14.229 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_recv_bytes": 6895, 2024-03-18T15:29:14.229 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_send_bytes": 520309, 2024-03-18T15:29:14.229 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_created_connections": 13, 2024-03-18T15:29:14.229 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_active_connections": 7, 2024-03-18T15:29:14.229 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_running_total_time": 0.015455540, 2024-03-18T15:29:14.229 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_running_send_time": 0.006933237, 2024-03-18T15:29:14.229 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_running_recv_time": 0.006324593, 2024-03-18T15:29:14.229 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_running_fast_dispatch_time": 0.000351465, 2024-03-18T15:29:14.229 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T15:29:14.229 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 213, 2024-03-18T15:29:14.229 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.039263159, 2024-03-18T15:29:14.229 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000184334 2024-03-18T15:29:14.229 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:14.229 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_handle_ack_lat": { 2024-03-18T15:29:14.230 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 39, 2024-03-18T15:29:14.230 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000006941, 2024-03-18T15:29:14.230 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000177 2024-03-18T15:29:14.230 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:14.230 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-18T15:29:14.230 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_send_encrypted_bytes": 463792 2024-03-18T15:29:14.230 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:14.230 INFO:teuthology.orchestra.run.smithi104.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T15:29:14.230 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_recv_messages": 331, 2024-03-18T15:29:14.230 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_send_messages": 337, 2024-03-18T15:29:14.230 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_recv_bytes": 1812499, 2024-03-18T15:29:14.230 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_send_bytes": 172099, 2024-03-18T15:29:14.230 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_created_connections": 6, 2024-03-18T15:29:14.230 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_active_connections": 5, 2024-03-18T15:29:14.231 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_running_total_time": 0.045899812, 2024-03-18T15:29:14.231 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_running_send_time": 0.010722677, 2024-03-18T15:29:14.231 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_running_recv_time": 0.049537415, 2024-03-18T15:29:14.231 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_running_fast_dispatch_time": 0.004490717, 2024-03-18T15:29:14.231 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T15:29:14.231 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 337, 2024-03-18T15:29:14.231 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.014051177, 2024-03-18T15:29:14.231 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000041694 2024-03-18T15:29:14.231 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:14.231 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_handle_ack_lat": { 2024-03-18T15:29:14.231 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 47, 2024-03-18T15:29:14.231 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000007202, 2024-03-18T15:29:14.231 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000153 2024-03-18T15:29:14.231 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:14.232 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_recv_encrypted_bytes": 1810272, 2024-03-18T15:29:14.232 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_send_encrypted_bytes": 163040 2024-03-18T15:29:14.232 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:14.232 INFO:teuthology.orchestra.run.smithi104.stdout: "cct": { 2024-03-18T15:29:14.232 INFO:teuthology.orchestra.run.smithi104.stdout: "total_workers": 1, 2024-03-18T15:29:14.232 INFO:teuthology.orchestra.run.smithi104.stdout: "unhealthy_workers": 0 2024-03-18T15:29:14.232 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:14.233 INFO:teuthology.orchestra.run.smithi104.stdout: "finisher-MDSRank": { 2024-03-18T15:29:14.233 INFO:teuthology.orchestra.run.smithi104.stdout: "queue_len": 0, 2024-03-18T15:29:14.233 INFO:teuthology.orchestra.run.smithi104.stdout: "complete_latency": { 2024-03-18T15:29:14.233 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 48, 2024-03-18T15:29:14.233 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.001954567, 2024-03-18T15:29:14.233 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000040720 2024-03-18T15:29:14.233 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-03-18T15:29:14.233 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:14.233 INFO:teuthology.orchestra.run.smithi104.stdout: "finisher-PurgeQueue": { 2024-03-18T15:29:14.233 INFO:teuthology.orchestra.run.smithi104.stdout: "queue_len": 0, 2024-03-18T15:29:14.233 INFO:teuthology.orchestra.run.smithi104.stdout: "complete_latency": { 2024-03-18T15:29:14.233 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 2, 2024-03-18T15:29:14.233 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000009574, 2024-03-18T15:29:14.233 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000004787 2024-03-18T15:29:14.234 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-03-18T15:29:14.234 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:14.234 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": { 2024-03-18T15:29:14.234 INFO:teuthology.orchestra.run.smithi104.stdout: "request": 7, 2024-03-18T15:29:14.234 INFO:teuthology.orchestra.run.smithi104.stdout: "reply": 3, 2024-03-18T15:29:14.234 INFO:teuthology.orchestra.run.smithi104.stdout: "reply_latency": { 2024-03-18T15:29:14.234 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 3, 2024-03-18T15:29:14.234 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000480120, 2024-03-18T15:29:14.234 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000160040 2024-03-18T15:29:14.234 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:14.234 INFO:teuthology.orchestra.run.smithi104.stdout: "slow_reply": 0, 2024-03-18T15:29:14.234 INFO:teuthology.orchestra.run.smithi104.stdout: "forward": 4, 2024-03-18T15:29:14.234 INFO:teuthology.orchestra.run.smithi104.stdout: "dir_fetch_complete": 0, 2024-03-18T15:29:14.235 INFO:teuthology.orchestra.run.smithi104.stdout: "dir_fetch_keys": 0, 2024-03-18T15:29:14.235 INFO:teuthology.orchestra.run.smithi104.stdout: "dir_commit": 11, 2024-03-18T15:29:14.235 INFO:teuthology.orchestra.run.smithi104.stdout: "dir_split": 0, 2024-03-18T15:29:14.235 INFO:teuthology.orchestra.run.smithi104.stdout: "dir_merge": 0, 2024-03-18T15:29:14.235 INFO:teuthology.orchestra.run.smithi104.stdout: "inodes": 17, 2024-03-18T15:29:14.235 INFO:teuthology.orchestra.run.smithi104.stdout: "inodes_top": 0, 2024-03-18T15:29:14.235 INFO:teuthology.orchestra.run.smithi104.stdout: "inodes_bottom": 17, 2024-03-18T15:29:14.235 INFO:teuthology.orchestra.run.smithi104.stdout: "inodes_pin_tail": 0, 2024-03-18T15:29:14.235 INFO:teuthology.orchestra.run.smithi104.stdout: "inodes_pinned": 13, 2024-03-18T15:29:14.235 INFO:teuthology.orchestra.run.smithi104.stdout: "inodes_expired": 4, 2024-03-18T15:29:14.235 INFO:teuthology.orchestra.run.smithi104.stdout: "inodes_with_caps": 1, 2024-03-18T15:29:14.235 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": 1, 2024-03-18T15:29:14.235 INFO:teuthology.orchestra.run.smithi104.stdout: "subtrees": 4, 2024-03-18T15:29:14.235 INFO:teuthology.orchestra.run.smithi104.stdout: "traverse": 18, 2024-03-18T15:29:14.235 INFO:teuthology.orchestra.run.smithi104.stdout: "traverse_hit": 11, 2024-03-18T15:29:14.236 INFO:teuthology.orchestra.run.smithi104.stdout: "traverse_forward": 1, 2024-03-18T15:29:14.236 INFO:teuthology.orchestra.run.smithi104.stdout: "traverse_discover": 5, 2024-03-18T15:29:14.236 INFO:teuthology.orchestra.run.smithi104.stdout: "traverse_dir_fetch": 0, 2024-03-18T15:29:14.236 INFO:teuthology.orchestra.run.smithi104.stdout: "traverse_remote_ino": 0, 2024-03-18T15:29:14.236 INFO:teuthology.orchestra.run.smithi104.stdout: "traverse_lock": 0, 2024-03-18T15:29:14.236 INFO:teuthology.orchestra.run.smithi104.stdout: "load_cent": 0, 2024-03-18T15:29:14.236 INFO:teuthology.orchestra.run.smithi104.stdout: "q": 0, 2024-03-18T15:29:14.236 INFO:teuthology.orchestra.run.smithi104.stdout: "exported": 0, 2024-03-18T15:29:14.236 INFO:teuthology.orchestra.run.smithi104.stdout: "exported_inodes": 0, 2024-03-18T15:29:14.236 INFO:teuthology.orchestra.run.smithi104.stdout: "imported": 0, 2024-03-18T15:29:14.236 INFO:teuthology.orchestra.run.smithi104.stdout: "imported_inodes": 0, 2024-03-18T15:29:14.236 INFO:teuthology.orchestra.run.smithi104.stdout: "openino_dir_fetch": 0, 2024-03-18T15:29:14.236 INFO:teuthology.orchestra.run.smithi104.stdout: "openino_backtrace_fetch": 0, 2024-03-18T15:29:14.237 INFO:teuthology.orchestra.run.smithi104.stdout: "openino_peer_discover": 0, 2024-03-18T15:29:14.237 INFO:teuthology.orchestra.run.smithi104.stdout: "root_rfiles": 4, 2024-03-18T15:29:14.237 INFO:teuthology.orchestra.run.smithi104.stdout: "root_rbytes": 234, 2024-03-18T15:29:14.237 INFO:teuthology.orchestra.run.smithi104.stdout: "root_rsnaps": 0, 2024-03-18T15:29:14.237 INFO:teuthology.orchestra.run.smithi104.stdout: "scrub_backtrace_fetch": 0, 2024-03-18T15:29:14.237 INFO:teuthology.orchestra.run.smithi104.stdout: "scrub_set_tag": 0, 2024-03-18T15:29:14.237 INFO:teuthology.orchestra.run.smithi104.stdout: "scrub_backtrace_repaired": 0, 2024-03-18T15:29:14.237 INFO:teuthology.orchestra.run.smithi104.stdout: "scrub_inotable_repaired": 0, 2024-03-18T15:29:14.237 INFO:teuthology.orchestra.run.smithi104.stdout: "scrub_dir_inodes": 0, 2024-03-18T15:29:14.237 INFO:teuthology.orchestra.run.smithi104.stdout: "scrub_dir_base_inodes": 0, 2024-03-18T15:29:14.237 INFO:teuthology.orchestra.run.smithi104.stdout: "scrub_dirfrag_rstats": 0, 2024-03-18T15:29:14.237 INFO:teuthology.orchestra.run.smithi104.stdout: "scrub_file_inodes": 0, 2024-03-18T15:29:14.237 INFO:teuthology.orchestra.run.smithi104.stdout: "handle_inode_file_caps": 0, 2024-03-18T15:29:14.237 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph_cap_op_revoke": 0, 2024-03-18T15:29:14.240 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph_cap_op_grant": 0, 2024-03-18T15:29:14.240 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph_cap_op_trunc": 0, 2024-03-18T15:29:14.240 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-18T15:29:14.240 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-18T15:29:14.240 INFO:teuthology.orchestra.run.smithi104.stdout: "handle_client_caps": 0, 2024-03-18T15:29:14.240 INFO:teuthology.orchestra.run.smithi104.stdout: "handle_client_caps_dirty": 0, 2024-03-18T15:29:14.240 INFO:teuthology.orchestra.run.smithi104.stdout: "handle_client_cap_release": 0, 2024-03-18T15:29:14.241 INFO:teuthology.orchestra.run.smithi104.stdout: "process_request_cap_release": 0 2024-03-18T15:29:14.241 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:14.241 INFO:teuthology.orchestra.run.smithi104.stdout: "mds_cache": { 2024-03-18T15:29:14.241 INFO:teuthology.orchestra.run.smithi104.stdout: "dir_update": 0, 2024-03-18T15:29:14.241 INFO:teuthology.orchestra.run.smithi104.stdout: "dir_update_receipt": 13, 2024-03-18T15:29:14.241 INFO:teuthology.orchestra.run.smithi104.stdout: "dir_try_discover": 9, 2024-03-18T15:29:14.241 INFO:teuthology.orchestra.run.smithi104.stdout: "dir_send_discover": 11, 2024-03-18T15:29:14.241 INFO:teuthology.orchestra.run.smithi104.stdout: "dir_handle_discover": 0, 2024-03-18T15:29:14.241 INFO:teuthology.orchestra.run.smithi104.stdout: "num_strays": 0, 2024-03-18T15:29:14.241 INFO:teuthology.orchestra.run.smithi104.stdout: "num_strays_delayed": 0, 2024-03-18T15:29:14.241 INFO:teuthology.orchestra.run.smithi104.stdout: "num_strays_enqueuing": 0, 2024-03-18T15:29:14.241 INFO:teuthology.orchestra.run.smithi104.stdout: "strays_created": 0, 2024-03-18T15:29:14.241 INFO:teuthology.orchestra.run.smithi104.stdout: "strays_enqueued": 0, 2024-03-18T15:29:14.241 INFO:teuthology.orchestra.run.smithi104.stdout: "strays_reintegrated": 0, 2024-03-18T15:29:14.241 INFO:teuthology.orchestra.run.smithi104.stdout: "strays_migrated": 0, 2024-03-18T15:29:14.242 INFO:teuthology.orchestra.run.smithi104.stdout: "num_recovering_processing": 0, 2024-03-18T15:29:14.242 INFO:teuthology.orchestra.run.smithi104.stdout: "num_recovering_enqueued": 0, 2024-03-18T15:29:14.242 INFO:teuthology.orchestra.run.smithi104.stdout: "num_recovering_prioritized": 0, 2024-03-18T15:29:14.242 INFO:teuthology.orchestra.run.smithi104.stdout: "recovery_started": 0, 2024-03-18T15:29:14.242 INFO:teuthology.orchestra.run.smithi104.stdout: "recovery_completed": 0, 2024-03-18T15:29:14.242 INFO:teuthology.orchestra.run.smithi104.stdout: "ireq_quiesce_path": 0, 2024-03-18T15:29:14.242 INFO:teuthology.orchestra.run.smithi104.stdout: "ireq_quiesce_inode": 0, 2024-03-18T15:29:14.242 INFO:teuthology.orchestra.run.smithi104.stdout: "ireq_enqueue_scrub": 0, 2024-03-18T15:29:14.242 INFO:teuthology.orchestra.run.smithi104.stdout: "ireq_exportdir": 0, 2024-03-18T15:29:14.242 INFO:teuthology.orchestra.run.smithi104.stdout: "ireq_flush": 0, 2024-03-18T15:29:14.242 INFO:teuthology.orchestra.run.smithi104.stdout: "ireq_fragmentdir": 0, 2024-03-18T15:29:14.242 INFO:teuthology.orchestra.run.smithi104.stdout: "ireq_fragstats": 0, 2024-03-18T15:29:14.242 INFO:teuthology.orchestra.run.smithi104.stdout: "ireq_inodestats": 0 2024-03-18T15:29:14.242 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:14.242 INFO:teuthology.orchestra.run.smithi104.stdout: "mds_log": { 2024-03-18T15:29:14.243 INFO:teuthology.orchestra.run.smithi104.stdout: "evlrg": 0, 2024-03-18T15:29:14.243 INFO:teuthology.orchestra.run.smithi104.stdout: "evadd": 4, 2024-03-18T15:29:14.243 INFO:teuthology.orchestra.run.smithi104.stdout: "evex": 0, 2024-03-18T15:29:14.243 INFO:teuthology.orchestra.run.smithi104.stdout: "evtrm": 0, 2024-03-18T15:29:14.243 INFO:teuthology.orchestra.run.smithi104.stdout: "ev": 4, 2024-03-18T15:29:14.243 INFO:teuthology.orchestra.run.smithi104.stdout: "evexg": 0, 2024-03-18T15:29:14.243 INFO:teuthology.orchestra.run.smithi104.stdout: "evexd": 0, 2024-03-18T15:29:14.243 INFO:teuthology.orchestra.run.smithi104.stdout: "segadd": 2, 2024-03-18T15:29:14.243 INFO:teuthology.orchestra.run.smithi104.stdout: "segex": 0, 2024-03-18T15:29:14.243 INFO:teuthology.orchestra.run.smithi104.stdout: "segtrm": 0, 2024-03-18T15:29:14.243 INFO:teuthology.orchestra.run.smithi104.stdout: "seg": 2, 2024-03-18T15:29:14.243 INFO:teuthology.orchestra.run.smithi104.stdout: "segmjr": 2, 2024-03-18T15:29:14.243 INFO:teuthology.orchestra.run.smithi104.stdout: "segexg": 0, 2024-03-18T15:29:14.243 INFO:teuthology.orchestra.run.smithi104.stdout: "segexd": 0, 2024-03-18T15:29:14.244 INFO:teuthology.orchestra.run.smithi104.stdout: "expos": 4194304, 2024-03-18T15:29:14.244 INFO:teuthology.orchestra.run.smithi104.stdout: "wrpos": 4195593, 2024-03-18T15:29:14.244 INFO:teuthology.orchestra.run.smithi104.stdout: "rdpos": 0, 2024-03-18T15:29:14.244 INFO:teuthology.orchestra.run.smithi104.stdout: "jlat": { 2024-03-18T15:29:14.244 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 3, 2024-03-18T15:29:14.244 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.004790000, 2024-03-18T15:29:14.244 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.001596666 2024-03-18T15:29:14.244 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:14.244 INFO:teuthology.orchestra.run.smithi104.stdout: "replayed": 0 2024-03-18T15:29:14.244 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:14.244 INFO:teuthology.orchestra.run.smithi104.stdout: "mds_mem": { 2024-03-18T15:29:14.244 INFO:teuthology.orchestra.run.smithi104.stdout: "ino": 22, 2024-03-18T15:29:14.244 INFO:teuthology.orchestra.run.smithi104.stdout: "ino+": 24, 2024-03-18T15:29:14.244 INFO:teuthology.orchestra.run.smithi104.stdout: "ino-": 2, 2024-03-18T15:29:14.244 INFO:teuthology.orchestra.run.smithi104.stdout: "dir": 21, 2024-03-18T15:29:14.245 INFO:teuthology.orchestra.run.smithi104.stdout: "dir+": 25, 2024-03-18T15:29:14.245 INFO:teuthology.orchestra.run.smithi104.stdout: "dir-": 4, 2024-03-18T15:29:14.245 INFO:teuthology.orchestra.run.smithi104.stdout: "dn": 17, 2024-03-18T15:29:14.245 INFO:teuthology.orchestra.run.smithi104.stdout: "dn+": 21, 2024-03-18T15:29:14.245 INFO:teuthology.orchestra.run.smithi104.stdout: "dn-": 4, 2024-03-18T15:29:14.245 INFO:teuthology.orchestra.run.smithi104.stdout: "cap": 1, 2024-03-18T15:29:14.245 INFO:teuthology.orchestra.run.smithi104.stdout: "cap+": 1, 2024-03-18T15:29:14.245 INFO:teuthology.orchestra.run.smithi104.stdout: "cap-": 0, 2024-03-18T15:29:14.245 INFO:teuthology.orchestra.run.smithi104.stdout: "rss": 50620, 2024-03-18T15:29:14.245 INFO:teuthology.orchestra.run.smithi104.stdout: "heap": 223612 2024-03-18T15:29:14.245 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:14.245 INFO:teuthology.orchestra.run.smithi104.stdout: "mds_server": { 2024-03-18T15:29:14.245 INFO:teuthology.orchestra.run.smithi104.stdout: "dispatch_client_request": 7, 2024-03-18T15:29:14.245 INFO:teuthology.orchestra.run.smithi104.stdout: "dispatch_server_request": 1, 2024-03-18T15:29:14.245 INFO:teuthology.orchestra.run.smithi104.stdout: "handle_client_request": 7, 2024-03-18T15:29:14.245 INFO:teuthology.orchestra.run.smithi104.stdout: "handle_client_session": 18, 2024-03-18T15:29:14.246 INFO:teuthology.orchestra.run.smithi104.stdout: "handle_peer_request": 2, 2024-03-18T15:29:14.246 INFO:teuthology.orchestra.run.smithi104.stdout: "req_create_latency": { 2024-03-18T15:29:14.246 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:14.246 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:14.246 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:14.246 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:14.246 INFO:teuthology.orchestra.run.smithi104.stdout: "req_getattr_latency": { 2024-03-18T15:29:14.246 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 3, 2024-03-18T15:29:14.246 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000480120, 2024-03-18T15:29:14.246 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000160040 2024-03-18T15:29:14.246 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:14.246 INFO:teuthology.orchestra.run.smithi104.stdout: "req_getfilelock_latency": { 2024-03-18T15:29:14.246 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:14.246 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:14.247 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:14.247 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:14.247 INFO:teuthology.orchestra.run.smithi104.stdout: "req_link_latency": { 2024-03-18T15:29:14.247 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:14.247 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:14.247 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:14.247 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:14.247 INFO:teuthology.orchestra.run.smithi104.stdout: "req_lookup_latency": { 2024-03-18T15:29:14.247 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:14.247 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:14.247 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:14.247 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:14.247 INFO:teuthology.orchestra.run.smithi104.stdout: "req_lookuphash_latency": { 2024-03-18T15:29:14.247 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:14.247 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:14.248 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:14.248 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:14.248 INFO:teuthology.orchestra.run.smithi104.stdout: "req_lookupino_latency": { 2024-03-18T15:29:14.248 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:14.248 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:14.248 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:14.248 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:14.248 INFO:teuthology.orchestra.run.smithi104.stdout: "req_lookupname_latency": { 2024-03-18T15:29:14.248 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:14.248 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:14.248 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:14.248 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:14.248 INFO:teuthology.orchestra.run.smithi104.stdout: "req_lookupparent_latency": { 2024-03-18T15:29:14.248 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:14.248 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:14.249 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:14.249 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:14.249 INFO:teuthology.orchestra.run.smithi104.stdout: "req_lookupsnap_latency": { 2024-03-18T15:29:14.249 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:14.249 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:14.249 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:14.249 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:14.249 INFO:teuthology.orchestra.run.smithi104.stdout: "req_lssnap_latency": { 2024-03-18T15:29:14.249 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:14.249 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:14.249 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:14.249 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:14.249 INFO:teuthology.orchestra.run.smithi104.stdout: "req_mkdir_latency": { 2024-03-18T15:29:14.249 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:14.250 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:14.250 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:14.250 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:14.250 INFO:teuthology.orchestra.run.smithi104.stdout: "req_mknod_latency": { 2024-03-18T15:29:14.250 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:14.251 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:14.251 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:14.251 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:14.251 INFO:teuthology.orchestra.run.smithi104.stdout: "req_mksnap_latency": { 2024-03-18T15:29:14.251 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:14.251 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:14.251 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:14.251 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:14.251 INFO:teuthology.orchestra.run.smithi104.stdout: "req_open_latency": { 2024-03-18T15:29:14.251 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:14.251 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:14.251 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:14.251 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:14.251 INFO:teuthology.orchestra.run.smithi104.stdout: "req_readdir_latency": { 2024-03-18T15:29:14.251 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:14.252 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:14.252 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:14.252 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:14.252 INFO:teuthology.orchestra.run.smithi104.stdout: "req_rename_latency": { 2024-03-18T15:29:14.252 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:14.252 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:14.252 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:14.252 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:14.252 INFO:teuthology.orchestra.run.smithi104.stdout: "req_renamesnap_latency": { 2024-03-18T15:29:14.252 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:14.252 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:14.252 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:14.252 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:14.252 INFO:teuthology.orchestra.run.smithi104.stdout: "req_snapdiff_latency": { 2024-03-18T15:29:14.253 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:14.253 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:14.253 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:14.253 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:14.253 INFO:teuthology.orchestra.run.smithi104.stdout: "req_rmdir_latency": { 2024-03-18T15:29:14.253 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:14.253 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:14.253 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:14.253 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:14.253 INFO:teuthology.orchestra.run.smithi104.stdout: "req_rmsnap_latency": { 2024-03-18T15:29:14.253 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:14.253 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:14.253 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:14.253 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:14.254 INFO:teuthology.orchestra.run.smithi104.stdout: "req_rmxattr_latency": { 2024-03-18T15:29:14.254 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:14.254 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:14.254 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:14.254 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:14.254 INFO:teuthology.orchestra.run.smithi104.stdout: "req_setattr_latency": { 2024-03-18T15:29:14.254 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:14.254 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:14.254 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:14.254 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:14.254 INFO:teuthology.orchestra.run.smithi104.stdout: "req_setdirlayout_latency": { 2024-03-18T15:29:14.254 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:14.254 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:14.254 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:14.255 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:14.255 INFO:teuthology.orchestra.run.smithi104.stdout: "req_setfilelock_latency": { 2024-03-18T15:29:14.255 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:14.255 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:14.255 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:14.255 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:14.255 INFO:teuthology.orchestra.run.smithi104.stdout: "req_setlayout_latency": { 2024-03-18T15:29:14.255 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:14.255 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:14.255 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:14.255 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:14.255 INFO:teuthology.orchestra.run.smithi104.stdout: "req_setxattr_latency": { 2024-03-18T15:29:14.255 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:14.255 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:14.255 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:14.256 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:14.256 INFO:teuthology.orchestra.run.smithi104.stdout: "req_symlink_latency": { 2024-03-18T15:29:14.256 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:14.256 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:14.256 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:14.256 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:14.256 INFO:teuthology.orchestra.run.smithi104.stdout: "req_unlink_latency": { 2024-03-18T15:29:14.256 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:14.256 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:14.256 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:14.256 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:14.256 INFO:teuthology.orchestra.run.smithi104.stdout: "cap_revoke_eviction": 0, 2024-03-18T15:29:14.256 INFO:teuthology.orchestra.run.smithi104.stdout: "cap_acquisition_throttle": 0, 2024-03-18T15:29:14.256 INFO:teuthology.orchestra.run.smithi104.stdout: "req_getvxattr_latency": { 2024-03-18T15:29:14.257 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:14.257 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:14.257 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:14.257 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-03-18T15:29:14.257 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:14.257 INFO:teuthology.orchestra.run.smithi104.stdout: "mds_sessions": { 2024-03-18T15:29:14.257 INFO:teuthology.orchestra.run.smithi104.stdout: "session_count": 2, 2024-03-18T15:29:14.257 INFO:teuthology.orchestra.run.smithi104.stdout: "session_add": 2, 2024-03-18T15:29:14.257 INFO:teuthology.orchestra.run.smithi104.stdout: "session_remove": 0, 2024-03-18T15:29:14.257 INFO:teuthology.orchestra.run.smithi104.stdout: "sessions_open": 2, 2024-03-18T15:29:14.257 INFO:teuthology.orchestra.run.smithi104.stdout: "sessions_stale": 0, 2024-03-18T15:29:14.257 INFO:teuthology.orchestra.run.smithi104.stdout: "total_load": 2, 2024-03-18T15:29:14.257 INFO:teuthology.orchestra.run.smithi104.stdout: "average_load": 1, 2024-03-18T15:29:14.257 INFO:teuthology.orchestra.run.smithi104.stdout: "avg_session_uptime": 634, 2024-03-18T15:29:14.257 INFO:teuthology.orchestra.run.smithi104.stdout: "mdthresh_evicted": 0 2024-03-18T15:29:14.258 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:14.258 INFO:teuthology.orchestra.run.smithi104.stdout: "mempool": { 2024-03-18T15:29:14.258 INFO:teuthology.orchestra.run.smithi104.stdout: "bloom_filter_bytes": 0, 2024-03-18T15:29:14.258 INFO:teuthology.orchestra.run.smithi104.stdout: "bloom_filter_items": 0, 2024-03-18T15:29:14.258 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T15:29:14.258 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_alloc_items": 0, 2024-03-18T15:29:14.258 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T15:29:14.258 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_cache_data_items": 0, 2024-03-18T15:29:14.258 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T15:29:14.258 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T15:29:14.258 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T15:29:14.258 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T15:29:14.258 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T15:29:14.258 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_cache_other_items": 0, 2024-03-18T15:29:14.259 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T15:29:14.259 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T15:29:14.259 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_extent_bytes": 0, 2024-03-18T15:29:14.259 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_extent_items": 0, 2024-03-18T15:29:14.259 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_blob_bytes": 0, 2024-03-18T15:29:14.259 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_blob_items": 0, 2024-03-18T15:29:14.259 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T15:29:14.259 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T15:29:14.259 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T15:29:14.259 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T15:29:14.259 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T15:29:14.259 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_fsck_items": 0, 2024-03-18T15:29:14.259 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_txc_bytes": 0, 2024-03-18T15:29:14.259 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_txc_items": 0, 2024-03-18T15:29:14.259 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T15:29:14.260 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T15:29:14.260 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_writing_bytes": 0, 2024-03-18T15:29:14.260 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_writing_items": 0, 2024-03-18T15:29:14.260 INFO:teuthology.orchestra.run.smithi104.stdout: "bluefs_bytes": 0, 2024-03-18T15:29:14.260 INFO:teuthology.orchestra.run.smithi104.stdout: "bluefs_items": 0, 2024-03-18T15:29:14.260 INFO:teuthology.orchestra.run.smithi104.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T15:29:14.260 INFO:teuthology.orchestra.run.smithi104.stdout: "bluefs_file_reader_items": 0, 2024-03-18T15:29:14.260 INFO:teuthology.orchestra.run.smithi104.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T15:29:14.260 INFO:teuthology.orchestra.run.smithi104.stdout: "bluefs_file_writer_items": 0, 2024-03-18T15:29:14.260 INFO:teuthology.orchestra.run.smithi104.stdout: "buffer_anon_bytes": 256709, 2024-03-18T15:29:14.260 INFO:teuthology.orchestra.run.smithi104.stdout: "buffer_anon_items": 97, 2024-03-18T15:29:14.260 INFO:teuthology.orchestra.run.smithi104.stdout: "buffer_meta_bytes": 176, 2024-03-18T15:29:14.260 INFO:teuthology.orchestra.run.smithi104.stdout: "buffer_meta_items": 2, 2024-03-18T15:29:14.260 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_bytes": 0, 2024-03-18T15:29:14.260 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_items": 0, 2024-03-18T15:29:14.261 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_mapbl_bytes": 0, 2024-03-18T15:29:14.261 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_mapbl_items": 0, 2024-03-18T15:29:14.261 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_pglog_bytes": 0, 2024-03-18T15:29:14.261 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_pglog_items": 0, 2024-03-18T15:29:14.261 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap_bytes": 10168, 2024-03-18T15:29:14.261 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap_items": 213, 2024-03-18T15:29:14.261 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T15:29:14.261 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap_mapping_items": 0, 2024-03-18T15:29:14.261 INFO:teuthology.orchestra.run.smithi104.stdout: "pgmap_bytes": 0, 2024-03-18T15:29:14.261 INFO:teuthology.orchestra.run.smithi104.stdout: "pgmap_items": 0, 2024-03-18T15:29:14.261 INFO:teuthology.orchestra.run.smithi104.stdout: "mds_co_bytes": 90293, 2024-03-18T15:29:14.261 INFO:teuthology.orchestra.run.smithi104.stdout: "mds_co_items": 349, 2024-03-18T15:29:14.261 INFO:teuthology.orchestra.run.smithi104.stdout: "unittest_1_bytes": 0, 2024-03-18T15:29:14.261 INFO:teuthology.orchestra.run.smithi104.stdout: "unittest_1_items": 0, 2024-03-18T15:29:14.261 INFO:teuthology.orchestra.run.smithi104.stdout: "unittest_2_bytes": 0, 2024-03-18T15:29:14.261 INFO:teuthology.orchestra.run.smithi104.stdout: "unittest_2_items": 0 2024-03-18T15:29:14.262 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:14.262 INFO:teuthology.orchestra.run.smithi104.stdout: "objecter": { 2024-03-18T15:29:14.262 INFO:teuthology.orchestra.run.smithi104.stdout: "op_active": 0, 2024-03-18T15:29:14.262 INFO:teuthology.orchestra.run.smithi104.stdout: "op_laggy": 0, 2024-03-18T15:29:14.262 INFO:teuthology.orchestra.run.smithi104.stdout: "op_send": 39, 2024-03-18T15:29:14.262 INFO:teuthology.orchestra.run.smithi104.stdout: "op_send_bytes": 11591, 2024-03-18T15:29:14.262 INFO:teuthology.orchestra.run.smithi104.stdout: "op_resend": 33, 2024-03-18T15:29:14.262 INFO:teuthology.orchestra.run.smithi104.stdout: "op_reply": 39, 2024-03-18T15:29:14.262 INFO:teuthology.orchestra.run.smithi104.stdout: "op_latency": { 2024-03-18T15:29:14.262 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 39, 2024-03-18T15:29:14.262 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.257996842, 2024-03-18T15:29:14.262 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.006615303 2024-03-18T15:29:14.262 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:14.262 INFO:teuthology.orchestra.run.smithi104.stdout: "op_inflight": 0, 2024-03-18T15:29:14.263 INFO:teuthology.orchestra.run.smithi104.stdout: "oplen_avg": { 2024-03-18T15:29:14.263 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 39, 2024-03-18T15:29:14.263 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 61 2024-03-18T15:29:14.263 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:14.263 INFO:teuthology.orchestra.run.smithi104.stdout: "op": 39, 2024-03-18T15:29:14.263 INFO:teuthology.orchestra.run.smithi104.stdout: "op_r": 0, 2024-03-18T15:29:14.263 INFO:teuthology.orchestra.run.smithi104.stdout: "op_w": 39, 2024-03-18T15:29:14.263 INFO:teuthology.orchestra.run.smithi104.stdout: "op_rmw": 0, 2024-03-18T15:29:14.263 INFO:teuthology.orchestra.run.smithi104.stdout: "op_pg": 0, 2024-03-18T15:29:14.263 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_stat": 0, 2024-03-18T15:29:14.263 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_create": 10, 2024-03-18T15:29:14.263 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_read": 0, 2024-03-18T15:29:14.263 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_write": 3, 2024-03-18T15:29:14.263 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_writefull": 7, 2024-03-18T15:29:14.263 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_writesame": 0, 2024-03-18T15:29:14.264 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_append": 0, 2024-03-18T15:29:14.264 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_zero": 0, 2024-03-18T15:29:14.264 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_truncate": 0, 2024-03-18T15:29:14.264 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_delete": 6, 2024-03-18T15:29:14.264 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_mapext": 0, 2024-03-18T15:29:14.264 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_sparse_read": 0, 2024-03-18T15:29:14.264 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_clonerange": 0, 2024-03-18T15:29:14.264 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_getxattr": 0, 2024-03-18T15:29:14.264 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_setxattr": 20, 2024-03-18T15:29:14.264 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_cmpxattr": 0, 2024-03-18T15:29:14.264 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_rmxattr": 0, 2024-03-18T15:29:14.264 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_resetxattrs": 0, 2024-03-18T15:29:14.264 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_call": 0, 2024-03-18T15:29:14.264 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_watch": 0, 2024-03-18T15:29:14.264 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_notify": 0, 2024-03-18T15:29:14.265 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_src_cmpxattr": 0, 2024-03-18T15:29:14.265 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_pgls": 0, 2024-03-18T15:29:14.265 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_pgls_filter": 0, 2024-03-18T15:29:14.265 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_other": 0, 2024-03-18T15:29:14.265 INFO:teuthology.orchestra.run.smithi104.stdout: "linger_active": 0, 2024-03-18T15:29:14.265 INFO:teuthology.orchestra.run.smithi104.stdout: "linger_send": 0, 2024-03-18T15:29:14.265 INFO:teuthology.orchestra.run.smithi104.stdout: "linger_resend": 0, 2024-03-18T15:29:14.265 INFO:teuthology.orchestra.run.smithi104.stdout: "linger_ping": 0, 2024-03-18T15:29:14.265 INFO:teuthology.orchestra.run.smithi104.stdout: "poolop_active": 0, 2024-03-18T15:29:14.265 INFO:teuthology.orchestra.run.smithi104.stdout: "poolop_send": 0, 2024-03-18T15:29:14.265 INFO:teuthology.orchestra.run.smithi104.stdout: "poolop_resend": 0, 2024-03-18T15:29:14.265 INFO:teuthology.orchestra.run.smithi104.stdout: "poolstat_active": 0, 2024-03-18T15:29:14.265 INFO:teuthology.orchestra.run.smithi104.stdout: "poolstat_send": 0, 2024-03-18T15:29:14.265 INFO:teuthology.orchestra.run.smithi104.stdout: "poolstat_resend": 0, 2024-03-18T15:29:14.265 INFO:teuthology.orchestra.run.smithi104.stdout: "statfs_active": 0, 2024-03-18T15:29:14.265 INFO:teuthology.orchestra.run.smithi104.stdout: "statfs_send": 0, 2024-03-18T15:29:14.266 INFO:teuthology.orchestra.run.smithi104.stdout: "statfs_resend": 0, 2024-03-18T15:29:14.266 INFO:teuthology.orchestra.run.smithi104.stdout: "command_active": 0, 2024-03-18T15:29:14.266 INFO:teuthology.orchestra.run.smithi104.stdout: "command_send": 0, 2024-03-18T15:29:14.266 INFO:teuthology.orchestra.run.smithi104.stdout: "command_resend": 0, 2024-03-18T15:29:14.266 INFO:teuthology.orchestra.run.smithi104.stdout: "map_epoch": 104, 2024-03-18T15:29:14.266 INFO:teuthology.orchestra.run.smithi104.stdout: "map_full": 0, 2024-03-18T15:29:14.266 INFO:teuthology.orchestra.run.smithi104.stdout: "map_inc": 15, 2024-03-18T15:29:14.266 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_sessions": 9, 2024-03-18T15:29:14.266 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_session_open": 9, 2024-03-18T15:29:14.266 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_session_close": 0, 2024-03-18T15:29:14.266 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_laggy": 0, 2024-03-18T15:29:14.266 INFO:teuthology.orchestra.run.smithi104.stdout: "omap_wr": 14, 2024-03-18T15:29:14.266 INFO:teuthology.orchestra.run.smithi104.stdout: "omap_rd": 0, 2024-03-18T15:29:14.266 INFO:teuthology.orchestra.run.smithi104.stdout: "omap_del": 1 2024-03-18T15:29:14.267 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:14.267 INFO:teuthology.orchestra.run.smithi104.stdout: "oft": { 2024-03-18T15:29:14.267 INFO:teuthology.orchestra.run.smithi104.stdout: "omap_total_objs": 1, 2024-03-18T15:29:14.267 INFO:teuthology.orchestra.run.smithi104.stdout: "omap_total_kv_pairs": 0, 2024-03-18T15:29:14.267 INFO:teuthology.orchestra.run.smithi104.stdout: "omap_total_updates": 0, 2024-03-18T15:29:14.267 INFO:teuthology.orchestra.run.smithi104.stdout: "omap_total_removes": 0 2024-03-18T15:29:14.267 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:14.267 INFO:teuthology.orchestra.run.smithi104.stdout: "osd-slow-ops": { 2024-03-18T15:29:14.267 INFO:teuthology.orchestra.run.smithi104.stdout: "slow_ops_count": 0 2024-03-18T15:29:14.267 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:14.267 INFO:teuthology.orchestra.run.smithi104.stdout: "purge_queue": { 2024-03-18T15:29:14.267 INFO:teuthology.orchestra.run.smithi104.stdout: "pq_executing_ops": 0, 2024-03-18T15:29:14.267 INFO:teuthology.orchestra.run.smithi104.stdout: "pq_executing_ops_high_water": 0, 2024-03-18T15:29:14.267 INFO:teuthology.orchestra.run.smithi104.stdout: "pq_executing": 0, 2024-03-18T15:29:14.267 INFO:teuthology.orchestra.run.smithi104.stdout: "pq_executing_high_water": 0, 2024-03-18T15:29:14.268 INFO:teuthology.orchestra.run.smithi104.stdout: "pq_executed_ops": 0, 2024-03-18T15:29:14.268 INFO:teuthology.orchestra.run.smithi104.stdout: "pq_executed": 0, 2024-03-18T15:29:14.268 INFO:teuthology.orchestra.run.smithi104.stdout: "pq_item_in_journal": 0 2024-03-18T15:29:14.268 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:14.268 INFO:teuthology.orchestra.run.smithi104.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T15:29:14.268 INFO:teuthology.orchestra.run.smithi104.stdout: "val": 0, 2024-03-18T15:29:14.268 INFO:teuthology.orchestra.run.smithi104.stdout: "max": 104857600, 2024-03-18T15:29:14.268 INFO:teuthology.orchestra.run.smithi104.stdout: "get_started": 0, 2024-03-18T15:29:14.268 INFO:teuthology.orchestra.run.smithi104.stdout: "get": 731, 2024-03-18T15:29:14.268 INFO:teuthology.orchestra.run.smithi104.stdout: "get_sum": 1892394, 2024-03-18T15:29:14.268 INFO:teuthology.orchestra.run.smithi104.stdout: "get_or_fail_fail": 0, 2024-03-18T15:29:14.268 INFO:teuthology.orchestra.run.smithi104.stdout: "get_or_fail_success": 731, 2024-03-18T15:29:14.268 INFO:teuthology.orchestra.run.smithi104.stdout: "take": 0, 2024-03-18T15:29:14.268 INFO:teuthology.orchestra.run.smithi104.stdout: "take_sum": 0, 2024-03-18T15:29:14.268 INFO:teuthology.orchestra.run.smithi104.stdout: "put": 731, 2024-03-18T15:29:14.269 INFO:teuthology.orchestra.run.smithi104.stdout: "put_sum": 1892394, 2024-03-18T15:29:14.269 INFO:teuthology.orchestra.run.smithi104.stdout: "wait": { 2024-03-18T15:29:14.269 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:14.269 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:14.269 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:14.269 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-03-18T15:29:14.269 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:14.269 INFO:teuthology.orchestra.run.smithi104.stdout: "throttle-objecter_bytes": { 2024-03-18T15:29:14.269 INFO:teuthology.orchestra.run.smithi104.stdout: "val": 0, 2024-03-18T15:29:14.269 INFO:teuthology.orchestra.run.smithi104.stdout: "max": 104857600, 2024-03-18T15:29:14.269 INFO:teuthology.orchestra.run.smithi104.stdout: "get_started": 0, 2024-03-18T15:29:14.269 INFO:teuthology.orchestra.run.smithi104.stdout: "get": 0, 2024-03-18T15:29:14.269 INFO:teuthology.orchestra.run.smithi104.stdout: "get_sum": 0, 2024-03-18T15:29:14.269 INFO:teuthology.orchestra.run.smithi104.stdout: "get_or_fail_fail": 0, 2024-03-18T15:29:14.269 INFO:teuthology.orchestra.run.smithi104.stdout: "get_or_fail_success": 0, 2024-03-18T15:29:14.270 INFO:teuthology.orchestra.run.smithi104.stdout: "take": 39, 2024-03-18T15:29:14.270 INFO:teuthology.orchestra.run.smithi104.stdout: "take_sum": 11591, 2024-03-18T15:29:14.270 INFO:teuthology.orchestra.run.smithi104.stdout: "put": 39, 2024-03-18T15:29:14.270 INFO:teuthology.orchestra.run.smithi104.stdout: "put_sum": 11591, 2024-03-18T15:29:14.270 INFO:teuthology.orchestra.run.smithi104.stdout: "wait": { 2024-03-18T15:29:14.270 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:14.270 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:14.270 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:14.270 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-03-18T15:29:14.270 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:14.270 INFO:teuthology.orchestra.run.smithi104.stdout: "throttle-objecter_ops": { 2024-03-18T15:29:14.270 INFO:teuthology.orchestra.run.smithi104.stdout: "val": 0, 2024-03-18T15:29:14.270 INFO:teuthology.orchestra.run.smithi104.stdout: "max": 1024, 2024-03-18T15:29:14.270 INFO:teuthology.orchestra.run.smithi104.stdout: "get_started": 0, 2024-03-18T15:29:14.270 INFO:teuthology.orchestra.run.smithi104.stdout: "get": 0, 2024-03-18T15:29:14.271 INFO:teuthology.orchestra.run.smithi104.stdout: "get_sum": 0, 2024-03-18T15:29:14.271 INFO:teuthology.orchestra.run.smithi104.stdout: "get_or_fail_fail": 0, 2024-03-18T15:29:14.271 INFO:teuthology.orchestra.run.smithi104.stdout: "get_or_fail_success": 0, 2024-03-18T15:29:14.271 INFO:teuthology.orchestra.run.smithi104.stdout: "take": 39, 2024-03-18T15:29:14.271 INFO:teuthology.orchestra.run.smithi104.stdout: "take_sum": 39, 2024-03-18T15:29:14.271 INFO:teuthology.orchestra.run.smithi104.stdout: "put": 39, 2024-03-18T15:29:14.271 INFO:teuthology.orchestra.run.smithi104.stdout: "put_sum": 39, 2024-03-18T15:29:14.271 INFO:teuthology.orchestra.run.smithi104.stdout: "wait": { 2024-03-18T15:29:14.271 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:14.271 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:14.271 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:14.271 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-03-18T15:29:14.271 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:14.271 INFO:teuthology.orchestra.run.smithi104.stdout: "throttle-write_buf_throttle": { 2024-03-18T15:29:14.271 INFO:teuthology.orchestra.run.smithi104.stdout: "val": 0, 2024-03-18T15:29:14.272 INFO:teuthology.orchestra.run.smithi104.stdout: "max": 3758096384, 2024-03-18T15:29:14.272 INFO:teuthology.orchestra.run.smithi104.stdout: "get_started": 0, 2024-03-18T15:29:14.272 INFO:teuthology.orchestra.run.smithi104.stdout: "get": 0, 2024-03-18T15:29:14.272 INFO:teuthology.orchestra.run.smithi104.stdout: "get_sum": 0, 2024-03-18T15:29:14.272 INFO:teuthology.orchestra.run.smithi104.stdout: "get_or_fail_fail": 0, 2024-03-18T15:29:14.272 INFO:teuthology.orchestra.run.smithi104.stdout: "get_or_fail_success": 0, 2024-03-18T15:29:14.272 INFO:teuthology.orchestra.run.smithi104.stdout: "take": 0, 2024-03-18T15:29:14.272 INFO:teuthology.orchestra.run.smithi104.stdout: "take_sum": 0, 2024-03-18T15:29:14.272 INFO:teuthology.orchestra.run.smithi104.stdout: "put": 0, 2024-03-18T15:29:14.272 INFO:teuthology.orchestra.run.smithi104.stdout: "put_sum": 0, 2024-03-18T15:29:14.272 INFO:teuthology.orchestra.run.smithi104.stdout: "wait": { 2024-03-18T15:29:14.273 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:14.273 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:14.273 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:14.273 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-03-18T15:29:14.273 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:14.273 INFO:teuthology.orchestra.run.smithi104.stdout: "throttle-write_buf_throttle-0x55b397b68500": { 2024-03-18T15:29:14.273 INFO:teuthology.orchestra.run.smithi104.stdout: "val": 0, 2024-03-18T15:29:14.273 INFO:teuthology.orchestra.run.smithi104.stdout: "max": 3758096384, 2024-03-18T15:29:14.273 INFO:teuthology.orchestra.run.smithi104.stdout: "get_started": 0, 2024-03-18T15:29:14.273 INFO:teuthology.orchestra.run.smithi104.stdout: "get": 4, 2024-03-18T15:29:14.273 INFO:teuthology.orchestra.run.smithi104.stdout: "get_sum": 1289, 2024-03-18T15:29:14.273 INFO:teuthology.orchestra.run.smithi104.stdout: "get_or_fail_fail": 0, 2024-03-18T15:29:14.273 INFO:teuthology.orchestra.run.smithi104.stdout: "get_or_fail_success": 4, 2024-03-18T15:29:14.273 INFO:teuthology.orchestra.run.smithi104.stdout: "take": 0, 2024-03-18T15:29:14.273 INFO:teuthology.orchestra.run.smithi104.stdout: "take_sum": 0, 2024-03-18T15:29:14.274 INFO:teuthology.orchestra.run.smithi104.stdout: "put": 3, 2024-03-18T15:29:14.274 INFO:teuthology.orchestra.run.smithi104.stdout: "put_sum": 1289, 2024-03-18T15:29:14.274 INFO:teuthology.orchestra.run.smithi104.stdout: "wait": { 2024-03-18T15:29:14.274 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:14.274 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:14.274 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:14.274 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-03-18T15:29:14.274 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-03-18T15:29:14.274 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-03-18T15:29:14.962 INFO:tasks.check_counter:Daemon mds.i mds.exported=0 2024-03-18T15:29:14.962 INFO:tasks.check_counter:Daemon mds.i mds.imported=0 2024-03-18T15:29:14.962 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update=0 2024-03-18T15:29:14.962 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update_receipt=13 2024-03-18T15:29:14.962 INFO:tasks.check_counter:Daemon mds.i mds.root_rsnaps=0 2024-03-18T15:29:14.962 INFO:tasks.check_counter:Daemon mds.i mds_server.req_mksnap_latency.avgcount=0 2024-03-18T15:29:14.962 INFO:tasks.check_counter:Daemon mds.i mds_server.req_rmsnap_latency.avgcount=0 2024-03-18T15:29:14.962 DEBUG:tasks.check_counter:Getting stats from l 2024-03-18T15:29:14.963 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 -- ceph daemon mds.l perf dump 2024-03-18T15:29:15.236 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T15:29:15.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:29:15 smithi104 ceph-mon[36070]: pgmap v335: 129 pgs: 129 active+clean; 1.8 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-18T15:29:15.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:29:15 smithi019 ceph-mon[31522]: pgmap v335: 129 pgs: 129 active+clean; 1.8 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-18T15:29:16.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:29:15 smithi033 ceph-mon[36648]: pgmap v335: 129 pgs: 129 active+clean; 1.8 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-18T15:29:17.098 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-03-18T15:29:17.099 INFO:teuthology.orchestra.run.smithi104.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T15:29:17.099 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_recv_messages": 1631, 2024-03-18T15:29:17.099 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_send_messages": 1631, 2024-03-18T15:29:17.099 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_recv_bytes": 491171, 2024-03-18T15:29:17.099 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_send_bytes": 517835, 2024-03-18T15:29:17.099 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_created_connections": 6, 2024-03-18T15:29:17.099 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_active_connections": 2, 2024-03-18T15:29:17.099 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_running_total_time": 0.106182284, 2024-03-18T15:29:17.099 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_running_send_time": 0.035211512, 2024-03-18T15:29:17.099 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_running_recv_time": 0.027395328, 2024-03-18T15:29:17.099 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_running_fast_dispatch_time": 0.034578622, 2024-03-18T15:29:17.099 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T15:29:17.099 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 1630, 2024-03-18T15:29:17.100 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.033604408, 2024-03-18T15:29:17.100 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000020616 2024-03-18T15:29:17.100 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:17.100 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_handle_ack_lat": { 2024-03-18T15:29:17.100 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:17.100 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:17.100 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:17.100 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:17.100 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_recv_encrypted_bytes": 3072, 2024-03-18T15:29:17.100 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_send_encrypted_bytes": 448 2024-03-18T15:29:17.100 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:17.100 INFO:teuthology.orchestra.run.smithi104.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T15:29:17.100 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_recv_messages": 935, 2024-03-18T15:29:17.100 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_send_messages": 918, 2024-03-18T15:29:17.100 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_recv_bytes": 1944458, 2024-03-18T15:29:17.101 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_send_bytes": 360202, 2024-03-18T15:29:17.101 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_created_connections": 4, 2024-03-18T15:29:17.101 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_active_connections": 3, 2024-03-18T15:29:17.101 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_running_total_time": 0.089458873, 2024-03-18T15:29:17.101 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_running_send_time": 0.025228282, 2024-03-18T15:29:17.101 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_running_recv_time": 0.050116651, 2024-03-18T15:29:17.101 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_running_fast_dispatch_time": 0.023161904, 2024-03-18T15:29:17.101 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T15:29:17.101 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 917, 2024-03-18T15:29:17.101 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.020525676, 2024-03-18T15:29:17.101 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000022383 2024-03-18T15:29:17.101 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:17.101 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_handle_ack_lat": { 2024-03-18T15:29:17.101 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:17.102 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:17.102 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:17.102 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:17.102 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_recv_encrypted_bytes": 1781200, 2024-03-18T15:29:17.102 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_send_encrypted_bytes": 149936 2024-03-18T15:29:17.102 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:17.102 INFO:teuthology.orchestra.run.smithi104.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T15:29:17.102 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_recv_messages": 3, 2024-03-18T15:29:17.102 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_send_messages": 184, 2024-03-18T15:29:17.102 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_recv_bytes": 588, 2024-03-18T15:29:17.102 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_send_bytes": 545070, 2024-03-18T15:29:17.102 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_created_connections": 7, 2024-03-18T15:29:17.102 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_active_connections": 4, 2024-03-18T15:29:17.102 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_running_total_time": 0.008690600, 2024-03-18T15:29:17.103 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_running_send_time": 0.005288651, 2024-03-18T15:29:17.103 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_running_recv_time": 0.001833175, 2024-03-18T15:29:17.103 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_running_fast_dispatch_time": 0.000016299, 2024-03-18T15:29:17.103 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T15:29:17.103 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 181, 2024-03-18T15:29:17.103 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.023732279, 2024-03-18T15:29:17.103 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000131117 2024-03-18T15:29:17.103 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:17.103 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_handle_ack_lat": { 2024-03-18T15:29:17.103 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:17.103 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:17.103 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:17.103 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:17.103 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-18T15:29:17.103 INFO:teuthology.orchestra.run.smithi104.stdout: "msgr_send_encrypted_bytes": 544752 2024-03-18T15:29:17.104 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:17.104 INFO:teuthology.orchestra.run.smithi104.stdout: "cct": { 2024-03-18T15:29:17.104 INFO:teuthology.orchestra.run.smithi104.stdout: "total_workers": 1, 2024-03-18T15:29:17.104 INFO:teuthology.orchestra.run.smithi104.stdout: "unhealthy_workers": 0 2024-03-18T15:29:17.104 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:17.104 INFO:teuthology.orchestra.run.smithi104.stdout: "finisher-MDSRank": { 2024-03-18T15:29:17.104 INFO:teuthology.orchestra.run.smithi104.stdout: "queue_len": 0, 2024-03-18T15:29:17.104 INFO:teuthology.orchestra.run.smithi104.stdout: "complete_latency": { 2024-03-18T15:29:17.104 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 3877, 2024-03-18T15:29:17.104 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.081922292, 2024-03-18T15:29:17.104 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000021130 2024-03-18T15:29:17.104 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-03-18T15:29:17.104 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:17.104 INFO:teuthology.orchestra.run.smithi104.stdout: "finisher-PurgeQueue": { 2024-03-18T15:29:17.105 INFO:teuthology.orchestra.run.smithi104.stdout: "queue_len": 0, 2024-03-18T15:29:17.105 INFO:teuthology.orchestra.run.smithi104.stdout: "complete_latency": { 2024-03-18T15:29:17.105 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:17.105 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:17.105 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:17.105 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-03-18T15:29:17.105 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:17.105 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": { 2024-03-18T15:29:17.105 INFO:teuthology.orchestra.run.smithi104.stdout: "request": 0, 2024-03-18T15:29:17.105 INFO:teuthology.orchestra.run.smithi104.stdout: "reply": 0, 2024-03-18T15:29:17.105 INFO:teuthology.orchestra.run.smithi104.stdout: "reply_latency": { 2024-03-18T15:29:17.105 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:17.105 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:17.105 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:17.105 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:17.106 INFO:teuthology.orchestra.run.smithi104.stdout: "slow_reply": 0, 2024-03-18T15:29:17.106 INFO:teuthology.orchestra.run.smithi104.stdout: "forward": 0, 2024-03-18T15:29:17.106 INFO:teuthology.orchestra.run.smithi104.stdout: "dir_fetch_complete": 0, 2024-03-18T15:29:17.106 INFO:teuthology.orchestra.run.smithi104.stdout: "dir_fetch_keys": 0, 2024-03-18T15:29:17.106 INFO:teuthology.orchestra.run.smithi104.stdout: "dir_commit": 0, 2024-03-18T15:29:17.106 INFO:teuthology.orchestra.run.smithi104.stdout: "dir_split": 0, 2024-03-18T15:29:17.106 INFO:teuthology.orchestra.run.smithi104.stdout: "dir_merge": 0, 2024-03-18T15:29:17.106 INFO:teuthology.orchestra.run.smithi104.stdout: "inodes": 0, 2024-03-18T15:29:17.106 INFO:teuthology.orchestra.run.smithi104.stdout: "inodes_top": 0, 2024-03-18T15:29:17.106 INFO:teuthology.orchestra.run.smithi104.stdout: "inodes_bottom": 0, 2024-03-18T15:29:17.106 INFO:teuthology.orchestra.run.smithi104.stdout: "inodes_pin_tail": 0, 2024-03-18T15:29:17.106 INFO:teuthology.orchestra.run.smithi104.stdout: "inodes_pinned": 0, 2024-03-18T15:29:17.106 INFO:teuthology.orchestra.run.smithi104.stdout: "inodes_expired": 0, 2024-03-18T15:29:17.106 INFO:teuthology.orchestra.run.smithi104.stdout: "inodes_with_caps": 0, 2024-03-18T15:29:17.107 INFO:teuthology.orchestra.run.smithi104.stdout: "caps": 0, 2024-03-18T15:29:17.107 INFO:teuthology.orchestra.run.smithi104.stdout: "subtrees": 1, 2024-03-18T15:29:17.107 INFO:teuthology.orchestra.run.smithi104.stdout: "traverse": 0, 2024-03-18T15:29:17.107 INFO:teuthology.orchestra.run.smithi104.stdout: "traverse_hit": 0, 2024-03-18T15:29:17.107 INFO:teuthology.orchestra.run.smithi104.stdout: "traverse_forward": 0, 2024-03-18T15:29:17.107 INFO:teuthology.orchestra.run.smithi104.stdout: "traverse_discover": 0, 2024-03-18T15:29:17.107 INFO:teuthology.orchestra.run.smithi104.stdout: "traverse_dir_fetch": 0, 2024-03-18T15:29:17.107 INFO:teuthology.orchestra.run.smithi104.stdout: "traverse_remote_ino": 0, 2024-03-18T15:29:17.107 INFO:teuthology.orchestra.run.smithi104.stdout: "traverse_lock": 0, 2024-03-18T15:29:17.107 INFO:teuthology.orchestra.run.smithi104.stdout: "load_cent": 0, 2024-03-18T15:29:17.107 INFO:teuthology.orchestra.run.smithi104.stdout: "q": 0, 2024-03-18T15:29:17.107 INFO:teuthology.orchestra.run.smithi104.stdout: "exported": 0, 2024-03-18T15:29:17.107 INFO:teuthology.orchestra.run.smithi104.stdout: "exported_inodes": 0, 2024-03-18T15:29:17.107 INFO:teuthology.orchestra.run.smithi104.stdout: "imported": 0, 2024-03-18T15:29:17.108 INFO:teuthology.orchestra.run.smithi104.stdout: "imported_inodes": 0, 2024-03-18T15:29:17.108 INFO:teuthology.orchestra.run.smithi104.stdout: "openino_dir_fetch": 0, 2024-03-18T15:29:17.108 INFO:teuthology.orchestra.run.smithi104.stdout: "openino_backtrace_fetch": 0, 2024-03-18T15:29:17.108 INFO:teuthology.orchestra.run.smithi104.stdout: "openino_peer_discover": 0, 2024-03-18T15:29:17.108 INFO:teuthology.orchestra.run.smithi104.stdout: "root_rfiles": 0, 2024-03-18T15:29:17.108 INFO:teuthology.orchestra.run.smithi104.stdout: "root_rbytes": 0, 2024-03-18T15:29:17.108 INFO:teuthology.orchestra.run.smithi104.stdout: "root_rsnaps": 0, 2024-03-18T15:29:17.108 INFO:teuthology.orchestra.run.smithi104.stdout: "scrub_backtrace_fetch": 0, 2024-03-18T15:29:17.108 INFO:teuthology.orchestra.run.smithi104.stdout: "scrub_set_tag": 0, 2024-03-18T15:29:17.108 INFO:teuthology.orchestra.run.smithi104.stdout: "scrub_backtrace_repaired": 0, 2024-03-18T15:29:17.108 INFO:teuthology.orchestra.run.smithi104.stdout: "scrub_inotable_repaired": 0, 2024-03-18T15:29:17.108 INFO:teuthology.orchestra.run.smithi104.stdout: "scrub_dir_inodes": 0, 2024-03-18T15:29:17.108 INFO:teuthology.orchestra.run.smithi104.stdout: "scrub_dir_base_inodes": 0, 2024-03-18T15:29:17.108 INFO:teuthology.orchestra.run.smithi104.stdout: "scrub_dirfrag_rstats": 0, 2024-03-18T15:29:17.108 INFO:teuthology.orchestra.run.smithi104.stdout: "scrub_file_inodes": 0, 2024-03-18T15:29:17.109 INFO:teuthology.orchestra.run.smithi104.stdout: "handle_inode_file_caps": 0, 2024-03-18T15:29:17.109 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph_cap_op_revoke": 0, 2024-03-18T15:29:17.109 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph_cap_op_grant": 0, 2024-03-18T15:29:17.109 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph_cap_op_trunc": 0, 2024-03-18T15:29:17.109 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-18T15:29:17.109 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-18T15:29:17.109 INFO:teuthology.orchestra.run.smithi104.stdout: "handle_client_caps": 0, 2024-03-18T15:29:17.109 INFO:teuthology.orchestra.run.smithi104.stdout: "handle_client_caps_dirty": 0, 2024-03-18T15:29:17.109 INFO:teuthology.orchestra.run.smithi104.stdout: "handle_client_cap_release": 0, 2024-03-18T15:29:17.109 INFO:teuthology.orchestra.run.smithi104.stdout: "process_request_cap_release": 0 2024-03-18T15:29:17.109 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:17.109 INFO:teuthology.orchestra.run.smithi104.stdout: "mds_cache": { 2024-03-18T15:29:17.109 INFO:teuthology.orchestra.run.smithi104.stdout: "dir_update": 0, 2024-03-18T15:29:17.109 INFO:teuthology.orchestra.run.smithi104.stdout: "dir_update_receipt": 0, 2024-03-18T15:29:17.109 INFO:teuthology.orchestra.run.smithi104.stdout: "dir_try_discover": 0, 2024-03-18T15:29:17.110 INFO:teuthology.orchestra.run.smithi104.stdout: "dir_send_discover": 0, 2024-03-18T15:29:17.110 INFO:teuthology.orchestra.run.smithi104.stdout: "dir_handle_discover": 0, 2024-03-18T15:29:17.110 INFO:teuthology.orchestra.run.smithi104.stdout: "num_strays": 0, 2024-03-18T15:29:17.110 INFO:teuthology.orchestra.run.smithi104.stdout: "num_strays_delayed": 0, 2024-03-18T15:29:17.110 INFO:teuthology.orchestra.run.smithi104.stdout: "num_strays_enqueuing": 0, 2024-03-18T15:29:17.110 INFO:teuthology.orchestra.run.smithi104.stdout: "strays_created": 0, 2024-03-18T15:29:17.110 INFO:teuthology.orchestra.run.smithi104.stdout: "strays_enqueued": 0, 2024-03-18T15:29:17.110 INFO:teuthology.orchestra.run.smithi104.stdout: "strays_reintegrated": 0, 2024-03-18T15:29:17.110 INFO:teuthology.orchestra.run.smithi104.stdout: "strays_migrated": 0, 2024-03-18T15:29:17.110 INFO:teuthology.orchestra.run.smithi104.stdout: "num_recovering_processing": 0, 2024-03-18T15:29:17.110 INFO:teuthology.orchestra.run.smithi104.stdout: "num_recovering_enqueued": 0, 2024-03-18T15:29:17.110 INFO:teuthology.orchestra.run.smithi104.stdout: "num_recovering_prioritized": 0, 2024-03-18T15:29:17.110 INFO:teuthology.orchestra.run.smithi104.stdout: "recovery_started": 0, 2024-03-18T15:29:17.110 INFO:teuthology.orchestra.run.smithi104.stdout: "recovery_completed": 0, 2024-03-18T15:29:17.111 INFO:teuthology.orchestra.run.smithi104.stdout: "ireq_quiesce_path": 0, 2024-03-18T15:29:17.111 INFO:teuthology.orchestra.run.smithi104.stdout: "ireq_quiesce_inode": 0, 2024-03-18T15:29:17.111 INFO:teuthology.orchestra.run.smithi104.stdout: "ireq_enqueue_scrub": 0, 2024-03-18T15:29:17.111 INFO:teuthology.orchestra.run.smithi104.stdout: "ireq_exportdir": 0, 2024-03-18T15:29:17.111 INFO:teuthology.orchestra.run.smithi104.stdout: "ireq_flush": 0, 2024-03-18T15:29:17.111 INFO:teuthology.orchestra.run.smithi104.stdout: "ireq_fragmentdir": 0, 2024-03-18T15:29:17.111 INFO:teuthology.orchestra.run.smithi104.stdout: "ireq_fragstats": 0, 2024-03-18T15:29:17.111 INFO:teuthology.orchestra.run.smithi104.stdout: "ireq_inodestats": 0 2024-03-18T15:29:17.111 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:17.111 INFO:teuthology.orchestra.run.smithi104.stdout: "mds_log": { 2024-03-18T15:29:17.111 INFO:teuthology.orchestra.run.smithi104.stdout: "evlrg": 0, 2024-03-18T15:29:17.111 INFO:teuthology.orchestra.run.smithi104.stdout: "evadd": 0, 2024-03-18T15:29:17.111 INFO:teuthology.orchestra.run.smithi104.stdout: "evex": 0, 2024-03-18T15:29:17.111 INFO:teuthology.orchestra.run.smithi104.stdout: "evtrm": 0, 2024-03-18T15:29:17.111 INFO:teuthology.orchestra.run.smithi104.stdout: "ev": 7, 2024-03-18T15:29:17.112 INFO:teuthology.orchestra.run.smithi104.stdout: "evexg": 0, 2024-03-18T15:29:17.112 INFO:teuthology.orchestra.run.smithi104.stdout: "evexd": 0, 2024-03-18T15:29:17.112 INFO:teuthology.orchestra.run.smithi104.stdout: "segadd": 0, 2024-03-18T15:29:17.112 INFO:teuthology.orchestra.run.smithi104.stdout: "segex": 0, 2024-03-18T15:29:17.112 INFO:teuthology.orchestra.run.smithi104.stdout: "segtrm": 0, 2024-03-18T15:29:17.112 INFO:teuthology.orchestra.run.smithi104.stdout: "seg": 2, 2024-03-18T15:29:17.112 INFO:teuthology.orchestra.run.smithi104.stdout: "segmjr": 2, 2024-03-18T15:29:17.112 INFO:teuthology.orchestra.run.smithi104.stdout: "segexg": 0, 2024-03-18T15:29:17.112 INFO:teuthology.orchestra.run.smithi104.stdout: "segexd": 0, 2024-03-18T15:29:17.112 INFO:teuthology.orchestra.run.smithi104.stdout: "expos": 4194304, 2024-03-18T15:29:17.112 INFO:teuthology.orchestra.run.smithi104.stdout: "wrpos": 4196266, 2024-03-18T15:29:17.112 INFO:teuthology.orchestra.run.smithi104.stdout: "rdpos": 4196125, 2024-03-18T15:29:17.112 INFO:teuthology.orchestra.run.smithi104.stdout: "jlat": { 2024-03-18T15:29:17.112 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:17.112 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:17.113 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:17.113 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:17.113 INFO:teuthology.orchestra.run.smithi104.stdout: "replayed": 7 2024-03-18T15:29:17.113 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:17.113 INFO:teuthology.orchestra.run.smithi104.stdout: "mds_mem": { 2024-03-18T15:29:17.113 INFO:teuthology.orchestra.run.smithi104.stdout: "ino": 3, 2024-03-18T15:29:17.113 INFO:teuthology.orchestra.run.smithi104.stdout: "ino+": 3, 2024-03-18T15:29:17.113 INFO:teuthology.orchestra.run.smithi104.stdout: "ino-": 0, 2024-03-18T15:29:17.113 INFO:teuthology.orchestra.run.smithi104.stdout: "dir": 1, 2024-03-18T15:29:17.113 INFO:teuthology.orchestra.run.smithi104.stdout: "dir+": 1, 2024-03-18T15:29:17.113 INFO:teuthology.orchestra.run.smithi104.stdout: "dir-": 0, 2024-03-18T15:29:17.113 INFO:teuthology.orchestra.run.smithi104.stdout: "dn": 0, 2024-03-18T15:29:17.113 INFO:teuthology.orchestra.run.smithi104.stdout: "dn+": 0, 2024-03-18T15:29:17.113 INFO:teuthology.orchestra.run.smithi104.stdout: "dn-": 0, 2024-03-18T15:29:17.113 INFO:teuthology.orchestra.run.smithi104.stdout: "cap": 0, 2024-03-18T15:29:17.114 INFO:teuthology.orchestra.run.smithi104.stdout: "cap+": 0, 2024-03-18T15:29:17.114 INFO:teuthology.orchestra.run.smithi104.stdout: "cap-": 0, 2024-03-18T15:29:17.114 INFO:teuthology.orchestra.run.smithi104.stdout: "rss": 49920, 2024-03-18T15:29:17.114 INFO:teuthology.orchestra.run.smithi104.stdout: "heap": 215420 2024-03-18T15:29:17.114 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:17.114 INFO:teuthology.orchestra.run.smithi104.stdout: "mds_server": { 2024-03-18T15:29:17.114 INFO:teuthology.orchestra.run.smithi104.stdout: "dispatch_client_request": 0, 2024-03-18T15:29:17.114 INFO:teuthology.orchestra.run.smithi104.stdout: "dispatch_server_request": 0, 2024-03-18T15:29:17.114 INFO:teuthology.orchestra.run.smithi104.stdout: "handle_client_request": 0, 2024-03-18T15:29:17.114 INFO:teuthology.orchestra.run.smithi104.stdout: "handle_client_session": 0, 2024-03-18T15:29:17.114 INFO:teuthology.orchestra.run.smithi104.stdout: "handle_peer_request": 0, 2024-03-18T15:29:17.114 INFO:teuthology.orchestra.run.smithi104.stdout: "req_create_latency": { 2024-03-18T15:29:17.117 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:17.117 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:17.117 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:17.117 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:17.118 INFO:teuthology.orchestra.run.smithi104.stdout: "req_getattr_latency": { 2024-03-18T15:29:17.118 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:17.118 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:17.118 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:17.118 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:17.118 INFO:teuthology.orchestra.run.smithi104.stdout: "req_getfilelock_latency": { 2024-03-18T15:29:17.118 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:17.119 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:17.119 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:17.119 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:17.119 INFO:teuthology.orchestra.run.smithi104.stdout: "req_link_latency": { 2024-03-18T15:29:17.119 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:17.119 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:17.119 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:17.119 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:17.119 INFO:teuthology.orchestra.run.smithi104.stdout: "req_lookup_latency": { 2024-03-18T15:29:17.119 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:17.119 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:17.120 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:17.120 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:17.120 INFO:teuthology.orchestra.run.smithi104.stdout: "req_lookuphash_latency": { 2024-03-18T15:29:17.120 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:17.120 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:17.120 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:17.120 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:17.120 INFO:teuthology.orchestra.run.smithi104.stdout: "req_lookupino_latency": { 2024-03-18T15:29:17.120 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:17.120 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:17.120 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:17.121 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:17.121 INFO:teuthology.orchestra.run.smithi104.stdout: "req_lookupname_latency": { 2024-03-18T15:29:17.121 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:17.121 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:17.121 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:17.121 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:17.121 INFO:teuthology.orchestra.run.smithi104.stdout: "req_lookupparent_latency": { 2024-03-18T15:29:17.121 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:17.121 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:17.121 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:17.121 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:17.121 INFO:teuthology.orchestra.run.smithi104.stdout: "req_lookupsnap_latency": { 2024-03-18T15:29:17.122 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:17.122 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:17.122 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:17.122 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:17.122 INFO:teuthology.orchestra.run.smithi104.stdout: "req_lssnap_latency": { 2024-03-18T15:29:17.122 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:17.122 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:17.122 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:17.122 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:17.122 INFO:teuthology.orchestra.run.smithi104.stdout: "req_mkdir_latency": { 2024-03-18T15:29:17.122 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:17.123 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:17.124 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:17.124 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:17.124 INFO:teuthology.orchestra.run.smithi104.stdout: "req_mknod_latency": { 2024-03-18T15:29:17.124 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:17.124 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:17.124 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:17.124 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:17.124 INFO:teuthology.orchestra.run.smithi104.stdout: "req_mksnap_latency": { 2024-03-18T15:29:17.124 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:17.124 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:17.125 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:17.125 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:17.125 INFO:teuthology.orchestra.run.smithi104.stdout: "req_open_latency": { 2024-03-18T15:29:17.125 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:17.125 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:17.125 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:17.125 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:17.125 INFO:teuthology.orchestra.run.smithi104.stdout: "req_readdir_latency": { 2024-03-18T15:29:17.125 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:17.125 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:17.126 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:17.126 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:17.126 INFO:teuthology.orchestra.run.smithi104.stdout: "req_rename_latency": { 2024-03-18T15:29:17.126 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:17.126 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:17.126 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:17.126 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:17.126 INFO:teuthology.orchestra.run.smithi104.stdout: "req_renamesnap_latency": { 2024-03-18T15:29:17.126 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:17.126 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:17.126 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:17.127 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:17.127 INFO:teuthology.orchestra.run.smithi104.stdout: "req_snapdiff_latency": { 2024-03-18T15:29:17.127 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:17.127 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:17.127 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:17.127 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:17.127 INFO:teuthology.orchestra.run.smithi104.stdout: "req_rmdir_latency": { 2024-03-18T15:29:17.127 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:17.128 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:17.128 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:17.128 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:17.128 INFO:teuthology.orchestra.run.smithi104.stdout: "req_rmsnap_latency": { 2024-03-18T15:29:17.128 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:17.128 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:17.128 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:17.128 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:17.128 INFO:teuthology.orchestra.run.smithi104.stdout: "req_rmxattr_latency": { 2024-03-18T15:29:17.128 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:17.128 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:17.129 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:17.129 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:17.129 INFO:teuthology.orchestra.run.smithi104.stdout: "req_setattr_latency": { 2024-03-18T15:29:17.129 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:17.129 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:17.129 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:17.129 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:17.129 INFO:teuthology.orchestra.run.smithi104.stdout: "req_setdirlayout_latency": { 2024-03-18T15:29:17.129 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:17.129 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:17.129 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:17.129 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:17.129 INFO:teuthology.orchestra.run.smithi104.stdout: "req_setfilelock_latency": { 2024-03-18T15:29:17.129 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:17.130 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:17.130 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:17.130 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:17.130 INFO:teuthology.orchestra.run.smithi104.stdout: "req_setlayout_latency": { 2024-03-18T15:29:17.130 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:17.130 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:17.130 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:17.130 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:17.130 INFO:teuthology.orchestra.run.smithi104.stdout: "req_setxattr_latency": { 2024-03-18T15:29:17.130 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:17.130 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:17.130 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:17.131 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:17.131 INFO:teuthology.orchestra.run.smithi104.stdout: "req_symlink_latency": { 2024-03-18T15:29:17.131 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:17.131 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:17.131 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:17.131 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:17.131 INFO:teuthology.orchestra.run.smithi104.stdout: "req_unlink_latency": { 2024-03-18T15:29:17.131 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:17.131 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:17.131 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:17.131 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:17.131 INFO:teuthology.orchestra.run.smithi104.stdout: "cap_revoke_eviction": 0, 2024-03-18T15:29:17.132 INFO:teuthology.orchestra.run.smithi104.stdout: "cap_acquisition_throttle": 0, 2024-03-18T15:29:17.132 INFO:teuthology.orchestra.run.smithi104.stdout: "req_getvxattr_latency": { 2024-03-18T15:29:17.132 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:17.132 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:17.132 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:17.132 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-03-18T15:29:17.132 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:17.132 INFO:teuthology.orchestra.run.smithi104.stdout: "mds_sessions": { 2024-03-18T15:29:17.132 INFO:teuthology.orchestra.run.smithi104.stdout: "session_count": 1, 2024-03-18T15:29:17.132 INFO:teuthology.orchestra.run.smithi104.stdout: "session_add": 3, 2024-03-18T15:29:17.132 INFO:teuthology.orchestra.run.smithi104.stdout: "session_remove": 2, 2024-03-18T15:29:17.132 INFO:teuthology.orchestra.run.smithi104.stdout: "sessions_open": 3, 2024-03-18T15:29:17.132 INFO:teuthology.orchestra.run.smithi104.stdout: "sessions_stale": 0, 2024-03-18T15:29:17.133 INFO:teuthology.orchestra.run.smithi104.stdout: "total_load": 0, 2024-03-18T15:29:17.133 INFO:teuthology.orchestra.run.smithi104.stdout: "average_load": 0, 2024-03-18T15:29:17.133 INFO:teuthology.orchestra.run.smithi104.stdout: "avg_session_uptime": 5585, 2024-03-18T15:29:17.133 INFO:teuthology.orchestra.run.smithi104.stdout: "mdthresh_evicted": 0 2024-03-18T15:29:17.133 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:17.133 INFO:teuthology.orchestra.run.smithi104.stdout: "mempool": { 2024-03-18T15:29:17.133 INFO:teuthology.orchestra.run.smithi104.stdout: "bloom_filter_bytes": 0, 2024-03-18T15:29:17.133 INFO:teuthology.orchestra.run.smithi104.stdout: "bloom_filter_items": 0, 2024-03-18T15:29:17.133 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T15:29:17.133 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_alloc_items": 0, 2024-03-18T15:29:17.133 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T15:29:17.133 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_cache_data_items": 0, 2024-03-18T15:29:17.133 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T15:29:17.133 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T15:29:17.133 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T15:29:17.134 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T15:29:17.134 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T15:29:17.134 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_cache_other_items": 0, 2024-03-18T15:29:17.134 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T15:29:17.134 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T15:29:17.134 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_extent_bytes": 0, 2024-03-18T15:29:17.134 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_extent_items": 0, 2024-03-18T15:29:17.134 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_blob_bytes": 0, 2024-03-18T15:29:17.134 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_blob_items": 0, 2024-03-18T15:29:17.134 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T15:29:17.134 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T15:29:17.134 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T15:29:17.135 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T15:29:17.135 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T15:29:17.135 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_fsck_items": 0, 2024-03-18T15:29:17.135 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_txc_bytes": 0, 2024-03-18T15:29:17.135 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_txc_items": 0, 2024-03-18T15:29:17.135 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T15:29:17.135 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T15:29:17.135 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_writing_bytes": 0, 2024-03-18T15:29:17.135 INFO:teuthology.orchestra.run.smithi104.stdout: "bluestore_writing_items": 0, 2024-03-18T15:29:17.135 INFO:teuthology.orchestra.run.smithi104.stdout: "bluefs_bytes": 0, 2024-03-18T15:29:17.135 INFO:teuthology.orchestra.run.smithi104.stdout: "bluefs_items": 0, 2024-03-18T15:29:17.135 INFO:teuthology.orchestra.run.smithi104.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T15:29:17.135 INFO:teuthology.orchestra.run.smithi104.stdout: "bluefs_file_reader_items": 0, 2024-03-18T15:29:17.136 INFO:teuthology.orchestra.run.smithi104.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T15:29:17.136 INFO:teuthology.orchestra.run.smithi104.stdout: "bluefs_file_writer_items": 0, 2024-03-18T15:29:17.136 INFO:teuthology.orchestra.run.smithi104.stdout: "buffer_anon_bytes": 190139, 2024-03-18T15:29:17.136 INFO:teuthology.orchestra.run.smithi104.stdout: "buffer_anon_items": 52, 2024-03-18T15:29:17.136 INFO:teuthology.orchestra.run.smithi104.stdout: "buffer_meta_bytes": 0, 2024-03-18T15:29:17.136 INFO:teuthology.orchestra.run.smithi104.stdout: "buffer_meta_items": 0, 2024-03-18T15:29:17.136 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_bytes": 0, 2024-03-18T15:29:17.136 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_items": 0, 2024-03-18T15:29:17.136 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_mapbl_bytes": 0, 2024-03-18T15:29:17.136 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_mapbl_items": 0, 2024-03-18T15:29:17.136 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_pglog_bytes": 0, 2024-03-18T15:29:17.136 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_pglog_items": 0, 2024-03-18T15:29:17.136 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap_bytes": 10168, 2024-03-18T15:29:17.137 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap_items": 213, 2024-03-18T15:29:17.137 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T15:29:17.137 INFO:teuthology.orchestra.run.smithi104.stdout: "osdmap_mapping_items": 0, 2024-03-18T15:29:17.137 INFO:teuthology.orchestra.run.smithi104.stdout: "pgmap_bytes": 0, 2024-03-18T15:29:17.137 INFO:teuthology.orchestra.run.smithi104.stdout: "pgmap_items": 0, 2024-03-18T15:29:17.137 INFO:teuthology.orchestra.run.smithi104.stdout: "mds_co_bytes": 8464, 2024-03-18T15:29:17.137 INFO:teuthology.orchestra.run.smithi104.stdout: "mds_co_items": 13, 2024-03-18T15:29:17.137 INFO:teuthology.orchestra.run.smithi104.stdout: "unittest_1_bytes": 0, 2024-03-18T15:29:17.137 INFO:teuthology.orchestra.run.smithi104.stdout: "unittest_1_items": 0, 2024-03-18T15:29:17.137 INFO:teuthology.orchestra.run.smithi104.stdout: "unittest_2_bytes": 0, 2024-03-18T15:29:17.137 INFO:teuthology.orchestra.run.smithi104.stdout: "unittest_2_items": 0 2024-03-18T15:29:17.137 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:17.137 INFO:teuthology.orchestra.run.smithi104.stdout: "objecter": { 2024-03-18T15:29:17.138 INFO:teuthology.orchestra.run.smithi104.stdout: "op_active": 0, 2024-03-18T15:29:17.138 INFO:teuthology.orchestra.run.smithi104.stdout: "op_laggy": 0, 2024-03-18T15:29:17.138 INFO:teuthology.orchestra.run.smithi104.stdout: "op_send": 2289, 2024-03-18T15:29:17.138 INFO:teuthology.orchestra.run.smithi104.stdout: "op_send_bytes": 21, 2024-03-18T15:29:17.138 INFO:teuthology.orchestra.run.smithi104.stdout: "op_resend": 3, 2024-03-18T15:29:17.138 INFO:teuthology.orchestra.run.smithi104.stdout: "op_reply": 2289, 2024-03-18T15:29:17.138 INFO:teuthology.orchestra.run.smithi104.stdout: "op_latency": { 2024-03-18T15:29:17.138 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 2289, 2024-03-18T15:29:17.138 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.888988997, 2024-03-18T15:29:17.138 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000388374 2024-03-18T15:29:17.138 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:17.138 INFO:teuthology.orchestra.run.smithi104.stdout: "op_inflight": 0, 2024-03-18T15:29:17.138 INFO:teuthology.orchestra.run.smithi104.stdout: "oplen_avg": { 2024-03-18T15:29:17.138 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 2289, 2024-03-18T15:29:17.139 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 2290 2024-03-18T15:29:17.139 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:17.139 INFO:teuthology.orchestra.run.smithi104.stdout: "op": 2289, 2024-03-18T15:29:17.139 INFO:teuthology.orchestra.run.smithi104.stdout: "op_r": 2289, 2024-03-18T15:29:17.139 INFO:teuthology.orchestra.run.smithi104.stdout: "op_w": 0, 2024-03-18T15:29:17.139 INFO:teuthology.orchestra.run.smithi104.stdout: "op_rmw": 0, 2024-03-18T15:29:17.139 INFO:teuthology.orchestra.run.smithi104.stdout: "op_pg": 0, 2024-03-18T15:29:17.139 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_stat": 1468, 2024-03-18T15:29:17.139 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_create": 0, 2024-03-18T15:29:17.139 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_read": 819, 2024-03-18T15:29:17.139 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_write": 0, 2024-03-18T15:29:17.139 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_writefull": 0, 2024-03-18T15:29:17.139 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_writesame": 0, 2024-03-18T15:29:17.139 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_append": 0, 2024-03-18T15:29:17.140 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_zero": 0, 2024-03-18T15:29:17.140 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_truncate": 0, 2024-03-18T15:29:17.140 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_delete": 0, 2024-03-18T15:29:17.140 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_mapext": 0, 2024-03-18T15:29:17.140 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_sparse_read": 0, 2024-03-18T15:29:17.140 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_clonerange": 0, 2024-03-18T15:29:17.140 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_getxattr": 1, 2024-03-18T15:29:17.140 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_setxattr": 0, 2024-03-18T15:29:17.140 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_cmpxattr": 0, 2024-03-18T15:29:17.140 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_rmxattr": 0, 2024-03-18T15:29:17.140 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_resetxattrs": 0, 2024-03-18T15:29:17.140 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_call": 0, 2024-03-18T15:29:17.140 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_watch": 0, 2024-03-18T15:29:17.140 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_notify": 0, 2024-03-18T15:29:17.141 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_src_cmpxattr": 0, 2024-03-18T15:29:17.141 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_pgls": 0, 2024-03-18T15:29:17.141 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_pgls_filter": 0, 2024-03-18T15:29:17.141 INFO:teuthology.orchestra.run.smithi104.stdout: "osdop_other": 0, 2024-03-18T15:29:17.141 INFO:teuthology.orchestra.run.smithi104.stdout: "linger_active": 0, 2024-03-18T15:29:17.141 INFO:teuthology.orchestra.run.smithi104.stdout: "linger_send": 0, 2024-03-18T15:29:17.141 INFO:teuthology.orchestra.run.smithi104.stdout: "linger_resend": 0, 2024-03-18T15:29:17.141 INFO:teuthology.orchestra.run.smithi104.stdout: "linger_ping": 0, 2024-03-18T15:29:17.141 INFO:teuthology.orchestra.run.smithi104.stdout: "poolop_active": 0, 2024-03-18T15:29:17.141 INFO:teuthology.orchestra.run.smithi104.stdout: "poolop_send": 0, 2024-03-18T15:29:17.141 INFO:teuthology.orchestra.run.smithi104.stdout: "poolop_resend": 0, 2024-03-18T15:29:17.141 INFO:teuthology.orchestra.run.smithi104.stdout: "poolstat_active": 0, 2024-03-18T15:29:17.141 INFO:teuthology.orchestra.run.smithi104.stdout: "poolstat_send": 0, 2024-03-18T15:29:17.142 INFO:teuthology.orchestra.run.smithi104.stdout: "poolstat_resend": 0, 2024-03-18T15:29:17.142 INFO:teuthology.orchestra.run.smithi104.stdout: "statfs_active": 0, 2024-03-18T15:29:17.142 INFO:teuthology.orchestra.run.smithi104.stdout: "statfs_send": 0, 2024-03-18T15:29:17.142 INFO:teuthology.orchestra.run.smithi104.stdout: "statfs_resend": 0, 2024-03-18T15:29:17.142 INFO:teuthology.orchestra.run.smithi104.stdout: "command_active": 0, 2024-03-18T15:29:17.142 INFO:teuthology.orchestra.run.smithi104.stdout: "command_send": 0, 2024-03-18T15:29:17.142 INFO:teuthology.orchestra.run.smithi104.stdout: "command_resend": 0, 2024-03-18T15:29:17.142 INFO:teuthology.orchestra.run.smithi104.stdout: "map_epoch": 104, 2024-03-18T15:29:17.142 INFO:teuthology.orchestra.run.smithi104.stdout: "map_full": 0, 2024-03-18T15:29:17.142 INFO:teuthology.orchestra.run.smithi104.stdout: "map_inc": 15, 2024-03-18T15:29:17.142 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_sessions": 5, 2024-03-18T15:29:17.142 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_session_open": 5, 2024-03-18T15:29:17.143 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_session_close": 0, 2024-03-18T15:29:17.143 INFO:teuthology.orchestra.run.smithi104.stdout: "osd_laggy": 0, 2024-03-18T15:29:17.143 INFO:teuthology.orchestra.run.smithi104.stdout: "omap_wr": 0, 2024-03-18T15:29:17.143 INFO:teuthology.orchestra.run.smithi104.stdout: "omap_rd": 2, 2024-03-18T15:29:17.143 INFO:teuthology.orchestra.run.smithi104.stdout: "omap_del": 0 2024-03-18T15:29:17.143 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:17.143 INFO:teuthology.orchestra.run.smithi104.stdout: "oft": { 2024-03-18T15:29:17.143 INFO:teuthology.orchestra.run.smithi104.stdout: "omap_total_objs": 0, 2024-03-18T15:29:17.143 INFO:teuthology.orchestra.run.smithi104.stdout: "omap_total_kv_pairs": 0, 2024-03-18T15:29:17.143 INFO:teuthology.orchestra.run.smithi104.stdout: "omap_total_updates": 0, 2024-03-18T15:29:17.143 INFO:teuthology.orchestra.run.smithi104.stdout: "omap_total_removes": 0 2024-03-18T15:29:17.144 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:17.144 INFO:teuthology.orchestra.run.smithi104.stdout: "osd-slow-ops": { 2024-03-18T15:29:17.144 INFO:teuthology.orchestra.run.smithi104.stdout: "slow_ops_count": 0 2024-03-18T15:29:17.144 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:17.144 INFO:teuthology.orchestra.run.smithi104.stdout: "purge_queue": { 2024-03-18T15:29:17.144 INFO:teuthology.orchestra.run.smithi104.stdout: "pq_executing_ops": 0, 2024-03-18T15:29:17.144 INFO:teuthology.orchestra.run.smithi104.stdout: "pq_executing_ops_high_water": 0, 2024-03-18T15:29:17.144 INFO:teuthology.orchestra.run.smithi104.stdout: "pq_executing": 0, 2024-03-18T15:29:17.144 INFO:teuthology.orchestra.run.smithi104.stdout: "pq_executing_high_water": 0, 2024-03-18T15:29:17.144 INFO:teuthology.orchestra.run.smithi104.stdout: "pq_executed_ops": 0, 2024-03-18T15:29:17.144 INFO:teuthology.orchestra.run.smithi104.stdout: "pq_executed": 0, 2024-03-18T15:29:17.145 INFO:teuthology.orchestra.run.smithi104.stdout: "pq_item_in_journal": 0 2024-03-18T15:29:17.145 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:17.145 INFO:teuthology.orchestra.run.smithi104.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T15:29:17.145 INFO:teuthology.orchestra.run.smithi104.stdout: "val": 0, 2024-03-18T15:29:17.145 INFO:teuthology.orchestra.run.smithi104.stdout: "max": 104857600, 2024-03-18T15:29:17.145 INFO:teuthology.orchestra.run.smithi104.stdout: "get_started": 0, 2024-03-18T15:29:17.145 INFO:teuthology.orchestra.run.smithi104.stdout: "get": 2565, 2024-03-18T15:29:17.145 INFO:teuthology.orchestra.run.smithi104.stdout: "get_sum": 2188274, 2024-03-18T15:29:17.145 INFO:teuthology.orchestra.run.smithi104.stdout: "get_or_fail_fail": 0, 2024-03-18T15:29:17.145 INFO:teuthology.orchestra.run.smithi104.stdout: "get_or_fail_success": 2565, 2024-03-18T15:29:17.145 INFO:teuthology.orchestra.run.smithi104.stdout: "take": 0, 2024-03-18T15:29:17.145 INFO:teuthology.orchestra.run.smithi104.stdout: "take_sum": 0, 2024-03-18T15:29:17.146 INFO:teuthology.orchestra.run.smithi104.stdout: "put": 2565, 2024-03-18T15:29:17.146 INFO:teuthology.orchestra.run.smithi104.stdout: "put_sum": 2188274, 2024-03-18T15:29:17.146 INFO:teuthology.orchestra.run.smithi104.stdout: "wait": { 2024-03-18T15:29:17.146 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:17.146 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:17.146 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:17.146 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-03-18T15:29:17.146 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:17.146 INFO:teuthology.orchestra.run.smithi104.stdout: "throttle-objecter_bytes": { 2024-03-18T15:29:17.146 INFO:teuthology.orchestra.run.smithi104.stdout: "val": 0, 2024-03-18T15:29:17.146 INFO:teuthology.orchestra.run.smithi104.stdout: "max": 104857600, 2024-03-18T15:29:17.146 INFO:teuthology.orchestra.run.smithi104.stdout: "get_started": 0, 2024-03-18T15:29:17.147 INFO:teuthology.orchestra.run.smithi104.stdout: "get": 0, 2024-03-18T15:29:17.147 INFO:teuthology.orchestra.run.smithi104.stdout: "get_sum": 0, 2024-03-18T15:29:17.147 INFO:teuthology.orchestra.run.smithi104.stdout: "get_or_fail_fail": 0, 2024-03-18T15:29:17.147 INFO:teuthology.orchestra.run.smithi104.stdout: "get_or_fail_success": 0, 2024-03-18T15:29:17.147 INFO:teuthology.orchestra.run.smithi104.stdout: "take": 2289, 2024-03-18T15:29:17.147 INFO:teuthology.orchestra.run.smithi104.stdout: "take_sum": 1967, 2024-03-18T15:29:17.147 INFO:teuthology.orchestra.run.smithi104.stdout: "put": 2289, 2024-03-18T15:29:17.147 INFO:teuthology.orchestra.run.smithi104.stdout: "put_sum": 1967, 2024-03-18T15:29:17.147 INFO:teuthology.orchestra.run.smithi104.stdout: "wait": { 2024-03-18T15:29:17.147 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:17.147 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:17.148 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:17.148 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-03-18T15:29:17.148 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:17.148 INFO:teuthology.orchestra.run.smithi104.stdout: "throttle-objecter_ops": { 2024-03-18T15:29:17.148 INFO:teuthology.orchestra.run.smithi104.stdout: "val": 0, 2024-03-18T15:29:17.148 INFO:teuthology.orchestra.run.smithi104.stdout: "max": 1024, 2024-03-18T15:29:17.148 INFO:teuthology.orchestra.run.smithi104.stdout: "get_started": 0, 2024-03-18T15:29:17.148 INFO:teuthology.orchestra.run.smithi104.stdout: "get": 0, 2024-03-18T15:29:17.148 INFO:teuthology.orchestra.run.smithi104.stdout: "get_sum": 0, 2024-03-18T15:29:17.148 INFO:teuthology.orchestra.run.smithi104.stdout: "get_or_fail_fail": 0, 2024-03-18T15:29:17.148 INFO:teuthology.orchestra.run.smithi104.stdout: "get_or_fail_success": 0, 2024-03-18T15:29:17.149 INFO:teuthology.orchestra.run.smithi104.stdout: "take": 2289, 2024-03-18T15:29:17.149 INFO:teuthology.orchestra.run.smithi104.stdout: "take_sum": 2289, 2024-03-18T15:29:17.149 INFO:teuthology.orchestra.run.smithi104.stdout: "put": 2289, 2024-03-18T15:29:17.149 INFO:teuthology.orchestra.run.smithi104.stdout: "put_sum": 2289, 2024-03-18T15:29:17.149 INFO:teuthology.orchestra.run.smithi104.stdout: "wait": { 2024-03-18T15:29:17.149 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:17.149 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:17.149 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:17.149 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-03-18T15:29:17.149 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:17.149 INFO:teuthology.orchestra.run.smithi104.stdout: "throttle-write_buf_throttle": { 2024-03-18T15:29:17.149 INFO:teuthology.orchestra.run.smithi104.stdout: "val": 0, 2024-03-18T15:29:17.150 INFO:teuthology.orchestra.run.smithi104.stdout: "max": 3758096384, 2024-03-18T15:29:17.150 INFO:teuthology.orchestra.run.smithi104.stdout: "get_started": 0, 2024-03-18T15:29:17.150 INFO:teuthology.orchestra.run.smithi104.stdout: "get": 0, 2024-03-18T15:29:17.150 INFO:teuthology.orchestra.run.smithi104.stdout: "get_sum": 0, 2024-03-18T15:29:17.150 INFO:teuthology.orchestra.run.smithi104.stdout: "get_or_fail_fail": 0, 2024-03-18T15:29:17.150 INFO:teuthology.orchestra.run.smithi104.stdout: "get_or_fail_success": 0, 2024-03-18T15:29:17.150 INFO:teuthology.orchestra.run.smithi104.stdout: "take": 0, 2024-03-18T15:29:17.150 INFO:teuthology.orchestra.run.smithi104.stdout: "take_sum": 0, 2024-03-18T15:29:17.150 INFO:teuthology.orchestra.run.smithi104.stdout: "put": 0, 2024-03-18T15:29:17.150 INFO:teuthology.orchestra.run.smithi104.stdout: "put_sum": 0, 2024-03-18T15:29:17.151 INFO:teuthology.orchestra.run.smithi104.stdout: "wait": { 2024-03-18T15:29:17.151 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:17.151 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:17.151 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:17.151 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-03-18T15:29:17.151 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-03-18T15:29:17.151 INFO:teuthology.orchestra.run.smithi104.stdout: "throttle-write_buf_throttle-0x55f81c6206e0": { 2024-03-18T15:29:17.151 INFO:teuthology.orchestra.run.smithi104.stdout: "val": 0, 2024-03-18T15:29:17.151 INFO:teuthology.orchestra.run.smithi104.stdout: "max": 3758096384, 2024-03-18T15:29:17.151 INFO:teuthology.orchestra.run.smithi104.stdout: "get_started": 0, 2024-03-18T15:29:17.151 INFO:teuthology.orchestra.run.smithi104.stdout: "get": 0, 2024-03-18T15:29:17.151 INFO:teuthology.orchestra.run.smithi104.stdout: "get_sum": 0, 2024-03-18T15:29:17.152 INFO:teuthology.orchestra.run.smithi104.stdout: "get_or_fail_fail": 0, 2024-03-18T15:29:17.152 INFO:teuthology.orchestra.run.smithi104.stdout: "get_or_fail_success": 0, 2024-03-18T15:29:17.152 INFO:teuthology.orchestra.run.smithi104.stdout: "take": 0, 2024-03-18T15:29:17.152 INFO:teuthology.orchestra.run.smithi104.stdout: "take_sum": 0, 2024-03-18T15:29:17.152 INFO:teuthology.orchestra.run.smithi104.stdout: "put": 0, 2024-03-18T15:29:17.152 INFO:teuthology.orchestra.run.smithi104.stdout: "put_sum": 0, 2024-03-18T15:29:17.152 INFO:teuthology.orchestra.run.smithi104.stdout: "wait": { 2024-03-18T15:29:17.152 INFO:teuthology.orchestra.run.smithi104.stdout: "avgcount": 0, 2024-03-18T15:29:17.152 INFO:teuthology.orchestra.run.smithi104.stdout: "sum": 0.000000000, 2024-03-18T15:29:17.152 INFO:teuthology.orchestra.run.smithi104.stdout: "avgtime": 0.000000000 2024-03-18T15:29:17.152 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-03-18T15:29:17.152 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-03-18T15:29:17.153 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-03-18T15:29:17.918 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:29:17 smithi104 ceph-mon[36070]: pgmap v336: 129 pgs: 129 active+clean; 1.8 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-18T15:29:17.918 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:29:17 smithi104 ceph-mon[36070]: osdmap e105: 12 total, 12 up, 12 in 2024-03-18T15:29:17.945 INFO:tasks.check_counter:Daemon mds.l mds.exported=0 2024-03-18T15:29:17.945 INFO:tasks.check_counter:Daemon mds.l mds.imported=0 2024-03-18T15:29:17.945 INFO:tasks.check_counter:Daemon mds.l mds_cache.dir_update=0 2024-03-18T15:29:17.945 INFO:tasks.check_counter:Daemon mds.l mds_cache.dir_update_receipt=0 2024-03-18T15:29:17.945 INFO:tasks.check_counter:Daemon mds.l mds.root_rsnaps=0 2024-03-18T15:29:17.945 INFO:tasks.check_counter:Daemon mds.l mds_server.req_mksnap_latency.avgcount=0 2024-03-18T15:29:17.945 INFO:tasks.check_counter:Daemon mds.l mds_server.req_rmsnap_latency.avgcount=0 2024-03-18T15:29:17.946 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-03-18T15:29:17.963 INFO:tasks.kclient:Unmounting kernel clients... 2024-03-18T15:29:17.963 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-03-18T15:29:17.963 DEBUG:teuthology.orchestra.run.smithi019:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T15:29:17.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:29:17 smithi019 ceph-mon[31522]: pgmap v336: 129 pgs: 129 active+clean; 1.8 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-18T15:29:17.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:29:17 smithi019 ceph-mon[31522]: osdmap e105: 12 total, 12 up, 12 in 2024-03-18T15:29:18.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:29:17 smithi033 ceph-mon[36648]: pgmap v336: 129 pgs: 129 active+clean; 1.8 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-18T15:29:18.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:29:17 smithi033 ceph-mon[36648]: osdmap e105: 12 total, 12 up, 12 in 2024-03-18T15:29:18.024 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-03-18T15:29:18.025 DEBUG:teuthology.orchestra.run.smithi019:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T15:29:18.076 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T15:29:18.339 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:18.339+0000 7f0454400640 1 -- 172.21.15.19:0/1135613389 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f044c107870 msgr2=0x7f044c107c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:29:18.339 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:18.339+0000 7f0454400640 1 --2- 172.21.15.19:0/1135613389 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f044c107870 0x7f044c107c50 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f0440009930 tx=0x7f044002bc20 comp rx=0 tx=0).stop 2024-03-18T15:29:18.339 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:18.339+0000 7f0454400640 1 -- 172.21.15.19:0/1135613389 shutdown_connections 2024-03-18T15:29:18.339 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:18.339+0000 7f0454400640 1 --2- 172.21.15.19:0/1135613389 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f044c0692e0 0x7f044c10e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:18.339 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:18.339+0000 7f0454400640 1 --2- 172.21.15.19:0/1135613389 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f044c068940 0x7f044c068da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:18.339 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:18.339+0000 7f0454400640 1 --2- 172.21.15.19:0/1135613389 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f044c107870 0x7f044c107c50 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:18.340 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:18.339+0000 7f0454400640 1 -- 172.21.15.19:0/1135613389 >> 172.21.15.19:0/1135613389 conn(0x7f044c0776c0 msgr2=0x7f044c077ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:29:18.340 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:18.339+0000 7f0454400640 1 -- 172.21.15.19:0/1135613389 shutdown_connections 2024-03-18T15:29:18.340 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:18.339+0000 7f0454400640 1 -- 172.21.15.19:0/1135613389 wait complete. 2024-03-18T15:29:18.341 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:18.340+0000 7f0454400640 1 Processor -- start 2024-03-18T15:29:18.341 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:18.340+0000 7f0454400640 1 -- start start 2024-03-18T15:29:18.341 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:18.340+0000 7f0454400640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f044c068940 0x7f044c19a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:29:18.341 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:18.340+0000 7f0454400640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f044c0692e0 0x7f044c19a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:29:18.341 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:18.340+0000 7f0454400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f044c107870 0x7f044c19eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:29:18.341 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:18.340+0000 7f0454400640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f044c111c30 con 0x7f044c0692e0 2024-03-18T15:29:18.342 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:18.340+0000 7f0454400640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f044c111ab0 con 0x7f044c107870 2024-03-18T15:29:18.342 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:18.340+0000 7f0454400640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f044c111db0 con 0x7f044c068940 2024-03-18T15:29:18.342 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:18.341+0000 7f0452400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f044c107870 0x7f044c19eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:29:18.342 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:18.341+0000 7f0451000640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f044c0692e0 0x7f044c19a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:29:18.342 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:18.341+0000 7f0452400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f044c107870 0x7f044c19eb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:48956/0 (socket says 172.21.15.19:48956) 2024-03-18T15:29:18.342 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:18.341+0000 7f0452400640 1 -- 172.21.15.19:0/4141923194 learned_addr learned my addr 172.21.15.19:0/4141923194 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:29:18.342 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:18.341+0000 7f0451a00640 1 --2- 172.21.15.19:0/4141923194 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f044c068940 0x7f044c19a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:29:18.342 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:18.341+0000 7f0452400640 1 -- 172.21.15.19:0/4141923194 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f044c068940 msgr2=0x7f044c19a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:29:18.342 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:18.341+0000 7f0452400640 1 --2- 172.21.15.19:0/4141923194 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f044c068940 0x7f044c19a320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:18.342 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:18.341+0000 7f0452400640 1 -- 172.21.15.19:0/4141923194 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f044c0692e0 msgr2=0x7f044c19a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:29:18.342 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:18.341+0000 7f0452400640 1 --2- 172.21.15.19:0/4141923194 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f044c0692e0 0x7f044c19a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:18.342 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:18.341+0000 7f0452400640 1 -- 172.21.15.19:0/4141923194 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f044c19f2c0 con 0x7f044c107870 2024-03-18T15:29:18.342 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:18.341+0000 7f0452400640 1 --2- 172.21.15.19:0/4141923194 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f044c107870 0x7f044c19eb90 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f044400eb10 tx=0x7f044400c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:29:18.343 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:18.342+0000 7f044a000640 1 -- 172.21.15.19:0/4141923194 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0444004b20 con 0x7f044c107870 2024-03-18T15:29:18.343 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:18.342+0000 7f044a000640 1 -- 172.21.15.19:0/4141923194 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0444004cc0 con 0x7f044c107870 2024-03-18T15:29:18.343 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:18.342+0000 7f0454400640 1 -- 172.21.15.19:0/4141923194 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f044c19f5b0 con 0x7f044c107870 2024-03-18T15:29:18.344 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:18.342+0000 7f044a000640 1 -- 172.21.15.19:0/4141923194 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0444010760 con 0x7f044c107870 2024-03-18T15:29:18.344 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:18.342+0000 7f0454400640 1 -- 172.21.15.19:0/4141923194 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f044c1a6e40 con 0x7f044c107870 2024-03-18T15:29:18.344 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:18.343+0000 7f0454400640 1 -- 172.21.15.19:0/4141923194 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f044c068640 con 0x7f044c107870 2024-03-18T15:29:18.344 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:18.345+0000 7f044a000640 1 -- 172.21.15.19:0/4141923194 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 27) ==== 137421+0+0 (secure 0 0 0) 0x7f0444005020 con 0x7f044c107870 2024-03-18T15:29:18.346 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:18.346+0000 7f044a000640 1 --2- 172.21.15.19:0/4141923194 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f04340a29f0 0x7f04340a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:29:18.346 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:18.346+0000 7f044a000640 1 -- 172.21.15.19:0/4141923194 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 9597+0+0 (secure 0 0 0) 0x7f04440cf7e0 con 0x7f044c107870 2024-03-18T15:29:18.346 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:18.347+0000 7f0451a00640 1 --2- 172.21.15.19:0/4141923194 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f04340a29f0 0x7f04340a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:29:18.347 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:18.347+0000 7f0451a00640 1 --2- 172.21.15.19:0/4141923194 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f04340a29f0 0x7f04340a4eb0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f0440009900 tx=0x7f0440005a20 comp rx=0 tx=0).ready entity=mgr.25466 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:29:18.350 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:18.350+0000 7f044a000640 1 -- 172.21.15.19:0/4141923194 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f044409c0b0 con 0x7f044c107870 2024-03-18T15:29:18.560 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:18.559+0000 7f0454400640 1 -- 172.21.15.19:0/4141923194 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f044c1a7070 con 0x7f044c107870 2024-03-18T15:29:18.561 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:18.561+0000 7f044a000640 1 -- 172.21.15.19:0/4141923194 <== mon.2 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v105) ==== 74+0+25433 (secure 0 0 0) 0x7f044409b9b0 con 0x7f044c107870 2024-03-18T15:29:18.562 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-18T15:29:18.562 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":105,"fsid":"d4dc2ba6-e538-11ee-95c9-87774f69a715","created":"2024-03-18T15:05:57.538551+0000","modified":"2024-03-18T15:29:17.289046+0000","last_up_change":"2024-03-18T15:14:12.009522+0000","last_in_change":"2024-03-18T15:13:52.555387+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T15:09:56.843349+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T15:15:08.941047+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T15:15:10.044671+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"104","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":104,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T15:15:13.621916+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"104","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":104,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"37b5a5cd-27ae-4e77-9b05-774b817b2ee9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6802","nonce":402333948},{"type":"v1","addr":"172.21.15.19:6803","nonce":402333948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6804","nonce":402333948},{"type":"v1","addr":"172.21.15.19:6805","nonce":402333948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6808","nonce":402333948},{"type":"v1","addr":"172.21.15.19:6809","nonce":402333948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6806","nonce":402333948},{"type":"v1","addr":"172.21.15.19:6807","nonce":402333948}]},"public_addr":"172.21.15.19:6803/402333948","cluster_addr":"172.21.15.19:6805/402333948","heartbeat_back_addr":"172.21.15.19:6809/402333948","heartbeat_front_addr":"172.21.15.19:6807/402333948","state":["exists","up"]},{"osd":1,"uuid":"3a1134e8-ede7-4dbd-9be1-eefd2cbb43bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1951880113},{"type":"v1","addr":"172.21.15.33:6801","nonce":1951880113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1951880113},{"type":"v1","addr":"172.21.15.33:6803","nonce":1951880113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1951880113},{"type":"v1","addr":"172.21.15.33:6807","nonce":1951880113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1951880113},{"type":"v1","addr":"172.21.15.33:6805","nonce":1951880113}]},"public_addr":"172.21.15.33:6801/1951880113","cluster_addr":"172.21.15.33:6803/1951880113","heartbeat_back_addr":"172.21.15.33:6807/1951880113","heartbeat_front_addr":"172.21.15.33:6805/1951880113","state":["exists","up"]},{"osd":2,"uuid":"f383a164-a3a0-49fd-a9e8-9c2178d8cfb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":1004727949},{"type":"v1","addr":"172.21.15.104:6801","nonce":1004727949}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":1004727949},{"type":"v1","addr":"172.21.15.104:6803","nonce":1004727949}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":1004727949},{"type":"v1","addr":"172.21.15.104:6807","nonce":1004727949}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":1004727949},{"type":"v1","addr":"172.21.15.104:6805","nonce":1004727949}]},"public_addr":"172.21.15.104:6801/1004727949","cluster_addr":"172.21.15.104:6803/1004727949","heartbeat_back_addr":"172.21.15.104:6807/1004727949","heartbeat_front_addr":"172.21.15.104:6805/1004727949","state":["exists","up"]},{"osd":3,"uuid":"c409e116-28bf-45dd-a791-539af22abe1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6810","nonce":3571525854},{"type":"v1","addr":"172.21.15.19:6811","nonce":3571525854}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6812","nonce":3571525854},{"type":"v1","addr":"172.21.15.19:6813","nonce":3571525854}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6816","nonce":3571525854},{"type":"v1","addr":"172.21.15.19:6817","nonce":3571525854}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6814","nonce":3571525854},{"type":"v1","addr":"172.21.15.19:6815","nonce":3571525854}]},"public_addr":"172.21.15.19:6811/3571525854","cluster_addr":"172.21.15.19:6813/3571525854","heartbeat_back_addr":"172.21.15.19:6817/3571525854","heartbeat_front_addr":"172.21.15.19:6815/3571525854","state":["exists","up"]},{"osd":4,"uuid":"696eb719-0b3e-43ae-b5f1-ccd73a3b2843","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":624901749},{"type":"v1","addr":"172.21.15.33:6809","nonce":624901749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":624901749},{"type":"v1","addr":"172.21.15.33:6811","nonce":624901749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":624901749},{"type":"v1","addr":"172.21.15.33:6815","nonce":624901749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":624901749},{"type":"v1","addr":"172.21.15.33:6813","nonce":624901749}]},"public_addr":"172.21.15.33:6809/624901749","cluster_addr":"172.21.15.33:6811/624901749","heartbeat_back_addr":"172.21.15.33:6815/624901749","heartbeat_front_addr":"172.21.15.33:6813/624901749","state":["exists","up"]},{"osd":5,"uuid":"21789779-8bb9-42c7-b9ee-437985f11e3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":1258208297},{"type":"v1","addr":"172.21.15.104:6809","nonce":1258208297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":1258208297},{"type":"v1","addr":"172.21.15.104:6811","nonce":1258208297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":1258208297},{"type":"v1","addr":"172.21.15.104:6815","nonce":1258208297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":1258208297},{"type":"v1","addr":"172.21.15.104:6813","nonce":1258208297}]},"public_addr":"172.21.15.104:6809/1258208297","cluster_addr":"172.21.15.104:6811/1258208297","heartbeat_back_addr":"172.21.15.104:6815/1258208297","heartbeat_front_addr":"172.21.15.104:6813/1258208297","state":["exists","up"]},{"osd":6,"uuid":"0e92a2ef-43cd-487c-9203-e219158aef8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6818","nonce":2782482301},{"type":"v1","addr":"172.21.15.19:6819","nonce":2782482301}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6820","nonce":2782482301},{"type":"v1","addr":"172.21.15.19:6821","nonce":2782482301}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6824","nonce":2782482301},{"type":"v1","addr":"172.21.15.19:6825","nonce":2782482301}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6822","nonce":2782482301},{"type":"v1","addr":"172.21.15.19:6823","nonce":2782482301}]},"public_addr":"172.21.15.19:6819/2782482301","cluster_addr":"172.21.15.19:6821/2782482301","heartbeat_back_addr":"172.21.15.19:6825/2782482301","heartbeat_front_addr":"172.21.15.19:6823/2782482301","state":["exists","up"]},{"osd":7,"uuid":"a16e39a1-3685-491d-b376-3e80a2084072","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":4238160988},{"type":"v1","addr":"172.21.15.33:6817","nonce":4238160988}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":4238160988},{"type":"v1","addr":"172.21.15.33:6819","nonce":4238160988}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":4238160988},{"type":"v1","addr":"172.21.15.33:6823","nonce":4238160988}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":4238160988},{"type":"v1","addr":"172.21.15.33:6821","nonce":4238160988}]},"public_addr":"172.21.15.33:6817/4238160988","cluster_addr":"172.21.15.33:6819/4238160988","heartbeat_back_addr":"172.21.15.33:6823/4238160988","heartbeat_front_addr":"172.21.15.33:6821/4238160988","state":["exists","up"]},{"osd":8,"uuid":"d5c3fa92-cd5d-4349-9c7d-d5666bacde16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":1014779813},{"type":"v1","addr":"172.21.15.104:6817","nonce":1014779813}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":1014779813},{"type":"v1","addr":"172.21.15.104:6819","nonce":1014779813}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":1014779813},{"type":"v1","addr":"172.21.15.104:6823","nonce":1014779813}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":1014779813},{"type":"v1","addr":"172.21.15.104:6821","nonce":1014779813}]},"public_addr":"172.21.15.104:6817/1014779813","cluster_addr":"172.21.15.104:6819/1014779813","heartbeat_back_addr":"172.21.15.104:6823/1014779813","heartbeat_front_addr":"172.21.15.104:6821/1014779813","state":["exists","up"]},{"osd":9,"uuid":"eed2e9b2-26d3-4c46-88ec-34b5596fe991","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6826","nonce":2641151009},{"type":"v1","addr":"172.21.15.19:6827","nonce":2641151009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6828","nonce":2641151009},{"type":"v1","addr":"172.21.15.19:6829","nonce":2641151009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6832","nonce":2641151009},{"type":"v1","addr":"172.21.15.19:6833","nonce":2641151009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6830","nonce":2641151009},{"type":"v1","addr":"172.21.15.19:6831","nonce":2641151009}]},"public_addr":"172.21.15.19:6827/2641151009","cluster_addr":"172.21.15.19:6829/2641151009","heartbeat_back_addr":"172.21.15.19:6833/2641151009","heartbeat_front_addr":"172.21.15.19:6831/2641151009","state":["exists","up"]},{"osd":10,"uuid":"f3c35ea5-fbe7-4dc4-93c5-b92d1a714db3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":1346058869},{"type":"v1","addr":"172.21.15.33:6825","nonce":1346058869}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":1346058869},{"type":"v1","addr":"172.21.15.33:6827","nonce":1346058869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":1346058869},{"type":"v1","addr":"172.21.15.33:6831","nonce":1346058869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":1346058869},{"type":"v1","addr":"172.21.15.33:6829","nonce":1346058869}]},"public_addr":"172.21.15.33:6825/1346058869","cluster_addr":"172.21.15.33:6827/1346058869","heartbeat_back_addr":"172.21.15.33:6831/1346058869","heartbeat_front_addr":"172.21.15.33:6829/1346058869","state":["exists","up"]},{"osd":11,"uuid":"c8b51571-f807-4b62-90b6-7529e09e59da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":3759498281},{"type":"v1","addr":"172.21.15.104:6825","nonce":3759498281}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":3759498281},{"type":"v1","addr":"172.21.15.104:6827","nonce":3759498281}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":3759498281},{"type":"v1","addr":"172.21.15.104:6831","nonce":3759498281}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":3759498281},{"type":"v1","addr":"172.21.15.104:6829","nonce":3759498281}]},"public_addr":"172.21.15.104:6825/3759498281","cluster_addr":"172.21.15.104:6827/3759498281","heartbeat_back_addr":"172.21.15.104:6831/3759498281","heartbeat_front_addr":"172.21.15.104:6829/3759498281","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:09:05.267795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:09:30.234068+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:09:53.870232+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:10:26.471296+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:10:53.180341+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:11:17.801361+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:11:51.364816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:12:18.306408+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:12:43.978744+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:13:18.213964+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:13:45.437544+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:14:10.148580+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":3,"to":6}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"3.37","mappings":[{"from":5,"to":8}]},{"pgid":"3.3a","mappings":[{"from":4,"to":7}]},{"pgid":"4.8","mappings":[{"from":5,"to":8}]},{"pgid":"4.f","mappings":[{"from":9,"to":10}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.19:0/1895666842":"2024-03-18T16:23:20.095509+0000","172.21.15.19:6801/3264337803":"2024-03-19T15:06:38.688405+0000","172.21.15.19:6800/3264337803":"2024-03-19T15:06:38.688405+0000","172.21.15.19:0/2793980440":"2024-03-19T15:06:38.688405+0000","172.21.15.19:6801/2776008962":"2024-03-19T15:07:14.150983+0000","172.21.15.19:0/2316969005":"2024-03-19T15:07:14.150983+0000","172.21.15.19:0/84203799":"2024-03-19T15:18:29.726021+0000","172.21.15.19:6800/2017714236":"2024-03-19T15:18:29.726021+0000","172.21.15.19:0/2737223213":"2024-03-19T15:07:14.150983+0000","172.21.15.19:0/3860783274":"2024-03-19T15:07:14.150983+0000","172.21.15.19:6800/2776008962":"2024-03-19T15:07:14.150983+0000","172.21.15.19:0/1024556841":"2024-03-19T15:18:29.726021+0000","172.21.15.19:0/3166705295":"2024-03-19T15:06:38.688405+0000","172.21.15.19:6801/2017714236":"2024-03-19T15:18:29.726021+0000","172.21.15.19:0/176699025":"2024-03-19T15:06:38.688405+0000","172.21.15.19:0/2730531676":"2024-03-19T15:18:29.726021+0000","172.21.15.19:0/2353244552":"2024-03-19T15:18:29.726021+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]},{"pool":4,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T15:29:18.564 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:18.564+0000 7f0454400640 1 -- 172.21.15.19:0/4141923194 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f04340a29f0 msgr2=0x7f04340a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:29:18.564 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:18.564+0000 7f0454400640 1 --2- 172.21.15.19:0/4141923194 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f04340a29f0 0x7f04340a4eb0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f0440009900 tx=0x7f0440005a20 comp rx=0 tx=0).stop 2024-03-18T15:29:18.564 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:18.564+0000 7f0454400640 1 -- 172.21.15.19:0/4141923194 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f044c107870 msgr2=0x7f044c19eb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:29:18.564 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:18.564+0000 7f0454400640 1 --2- 172.21.15.19:0/4141923194 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f044c107870 0x7f044c19eb90 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f044400eb10 tx=0x7f044400c2d0 comp rx=0 tx=0).stop 2024-03-18T15:29:18.564 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:18.564+0000 7f0454400640 1 -- 172.21.15.19:0/4141923194 shutdown_connections 2024-03-18T15:29:18.564 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:18.564+0000 7f0454400640 1 --2- 172.21.15.19:0/4141923194 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f04340a29f0 0x7f04340a4eb0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:18.565 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:18.564+0000 7f0454400640 1 --2- 172.21.15.19:0/4141923194 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f044c107870 0x7f044c19eb90 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:18.565 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:18.564+0000 7f0454400640 1 --2- 172.21.15.19:0/4141923194 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f044c0692e0 0x7f044c19a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:18.565 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:18.564+0000 7f0454400640 1 --2- 172.21.15.19:0/4141923194 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f044c068940 0x7f044c19a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:18.565 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:18.564+0000 7f0454400640 1 -- 172.21.15.19:0/4141923194 >> 172.21.15.19:0/4141923194 conn(0x7f044c0776c0 msgr2=0x7f044c0ff520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:29:18.565 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:18.564+0000 7f0454400640 1 -- 172.21.15.19:0/4141923194 shutdown_connections 2024-03-18T15:29:18.565 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:18.564+0000 7f0454400640 1 -- 172.21.15.19:0/4141923194 wait complete. 2024-03-18T15:29:18.591 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T15:09:56.843349+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T15:15:08.941047+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T15:15:10.044671+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '104', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 17, 'snap_epoch': 104, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-18T15:15:13.621916+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '104', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 17, 'snap_epoch': 104, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-18T15:29:18.592 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T15:29:18.824 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:18.824+0000 7fb71d000640 1 -- 172.21.15.19:0/1461205855 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb718102010 msgr2=0x7fb71810e520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:29:18.825 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:18.824+0000 7fb71d000640 1 --2- 172.21.15.19:0/1461205855 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb718102010 0x7fb71810e520 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fb710009cd0 tx=0x7fb71002c0e0 comp rx=0 tx=0).stop 2024-03-18T15:29:18.825 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:18.824+0000 7fb71d000640 1 -- 172.21.15.19:0/1461205855 shutdown_connections 2024-03-18T15:29:18.825 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:18.824+0000 7fb71d000640 1 --2- 172.21.15.19:0/1461205855 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb718102010 0x7fb71810e520 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:18.825 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:18.824+0000 7fb71d000640 1 --2- 172.21.15.19:0/1461205855 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb718101650 0x7fb718101ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:18.825 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:18.824+0000 7fb71d000640 1 --2- 172.21.15.19:0/1461205855 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb718107660 0x7fb718107a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:18.826 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:18.824+0000 7fb71d000640 1 -- 172.21.15.19:0/1461205855 >> 172.21.15.19:0/1461205855 conn(0x7fb7180fd180 msgr2=0x7fb7180ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:29:18.826 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:18.824+0000 7fb71d000640 1 -- 172.21.15.19:0/1461205855 shutdown_connections 2024-03-18T15:29:18.826 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:18.824+0000 7fb71d000640 1 -- 172.21.15.19:0/1461205855 wait complete. 2024-03-18T15:29:18.829 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:18.825+0000 7fb71d000640 1 Processor -- start 2024-03-18T15:29:18.829 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:18.825+0000 7fb71d000640 1 -- start start 2024-03-18T15:29:18.829 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:18.826+0000 7fb71d000640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb718101650 0x7fb71819a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:29:18.829 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:18.826+0000 7fb71d000640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb718102010 0x7fb71819a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:29:18.829 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:18.826+0000 7fb71d000640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb718107660 0x7fb71819e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:29:18.829 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:18.826+0000 7fb71d000640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fb718111a30 con 0x7fb718102010 2024-03-18T15:29:18.829 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:18.826+0000 7fb71d000640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb7181118b0 con 0x7fb718101650 2024-03-18T15:29:18.830 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:18.826+0000 7fb71d000640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fb718111bb0 con 0x7fb718107660 2024-03-18T15:29:18.830 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:18.826+0000 7fb717400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb718101650 0x7fb71819a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:29:18.830 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:18.826+0000 7fb717400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb718101650 0x7fb71819a0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:48980/0 (socket says 172.21.15.19:48980) 2024-03-18T15:29:18.830 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:18.826+0000 7fb717400640 1 -- 172.21.15.19:0/3422504909 learned_addr learned my addr 172.21.15.19:0/3422504909 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:29:18.830 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:18.826+0000 7fb716a00640 1 --2- 172.21.15.19:0/3422504909 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb718102010 0x7fb71819a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:29:18.830 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:18.826+0000 7fb717e00640 1 --2- 172.21.15.19:0/3422504909 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb718107660 0x7fb71819e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:29:18.830 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:18.827+0000 7fb717400640 1 -- 172.21.15.19:0/3422504909 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb718107660 msgr2=0x7fb71819e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:29:18.830 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:18.827+0000 7fb717400640 1 --2- 172.21.15.19:0/3422504909 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb718107660 0x7fb71819e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:18.830 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:18.827+0000 7fb717400640 1 -- 172.21.15.19:0/3422504909 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb718102010 msgr2=0x7fb71819a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:29:18.830 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:18.827+0000 7fb717400640 1 --2- 172.21.15.19:0/3422504909 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb718102010 0x7fb71819a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:18.830 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:18.827+0000 7fb717400640 1 -- 172.21.15.19:0/3422504909 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb71819f0a0 con 0x7fb718101650 2024-03-18T15:29:18.830 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:18.827+0000 7fb717400640 1 --2- 172.21.15.19:0/3422504909 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb718101650 0x7fb71819a0b0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fb70800e9f0 tx=0x7fb70800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:29:18.831 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:18.827+0000 7fb70fe00640 1 -- 172.21.15.19:0/3422504909 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb70800cd80 con 0x7fb718101650 2024-03-18T15:29:18.831 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:18.827+0000 7fb70fe00640 1 -- 172.21.15.19:0/3422504909 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb708010910 con 0x7fb718101650 2024-03-18T15:29:18.831 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:18.827+0000 7fb70fe00640 1 -- 172.21.15.19:0/3422504909 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb708018990 con 0x7fb718101650 2024-03-18T15:29:18.831 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:18.827+0000 7fb71d000640 1 -- 172.21.15.19:0/3422504909 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb71819f390 con 0x7fb718101650 2024-03-18T15:29:18.831 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:18.827+0000 7fb71d000640 1 -- 172.21.15.19:0/3422504909 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7181a6c20 con 0x7fb718101650 2024-03-18T15:29:18.831 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:18.829+0000 7fb71d000640 1 -- 172.21.15.19:0/3422504909 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7180684f0 con 0x7fb718101650 2024-03-18T15:29:18.832 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:18.830+0000 7fb70fe00640 1 -- 172.21.15.19:0/3422504909 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 27) ==== 137421+0+0 (secure 0 0 0) 0x7fb708010ac0 con 0x7fb718101650 2024-03-18T15:29:18.832 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:18.831+0000 7fb70fe00640 1 --2- 172.21.15.19:0/3422504909 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7fb6fc0a2970 0x7fb6fc0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:29:18.832 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:18.832+0000 7fb70fe00640 1 -- 172.21.15.19:0/3422504909 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 9597+0+0 (secure 0 0 0) 0x7fb7080cf5b0 con 0x7fb718101650 2024-03-18T15:29:18.832 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:18.832+0000 7fb716a00640 1 --2- 172.21.15.19:0/3422504909 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7fb6fc0a2970 0x7fb6fc0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:29:18.832 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:18.833+0000 7fb716a00640 1 --2- 172.21.15.19:0/3422504909 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7fb6fc0a2970 0x7fb6fc0a4e30 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb71810e8e0 tx=0x7fb704005f90 comp rx=0 tx=0).ready entity=mgr.25466 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:29:18.835 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:18.836+0000 7fb70fe00640 1 -- 172.21.15.19:0/3422504909 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb70809b830 con 0x7fb718101650 2024-03-18T15:29:18.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:29:18 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/4141923194' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:29:18.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:29:18 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/4141923194' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:29:19.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:29:18 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/4141923194' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:29:19.041 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.041+0000 7fb71d000640 1 -- 172.21.15.19:0/3422504909 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb7181a6e70 con 0x7fb718101650 2024-03-18T15:29:19.042 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.042+0000 7fb70fe00640 1 -- 172.21.15.19:0/3422504909 <== mon.2 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v105) ==== 93+0+10 (secure 0 0 0) 0x7fb70809f8d0 con 0x7fb718101650 2024-03-18T15:29:19.042 INFO:teuthology.orchestra.run.smithi019.stdout:pg_num: 1 2024-03-18T15:29:19.045 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.045+0000 7fb71d000640 1 -- 172.21.15.19:0/3422504909 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7fb6fc0a2970 msgr2=0x7fb6fc0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:29:19.045 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.045+0000 7fb71d000640 1 --2- 172.21.15.19:0/3422504909 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7fb6fc0a2970 0x7fb6fc0a4e30 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb71810e8e0 tx=0x7fb704005f90 comp rx=0 tx=0).stop 2024-03-18T15:29:19.045 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.045+0000 7fb71d000640 1 -- 172.21.15.19:0/3422504909 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb718101650 msgr2=0x7fb71819a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:29:19.045 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.045+0000 7fb71d000640 1 --2- 172.21.15.19:0/3422504909 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb718101650 0x7fb71819a0b0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fb70800e9f0 tx=0x7fb70800eeb0 comp rx=0 tx=0).stop 2024-03-18T15:29:19.045 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.045+0000 7fb71d000640 1 -- 172.21.15.19:0/3422504909 shutdown_connections 2024-03-18T15:29:19.045 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.045+0000 7fb71d000640 1 --2- 172.21.15.19:0/3422504909 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7fb6fc0a2970 0x7fb6fc0a4e30 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:19.045 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.045+0000 7fb71d000640 1 --2- 172.21.15.19:0/3422504909 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb718107660 0x7fb71819e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:19.045 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.045+0000 7fb71d000640 1 --2- 172.21.15.19:0/3422504909 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb718102010 0x7fb71819a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:19.046 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.045+0000 7fb71d000640 1 --2- 172.21.15.19:0/3422504909 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb718101650 0x7fb71819a0b0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:19.046 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.045+0000 7fb71d000640 1 -- 172.21.15.19:0/3422504909 >> 172.21.15.19:0/3422504909 conn(0x7fb7180fd180 msgr2=0x7fb71810cea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:29:19.046 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.045+0000 7fb71d000640 1 -- 172.21.15.19:0/3422504909 shutdown_connections 2024-03-18T15:29:19.046 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.045+0000 7fb71d000640 1 -- 172.21.15.19:0/3422504909 wait complete. 2024-03-18T15:29:19.065 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T15:29:19.264 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.261+0000 7f2019e00640 1 -- 172.21.15.19:0/139334375 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2014107650 msgr2=0x7f2014107a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:29:19.265 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.261+0000 7f2019e00640 1 --2- 172.21.15.19:0/139334375 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2014107650 0x7f2014107a30 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f2008009ca0 tx=0x7f200802c100 comp rx=0 tx=0).stop 2024-03-18T15:29:19.265 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.262+0000 7f2019e00640 1 -- 172.21.15.19:0/139334375 shutdown_connections 2024-03-18T15:29:19.265 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.262+0000 7f2019e00640 1 --2- 172.21.15.19:0/139334375 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2014102000 0x7f201410e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:19.265 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.262+0000 7f2019e00640 1 --2- 172.21.15.19:0/139334375 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2014101640 0x7f2014101ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:19.265 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.262+0000 7f2019e00640 1 --2- 172.21.15.19:0/139334375 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2014107650 0x7f2014107a30 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:19.265 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.262+0000 7f2019e00640 1 -- 172.21.15.19:0/139334375 >> 172.21.15.19:0/139334375 conn(0x7f20140fd150 msgr2=0x7f20140ff570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:29:19.265 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.262+0000 7f2019e00640 1 -- 172.21.15.19:0/139334375 shutdown_connections 2024-03-18T15:29:19.265 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.262+0000 7f2019e00640 1 -- 172.21.15.19:0/139334375 wait complete. 2024-03-18T15:29:19.266 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.263+0000 7f2019e00640 1 Processor -- start 2024-03-18T15:29:19.266 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.263+0000 7f2019e00640 1 -- start start 2024-03-18T15:29:19.266 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.263+0000 7f2019e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2014101640 0x7f2014073ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:29:19.266 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.263+0000 7f2019e00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2014102000 0x7f2014074020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:29:19.266 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.263+0000 7f2019e00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2014107650 0x7f2014074560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:29:19.266 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.263+0000 7f2019e00640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f2014111a20 con 0x7f2014102000 2024-03-18T15:29:19.266 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.263+0000 7f2019e00640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f20141118a0 con 0x7f2014101640 2024-03-18T15:29:19.266 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.263+0000 7f2019e00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f2014111ba0 con 0x7f2014107650 2024-03-18T15:29:19.266 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.263+0000 7f2018a00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2014101640 0x7f2014073ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:29:19.266 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.263+0000 7f2018a00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2014101640 0x7f2014073ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:48996/0 (socket says 172.21.15.19:48996) 2024-03-18T15:29:19.266 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.263+0000 7f2018a00640 1 -- 172.21.15.19:0/1152537234 learned_addr learned my addr 172.21.15.19:0/1152537234 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:29:19.266 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.263+0000 7f2019400640 1 --2- 172.21.15.19:0/1152537234 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2014107650 0x7f2014074560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:29:19.267 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.264+0000 7f2013e00640 1 --2- 172.21.15.19:0/1152537234 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2014102000 0x7f2014074020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:29:19.267 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.264+0000 7f2018a00640 1 -- 172.21.15.19:0/1152537234 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2014107650 msgr2=0x7f2014074560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:29:19.267 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.264+0000 7f2018a00640 1 --2- 172.21.15.19:0/1152537234 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2014107650 0x7f2014074560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:19.267 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.264+0000 7f2018a00640 1 -- 172.21.15.19:0/1152537234 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2014102000 msgr2=0x7f2014074020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:29:19.267 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.264+0000 7f2018a00640 1 --2- 172.21.15.19:0/1152537234 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2014102000 0x7f2014074020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:19.267 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.264+0000 7f2018a00640 1 -- 172.21.15.19:0/1152537234 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20141a6420 con 0x7f2014101640 2024-03-18T15:29:19.267 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.264+0000 7f2013e00640 1 --2- 172.21.15.19:0/1152537234 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2014102000 0x7f2014074020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T15:29:19.267 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.264+0000 7f2018a00640 1 --2- 172.21.15.19:0/1152537234 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2014101640 0x7f2014073ae0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f2008005e30 tx=0x7f20080346f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:29:19.267 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.264+0000 7f2011600640 1 -- 172.21.15.19:0/1152537234 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2008039050 con 0x7f2014101640 2024-03-18T15:29:19.268 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.264+0000 7f2011600640 1 -- 172.21.15.19:0/1152537234 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2008045070 con 0x7f2014101640 2024-03-18T15:29:19.268 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.264+0000 7f2019e00640 1 -- 172.21.15.19:0/1152537234 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20141a66b0 con 0x7f2014101640 2024-03-18T15:29:19.268 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.265+0000 7f2011600640 1 -- 172.21.15.19:0/1152537234 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20080046c0 con 0x7f2014101640 2024-03-18T15:29:19.268 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.265+0000 7f2019e00640 1 -- 172.21.15.19:0/1152537234 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20141a6b70 con 0x7f2014101640 2024-03-18T15:29:19.270 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.266+0000 7f2019e00640 1 -- 172.21.15.19:0/1152537234 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20140684f0 con 0x7f2014101640 2024-03-18T15:29:19.270 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.268+0000 7f2011600640 1 -- 172.21.15.19:0/1152537234 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 27) ==== 137421+0+0 (secure 0 0 0) 0x7f2008003970 con 0x7f2014101640 2024-03-18T15:29:19.270 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.269+0000 7f2011600640 1 --2- 172.21.15.19:0/1152537234 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f1ffc0a2970 0x7f1ffc0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:29:19.271 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.269+0000 7f2011600640 1 -- 172.21.15.19:0/1152537234 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 9597+0+0 (secure 0 0 0) 0x7f20080f2740 con 0x7f2014101640 2024-03-18T15:29:19.271 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.269+0000 7f2013e00640 1 --2- 172.21.15.19:0/1152537234 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f1ffc0a2970 0x7f1ffc0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:29:19.271 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.270+0000 7f2013e00640 1 --2- 172.21.15.19:0/1152537234 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f1ffc0a2970 0x7f1ffc0a4e30 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f2014075570 tx=0x7f2000005f90 comp rx=0 tx=0).ready entity=mgr.25466 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:29:19.276 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.276+0000 7f2011600640 1 -- 172.21.15.19:0/1152537234 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f20080bf120 con 0x7f2014101640 2024-03-18T15:29:19.477 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.476+0000 7f2019e00640 1 -- 172.21.15.19:0/1152537234 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f2014109bb0 con 0x7f2014101640 2024-03-18T15:29:19.477 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.477+0000 7f2011600640 1 -- 172.21.15.19:0/1152537234 <== mon.2 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v105) ==== 104+0+11 (secure 0 0 0) 0x7f20080bea20 con 0x7f2014101640 2024-03-18T15:29:19.478 INFO:teuthology.orchestra.run.smithi019.stdout:pg_num: 32 2024-03-18T15:29:19.480 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.480+0000 7f2019e00640 1 -- 172.21.15.19:0/1152537234 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f1ffc0a2970 msgr2=0x7f1ffc0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:29:19.481 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.480+0000 7f2019e00640 1 --2- 172.21.15.19:0/1152537234 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f1ffc0a2970 0x7f1ffc0a4e30 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f2014075570 tx=0x7f2000005f90 comp rx=0 tx=0).stop 2024-03-18T15:29:19.481 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.480+0000 7f2019e00640 1 -- 172.21.15.19:0/1152537234 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2014101640 msgr2=0x7f2014073ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:29:19.481 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.480+0000 7f2019e00640 1 --2- 172.21.15.19:0/1152537234 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2014101640 0x7f2014073ae0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f2008005e30 tx=0x7f20080346f0 comp rx=0 tx=0).stop 2024-03-18T15:29:19.481 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.480+0000 7f2019e00640 1 -- 172.21.15.19:0/1152537234 shutdown_connections 2024-03-18T15:29:19.481 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.480+0000 7f2019e00640 1 --2- 172.21.15.19:0/1152537234 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f1ffc0a2970 0x7f1ffc0a4e30 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:19.481 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.480+0000 7f2019e00640 1 --2- 172.21.15.19:0/1152537234 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2014107650 0x7f2014074560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:19.481 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.480+0000 7f2019e00640 1 --2- 172.21.15.19:0/1152537234 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2014102000 0x7f2014074020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:19.481 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.480+0000 7f2019e00640 1 --2- 172.21.15.19:0/1152537234 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2014101640 0x7f2014073ae0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:19.481 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.480+0000 7f2019e00640 1 -- 172.21.15.19:0/1152537234 >> 172.21.15.19:0/1152537234 conn(0x7f20140fd150 msgr2=0x7f20140fd940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:29:19.481 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.481+0000 7f2019e00640 1 -- 172.21.15.19:0/1152537234 shutdown_connections 2024-03-18T15:29:19.481 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.481+0000 7f2019e00640 1 -- 172.21.15.19:0/1152537234 wait complete. 2024-03-18T15:29:19.498 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T15:29:19.725 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.724+0000 7fb598200640 1 -- 172.21.15.19:0/955905797 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb5900689e0 msgr2=0x7fb590068e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:29:19.725 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.724+0000 7fb598200640 1 --2- 172.21.15.19:0/955905797 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb5900689e0 0x7fb590068e60 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fb57c009960 tx=0x7fb57c02bc20 comp rx=0 tx=0).stop 2024-03-18T15:29:19.725 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.725+0000 7fb598200640 1 -- 172.21.15.19:0/955905797 shutdown_connections 2024-03-18T15:29:19.725 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.725+0000 7fb598200640 1 --2- 172.21.15.19:0/955905797 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb5900693a0 0x7fb59010e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:19.726 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.725+0000 7fb598200640 1 --2- 172.21.15.19:0/955905797 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb5900689e0 0x7fb590068e60 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:19.726 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.725+0000 7fb598200640 1 --2- 172.21.15.19:0/955905797 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb590107830 0x7fb590107c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:19.726 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.725+0000 7fb598200640 1 -- 172.21.15.19:0/955905797 >> 172.21.15.19:0/955905797 conn(0x7fb5900777e0 msgr2=0x7fb590077bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:29:19.726 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.725+0000 7fb598200640 1 -- 172.21.15.19:0/955905797 shutdown_connections 2024-03-18T15:29:19.726 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.725+0000 7fb598200640 1 -- 172.21.15.19:0/955905797 wait complete. 2024-03-18T15:29:19.728 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.726+0000 7fb598200640 1 Processor -- start 2024-03-18T15:29:19.728 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.726+0000 7fb598200640 1 -- start start 2024-03-18T15:29:19.728 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.726+0000 7fb598200640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb5900689e0 0x7fb59019a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:29:19.729 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.727+0000 7fb598200640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb5900693a0 0x7fb59019a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:29:19.729 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.727+0000 7fb598200640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb590107830 0x7fb59019eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:29:19.729 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.727+0000 7fb598200640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fb590111b90 con 0x7fb5900689e0 2024-03-18T15:29:19.729 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.727+0000 7fb598200640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb590111a10 con 0x7fb590107830 2024-03-18T15:29:19.729 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.727+0000 7fb598200640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fb590111d10 con 0x7fb5900693a0 2024-03-18T15:29:19.729 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.727+0000 7fb594e00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb5900693a0 0x7fb59019a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:29:19.729 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.727+0000 7fb595800640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb5900689e0 0x7fb59019a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:29:19.729 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.727+0000 7fb596200640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb590107830 0x7fb59019eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:29:19.729 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.727+0000 7fb594e00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb5900693a0 0x7fb59019a7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.19:60594/0 (socket says 172.21.15.19:60594) 2024-03-18T15:29:19.729 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.727+0000 7fb594e00640 1 -- 172.21.15.19:0/4096144331 learned_addr learned my addr 172.21.15.19:0/4096144331 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:29:19.730 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.727+0000 7fb594e00640 1 -- 172.21.15.19:0/4096144331 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb590107830 msgr2=0x7fb59019eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:29:19.730 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.727+0000 7fb594e00640 1 --2- 172.21.15.19:0/4096144331 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb590107830 0x7fb59019eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:19.730 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.727+0000 7fb594e00640 1 -- 172.21.15.19:0/4096144331 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb5900689e0 msgr2=0x7fb59019a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:29:19.730 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.727+0000 7fb594e00640 1 --2- 172.21.15.19:0/4096144331 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb5900689e0 0x7fb59019a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:19.730 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.727+0000 7fb594e00640 1 -- 172.21.15.19:0/4096144331 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb59019f280 con 0x7fb5900693a0 2024-03-18T15:29:19.730 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.728+0000 7fb594e00640 1 --2- 172.21.15.19:0/4096144331 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb5900693a0 0x7fb59019a7d0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fb57c02c230 tx=0x7fb57c004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:29:19.730 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.728+0000 7fb58e000640 1 -- 172.21.15.19:0/4096144331 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb57c043020 con 0x7fb5900693a0 2024-03-18T15:29:19.730 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.728+0000 7fb58e000640 1 -- 172.21.15.19:0/4096144331 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb57c005000 con 0x7fb5900693a0 2024-03-18T15:29:19.730 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.728+0000 7fb598200640 1 -- 172.21.15.19:0/4096144331 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb59019f510 con 0x7fb5900693a0 2024-03-18T15:29:19.730 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.728+0000 7fb58e000640 1 -- 172.21.15.19:0/4096144331 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb57c003b50 con 0x7fb5900693a0 2024-03-18T15:29:19.730 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.728+0000 7fb598200640 1 -- 172.21.15.19:0/4096144331 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5901a6e00 con 0x7fb5900693a0 2024-03-18T15:29:19.731 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.729+0000 7fb598200640 1 -- 172.21.15.19:0/4096144331 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5900685d0 con 0x7fb5900693a0 2024-03-18T15:29:19.731 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.731+0000 7fb58e000640 1 -- 172.21.15.19:0/4096144331 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 27) ==== 137421+0+0 (secure 0 0 0) 0x7fb57c046020 con 0x7fb5900693a0 2024-03-18T15:29:19.732 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.732+0000 7fb58e000640 1 --2- 172.21.15.19:0/4096144331 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7fb5780a29f0 0x7fb5780a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:29:19.732 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.732+0000 7fb58e000640 1 -- 172.21.15.19:0/4096144331 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 9597+0+0 (secure 0 0 0) 0x7fb57c0f15e0 con 0x7fb5900693a0 2024-03-18T15:29:19.732 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.733+0000 7fb595800640 1 --2- 172.21.15.19:0/4096144331 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7fb5780a29f0 0x7fb5780a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:29:19.733 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.733+0000 7fb595800640 1 --2- 172.21.15.19:0/4096144331 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7fb5780a29f0 0x7fb5780a4eb0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb584009bd0 tx=0x7fb584005f90 comp rx=0 tx=0).ready entity=mgr.25466 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:29:19.735 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.735+0000 7fb58e000640 1 -- 172.21.15.19:0/4096144331 <== mon.1 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb57c04d020 con 0x7fb5900693a0 2024-03-18T15:29:19.938 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.938+0000 7fb598200640 1 -- 172.21.15.19:0/4096144331 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fb5900ff1b0 con 0x7fb5900693a0 2024-03-18T15:29:19.938 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.939+0000 7fb58e000640 1 -- 172.21.15.19:0/4096144331 <== mon.1 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v105) ==== 100+0+11 (secure 0 0 0) 0x7fb57c0be020 con 0x7fb5900693a0 2024-03-18T15:29:19.939 INFO:teuthology.orchestra.run.smithi019.stdout:pg_num: 64 2024-03-18T15:29:19.942 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.941+0000 7fb598200640 1 -- 172.21.15.19:0/4096144331 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7fb5780a29f0 msgr2=0x7fb5780a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:29:19.942 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.941+0000 7fb598200640 1 --2- 172.21.15.19:0/4096144331 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7fb5780a29f0 0x7fb5780a4eb0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb584009bd0 tx=0x7fb584005f90 comp rx=0 tx=0).stop 2024-03-18T15:29:19.942 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.941+0000 7fb598200640 1 -- 172.21.15.19:0/4096144331 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb5900693a0 msgr2=0x7fb59019a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:29:19.942 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.941+0000 7fb598200640 1 --2- 172.21.15.19:0/4096144331 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb5900693a0 0x7fb59019a7d0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fb57c02c230 tx=0x7fb57c004270 comp rx=0 tx=0).stop 2024-03-18T15:29:19.942 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.942+0000 7fb598200640 1 -- 172.21.15.19:0/4096144331 shutdown_connections 2024-03-18T15:29:19.942 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.942+0000 7fb598200640 1 --2- 172.21.15.19:0/4096144331 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7fb5780a29f0 0x7fb5780a4eb0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:19.942 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.942+0000 7fb598200640 1 --2- 172.21.15.19:0/4096144331 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb590107830 0x7fb59019eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:19.942 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.942+0000 7fb598200640 1 --2- 172.21.15.19:0/4096144331 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb5900693a0 0x7fb59019a7d0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:19.943 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.942+0000 7fb598200640 1 --2- 172.21.15.19:0/4096144331 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb5900689e0 0x7fb59019a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:19.943 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.942+0000 7fb598200640 1 -- 172.21.15.19:0/4096144331 >> 172.21.15.19:0/4096144331 conn(0x7fb5900777e0 msgr2=0x7fb59010e330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:29:19.943 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.942+0000 7fb598200640 1 -- 172.21.15.19:0/4096144331 shutdown_connections 2024-03-18T15:29:19.943 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:19.942+0000 7fb598200640 1 -- 172.21.15.19:0/4096144331 wait complete. 2024-03-18T15:29:19.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:29:19 smithi104 ceph-mon[36070]: pgmap v338: 129 pgs: 129 active+clean; 1.8 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 242 B/s wr, 12 op/s 2024-03-18T15:29:19.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:29:19 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/3422504909' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T15:29:19.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:29:19 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/1152537234' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T15:29:19.958 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:29:19 smithi019 ceph-mon[31522]: pgmap v338: 129 pgs: 129 active+clean; 1.8 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 242 B/s wr, 12 op/s 2024-03-18T15:29:19.958 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:29:19 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/3422504909' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T15:29:19.958 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:29:19 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/1152537234' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T15:29:19.961 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-18T15:29:20.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:29:19 smithi033 ceph-mon[36648]: pgmap v338: 129 pgs: 129 active+clean; 1.8 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 242 B/s wr, 12 op/s 2024-03-18T15:29:20.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:29:19 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/3422504909' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T15:29:20.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:29:19 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/1152537234' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T15:29:20.174 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.174+0000 7f8c26c00640 1 -- 172.21.15.19:0/1308008709 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8c20102010 msgr2=0x7f8c2010e520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:29:20.174 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.174+0000 7f8c26c00640 1 --2- 172.21.15.19:0/1308008709 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8c20102010 0x7f8c2010e520 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f8c18009be0 tx=0x7f8c1802bef0 comp rx=0 tx=0).stop 2024-03-18T15:29:20.174 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.174+0000 7f8c26c00640 1 -- 172.21.15.19:0/1308008709 shutdown_connections 2024-03-18T15:29:20.175 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.174+0000 7f8c26c00640 1 --2- 172.21.15.19:0/1308008709 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8c20102010 0x7f8c2010e520 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:20.175 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.174+0000 7f8c26c00640 1 --2- 172.21.15.19:0/1308008709 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8c20101650 0x7f8c20101ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:20.175 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.174+0000 7f8c26c00640 1 --2- 172.21.15.19:0/1308008709 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8c20107660 0x7f8c20107a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:20.175 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.174+0000 7f8c26c00640 1 -- 172.21.15.19:0/1308008709 >> 172.21.15.19:0/1308008709 conn(0x7f8c200fd140 msgr2=0x7f8c200ff560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:29:20.175 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.174+0000 7f8c26c00640 1 -- 172.21.15.19:0/1308008709 shutdown_connections 2024-03-18T15:29:20.175 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.174+0000 7f8c26c00640 1 -- 172.21.15.19:0/1308008709 wait complete. 2024-03-18T15:29:20.177 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.175+0000 7f8c26c00640 1 Processor -- start 2024-03-18T15:29:20.178 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.175+0000 7f8c26c00640 1 -- start start 2024-03-18T15:29:20.178 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.175+0000 7f8c26c00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8c20101650 0x7f8c20073ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:29:20.178 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.175+0000 7f8c26c00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8c20102010 0x7f8c20074020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:29:20.178 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.175+0000 7f8c26c00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8c20107660 0x7f8c20074560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:29:20.178 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.175+0000 7f8c26c00640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f8c20111a70 con 0x7f8c20107660 2024-03-18T15:29:20.178 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.175+0000 7f8c26c00640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8c201118f0 con 0x7f8c20101650 2024-03-18T15:29:20.178 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.175+0000 7f8c26c00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f8c20111bf0 con 0x7f8c20102010 2024-03-18T15:29:20.178 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.176+0000 7f8c25800640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8c20101650 0x7f8c20073ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:29:20.178 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.176+0000 7f8c24e00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8c20102010 0x7f8c20074020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:29:20.178 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.176+0000 7f8c25800640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8c20101650 0x7f8c20073ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:49024/0 (socket says 172.21.15.19:49024) 2024-03-18T15:29:20.179 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.176+0000 7f8c25800640 1 -- 172.21.15.19:0/948760935 learned_addr learned my addr 172.21.15.19:0/948760935 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:29:20.179 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.176+0000 7f8c26200640 1 --2- 172.21.15.19:0/948760935 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8c20107660 0x7f8c20074560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:29:20.179 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.176+0000 7f8c25800640 1 -- 172.21.15.19:0/948760935 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8c20102010 msgr2=0x7f8c20074020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:29:20.179 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.176+0000 7f8c25800640 1 --2- 172.21.15.19:0/948760935 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8c20102010 0x7f8c20074020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:20.179 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.176+0000 7f8c25800640 1 -- 172.21.15.19:0/948760935 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8c20107660 msgr2=0x7f8c20074560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:29:20.179 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.176+0000 7f8c25800640 1 --2- 172.21.15.19:0/948760935 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8c20107660 0x7f8c20074560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:20.179 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.176+0000 7f8c25800640 1 -- 172.21.15.19:0/948760935 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c201a63d0 con 0x7f8c20101650 2024-03-18T15:29:20.179 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.176+0000 7f8c26200640 1 --2- 172.21.15.19:0/948760935 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8c20107660 0x7f8c20074560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T15:29:20.179 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.176+0000 7f8c25800640 1 --2- 172.21.15.19:0/948760935 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8c20101650 0x7f8c20073ae0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f8c1400e9c0 tx=0x7f8c1400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:29:20.179 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.176+0000 7f8c1e000640 1 -- 172.21.15.19:0/948760935 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c1400cde0 con 0x7f8c20101650 2024-03-18T15:29:20.179 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.176+0000 7f8c1e000640 1 -- 172.21.15.19:0/948760935 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8c140035f0 con 0x7f8c20101650 2024-03-18T15:29:20.179 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.176+0000 7f8c1e000640 1 -- 172.21.15.19:0/948760935 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c14010690 con 0x7f8c20101650 2024-03-18T15:29:20.180 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.176+0000 7f8c26c00640 1 -- 172.21.15.19:0/948760935 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c201a66c0 con 0x7f8c20101650 2024-03-18T15:29:20.180 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.177+0000 7f8c26c00640 1 -- 172.21.15.19:0/948760935 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c201a6c10 con 0x7f8c20101650 2024-03-18T15:29:20.180 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.178+0000 7f8c26c00640 1 -- 172.21.15.19:0/948760935 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c20068940 con 0x7f8c20101650 2024-03-18T15:29:20.180 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.180+0000 7f8c1e000640 1 -- 172.21.15.19:0/948760935 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 27) ==== 137421+0+0 (secure 0 0 0) 0x7f8c140037a0 con 0x7f8c20101650 2024-03-18T15:29:20.180 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.181+0000 7f8c1e000640 1 --2- 172.21.15.19:0/948760935 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f8c080a2970 0x7f8c080a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:29:20.181 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.181+0000 7f8c1e000640 1 -- 172.21.15.19:0/948760935 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 9597+0+0 (secure 0 0 0) 0x7f8c140cf5e0 con 0x7f8c20101650 2024-03-18T15:29:20.181 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.181+0000 7f8c24e00640 1 --2- 172.21.15.19:0/948760935 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f8c080a2970 0x7f8c080a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:29:20.181 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.182+0000 7f8c24e00640 1 --2- 172.21.15.19:0/948760935 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f8c080a2970 0x7f8c080a4e30 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8c20075570 tx=0x7f8c0c005f90 comp rx=0 tx=0).ready entity=mgr.25466 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:29:20.190 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.190+0000 7f8c1e000640 1 -- 172.21.15.19:0/948760935 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8c1409bf10 con 0x7f8c20101650 2024-03-18T15:29:20.395 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.395+0000 7f8c26c00640 1 -- 172.21.15.19:0/948760935 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f8c20109bc0 con 0x7f8c20101650 2024-03-18T15:29:20.395 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.395+0000 7f8c1e000640 1 -- 172.21.15.19:0/948760935 <== mon.2 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v105) ==== 103+0+11 (secure 0 0 0) 0x7f8c1409b810 con 0x7f8c20101650 2024-03-18T15:29:20.395 INFO:teuthology.orchestra.run.smithi019.stdout:pg_num: 32 2024-03-18T15:29:20.398 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.398+0000 7f8c26c00640 1 -- 172.21.15.19:0/948760935 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f8c080a2970 msgr2=0x7f8c080a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:29:20.398 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.398+0000 7f8c26c00640 1 --2- 172.21.15.19:0/948760935 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f8c080a2970 0x7f8c080a4e30 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8c20075570 tx=0x7f8c0c005f90 comp rx=0 tx=0).stop 2024-03-18T15:29:20.399 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.398+0000 7f8c26c00640 1 -- 172.21.15.19:0/948760935 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8c20101650 msgr2=0x7f8c20073ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:29:20.399 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.398+0000 7f8c26c00640 1 --2- 172.21.15.19:0/948760935 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8c20101650 0x7f8c20073ae0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f8c1400e9c0 tx=0x7f8c1400ee80 comp rx=0 tx=0).stop 2024-03-18T15:29:20.399 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.398+0000 7f8c26c00640 1 -- 172.21.15.19:0/948760935 shutdown_connections 2024-03-18T15:29:20.399 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.398+0000 7f8c26c00640 1 --2- 172.21.15.19:0/948760935 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f8c080a2970 0x7f8c080a4e30 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:20.399 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.398+0000 7f8c26c00640 1 --2- 172.21.15.19:0/948760935 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8c20107660 0x7f8c20074560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:20.399 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.398+0000 7f8c26c00640 1 --2- 172.21.15.19:0/948760935 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8c20102010 0x7f8c20074020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:20.399 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.398+0000 7f8c26c00640 1 --2- 172.21.15.19:0/948760935 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8c20101650 0x7f8c20073ae0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:20.399 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.398+0000 7f8c26c00640 1 -- 172.21.15.19:0/948760935 >> 172.21.15.19:0/948760935 conn(0x7f8c200fd140 msgr2=0x7f8c200fd930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:29:20.399 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.399+0000 7f8c26c00640 1 -- 172.21.15.19:0/948760935 shutdown_connections 2024-03-18T15:29:20.399 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.399+0000 7f8c26c00640 1 -- 172.21.15.19:0/948760935 wait complete. 2024-03-18T15:29:20.419 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-18T15:29:20.648 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.646+0000 7ff936400640 1 -- 172.21.15.19:0/3480498715 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff930103980 msgr2=0x7ff930103e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:29:20.648 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.646+0000 7ff936400640 1 --2- 172.21.15.19:0/3480498715 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff930103980 0x7ff930103e00 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7ff918009900 tx=0x7ff91802bc50 comp rx=0 tx=0).stop 2024-03-18T15:29:20.648 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.646+0000 7ff936400640 1 -- 172.21.15.19:0/3480498715 shutdown_connections 2024-03-18T15:29:20.648 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.646+0000 7ff936400640 1 --2- 172.21.15.19:0/3480498715 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff930104340 0x7ff930110850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:20.648 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.646+0000 7ff936400640 1 --2- 172.21.15.19:0/3480498715 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff930103980 0x7ff930103e00 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:20.648 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.646+0000 7ff936400640 1 --2- 172.21.15.19:0/3480498715 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff930109990 0x7ff930109d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:20.648 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.646+0000 7ff936400640 1 -- 172.21.15.19:0/3480498715 >> 172.21.15.19:0/3480498715 conn(0x7ff9300ff490 msgr2=0x7ff9301018b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:29:20.648 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.646+0000 7ff936400640 1 -- 172.21.15.19:0/3480498715 shutdown_connections 2024-03-18T15:29:20.648 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.646+0000 7ff936400640 1 -- 172.21.15.19:0/3480498715 wait complete. 2024-03-18T15:29:20.648 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.647+0000 7ff936400640 1 Processor -- start 2024-03-18T15:29:20.649 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.647+0000 7ff936400640 1 -- start start 2024-03-18T15:29:20.649 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.647+0000 7ff936400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff930103980 0x7ff93019a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:29:20.649 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.648+0000 7ff936400640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff930104340 0x7ff93019a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:29:20.649 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.648+0000 7ff936400640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff930109990 0x7ff93019eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:29:20.649 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.648+0000 7ff936400640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7ff930113df0 con 0x7ff930104340 2024-03-18T15:29:20.649 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.648+0000 7ff936400640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff930113c70 con 0x7ff930103980 2024-03-18T15:29:20.649 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.648+0000 7ff936400640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7ff930113f70 con 0x7ff930109990 2024-03-18T15:29:20.649 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.648+0000 7ff92f400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff930103980 0x7ff93019a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:29:20.649 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.648+0000 7ff92f400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff930103980 0x7ff93019a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:49038/0 (socket says 172.21.15.19:49038) 2024-03-18T15:29:20.649 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.648+0000 7ff92fe00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff930109990 0x7ff93019eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:29:20.649 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.648+0000 7ff92f400640 1 -- 172.21.15.19:0/3608112181 learned_addr learned my addr 172.21.15.19:0/3608112181 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:29:20.649 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.648+0000 7ff92ea00640 1 --2- 172.21.15.19:0/3608112181 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff930104340 0x7ff93019a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:29:20.651 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.648+0000 7ff92fe00640 1 -- 172.21.15.19:0/3608112181 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff930103980 msgr2=0x7ff93019a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:29:20.651 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.648+0000 7ff92fe00640 1 --2- 172.21.15.19:0/3608112181 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff930103980 0x7ff93019a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:20.651 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.648+0000 7ff92fe00640 1 -- 172.21.15.19:0/3608112181 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff930104340 msgr2=0x7ff93019a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:29:20.651 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.648+0000 7ff92fe00640 1 --2- 172.21.15.19:0/3608112181 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff930104340 0x7ff93019a810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:20.651 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.648+0000 7ff92fe00640 1 -- 172.21.15.19:0/3608112181 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff93019f2c0 con 0x7ff930109990 2024-03-18T15:29:20.651 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.648+0000 7ff92ea00640 1 --2- 172.21.15.19:0/3608112181 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff930104340 0x7ff93019a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T15:29:20.651 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.648+0000 7ff92f400640 1 --2- 172.21.15.19:0/3608112181 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff930103980 0x7ff93019a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:29:20.651 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.649+0000 7ff92fe00640 1 --2- 172.21.15.19:0/3608112181 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff930109990 0x7ff93019eb90 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7ff92800e940 tx=0x7ff92800ee00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:29:20.651 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.649+0000 7ff927e00640 1 -- 172.21.15.19:0/3608112181 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff928004d20 con 0x7ff930109990 2024-03-18T15:29:20.651 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.649+0000 7ff927e00640 1 -- 172.21.15.19:0/3608112181 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff928005020 con 0x7ff930109990 2024-03-18T15:29:20.651 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.649+0000 7ff936400640 1 -- 172.21.15.19:0/3608112181 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff93019f5b0 con 0x7ff930109990 2024-03-18T15:29:20.652 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.649+0000 7ff927e00640 1 -- 172.21.15.19:0/3608112181 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9280116b0 con 0x7ff930109990 2024-03-18T15:29:20.652 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.649+0000 7ff936400640 1 -- 172.21.15.19:0/3608112181 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9301a6e40 con 0x7ff930109990 2024-03-18T15:29:20.652 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.650+0000 7ff936400640 1 -- 172.21.15.19:0/3608112181 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff930068a20 con 0x7ff930109990 2024-03-18T15:29:20.652 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.652+0000 7ff927e00640 1 -- 172.21.15.19:0/3608112181 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 27) ==== 137421+0+0 (secure 0 0 0) 0x7ff9280054d0 con 0x7ff930109990 2024-03-18T15:29:20.653 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.653+0000 7ff927e00640 1 --2- 172.21.15.19:0/3608112181 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7ff9140a2970 0x7ff9140a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:29:20.653 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.654+0000 7ff927e00640 1 -- 172.21.15.19:0/3608112181 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 9597+0+0 (secure 0 0 0) 0x7ff9280cf3e0 con 0x7ff930109990 2024-03-18T15:29:20.653 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.654+0000 7ff92f400640 1 --2- 172.21.15.19:0/3608112181 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7ff9140a2970 0x7ff9140a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:29:20.654 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.654+0000 7ff92f400640 1 --2- 172.21.15.19:0/3608112181 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7ff9140a2970 0x7ff9140a4e30 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff920011490 tx=0x7ff920005f90 comp rx=0 tx=0).ready entity=mgr.25466 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:29:20.657 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.658+0000 7ff927e00640 1 -- 172.21.15.19:0/3608112181 <== mon.1 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff928016030 con 0x7ff930109990 2024-03-18T15:29:20.877 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.877+0000 7ff936400640 1 -- 172.21.15.19:0/3608112181 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7ff93002d1b0 con 0x7ff930109990 2024-03-18T15:29:20.877 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.878+0000 7ff927e00640 1 -- 172.21.15.19:0/3608112181 <== mon.1 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 17 entries v105) ==== 81+0+1001 (secure 0 0 0) 0x7ff92809bcb0 con 0x7ff930109990 2024-03-18T15:29:20.878 INFO:teuthology.orchestra.run.smithi019.stdout:172.21.15.19:0/1895666842 2024-03-18T16:23:20.095509+0000 2024-03-18T15:29:20.878 INFO:teuthology.orchestra.run.smithi019.stdout:172.21.15.19:6801/3264337803 2024-03-19T15:06:38.688405+0000 2024-03-18T15:29:20.878 INFO:teuthology.orchestra.run.smithi019.stdout:172.21.15.19:6800/3264337803 2024-03-19T15:06:38.688405+0000 2024-03-18T15:29:20.878 INFO:teuthology.orchestra.run.smithi019.stdout:172.21.15.19:0/2793980440 2024-03-19T15:06:38.688405+0000 2024-03-18T15:29:20.878 INFO:teuthology.orchestra.run.smithi019.stdout:172.21.15.19:6801/2776008962 2024-03-19T15:07:14.150983+0000 2024-03-18T15:29:20.878 INFO:teuthology.orchestra.run.smithi019.stdout:172.21.15.19:0/2316969005 2024-03-19T15:07:14.150983+0000 2024-03-18T15:29:20.878 INFO:teuthology.orchestra.run.smithi019.stdout:172.21.15.19:0/84203799 2024-03-19T15:18:29.726021+0000 2024-03-18T15:29:20.878 INFO:teuthology.orchestra.run.smithi019.stdout:172.21.15.19:6800/2017714236 2024-03-19T15:18:29.726021+0000 2024-03-18T15:29:20.878 INFO:teuthology.orchestra.run.smithi019.stdout:172.21.15.19:0/2737223213 2024-03-19T15:07:14.150983+0000 2024-03-18T15:29:20.879 INFO:teuthology.orchestra.run.smithi019.stdout:172.21.15.19:0/3860783274 2024-03-19T15:07:14.150983+0000 2024-03-18T15:29:20.879 INFO:teuthology.orchestra.run.smithi019.stdout:172.21.15.19:6800/2776008962 2024-03-19T15:07:14.150983+0000 2024-03-18T15:29:20.879 INFO:teuthology.orchestra.run.smithi019.stdout:172.21.15.19:0/1024556841 2024-03-19T15:18:29.726021+0000 2024-03-18T15:29:20.879 INFO:teuthology.orchestra.run.smithi019.stdout:172.21.15.19:0/3166705295 2024-03-19T15:06:38.688405+0000 2024-03-18T15:29:20.879 INFO:teuthology.orchestra.run.smithi019.stdout:172.21.15.19:6801/2017714236 2024-03-19T15:18:29.726021+0000 2024-03-18T15:29:20.879 INFO:teuthology.orchestra.run.smithi019.stdout:172.21.15.19:0/176699025 2024-03-19T15:06:38.688405+0000 2024-03-18T15:29:20.879 INFO:teuthology.orchestra.run.smithi019.stdout:172.21.15.19:0/2730531676 2024-03-19T15:18:29.726021+0000 2024-03-18T15:29:20.879 INFO:teuthology.orchestra.run.smithi019.stdout:172.21.15.19:0/2353244552 2024-03-19T15:18:29.726021+0000 2024-03-18T15:29:20.879 INFO:teuthology.orchestra.run.smithi019.stderr:listed 17 entries 2024-03-18T15:29:20.881 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.881+0000 7ff936400640 1 -- 172.21.15.19:0/3608112181 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7ff9140a2970 msgr2=0x7ff9140a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:29:20.881 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.881+0000 7ff936400640 1 --2- 172.21.15.19:0/3608112181 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7ff9140a2970 0x7ff9140a4e30 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff920011490 tx=0x7ff920005f90 comp rx=0 tx=0).stop 2024-03-18T15:29:20.881 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.881+0000 7ff936400640 1 -- 172.21.15.19:0/3608112181 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff930109990 msgr2=0x7ff93019eb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:29:20.881 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.881+0000 7ff936400640 1 --2- 172.21.15.19:0/3608112181 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff930109990 0x7ff93019eb90 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7ff92800e940 tx=0x7ff92800ee00 comp rx=0 tx=0).stop 2024-03-18T15:29:20.881 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.881+0000 7ff936400640 1 -- 172.21.15.19:0/3608112181 shutdown_connections 2024-03-18T15:29:20.881 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.881+0000 7ff936400640 1 --2- 172.21.15.19:0/3608112181 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7ff9140a2970 0x7ff9140a4e30 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:20.882 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.881+0000 7ff936400640 1 --2- 172.21.15.19:0/3608112181 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff930109990 0x7ff93019eb90 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:20.882 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.881+0000 7ff936400640 1 --2- 172.21.15.19:0/3608112181 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff930104340 0x7ff93019a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:20.882 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.881+0000 7ff936400640 1 --2- 172.21.15.19:0/3608112181 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff930103980 0x7ff93019a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:20.882 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.881+0000 7ff936400640 1 -- 172.21.15.19:0/3608112181 >> 172.21.15.19:0/3608112181 conn(0x7ff9300ff490 msgr2=0x7ff9300ffcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:29:20.882 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.881+0000 7ff936400640 1 -- 172.21.15.19:0/3608112181 shutdown_connections 2024-03-18T15:29:20.882 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:20.881+0000 7ff936400640 1 -- 172.21.15.19:0/3608112181 wait complete. 2024-03-18T15:29:20.901 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-18T15:29:20.902 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T15:29:20.902 DEBUG:teuthology.orchestra.run.smithi019:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-18T15:29:20.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:29:20 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/4096144331' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T15:29:20.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:29:20 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/948760935' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T15:29:20.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:29:20 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/4096144331' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T15:29:20.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:29:20 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/948760935' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T15:29:21.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:29:20 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/4096144331' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T15:29:21.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:29:20 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/948760935' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T15:29:21.054 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi019.front.sepia.ceph.com 2024-03-18T15:29:21.054 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T15:29:21.054 DEBUG:teuthology.orchestra.run.smithi019:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-18T15:29:21.093 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T15:29:21.094 DEBUG:teuthology.orchestra.run.smithi019:> ip netns list 2024-03-18T15:29:21.176 INFO:teuthology.orchestra.run.smithi019.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-18T15:29:21.176 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T15:29:21.176 DEBUG:teuthology.orchestra.run.smithi019:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-18T15:29:21.317 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T15:29:21.318 DEBUG:teuthology.orchestra.run.smithi019:> sudo ip link delete ceph-brx 2024-03-18T15:29:21.433 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-18T15:29:21.444 INFO:tasks.cephadm:Teardown begin 2024-03-18T15:29:21.444 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-18T15:29:21.445 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T15:29:21.651 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:21.650+0000 7f4380c00640 1 -- 172.21.15.19:0/1545249651 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f437c109980 msgr2=0x7f437c109d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:29:21.651 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:21.650+0000 7f4380c00640 1 --2- 172.21.15.19:0/1545249651 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f437c109980 0x7f437c109d60 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f436c009900 tx=0x7f436c02bc40 comp rx=0 tx=0).stop 2024-03-18T15:29:21.651 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:21.651+0000 7f4380c00640 1 -- 172.21.15.19:0/1545249651 shutdown_connections 2024-03-18T15:29:21.651 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:21.651+0000 7f4380c00640 1 --2- 172.21.15.19:0/1545249651 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f437c104330 0x7f437c110840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:21.651 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:21.651+0000 7f4380c00640 1 --2- 172.21.15.19:0/1545249651 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f437c103970 0x7f437c103df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:21.651 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:21.651+0000 7f4380c00640 1 --2- 172.21.15.19:0/1545249651 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f437c109980 0x7f437c109d60 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:21.651 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:21.651+0000 7f4380c00640 1 -- 172.21.15.19:0/1545249651 >> 172.21.15.19:0/1545249651 conn(0x7f437c0ff4a0 msgr2=0x7f437c1018c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:29:21.651 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:21.651+0000 7f4380c00640 1 -- 172.21.15.19:0/1545249651 shutdown_connections 2024-03-18T15:29:21.651 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:21.651+0000 7f4380c00640 1 -- 172.21.15.19:0/1545249651 wait complete. 2024-03-18T15:29:21.653 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:21.652+0000 7f4380c00640 1 Processor -- start 2024-03-18T15:29:21.654 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:21.652+0000 7f4380c00640 1 -- start start 2024-03-18T15:29:21.654 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:21.652+0000 7f4380c00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f437c103970 0x7f437c19a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:29:21.654 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:21.652+0000 7f4380c00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f437c104330 0x7f437c19a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:29:21.654 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:21.652+0000 7f4380c00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f437c109980 0x7f437c19eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:29:21.654 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:21.652+0000 7f4380c00640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f437c113d90 con 0x7f437c109980 2024-03-18T15:29:21.654 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:21.652+0000 7f4380c00640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f437c113c10 con 0x7f437c103970 2024-03-18T15:29:21.654 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:21.652+0000 7f4380c00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f437c113f10 con 0x7f437c104330 2024-03-18T15:29:21.654 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:21.652+0000 7f4379e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f437c103970 0x7f437c19a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:29:21.654 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:21.652+0000 7f4379400640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f437c104330 0x7f437c19a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:29:21.654 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:21.652+0000 7f4379400640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f437c104330 0x7f437c19a810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.19:60646/0 (socket says 172.21.15.19:60646) 2024-03-18T15:29:21.654 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:21.652+0000 7f4379e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f437c103970 0x7f437c19a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:49046/0 (socket says 172.21.15.19:49046) 2024-03-18T15:29:21.654 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:21.652+0000 7f4379400640 1 -- 172.21.15.19:0/1942199253 learned_addr learned my addr 172.21.15.19:0/1942199253 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:29:21.654 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:21.652+0000 7f437a800640 1 --2- 172.21.15.19:0/1942199253 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f437c109980 0x7f437c19eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:29:21.655 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:21.653+0000 7f4379e00640 1 -- 172.21.15.19:0/1942199253 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f437c104330 msgr2=0x7f437c19a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:29:21.655 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:21.653+0000 7f4379e00640 1 --2- 172.21.15.19:0/1942199253 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f437c104330 0x7f437c19a810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:21.655 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:21.653+0000 7f4379e00640 1 -- 172.21.15.19:0/1942199253 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f437c109980 msgr2=0x7f437c19eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:29:21.655 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:21.653+0000 7f4379e00640 1 --2- 172.21.15.19:0/1942199253 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f437c109980 0x7f437c19eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:21.655 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:21.653+0000 7f4379e00640 1 -- 172.21.15.19:0/1942199253 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f437c19f2c0 con 0x7f437c103970 2024-03-18T15:29:21.655 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:21.653+0000 7f4379e00640 1 --2- 172.21.15.19:0/1942199253 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f437c103970 0x7f437c19a2d0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f436c000c00 tx=0x7f436c004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:29:21.655 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:21.653+0000 7f4372a00640 1 -- 172.21.15.19:0/1942199253 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f436c041070 con 0x7f437c103970 2024-03-18T15:29:21.655 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:21.653+0000 7f4372a00640 1 -- 172.21.15.19:0/1942199253 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f436c0054e0 con 0x7f437c103970 2024-03-18T15:29:21.655 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:21.653+0000 7f4372a00640 1 -- 172.21.15.19:0/1942199253 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f436c003c60 con 0x7f437c103970 2024-03-18T15:29:21.655 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:21.653+0000 7f4380c00640 1 -- 172.21.15.19:0/1942199253 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f437c19f550 con 0x7f437c103970 2024-03-18T15:29:21.655 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:21.654+0000 7f4380c00640 1 -- 172.21.15.19:0/1942199253 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f437c1a6e40 con 0x7f437c103970 2024-03-18T15:29:21.656 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:21.655+0000 7f4380c00640 1 -- 172.21.15.19:0/1942199253 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f437c1050b0 con 0x7f437c103970 2024-03-18T15:29:21.656 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:21.657+0000 7f4372a00640 1 -- 172.21.15.19:0/1942199253 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 27) ==== 137421+0+0 (secure 0 0 0) 0x7f436c044070 con 0x7f437c103970 2024-03-18T15:29:21.658 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:21.658+0000 7f4372a00640 1 --2- 172.21.15.19:0/1942199253 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f43600a29f0 0x7f43600a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:29:21.658 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:21.658+0000 7f4372a00640 1 -- 172.21.15.19:0/1942199253 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 9597+0+0 (secure 0 0 0) 0x7f436c0f0ec0 con 0x7f437c103970 2024-03-18T15:29:21.658 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:21.658+0000 7f4379400640 1 --2- 172.21.15.19:0/1942199253 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f43600a29f0 0x7f43600a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:29:21.658 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:21.659+0000 7f4379400640 1 --2- 172.21.15.19:0/1942199253 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f43600a29f0 0x7f43600a4eb0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f437c1029a0 tx=0x7f4368005f90 comp rx=0 tx=0).ready entity=mgr.25466 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:29:21.660 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:21.660+0000 7f4372a00640 1 -- 172.21.15.19:0/1942199253 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f436c0bd250 con 0x7f437c103970 2024-03-18T15:29:21.909 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:21.909+0000 7f4380c00640 1 -- 172.21.15.19:0/1942199253 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f437c068a90 con 0x7f437c103970 2024-03-18T15:29:21.911 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:21.911+0000 7f4372a00640 1 -- 172.21.15.19:0/1942199253 <== mon.2 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11927 (secure 0 0 0) 0x7f436c0c12f0 con 0x7f437c103970 2024-03-18T15:29:21.911 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-18T15:29:21.912 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6835/1135329545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6834","nonce":1135329545},{"type":"v1","addr":"172.21.15.19:6835","nonce":1135329545}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14457,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6839/3833518557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6838","nonce":3833518557},{"type":"v1","addr":"172.21.15.19:6839","nonce":3833518557}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T15:15:10.589737+0000","modified":"2024-03-18T15:22:32.186227+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24560,"mds_1":24529,"mds_2":24511,"mds_3":24506,"mds_4":24499},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14475":{"gid":14475,"name":"a","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.19:6841/3176583319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6840","nonce":3176583319},{"type":"v1","addr":"172.21.15.19:6841","nonce":3176583319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24433":{"gid":24433,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6833/482725809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":482725809},{"type":"v1","addr":"172.21.15.104:6833","nonce":482725809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24434":{"gid":24434,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/2272279159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":2272279159},{"type":"v1","addr":"172.21.15.33:6833","nonce":2272279159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24469":{"gid":24469,"name":"d","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.19:6837/1778019859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6836","nonce":1778019859},{"type":"v1","addr":"172.21.15.19:6837","nonce":1778019859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6835/4051613151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":4051613151},{"type":"v1","addr":"172.21.15.33:6835","nonce":4051613151}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":17,"addr":"172.21.15.104:6835/3788595957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3788595957},{"type":"v1","addr":"172.21.15.104:6835","nonce":3788595957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":15,"addr":"172.21.15.33:6837/160187378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":160187378},{"type":"v1","addr":"172.21.15.33:6837","nonce":160187378}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24511":{"gid":24511,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":16,"addr":"172.21.15.104:6837/260984674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":260984674},{"type":"v1","addr":"172.21.15.104:6837","nonce":260984674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24529":{"gid":24529,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.104:6839/805341436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6838","nonce":805341436},{"type":"v1","addr":"172.21.15.104:6839","nonce":805341436}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.33:6839/1903591535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":1903591535},{"type":"v1","addr":"172.21.15.33:6839","nonce":1903591535}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24560,"qdb_cluster":[24560,24511,24529,24506,24499]},"id":1}]} 2024-03-18T15:29:21.912 INFO:teuthology.orchestra.run.smithi019.stderr:dumped fsmap epoch 31 2024-03-18T15:29:21.915 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:21.914+0000 7f4380c00640 1 -- 172.21.15.19:0/1942199253 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f43600a29f0 msgr2=0x7f43600a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:29:21.915 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:21.914+0000 7f4380c00640 1 --2- 172.21.15.19:0/1942199253 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f43600a29f0 0x7f43600a4eb0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f437c1029a0 tx=0x7f4368005f90 comp rx=0 tx=0).stop 2024-03-18T15:29:21.915 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:21.914+0000 7f4380c00640 1 -- 172.21.15.19:0/1942199253 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f437c103970 msgr2=0x7f437c19a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:29:21.915 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:21.914+0000 7f4380c00640 1 --2- 172.21.15.19:0/1942199253 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f437c103970 0x7f437c19a2d0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f436c000c00 tx=0x7f436c004270 comp rx=0 tx=0).stop 2024-03-18T15:29:21.915 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:21.914+0000 7f4380c00640 1 -- 172.21.15.19:0/1942199253 shutdown_connections 2024-03-18T15:29:21.915 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:21.914+0000 7f4380c00640 1 --2- 172.21.15.19:0/1942199253 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f43600a29f0 0x7f43600a4eb0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:21.915 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:21.914+0000 7f4380c00640 1 --2- 172.21.15.19:0/1942199253 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f437c109980 0x7f437c19eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:21.915 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:21.914+0000 7f4380c00640 1 --2- 172.21.15.19:0/1942199253 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f437c104330 0x7f437c19a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:21.916 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:21.914+0000 7f4380c00640 1 --2- 172.21.15.19:0/1942199253 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f437c103970 0x7f437c19a2d0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:21.916 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:21.914+0000 7f4380c00640 1 -- 172.21.15.19:0/1942199253 >> 172.21.15.19:0/1942199253 conn(0x7f437c0ff4a0 msgr2=0x7f437c0ffae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:29:21.916 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:21.915+0000 7f4380c00640 1 -- 172.21.15.19:0/1942199253 shutdown_connections 2024-03-18T15:29:21.916 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:21.915+0000 7f4380c00640 1 -- 172.21.15.19:0/1942199253 wait complete. 2024-03-18T15:29:21.935 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T15:29:21.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:29:21 smithi104 ceph-mon[36070]: pgmap v339: 129 pgs: 129 active+clean; 1.8 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-18T15:29:21.944 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:29:21 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/3608112181' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-18T15:29:21.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:29:21 smithi019 ceph-mon[31522]: pgmap v339: 129 pgs: 129 active+clean; 1.8 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-18T15:29:21.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:29:21 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/3608112181' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-18T15:29:22.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:29:21 smithi033 ceph-mon[36648]: pgmap v339: 129 pgs: 129 active+clean; 1.8 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-18T15:29:22.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:29:21 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/3608112181' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-18T15:29:22.162 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.162+0000 7fa161e00640 1 -- 172.21.15.19:0/2445088812 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa15c10cee0 msgr2=0x7fa15c114ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:29:22.163 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.162+0000 7fa161e00640 1 --2- 172.21.15.19:0/2445088812 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa15c10cee0 0x7fa15c114ab0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fa154009ca0 tx=0x7fa15402c100 comp rx=0 tx=0).stop 2024-03-18T15:29:22.163 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.162+0000 7fa161e00640 1 -- 172.21.15.19:0/2445088812 shutdown_connections 2024-03-18T15:29:22.163 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.162+0000 7fa161e00640 1 --2- 172.21.15.19:0/2445088812 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa15c10cee0 0x7fa15c114ab0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:22.163 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.162+0000 7fa161e00640 1 --2- 172.21.15.19:0/2445088812 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa15c1087c0 0x7fa15c10c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:22.163 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.162+0000 7fa161e00640 1 --2- 172.21.15.19:0/2445088812 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa15c107e10 0x7fa15c1081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:22.163 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.162+0000 7fa161e00640 1 -- 172.21.15.19:0/2445088812 >> 172.21.15.19:0/2445088812 conn(0x7fa15c077300 msgr2=0x7fa15c079720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:29:22.163 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.162+0000 7fa161e00640 1 -- 172.21.15.19:0/2445088812 shutdown_connections 2024-03-18T15:29:22.163 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.162+0000 7fa161e00640 1 -- 172.21.15.19:0/2445088812 wait complete. 2024-03-18T15:29:22.165 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.163+0000 7fa161e00640 1 Processor -- start 2024-03-18T15:29:22.165 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.163+0000 7fa161e00640 1 -- start start 2024-03-18T15:29:22.165 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.164+0000 7fa161e00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa15c107e10 0x7fa15c19e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:29:22.165 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.164+0000 7fa161e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa15c10cee0 0x7fa15c19e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:29:22.165 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.164+0000 7fa161e00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa15c1a2cc0 0x7fa15c1a3170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:29:22.166 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.164+0000 7fa161e00640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fa15c117f90 con 0x7fa15c1a2cc0 2024-03-18T15:29:22.166 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.164+0000 7fa161e00640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa15c117e10 con 0x7fa15c10cee0 2024-03-18T15:29:22.166 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.164+0000 7fa161e00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fa15c118110 con 0x7fa15c107e10 2024-03-18T15:29:22.166 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.164+0000 7fa160a00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa15c107e10 0x7fa15c19e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:29:22.166 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.164+0000 7fa160a00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa15c107e10 0x7fa15c19e450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.19:33110/0 (socket says 172.21.15.19:33110) 2024-03-18T15:29:22.166 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.164+0000 7fa153e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa15c10cee0 0x7fa15c19e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:29:22.166 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.164+0000 7fa160a00640 1 -- 172.21.15.19:0/162178797 learned_addr learned my addr 172.21.15.19:0/162178797 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:29:22.166 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.164+0000 7fa161400640 1 --2- 172.21.15.19:0/162178797 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa15c1a2cc0 0x7fa15c1a3170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:29:22.166 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.164+0000 7fa160a00640 1 -- 172.21.15.19:0/162178797 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa15c10cee0 msgr2=0x7fa15c19e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:29:22.166 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.164+0000 7fa160a00640 1 --2- 172.21.15.19:0/162178797 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa15c10cee0 0x7fa15c19e990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:22.166 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.164+0000 7fa160a00640 1 -- 172.21.15.19:0/162178797 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa15c1a2cc0 msgr2=0x7fa15c1a3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:29:22.166 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.164+0000 7fa160a00640 1 --2- 172.21.15.19:0/162178797 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa15c1a2cc0 0x7fa15c1a3170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:22.166 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.164+0000 7fa160a00640 1 -- 172.21.15.19:0/162178797 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa15c1a37b0 con 0x7fa15c107e10 2024-03-18T15:29:22.167 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.165+0000 7fa160a00640 1 --2- 172.21.15.19:0/162178797 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa15c107e10 0x7fa15c19e450 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fa14c00e9f0 tx=0x7fa14c00edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:29:22.167 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.165+0000 7fa15a000640 1 -- 172.21.15.19:0/162178797 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa14c00cd80 con 0x7fa15c107e10 2024-03-18T15:29:22.167 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.165+0000 7fa161e00640 1 -- 172.21.15.19:0/162178797 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa15c1a3aa0 con 0x7fa15c107e10 2024-03-18T15:29:22.167 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.165+0000 7fa15a000640 1 -- 172.21.15.19:0/162178797 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa14c005500 con 0x7fa15c107e10 2024-03-18T15:29:22.167 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.165+0000 7fa15a000640 1 -- 172.21.15.19:0/162178797 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa14c010690 con 0x7fa15c107e10 2024-03-18T15:29:22.167 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.165+0000 7fa161e00640 1 -- 172.21.15.19:0/162178797 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa15c1ab1e0 con 0x7fa15c107e10 2024-03-18T15:29:22.167 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.166+0000 7fa161e00640 1 -- 172.21.15.19:0/162178797 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa15c19f750 con 0x7fa15c107e10 2024-03-18T15:29:22.168 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.168+0000 7fa15a000640 1 -- 172.21.15.19:0/162178797 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 27) ==== 137421+0+0 (secure 0 0 0) 0x7fa14c0046c0 con 0x7fa15c107e10 2024-03-18T15:29:22.170 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.170+0000 7fa15a000640 1 --2- 172.21.15.19:0/162178797 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7fa1400a2970 0x7fa1400a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:29:22.170 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.170+0000 7fa15a000640 1 -- 172.21.15.19:0/162178797 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 9597+0+0 (secure 0 0 0) 0x7fa14c0cf6e0 con 0x7fa15c107e10 2024-03-18T15:29:22.170 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.170+0000 7fa153e00640 1 --2- 172.21.15.19:0/162178797 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7fa1400a2970 0x7fa1400a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:29:22.170 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.171+0000 7fa153e00640 1 --2- 172.21.15.19:0/162178797 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7fa1400a2970 0x7fa1400a4e30 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa15c1070c0 tx=0x7fa14400b390 comp rx=0 tx=0).ready entity=mgr.25466 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:29:22.174 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.175+0000 7fa15a000640 1 -- 172.21.15.19:0/162178797 <== mon.1 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa14c09bfb0 con 0x7fa15c107e10 2024-03-18T15:29:22.419 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.419+0000 7fa161e00640 1 -- 172.21.15.19:0/162178797 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa15c068920 con 0x7fa15c107e10 2024-03-18T15:29:22.420 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.420+0000 7fa15a000640 1 -- 172.21.15.19:0/162178797 <== mon.1 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11927 (secure 0 0 0) 0x7fa14c09b8b0 con 0x7fa15c107e10 2024-03-18T15:29:22.420 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-18T15:29:22.420 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6835/1135329545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6834","nonce":1135329545},{"type":"v1","addr":"172.21.15.19:6835","nonce":1135329545}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14457,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6839/3833518557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6838","nonce":3833518557},{"type":"v1","addr":"172.21.15.19:6839","nonce":3833518557}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T15:15:10.589737+0000","modified":"2024-03-18T15:22:32.186227+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24560,"mds_1":24529,"mds_2":24511,"mds_3":24506,"mds_4":24499},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14475":{"gid":14475,"name":"a","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.19:6841/3176583319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6840","nonce":3176583319},{"type":"v1","addr":"172.21.15.19:6841","nonce":3176583319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24433":{"gid":24433,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6833/482725809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":482725809},{"type":"v1","addr":"172.21.15.104:6833","nonce":482725809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24434":{"gid":24434,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6833/2272279159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":2272279159},{"type":"v1","addr":"172.21.15.33:6833","nonce":2272279159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24469":{"gid":24469,"name":"d","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.19:6837/1778019859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6836","nonce":1778019859},{"type":"v1","addr":"172.21.15.19:6837","nonce":1778019859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.33:6835/4051613151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":4051613151},{"type":"v1","addr":"172.21.15.33:6835","nonce":4051613151}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":17,"addr":"172.21.15.104:6835/3788595957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3788595957},{"type":"v1","addr":"172.21.15.104:6835","nonce":3788595957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":15,"addr":"172.21.15.33:6837/160187378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":160187378},{"type":"v1","addr":"172.21.15.33:6837","nonce":160187378}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24511":{"gid":24511,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":16,"addr":"172.21.15.104:6837/260984674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":260984674},{"type":"v1","addr":"172.21.15.104:6837","nonce":260984674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24529":{"gid":24529,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.104:6839/805341436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6838","nonce":805341436},{"type":"v1","addr":"172.21.15.104:6839","nonce":805341436}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.33:6839/1903591535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":1903591535},{"type":"v1","addr":"172.21.15.33:6839","nonce":1903591535}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24560,"qdb_cluster":[24560,24511,24529,24506,24499]},"id":1}]} 2024-03-18T15:29:22.420 INFO:teuthology.orchestra.run.smithi019.stderr:dumped fsmap epoch 31 2024-03-18T15:29:22.423 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.423+0000 7fa161e00640 1 -- 172.21.15.19:0/162178797 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7fa1400a2970 msgr2=0x7fa1400a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:29:22.423 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.423+0000 7fa161e00640 1 --2- 172.21.15.19:0/162178797 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7fa1400a2970 0x7fa1400a4e30 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa15c1070c0 tx=0x7fa14400b390 comp rx=0 tx=0).stop 2024-03-18T15:29:22.424 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.423+0000 7fa161e00640 1 -- 172.21.15.19:0/162178797 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa15c107e10 msgr2=0x7fa15c19e450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:29:22.424 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.423+0000 7fa161e00640 1 --2- 172.21.15.19:0/162178797 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa15c107e10 0x7fa15c19e450 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fa14c00e9f0 tx=0x7fa14c00edf0 comp rx=0 tx=0).stop 2024-03-18T15:29:22.424 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.423+0000 7fa161e00640 1 -- 172.21.15.19:0/162178797 shutdown_connections 2024-03-18T15:29:22.424 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.423+0000 7fa161e00640 1 --2- 172.21.15.19:0/162178797 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7fa1400a2970 0x7fa1400a4e30 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:22.424 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.423+0000 7fa161e00640 1 --2- 172.21.15.19:0/162178797 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa15c1a2cc0 0x7fa15c1a3170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:22.424 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.423+0000 7fa161e00640 1 --2- 172.21.15.19:0/162178797 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa15c10cee0 0x7fa15c19e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:22.424 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.423+0000 7fa161e00640 1 --2- 172.21.15.19:0/162178797 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa15c107e10 0x7fa15c19e450 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:22.424 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.423+0000 7fa161e00640 1 -- 172.21.15.19:0/162178797 >> 172.21.15.19:0/162178797 conn(0x7fa15c077300 msgr2=0x7fa15c0795c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:29:22.424 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.424+0000 7fa161e00640 1 -- 172.21.15.19:0/162178797 shutdown_connections 2024-03-18T15:29:22.424 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.424+0000 7fa161e00640 1 -- 172.21.15.19:0/162178797 wait complete. 2024-03-18T15:29:22.445 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T15:29:22.710 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.709+0000 7f249a400640 1 -- 172.21.15.19:0/2177582120 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2494107840 msgr2=0x7f2494107c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:29:22.710 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.709+0000 7f249a400640 1 --2- 172.21.15.19:0/2177582120 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2494107840 0x7f2494107c20 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f2484009890 tx=0x7f248402bb70 comp rx=0 tx=0).stop 2024-03-18T15:29:22.710 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.709+0000 7f249a400640 1 -- 172.21.15.19:0/2177582120 shutdown_connections 2024-03-18T15:29:22.710 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.709+0000 7f249a400640 1 --2- 172.21.15.19:0/2177582120 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f249406a900 0x7f249410e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:22.710 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.709+0000 7f249a400640 1 --2- 172.21.15.19:0/2177582120 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2494069f40 0x7f249406a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:22.710 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.709+0000 7f249a400640 1 --2- 172.21.15.19:0/2177582120 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2494107840 0x7f2494107c20 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:22.710 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.709+0000 7f249a400640 1 -- 172.21.15.19:0/2177582120 >> 172.21.15.19:0/2177582120 conn(0x7f2494076a60 msgr2=0x7f2494076e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:29:22.710 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.709+0000 7f249a400640 1 -- 172.21.15.19:0/2177582120 shutdown_connections 2024-03-18T15:29:22.710 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.710+0000 7f249a400640 1 -- 172.21.15.19:0/2177582120 wait complete. 2024-03-18T15:29:22.714 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.711+0000 7f249a400640 1 Processor -- start 2024-03-18T15:29:22.715 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.711+0000 7f249a400640 1 -- start start 2024-03-18T15:29:22.715 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.711+0000 7f249a400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2494069f40 0x7f249419a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:29:22.715 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.711+0000 7f249a400640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f249406a900 0x7f249419a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:29:22.715 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.712+0000 7f249a400640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2494107840 0x7f249419eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:29:22.715 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.712+0000 7f249a400640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f2494111b50 con 0x7f249406a900 2024-03-18T15:29:22.715 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.712+0000 7f249a400640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f24941119d0 con 0x7f2494069f40 2024-03-18T15:29:22.715 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.712+0000 7f249a400640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f2494111cd0 con 0x7f2494107840 2024-03-18T15:29:22.715 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.712+0000 7f2493400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2494069f40 0x7f249419a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:29:22.715 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.712+0000 7f2493400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2494069f40 0x7f249419a1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.19:50384/0 (socket says 172.21.15.19:50384) 2024-03-18T15:29:22.715 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.712+0000 7f2493400640 1 -- 172.21.15.19:0/1248040415 learned_addr learned my addr 172.21.15.19:0/1248040415 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:29:22.715 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.712+0000 7f2493e00640 1 --2- 172.21.15.19:0/1248040415 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2494107840 0x7f249419eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:29:22.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.712+0000 7f248be00640 1 --2- 172.21.15.19:0/1248040415 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f249406a900 0x7f249419a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:29:22.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.712+0000 7f2493400640 1 -- 172.21.15.19:0/1248040415 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2494107840 msgr2=0x7f249419eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:29:22.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.712+0000 7f2493400640 1 --2- 172.21.15.19:0/1248040415 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2494107840 0x7f249419eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:22.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.712+0000 7f2493400640 1 -- 172.21.15.19:0/1248040415 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f249406a900 msgr2=0x7f249419a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:29:22.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.712+0000 7f2493400640 1 --2- 172.21.15.19:0/1248040415 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f249406a900 0x7f249419a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:22.717 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.712+0000 7f2493400640 1 -- 172.21.15.19:0/1248040415 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f249419f1e0 con 0x7f2494069f40 2024-03-18T15:29:22.717 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.712+0000 7f248be00640 1 --2- 172.21.15.19:0/1248040415 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f249406a900 0x7f249419a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T15:29:22.717 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.713+0000 7f2493400640 1 --2- 172.21.15.19:0/1248040415 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2494069f40 0x7f249419a1f0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f24840057c0 tx=0x7f24840045d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:29:22.717 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.713+0000 7f2490c00640 1 -- 172.21.15.19:0/1248040415 <== mon.2 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2484041020 con 0x7f2494069f40 2024-03-18T15:29:22.717 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.713+0000 7f2490c00640 1 -- 172.21.15.19:0/1248040415 <== mon.2 v2:172.21.15.33:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f248402c9f0 con 0x7f2494069f40 2024-03-18T15:29:22.717 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.713+0000 7f2490c00640 1 -- 172.21.15.19:0/1248040415 <== mon.2 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2484039040 con 0x7f2494069f40 2024-03-18T15:29:22.717 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.713+0000 7f249a400640 1 -- 172.21.15.19:0/1248040415 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f249419f470 con 0x7f2494069f40 2024-03-18T15:29:22.717 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.713+0000 7f249a400640 1 -- 172.21.15.19:0/1248040415 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24941a6d60 con 0x7f2494069f40 2024-03-18T15:29:22.717 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.714+0000 7f249a400640 1 -- 172.21.15.19:0/1248040415 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2494068a90 con 0x7f2494069f40 2024-03-18T15:29:22.719 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.716+0000 7f2490c00640 1 -- 172.21.15.19:0/1248040415 <== mon.2 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 27) ==== 137421+0+0 (secure 0 0 0) 0x7f2484044020 con 0x7f2494069f40 2024-03-18T15:29:22.719 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.717+0000 7f2490c00640 1 --2- 172.21.15.19:0/1248040415 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f24780a29f0 0x7f24780a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:29:22.719 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.717+0000 7f2490c00640 1 -- 172.21.15.19:0/1248040415 <== mon.2 v2:172.21.15.33:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 9597+0+0 (secure 0 0 0) 0x7f24840f0a10 con 0x7f2494069f40 2024-03-18T15:29:22.719 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.718+0000 7f248be00640 1 --2- 172.21.15.19:0/1248040415 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f24780a29f0 0x7f24780a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:29:22.719 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.718+0000 7f248be00640 1 --2- 172.21.15.19:0/1248040415 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f24780a29f0 0x7f24780a4eb0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f24940fda20 tx=0x7f247c009260 comp rx=0 tx=0).ready entity=mgr.25466 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:29:22.721 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.721+0000 7f2490c00640 1 -- 172.21.15.19:0/1248040415 <== mon.2 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f248404b020 con 0x7f2494069f40 2024-03-18T15:29:22.924 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.924+0000 7f249a400640 1 -- 172.21.15.19:0/1248040415 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f24941a6f80 con 0x7f2494069f40 2024-03-18T15:29:22.927 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.926+0000 7f2490c00640 1 -- 172.21.15.19:0/1248040415 <== mon.2 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v105) ==== 74+0+25433 (secure 0 0 0) 0x7f24840bd3f0 con 0x7f2494069f40 2024-03-18T15:29:22.928 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-18T15:29:22.929 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":105,"fsid":"d4dc2ba6-e538-11ee-95c9-87774f69a715","created":"2024-03-18T15:05:57.538551+0000","modified":"2024-03-18T15:29:17.289046+0000","last_up_change":"2024-03-18T15:14:12.009522+0000","last_in_change":"2024-03-18T15:13:52.555387+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T15:09:56.843349+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T15:15:08.941047+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T15:15:10.044671+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"104","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":104,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T15:15:13.621916+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"104","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":104,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"37b5a5cd-27ae-4e77-9b05-774b817b2ee9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6802","nonce":402333948},{"type":"v1","addr":"172.21.15.19:6803","nonce":402333948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6804","nonce":402333948},{"type":"v1","addr":"172.21.15.19:6805","nonce":402333948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6808","nonce":402333948},{"type":"v1","addr":"172.21.15.19:6809","nonce":402333948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6806","nonce":402333948},{"type":"v1","addr":"172.21.15.19:6807","nonce":402333948}]},"public_addr":"172.21.15.19:6803/402333948","cluster_addr":"172.21.15.19:6805/402333948","heartbeat_back_addr":"172.21.15.19:6809/402333948","heartbeat_front_addr":"172.21.15.19:6807/402333948","state":["exists","up"]},{"osd":1,"uuid":"3a1134e8-ede7-4dbd-9be1-eefd2cbb43bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1951880113},{"type":"v1","addr":"172.21.15.33:6801","nonce":1951880113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1951880113},{"type":"v1","addr":"172.21.15.33:6803","nonce":1951880113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1951880113},{"type":"v1","addr":"172.21.15.33:6807","nonce":1951880113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1951880113},{"type":"v1","addr":"172.21.15.33:6805","nonce":1951880113}]},"public_addr":"172.21.15.33:6801/1951880113","cluster_addr":"172.21.15.33:6803/1951880113","heartbeat_back_addr":"172.21.15.33:6807/1951880113","heartbeat_front_addr":"172.21.15.33:6805/1951880113","state":["exists","up"]},{"osd":2,"uuid":"f383a164-a3a0-49fd-a9e8-9c2178d8cfb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":1004727949},{"type":"v1","addr":"172.21.15.104:6801","nonce":1004727949}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":1004727949},{"type":"v1","addr":"172.21.15.104:6803","nonce":1004727949}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":1004727949},{"type":"v1","addr":"172.21.15.104:6807","nonce":1004727949}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":1004727949},{"type":"v1","addr":"172.21.15.104:6805","nonce":1004727949}]},"public_addr":"172.21.15.104:6801/1004727949","cluster_addr":"172.21.15.104:6803/1004727949","heartbeat_back_addr":"172.21.15.104:6807/1004727949","heartbeat_front_addr":"172.21.15.104:6805/1004727949","state":["exists","up"]},{"osd":3,"uuid":"c409e116-28bf-45dd-a791-539af22abe1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6810","nonce":3571525854},{"type":"v1","addr":"172.21.15.19:6811","nonce":3571525854}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6812","nonce":3571525854},{"type":"v1","addr":"172.21.15.19:6813","nonce":3571525854}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6816","nonce":3571525854},{"type":"v1","addr":"172.21.15.19:6817","nonce":3571525854}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6814","nonce":3571525854},{"type":"v1","addr":"172.21.15.19:6815","nonce":3571525854}]},"public_addr":"172.21.15.19:6811/3571525854","cluster_addr":"172.21.15.19:6813/3571525854","heartbeat_back_addr":"172.21.15.19:6817/3571525854","heartbeat_front_addr":"172.21.15.19:6815/3571525854","state":["exists","up"]},{"osd":4,"uuid":"696eb719-0b3e-43ae-b5f1-ccd73a3b2843","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":624901749},{"type":"v1","addr":"172.21.15.33:6809","nonce":624901749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":624901749},{"type":"v1","addr":"172.21.15.33:6811","nonce":624901749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":624901749},{"type":"v1","addr":"172.21.15.33:6815","nonce":624901749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":624901749},{"type":"v1","addr":"172.21.15.33:6813","nonce":624901749}]},"public_addr":"172.21.15.33:6809/624901749","cluster_addr":"172.21.15.33:6811/624901749","heartbeat_back_addr":"172.21.15.33:6815/624901749","heartbeat_front_addr":"172.21.15.33:6813/624901749","state":["exists","up"]},{"osd":5,"uuid":"21789779-8bb9-42c7-b9ee-437985f11e3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":1258208297},{"type":"v1","addr":"172.21.15.104:6809","nonce":1258208297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":1258208297},{"type":"v1","addr":"172.21.15.104:6811","nonce":1258208297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":1258208297},{"type":"v1","addr":"172.21.15.104:6815","nonce":1258208297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":1258208297},{"type":"v1","addr":"172.21.15.104:6813","nonce":1258208297}]},"public_addr":"172.21.15.104:6809/1258208297","cluster_addr":"172.21.15.104:6811/1258208297","heartbeat_back_addr":"172.21.15.104:6815/1258208297","heartbeat_front_addr":"172.21.15.104:6813/1258208297","state":["exists","up"]},{"osd":6,"uuid":"0e92a2ef-43cd-487c-9203-e219158aef8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6818","nonce":2782482301},{"type":"v1","addr":"172.21.15.19:6819","nonce":2782482301}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6820","nonce":2782482301},{"type":"v1","addr":"172.21.15.19:6821","nonce":2782482301}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6824","nonce":2782482301},{"type":"v1","addr":"172.21.15.19:6825","nonce":2782482301}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6822","nonce":2782482301},{"type":"v1","addr":"172.21.15.19:6823","nonce":2782482301}]},"public_addr":"172.21.15.19:6819/2782482301","cluster_addr":"172.21.15.19:6821/2782482301","heartbeat_back_addr":"172.21.15.19:6825/2782482301","heartbeat_front_addr":"172.21.15.19:6823/2782482301","state":["exists","up"]},{"osd":7,"uuid":"a16e39a1-3685-491d-b376-3e80a2084072","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":4238160988},{"type":"v1","addr":"172.21.15.33:6817","nonce":4238160988}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":4238160988},{"type":"v1","addr":"172.21.15.33:6819","nonce":4238160988}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":4238160988},{"type":"v1","addr":"172.21.15.33:6823","nonce":4238160988}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":4238160988},{"type":"v1","addr":"172.21.15.33:6821","nonce":4238160988}]},"public_addr":"172.21.15.33:6817/4238160988","cluster_addr":"172.21.15.33:6819/4238160988","heartbeat_back_addr":"172.21.15.33:6823/4238160988","heartbeat_front_addr":"172.21.15.33:6821/4238160988","state":["exists","up"]},{"osd":8,"uuid":"d5c3fa92-cd5d-4349-9c7d-d5666bacde16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":1014779813},{"type":"v1","addr":"172.21.15.104:6817","nonce":1014779813}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":1014779813},{"type":"v1","addr":"172.21.15.104:6819","nonce":1014779813}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":1014779813},{"type":"v1","addr":"172.21.15.104:6823","nonce":1014779813}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":1014779813},{"type":"v1","addr":"172.21.15.104:6821","nonce":1014779813}]},"public_addr":"172.21.15.104:6817/1014779813","cluster_addr":"172.21.15.104:6819/1014779813","heartbeat_back_addr":"172.21.15.104:6823/1014779813","heartbeat_front_addr":"172.21.15.104:6821/1014779813","state":["exists","up"]},{"osd":9,"uuid":"eed2e9b2-26d3-4c46-88ec-34b5596fe991","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6826","nonce":2641151009},{"type":"v1","addr":"172.21.15.19:6827","nonce":2641151009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6828","nonce":2641151009},{"type":"v1","addr":"172.21.15.19:6829","nonce":2641151009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6832","nonce":2641151009},{"type":"v1","addr":"172.21.15.19:6833","nonce":2641151009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6830","nonce":2641151009},{"type":"v1","addr":"172.21.15.19:6831","nonce":2641151009}]},"public_addr":"172.21.15.19:6827/2641151009","cluster_addr":"172.21.15.19:6829/2641151009","heartbeat_back_addr":"172.21.15.19:6833/2641151009","heartbeat_front_addr":"172.21.15.19:6831/2641151009","state":["exists","up"]},{"osd":10,"uuid":"f3c35ea5-fbe7-4dc4-93c5-b92d1a714db3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":1346058869},{"type":"v1","addr":"172.21.15.33:6825","nonce":1346058869}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":1346058869},{"type":"v1","addr":"172.21.15.33:6827","nonce":1346058869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":1346058869},{"type":"v1","addr":"172.21.15.33:6831","nonce":1346058869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":1346058869},{"type":"v1","addr":"172.21.15.33:6829","nonce":1346058869}]},"public_addr":"172.21.15.33:6825/1346058869","cluster_addr":"172.21.15.33:6827/1346058869","heartbeat_back_addr":"172.21.15.33:6831/1346058869","heartbeat_front_addr":"172.21.15.33:6829/1346058869","state":["exists","up"]},{"osd":11,"uuid":"c8b51571-f807-4b62-90b6-7529e09e59da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":3759498281},{"type":"v1","addr":"172.21.15.104:6825","nonce":3759498281}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":3759498281},{"type":"v1","addr":"172.21.15.104:6827","nonce":3759498281}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":3759498281},{"type":"v1","addr":"172.21.15.104:6831","nonce":3759498281}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":3759498281},{"type":"v1","addr":"172.21.15.104:6829","nonce":3759498281}]},"public_addr":"172.21.15.104:6825/3759498281","cluster_addr":"172.21.15.104:6827/3759498281","heartbeat_back_addr":"172.21.15.104:6831/3759498281","heartbeat_front_addr":"172.21.15.104:6829/3759498281","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:09:05.267795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:09:30.234068+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:09:53.870232+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:10:26.471296+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:10:53.180341+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:11:17.801361+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:11:51.364816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:12:18.306408+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:12:43.978744+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:13:18.213964+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:13:45.437544+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:14:10.148580+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":3,"to":6}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"3.37","mappings":[{"from":5,"to":8}]},{"pgid":"3.3a","mappings":[{"from":4,"to":7}]},{"pgid":"4.8","mappings":[{"from":5,"to":8}]},{"pgid":"4.f","mappings":[{"from":9,"to":10}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.19:0/1895666842":"2024-03-18T16:23:20.095509+0000","172.21.15.19:6801/3264337803":"2024-03-19T15:06:38.688405+0000","172.21.15.19:6800/3264337803":"2024-03-19T15:06:38.688405+0000","172.21.15.19:0/2793980440":"2024-03-19T15:06:38.688405+0000","172.21.15.19:6801/2776008962":"2024-03-19T15:07:14.150983+0000","172.21.15.19:0/2316969005":"2024-03-19T15:07:14.150983+0000","172.21.15.19:0/84203799":"2024-03-19T15:18:29.726021+0000","172.21.15.19:6800/2017714236":"2024-03-19T15:18:29.726021+0000","172.21.15.19:0/2737223213":"2024-03-19T15:07:14.150983+0000","172.21.15.19:0/3860783274":"2024-03-19T15:07:14.150983+0000","172.21.15.19:6800/2776008962":"2024-03-19T15:07:14.150983+0000","172.21.15.19:0/1024556841":"2024-03-19T15:18:29.726021+0000","172.21.15.19:0/3166705295":"2024-03-19T15:06:38.688405+0000","172.21.15.19:6801/2017714236":"2024-03-19T15:18:29.726021+0000","172.21.15.19:0/176699025":"2024-03-19T15:06:38.688405+0000","172.21.15.19:0/2730531676":"2024-03-19T15:18:29.726021+0000","172.21.15.19:0/2353244552":"2024-03-19T15:18:29.726021+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]},{"pool":4,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T15:29:22.930 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.929+0000 7f249a400640 1 -- 172.21.15.19:0/1248040415 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f24780a29f0 msgr2=0x7f24780a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:29:22.930 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.929+0000 7f249a400640 1 --2- 172.21.15.19:0/1248040415 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f24780a29f0 0x7f24780a4eb0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f24940fda20 tx=0x7f247c009260 comp rx=0 tx=0).stop 2024-03-18T15:29:22.930 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.929+0000 7f249a400640 1 -- 172.21.15.19:0/1248040415 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2494069f40 msgr2=0x7f249419a1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:29:22.930 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.929+0000 7f249a400640 1 --2- 172.21.15.19:0/1248040415 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2494069f40 0x7f249419a1f0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f24840057c0 tx=0x7f24840045d0 comp rx=0 tx=0).stop 2024-03-18T15:29:22.930 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.929+0000 7f249a400640 1 -- 172.21.15.19:0/1248040415 shutdown_connections 2024-03-18T15:29:22.930 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.929+0000 7f249a400640 1 --2- 172.21.15.19:0/1248040415 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f24780a29f0 0x7f24780a4eb0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:22.930 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.929+0000 7f249a400640 1 --2- 172.21.15.19:0/1248040415 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2494107840 0x7f249419eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:22.930 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.929+0000 7f249a400640 1 --2- 172.21.15.19:0/1248040415 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f249406a900 0x7f249419a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:22.930 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.929+0000 7f249a400640 1 --2- 172.21.15.19:0/1248040415 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2494069f40 0x7f249419a1f0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:22.930 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.929+0000 7f249a400640 1 -- 172.21.15.19:0/1248040415 >> 172.21.15.19:0/1248040415 conn(0x7f2494076a60 msgr2=0x7f249410cef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:29:22.931 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.929+0000 7f249a400640 1 -- 172.21.15.19:0/1248040415 shutdown_connections 2024-03-18T15:29:22.931 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:22.930+0000 7f249a400640 1 -- 172.21.15.19:0/1248040415 wait complete. 2024-03-18T15:29:22.949 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-18T15:29:22.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:29:22 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/1942199253' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:29:22.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:29:22 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/162178797' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:29:23.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:29:22 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/1942199253' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:29:23.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:29:22 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/162178797' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:29:23.182 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:23.179+0000 7fa7c1e00640 1 -- 172.21.15.19:0/1506966628 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa7bc105c10 msgr2=0x7fa7bc1060c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:29:23.182 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:23.179+0000 7fa7c1e00640 1 --2- 172.21.15.19:0/1506966628 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa7bc105c10 0x7fa7bc1060c0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fa7b4009900 tx=0x7fa7b402bcc0 comp rx=0 tx=0).stop 2024-03-18T15:29:23.182 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:23.179+0000 7fa7c1e00640 1 -- 172.21.15.19:0/1506966628 shutdown_connections 2024-03-18T15:29:23.182 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:23.179+0000 7fa7c1e00640 1 --2- 172.21.15.19:0/1506966628 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa7bc105c10 0x7fa7bc1060c0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:23.182 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:23.179+0000 7fa7c1e00640 1 --2- 172.21.15.19:0/1506966628 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa7bc101130 0x7fa7bc1055f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:23.182 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:23.179+0000 7fa7c1e00640 1 --2- 172.21.15.19:0/1506966628 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa7bc100810 0x7fa7bc100bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:23.182 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:23.179+0000 7fa7c1e00640 1 -- 172.21.15.19:0/1506966628 >> 172.21.15.19:0/1506966628 conn(0x7fa7bc076a60 msgr2=0x7fa7bc076e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:29:23.183 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:23.179+0000 7fa7c1e00640 1 -- 172.21.15.19:0/1506966628 shutdown_connections 2024-03-18T15:29:23.183 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:23.179+0000 7fa7c1e00640 1 -- 172.21.15.19:0/1506966628 wait complete. 2024-03-18T15:29:23.183 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:23.180+0000 7fa7c1e00640 1 Processor -- start 2024-03-18T15:29:23.183 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:23.180+0000 7fa7c1e00640 1 -- start start 2024-03-18T15:29:23.183 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:23.181+0000 7fa7c1e00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa7bc100810 0x7fa7bc19e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:29:23.183 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:23.181+0000 7fa7c1e00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa7bc101130 0x7fa7bc19eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:29:23.183 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:23.181+0000 7fa7c1e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa7bc105c10 0x7fa7bc1a2ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:29:23.184 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:23.181+0000 7fa7c1e00640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fa7bc118140 con 0x7fa7bc100810 2024-03-18T15:29:23.184 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:23.181+0000 7fa7c1e00640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa7bc117fc0 con 0x7fa7bc105c10 2024-03-18T15:29:23.184 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:23.181+0000 7fa7c1e00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fa7bc1182c0 con 0x7fa7bc101130 2024-03-18T15:29:23.184 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:23.181+0000 7fa7ba000640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa7bc101130 0x7fa7bc19eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:29:23.184 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:23.181+0000 7fa7baa00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa7bc100810 0x7fa7bc19e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:29:23.184 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:23.181+0000 7fa7ba000640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa7bc101130 0x7fa7bc19eba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.19:33142/0 (socket says 172.21.15.19:33142) 2024-03-18T15:29:23.184 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:23.181+0000 7fa7ba000640 1 -- 172.21.15.19:0/1220219657 learned_addr learned my addr 172.21.15.19:0/1220219657 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:29:23.184 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:23.181+0000 7fa7bb400640 1 --2- 172.21.15.19:0/1220219657 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa7bc105c10 0x7fa7bc1a2ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:29:23.184 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:23.181+0000 7fa7ba000640 1 -- 172.21.15.19:0/1220219657 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa7bc105c10 msgr2=0x7fa7bc1a2ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:29:23.185 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:23.181+0000 7fa7ba000640 1 --2- 172.21.15.19:0/1220219657 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa7bc105c10 0x7fa7bc1a2ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:23.185 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:23.181+0000 7fa7ba000640 1 -- 172.21.15.19:0/1220219657 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa7bc100810 msgr2=0x7fa7bc19e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:29:23.185 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:23.181+0000 7fa7ba000640 1 --2- 172.21.15.19:0/1220219657 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa7bc100810 0x7fa7bc19e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:23.185 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:23.182+0000 7fa7ba000640 1 -- 172.21.15.19:0/1220219657 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7bc1a3600 con 0x7fa7bc101130 2024-03-18T15:29:23.185 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:23.182+0000 7fa7bb400640 1 --2- 172.21.15.19:0/1220219657 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa7bc105c10 0x7fa7bc1a2ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:29:23.187 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:23.182+0000 7fa7ba000640 1 --2- 172.21.15.19:0/1220219657 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa7bc101130 0x7fa7bc19eba0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fa7a400ebc0 tx=0x7fa7a400c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:29:23.187 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:23.182+0000 7fa7c0a00640 1 -- 172.21.15.19:0/1220219657 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7a4010630 con 0x7fa7bc101130 2024-03-18T15:29:23.187 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:23.182+0000 7fa7c1e00640 1 -- 172.21.15.19:0/1220219657 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7bc1a38f0 con 0x7fa7bc101130 2024-03-18T15:29:23.187 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:23.182+0000 7fa7c0a00640 1 -- 172.21.15.19:0/1220219657 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa7a4010cb0 con 0x7fa7bc101130 2024-03-18T15:29:23.187 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:23.182+0000 7fa7c0a00640 1 -- 172.21.15.19:0/1220219657 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7a4018bd0 con 0x7fa7bc101130 2024-03-18T15:29:23.188 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:23.182+0000 7fa7c1e00640 1 -- 172.21.15.19:0/1220219657 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7bc1ab180 con 0x7fa7bc101130 2024-03-18T15:29:23.188 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:23.183+0000 7fa7c1e00640 1 -- 172.21.15.19:0/1220219657 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7bc06a5d0 con 0x7fa7bc101130 2024-03-18T15:29:23.188 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:23.186+0000 7fa7c0a00640 1 -- 172.21.15.19:0/1220219657 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 27) ==== 137421+0+0 (secure 0 0 0) 0x7fa7a40107d0 con 0x7fa7bc101130 2024-03-18T15:29:23.189 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:23.187+0000 7fa7c0a00640 1 --2- 172.21.15.19:0/1220219657 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7fa79c0a29f0 0x7fa79c0a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:29:23.189 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:23.188+0000 7fa7c0a00640 1 -- 172.21.15.19:0/1220219657 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 9597+0+0 (secure 0 0 0) 0x7fa7a40cf9f0 con 0x7fa7bc101130 2024-03-18T15:29:23.189 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:23.188+0000 7fa7baa00640 1 --2- 172.21.15.19:0/1220219657 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7fa79c0a29f0 0x7fa79c0a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:29:23.189 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:23.189+0000 7fa7baa00640 1 --2- 172.21.15.19:0/1220219657 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7fa79c0a29f0 0x7fa79c0a4eb0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa7ac009bd0 tx=0x7fa7ac005f90 comp rx=0 tx=0).ready entity=mgr.25466 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:29:23.193 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:23.193+0000 7fa7c0a00640 1 -- 172.21.15.19:0/1220219657 <== mon.1 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa7a409bc70 con 0x7fa7bc101130 2024-03-18T15:29:23.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:29:22 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/1942199253' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:29:23.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:29:22 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/162178797' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:29:23.439 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:23.438+0000 7fa7c1e00640 1 -- 172.21.15.19:0/1220219657 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fa7bc19f960 con 0x7fa7bc101130 2024-03-18T15:29:23.448 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:23.448+0000 7fa7c0a00640 1 -- 172.21.15.19:0/1220219657 <== mon.1 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v32) ==== 161+0+0 (secure 0 0 0) 0x7fa7a409fd10 con 0x7fa7bc101130 2024-03-18T15:29:23.449 INFO:teuthology.orchestra.run.smithi019.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-18T15:29:23.454 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:23.452+0000 7fa7c1e00640 1 -- 172.21.15.19:0/1220219657 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7fa79c0a29f0 msgr2=0x7fa79c0a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:29:23.455 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:23.452+0000 7fa7c1e00640 1 --2- 172.21.15.19:0/1220219657 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7fa79c0a29f0 0x7fa79c0a4eb0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa7ac009bd0 tx=0x7fa7ac005f90 comp rx=0 tx=0).stop 2024-03-18T15:29:23.455 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:23.452+0000 7fa7c1e00640 1 -- 172.21.15.19:0/1220219657 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa7bc101130 msgr2=0x7fa7bc19eba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:29:23.455 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:23.452+0000 7fa7c1e00640 1 --2- 172.21.15.19:0/1220219657 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa7bc101130 0x7fa7bc19eba0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fa7a400ebc0 tx=0x7fa7a400c6a0 comp rx=0 tx=0).stop 2024-03-18T15:29:23.455 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:23.452+0000 7fa7c1e00640 1 -- 172.21.15.19:0/1220219657 shutdown_connections 2024-03-18T15:29:23.456 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:23.452+0000 7fa7c1e00640 1 --2- 172.21.15.19:0/1220219657 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7fa79c0a29f0 0x7fa79c0a4eb0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:23.456 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:23.452+0000 7fa7c1e00640 1 --2- 172.21.15.19:0/1220219657 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa7bc105c10 0x7fa7bc1a2ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:23.456 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:23.452+0000 7fa7c1e00640 1 --2- 172.21.15.19:0/1220219657 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa7bc101130 0x7fa7bc19eba0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:23.457 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:23.452+0000 7fa7c1e00640 1 --2- 172.21.15.19:0/1220219657 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa7bc100810 0x7fa7bc19e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:23.457 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:23.452+0000 7fa7c1e00640 1 -- 172.21.15.19:0/1220219657 >> 172.21.15.19:0/1220219657 conn(0x7fa7bc076a60 msgr2=0x7fa7bc1035c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:29:23.457 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:23.453+0000 7fa7c1e00640 1 -- 172.21.15.19:0/1220219657 shutdown_connections 2024-03-18T15:29:23.457 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:23.453+0000 7fa7c1e00640 1 -- 172.21.15.19:0/1220219657 wait complete. 2024-03-18T15:29:23.472 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-18T15:29:23.702 INFO:journalctl@ceph.mds.a.smithi019.stdout:Mar 18 15:29:23 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-a[73284]: ignoring --setuser ceph since I am not root 2024-03-18T15:29:23.702 INFO:journalctl@ceph.mds.a.smithi019.stdout:Mar 18 15:29:23 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-a[73284]: ignoring --setgroup ceph since I am not root 2024-03-18T15:29:23.702 INFO:journalctl@ceph.mds.a.smithi019.stdout:Mar 18 15:29:23 smithi019 ceph-mds[73307]: -- 172.21.15.19:0/1869663561 <== mon.2 v2:172.21.15.33:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559abeab1a40 con 0x559abea84800 2024-03-18T15:29:23.702 INFO:journalctl@ceph.mds.a.smithi019.stdout:Mar 18 15:29:23 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-a[73284]: starting mds.a at 2024-03-18T15:29:23.703 INFO:journalctl@ceph.mds.d.smithi019.stdout:Mar 18 15:29:23 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-d[72277]: ignoring --setuser ceph since I am not root 2024-03-18T15:29:23.703 INFO:journalctl@ceph.mds.d.smithi019.stdout:Mar 18 15:29:23 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-d[72277]: ignoring --setgroup ceph since I am not root 2024-03-18T15:29:23.703 INFO:journalctl@ceph.mds.d.smithi019.stdout:Mar 18 15:29:23 smithi019 ceph-mds[72297]: -- 172.21.15.19:0/1943962886 <== mon.1 v2:172.21.15.104:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5612027c5a40 con 0x561202798c00 2024-03-18T15:29:23.703 INFO:journalctl@ceph.mds.d.smithi019.stdout:Mar 18 15:29:23 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-d[72277]: starting mds.d at 2024-03-18T15:29:23.703 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:29:23 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mon-a[31518]: 2024-03-18T15:29:23.440+0000 7ffac3600700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-18T15:29:23.709 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:23.709+0000 7f3ee0c00640 1 -- 172.21.15.19:0/1101727924 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3edc102a70 msgr2=0x7f3edc102e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:29:23.709 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:23.709+0000 7f3ee0c00640 1 --2- 172.21.15.19:0/1101727924 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3edc102a70 0x7f3edc102e50 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f3ecc009900 tx=0x7f3ecc02bcd0 comp rx=0 tx=0).stop 2024-03-18T15:29:23.710 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:23.709+0000 7f3ee0c00640 1 -- 172.21.15.19:0/1101727924 shutdown_connections 2024-03-18T15:29:23.710 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:23.709+0000 7f3ee0c00640 1 --2- 172.21.15.19:0/1101727924 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3edc10bdb0 0x7f3edc10e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:23.710 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:23.709+0000 7f3ee0c00640 1 --2- 172.21.15.19:0/1101727924 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3edc103390 0x7f3edc10b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:23.710 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:23.709+0000 7f3ee0c00640 1 --2- 172.21.15.19:0/1101727924 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3edc102a70 0x7f3edc102e50 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:23.710 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:23.709+0000 7f3ee0c00640 1 -- 172.21.15.19:0/1101727924 >> 172.21.15.19:0/1101727924 conn(0x7f3edc077840 msgr2=0x7f3edc077c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:29:23.710 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:23.709+0000 7f3ee0c00640 1 -- 172.21.15.19:0/1101727924 shutdown_connections 2024-03-18T15:29:23.710 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:23.709+0000 7f3ee0c00640 1 -- 172.21.15.19:0/1101727924 wait complete. 2024-03-18T15:29:23.712 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:23.710+0000 7f3ee0c00640 1 Processor -- start 2024-03-18T15:29:23.712 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:23.710+0000 7f3ee0c00640 1 -- start start 2024-03-18T15:29:23.712 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:23.711+0000 7f3ee0c00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3edc102a70 0x7f3edc1006c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:29:23.712 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:23.711+0000 7f3ee0c00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3edc103390 0x7f3edc100c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:29:23.712 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:23.711+0000 7f3ee0c00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3edc10bdb0 0x7f3edc101140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:29:23.713 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:23.711+0000 7f3ee0c00640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f3edc111bd0 con 0x7f3edc103390 2024-03-18T15:29:23.713 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:23.711+0000 7f3ee0c00640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f3edc111a50 con 0x7f3edc10bdb0 2024-03-18T15:29:23.713 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:23.711+0000 7f3ee0c00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f3edc111d50 con 0x7f3edc102a70 2024-03-18T15:29:23.713 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:23.711+0000 7f3ed9400640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3edc103390 0x7f3edc100c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:29:23.713 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:23.711+0000 7f3ed9400640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3edc103390 0x7f3edc100c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:60484/0 (socket says 172.21.15.19:60484) 2024-03-18T15:29:23.713 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:23.711+0000 7f3ed9400640 1 -- 172.21.15.19:0/3642234977 learned_addr learned my addr 172.21.15.19:0/3642234977 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:29:23.713 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:23.711+0000 7f3ed9e00640 1 --2- 172.21.15.19:0/3642234977 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3edc102a70 0x7f3edc1006c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:29:23.713 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:23.711+0000 7f3eda800640 1 --2- 172.21.15.19:0/3642234977 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3edc10bdb0 0x7f3edc101140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:29:23.713 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:23.712+0000 7f3ed9e00640 1 -- 172.21.15.19:0/3642234977 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3edc10bdb0 msgr2=0x7f3edc101140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:29:23.713 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:23.712+0000 7f3ed9e00640 1 --2- 172.21.15.19:0/3642234977 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3edc10bdb0 0x7f3edc101140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:23.713 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:23.712+0000 7f3ed9e00640 1 -- 172.21.15.19:0/3642234977 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3edc103390 msgr2=0x7f3edc100c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:29:23.713 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:23.712+0000 7f3ed9e00640 1 --2- 172.21.15.19:0/3642234977 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3edc103390 0x7f3edc100c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:23.713 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:23.712+0000 7f3ed9e00640 1 -- 172.21.15.19:0/3642234977 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3edc1a6550 con 0x7f3edc102a70 2024-03-18T15:29:23.713 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:23.712+0000 7f3ed9e00640 1 --2- 172.21.15.19:0/3642234977 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3edc102a70 0x7f3edc1006c0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f3ecc02c210 tx=0x7f3ecc004360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:29:23.714 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:23.712+0000 7f3ed2a00640 1 -- 172.21.15.19:0/3642234977 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ecc042020 con 0x7f3edc102a70 2024-03-18T15:29:23.714 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:23.712+0000 7f3ed2a00640 1 -- 172.21.15.19:0/3642234977 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ecc0054e0 con 0x7f3edc102a70 2024-03-18T15:29:23.714 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:23.712+0000 7f3ed2a00640 1 -- 172.21.15.19:0/3642234977 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ecc003b00 con 0x7f3edc102a70 2024-03-18T15:29:23.714 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:23.712+0000 7f3ee0c00640 1 -- 172.21.15.19:0/3642234977 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3edc1a67e0 con 0x7f3edc102a70 2024-03-18T15:29:23.714 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:23.712+0000 7f3ee0c00640 1 -- 172.21.15.19:0/3642234977 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3edc1a6cb0 con 0x7f3edc102a70 2024-03-18T15:29:23.714 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:23.714+0000 7f3ee0c00640 1 -- 172.21.15.19:0/3642234977 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3edc067d40 con 0x7f3edc102a70 2024-03-18T15:29:23.715 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:23.715+0000 7f3ed2a00640 1 -- 172.21.15.19:0/3642234977 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 27) ==== 137421+0+0 (secure 0 0 0) 0x7f3ecc045020 con 0x7f3edc102a70 2024-03-18T15:29:23.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:23.716+0000 7f3ed2a00640 1 --2- 172.21.15.19:0/3642234977 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f3ec00a2a20 0x7f3ec00a4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:29:23.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:23.716+0000 7f3ed2a00640 1 -- 172.21.15.19:0/3642234977 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9939+0+0 (secure 0 0 0) 0x7f3ecc0f0d80 con 0x7f3edc102a70 2024-03-18T15:29:23.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:23.717+0000 7f3ed9400640 1 --2- 172.21.15.19:0/3642234977 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f3ec00a2a20 0x7f3ec00a4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:29:23.717 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:23.717+0000 7f3ed9400640 1 --2- 172.21.15.19:0/3642234977 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f3ec00a2a20 0x7f3ec00a4ee0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f3edc102100 tx=0x7f3ec8005f90 comp rx=0 tx=0).ready entity=mgr.25466 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:29:23.720 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:23.720+0000 7f3ed2a00640 1 -- 172.21.15.19:0/3642234977 <== mon.1 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3ecc04c020 con 0x7f3edc102a70 2024-03-18T15:29:23.757 INFO:journalctl@ceph.mds.k.smithi033.stdout:Mar 18 15:29:23 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-k[66659]: ignoring --setuser ceph since I am not root 2024-03-18T15:29:23.757 INFO:journalctl@ceph.mds.k.smithi033.stdout:Mar 18 15:29:23 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-k[66659]: ignoring --setgroup ceph since I am not root 2024-03-18T15:29:23.757 INFO:journalctl@ceph.mds.k.smithi033.stdout:Mar 18 15:29:23 smithi033 ceph-mds[66663]: -- 172.21.15.33:0/2003279206 <== mon.2 v2:172.21.15.33:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f6ecf1ba40 con 0x55f6eceef000 2024-03-18T15:29:23.757 INFO:journalctl@ceph.mds.k.smithi033.stdout:Mar 18 15:29:23 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-k[66659]: starting mds.k at 2024-03-18T15:29:23.757 INFO:journalctl@ceph.mds.e.smithi033.stdout:Mar 18 15:29:23 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-e[66984]: ignoring --setuser ceph since I am not root 2024-03-18T15:29:23.758 INFO:journalctl@ceph.mds.e.smithi033.stdout:Mar 18 15:29:23 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-e[66984]: ignoring --setgroup ceph since I am not root 2024-03-18T15:29:23.758 INFO:journalctl@ceph.mds.e.smithi033.stdout:Mar 18 15:29:23 smithi033 ceph-mds[66988]: -- 172.21.15.33:0/3076701990 <== mon.2 v2:172.21.15.33:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e1ee6eba40 con 0x55e1ee6be800 2024-03-18T15:29:23.758 INFO:journalctl@ceph.mds.e.smithi033.stdout:Mar 18 15:29:23 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-e[66984]: starting mds.e at 2024-03-18T15:29:23.759 INFO:journalctl@ceph.mds.b.smithi033.stdout:Mar 18 15:29:23 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-b[67599]: ignoring --setuser ceph since I am not root 2024-03-18T15:29:23.759 INFO:journalctl@ceph.mds.b.smithi033.stdout:Mar 18 15:29:23 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-b[67599]: ignoring --setgroup ceph since I am not root 2024-03-18T15:29:23.759 INFO:journalctl@ceph.mds.b.smithi033.stdout:Mar 18 15:29:23 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-b[67599]: starting mds.b at 2024-03-18T15:29:23.760 INFO:journalctl@ceph.mds.h.smithi033.stdout:Mar 18 15:29:23 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-h[67294]: ignoring --setuser ceph since I am not root 2024-03-18T15:29:23.760 INFO:journalctl@ceph.mds.h.smithi033.stdout:Mar 18 15:29:23 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-h[67294]: ignoring --setgroup ceph since I am not root 2024-03-18T15:29:23.760 INFO:journalctl@ceph.mds.h.smithi033.stdout:Mar 18 15:29:23 smithi033 ceph-mds[67298]: -- 172.21.15.33:0/2317173984 <== mon.2 v2:172.21.15.33:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5629ab8c7a40 con 0x5629ab89a800 2024-03-18T15:29:23.760 INFO:journalctl@ceph.mds.h.smithi033.stdout:Mar 18 15:29:23 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-h[67294]: starting mds.h at 2024-03-18T15:29:23.760 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:29:23 smithi033 ceph-mon[36648]: pgmap v340: 129 pgs: 129 active+clean; 1.8 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-18T15:29:23.760 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:29:23 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/1248040415' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:29:23.760 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:29:23 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/1220219657' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-18T15:29:23.760 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:29:23 smithi033 ceph-mon[36648]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-18T15:29:23.760 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:29:23 smithi033 ceph-mon[36648]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-18T15:29:23.761 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:29:23 smithi033 ceph-mon[36648]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-18T15:29:23.761 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:29:23 smithi033 ceph-mon[36648]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-18T15:29:23.761 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:29:23 smithi033 ceph-mon[36648]: osdmap e106: 12 total, 12 up, 12 in 2024-03-18T15:29:23.761 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:29:23 smithi033 ceph-mon[36648]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-03-18T15:29:23.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:29:23 smithi104 ceph-mon[36070]: pgmap v340: 129 pgs: 129 active+clean; 1.8 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-18T15:29:23.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:29:23 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/1248040415' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:29:23.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:29:23 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/1220219657' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-18T15:29:23.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:29:23 smithi104 ceph-mon[36070]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-18T15:29:23.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:29:23 smithi104 ceph-mon[36070]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-18T15:29:23.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:29:23 smithi104 ceph-mon[36070]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-18T15:29:23.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:29:23 smithi104 ceph-mon[36070]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-18T15:29:23.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:29:23 smithi104 ceph-mon[36070]: osdmap e106: 12 total, 12 up, 12 in 2024-03-18T15:29:23.945 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:29:23 smithi104 ceph-mon[36070]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-03-18T15:29:23.946 INFO:journalctl@ceph.mds.c.smithi104.stdout:Mar 18 15:29:23 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-c[66920]: ignoring --setuser ceph since I am not root 2024-03-18T15:29:23.946 INFO:journalctl@ceph.mds.c.smithi104.stdout:Mar 18 15:29:23 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-c[66920]: ignoring --setgroup ceph since I am not root 2024-03-18T15:29:23.946 INFO:journalctl@ceph.mds.c.smithi104.stdout:Mar 18 15:29:23 smithi104 ceph-mds[66924]: -- 172.21.15.104:0/3582022856 <== mon.1 v2:172.21.15.104:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c0ddd93a40 con 0x55c0ddd67000 2024-03-18T15:29:23.946 INFO:journalctl@ceph.mds.c.smithi104.stdout:Mar 18 15:29:23 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-c[66920]: starting mds.c at 2024-03-18T15:29:23.946 INFO:journalctl@ceph.mds.f.smithi104.stdout:Mar 18 15:29:23 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-f[66607]: ignoring --setuser ceph since I am not root 2024-03-18T15:29:23.946 INFO:journalctl@ceph.mds.f.smithi104.stdout:Mar 18 15:29:23 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-f[66607]: ignoring --setgroup ceph since I am not root 2024-03-18T15:29:23.946 INFO:journalctl@ceph.mds.f.smithi104.stdout:Mar 18 15:29:23 smithi104 ceph-mds[66611]: -- 172.21.15.104:0/2472054896 <== mon.1 v2:172.21.15.104:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5615d9281a40 con 0x5615d9254c00 2024-03-18T15:29:23.947 INFO:journalctl@ceph.mds.f.smithi104.stdout:Mar 18 15:29:23 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-f[66607]: starting mds.f at 2024-03-18T15:29:23.947 INFO:journalctl@ceph.mds.l.smithi104.stdout:Mar 18 15:29:23 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-l[66300]: ignoring --setuser ceph since I am not root 2024-03-18T15:29:23.947 INFO:journalctl@ceph.mds.l.smithi104.stdout:Mar 18 15:29:23 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-l[66300]: ignoring --setgroup ceph since I am not root 2024-03-18T15:29:23.947 INFO:journalctl@ceph.mds.l.smithi104.stdout:Mar 18 15:29:23 smithi104 ceph-mds[66304]: -- 172.21.15.104:0/2535930375 <== mon.1 v2:172.21.15.104:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56042ad37a40 con 0x56042ad0b000 2024-03-18T15:29:23.947 INFO:journalctl@ceph.mds.l.smithi104.stdout:Mar 18 15:29:23 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-l[66300]: starting mds.l at 2024-03-18T15:29:23.947 INFO:journalctl@ceph.mds.i.smithi104.stdout:Mar 18 15:29:23 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-i[67232]: ignoring --setuser ceph since I am not root 2024-03-18T15:29:23.947 INFO:journalctl@ceph.mds.i.smithi104.stdout:Mar 18 15:29:23 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-i[67232]: ignoring --setgroup ceph since I am not root 2024-03-18T15:29:23.948 INFO:journalctl@ceph.mds.i.smithi104.stdout:Mar 18 15:29:23 smithi104 ceph-mds[67236]: -- 172.21.15.104:0/1658477268 <== mon.1 v2:172.21.15.104:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564586445a40 con 0x564586418c00 2024-03-18T15:29:23.948 INFO:journalctl@ceph.mds.i.smithi104.stdout:Mar 18 15:29:23 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-i[67232]: starting mds.i at 2024-03-18T15:29:23.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:29:23 smithi019 ceph-mon[31522]: pgmap v340: 129 pgs: 129 active+clean; 1.8 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-18T15:29:23.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:29:23 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/1248040415' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:29:23.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:29:23 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/1220219657' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-18T15:29:23.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:29:23 smithi019 ceph-mon[31522]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-18T15:29:23.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:29:23 smithi019 ceph-mon[31522]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-18T15:29:23.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:29:23 smithi019 ceph-mon[31522]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-18T15:29:23.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:29:23 smithi019 ceph-mon[31522]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-18T15:29:23.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:29:23 smithi019 ceph-mon[31522]: osdmap e106: 12 total, 12 up, 12 in 2024-03-18T15:29:23.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:29:23 smithi019 ceph-mon[31522]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-03-18T15:29:23.968 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:23.968+0000 7f3ee0c00640 1 -- 172.21.15.19:0/3642234977 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f3edc106f30 con 0x7f3edc102a70 2024-03-18T15:29:24.450 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:24.450+0000 7f3ed2a00640 1 -- 172.21.15.19:0/3642234977 <== mon.1 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v33) ==== 104+0+0 (secure 0 0 0) 0x7f3ecc0bd600 con 0x7f3edc102a70 2024-03-18T15:29:24.455 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:24.455+0000 7f3ee0c00640 1 -- 172.21.15.19:0/3642234977 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f3ec00a2a20 msgr2=0x7f3ec00a4ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:29:24.455 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:24.455+0000 7f3ee0c00640 1 --2- 172.21.15.19:0/3642234977 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f3ec00a2a20 0x7f3ec00a4ee0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f3edc102100 tx=0x7f3ec8005f90 comp rx=0 tx=0).stop 2024-03-18T15:29:24.455 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:24.455+0000 7f3ee0c00640 1 -- 172.21.15.19:0/3642234977 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3edc102a70 msgr2=0x7f3edc1006c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:29:24.455 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:24.455+0000 7f3ee0c00640 1 --2- 172.21.15.19:0/3642234977 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3edc102a70 0x7f3edc1006c0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f3ecc02c210 tx=0x7f3ecc004360 comp rx=0 tx=0).stop 2024-03-18T15:29:24.455 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:24.455+0000 7f3ee0c00640 1 -- 172.21.15.19:0/3642234977 shutdown_connections 2024-03-18T15:29:24.456 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:24.455+0000 7f3ee0c00640 1 --2- 172.21.15.19:0/3642234977 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f3ec00a2a20 0x7f3ec00a4ee0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:24.456 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:24.455+0000 7f3ee0c00640 1 --2- 172.21.15.19:0/3642234977 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3edc10bdb0 0x7f3edc101140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:24.456 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:24.455+0000 7f3ee0c00640 1 --2- 172.21.15.19:0/3642234977 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3edc103390 0x7f3edc100c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:24.456 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:24.455+0000 7f3ee0c00640 1 --2- 172.21.15.19:0/3642234977 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3edc102a70 0x7f3edc1006c0 secure :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f3ecc02c210 tx=0x7f3ecc004360 comp rx=0 tx=0).stop 2024-03-18T15:29:24.456 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:24.455+0000 7f3ee0c00640 1 -- 172.21.15.19:0/3642234977 >> 172.21.15.19:0/3642234977 conn(0x7f3edc077840 msgr2=0x7f3edc0fd7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:29:24.456 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:24.456+0000 7f3ee0c00640 1 -- 172.21.15.19:0/3642234977 shutdown_connections 2024-03-18T15:29:24.456 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:24.457+0000 7f3ee0c00640 1 -- 172.21.15.19:0/3642234977 wait complete. 2024-03-18T15:29:24.479 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-18T15:29:24.695 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:24.694+0000 7fd783e00640 1 -- 172.21.15.19:0/3467445179 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd784073b90 msgr2=0x7fd784073f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:29:24.695 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:24.694+0000 7fd783e00640 1 --2- 172.21.15.19:0/3467445179 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd784073b90 0x7fd784073f70 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fd774009860 tx=0x7fd77402bc00 comp rx=0 tx=0).stop 2024-03-18T15:29:24.695 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:24.694+0000 7fd783e00640 1 -- 172.21.15.19:0/3467445179 shutdown_connections 2024-03-18T15:29:24.695 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:24.694+0000 7fd783e00640 1 --2- 172.21.15.19:0/3467445179 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd78410df10 0x7fd784110300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:24.695 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:24.694+0000 7fd783e00640 1 --2- 172.21.15.19:0/3467445179 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd7840744b0 0x7fd78410d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:24.695 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:24.694+0000 7fd783e00640 1 --2- 172.21.15.19:0/3467445179 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd784073b90 0x7fd784073f70 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:24.695 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:24.694+0000 7fd783e00640 1 -- 172.21.15.19:0/3467445179 >> 172.21.15.19:0/3467445179 conn(0x7fd7840fd180 msgr2=0x7fd7840ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:29:24.695 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:24.694+0000 7fd783e00640 1 -- 172.21.15.19:0/3467445179 shutdown_connections 2024-03-18T15:29:24.695 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:24.695+0000 7fd783e00640 1 -- 172.21.15.19:0/3467445179 wait complete. 2024-03-18T15:29:24.697 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:24.695+0000 7fd783e00640 1 Processor -- start 2024-03-18T15:29:24.698 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:24.696+0000 7fd783e00640 1 -- start start 2024-03-18T15:29:24.698 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:24.696+0000 7fd783e00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd784073b90 0x7fd78419a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:29:24.698 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:24.696+0000 7fd783e00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd7840744b0 0x7fd78419a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:29:24.698 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:24.696+0000 7fd783e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd78410df10 0x7fd78419ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:29:24.698 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:24.696+0000 7fd783e00640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fd784113c30 con 0x7fd7840744b0 2024-03-18T15:29:24.698 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:24.696+0000 7fd783e00640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd784113ab0 con 0x7fd78410df10 2024-03-18T15:29:24.698 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:24.696+0000 7fd783e00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fd784113db0 con 0x7fd784073b90 2024-03-18T15:29:24.698 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:24.696+0000 7fd782000640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd7840744b0 0x7fd78419a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:29:24.698 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:24.696+0000 7fd783400640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd78410df10 0x7fd78419ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:29:24.698 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:24.696+0000 7fd782000640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd7840744b0 0x7fd78419a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:60500/0 (socket says 172.21.15.19:60500) 2024-03-18T15:29:24.698 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:24.696+0000 7fd782000640 1 -- 172.21.15.19:0/869932086 learned_addr learned my addr 172.21.15.19:0/869932086 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:29:24.698 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:24.696+0000 7fd782a00640 1 --2- 172.21.15.19:0/869932086 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd784073b90 0x7fd78419a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:29:24.699 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:24.697+0000 7fd782000640 1 -- 172.21.15.19:0/869932086 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd784073b90 msgr2=0x7fd78419a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:29:24.699 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:24.697+0000 7fd782000640 1 --2- 172.21.15.19:0/869932086 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd784073b90 0x7fd78419a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:24.699 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:24.697+0000 7fd782000640 1 -- 172.21.15.19:0/869932086 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd78410df10 msgr2=0x7fd78419ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:29:24.699 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:24.697+0000 7fd782000640 1 --2- 172.21.15.19:0/869932086 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd78410df10 0x7fd78419ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:24.699 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:24.697+0000 7fd782000640 1 -- 172.21.15.19:0/869932086 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd78419f1c0 con 0x7fd7840744b0 2024-03-18T15:29:24.699 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:24.697+0000 7fd782a00640 1 --2- 172.21.15.19:0/869932086 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd784073b90 0x7fd78419a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:29:24.699 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:24.697+0000 7fd782000640 1 --2- 172.21.15.19:0/869932086 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd7840744b0 0x7fd78419a710 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fd77000e9c0 tx=0x7fd77000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:29:24.699 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:24.697+0000 7fd77b400640 1 -- 172.21.15.19:0/869932086 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd77000ccc0 con 0x7fd7840744b0 2024-03-18T15:29:24.699 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:24.698+0000 7fd783e00640 1 -- 172.21.15.19:0/869932086 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd78419f4b0 con 0x7fd7840744b0 2024-03-18T15:29:24.699 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:24.698+0000 7fd77b400640 1 -- 172.21.15.19:0/869932086 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd770005500 con 0x7fd7840744b0 2024-03-18T15:29:24.699 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:24.698+0000 7fd77b400640 1 -- 172.21.15.19:0/869932086 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd770010430 con 0x7fd7840744b0 2024-03-18T15:29:24.699 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:24.698+0000 7fd783e00640 1 -- 172.21.15.19:0/869932086 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7841a6d40 con 0x7fd7840744b0 2024-03-18T15:29:24.700 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:24.699+0000 7fd783e00640 1 -- 172.21.15.19:0/869932086 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7840684f0 con 0x7fd7840744b0 2024-03-18T15:29:24.702 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:24.702+0000 7fd77b400640 1 -- 172.21.15.19:0/869932086 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 27) ==== 137421+0+0 (secure 0 0 0) 0x7fd7700046c0 con 0x7fd7840744b0 2024-03-18T15:29:24.703 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:24.703+0000 7fd77b400640 1 --2- 172.21.15.19:0/869932086 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7fd7680a2970 0x7fd7680a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:29:24.703 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:24.703+0000 7fd77b400640 1 -- 172.21.15.19:0/869932086 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9939+0+0 (secure 0 0 0) 0x7fd7700cf740 con 0x7fd7840744b0 2024-03-18T15:29:24.703 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:24.704+0000 7fd782a00640 1 --2- 172.21.15.19:0/869932086 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7fd7680a2970 0x7fd7680a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:29:24.704 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:24.704+0000 7fd782a00640 1 --2- 172.21.15.19:0/869932086 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7fd7680a2970 0x7fd7680a4e30 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fd77402c210 tx=0x7fd77402e040 comp rx=0 tx=0).ready entity=mgr.25466 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:29:24.708 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:24.708+0000 7fd77b400640 1 -- 172.21.15.19:0/869932086 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd77009beb0 con 0x7fd7840744b0 2024-03-18T15:29:24.917 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:24.917+0000 7fd783e00640 1 -- 172.21.15.19:0/869932086 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fd784109090 con 0x7fd7840744b0 2024-03-18T15:29:24.966 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:29:24 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/3642234977' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-18T15:29:24.967 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:29:24 smithi019 ceph-mon[31522]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-18T15:29:24.967 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:29:24 smithi019 ceph-mon[31522]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-18T15:29:24.967 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:29:24 smithi019 ceph-mon[31522]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-18T15:29:24.967 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:29:24 smithi019 ceph-mon[31522]: Cluster is now healthy 2024-03-18T15:29:24.967 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:29:24 smithi019 ceph-mon[31522]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-18T15:29:24.967 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:29:24 smithi019 ceph-mon[31522]: mds.? [v2:172.21.15.104:6835/1331087965,v1:172.21.15.104:6837/1331087965] up:boot 2024-03-18T15:29:24.967 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:29:24 smithi019 ceph-mon[31522]: mds.? [v2:172.21.15.104:6838/4189820367,v1:172.21.15.104:6839/4189820367] up:boot 2024-03-18T15:29:24.967 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:29:24 smithi019 ceph-mon[31522]: mds.? [v2:172.21.15.33:6836/301488147,v1:172.21.15.33:6839/301488147] up:boot 2024-03-18T15:29:24.967 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:29:24 smithi019 ceph-mon[31522]: mds.? [v2:172.21.15.33:6835/442844527,v1:172.21.15.33:6838/442844527] up:boot 2024-03-18T15:29:24.967 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:29:24 smithi019 ceph-mon[31522]: mds.? [v2:172.21.15.33:6832/3402863498,v1:172.21.15.33:6833/3402863498] up:boot 2024-03-18T15:29:24.967 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:29:24 smithi019 ceph-mon[31522]: mds.? [v2:172.21.15.104:6832/3998193571,v1:172.21.15.104:6833/3998193571] up:boot 2024-03-18T15:29:24.967 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:29:24 smithi019 ceph-mon[31522]: mds.? [v2:172.21.15.104:6834/3767519313,v1:172.21.15.104:6836/3767519313] up:boot 2024-03-18T15:29:24.968 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:29:24 smithi019 ceph-mon[31522]: mds.? [v2:172.21.15.33:6834/3891003691,v1:172.21.15.33:6837/3891003691] up:boot 2024-03-18T15:29:24.968 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:29:24 smithi019 ceph-mon[31522]: mds.? [v2:172.21.15.19:6836/1872790917,v1:172.21.15.19:6837/1872790917] up:boot 2024-03-18T15:29:24.968 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:29:24 smithi019 ceph-mon[31522]: mds.? [v2:172.21.15.19:6840/2170609358,v1:172.21.15.19:6841/2170609358] up:boot 2024-03-18T15:29:24.968 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:29:24 smithi019 ceph-mon[31522]: fsmap 12 up:standby 2024-03-18T15:29:24.968 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:29:24 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-18T15:29:24.968 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:29:24 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-18T15:29:24.968 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:29:24 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-18T15:29:24.968 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:29:24 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-18T15:29:24.968 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:29:24 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-18T15:29:24.968 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:29:24 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-18T15:29:24.968 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:29:24 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-18T15:29:24.968 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:29:24 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-18T15:29:24.968 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:29:24 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-18T15:29:24.968 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:29:24 smithi019 ceph-mon[31522]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-18T15:29:25.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:29:24 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/3642234977' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-18T15:29:25.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:29:24 smithi033 ceph-mon[36648]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-18T15:29:25.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:29:24 smithi033 ceph-mon[36648]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-18T15:29:25.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:29:24 smithi033 ceph-mon[36648]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-18T15:29:25.008 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:29:24 smithi033 ceph-mon[36648]: Cluster is now healthy 2024-03-18T15:29:25.008 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:29:24 smithi033 ceph-mon[36648]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-18T15:29:25.008 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:29:24 smithi033 ceph-mon[36648]: mds.? [v2:172.21.15.104:6835/1331087965,v1:172.21.15.104:6837/1331087965] up:boot 2024-03-18T15:29:25.008 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:29:24 smithi033 ceph-mon[36648]: mds.? [v2:172.21.15.104:6838/4189820367,v1:172.21.15.104:6839/4189820367] up:boot 2024-03-18T15:29:25.008 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:29:24 smithi033 ceph-mon[36648]: mds.? [v2:172.21.15.33:6836/301488147,v1:172.21.15.33:6839/301488147] up:boot 2024-03-18T15:29:25.008 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:29:24 smithi033 ceph-mon[36648]: mds.? [v2:172.21.15.33:6835/442844527,v1:172.21.15.33:6838/442844527] up:boot 2024-03-18T15:29:25.008 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:29:24 smithi033 ceph-mon[36648]: mds.? [v2:172.21.15.33:6832/3402863498,v1:172.21.15.33:6833/3402863498] up:boot 2024-03-18T15:29:25.008 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:29:24 smithi033 ceph-mon[36648]: mds.? [v2:172.21.15.104:6832/3998193571,v1:172.21.15.104:6833/3998193571] up:boot 2024-03-18T15:29:25.008 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:29:24 smithi033 ceph-mon[36648]: mds.? [v2:172.21.15.104:6834/3767519313,v1:172.21.15.104:6836/3767519313] up:boot 2024-03-18T15:29:25.008 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:29:24 smithi033 ceph-mon[36648]: mds.? [v2:172.21.15.33:6834/3891003691,v1:172.21.15.33:6837/3891003691] up:boot 2024-03-18T15:29:25.009 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:29:24 smithi033 ceph-mon[36648]: mds.? [v2:172.21.15.19:6836/1872790917,v1:172.21.15.19:6837/1872790917] up:boot 2024-03-18T15:29:25.009 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:29:24 smithi033 ceph-mon[36648]: mds.? [v2:172.21.15.19:6840/2170609358,v1:172.21.15.19:6841/2170609358] up:boot 2024-03-18T15:29:25.009 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:29:24 smithi033 ceph-mon[36648]: fsmap 12 up:standby 2024-03-18T15:29:25.009 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:29:24 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-18T15:29:25.009 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:29:24 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-18T15:29:25.009 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:29:24 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-18T15:29:25.009 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:29:24 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-18T15:29:25.009 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:29:24 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-18T15:29:25.009 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:29:24 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-18T15:29:25.009 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:29:24 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-18T15:29:25.010 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:29:24 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-18T15:29:25.010 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:29:24 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-18T15:29:25.010 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:29:24 smithi033 ceph-mon[36648]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-18T15:29:25.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:29:24 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/3642234977' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-18T15:29:25.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:29:24 smithi104 ceph-mon[36070]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-18T15:29:25.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:29:24 smithi104 ceph-mon[36070]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-18T15:29:25.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:29:24 smithi104 ceph-mon[36070]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-18T15:29:25.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:29:24 smithi104 ceph-mon[36070]: Cluster is now healthy 2024-03-18T15:29:25.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:29:24 smithi104 ceph-mon[36070]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-18T15:29:25.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:29:24 smithi104 ceph-mon[36070]: mds.? [v2:172.21.15.104:6835/1331087965,v1:172.21.15.104:6837/1331087965] up:boot 2024-03-18T15:29:25.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:29:24 smithi104 ceph-mon[36070]: mds.? [v2:172.21.15.104:6838/4189820367,v1:172.21.15.104:6839/4189820367] up:boot 2024-03-18T15:29:25.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:29:24 smithi104 ceph-mon[36070]: mds.? [v2:172.21.15.33:6836/301488147,v1:172.21.15.33:6839/301488147] up:boot 2024-03-18T15:29:25.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:29:24 smithi104 ceph-mon[36070]: mds.? [v2:172.21.15.33:6835/442844527,v1:172.21.15.33:6838/442844527] up:boot 2024-03-18T15:29:25.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:29:24 smithi104 ceph-mon[36070]: mds.? [v2:172.21.15.33:6832/3402863498,v1:172.21.15.33:6833/3402863498] up:boot 2024-03-18T15:29:25.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:29:24 smithi104 ceph-mon[36070]: mds.? [v2:172.21.15.104:6832/3998193571,v1:172.21.15.104:6833/3998193571] up:boot 2024-03-18T15:29:25.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:29:24 smithi104 ceph-mon[36070]: mds.? [v2:172.21.15.104:6834/3767519313,v1:172.21.15.104:6836/3767519313] up:boot 2024-03-18T15:29:25.196 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:29:24 smithi104 ceph-mon[36070]: mds.? [v2:172.21.15.33:6834/3891003691,v1:172.21.15.33:6837/3891003691] up:boot 2024-03-18T15:29:25.196 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:29:24 smithi104 ceph-mon[36070]: mds.? [v2:172.21.15.19:6836/1872790917,v1:172.21.15.19:6837/1872790917] up:boot 2024-03-18T15:29:25.196 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:29:24 smithi104 ceph-mon[36070]: mds.? [v2:172.21.15.19:6840/2170609358,v1:172.21.15.19:6841/2170609358] up:boot 2024-03-18T15:29:25.196 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:29:24 smithi104 ceph-mon[36070]: fsmap 12 up:standby 2024-03-18T15:29:25.196 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:29:24 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-18T15:29:25.196 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:29:24 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-18T15:29:25.196 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:29:24 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-18T15:29:25.196 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:29:24 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-18T15:29:25.196 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:29:24 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-18T15:29:25.196 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:29:24 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-18T15:29:25.196 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:29:24 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-18T15:29:25.196 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:29:24 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-18T15:29:25.196 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:29:24 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-18T15:29:25.196 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:29:24 smithi104 ceph-mon[36070]: from='mgr.25466 172.21.15.19:0/381385384' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-18T15:29:25.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:25.716+0000 7fd77b400640 1 -- 172.21.15.19:0/869932086 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v107) ==== 181+0+0 (secure 0 0 0) 0x7fd77009b7b0 con 0x7fd7840744b0 2024-03-18T15:29:25.717 INFO:teuthology.orchestra.run.smithi019.stderr:pool 'cephfs_metadata' removed 2024-03-18T15:29:25.722 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:25.722+0000 7fd783e00640 1 -- 172.21.15.19:0/869932086 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7fd7680a2970 msgr2=0x7fd7680a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:29:25.722 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:25.722+0000 7fd783e00640 1 --2- 172.21.15.19:0/869932086 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7fd7680a2970 0x7fd7680a4e30 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fd77402c210 tx=0x7fd77402e040 comp rx=0 tx=0).stop 2024-03-18T15:29:25.722 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:25.722+0000 7fd783e00640 1 -- 172.21.15.19:0/869932086 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd7840744b0 msgr2=0x7fd78419a710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:29:25.723 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:25.722+0000 7fd783e00640 1 --2- 172.21.15.19:0/869932086 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd7840744b0 0x7fd78419a710 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fd77000e9c0 tx=0x7fd77000ee80 comp rx=0 tx=0).stop 2024-03-18T15:29:25.723 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:25.722+0000 7fd783e00640 1 -- 172.21.15.19:0/869932086 shutdown_connections 2024-03-18T15:29:25.723 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:25.723+0000 7fd783e00640 1 --2- 172.21.15.19:0/869932086 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7fd7680a2970 0x7fd7680a4e30 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:25.723 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:25.723+0000 7fd783e00640 1 --2- 172.21.15.19:0/869932086 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd78410df10 0x7fd78419ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:25.723 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:25.723+0000 7fd783e00640 1 --2- 172.21.15.19:0/869932086 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd7840744b0 0x7fd78419a710 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:25.723 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:25.723+0000 7fd783e00640 1 --2- 172.21.15.19:0/869932086 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd784073b90 0x7fd78419a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:25.724 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:25.723+0000 7fd783e00640 1 -- 172.21.15.19:0/869932086 >> 172.21.15.19:0/869932086 conn(0x7fd7840fd180 msgr2=0x7fd7840fdf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:29:25.724 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:25.723+0000 7fd783e00640 1 -- 172.21.15.19:0/869932086 shutdown_connections 2024-03-18T15:29:25.724 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:25.724+0000 7fd783e00640 1 -- 172.21.15.19:0/869932086 wait complete. 2024-03-18T15:29:25.749 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-18T15:29:25.973 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:25.973+0000 7f2a82c00640 1 -- 172.21.15.19:0/1896680629 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2a7c104340 msgr2=0x7f2a7c110850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:29:25.974 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:25.973+0000 7f2a82c00640 1 --2- 172.21.15.19:0/1896680629 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2a7c104340 0x7f2a7c110850 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f2a74009be0 tx=0x7f2a7402bef0 comp rx=0 tx=0).stop 2024-03-18T15:29:25.974 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:25.973+0000 7f2a82c00640 1 -- 172.21.15.19:0/1896680629 shutdown_connections 2024-03-18T15:29:25.974 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:25.973+0000 7f2a82c00640 1 --2- 172.21.15.19:0/1896680629 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2a7c104340 0x7f2a7c110850 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:25.974 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:25.973+0000 7f2a82c00640 1 --2- 172.21.15.19:0/1896680629 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2a7c103980 0x7f2a7c103e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:25.974 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:25.973+0000 7f2a82c00640 1 --2- 172.21.15.19:0/1896680629 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2a7c109990 0x7f2a7c109d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:25.974 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:25.973+0000 7f2a82c00640 1 -- 172.21.15.19:0/1896680629 >> 172.21.15.19:0/1896680629 conn(0x7f2a7c0ff450 msgr2=0x7f2a7c101870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:29:25.974 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:25.973+0000 7f2a82c00640 1 -- 172.21.15.19:0/1896680629 shutdown_connections 2024-03-18T15:29:25.974 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:25.973+0000 7f2a82c00640 1 -- 172.21.15.19:0/1896680629 wait complete. 2024-03-18T15:29:25.976 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:25.974+0000 7f2a82c00640 1 Processor -- start 2024-03-18T15:29:25.976 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:25.974+0000 7f2a82c00640 1 -- start start 2024-03-18T15:29:25.976 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:25.975+0000 7f2a82c00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2a7c103980 0x7f2a7c19a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:29:25.976 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:25.975+0000 7f2a82c00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2a7c104340 0x7f2a7c19a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:29:25.977 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:25.975+0000 7f2a82c00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2a7c109990 0x7f2a7c19eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:29:25.977 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:25.975+0000 7f2a82c00640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f2a7c113df0 con 0x7f2a7c109990 2024-03-18T15:29:25.977 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:25.975+0000 7f2a82c00640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2a7c113c70 con 0x7f2a7c103980 2024-03-18T15:29:25.977 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:25.975+0000 7f2a82c00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f2a7c113f70 con 0x7f2a7c104340 2024-03-18T15:29:25.977 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:25.975+0000 7f2a80c00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2a7c109990 0x7f2a7c19eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:29:25.977 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:25.975+0000 7f2a80c00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2a7c109990 0x7f2a7c19eaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:60524/0 (socket says 172.21.15.19:60524) 2024-03-18T15:29:25.977 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:25.975+0000 7f2a80c00640 1 -- 172.21.15.19:0/3097838481 learned_addr learned my addr 172.21.15.19:0/3097838481 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:29:25.977 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:25.975+0000 7f2a7be00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2a7c103980 0x7f2a7c19a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:29:25.977 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:25.975+0000 7f2a73400640 1 --2- 172.21.15.19:0/3097838481 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2a7c104340 0x7f2a7c19a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:29:25.977 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:25.976+0000 7f2a80c00640 1 -- 172.21.15.19:0/3097838481 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2a7c104340 msgr2=0x7f2a7c19a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:29:25.977 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:25.976+0000 7f2a80c00640 1 --2- 172.21.15.19:0/3097838481 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2a7c104340 0x7f2a7c19a770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:25.977 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:25.976+0000 7f2a80c00640 1 -- 172.21.15.19:0/3097838481 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2a7c103980 msgr2=0x7f2a7c19a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:29:25.977 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:25.976+0000 7f2a80c00640 1 --2- 172.21.15.19:0/3097838481 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2a7c103980 0x7f2a7c19a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:25.977 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:25.976+0000 7f2a80c00640 1 -- 172.21.15.19:0/3097838481 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a7c19f220 con 0x7f2a7c109990 2024-03-18T15:29:25.978 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:25.976+0000 7f2a80c00640 1 --2- 172.21.15.19:0/3097838481 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2a7c109990 0x7f2a7c19eaf0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f2a74033420 tx=0x7f2a74033720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:29:25.978 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:25.976+0000 7f2a79600640 1 -- 172.21.15.19:0/3097838481 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a74043020 con 0x7f2a7c109990 2024-03-18T15:29:25.978 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:25.976+0000 7f2a79600640 1 -- 172.21.15.19:0/3097838481 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2a74003e50 con 0x7f2a7c109990 2024-03-18T15:29:25.978 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:25.976+0000 7f2a82c00640 1 -- 172.21.15.19:0/3097838481 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a7c19f4b0 con 0x7f2a7c109990 2024-03-18T15:29:25.978 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:25.976+0000 7f2a79600640 1 -- 172.21.15.19:0/3097838481 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a7403e6b0 con 0x7f2a7c109990 2024-03-18T15:29:25.978 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:25.977+0000 7f2a82c00640 1 -- 172.21.15.19:0/3097838481 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a7c1a6da0 con 0x7f2a7c109990 2024-03-18T15:29:25.978 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:25.978+0000 7f2a82c00640 1 -- 172.21.15.19:0/3097838481 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a7c068a20 con 0x7f2a7c109990 2024-03-18T15:29:25.980 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:25.980+0000 7f2a79600640 1 -- 172.21.15.19:0/3097838481 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 27) ==== 137421+0+0 (secure 0 0 0) 0x7f2a74046020 con 0x7f2a7c109990 2024-03-18T15:29:25.981 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:25.981+0000 7f2a79600640 1 --2- 172.21.15.19:0/3097838481 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f2a600a2970 0x7f2a600a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:29:25.981 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:25.981+0000 7f2a79600640 1 -- 172.21.15.19:0/3097838481 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9414+0+0 (secure 0 0 0) 0x7f2a740f13c0 con 0x7f2a7c109990 2024-03-18T15:29:25.981 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:25.981+0000 7f2a7be00640 1 --2- 172.21.15.19:0/3097838481 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f2a600a2970 0x7f2a600a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:29:25.982 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:25.982+0000 7f2a7be00640 1 --2- 172.21.15.19:0/3097838481 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f2a600a2970 0x7f2a600a4e30 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f2a6c00bb60 tx=0x7f2a6c00b390 comp rx=0 tx=0).ready entity=mgr.25466 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:29:25.986 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:25.986+0000 7f2a79600640 1 -- 172.21.15.19:0/3097838481 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2a7404d030 con 0x7f2a7c109990 2024-03-18T15:29:26.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:29:25 smithi033 ceph-mon[36648]: pgmap v342: 129 pgs: 129 active+clean; 1.8 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-18T15:29:26.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:29:25 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/869932086' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-18T15:29:26.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:29:25 smithi104 ceph-mon[36070]: pgmap v342: 129 pgs: 129 active+clean; 1.8 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-18T15:29:26.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:29:25 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/869932086' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-18T15:29:26.195 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:26.194+0000 7f2a82c00640 1 -- 172.21.15.19:0/3097838481 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f2a7c1a7060 con 0x7f2a7c109990 2024-03-18T15:29:26.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:29:25 smithi019 ceph-mon[31522]: pgmap v342: 129 pgs: 129 active+clean; 1.8 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-18T15:29:26.215 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:29:25 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/869932086' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-18T15:29:26.736 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:26.736+0000 7f2a79600640 1 -- 172.21.15.19:0/3097838481 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v108) ==== 169+0+0 (secure 0 0 0) 0x7f2a740bde50 con 0x7f2a7c109990 2024-03-18T15:29:26.737 INFO:teuthology.orchestra.run.smithi019.stderr:pool 'cephfs_data' removed 2024-03-18T15:29:26.741 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:26.741+0000 7f2a82c00640 1 -- 172.21.15.19:0/3097838481 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f2a600a2970 msgr2=0x7f2a600a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:29:26.741 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:26.741+0000 7f2a82c00640 1 --2- 172.21.15.19:0/3097838481 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f2a600a2970 0x7f2a600a4e30 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f2a6c00bb60 tx=0x7f2a6c00b390 comp rx=0 tx=0).stop 2024-03-18T15:29:26.741 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:26.741+0000 7f2a82c00640 1 -- 172.21.15.19:0/3097838481 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2a7c109990 msgr2=0x7f2a7c19eaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:29:26.741 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:26.741+0000 7f2a82c00640 1 --2- 172.21.15.19:0/3097838481 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2a7c109990 0x7f2a7c19eaf0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f2a74033420 tx=0x7f2a74033720 comp rx=0 tx=0).stop 2024-03-18T15:29:26.742 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:26.742+0000 7f2a82c00640 1 -- 172.21.15.19:0/3097838481 shutdown_connections 2024-03-18T15:29:26.742 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:26.742+0000 7f2a82c00640 1 --2- 172.21.15.19:0/3097838481 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f2a600a2970 0x7f2a600a4e30 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:26.742 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:26.742+0000 7f2a82c00640 1 --2- 172.21.15.19:0/3097838481 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2a7c109990 0x7f2a7c19eaf0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:26.742 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:26.742+0000 7f2a82c00640 1 --2- 172.21.15.19:0/3097838481 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2a7c104340 0x7f2a7c19a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:26.742 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:26.742+0000 7f2a82c00640 1 --2- 172.21.15.19:0/3097838481 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2a7c103980 0x7f2a7c19a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:26.742 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:26.742+0000 7f2a82c00640 1 -- 172.21.15.19:0/3097838481 >> 172.21.15.19:0/3097838481 conn(0x7f2a7c0ff450 msgr2=0x7f2a7c0ff970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:29:26.742 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:26.742+0000 7f2a82c00640 1 -- 172.21.15.19:0/3097838481 shutdown_connections 2024-03-18T15:29:26.743 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:26.743+0000 7f2a82c00640 1 -- 172.21.15.19:0/3097838481 wait complete. 2024-03-18T15:29:26.767 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-03-18T15:29:26.981 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:26.980+0000 7f7998e00640 1 -- 172.21.15.19:0/1126536090 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f799410d0c0 msgr2=0x7f7994114c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:29:26.981 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:26.980+0000 7f7998e00640 1 --2- 172.21.15.19:0/1126536090 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f799410d0c0 0x7f7994114c90 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f798c009be0 tx=0x7f798c02bef0 comp rx=0 tx=0).stop 2024-03-18T15:29:26.981 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:26.980+0000 7f7998e00640 1 -- 172.21.15.19:0/1126536090 shutdown_connections 2024-03-18T15:29:26.981 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:26.980+0000 7f7998e00640 1 --2- 172.21.15.19:0/1126536090 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f799410d0c0 0x7f7994114c90 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:26.981 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:26.980+0000 7f7998e00640 1 --2- 172.21.15.19:0/1126536090 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f79941089a0 0x7f799410c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:26.981 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:26.980+0000 7f7998e00640 1 --2- 172.21.15.19:0/1126536090 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7994107ff0 0x7f79941083d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:26.981 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:26.980+0000 7f7998e00640 1 -- 172.21.15.19:0/1126536090 >> 172.21.15.19:0/1126536090 conn(0x7f7994075d20 msgr2=0x7f7994078140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:29:26.981 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:26.981+0000 7f7998e00640 1 -- 172.21.15.19:0/1126536090 shutdown_connections 2024-03-18T15:29:26.981 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:26.981+0000 7f7998e00640 1 -- 172.21.15.19:0/1126536090 wait complete. 2024-03-18T15:29:26.983 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:26.982+0000 7f7998e00640 1 Processor -- start 2024-03-18T15:29:26.983 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:26.982+0000 7f7998e00640 1 -- start start 2024-03-18T15:29:26.984 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:26.982+0000 7f7998e00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7994107ff0 0x7f799419e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:29:26.984 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:26.982+0000 7f7998e00640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f799410d0c0 0x7f799419ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:29:26.984 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:26.982+0000 7f7998e00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f79941a2f10 0x7f79941a33c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:29:26.984 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:26.982+0000 7f7998e00640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f7994118180 con 0x7f79941a2f10 2024-03-18T15:29:26.984 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:26.982+0000 7f7998e00640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7994118000 con 0x7f799410d0c0 2024-03-18T15:29:26.984 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:26.982+0000 7f7998e00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f7994118300 con 0x7f7994107ff0 2024-03-18T15:29:26.984 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:26.982+0000 7f7991e00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7994107ff0 0x7f799419e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:29:26.984 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:26.982+0000 7f7992800640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f79941a2f10 0x7f79941a33c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:29:26.984 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:26.982+0000 7f7991e00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7994107ff0 0x7f799419e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.19:33258/0 (socket says 172.21.15.19:33258) 2024-03-18T15:29:26.984 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:26.982+0000 7f7991e00640 1 -- 172.21.15.19:0/2060608696 learned_addr learned my addr 172.21.15.19:0/2060608696 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-03-18T15:29:26.984 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:26.982+0000 7f7991400640 1 --2- 172.21.15.19:0/2060608696 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f799410d0c0 0x7f799419ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:29:26.984 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:26.983+0000 7f7992800640 1 -- 172.21.15.19:0/2060608696 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7994107ff0 msgr2=0x7f799419e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:29:26.984 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:26.983+0000 7f7992800640 1 --2- 172.21.15.19:0/2060608696 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7994107ff0 0x7f799419e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:26.985 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:26.983+0000 7f7992800640 1 -- 172.21.15.19:0/2060608696 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f799410d0c0 msgr2=0x7f799419ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:29:26.985 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:26.983+0000 7f7992800640 1 --2- 172.21.15.19:0/2060608696 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f799410d0c0 0x7f799419ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:26.985 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:26.983+0000 7f7992800640 1 -- 172.21.15.19:0/2060608696 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79941a3a00 con 0x7f79941a2f10 2024-03-18T15:29:26.985 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:26.983+0000 7f7992800640 1 --2- 172.21.15.19:0/2060608696 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f79941a2f10 0x7f79941a33c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f798c033420 tx=0x7f798c0336b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:29:26.985 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:26.983+0000 7f798aa00640 1 -- 172.21.15.19:0/2060608696 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f798c043020 con 0x7f79941a2f10 2024-03-18T15:29:26.985 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:26.984+0000 7f798aa00640 1 -- 172.21.15.19:0/2060608696 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f798c02ca80 con 0x7f79941a2f10 2024-03-18T15:29:26.985 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:26.984+0000 7f7998e00640 1 -- 172.21.15.19:0/2060608696 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79941a3c90 con 0x7f79941a2f10 2024-03-18T15:29:26.985 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:26.984+0000 7f798aa00640 1 -- 172.21.15.19:0/2060608696 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f798c03e6b0 con 0x7f79941a2f10 2024-03-18T15:29:26.985 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:26.984+0000 7f7998e00640 1 -- 172.21.15.19:0/2060608696 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79941ab450 con 0x7f79941a2f10 2024-03-18T15:29:26.985 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:26.985+0000 7f7998e00640 1 -- 172.21.15.19:0/2060608696 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7994068a20 con 0x7f79941a2f10 2024-03-18T15:29:26.986 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:26.987+0000 7f798aa00640 1 -- 172.21.15.19:0/2060608696 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 27) ==== 137421+0+0 (secure 0 0 0) 0x7f798c046020 con 0x7f79941a2f10 2024-03-18T15:29:26.988 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:26.988+0000 7f798aa00640 1 --2- 172.21.15.19:0/2060608696 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f79780a2970 0x7f79780a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:29:26.988 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:26.988+0000 7f798aa00640 1 -- 172.21.15.19:0/2060608696 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 8797+0+0 (secure 0 0 0) 0x7f798c0f13b0 con 0x7f79941a2f10 2024-03-18T15:29:26.988 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:26.988+0000 7f7991e00640 1 --2- 172.21.15.19:0/2060608696 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f79780a2970 0x7f79780a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:29:26.989 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:26.989+0000 7f7991e00640 1 --2- 172.21.15.19:0/2060608696 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f79780a2970 0x7f79780a4e30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f7984009cd0 tx=0x7f7984005f90 comp rx=0 tx=0).ready entity=mgr.25466 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:29:26.992 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:26.992+0000 7f798aa00640 1 -- 172.21.15.19:0/2060608696 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f798c04d020 con 0x7f79941a2f10 2024-03-18T15:29:27.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:29:26 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/869932086' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-18T15:29:27.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:29:26 smithi033 ceph-mon[36648]: osdmap e107: 12 total, 12 up, 12 in 2024-03-18T15:29:27.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:29:26 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/3097838481' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-18T15:29:27.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:29:26 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/869932086' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-18T15:29:27.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:29:26 smithi104 ceph-mon[36070]: osdmap e107: 12 total, 12 up, 12 in 2024-03-18T15:29:27.195 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:29:26 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/3097838481' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-18T15:29:27.195 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:27.195+0000 7f7998e00640 1 -- 172.21.15.19:0/2060608696 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) -- 0x7f79941089a0 con 0x7f79941a2f10 2024-03-18T15:29:27.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:29:26 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/869932086' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-18T15:29:27.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:29:26 smithi019 ceph-mon[31522]: osdmap e107: 12 total, 12 up, 12 in 2024-03-18T15:29:27.216 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:29:26 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/3097838481' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-18T15:29:27.739 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:27.739+0000 7f798aa00640 1 -- 172.21.15.19:0/2060608696 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v109) ==== 178+0+0 (secure 0 0 0) 0x7f798c0be0b0 con 0x7f79941a2f10 2024-03-18T15:29:27.739 INFO:teuthology.orchestra.run.smithi019.stderr:pool 'cephfs_data_ec' removed 2024-03-18T15:29:27.743 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:27.743+0000 7f7998e00640 1 -- 172.21.15.19:0/2060608696 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f79780a2970 msgr2=0x7f79780a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:29:27.743 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:27.743+0000 7f7998e00640 1 --2- 172.21.15.19:0/2060608696 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f79780a2970 0x7f79780a4e30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f7984009cd0 tx=0x7f7984005f90 comp rx=0 tx=0).stop 2024-03-18T15:29:27.743 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:27.743+0000 7f7998e00640 1 -- 172.21.15.19:0/2060608696 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f79941a2f10 msgr2=0x7f79941a33c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:29:27.743 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:27.743+0000 7f7998e00640 1 --2- 172.21.15.19:0/2060608696 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f79941a2f10 0x7f79941a33c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f798c033420 tx=0x7f798c0336b0 comp rx=0 tx=0).stop 2024-03-18T15:29:27.744 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:27.744+0000 7f7998e00640 1 -- 172.21.15.19:0/2060608696 shutdown_connections 2024-03-18T15:29:27.745 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:27.744+0000 7f7998e00640 1 --2- 172.21.15.19:0/2060608696 >> [v2:172.21.15.19:6800/3894323675,v1:172.21.15.19:6801/3894323675] conn(0x7f79780a2970 0x7f79780a4e30 secure :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f7984009cd0 tx=0x7f7984005f90 comp rx=0 tx=0).stop 2024-03-18T15:29:27.745 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:27.744+0000 7f7998e00640 1 --2- 172.21.15.19:0/2060608696 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f79941a2f10 0x7f79941a33c0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:27.745 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:27.744+0000 7f7998e00640 1 --2- 172.21.15.19:0/2060608696 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f799410d0c0 0x7f799419ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:27.745 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:27.744+0000 7f7998e00640 1 --2- 172.21.15.19:0/2060608696 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7994107ff0 0x7f799419e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:29:27.745 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:27.744+0000 7f7998e00640 1 -- 172.21.15.19:0/2060608696 >> 172.21.15.19:0/2060608696 conn(0x7f7994075d20 msgr2=0x7f7994078060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:29:27.746 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:27.746+0000 7f7998e00640 1 -- 172.21.15.19:0/2060608696 shutdown_connections 2024-03-18T15:29:27.746 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-18T15:29:27.746+0000 7f7998e00640 1 -- 172.21.15.19:0/2060608696 wait complete. 2024-03-18T15:29:27.776 DEBUG:teuthology.orchestra.run.smithi019:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-18T15:29:27.873 DEBUG:teuthology.orchestra.run.smithi033:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-18T15:29:27.913 DEBUG:teuthology.orchestra.run.smithi104:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-18T15:29:27.948 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-18T15:29:27.949 DEBUG:teuthology.orchestra.run.smithi019:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-18T15:29:28.006 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:29:27 smithi033 ceph-mon[36648]: pgmap v344: 97 pgs: 97 active+clean; 577 KiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:29:28.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:29:27 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/3097838481' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-03-18T15:29:28.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:29:27 smithi033 ceph-mon[36648]: osdmap e108: 12 total, 12 up, 12 in 2024-03-18T15:29:28.007 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:29:27 smithi033 ceph-mon[36648]: from='client.? 172.21.15.19:0/2060608696' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-18T15:29:28.009 DEBUG:teuthology.orchestra.run.smithi033:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-18T15:29:28.027 DEBUG:teuthology.orchestra.run.smithi104:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-18T15:29:28.044 INFO:tasks.cephadm:Stopping all daemons... 2024-03-18T15:29:28.045 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-18T15:29:28.045 DEBUG:teuthology.orchestra.run.smithi019:> sudo systemctl stop ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@mon.a 2024-03-18T15:29:28.129 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:29:27 smithi019 ceph-mon[31522]: pgmap v344: 97 pgs: 97 active+clean; 577 KiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:29:28.129 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:29:27 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/3097838481' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-03-18T15:29:28.129 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:29:27 smithi019 ceph-mon[31522]: osdmap e108: 12 total, 12 up, 12 in 2024-03-18T15:29:28.129 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:29:27 smithi019 ceph-mon[31522]: from='client.? 172.21.15.19:0/2060608696' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-18T15:29:28.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:29:27 smithi104 ceph-mon[36070]: pgmap v344: 97 pgs: 97 active+clean; 577 KiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:29:28.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:29:27 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/3097838481' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-03-18T15:29:28.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:29:27 smithi104 ceph-mon[36070]: osdmap e108: 12 total, 12 up, 12 in 2024-03-18T15:29:28.194 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:29:27 smithi104 ceph-mon[36070]: from='client.? 172.21.15.19:0/2060608696' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-18T15:29:28.465 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:29:28 smithi019 systemd[1]: Stopping Ceph mon.a for d4dc2ba6-e538-11ee-95c9-87774f69a715... 2024-03-18T15:29:28.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:29:28 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mon-a[31518]: 2024-03-18T15:29:28.649+0000 7ffaca400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T15:29:28.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:29:28 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mon-a[31518]: 2024-03-18T15:29:28.649+0000 7ffaca400700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-18T15:29:28.965 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:29:28 smithi019 podman[92141]: 2024-03-18 15:29:28.711806751 +0000 UTC m=+0.207157944 container died 21fe4b08ac45f6975b99831acf5267fbbe1580ae6db5a8c7857a30b26b5f9d7b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mon-a, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T15:29:29.434 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:29:29 smithi019 podman[92141]: 2024-03-18 15:29:29.155989481 +0000 UTC m=+0.651340701 container cleanup 21fe4b08ac45f6975b99831acf5267fbbe1580ae6db5a8c7857a30b26b5f9d7b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mon-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5) 2024-03-18T15:29:29.435 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:29:29 smithi019 bash[92141]: ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mon-a 2024-03-18T15:29:29.715 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:29:29 smithi019 podman[92154]: 2024-03-18 15:29:29.432021358 +0000 UTC m=+0.709990088 container remove 21fe4b08ac45f6975b99831acf5267fbbe1580ae6db5a8c7857a30b26b5f9d7b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mon-a, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307) 2024-03-18T15:29:29.920 DEBUG:teuthology.orchestra.run.smithi019:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@mon.a.service' 2024-03-18T15:29:30.016 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:29:29 smithi019 systemd[1]: ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-18T15:29:30.016 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:29:29 smithi019 systemd[1]: Stopped Ceph mon.a for d4dc2ba6-e538-11ee-95c9-87774f69a715. 2024-03-18T15:29:30.016 INFO:journalctl@ceph.mon.a.smithi019.stdout:Mar 18 15:29:29 smithi019 systemd[1]: ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@mon.a.service: Consumed 45.986s CPU time. 2024-03-18T15:29:30.069 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T15:29:30.069 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-18T15:29:30.069 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-03-18T15:29:30.070 DEBUG:teuthology.orchestra.run.smithi033:> sudo systemctl stop ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@mon.b 2024-03-18T15:29:30.381 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:29:30 smithi033 systemd[1]: Stopping Ceph mon.b for d4dc2ba6-e538-11ee-95c9-87774f69a715... 2024-03-18T15:29:30.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:29:30 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mon-b[36644]: 2024-03-18T15:29:30.380+0000 7f67ca67c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T15:29:30.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:29:30 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mon-b[36644]: 2024-03-18T15:29:30.380+0000 7f67ca67c700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-03-18T15:29:30.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:29:30 smithi033 podman[73702]: 2024-03-18 15:29:30.473447728 +0000 UTC m=+0.202117930 container died 5d7d00c66554d26cf7bbeb62dc6eedbbb10125147a831a90d65f4722c6488094 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mon-b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-03-18T15:29:31.141 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:29:30 smithi033 podman[73702]: 2024-03-18 15:29:30.848257662 +0000 UTC m=+0.576927864 container cleanup 5d7d00c66554d26cf7bbeb62dc6eedbbb10125147a831a90d65f4722c6488094 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mon-b, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-03-18T15:29:31.141 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 18 15:29:30 smithi033 bash[73702]: ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mon-b 2024-03-18T15:29:31.277 DEBUG:teuthology.orchestra.run.smithi033:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@mon.b.service' 2024-03-18T15:29:31.333 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T15:29:31.334 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-03-18T15:29:31.334 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-03-18T15:29:31.334 DEBUG:teuthology.orchestra.run.smithi104:> sudo systemctl stop ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@mon.c 2024-03-18T15:29:31.633 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:29:31 smithi104 systemd[1]: Stopping Ceph mon.c for d4dc2ba6-e538-11ee-95c9-87774f69a715... 2024-03-18T15:29:31.903 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:29:31 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mon-c[36053]: 2024-03-18T15:29:31.632+0000 7fbad5faa700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T15:29:31.903 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:29:31 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mon-c[36053]: 2024-03-18T15:29:31.632+0000 7fbad5faa700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-03-18T15:29:31.903 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:29:31 smithi104 podman[73214]: 2024-03-18 15:29:31.816512945 +0000 UTC m=+0.292308136 container died b6b4027b63821f2b12853abc725550da8b7f89ecf8e4ce4543d33e802404ed2f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mon-c, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T15:29:32.632 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:29:32 smithi104 podman[73214]: 2024-03-18 15:29:32.217199432 +0000 UTC m=+0.692994607 container cleanup b6b4027b63821f2b12853abc725550da8b7f89ecf8e4ce4543d33e802404ed2f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mon-c, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T15:29:32.632 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:29:32 smithi104 bash[73214]: ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mon-c 2024-03-18T15:29:32.632 INFO:journalctl@ceph.mon.c.smithi104.stdout:Mar 18 15:29:32 smithi104 podman[73229]: 2024-03-18 15:29:32.420634823 +0000 UTC m=+0.605309558 container remove b6b4027b63821f2b12853abc725550da8b7f89ecf8e4ce4543d33e802404ed2f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mon-c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T15:29:32.652 DEBUG:teuthology.orchestra.run.smithi104:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@mon.c.service' 2024-03-18T15:29:32.718 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T15:29:32.718 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-03-18T15:29:32.719 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-18T15:29:32.719 DEBUG:teuthology.orchestra.run.smithi019:> sudo systemctl stop ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@mgr.x 2024-03-18T15:29:33.215 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:29:32 smithi019 systemd[1]: Stopping Ceph mgr.x for d4dc2ba6-e538-11ee-95c9-87774f69a715... 2024-03-18T15:29:33.715 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:29:33 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:29:33.309+0000 7f8fb1800700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T15:29:33.716 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:29:33 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x[31750]: 2024-03-18T15:29:33.309+0000 7f8fb1800700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-18T15:29:33.716 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:29:33 smithi019 podman[92263]: 2024-03-18 15:29:33.414303286 +0000 UTC m=+0.250277965 container died 8574ee959649a93587a0fd2a8e544ed301f945e10af3b1244982598b1fa93a50 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-18T15:29:34.215 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:29:34 smithi019 podman[92263]: 2024-03-18 15:29:34.077087547 +0000 UTC m=+0.913062211 container cleanup 8574ee959649a93587a0fd2a8e544ed301f945e10af3b1244982598b1fa93a50 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2) 2024-03-18T15:29:34.215 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:29:34 smithi019 bash[92263]: ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x 2024-03-18T15:29:34.715 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Mar 18 15:29:34 smithi019 podman[92277]: 2024-03-18 15:29:34.345287387 +0000 UTC m=+0.911400496 container remove 8574ee959649a93587a0fd2a8e544ed301f945e10af3b1244982598b1fa93a50 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-x, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-03-18T15:29:34.900 DEBUG:teuthology.orchestra.run.smithi019:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@mgr.x.service' 2024-03-18T15:29:35.033 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T15:29:35.034 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-18T15:29:35.034 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-18T15:29:35.034 DEBUG:teuthology.orchestra.run.smithi033:> sudo systemctl stop ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@mgr.y 2024-03-18T15:29:35.373 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Mar 18 15:29:35 smithi033 systemd[1]: Stopping Ceph mgr.y for d4dc2ba6-e538-11ee-95c9-87774f69a715... 2024-03-18T15:29:35.756 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Mar 18 15:29:35 smithi033 podman[73826]: 2024-03-18 15:29:35.373554889 +0000 UTC m=+0.141872335 container died 1946eb032c4c4d1671275b5e16e8d207923313bb123046370624ba414b31bdaf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-y, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-18T15:29:36.231 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Mar 18 15:29:35 smithi033 podman[73826]: 2024-03-18 15:29:35.970887523 +0000 UTC m=+0.739204962 container cleanup 1946eb032c4c4d1671275b5e16e8d207923313bb123046370624ba414b31bdaf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-y, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2) 2024-03-18T15:29:36.231 INFO:journalctl@ceph.mgr.y.smithi033.stdout:Mar 18 15:29:35 smithi033 bash[73826]: ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-y 2024-03-18T15:29:36.366 DEBUG:teuthology.orchestra.run.smithi033:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@mgr.y.service' 2024-03-18T15:29:36.410 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T15:29:36.410 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-18T15:29:36.410 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-03-18T15:29:36.410 DEBUG:teuthology.orchestra.run.smithi104:> sudo systemctl stop ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@mgr.z 2024-03-18T15:29:36.694 INFO:journalctl@ceph.mgr.z.smithi104.stdout:Mar 18 15:29:36 smithi104 systemd[1]: Stopping Ceph mgr.z for d4dc2ba6-e538-11ee-95c9-87774f69a715... 2024-03-18T15:29:37.024 INFO:journalctl@ceph.mgr.z.smithi104.stdout:Mar 18 15:29:36 smithi104 podman[73347]: 2024-03-18 15:29:36.761052744 +0000 UTC m=+0.160128997 container died 363a7311517323a28c3c4befeea29ecdaf790735c0ccef1a57c1f5964b89a784 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-z, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T15:29:37.434 INFO:journalctl@ceph.mgr.z.smithi104.stdout:Mar 18 15:29:37 smithi104 podman[73347]: 2024-03-18 15:29:37.123930416 +0000 UTC m=+0.523006660 container cleanup 363a7311517323a28c3c4befeea29ecdaf790735c0ccef1a57c1f5964b89a784 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-z, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T15:29:37.434 INFO:journalctl@ceph.mgr.z.smithi104.stdout:Mar 18 15:29:37 smithi104 bash[73347]: ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mgr-z 2024-03-18T15:29:37.594 DEBUG:teuthology.orchestra.run.smithi104:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@mgr.z.service' 2024-03-18T15:29:37.632 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T15:29:37.632 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-03-18T15:29:37.632 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-18T15:29:37.632 DEBUG:teuthology.orchestra.run.smithi019:> sudo systemctl stop ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@osd.0 2024-03-18T15:29:38.215 INFO:journalctl@ceph.osd.0.smithi019.stdout:Mar 18 15:29:37 smithi019 systemd[1]: Stopping Ceph osd.0 for d4dc2ba6-e538-11ee-95c9-87774f69a715... 2024-03-18T15:29:38.718 INFO:journalctl@ceph.osd.0.smithi019.stdout:Mar 18 15:29:38 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-0[45979]: 2024-03-18T15:29:38.229+0000 7f04b0400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T15:29:38.719 INFO:journalctl@ceph.osd.0.smithi019.stdout:Mar 18 15:29:38 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-0[45979]: 2024-03-18T15:29:38.229+0000 7f04b0400700 -1 osd.0 109 *** Got signal Terminated *** 2024-03-18T15:29:38.719 INFO:journalctl@ceph.osd.0.smithi019.stdout:Mar 18 15:29:38 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-0[45979]: 2024-03-18T15:29:38.229+0000 7f04b0400700 -1 osd.0 109 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T15:29:44.071 INFO:journalctl@ceph.osd.0.smithi019.stdout:Mar 18 15:29:43 smithi019 podman[92382]: 2024-03-18 15:29:43.749196251 +0000 UTC m=+5.660322092 container died 8fd8596f32038adbe09f58fcce3ec74791d2576fd694a258c37ed57e961f7f4e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-0, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-18T15:29:44.442 INFO:journalctl@ceph.osd.0.smithi019.stdout:Mar 18 15:29:44 smithi019 podman[92382]: 2024-03-18 15:29:44.215113572 +0000 UTC m=+6.126239442 container cleanup 8fd8596f32038adbe09f58fcce3ec74791d2576fd694a258c37ed57e961f7f4e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-18T15:29:44.442 INFO:journalctl@ceph.osd.0.smithi019.stdout:Mar 18 15:29:44 smithi019 bash[92382]: ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-0 2024-03-18T15:29:44.715 INFO:journalctl@ceph.osd.0.smithi019.stdout:Mar 18 15:29:44 smithi019 podman[92401]: 2024-03-18 15:29:44.441261301 +0000 UTC m=+0.680816111 container remove 8fd8596f32038adbe09f58fcce3ec74791d2576fd694a258c37ed57e961f7f4e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-0, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-18T15:29:45.591 INFO:journalctl@ceph.osd.0.smithi019.stdout:Mar 18 15:29:45 smithi019 podman[92485]: 2024-03-18 15:29:45.154555284 +0000 UTC m=+0.044763486 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T15:29:45.591 INFO:journalctl@ceph.osd.0.smithi019.stdout:Mar 18 15:29:45 smithi019 podman[92485]: 2024-03-18 15:29:45.275619793 +0000 UTC m=+0.165828020 container create d399a587c0b793ebfc4b74ccad7ba541e767158bb341c2b83ccd8389d3113ead (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-18T15:29:45.592 INFO:journalctl@ceph.osd.0.smithi019.stdout:Mar 18 15:29:45 smithi019 podman[92485]: 2024-03-18 15:29:45.589560251 +0000 UTC m=+0.479768495 container init d399a587c0b793ebfc4b74ccad7ba541e767158bb341c2b83ccd8389d3113ead (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-0-deactivate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T15:29:45.965 INFO:journalctl@ceph.osd.0.smithi019.stdout:Mar 18 15:29:45 smithi019 podman[92485]: 2024-03-18 15:29:45.604695225 +0000 UTC m=+0.494903405 container start d399a587c0b793ebfc4b74ccad7ba541e767158bb341c2b83ccd8389d3113ead (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-0-deactivate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-18T15:29:45.965 INFO:journalctl@ceph.osd.0.smithi019.stdout:Mar 18 15:29:45 smithi019 podman[92485]: 2024-03-18 15:29:45.820168638 +0000 UTC m=+0.710376868 container attach d399a587c0b793ebfc4b74ccad7ba541e767158bb341c2b83ccd8389d3113ead (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-0-deactivate, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-03-18T15:29:46.453 INFO:journalctl@ceph.osd.0.smithi019.stdout:Mar 18 15:29:46 smithi019 podman[92485]: 2024-03-18 15:29:46.057020707 +0000 UTC m=+0.947228951 container died d399a587c0b793ebfc4b74ccad7ba541e767158bb341c2b83ccd8389d3113ead (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-0-deactivate, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240307) 2024-03-18T15:29:46.842 DEBUG:teuthology.orchestra.run.smithi019:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@osd.0.service' 2024-03-18T15:29:46.965 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T15:29:46.966 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-18T15:29:46.966 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-18T15:29:46.966 DEBUG:teuthology.orchestra.run.smithi033:> sudo systemctl stop ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@osd.1 2024-03-18T15:29:47.257 INFO:journalctl@ceph.osd.1.smithi033.stdout:Mar 18 15:29:47 smithi033 systemd[1]: Stopping Ceph osd.1 for d4dc2ba6-e538-11ee-95c9-87774f69a715... 2024-03-18T15:29:47.757 INFO:journalctl@ceph.osd.1.smithi033.stdout:Mar 18 15:29:47 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-1[41252]: 2024-03-18T15:29:47.283+0000 7f4260f6d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T15:29:47.757 INFO:journalctl@ceph.osd.1.smithi033.stdout:Mar 18 15:29:47 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-1[41252]: 2024-03-18T15:29:47.283+0000 7f4260f6d700 -1 osd.1 109 *** Got signal Terminated *** 2024-03-18T15:29:47.757 INFO:journalctl@ceph.osd.1.smithi033.stdout:Mar 18 15:29:47 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-1[41252]: 2024-03-18T15:29:47.283+0000 7f4260f6d700 -1 osd.1 109 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T15:29:52.951 INFO:journalctl@ceph.osd.1.smithi033.stdout:Mar 18 15:29:52 smithi033 podman[73962]: 2024-03-18 15:29:52.641230784 +0000 UTC m=+5.481961290 container died aaaeef516cade91e719c8a6faaf5278ce7a1be3397e84173deea26a80e25aece (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-1, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-03-18T15:29:53.256 INFO:journalctl@ceph.osd.1.smithi033.stdout:Mar 18 15:29:53 smithi033 podman[73962]: 2024-03-18 15:29:53.033947036 +0000 UTC m=+5.874677553 container cleanup aaaeef516cade91e719c8a6faaf5278ce7a1be3397e84173deea26a80e25aece (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-1, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T15:29:53.257 INFO:journalctl@ceph.osd.1.smithi033.stdout:Mar 18 15:29:53 smithi033 bash[73962]: ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-1 2024-03-18T15:29:53.609 INFO:journalctl@ceph.osd.1.smithi033.stdout:Mar 18 15:29:53 smithi033 podman[73979]: 2024-03-18 15:29:53.294001291 +0000 UTC m=+0.652257031 container remove aaaeef516cade91e719c8a6faaf5278ce7a1be3397e84173deea26a80e25aece (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-1, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS) 2024-03-18T15:29:53.881 INFO:journalctl@ceph.osd.1.smithi033.stdout:Mar 18 15:29:53 smithi033 podman[74069]: 2024-03-18 15:29:53.511971861 +0000 UTC m=+0.018359826 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T15:29:53.881 INFO:journalctl@ceph.osd.1.smithi033.stdout:Mar 18 15:29:53 smithi033 podman[74069]: 2024-03-18 15:29:53.619934595 +0000 UTC m=+0.126322549 container create 95f153ef79b528973adfe7d7a8e1ac33df5942ce199a7654aeea88ecba2bb0f6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-1-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T15:29:53.881 INFO:journalctl@ceph.osd.1.smithi033.stdout:Mar 18 15:29:53 smithi033 podman[74069]: 2024-03-18 15:29:53.881133137 +0000 UTC m=+0.387521098 container init 95f153ef79b528973adfe7d7a8e1ac33df5942ce199a7654aeea88ecba2bb0f6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-1-deactivate, org.label-schema.build-date=20240307, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T15:29:54.185 INFO:journalctl@ceph.osd.1.smithi033.stdout:Mar 18 15:29:53 smithi033 podman[74069]: 2024-03-18 15:29:53.884981573 +0000 UTC m=+0.391369535 container start 95f153ef79b528973adfe7d7a8e1ac33df5942ce199a7654aeea88ecba2bb0f6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-1-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-18T15:29:54.185 INFO:journalctl@ceph.osd.1.smithi033.stdout:Mar 18 15:29:53 smithi033 podman[74069]: 2024-03-18 15:29:53.99791071 +0000 UTC m=+0.504298677 container attach 95f153ef79b528973adfe7d7a8e1ac33df5942ce199a7654aeea88ecba2bb0f6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-1-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T15:29:54.462 INFO:journalctl@ceph.osd.1.smithi033.stdout:Mar 18 15:29:54 smithi033 podman[74069]: 2024-03-18 15:29:54.185868649 +0000 UTC m=+0.692256618 container died 95f153ef79b528973adfe7d7a8e1ac33df5942ce199a7654aeea88ecba2bb0f6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, ceph=True, org.label-schema.license=GPLv2) 2024-03-18T15:29:54.824 DEBUG:teuthology.orchestra.run.smithi033:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@osd.1.service' 2024-03-18T15:29:54.871 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T15:29:54.871 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-18T15:29:54.871 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-18T15:29:54.872 DEBUG:teuthology.orchestra.run.smithi104:> sudo systemctl stop ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@osd.2 2024-03-18T15:29:55.194 INFO:journalctl@ceph.osd.2.smithi104.stdout:Mar 18 15:29:54 smithi104 systemd[1]: Stopping Ceph osd.2 for d4dc2ba6-e538-11ee-95c9-87774f69a715... 2024-03-18T15:29:55.694 INFO:journalctl@ceph.osd.2.smithi104.stdout:Mar 18 15:29:55 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-2[41206]: 2024-03-18T15:29:55.193+0000 7f0f88d5b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T15:29:55.694 INFO:journalctl@ceph.osd.2.smithi104.stdout:Mar 18 15:29:55 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-2[41206]: 2024-03-18T15:29:55.193+0000 7f0f88d5b700 -1 osd.2 109 *** Got signal Terminated *** 2024-03-18T15:29:55.694 INFO:journalctl@ceph.osd.2.smithi104.stdout:Mar 18 15:29:55 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-2[41206]: 2024-03-18T15:29:55.193+0000 7f0f88d5b700 -1 osd.2 109 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T15:30:01.004 INFO:journalctl@ceph.osd.2.smithi104.stdout:Mar 18 15:30:00 smithi104 podman[73470]: 2024-03-18 15:30:00.721163996 +0000 UTC m=+5.659799106 container died 76e6f4e2e8c68407dff35d86ee4d80d63ac9966254f4c24fa8536483af9aeadc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD) 2024-03-18T15:30:01.404 INFO:journalctl@ceph.osd.2.smithi104.stdout:Mar 18 15:30:01 smithi104 podman[73470]: 2024-03-18 15:30:01.110800467 +0000 UTC m=+6.049435578 container cleanup 76e6f4e2e8c68407dff35d86ee4d80d63ac9966254f4c24fa8536483af9aeadc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5) 2024-03-18T15:30:01.404 INFO:journalctl@ceph.osd.2.smithi104.stdout:Mar 18 15:30:01 smithi104 bash[73470]: ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-2 2024-03-18T15:30:01.695 INFO:journalctl@ceph.osd.2.smithi104.stdout:Mar 18 15:30:01 smithi104 podman[73488]: 2024-03-18 15:30:01.404117844 +0000 UTC m=+0.681841900 container remove 76e6f4e2e8c68407dff35d86ee4d80d63ac9966254f4c24fa8536483af9aeadc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T15:30:02.008 INFO:journalctl@ceph.osd.2.smithi104.stdout:Mar 18 15:30:01 smithi104 podman[73578]: 2024-03-18 15:30:01.638266338 +0000 UTC m=+0.017570043 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T15:30:02.008 INFO:journalctl@ceph.osd.2.smithi104.stdout:Mar 18 15:30:01 smithi104 podman[73578]: 2024-03-18 15:30:01.755378057 +0000 UTC m=+0.134681760 container create d9960d66695054fcabf10729d91ab8e9db6506dfc0f128fbc73c8a001cb368a9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-2-deactivate, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-18T15:30:02.300 INFO:journalctl@ceph.osd.2.smithi104.stdout:Mar 18 15:30:02 smithi104 podman[73578]: 2024-03-18 15:30:02.008282219 +0000 UTC m=+0.387585944 container init d9960d66695054fcabf10729d91ab8e9db6506dfc0f128fbc73c8a001cb368a9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-2-deactivate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-18T15:30:02.300 INFO:journalctl@ceph.osd.2.smithi104.stdout:Mar 18 15:30:02 smithi104 podman[73578]: 2024-03-18 15:30:02.011726682 +0000 UTC m=+0.391030380 container start d9960d66695054fcabf10729d91ab8e9db6506dfc0f128fbc73c8a001cb368a9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-2-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, org.label-schema.vendor=CentOS) 2024-03-18T15:30:02.300 INFO:journalctl@ceph.osd.2.smithi104.stdout:Mar 18 15:30:02 smithi104 podman[73578]: 2024-03-18 15:30:02.108131772 +0000 UTC m=+0.487435484 container attach d9960d66695054fcabf10729d91ab8e9db6506dfc0f128fbc73c8a001cb368a9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-2-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T15:30:02.564 INFO:journalctl@ceph.osd.2.smithi104.stdout:Mar 18 15:30:02 smithi104 podman[73578]: 2024-03-18 15:30:02.300694368 +0000 UTC m=+0.679998072 container died d9960d66695054fcabf10729d91ab8e9db6506dfc0f128fbc73c8a001cb368a9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-2-deactivate, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-03-18T15:30:02.952 DEBUG:teuthology.orchestra.run.smithi104:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@osd.2.service' 2024-03-18T15:30:02.994 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T15:30:02.994 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-18T15:30:02.994 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-18T15:30:02.995 DEBUG:teuthology.orchestra.run.smithi019:> sudo systemctl stop ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@osd.3 2024-03-18T15:30:03.466 INFO:journalctl@ceph.osd.3.smithi019.stdout:Mar 18 15:30:03 smithi019 systemd[1]: Stopping Ceph osd.3 for d4dc2ba6-e538-11ee-95c9-87774f69a715... 2024-03-18T15:30:03.965 INFO:journalctl@ceph.osd.3.smithi019.stdout:Mar 18 15:30:03 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-3[52790]: 2024-03-18T15:30:03.598+0000 7f6bbfc00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T15:30:03.966 INFO:journalctl@ceph.osd.3.smithi019.stdout:Mar 18 15:30:03 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-3[52790]: 2024-03-18T15:30:03.598+0000 7f6bbfc00700 -1 osd.3 109 *** Got signal Terminated *** 2024-03-18T15:30:03.966 INFO:journalctl@ceph.osd.3.smithi019.stdout:Mar 18 15:30:03 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-3[52790]: 2024-03-18T15:30:03.598+0000 7f6bbfc00700 -1 osd.3 109 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T15:30:05.195 INFO:journalctl@ceph.osd.5.smithi104.stdout:Mar 18 15:30:04 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-5[48148]: 2024-03-18T15:30:04.878+0000 7f6ec6d5c700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:40.553008+0000 front 2024-03-18T15:29:40.552917+0000 (oldest deadline 2024-03-18T15:30:04.052932+0000) 2024-03-18T15:30:05.195 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:04 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:04.945+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:41.404783+0000 front 2024-03-18T15:29:41.404661+0000 (oldest deadline 2024-03-18T15:30:04.304572+0000) 2024-03-18T15:30:05.716 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:05 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:05.284+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:39.237875+0000 front 2024-03-18T15:29:39.238431+0000 (oldest deadline 2024-03-18T15:30:05.138032+0000) 2024-03-18T15:30:06.194 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:05 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:05.952+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:41.404783+0000 front 2024-03-18T15:29:41.404661+0000 (oldest deadline 2024-03-18T15:30:04.304572+0000) 2024-03-18T15:30:06.194 INFO:journalctl@ceph.osd.5.smithi104.stdout:Mar 18 15:30:05 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-5[48148]: 2024-03-18T15:30:05.895+0000 7f6ec6d5c700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:40.553008+0000 front 2024-03-18T15:29:40.552917+0000 (oldest deadline 2024-03-18T15:30:04.052932+0000) 2024-03-18T15:30:06.276 INFO:journalctl@ceph.osd.3.smithi019.stdout:Mar 18 15:30:05 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-3[52790]: 2024-03-18T15:30:05.981+0000 7f6bbb400700 -1 osd.3 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:41.767445+0000 front 2024-03-18T15:29:41.766309+0000 (oldest deadline 2024-03-18T15:30:05.266565+0000) 2024-03-18T15:30:06.715 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:06 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:06.275+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:39.237875+0000 front 2024-03-18T15:29:39.238431+0000 (oldest deadline 2024-03-18T15:30:05.138032+0000) 2024-03-18T15:30:07.194 INFO:journalctl@ceph.osd.5.smithi104.stdout:Mar 18 15:30:06 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-5[48148]: 2024-03-18T15:30:06.880+0000 7f6ec6d5c700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:40.553008+0000 front 2024-03-18T15:29:40.552917+0000 (oldest deadline 2024-03-18T15:30:04.052932+0000) 2024-03-18T15:30:07.195 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:06 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:06.968+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:41.404783+0000 front 2024-03-18T15:29:41.404661+0000 (oldest deadline 2024-03-18T15:30:04.304572+0000) 2024-03-18T15:30:07.255 INFO:journalctl@ceph.osd.3.smithi019.stdout:Mar 18 15:30:06 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-3[52790]: 2024-03-18T15:30:06.982+0000 7f6bbb400700 -1 osd.3 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:41.767445+0000 front 2024-03-18T15:29:41.766309+0000 (oldest deadline 2024-03-18T15:30:05.266565+0000) 2024-03-18T15:30:07.715 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:07 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:07.254+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:39.237875+0000 front 2024-03-18T15:29:39.238431+0000 (oldest deadline 2024-03-18T15:30:05.138032+0000) 2024-03-18T15:30:08.194 INFO:journalctl@ceph.osd.5.smithi104.stdout:Mar 18 15:30:07 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-5[48148]: 2024-03-18T15:30:07.874+0000 7f6ec6d5c700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:40.553008+0000 front 2024-03-18T15:29:40.552917+0000 (oldest deadline 2024-03-18T15:30:04.052932+0000) 2024-03-18T15:30:08.198 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:07 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:07.949+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:41.404783+0000 front 2024-03-18T15:29:41.404661+0000 (oldest deadline 2024-03-18T15:30:04.304572+0000) 2024-03-18T15:30:08.228 INFO:journalctl@ceph.osd.3.smithi019.stdout:Mar 18 15:30:07 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-3[52790]: 2024-03-18T15:30:07.969+0000 7f6bbb400700 -1 osd.3 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:41.767445+0000 front 2024-03-18T15:29:41.766309+0000 (oldest deadline 2024-03-18T15:30:05.266565+0000) 2024-03-18T15:30:08.715 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:08 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:08.226+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:39.237875+0000 front 2024-03-18T15:29:39.238431+0000 (oldest deadline 2024-03-18T15:30:05.138032+0000) 2024-03-18T15:30:09.194 INFO:journalctl@ceph.osd.5.smithi104.stdout:Mar 18 15:30:08 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-5[48148]: 2024-03-18T15:30:08.901+0000 7f6ec6d5c700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:40.553008+0000 front 2024-03-18T15:29:40.552917+0000 (oldest deadline 2024-03-18T15:30:04.052932+0000) 2024-03-18T15:30:09.195 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:08 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:08.996+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:41.404783+0000 front 2024-03-18T15:29:41.404661+0000 (oldest deadline 2024-03-18T15:30:04.304572+0000) 2024-03-18T15:30:09.466 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:09 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:09.260+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:39.237875+0000 front 2024-03-18T15:29:39.238431+0000 (oldest deadline 2024-03-18T15:30:05.138032+0000) 2024-03-18T15:30:09.466 INFO:journalctl@ceph.osd.3.smithi019.stdout:Mar 18 15:30:09 smithi019 podman[92604]: 2024-03-18 15:30:09.113507223 +0000 UTC m=+5.653055747 container stop e7106c105729f163e674cc27d7baee99bff7680c3653fdefcb37b5a3d0ab1125 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-3, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T15:30:09.466 INFO:journalctl@ceph.osd.3.smithi019.stdout:Mar 18 15:30:09 smithi019 podman[92604]: 2024-03-18 15:30:09.140927963 +0000 UTC m=+5.680476456 container died e7106c105729f163e674cc27d7baee99bff7680c3653fdefcb37b5a3d0ab1125 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-3, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T15:30:09.965 INFO:journalctl@ceph.osd.3.smithi019.stdout:Mar 18 15:30:09 smithi019 podman[92604]: 2024-03-18 15:30:09.766981924 +0000 UTC m=+6.306530444 container cleanup e7106c105729f163e674cc27d7baee99bff7680c3653fdefcb37b5a3d0ab1125 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-3, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-03-18T15:30:09.966 INFO:journalctl@ceph.osd.3.smithi019.stdout:Mar 18 15:30:09 smithi019 bash[92604]: ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-3 2024-03-18T15:30:10.194 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:10 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:09.999+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:41.404783+0000 front 2024-03-18T15:29:41.404661+0000 (oldest deadline 2024-03-18T15:30:04.304572+0000) 2024-03-18T15:30:10.194 INFO:journalctl@ceph.osd.5.smithi104.stdout:Mar 18 15:30:09 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-5[48148]: 2024-03-18T15:30:09.882+0000 7f6ec6d5c700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:40.553008+0000 front 2024-03-18T15:29:40.552917+0000 (oldest deadline 2024-03-18T15:30:04.052932+0000) 2024-03-18T15:30:10.308 INFO:journalctl@ceph.osd.3.smithi019.stdout:Mar 18 15:30:09 smithi019 podman[92622]: 2024-03-18 15:30:09.99312313 +0000 UTC m=+0.857313990 container remove e7106c105729f163e674cc27d7baee99bff7680c3653fdefcb37b5a3d0ab1125 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-3, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-18T15:30:10.715 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:10 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:10.306+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:39.237875+0000 front 2024-03-18T15:29:39.238431+0000 (oldest deadline 2024-03-18T15:30:05.138032+0000) 2024-03-18T15:30:11.189 INFO:journalctl@ceph.osd.3.smithi019.stdout:Mar 18 15:30:10 smithi019 podman[92713]: 2024-03-18 15:30:10.769239446 +0000 UTC m=+0.038855148 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T15:30:11.189 INFO:journalctl@ceph.osd.3.smithi019.stdout:Mar 18 15:30:11 smithi019 podman[92713]: 2024-03-18 15:30:11.002378381 +0000 UTC m=+0.271994108 container create 3fcf0b95174322f840468ecce17cb25dcafd3679b82d4d439ff237ecef1129b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-3-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T15:30:11.194 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:10 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:10.961+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:41.404783+0000 front 2024-03-18T15:29:41.404661+0000 (oldest deadline 2024-03-18T15:30:04.304572+0000) 2024-03-18T15:30:11.194 INFO:journalctl@ceph.osd.5.smithi104.stdout:Mar 18 15:30:10 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-5[48148]: 2024-03-18T15:30:10.840+0000 7f6ec6d5c700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:40.553008+0000 front 2024-03-18T15:29:40.552917+0000 (oldest deadline 2024-03-18T15:30:04.052932+0000) 2024-03-18T15:30:11.465 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:11 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:11.294+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:39.237875+0000 front 2024-03-18T15:29:39.238431+0000 (oldest deadline 2024-03-18T15:30:05.138032+0000) 2024-03-18T15:30:11.465 INFO:journalctl@ceph.osd.3.smithi019.stdout:Mar 18 15:30:11 smithi019 podman[92713]: 2024-03-18 15:30:11.297893306 +0000 UTC m=+0.567509039 container init 3fcf0b95174322f840468ecce17cb25dcafd3679b82d4d439ff237ecef1129b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-3-deactivate, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-03-18T15:30:11.466 INFO:journalctl@ceph.osd.3.smithi019.stdout:Mar 18 15:30:11 smithi019 podman[92713]: 2024-03-18 15:30:11.30898442 +0000 UTC m=+0.578600139 container start 3fcf0b95174322f840468ecce17cb25dcafd3679b82d4d439ff237ecef1129b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-3-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-03-18T15:30:11.466 INFO:journalctl@ceph.osd.3.smithi019.stdout:Mar 18 15:30:11 smithi019 podman[92713]: 2024-03-18 15:30:11.404754783 +0000 UTC m=+0.674370517 container attach 3fcf0b95174322f840468ecce17cb25dcafd3679b82d4d439ff237ecef1129b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-3-deactivate, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-18T15:30:11.891 INFO:journalctl@ceph.osd.3.smithi019.stdout:Mar 18 15:30:11 smithi019 podman[92713]: 2024-03-18 15:30:11.785529275 +0000 UTC m=+1.055144992 container died 3fcf0b95174322f840468ecce17cb25dcafd3679b82d4d439ff237ecef1129b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-3-deactivate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0) 2024-03-18T15:30:12.194 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:11 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:11.984+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:41.404783+0000 front 2024-03-18T15:29:41.404661+0000 (oldest deadline 2024-03-18T15:30:04.304572+0000) 2024-03-18T15:30:12.194 INFO:journalctl@ceph.osd.5.smithi104.stdout:Mar 18 15:30:11 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-5[48148]: 2024-03-18T15:30:11.877+0000 7f6ec6d5c700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:40.553008+0000 front 2024-03-18T15:29:40.552917+0000 (oldest deadline 2024-03-18T15:30:04.052932+0000) 2024-03-18T15:30:12.660 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:12 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:12.285+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:39.237875+0000 front 2024-03-18T15:29:39.238431+0000 (oldest deadline 2024-03-18T15:30:05.138032+0000) 2024-03-18T15:30:12.696 DEBUG:teuthology.orchestra.run.smithi019:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@osd.3.service' 2024-03-18T15:30:12.808 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T15:30:12.808 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-18T15:30:12.808 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-18T15:30:12.808 DEBUG:teuthology.orchestra.run.smithi033:> sudo systemctl stop ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@osd.4 2024-03-18T15:30:13.148 INFO:journalctl@ceph.osd.4.smithi033.stdout:Mar 18 15:30:12 smithi033 systemd[1]: Stopping Ceph osd.4 for d4dc2ba6-e538-11ee-95c9-87774f69a715... 2024-03-18T15:30:13.194 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:12 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:12.972+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:41.404783+0000 front 2024-03-18T15:29:41.404661+0000 (oldest deadline 2024-03-18T15:30:04.304572+0000) 2024-03-18T15:30:13.194 INFO:journalctl@ceph.osd.5.smithi104.stdout:Mar 18 15:30:12 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-5[48148]: 2024-03-18T15:30:12.879+0000 7f6ec6d5c700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:40.553008+0000 front 2024-03-18T15:29:40.552917+0000 (oldest deadline 2024-03-18T15:30:04.052932+0000) 2024-03-18T15:30:13.507 INFO:journalctl@ceph.osd.4.smithi033.stdout:Mar 18 15:30:13 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-4[47810]: 2024-03-18T15:30:13.146+0000 7fe6d732e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T15:30:13.507 INFO:journalctl@ceph.osd.4.smithi033.stdout:Mar 18 15:30:13 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-4[47810]: 2024-03-18T15:30:13.146+0000 7fe6d732e700 -1 osd.4 109 *** Got signal Terminated *** 2024-03-18T15:30:13.507 INFO:journalctl@ceph.osd.4.smithi033.stdout:Mar 18 15:30:13 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-4[47810]: 2024-03-18T15:30:13.146+0000 7fe6d732e700 -1 osd.4 109 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T15:30:13.694 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:13 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:13.387+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:49.191521+0000 front 2024-03-18T15:29:49.191660+0000 (oldest deadline 2024-03-18T15:30:12.691483+0000) 2024-03-18T15:30:13.715 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:13 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:13.256+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:39.237875+0000 front 2024-03-18T15:29:39.238431+0000 (oldest deadline 2024-03-18T15:30:05.138032+0000) 2024-03-18T15:30:14.194 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:13 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:13.987+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:41.404783+0000 front 2024-03-18T15:29:41.404661+0000 (oldest deadline 2024-03-18T15:30:04.304572+0000) 2024-03-18T15:30:14.194 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:13 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:13.987+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.207053+0000 front 2024-03-18T15:29:51.206898+0000 (oldest deadline 2024-03-18T15:30:13.507003+0000) 2024-03-18T15:30:14.195 INFO:journalctl@ceph.osd.5.smithi104.stdout:Mar 18 15:30:13 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-5[48148]: 2024-03-18T15:30:13.918+0000 7f6ec6d5c700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:40.553008+0000 front 2024-03-18T15:29:40.552917+0000 (oldest deadline 2024-03-18T15:30:04.052932+0000) 2024-03-18T15:30:14.195 INFO:journalctl@ceph.osd.5.smithi104.stdout:Mar 18 15:30:13 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-5[48148]: 2024-03-18T15:30:13.918+0000 7f6ec6d5c700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:47.955561+0000 front 2024-03-18T15:29:47.955477+0000 (oldest deadline 2024-03-18T15:30:13.255470+0000) 2024-03-18T15:30:14.256 INFO:journalctl@ceph.osd.4.smithi033.stdout:Mar 18 15:30:13 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-4[47810]: 2024-03-18T15:30:13.953+0000 7fe6cf91a700 -1 osd.4 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:48.333651+0000 front 2024-03-18T15:29:48.333562+0000 (oldest deadline 2024-03-18T15:30:13.033781+0000) 2024-03-18T15:30:14.465 INFO:journalctl@ceph.osd.6.smithi019.stdout:Mar 18 15:30:14 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-6[59600]: 2024-03-18T15:30:14.180+0000 7fde4d400700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:52.056785+0000 front 2024-03-18T15:29:52.056368+0000 (oldest deadline 2024-03-18T15:30:13.756023+0000) 2024-03-18T15:30:14.465 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:14 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:14.241+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:39.237875+0000 front 2024-03-18T15:29:39.238431+0000 (oldest deadline 2024-03-18T15:30:05.138032+0000) 2024-03-18T15:30:14.466 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:14 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:14.241+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:50.743126+0000 front 2024-03-18T15:29:50.743226+0000 (oldest deadline 2024-03-18T15:30:13.643377+0000) 2024-03-18T15:30:14.694 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:14 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:14.394+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:49.191521+0000 front 2024-03-18T15:29:49.191660+0000 (oldest deadline 2024-03-18T15:30:12.691483+0000) 2024-03-18T15:30:15.194 INFO:journalctl@ceph.osd.5.smithi104.stdout:Mar 18 15:30:14 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-5[48148]: 2024-03-18T15:30:14.906+0000 7f6ec6d5c700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:40.553008+0000 front 2024-03-18T15:29:40.552917+0000 (oldest deadline 2024-03-18T15:30:04.052932+0000) 2024-03-18T15:30:15.194 INFO:journalctl@ceph.osd.5.smithi104.stdout:Mar 18 15:30:14 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-5[48148]: 2024-03-18T15:30:14.906+0000 7f6ec6d5c700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:47.955561+0000 front 2024-03-18T15:29:47.955477+0000 (oldest deadline 2024-03-18T15:30:13.255470+0000) 2024-03-18T15:30:15.195 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:14 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:14.996+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:41.404783+0000 front 2024-03-18T15:29:41.404661+0000 (oldest deadline 2024-03-18T15:30:04.304572+0000) 2024-03-18T15:30:15.195 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:14 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:14.996+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.207053+0000 front 2024-03-18T15:29:51.206898+0000 (oldest deadline 2024-03-18T15:30:13.507003+0000) 2024-03-18T15:30:15.257 INFO:journalctl@ceph.osd.4.smithi033.stdout:Mar 18 15:30:15 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-4[47810]: 2024-03-18T15:30:15.000+0000 7fe6cf91a700 -1 osd.4 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:48.333651+0000 front 2024-03-18T15:29:48.333562+0000 (oldest deadline 2024-03-18T15:30:13.033781+0000) 2024-03-18T15:30:15.465 INFO:journalctl@ceph.osd.6.smithi019.stdout:Mar 18 15:30:15 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-6[59600]: 2024-03-18T15:30:15.179+0000 7fde4d400700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:52.056785+0000 front 2024-03-18T15:29:52.056368+0000 (oldest deadline 2024-03-18T15:30:13.756023+0000) 2024-03-18T15:30:15.465 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:15 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:15.264+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:39.237875+0000 front 2024-03-18T15:29:39.238431+0000 (oldest deadline 2024-03-18T15:30:05.138032+0000) 2024-03-18T15:30:15.465 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:15 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:15.264+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:50.743126+0000 front 2024-03-18T15:29:50.743226+0000 (oldest deadline 2024-03-18T15:30:13.643377+0000) 2024-03-18T15:30:15.694 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:15 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:15.415+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:49.191521+0000 front 2024-03-18T15:29:49.191660+0000 (oldest deadline 2024-03-18T15:30:12.691483+0000) 2024-03-18T15:30:15.979 INFO:journalctl@ceph.osd.7.smithi033.stdout:Mar 18 15:30:15 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-7[54818]: 2024-03-18T15:30:15.675+0000 7f480d187700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:50.578091+0000 front 2024-03-18T15:29:50.578050+0000 (oldest deadline 2024-03-18T15:30:15.278156+0000) 2024-03-18T15:30:16.194 INFO:journalctl@ceph.osd.5.smithi104.stdout:Mar 18 15:30:15 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-5[48148]: 2024-03-18T15:30:15.880+0000 7f6ec6d5c700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:40.553008+0000 front 2024-03-18T15:29:40.552917+0000 (oldest deadline 2024-03-18T15:30:04.052932+0000) 2024-03-18T15:30:16.194 INFO:journalctl@ceph.osd.5.smithi104.stdout:Mar 18 15:30:15 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-5[48148]: 2024-03-18T15:30:15.880+0000 7f6ec6d5c700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:47.955561+0000 front 2024-03-18T15:29:47.955477+0000 (oldest deadline 2024-03-18T15:30:13.255470+0000) 2024-03-18T15:30:16.195 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:16 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:16.018+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:41.404783+0000 front 2024-03-18T15:29:41.404661+0000 (oldest deadline 2024-03-18T15:30:04.304572+0000) 2024-03-18T15:30:16.195 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:16 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:16.018+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.207053+0000 front 2024-03-18T15:29:51.206898+0000 (oldest deadline 2024-03-18T15:30:13.507003+0000) 2024-03-18T15:30:16.236 INFO:journalctl@ceph.osd.4.smithi033.stdout:Mar 18 15:30:15 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-4[47810]: 2024-03-18T15:30:15.978+0000 7fe6cf91a700 -1 osd.4 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:48.333651+0000 front 2024-03-18T15:29:48.333562+0000 (oldest deadline 2024-03-18T15:30:13.033781+0000) 2024-03-18T15:30:16.465 INFO:journalctl@ceph.osd.6.smithi019.stdout:Mar 18 15:30:16 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-6[59600]: 2024-03-18T15:30:16.144+0000 7fde4d400700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:52.056785+0000 front 2024-03-18T15:29:52.056368+0000 (oldest deadline 2024-03-18T15:30:13.756023+0000) 2024-03-18T15:30:16.466 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:16 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:16.275+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:39.237875+0000 front 2024-03-18T15:29:39.238431+0000 (oldest deadline 2024-03-18T15:30:05.138032+0000) 2024-03-18T15:30:16.466 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:16 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:16.275+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:50.743126+0000 front 2024-03-18T15:29:50.743226+0000 (oldest deadline 2024-03-18T15:30:13.643377+0000) 2024-03-18T15:30:16.694 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:16 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:16.421+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:49.191521+0000 front 2024-03-18T15:29:49.191660+0000 (oldest deadline 2024-03-18T15:30:12.691483+0000) 2024-03-18T15:30:16.955 INFO:journalctl@ceph.osd.7.smithi033.stdout:Mar 18 15:30:16 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-7[54818]: 2024-03-18T15:30:16.688+0000 7f480d187700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:50.578091+0000 front 2024-03-18T15:29:50.578050+0000 (oldest deadline 2024-03-18T15:30:15.278156+0000) 2024-03-18T15:30:17.197 INFO:journalctl@ceph.osd.5.smithi104.stdout:Mar 18 15:30:16 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-5[48148]: 2024-03-18T15:30:16.930+0000 7f6ec6d5c700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:40.553008+0000 front 2024-03-18T15:29:40.552917+0000 (oldest deadline 2024-03-18T15:30:04.052932+0000) 2024-03-18T15:30:17.197 INFO:journalctl@ceph.osd.5.smithi104.stdout:Mar 18 15:30:16 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-5[48148]: 2024-03-18T15:30:16.930+0000 7f6ec6d5c700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:47.955561+0000 front 2024-03-18T15:29:47.955477+0000 (oldest deadline 2024-03-18T15:30:13.255470+0000) 2024-03-18T15:30:17.198 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:16 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:16.996+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:41.404783+0000 front 2024-03-18T15:29:41.404661+0000 (oldest deadline 2024-03-18T15:30:04.304572+0000) 2024-03-18T15:30:17.198 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:16 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:16.996+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.207053+0000 front 2024-03-18T15:29:51.206898+0000 (oldest deadline 2024-03-18T15:30:13.507003+0000) 2024-03-18T15:30:17.256 INFO:journalctl@ceph.osd.4.smithi033.stdout:Mar 18 15:30:16 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-4[47810]: 2024-03-18T15:30:16.954+0000 7fe6cf91a700 -1 osd.4 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:48.333651+0000 front 2024-03-18T15:29:48.333562+0000 (oldest deadline 2024-03-18T15:30:13.033781+0000) 2024-03-18T15:30:17.465 INFO:journalctl@ceph.osd.6.smithi019.stdout:Mar 18 15:30:17 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-6[59600]: 2024-03-18T15:30:17.167+0000 7fde4d400700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:52.056785+0000 front 2024-03-18T15:29:52.056368+0000 (oldest deadline 2024-03-18T15:30:13.756023+0000) 2024-03-18T15:30:17.465 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:17 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:17.253+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:39.237875+0000 front 2024-03-18T15:29:39.238431+0000 (oldest deadline 2024-03-18T15:30:05.138032+0000) 2024-03-18T15:30:17.465 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:17 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:17.253+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:50.743126+0000 front 2024-03-18T15:29:50.743226+0000 (oldest deadline 2024-03-18T15:30:13.643377+0000) 2024-03-18T15:30:17.694 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:17 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:17.415+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:49.191521+0000 front 2024-03-18T15:29:49.191660+0000 (oldest deadline 2024-03-18T15:30:12.691483+0000) 2024-03-18T15:30:18.003 INFO:journalctl@ceph.osd.7.smithi033.stdout:Mar 18 15:30:17 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-7[54818]: 2024-03-18T15:30:17.726+0000 7f480d187700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:50.578091+0000 front 2024-03-18T15:29:50.578050+0000 (oldest deadline 2024-03-18T15:30:15.278156+0000) 2024-03-18T15:30:18.257 INFO:journalctl@ceph.osd.4.smithi033.stdout:Mar 18 15:30:18 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-4[47810]: 2024-03-18T15:30:18.002+0000 7fe6cf91a700 -1 osd.4 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:48.333651+0000 front 2024-03-18T15:29:48.333562+0000 (oldest deadline 2024-03-18T15:30:13.033781+0000) 2024-03-18T15:30:18.257 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:18 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:18.166+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.612085+0000 front 2024-03-18T15:29:51.612164+0000 (oldest deadline 2024-03-18T15:30:17.512254+0000) 2024-03-18T15:30:18.413 INFO:journalctl@ceph.osd.5.smithi104.stdout:Mar 18 15:30:17 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-5[48148]: 2024-03-18T15:30:17.960+0000 7f6ec6d5c700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:40.553008+0000 front 2024-03-18T15:29:40.552917+0000 (oldest deadline 2024-03-18T15:30:04.052932+0000) 2024-03-18T15:30:18.413 INFO:journalctl@ceph.osd.5.smithi104.stdout:Mar 18 15:30:17 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-5[48148]: 2024-03-18T15:30:17.960+0000 7f6ec6d5c700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:47.955561+0000 front 2024-03-18T15:29:47.955477+0000 (oldest deadline 2024-03-18T15:30:13.255470+0000) 2024-03-18T15:30:18.413 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:17 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:17.973+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:41.404783+0000 front 2024-03-18T15:29:41.404661+0000 (oldest deadline 2024-03-18T15:30:04.304572+0000) 2024-03-18T15:30:18.413 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:17 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:17.973+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.207053+0000 front 2024-03-18T15:29:51.206898+0000 (oldest deadline 2024-03-18T15:30:13.507003+0000) 2024-03-18T15:30:18.465 INFO:journalctl@ceph.osd.6.smithi019.stdout:Mar 18 15:30:18 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-6[59600]: 2024-03-18T15:30:18.126+0000 7fde4d400700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:52.056785+0000 front 2024-03-18T15:29:52.056368+0000 (oldest deadline 2024-03-18T15:30:13.756023+0000) 2024-03-18T15:30:18.465 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:18 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:18.250+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:39.237875+0000 front 2024-03-18T15:29:39.238431+0000 (oldest deadline 2024-03-18T15:30:05.138032+0000) 2024-03-18T15:30:18.465 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:18 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:18.250+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:50.743126+0000 front 2024-03-18T15:29:50.743226+0000 (oldest deadline 2024-03-18T15:30:13.643377+0000) 2024-03-18T15:30:18.694 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:18 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:18.412+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:49.191521+0000 front 2024-03-18T15:29:49.191660+0000 (oldest deadline 2024-03-18T15:30:12.691483+0000) 2024-03-18T15:30:18.948 INFO:journalctl@ceph.osd.4.smithi033.stdout:Mar 18 15:30:18 smithi033 podman[74200]: 2024-03-18 15:30:18.631416508 +0000 UTC m=+5.613301789 container died e70bbf90c568763a104598393ffbcbca1b2313de06135c401d07813c47999f02 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-4, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, org.label-schema.build-date=20240307) 2024-03-18T15:30:18.948 INFO:journalctl@ceph.osd.7.smithi033.stdout:Mar 18 15:30:18 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-7[54818]: 2024-03-18T15:30:18.754+0000 7f480d187700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:50.578091+0000 front 2024-03-18T15:29:50.578050+0000 (oldest deadline 2024-03-18T15:30:15.278156+0000) 2024-03-18T15:30:19.256 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:19 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:19.134+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.612085+0000 front 2024-03-18T15:29:51.612164+0000 (oldest deadline 2024-03-18T15:30:17.512254+0000) 2024-03-18T15:30:19.257 INFO:journalctl@ceph.osd.4.smithi033.stdout:Mar 18 15:30:19 smithi033 podman[74200]: 2024-03-18 15:30:19.047295948 +0000 UTC m=+6.029181236 container cleanup e70bbf90c568763a104598393ffbcbca1b2313de06135c401d07813c47999f02 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-4, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-18T15:30:19.257 INFO:journalctl@ceph.osd.4.smithi033.stdout:Mar 18 15:30:19 smithi033 bash[74200]: ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-4 2024-03-18T15:30:19.409 INFO:journalctl@ceph.osd.5.smithi104.stdout:Mar 18 15:30:19 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-5[48148]: 2024-03-18T15:30:19.006+0000 7f6ec6d5c700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:40.553008+0000 front 2024-03-18T15:29:40.552917+0000 (oldest deadline 2024-03-18T15:30:04.052932+0000) 2024-03-18T15:30:19.409 INFO:journalctl@ceph.osd.5.smithi104.stdout:Mar 18 15:30:19 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-5[48148]: 2024-03-18T15:30:19.006+0000 7f6ec6d5c700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:47.955561+0000 front 2024-03-18T15:29:47.955477+0000 (oldest deadline 2024-03-18T15:30:13.255470+0000) 2024-03-18T15:30:19.409 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:18 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:18.961+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:41.404783+0000 front 2024-03-18T15:29:41.404661+0000 (oldest deadline 2024-03-18T15:30:04.304572+0000) 2024-03-18T15:30:19.409 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:18 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:18.961+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.207053+0000 front 2024-03-18T15:29:51.206898+0000 (oldest deadline 2024-03-18T15:30:13.507003+0000) 2024-03-18T15:30:19.465 INFO:journalctl@ceph.osd.6.smithi019.stdout:Mar 18 15:30:19 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-6[59600]: 2024-03-18T15:30:19.077+0000 7fde4d400700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:52.056785+0000 front 2024-03-18T15:29:52.056368+0000 (oldest deadline 2024-03-18T15:30:13.756023+0000) 2024-03-18T15:30:19.465 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:19 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:19.264+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:39.237875+0000 front 2024-03-18T15:29:39.238431+0000 (oldest deadline 2024-03-18T15:30:05.138032+0000) 2024-03-18T15:30:19.465 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:19 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:19.264+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:50.743126+0000 front 2024-03-18T15:29:50.743226+0000 (oldest deadline 2024-03-18T15:30:13.643377+0000) 2024-03-18T15:30:19.673 INFO:journalctl@ceph.osd.4.smithi033.stdout:Mar 18 15:30:19 smithi033 podman[74218]: 2024-03-18 15:30:19.265444111 +0000 UTC m=+0.636561004 container remove e70bbf90c568763a104598393ffbcbca1b2313de06135c401d07813c47999f02 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-4, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-18T15:30:19.694 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:19 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:19.408+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:49.191521+0000 front 2024-03-18T15:29:49.191660+0000 (oldest deadline 2024-03-18T15:30:12.691483+0000) 2024-03-18T15:30:19.929 INFO:journalctl@ceph.osd.7.smithi033.stdout:Mar 18 15:30:19 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-7[54818]: 2024-03-18T15:30:19.801+0000 7f480d187700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:50.578091+0000 front 2024-03-18T15:29:50.578050+0000 (oldest deadline 2024-03-18T15:30:15.278156+0000) 2024-03-18T15:30:19.930 INFO:journalctl@ceph.osd.4.smithi033.stdout:Mar 18 15:30:19 smithi033 podman[74307]: 2024-03-18 15:30:19.576682538 +0000 UTC m=+0.018032676 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T15:30:19.930 INFO:journalctl@ceph.osd.4.smithi033.stdout:Mar 18 15:30:19 smithi033 podman[74307]: 2024-03-18 15:30:19.683173479 +0000 UTC m=+0.124523642 container create 66ae6a62435a822ca03233609c57fb30eb4e708cd6ad20a62da9a41e37c22361 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-4-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-18T15:30:20.230 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:20 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:20.142+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.612085+0000 front 2024-03-18T15:29:51.612164+0000 (oldest deadline 2024-03-18T15:30:17.512254+0000) 2024-03-18T15:30:20.231 INFO:journalctl@ceph.osd.4.smithi033.stdout:Mar 18 15:30:19 smithi033 podman[74307]: 2024-03-18 15:30:19.928129231 +0000 UTC m=+0.369479344 container init 66ae6a62435a822ca03233609c57fb30eb4e708cd6ad20a62da9a41e37c22361 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-4-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-03-18T15:30:20.231 INFO:journalctl@ceph.osd.4.smithi033.stdout:Mar 18 15:30:19 smithi033 podman[74307]: 2024-03-18 15:30:19.931561575 +0000 UTC m=+0.372911696 container start 66ae6a62435a822ca03233609c57fb30eb4e708cd6ad20a62da9a41e37c22361 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-4-deactivate, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-03-18T15:30:20.231 INFO:journalctl@ceph.osd.4.smithi033.stdout:Mar 18 15:30:20 smithi033 podman[74307]: 2024-03-18 15:30:20.035780622 +0000 UTC m=+0.477130745 container attach 66ae6a62435a822ca03233609c57fb30eb4e708cd6ad20a62da9a41e37c22361 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-4-deactivate, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True) 2024-03-18T15:30:20.444 INFO:journalctl@ceph.osd.5.smithi104.stdout:Mar 18 15:30:19 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-5[48148]: 2024-03-18T15:30:19.984+0000 7f6ec6d5c700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:40.553008+0000 front 2024-03-18T15:29:40.552917+0000 (oldest deadline 2024-03-18T15:30:04.052932+0000) 2024-03-18T15:30:20.444 INFO:journalctl@ceph.osd.5.smithi104.stdout:Mar 18 15:30:19 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-5[48148]: 2024-03-18T15:30:19.984+0000 7f6ec6d5c700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:47.955561+0000 front 2024-03-18T15:29:47.955477+0000 (oldest deadline 2024-03-18T15:30:13.255470+0000) 2024-03-18T15:30:20.444 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:20 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:20.009+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:41.404783+0000 front 2024-03-18T15:29:41.404661+0000 (oldest deadline 2024-03-18T15:30:04.304572+0000) 2024-03-18T15:30:20.445 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:20 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:20.009+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.207053+0000 front 2024-03-18T15:29:51.206898+0000 (oldest deadline 2024-03-18T15:30:13.507003+0000) 2024-03-18T15:30:20.465 INFO:journalctl@ceph.osd.6.smithi019.stdout:Mar 18 15:30:20 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-6[59600]: 2024-03-18T15:30:20.118+0000 7fde4d400700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:52.056785+0000 front 2024-03-18T15:29:52.056368+0000 (oldest deadline 2024-03-18T15:30:13.756023+0000) 2024-03-18T15:30:20.465 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:20 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:20.276+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:39.237875+0000 front 2024-03-18T15:29:39.238431+0000 (oldest deadline 2024-03-18T15:30:05.138032+0000) 2024-03-18T15:30:20.466 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:20 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:20.276+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:50.743126+0000 front 2024-03-18T15:29:50.743226+0000 (oldest deadline 2024-03-18T15:30:13.643377+0000) 2024-03-18T15:30:20.507 INFO:journalctl@ceph.osd.4.smithi033.stdout:Mar 18 15:30:20 smithi033 podman[74307]: 2024-03-18 15:30:20.231401661 +0000 UTC m=+0.672751777 container died 66ae6a62435a822ca03233609c57fb30eb4e708cd6ad20a62da9a41e37c22361 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-4-deactivate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-18T15:30:20.763 DEBUG:teuthology.orchestra.run.smithi033:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@osd.4.service' 2024-03-18T15:30:20.792 INFO:journalctl@ceph.osd.7.smithi033.stdout:Mar 18 15:30:20 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-7[54818]: 2024-03-18T15:30:20.761+0000 7f480d187700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:50.578091+0000 front 2024-03-18T15:29:50.578050+0000 (oldest deadline 2024-03-18T15:30:15.278156+0000) 2024-03-18T15:30:20.793 INFO:journalctl@ceph.osd.4.smithi033.stdout:Mar 18 15:30:20 smithi033 podman[74334]: 2024-03-18 15:30:20.748207388 +0000 UTC m=+0.511932307 container remove 66ae6a62435a822ca03233609c57fb30eb4e708cd6ad20a62da9a41e37c22361 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-4-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T15:30:20.793 INFO:journalctl@ceph.osd.4.smithi033.stdout:Mar 18 15:30:20 smithi033 systemd[1]: ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@osd.4.service: Deactivated successfully. 2024-03-18T15:30:20.793 INFO:journalctl@ceph.osd.4.smithi033.stdout:Mar 18 15:30:20 smithi033 systemd[1]: Stopped Ceph osd.4 for d4dc2ba6-e538-11ee-95c9-87774f69a715. 2024-03-18T15:30:20.793 INFO:journalctl@ceph.osd.4.smithi033.stdout:Mar 18 15:30:20 smithi033 systemd[1]: ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@osd.4.service: Consumed 18.460s CPU time. 2024-03-18T15:30:20.808 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T15:30:20.809 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-18T15:30:20.809 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-18T15:30:20.809 DEBUG:teuthology.orchestra.run.smithi104:> sudo systemctl stop ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@osd.5 2024-03-18T15:30:20.838 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:20 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:20.456+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:49.191521+0000 front 2024-03-18T15:29:49.191660+0000 (oldest deadline 2024-03-18T15:30:12.691483+0000) 2024-03-18T15:30:21.130 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:21 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:21.018+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:41.404783+0000 front 2024-03-18T15:29:41.404661+0000 (oldest deadline 2024-03-18T15:30:04.304572+0000) 2024-03-18T15:30:21.131 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:21 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:21.018+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.207053+0000 front 2024-03-18T15:29:51.206898+0000 (oldest deadline 2024-03-18T15:30:13.507003+0000) 2024-03-18T15:30:21.131 INFO:journalctl@ceph.osd.5.smithi104.stdout:Mar 18 15:30:20 smithi104 systemd[1]: Stopping Ceph osd.5 for d4dc2ba6-e538-11ee-95c9-87774f69a715... 2024-03-18T15:30:21.131 INFO:journalctl@ceph.osd.5.smithi104.stdout:Mar 18 15:30:20 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-5[48148]: 2024-03-18T15:30:20.992+0000 7f6ec6d5c700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:40.553008+0000 front 2024-03-18T15:29:40.552917+0000 (oldest deadline 2024-03-18T15:30:04.052932+0000) 2024-03-18T15:30:21.131 INFO:journalctl@ceph.osd.5.smithi104.stdout:Mar 18 15:30:20 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-5[48148]: 2024-03-18T15:30:20.992+0000 7f6ec6d5c700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:47.955561+0000 front 2024-03-18T15:29:47.955477+0000 (oldest deadline 2024-03-18T15:30:13.255470+0000) 2024-03-18T15:30:21.411 INFO:journalctl@ceph.osd.5.smithi104.stdout:Mar 18 15:30:21 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-5[48148]: 2024-03-18T15:30:21.129+0000 7f6eca768700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T15:30:21.411 INFO:journalctl@ceph.osd.5.smithi104.stdout:Mar 18 15:30:21 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-5[48148]: 2024-03-18T15:30:21.129+0000 7f6eca768700 -1 osd.5 109 *** Got signal Terminated *** 2024-03-18T15:30:21.412 INFO:journalctl@ceph.osd.5.smithi104.stdout:Mar 18 15:30:21 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-5[48148]: 2024-03-18T15:30:21.129+0000 7f6eca768700 -1 osd.5 109 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T15:30:21.465 INFO:journalctl@ceph.osd.6.smithi019.stdout:Mar 18 15:30:21 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-6[59600]: 2024-03-18T15:30:21.095+0000 7fde4d400700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:52.056785+0000 front 2024-03-18T15:29:52.056368+0000 (oldest deadline 2024-03-18T15:30:13.756023+0000) 2024-03-18T15:30:21.465 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:21 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:21.307+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:39.237875+0000 front 2024-03-18T15:29:39.238431+0000 (oldest deadline 2024-03-18T15:30:05.138032+0000) 2024-03-18T15:30:21.465 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:21 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:21.307+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:50.743126+0000 front 2024-03-18T15:29:50.743226+0000 (oldest deadline 2024-03-18T15:30:13.643377+0000) 2024-03-18T15:30:21.506 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:21 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:21.170+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.612085+0000 front 2024-03-18T15:29:51.612164+0000 (oldest deadline 2024-03-18T15:30:17.512254+0000) 2024-03-18T15:30:21.694 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:21 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:21.411+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:49.191521+0000 front 2024-03-18T15:29:49.191660+0000 (oldest deadline 2024-03-18T15:30:12.691483+0000) 2024-03-18T15:30:22.155 INFO:journalctl@ceph.osd.7.smithi033.stdout:Mar 18 15:30:21 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-7[54818]: 2024-03-18T15:30:21.785+0000 7f480d187700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:50.578091+0000 front 2024-03-18T15:29:50.578050+0000 (oldest deadline 2024-03-18T15:30:15.278156+0000) 2024-03-18T15:30:22.444 INFO:journalctl@ceph.osd.5.smithi104.stdout:Mar 18 15:30:21 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-5[48148]: 2024-03-18T15:30:21.985+0000 7f6ec6d5c700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:40.553008+0000 front 2024-03-18T15:29:40.552917+0000 (oldest deadline 2024-03-18T15:30:04.052932+0000) 2024-03-18T15:30:22.444 INFO:journalctl@ceph.osd.5.smithi104.stdout:Mar 18 15:30:21 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-5[48148]: 2024-03-18T15:30:21.985+0000 7f6ec6d5c700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:47.955561+0000 front 2024-03-18T15:29:47.955477+0000 (oldest deadline 2024-03-18T15:30:13.255470+0000) 2024-03-18T15:30:22.445 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:22 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:22.011+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:41.404783+0000 front 2024-03-18T15:29:41.404661+0000 (oldest deadline 2024-03-18T15:30:04.304572+0000) 2024-03-18T15:30:22.445 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:22 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:22.011+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.207053+0000 front 2024-03-18T15:29:51.206898+0000 (oldest deadline 2024-03-18T15:30:13.507003+0000) 2024-03-18T15:30:22.445 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:22 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:22.011+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:59.809088+0000 front 2024-03-18T15:29:59.809253+0000 (oldest deadline 2024-03-18T15:30:21.509244+0000) 2024-03-18T15:30:22.465 INFO:journalctl@ceph.osd.6.smithi019.stdout:Mar 18 15:30:22 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-6[59600]: 2024-03-18T15:30:22.101+0000 7fde4d400700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:52.056785+0000 front 2024-03-18T15:29:52.056368+0000 (oldest deadline 2024-03-18T15:30:13.756023+0000) 2024-03-18T15:30:22.466 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:22 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:22.272+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:39.237875+0000 front 2024-03-18T15:29:39.238431+0000 (oldest deadline 2024-03-18T15:30:05.138032+0000) 2024-03-18T15:30:22.466 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:22 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:22.272+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:50.743126+0000 front 2024-03-18T15:29:50.743226+0000 (oldest deadline 2024-03-18T15:30:13.643377+0000) 2024-03-18T15:30:22.466 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:22 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:22.272+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:56.947757+0000 front 2024-03-18T15:29:56.947820+0000 (oldest deadline 2024-03-18T15:30:22.248013+0000) 2024-03-18T15:30:22.506 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:22 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:22.155+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.612085+0000 front 2024-03-18T15:29:51.612164+0000 (oldest deadline 2024-03-18T15:30:17.512254+0000) 2024-03-18T15:30:22.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:22 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:22.155+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:57.512824+0000 front 2024-03-18T15:29:57.512967+0000 (oldest deadline 2024-03-18T15:30:21.612923+0000) 2024-03-18T15:30:22.944 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:22 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:22.443+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:49.191521+0000 front 2024-03-18T15:29:49.191660+0000 (oldest deadline 2024-03-18T15:30:12.691483+0000) 2024-03-18T15:30:22.944 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:22 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:22.443+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:59.092785+0000 front 2024-03-18T15:29:59.092921+0000 (oldest deadline 2024-03-18T15:30:21.992976+0000) 2024-03-18T15:30:23.158 INFO:journalctl@ceph.osd.7.smithi033.stdout:Mar 18 15:30:22 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-7[54818]: 2024-03-18T15:30:22.805+0000 7f480d187700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:50.578091+0000 front 2024-03-18T15:29:50.578050+0000 (oldest deadline 2024-03-18T15:30:15.278156+0000) 2024-03-18T15:30:23.215 INFO:journalctl@ceph.osd.6.smithi019.stdout:Mar 18 15:30:23 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-6[59600]: 2024-03-18T15:30:23.084+0000 7fde4d400700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:52.056785+0000 front 2024-03-18T15:29:52.056368+0000 (oldest deadline 2024-03-18T15:30:13.756023+0000) 2024-03-18T15:30:23.215 INFO:journalctl@ceph.osd.6.smithi019.stdout:Mar 18 15:30:23 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-6[59600]: 2024-03-18T15:30:23.084+0000 7fde4d400700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:57.758902+0000 front 2024-03-18T15:29:57.759391+0000 (oldest deadline 2024-03-18T15:30:23.059128+0000) 2024-03-18T15:30:23.414 INFO:journalctl@ceph.osd.5.smithi104.stdout:Mar 18 15:30:22 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-5[48148]: 2024-03-18T15:30:22.994+0000 7f6ec6d5c700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:40.553008+0000 front 2024-03-18T15:29:40.552917+0000 (oldest deadline 2024-03-18T15:30:04.052932+0000) 2024-03-18T15:30:23.414 INFO:journalctl@ceph.osd.5.smithi104.stdout:Mar 18 15:30:22 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-5[48148]: 2024-03-18T15:30:22.994+0000 7f6ec6d5c700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:47.955561+0000 front 2024-03-18T15:29:47.955477+0000 (oldest deadline 2024-03-18T15:30:13.255470+0000) 2024-03-18T15:30:23.414 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:23 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:23.045+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:41.404783+0000 front 2024-03-18T15:29:41.404661+0000 (oldest deadline 2024-03-18T15:30:04.304572+0000) 2024-03-18T15:30:23.415 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:23 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:23.045+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.207053+0000 front 2024-03-18T15:29:51.206898+0000 (oldest deadline 2024-03-18T15:30:13.507003+0000) 2024-03-18T15:30:23.415 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:23 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:23.045+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:59.809088+0000 front 2024-03-18T15:29:59.809253+0000 (oldest deadline 2024-03-18T15:30:21.509244+0000) 2024-03-18T15:30:23.506 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:23 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:23.158+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.612085+0000 front 2024-03-18T15:29:51.612164+0000 (oldest deadline 2024-03-18T15:30:17.512254+0000) 2024-03-18T15:30:23.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:23 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:23.158+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:57.512824+0000 front 2024-03-18T15:29:57.512967+0000 (oldest deadline 2024-03-18T15:30:21.612923+0000) 2024-03-18T15:30:23.694 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:23 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:23.413+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:49.191521+0000 front 2024-03-18T15:29:49.191660+0000 (oldest deadline 2024-03-18T15:30:12.691483+0000) 2024-03-18T15:30:23.694 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:23 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:23.413+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:59.092785+0000 front 2024-03-18T15:29:59.092921+0000 (oldest deadline 2024-03-18T15:30:21.992976+0000) 2024-03-18T15:30:23.715 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:23 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:23.236+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:39.237875+0000 front 2024-03-18T15:29:39.238431+0000 (oldest deadline 2024-03-18T15:30:05.138032+0000) 2024-03-18T15:30:23.715 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:23 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:23.236+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:50.743126+0000 front 2024-03-18T15:29:50.743226+0000 (oldest deadline 2024-03-18T15:30:13.643377+0000) 2024-03-18T15:30:23.715 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:23 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:23.236+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:56.947757+0000 front 2024-03-18T15:29:56.947820+0000 (oldest deadline 2024-03-18T15:30:22.248013+0000) 2024-03-18T15:30:24.184 INFO:journalctl@ceph.osd.7.smithi033.stdout:Mar 18 15:30:23 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-7[54818]: 2024-03-18T15:30:23.800+0000 7f480d187700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:50.578091+0000 front 2024-03-18T15:29:50.578050+0000 (oldest deadline 2024-03-18T15:30:15.278156+0000) 2024-03-18T15:30:24.184 INFO:journalctl@ceph.osd.7.smithi033.stdout:Mar 18 15:30:23 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-7[54818]: 2024-03-18T15:30:23.800+0000 7f480d187700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:57.579372+0000 front 2024-03-18T15:29:57.579232+0000 (oldest deadline 2024-03-18T15:30:22.879178+0000) 2024-03-18T15:30:24.366 INFO:journalctl@ceph.osd.5.smithi104.stdout:Mar 18 15:30:23 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-5[48148]: 2024-03-18T15:30:23.955+0000 7f6ec6d5c700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:40.553008+0000 front 2024-03-18T15:29:40.552917+0000 (oldest deadline 2024-03-18T15:30:04.052932+0000) 2024-03-18T15:30:24.366 INFO:journalctl@ceph.osd.5.smithi104.stdout:Mar 18 15:30:23 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-5[48148]: 2024-03-18T15:30:23.955+0000 7f6ec6d5c700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:47.955561+0000 front 2024-03-18T15:29:47.955477+0000 (oldest deadline 2024-03-18T15:30:13.255470+0000) 2024-03-18T15:30:24.367 INFO:journalctl@ceph.osd.5.smithi104.stdout:Mar 18 15:30:23 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-5[48148]: 2024-03-18T15:30:23.955+0000 7f6ec6d5c700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:59.056937+0000 front 2024-03-18T15:29:59.056960+0000 (oldest deadline 2024-03-18T15:30:23.157090+0000) 2024-03-18T15:30:24.367 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:24 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:24.021+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:41.404783+0000 front 2024-03-18T15:29:41.404661+0000 (oldest deadline 2024-03-18T15:30:04.304572+0000) 2024-03-18T15:30:24.367 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:24 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:24.021+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.207053+0000 front 2024-03-18T15:29:51.206898+0000 (oldest deadline 2024-03-18T15:30:13.507003+0000) 2024-03-18T15:30:24.367 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:24 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:24.021+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:59.809088+0000 front 2024-03-18T15:29:59.809253+0000 (oldest deadline 2024-03-18T15:30:21.509244+0000) 2024-03-18T15:30:24.465 INFO:journalctl@ceph.osd.6.smithi019.stdout:Mar 18 15:30:24 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-6[59600]: 2024-03-18T15:30:24.094+0000 7fde4d400700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:52.056785+0000 front 2024-03-18T15:29:52.056368+0000 (oldest deadline 2024-03-18T15:30:13.756023+0000) 2024-03-18T15:30:24.465 INFO:journalctl@ceph.osd.6.smithi019.stdout:Mar 18 15:30:24 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-6[59600]: 2024-03-18T15:30:24.094+0000 7fde4d400700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:57.758902+0000 front 2024-03-18T15:29:57.759391+0000 (oldest deadline 2024-03-18T15:30:23.059128+0000) 2024-03-18T15:30:24.466 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:24 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:24.235+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:39.237875+0000 front 2024-03-18T15:29:39.238431+0000 (oldest deadline 2024-03-18T15:30:05.138032+0000) 2024-03-18T15:30:24.466 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:24 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:24.235+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:50.743126+0000 front 2024-03-18T15:29:50.743226+0000 (oldest deadline 2024-03-18T15:30:13.643377+0000) 2024-03-18T15:30:24.466 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:24 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:24.235+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:56.947757+0000 front 2024-03-18T15:29:56.947820+0000 (oldest deadline 2024-03-18T15:30:22.248013+0000) 2024-03-18T15:30:24.506 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:24 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:24.183+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.612085+0000 front 2024-03-18T15:29:51.612164+0000 (oldest deadline 2024-03-18T15:30:17.512254+0000) 2024-03-18T15:30:24.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:24 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:24.183+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:57.512824+0000 front 2024-03-18T15:29:57.512967+0000 (oldest deadline 2024-03-18T15:30:21.612923+0000) 2024-03-18T15:30:24.694 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:24 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:24.365+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:49.191521+0000 front 2024-03-18T15:29:49.191660+0000 (oldest deadline 2024-03-18T15:30:12.691483+0000) 2024-03-18T15:30:24.695 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:24 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:24.365+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:59.092785+0000 front 2024-03-18T15:29:59.092921+0000 (oldest deadline 2024-03-18T15:30:21.992976+0000) 2024-03-18T15:30:25.147 INFO:journalctl@ceph.osd.7.smithi033.stdout:Mar 18 15:30:24 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-7[54818]: 2024-03-18T15:30:24.834+0000 7f480d187700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:50.578091+0000 front 2024-03-18T15:29:50.578050+0000 (oldest deadline 2024-03-18T15:30:15.278156+0000) 2024-03-18T15:30:25.147 INFO:journalctl@ceph.osd.7.smithi033.stdout:Mar 18 15:30:24 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-7[54818]: 2024-03-18T15:30:24.834+0000 7f480d187700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:57.579372+0000 front 2024-03-18T15:29:57.579232+0000 (oldest deadline 2024-03-18T15:30:22.879178+0000) 2024-03-18T15:30:25.325 INFO:journalctl@ceph.osd.5.smithi104.stdout:Mar 18 15:30:24 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-5[48148]: 2024-03-18T15:30:24.950+0000 7f6ec6d5c700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:40.553008+0000 front 2024-03-18T15:29:40.552917+0000 (oldest deadline 2024-03-18T15:30:04.052932+0000) 2024-03-18T15:30:25.325 INFO:journalctl@ceph.osd.5.smithi104.stdout:Mar 18 15:30:24 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-5[48148]: 2024-03-18T15:30:24.950+0000 7f6ec6d5c700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:47.955561+0000 front 2024-03-18T15:29:47.955477+0000 (oldest deadline 2024-03-18T15:30:13.255470+0000) 2024-03-18T15:30:25.325 INFO:journalctl@ceph.osd.5.smithi104.stdout:Mar 18 15:30:24 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-5[48148]: 2024-03-18T15:30:24.950+0000 7f6ec6d5c700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:59.056937+0000 front 2024-03-18T15:29:59.056960+0000 (oldest deadline 2024-03-18T15:30:23.157090+0000) 2024-03-18T15:30:25.325 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:25 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:25.062+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:41.404783+0000 front 2024-03-18T15:29:41.404661+0000 (oldest deadline 2024-03-18T15:30:04.304572+0000) 2024-03-18T15:30:25.325 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:25 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:25.062+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.207053+0000 front 2024-03-18T15:29:51.206898+0000 (oldest deadline 2024-03-18T15:30:13.507003+0000) 2024-03-18T15:30:25.325 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:25 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:25.062+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:59.809088+0000 front 2024-03-18T15:29:59.809253+0000 (oldest deadline 2024-03-18T15:30:21.509244+0000) 2024-03-18T15:30:25.465 INFO:journalctl@ceph.osd.6.smithi019.stdout:Mar 18 15:30:25 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-6[59600]: 2024-03-18T15:30:25.068+0000 7fde4d400700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:52.056785+0000 front 2024-03-18T15:29:52.056368+0000 (oldest deadline 2024-03-18T15:30:13.756023+0000) 2024-03-18T15:30:25.465 INFO:journalctl@ceph.osd.6.smithi019.stdout:Mar 18 15:30:25 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-6[59600]: 2024-03-18T15:30:25.068+0000 7fde4d400700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:57.758902+0000 front 2024-03-18T15:29:57.759391+0000 (oldest deadline 2024-03-18T15:30:23.059128+0000) 2024-03-18T15:30:25.465 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:25 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:25.262+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:39.237875+0000 front 2024-03-18T15:29:39.238431+0000 (oldest deadline 2024-03-18T15:30:05.138032+0000) 2024-03-18T15:30:25.466 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:25 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:25.262+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:50.743126+0000 front 2024-03-18T15:29:50.743226+0000 (oldest deadline 2024-03-18T15:30:13.643377+0000) 2024-03-18T15:30:25.466 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:25 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:25.262+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:56.947757+0000 front 2024-03-18T15:29:56.947820+0000 (oldest deadline 2024-03-18T15:30:22.248013+0000) 2024-03-18T15:30:25.506 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:25 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:25.147+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.612085+0000 front 2024-03-18T15:29:51.612164+0000 (oldest deadline 2024-03-18T15:30:17.512254+0000) 2024-03-18T15:30:25.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:25 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:25.147+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:57.512824+0000 front 2024-03-18T15:29:57.512967+0000 (oldest deadline 2024-03-18T15:30:21.612923+0000) 2024-03-18T15:30:25.694 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:25 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:25.324+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:49.191521+0000 front 2024-03-18T15:29:49.191660+0000 (oldest deadline 2024-03-18T15:30:12.691483+0000) 2024-03-18T15:30:25.694 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:25 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:25.324+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:59.092785+0000 front 2024-03-18T15:29:59.092921+0000 (oldest deadline 2024-03-18T15:30:21.992976+0000) 2024-03-18T15:30:26.189 INFO:journalctl@ceph.osd.7.smithi033.stdout:Mar 18 15:30:25 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-7[54818]: 2024-03-18T15:30:25.862+0000 7f480d187700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:50.578091+0000 front 2024-03-18T15:29:50.578050+0000 (oldest deadline 2024-03-18T15:30:15.278156+0000) 2024-03-18T15:30:26.189 INFO:journalctl@ceph.osd.7.smithi033.stdout:Mar 18 15:30:25 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-7[54818]: 2024-03-18T15:30:25.862+0000 7f480d187700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:57.579372+0000 front 2024-03-18T15:29:57.579232+0000 (oldest deadline 2024-03-18T15:30:22.879178+0000) 2024-03-18T15:30:26.332 INFO:journalctl@ceph.osd.5.smithi104.stdout:Mar 18 15:30:25 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-5[48148]: 2024-03-18T15:30:25.990+0000 7f6ec6d5c700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:40.553008+0000 front 2024-03-18T15:29:40.552917+0000 (oldest deadline 2024-03-18T15:30:04.052932+0000) 2024-03-18T15:30:26.333 INFO:journalctl@ceph.osd.5.smithi104.stdout:Mar 18 15:30:25 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-5[48148]: 2024-03-18T15:30:25.990+0000 7f6ec6d5c700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:47.955561+0000 front 2024-03-18T15:29:47.955477+0000 (oldest deadline 2024-03-18T15:30:13.255470+0000) 2024-03-18T15:30:26.333 INFO:journalctl@ceph.osd.5.smithi104.stdout:Mar 18 15:30:25 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-5[48148]: 2024-03-18T15:30:25.990+0000 7f6ec6d5c700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:59.056937+0000 front 2024-03-18T15:29:59.056960+0000 (oldest deadline 2024-03-18T15:30:23.157090+0000) 2024-03-18T15:30:26.333 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:26 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:26.051+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:41.404783+0000 front 2024-03-18T15:29:41.404661+0000 (oldest deadline 2024-03-18T15:30:04.304572+0000) 2024-03-18T15:30:26.333 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:26 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:26.051+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.207053+0000 front 2024-03-18T15:29:51.206898+0000 (oldest deadline 2024-03-18T15:30:13.507003+0000) 2024-03-18T15:30:26.333 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:26 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:26.051+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:59.809088+0000 front 2024-03-18T15:29:59.809253+0000 (oldest deadline 2024-03-18T15:30:21.509244+0000) 2024-03-18T15:30:26.466 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:26 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:26.279+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:39.237875+0000 front 2024-03-18T15:29:39.238431+0000 (oldest deadline 2024-03-18T15:30:05.138032+0000) 2024-03-18T15:30:26.466 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:26 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:26.279+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:50.743126+0000 front 2024-03-18T15:29:50.743226+0000 (oldest deadline 2024-03-18T15:30:13.643377+0000) 2024-03-18T15:30:26.466 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:26 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:26.279+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:56.947757+0000 front 2024-03-18T15:29:56.947820+0000 (oldest deadline 2024-03-18T15:30:22.248013+0000) 2024-03-18T15:30:26.466 INFO:journalctl@ceph.osd.6.smithi019.stdout:Mar 18 15:30:26 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-6[59600]: 2024-03-18T15:30:26.085+0000 7fde4d400700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:52.056785+0000 front 2024-03-18T15:29:52.056368+0000 (oldest deadline 2024-03-18T15:30:13.756023+0000) 2024-03-18T15:30:26.466 INFO:journalctl@ceph.osd.6.smithi019.stdout:Mar 18 15:30:26 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-6[59600]: 2024-03-18T15:30:26.085+0000 7fde4d400700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:57.758902+0000 front 2024-03-18T15:29:57.759391+0000 (oldest deadline 2024-03-18T15:30:23.059128+0000) 2024-03-18T15:30:26.506 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:26 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:26.188+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.612085+0000 front 2024-03-18T15:29:51.612164+0000 (oldest deadline 2024-03-18T15:30:17.512254+0000) 2024-03-18T15:30:26.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:26 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:26.188+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:57.512824+0000 front 2024-03-18T15:29:57.512967+0000 (oldest deadline 2024-03-18T15:30:21.612923+0000) 2024-03-18T15:30:26.657 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:26 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:26.332+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:49.191521+0000 front 2024-03-18T15:29:49.191660+0000 (oldest deadline 2024-03-18T15:30:12.691483+0000) 2024-03-18T15:30:26.657 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:26 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:26.332+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:59.092785+0000 front 2024-03-18T15:29:59.092921+0000 (oldest deadline 2024-03-18T15:30:21.992976+0000) 2024-03-18T15:30:26.939 INFO:journalctl@ceph.osd.5.smithi104.stdout:Mar 18 15:30:26 smithi104 podman[73698]: 2024-03-18 15:30:26.656967403 +0000 UTC m=+5.654145797 container died 2af319d597e45dee93a5eae0703fdc8254cc7dca16d24739c50fe764cde59c49 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS) 2024-03-18T15:30:27.194 INFO:journalctl@ceph.osd.5.smithi104.stdout:Mar 18 15:30:27 smithi104 podman[73698]: 2024-03-18 15:30:27.037841216 +0000 UTC m=+6.035019595 container cleanup 2af319d597e45dee93a5eae0703fdc8254cc7dca16d24739c50fe764cde59c49 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T15:30:27.194 INFO:journalctl@ceph.osd.5.smithi104.stdout:Mar 18 15:30:27 smithi104 bash[73698]: ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-5 2024-03-18T15:30:27.195 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:27 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:27.068+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:41.404783+0000 front 2024-03-18T15:29:41.404661+0000 (oldest deadline 2024-03-18T15:30:04.304572+0000) 2024-03-18T15:30:27.195 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:27 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:27.068+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.207053+0000 front 2024-03-18T15:29:51.206898+0000 (oldest deadline 2024-03-18T15:30:13.507003+0000) 2024-03-18T15:30:27.195 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:27 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:27.068+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:59.809088+0000 front 2024-03-18T15:29:59.809253+0000 (oldest deadline 2024-03-18T15:30:21.509244+0000) 2024-03-18T15:30:27.218 INFO:journalctl@ceph.osd.7.smithi033.stdout:Mar 18 15:30:26 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-7[54818]: 2024-03-18T15:30:26.845+0000 7f480d187700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:50.578091+0000 front 2024-03-18T15:29:50.578050+0000 (oldest deadline 2024-03-18T15:30:15.278156+0000) 2024-03-18T15:30:27.218 INFO:journalctl@ceph.osd.7.smithi033.stdout:Mar 18 15:30:26 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-7[54818]: 2024-03-18T15:30:26.845+0000 7f480d187700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:57.579372+0000 front 2024-03-18T15:29:57.579232+0000 (oldest deadline 2024-03-18T15:30:22.879178+0000) 2024-03-18T15:30:27.465 INFO:journalctl@ceph.osd.6.smithi019.stdout:Mar 18 15:30:27 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-6[59600]: 2024-03-18T15:30:27.113+0000 7fde4d400700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:52.056785+0000 front 2024-03-18T15:29:52.056368+0000 (oldest deadline 2024-03-18T15:30:13.756023+0000) 2024-03-18T15:30:27.465 INFO:journalctl@ceph.osd.6.smithi019.stdout:Mar 18 15:30:27 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-6[59600]: 2024-03-18T15:30:27.113+0000 7fde4d400700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:57.758902+0000 front 2024-03-18T15:29:57.759391+0000 (oldest deadline 2024-03-18T15:30:23.059128+0000) 2024-03-18T15:30:27.466 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:27 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:27.281+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:39.237875+0000 front 2024-03-18T15:29:39.238431+0000 (oldest deadline 2024-03-18T15:30:05.138032+0000) 2024-03-18T15:30:27.466 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:27 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:27.281+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:50.743126+0000 front 2024-03-18T15:29:50.743226+0000 (oldest deadline 2024-03-18T15:30:13.643377+0000) 2024-03-18T15:30:27.466 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:27 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:27.281+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:56.947757+0000 front 2024-03-18T15:29:56.947820+0000 (oldest deadline 2024-03-18T15:30:22.248013+0000) 2024-03-18T15:30:27.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:27 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:27.217+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.612085+0000 front 2024-03-18T15:29:51.612164+0000 (oldest deadline 2024-03-18T15:30:17.512254+0000) 2024-03-18T15:30:27.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:27 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:27.217+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:57.512824+0000 front 2024-03-18T15:29:57.512967+0000 (oldest deadline 2024-03-18T15:30:21.612923+0000) 2024-03-18T15:30:27.650 INFO:journalctl@ceph.osd.5.smithi104.stdout:Mar 18 15:30:27 smithi104 podman[73716]: 2024-03-18 15:30:27.322661262 +0000 UTC m=+0.666320913 container remove 2af319d597e45dee93a5eae0703fdc8254cc7dca16d24739c50fe764cde59c49 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T15:30:27.650 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:27 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:27.379+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:49.191521+0000 front 2024-03-18T15:29:49.191660+0000 (oldest deadline 2024-03-18T15:30:12.691483+0000) 2024-03-18T15:30:27.650 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:27 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:27.379+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:59.092785+0000 front 2024-03-18T15:29:59.092921+0000 (oldest deadline 2024-03-18T15:30:21.992976+0000) 2024-03-18T15:30:27.902 INFO:journalctl@ceph.osd.5.smithi104.stdout:Mar 18 15:30:27 smithi104 podman[73807]: 2024-03-18 15:30:27.552988499 +0000 UTC m=+0.018395107 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T15:30:27.902 INFO:journalctl@ceph.osd.5.smithi104.stdout:Mar 18 15:30:27 smithi104 podman[73807]: 2024-03-18 15:30:27.657191 +0000 UTC m=+0.122597651 container create 3e49b7be0781b5dab13ca34c24c932bc069fd3651361f031a1eaf7852953c419 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-5-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240307, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-18T15:30:28.189 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:28 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:28.116+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:41.404783+0000 front 2024-03-18T15:29:41.404661+0000 (oldest deadline 2024-03-18T15:30:04.304572+0000) 2024-03-18T15:30:28.189 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:28 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:28.116+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.207053+0000 front 2024-03-18T15:29:51.206898+0000 (oldest deadline 2024-03-18T15:30:13.507003+0000) 2024-03-18T15:30:28.189 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:28 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:28.116+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:59.809088+0000 front 2024-03-18T15:29:59.809253+0000 (oldest deadline 2024-03-18T15:30:21.509244+0000) 2024-03-18T15:30:28.189 INFO:journalctl@ceph.osd.5.smithi104.stdout:Mar 18 15:30:27 smithi104 podman[73807]: 2024-03-18 15:30:27.901833571 +0000 UTC m=+0.367240194 container init 3e49b7be0781b5dab13ca34c24c932bc069fd3651361f031a1eaf7852953c419 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-5-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T15:30:28.190 INFO:journalctl@ceph.osd.5.smithi104.stdout:Mar 18 15:30:27 smithi104 podman[73807]: 2024-03-18 15:30:27.905315608 +0000 UTC m=+0.370722219 container start 3e49b7be0781b5dab13ca34c24c932bc069fd3651361f031a1eaf7852953c419 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-5-deactivate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T15:30:28.190 INFO:journalctl@ceph.osd.5.smithi104.stdout:Mar 18 15:30:28 smithi104 podman[73807]: 2024-03-18 15:30:28.009743134 +0000 UTC m=+0.475149752 container attach 3e49b7be0781b5dab13ca34c24c932bc069fd3651361f031a1eaf7852953c419 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-5-deactivate, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-18T15:30:28.211 INFO:journalctl@ceph.osd.7.smithi033.stdout:Mar 18 15:30:27 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-7[54818]: 2024-03-18T15:30:27.825+0000 7f480d187700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:50.578091+0000 front 2024-03-18T15:29:50.578050+0000 (oldest deadline 2024-03-18T15:30:15.278156+0000) 2024-03-18T15:30:28.211 INFO:journalctl@ceph.osd.7.smithi033.stdout:Mar 18 15:30:27 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-7[54818]: 2024-03-18T15:30:27.825+0000 7f480d187700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:57.579372+0000 front 2024-03-18T15:29:57.579232+0000 (oldest deadline 2024-03-18T15:30:22.879178+0000) 2024-03-18T15:30:28.444 INFO:journalctl@ceph.osd.5.smithi104.stdout:Mar 18 15:30:28 smithi104 podman[73807]: 2024-03-18 15:30:28.189673413 +0000 UTC m=+0.655080041 container died 3e49b7be0781b5dab13ca34c24c932bc069fd3651361f031a1eaf7852953c419 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-5-deactivate, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T15:30:28.444 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:28 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:28.413+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:49.191521+0000 front 2024-03-18T15:29:49.191660+0000 (oldest deadline 2024-03-18T15:30:12.691483+0000) 2024-03-18T15:30:28.444 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:28 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:28.413+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:59.092785+0000 front 2024-03-18T15:29:59.092921+0000 (oldest deadline 2024-03-18T15:30:21.992976+0000) 2024-03-18T15:30:28.465 INFO:journalctl@ceph.osd.6.smithi019.stdout:Mar 18 15:30:28 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-6[59600]: 2024-03-18T15:30:28.102+0000 7fde4d400700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:52.056785+0000 front 2024-03-18T15:29:52.056368+0000 (oldest deadline 2024-03-18T15:30:13.756023+0000) 2024-03-18T15:30:28.465 INFO:journalctl@ceph.osd.6.smithi019.stdout:Mar 18 15:30:28 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-6[59600]: 2024-03-18T15:30:28.102+0000 7fde4d400700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:57.758902+0000 front 2024-03-18T15:29:57.759391+0000 (oldest deadline 2024-03-18T15:30:23.059128+0000) 2024-03-18T15:30:28.465 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:28 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:28.253+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:39.237875+0000 front 2024-03-18T15:29:39.238431+0000 (oldest deadline 2024-03-18T15:30:05.138032+0000) 2024-03-18T15:30:28.466 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:28 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:28.253+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:50.743126+0000 front 2024-03-18T15:29:50.743226+0000 (oldest deadline 2024-03-18T15:30:13.643377+0000) 2024-03-18T15:30:28.466 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:28 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:28.253+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:56.947757+0000 front 2024-03-18T15:29:56.947820+0000 (oldest deadline 2024-03-18T15:30:22.248013+0000) 2024-03-18T15:30:28.506 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:28 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:28.211+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.612085+0000 front 2024-03-18T15:29:51.612164+0000 (oldest deadline 2024-03-18T15:30:17.512254+0000) 2024-03-18T15:30:28.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:28 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:28.211+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:57.512824+0000 front 2024-03-18T15:29:57.512967+0000 (oldest deadline 2024-03-18T15:30:21.612923+0000) 2024-03-18T15:30:28.850 DEBUG:teuthology.orchestra.run.smithi104:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@osd.5.service' 2024-03-18T15:30:28.933 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T15:30:28.934 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-18T15:30:28.934 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-18T15:30:28.934 DEBUG:teuthology.orchestra.run.smithi019:> sudo systemctl stop ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@osd.6 2024-03-18T15:30:29.221 INFO:journalctl@ceph.osd.7.smithi033.stdout:Mar 18 15:30:28 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-7[54818]: 2024-03-18T15:30:28.826+0000 7f480d187700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:50.578091+0000 front 2024-03-18T15:29:50.578050+0000 (oldest deadline 2024-03-18T15:30:15.278156+0000) 2024-03-18T15:30:29.221 INFO:journalctl@ceph.osd.7.smithi033.stdout:Mar 18 15:30:28 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-7[54818]: 2024-03-18T15:30:28.826+0000 7f480d187700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:57.579372+0000 front 2024-03-18T15:29:57.579232+0000 (oldest deadline 2024-03-18T15:30:22.879178+0000) 2024-03-18T15:30:29.435 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:29 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:29.112+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:41.404783+0000 front 2024-03-18T15:29:41.404661+0000 (oldest deadline 2024-03-18T15:30:04.304572+0000) 2024-03-18T15:30:29.435 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:29 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:29.112+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.207053+0000 front 2024-03-18T15:29:51.206898+0000 (oldest deadline 2024-03-18T15:30:13.507003+0000) 2024-03-18T15:30:29.435 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:29 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:29.112+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:59.809088+0000 front 2024-03-18T15:29:59.809253+0000 (oldest deadline 2024-03-18T15:30:21.509244+0000) 2024-03-18T15:30:29.465 INFO:journalctl@ceph.osd.6.smithi019.stdout:Mar 18 15:30:29 smithi019 systemd[1]: Stopping Ceph osd.6 for d4dc2ba6-e538-11ee-95c9-87774f69a715... 2024-03-18T15:30:29.465 INFO:journalctl@ceph.osd.6.smithi019.stdout:Mar 18 15:30:29 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-6[59600]: 2024-03-18T15:30:29.104+0000 7fde4d400700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:52.056785+0000 front 2024-03-18T15:29:52.056368+0000 (oldest deadline 2024-03-18T15:30:13.756023+0000) 2024-03-18T15:30:29.465 INFO:journalctl@ceph.osd.6.smithi019.stdout:Mar 18 15:30:29 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-6[59600]: 2024-03-18T15:30:29.104+0000 7fde4d400700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:57.758902+0000 front 2024-03-18T15:29:57.759391+0000 (oldest deadline 2024-03-18T15:30:23.059128+0000) 2024-03-18T15:30:29.466 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:29 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:29.262+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:39.237875+0000 front 2024-03-18T15:29:39.238431+0000 (oldest deadline 2024-03-18T15:30:05.138032+0000) 2024-03-18T15:30:29.466 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:29 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:29.262+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:50.743126+0000 front 2024-03-18T15:29:50.743226+0000 (oldest deadline 2024-03-18T15:30:13.643377+0000) 2024-03-18T15:30:29.466 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:29 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:29.262+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:56.947757+0000 front 2024-03-18T15:29:56.947820+0000 (oldest deadline 2024-03-18T15:30:22.248013+0000) 2024-03-18T15:30:29.506 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:29 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:29.220+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.612085+0000 front 2024-03-18T15:29:51.612164+0000 (oldest deadline 2024-03-18T15:30:17.512254+0000) 2024-03-18T15:30:29.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:29 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:29.220+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:57.512824+0000 front 2024-03-18T15:29:57.512967+0000 (oldest deadline 2024-03-18T15:30:21.612923+0000) 2024-03-18T15:30:29.694 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:29 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:29.434+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:49.191521+0000 front 2024-03-18T15:29:49.191660+0000 (oldest deadline 2024-03-18T15:30:12.691483+0000) 2024-03-18T15:30:29.694 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:29 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:29.434+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:59.092785+0000 front 2024-03-18T15:29:59.092921+0000 (oldest deadline 2024-03-18T15:30:21.992976+0000) 2024-03-18T15:30:29.965 INFO:journalctl@ceph.osd.6.smithi019.stdout:Mar 18 15:30:29 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-6[59600]: 2024-03-18T15:30:29.602+0000 7fde56c00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T15:30:29.966 INFO:journalctl@ceph.osd.6.smithi019.stdout:Mar 18 15:30:29 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-6[59600]: 2024-03-18T15:30:29.602+0000 7fde56c00700 -1 osd.6 109 *** Got signal Terminated *** 2024-03-18T15:30:29.966 INFO:journalctl@ceph.osd.6.smithi019.stdout:Mar 18 15:30:29 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-6[59600]: 2024-03-18T15:30:29.602+0000 7fde56c00700 -1 osd.6 109 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T15:30:30.182 INFO:journalctl@ceph.osd.7.smithi033.stdout:Mar 18 15:30:29 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-7[54818]: 2024-03-18T15:30:29.804+0000 7f480d187700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:50.578091+0000 front 2024-03-18T15:29:50.578050+0000 (oldest deadline 2024-03-18T15:30:15.278156+0000) 2024-03-18T15:30:30.182 INFO:journalctl@ceph.osd.7.smithi033.stdout:Mar 18 15:30:29 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-7[54818]: 2024-03-18T15:30:29.804+0000 7f480d187700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:57.579372+0000 front 2024-03-18T15:29:57.579232+0000 (oldest deadline 2024-03-18T15:30:22.879178+0000) 2024-03-18T15:30:30.182 INFO:journalctl@ceph.osd.7.smithi033.stdout:Mar 18 15:30:29 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-7[54818]: 2024-03-18T15:30:29.804+0000 7f480d187700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:06.280853+0000 front 2024-03-18T15:30:06.280754+0000 (oldest deadline 2024-03-18T15:30:29.780617+0000) 2024-03-18T15:30:30.440 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:30 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:30.095+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:41.404783+0000 front 2024-03-18T15:29:41.404661+0000 (oldest deadline 2024-03-18T15:30:04.304572+0000) 2024-03-18T15:30:30.440 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:30 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:30.095+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.207053+0000 front 2024-03-18T15:29:51.206898+0000 (oldest deadline 2024-03-18T15:30:13.507003+0000) 2024-03-18T15:30:30.440 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:30 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:30.095+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:59.809088+0000 front 2024-03-18T15:29:59.809253+0000 (oldest deadline 2024-03-18T15:30:21.509244+0000) 2024-03-18T15:30:30.465 INFO:journalctl@ceph.osd.6.smithi019.stdout:Mar 18 15:30:30 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-6[59600]: 2024-03-18T15:30:30.113+0000 7fde4d400700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:52.056785+0000 front 2024-03-18T15:29:52.056368+0000 (oldest deadline 2024-03-18T15:30:13.756023+0000) 2024-03-18T15:30:30.465 INFO:journalctl@ceph.osd.6.smithi019.stdout:Mar 18 15:30:30 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-6[59600]: 2024-03-18T15:30:30.114+0000 7fde4d400700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:57.758902+0000 front 2024-03-18T15:29:57.759391+0000 (oldest deadline 2024-03-18T15:30:23.059128+0000) 2024-03-18T15:30:30.466 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:30 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:30.242+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:39.237875+0000 front 2024-03-18T15:29:39.238431+0000 (oldest deadline 2024-03-18T15:30:05.138032+0000) 2024-03-18T15:30:30.466 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:30 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:30.242+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:50.743126+0000 front 2024-03-18T15:29:50.743226+0000 (oldest deadline 2024-03-18T15:30:13.643377+0000) 2024-03-18T15:30:30.466 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:30 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:30.242+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:56.947757+0000 front 2024-03-18T15:29:56.947820+0000 (oldest deadline 2024-03-18T15:30:22.248013+0000) 2024-03-18T15:30:30.506 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:30 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:30.181+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.612085+0000 front 2024-03-18T15:29:51.612164+0000 (oldest deadline 2024-03-18T15:30:17.512254+0000) 2024-03-18T15:30:30.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:30 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:30.181+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:57.512824+0000 front 2024-03-18T15:29:57.512967+0000 (oldest deadline 2024-03-18T15:30:21.612923+0000) 2024-03-18T15:30:30.694 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:30 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:30.439+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:49.191521+0000 front 2024-03-18T15:29:49.191660+0000 (oldest deadline 2024-03-18T15:30:12.691483+0000) 2024-03-18T15:30:30.695 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:30 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:30.439+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:59.092785+0000 front 2024-03-18T15:29:59.092921+0000 (oldest deadline 2024-03-18T15:30:21.992976+0000) 2024-03-18T15:30:31.171 INFO:journalctl@ceph.osd.7.smithi033.stdout:Mar 18 15:30:30 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-7[54818]: 2024-03-18T15:30:30.828+0000 7f480d187700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:50.578091+0000 front 2024-03-18T15:29:50.578050+0000 (oldest deadline 2024-03-18T15:30:15.278156+0000) 2024-03-18T15:30:31.171 INFO:journalctl@ceph.osd.7.smithi033.stdout:Mar 18 15:30:30 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-7[54818]: 2024-03-18T15:30:30.828+0000 7f480d187700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:57.579372+0000 front 2024-03-18T15:29:57.579232+0000 (oldest deadline 2024-03-18T15:30:22.879178+0000) 2024-03-18T15:30:31.171 INFO:journalctl@ceph.osd.7.smithi033.stdout:Mar 18 15:30:30 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-7[54818]: 2024-03-18T15:30:30.828+0000 7f480d187700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:06.280853+0000 front 2024-03-18T15:30:06.280754+0000 (oldest deadline 2024-03-18T15:30:29.780617+0000) 2024-03-18T15:30:31.420 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:31 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:31.141+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:41.404783+0000 front 2024-03-18T15:29:41.404661+0000 (oldest deadline 2024-03-18T15:30:04.304572+0000) 2024-03-18T15:30:31.420 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:31 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:31.141+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.207053+0000 front 2024-03-18T15:29:51.206898+0000 (oldest deadline 2024-03-18T15:30:13.507003+0000) 2024-03-18T15:30:31.420 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:31 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:31.141+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:59.809088+0000 front 2024-03-18T15:29:59.809253+0000 (oldest deadline 2024-03-18T15:30:21.509244+0000) 2024-03-18T15:30:31.465 INFO:journalctl@ceph.osd.6.smithi019.stdout:Mar 18 15:30:31 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-6[59600]: 2024-03-18T15:30:31.103+0000 7fde4d400700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:52.056785+0000 front 2024-03-18T15:29:52.056368+0000 (oldest deadline 2024-03-18T15:30:13.756023+0000) 2024-03-18T15:30:31.465 INFO:journalctl@ceph.osd.6.smithi019.stdout:Mar 18 15:30:31 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-6[59600]: 2024-03-18T15:30:31.103+0000 7fde4d400700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:57.758902+0000 front 2024-03-18T15:29:57.759391+0000 (oldest deadline 2024-03-18T15:30:23.059128+0000) 2024-03-18T15:30:31.466 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:31 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:31.272+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:39.237875+0000 front 2024-03-18T15:29:39.238431+0000 (oldest deadline 2024-03-18T15:30:05.138032+0000) 2024-03-18T15:30:31.466 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:31 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:31.272+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:50.743126+0000 front 2024-03-18T15:29:50.743226+0000 (oldest deadline 2024-03-18T15:30:13.643377+0000) 2024-03-18T15:30:31.466 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:31 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:31.272+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:56.947757+0000 front 2024-03-18T15:29:56.947820+0000 (oldest deadline 2024-03-18T15:30:22.248013+0000) 2024-03-18T15:30:31.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:31 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:31.170+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.612085+0000 front 2024-03-18T15:29:51.612164+0000 (oldest deadline 2024-03-18T15:30:17.512254+0000) 2024-03-18T15:30:31.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:31 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:31.170+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:57.512824+0000 front 2024-03-18T15:29:57.512967+0000 (oldest deadline 2024-03-18T15:30:21.612923+0000) 2024-03-18T15:30:31.689 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:31 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:31.419+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:49.191521+0000 front 2024-03-18T15:29:49.191660+0000 (oldest deadline 2024-03-18T15:30:12.691483+0000) 2024-03-18T15:30:31.689 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:31 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:31.419+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:59.092785+0000 front 2024-03-18T15:29:59.092921+0000 (oldest deadline 2024-03-18T15:30:21.992976+0000) 2024-03-18T15:30:31.689 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:31 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:31.419+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:04.894647+0000 front 2024-03-18T15:30:04.895189+0000 (oldest deadline 2024-03-18T15:30:30.794498+0000) 2024-03-18T15:30:32.133 INFO:journalctl@ceph.osd.7.smithi033.stdout:Mar 18 15:30:31 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-7[54818]: 2024-03-18T15:30:31.790+0000 7f480d187700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:50.578091+0000 front 2024-03-18T15:29:50.578050+0000 (oldest deadline 2024-03-18T15:30:15.278156+0000) 2024-03-18T15:30:32.133 INFO:journalctl@ceph.osd.7.smithi033.stdout:Mar 18 15:30:31 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-7[54818]: 2024-03-18T15:30:31.790+0000 7f480d187700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:57.579372+0000 front 2024-03-18T15:29:57.579232+0000 (oldest deadline 2024-03-18T15:30:22.879178+0000) 2024-03-18T15:30:32.133 INFO:journalctl@ceph.osd.7.smithi033.stdout:Mar 18 15:30:31 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-7[54818]: 2024-03-18T15:30:31.790+0000 7f480d187700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:06.280853+0000 front 2024-03-18T15:30:06.280754+0000 (oldest deadline 2024-03-18T15:30:29.780617+0000) 2024-03-18T15:30:32.444 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:32 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:32.373+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:49.191521+0000 front 2024-03-18T15:29:49.191660+0000 (oldest deadline 2024-03-18T15:30:12.691483+0000) 2024-03-18T15:30:32.445 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:32 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:32.373+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:59.092785+0000 front 2024-03-18T15:29:59.092921+0000 (oldest deadline 2024-03-18T15:30:21.992976+0000) 2024-03-18T15:30:32.445 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:32 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:32.373+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:04.894647+0000 front 2024-03-18T15:30:04.895189+0000 (oldest deadline 2024-03-18T15:30:30.794498+0000) 2024-03-18T15:30:32.445 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:32 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:32.158+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:41.404783+0000 front 2024-03-18T15:29:41.404661+0000 (oldest deadline 2024-03-18T15:30:04.304572+0000) 2024-03-18T15:30:32.445 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:32 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:32.158+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.207053+0000 front 2024-03-18T15:29:51.206898+0000 (oldest deadline 2024-03-18T15:30:13.507003+0000) 2024-03-18T15:30:32.445 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:32 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:32.158+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:59.809088+0000 front 2024-03-18T15:29:59.809253+0000 (oldest deadline 2024-03-18T15:30:21.509244+0000) 2024-03-18T15:30:32.465 INFO:journalctl@ceph.osd.6.smithi019.stdout:Mar 18 15:30:32 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-6[59600]: 2024-03-18T15:30:32.058+0000 7fde4d400700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:52.056785+0000 front 2024-03-18T15:29:52.056368+0000 (oldest deadline 2024-03-18T15:30:13.756023+0000) 2024-03-18T15:30:32.465 INFO:journalctl@ceph.osd.6.smithi019.stdout:Mar 18 15:30:32 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-6[59600]: 2024-03-18T15:30:32.058+0000 7fde4d400700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:57.758902+0000 front 2024-03-18T15:29:57.759391+0000 (oldest deadline 2024-03-18T15:30:23.059128+0000) 2024-03-18T15:30:32.465 INFO:journalctl@ceph.osd.6.smithi019.stdout:Mar 18 15:30:32 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-6[59600]: 2024-03-18T15:30:32.058+0000 7fde4d400700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:03.060880+0000 front 2024-03-18T15:30:03.060047+0000 (oldest deadline 2024-03-18T15:30:31.861502+0000) 2024-03-18T15:30:32.466 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:32 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:32.286+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:39.237875+0000 front 2024-03-18T15:29:39.238431+0000 (oldest deadline 2024-03-18T15:30:05.138032+0000) 2024-03-18T15:30:32.466 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:32 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:32.286+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:50.743126+0000 front 2024-03-18T15:29:50.743226+0000 (oldest deadline 2024-03-18T15:30:13.643377+0000) 2024-03-18T15:30:32.466 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:32 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:32.286+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:56.947757+0000 front 2024-03-18T15:29:56.947820+0000 (oldest deadline 2024-03-18T15:30:22.248013+0000) 2024-03-18T15:30:32.466 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:32 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:32.286+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:06.950015+0000 front 2024-03-18T15:30:06.950486+0000 (oldest deadline 2024-03-18T15:30:31.650112+0000) 2024-03-18T15:30:32.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:32 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:32.132+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.612085+0000 front 2024-03-18T15:29:51.612164+0000 (oldest deadline 2024-03-18T15:30:17.512254+0000) 2024-03-18T15:30:32.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:32 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:32.132+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:57.512824+0000 front 2024-03-18T15:29:57.512967+0000 (oldest deadline 2024-03-18T15:30:21.612923+0000) 2024-03-18T15:30:33.007 INFO:journalctl@ceph.osd.7.smithi033.stdout:Mar 18 15:30:32 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-7[54818]: 2024-03-18T15:30:32.741+0000 7f480d187700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:50.578091+0000 front 2024-03-18T15:29:50.578050+0000 (oldest deadline 2024-03-18T15:30:15.278156+0000) 2024-03-18T15:30:33.007 INFO:journalctl@ceph.osd.7.smithi033.stdout:Mar 18 15:30:32 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-7[54818]: 2024-03-18T15:30:32.741+0000 7f480d187700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:57.579372+0000 front 2024-03-18T15:29:57.579232+0000 (oldest deadline 2024-03-18T15:30:22.879178+0000) 2024-03-18T15:30:33.007 INFO:journalctl@ceph.osd.7.smithi033.stdout:Mar 18 15:30:32 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-7[54818]: 2024-03-18T15:30:32.741+0000 7f480d187700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:06.280853+0000 front 2024-03-18T15:30:06.280754+0000 (oldest deadline 2024-03-18T15:30:29.780617+0000) 2024-03-18T15:30:33.388 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:33 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:33.136+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:41.404783+0000 front 2024-03-18T15:29:41.404661+0000 (oldest deadline 2024-03-18T15:30:04.304572+0000) 2024-03-18T15:30:33.388 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:33 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:33.136+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.207053+0000 front 2024-03-18T15:29:51.206898+0000 (oldest deadline 2024-03-18T15:30:13.507003+0000) 2024-03-18T15:30:33.388 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:33 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:33.136+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:59.809088+0000 front 2024-03-18T15:29:59.809253+0000 (oldest deadline 2024-03-18T15:30:21.509244+0000) 2024-03-18T15:30:33.465 INFO:journalctl@ceph.osd.6.smithi019.stdout:Mar 18 15:30:33 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-6[59600]: 2024-03-18T15:30:33.108+0000 7fde4d400700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:52.056785+0000 front 2024-03-18T15:29:52.056368+0000 (oldest deadline 2024-03-18T15:30:13.756023+0000) 2024-03-18T15:30:33.465 INFO:journalctl@ceph.osd.6.smithi019.stdout:Mar 18 15:30:33 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-6[59600]: 2024-03-18T15:30:33.108+0000 7fde4d400700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:57.758902+0000 front 2024-03-18T15:29:57.759391+0000 (oldest deadline 2024-03-18T15:30:23.059128+0000) 2024-03-18T15:30:33.465 INFO:journalctl@ceph.osd.6.smithi019.stdout:Mar 18 15:30:33 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-6[59600]: 2024-03-18T15:30:33.108+0000 7fde4d400700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:03.060880+0000 front 2024-03-18T15:30:03.060047+0000 (oldest deadline 2024-03-18T15:30:31.861502+0000) 2024-03-18T15:30:33.466 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:33 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:33.267+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:39.237875+0000 front 2024-03-18T15:29:39.238431+0000 (oldest deadline 2024-03-18T15:30:05.138032+0000) 2024-03-18T15:30:33.466 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:33 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:33.267+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:50.743126+0000 front 2024-03-18T15:29:50.743226+0000 (oldest deadline 2024-03-18T15:30:13.643377+0000) 2024-03-18T15:30:33.466 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:33 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:33.267+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:56.947757+0000 front 2024-03-18T15:29:56.947820+0000 (oldest deadline 2024-03-18T15:30:22.248013+0000) 2024-03-18T15:30:33.466 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:33 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:33.267+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:06.950015+0000 front 2024-03-18T15:30:06.950486+0000 (oldest deadline 2024-03-18T15:30:31.650112+0000) 2024-03-18T15:30:33.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:33 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:33.095+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.612085+0000 front 2024-03-18T15:29:51.612164+0000 (oldest deadline 2024-03-18T15:30:17.512254+0000) 2024-03-18T15:30:33.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:33 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:33.095+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:57.512824+0000 front 2024-03-18T15:29:57.512967+0000 (oldest deadline 2024-03-18T15:30:21.612923+0000) 2024-03-18T15:30:33.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:33 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:33.095+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:07.414865+0000 front 2024-03-18T15:30:07.415069+0000 (oldest deadline 2024-03-18T15:30:32.714829+0000) 2024-03-18T15:30:33.694 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:33 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:33.387+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:49.191521+0000 front 2024-03-18T15:29:49.191660+0000 (oldest deadline 2024-03-18T15:30:12.691483+0000) 2024-03-18T15:30:33.694 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:33 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:33.387+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:59.092785+0000 front 2024-03-18T15:29:59.092921+0000 (oldest deadline 2024-03-18T15:30:21.992976+0000) 2024-03-18T15:30:33.694 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:33 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:33.387+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:04.894647+0000 front 2024-03-18T15:30:04.895189+0000 (oldest deadline 2024-03-18T15:30:30.794498+0000) 2024-03-18T15:30:34.007 INFO:journalctl@ceph.osd.7.smithi033.stdout:Mar 18 15:30:33 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-7[54818]: 2024-03-18T15:30:33.731+0000 7f480d187700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:50.578091+0000 front 2024-03-18T15:29:50.578050+0000 (oldest deadline 2024-03-18T15:30:15.278156+0000) 2024-03-18T15:30:34.007 INFO:journalctl@ceph.osd.7.smithi033.stdout:Mar 18 15:30:33 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-7[54818]: 2024-03-18T15:30:33.731+0000 7f480d187700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:57.579372+0000 front 2024-03-18T15:29:57.579232+0000 (oldest deadline 2024-03-18T15:30:22.879178+0000) 2024-03-18T15:30:34.007 INFO:journalctl@ceph.osd.7.smithi033.stdout:Mar 18 15:30:33 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-7[54818]: 2024-03-18T15:30:33.731+0000 7f480d187700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:06.280853+0000 front 2024-03-18T15:30:06.280754+0000 (oldest deadline 2024-03-18T15:30:29.780617+0000) 2024-03-18T15:30:34.434 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:34 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:34.176+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:41.404783+0000 front 2024-03-18T15:29:41.404661+0000 (oldest deadline 2024-03-18T15:30:04.304572+0000) 2024-03-18T15:30:34.437 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:34 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:34.176+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.207053+0000 front 2024-03-18T15:29:51.206898+0000 (oldest deadline 2024-03-18T15:30:13.507003+0000) 2024-03-18T15:30:34.437 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:34 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:34.176+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:59.809088+0000 front 2024-03-18T15:29:59.809253+0000 (oldest deadline 2024-03-18T15:30:21.509244+0000) 2024-03-18T15:30:34.465 INFO:journalctl@ceph.osd.6.smithi019.stdout:Mar 18 15:30:34 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-6[59600]: 2024-03-18T15:30:34.073+0000 7fde4d400700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:52.056785+0000 front 2024-03-18T15:29:52.056368+0000 (oldest deadline 2024-03-18T15:30:13.756023+0000) 2024-03-18T15:30:34.465 INFO:journalctl@ceph.osd.6.smithi019.stdout:Mar 18 15:30:34 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-6[59600]: 2024-03-18T15:30:34.073+0000 7fde4d400700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:57.758902+0000 front 2024-03-18T15:29:57.759391+0000 (oldest deadline 2024-03-18T15:30:23.059128+0000) 2024-03-18T15:30:34.465 INFO:journalctl@ceph.osd.6.smithi019.stdout:Mar 18 15:30:34 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-6[59600]: 2024-03-18T15:30:34.073+0000 7fde4d400700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:03.060880+0000 front 2024-03-18T15:30:03.060047+0000 (oldest deadline 2024-03-18T15:30:31.861502+0000) 2024-03-18T15:30:34.466 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:34 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:34.290+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:39.237875+0000 front 2024-03-18T15:29:39.238431+0000 (oldest deadline 2024-03-18T15:30:05.138032+0000) 2024-03-18T15:30:34.466 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:34 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:34.290+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:50.743126+0000 front 2024-03-18T15:29:50.743226+0000 (oldest deadline 2024-03-18T15:30:13.643377+0000) 2024-03-18T15:30:34.466 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:34 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:34.290+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:56.947757+0000 front 2024-03-18T15:29:56.947820+0000 (oldest deadline 2024-03-18T15:30:22.248013+0000) 2024-03-18T15:30:34.466 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:34 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:34.290+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:06.950015+0000 front 2024-03-18T15:30:06.950486+0000 (oldest deadline 2024-03-18T15:30:31.650112+0000) 2024-03-18T15:30:34.506 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:34 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:34.090+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.612085+0000 front 2024-03-18T15:29:51.612164+0000 (oldest deadline 2024-03-18T15:30:17.512254+0000) 2024-03-18T15:30:34.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:34 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:34.090+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:57.512824+0000 front 2024-03-18T15:29:57.512967+0000 (oldest deadline 2024-03-18T15:30:21.612923+0000) 2024-03-18T15:30:34.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:34 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:34.090+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:07.414865+0000 front 2024-03-18T15:30:07.415069+0000 (oldest deadline 2024-03-18T15:30:32.714829+0000) 2024-03-18T15:30:34.694 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:34 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:34.433+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:49.191521+0000 front 2024-03-18T15:29:49.191660+0000 (oldest deadline 2024-03-18T15:30:12.691483+0000) 2024-03-18T15:30:34.695 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:34 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:34.433+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:59.092785+0000 front 2024-03-18T15:29:59.092921+0000 (oldest deadline 2024-03-18T15:30:21.992976+0000) 2024-03-18T15:30:34.695 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:34 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:34.433+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:04.894647+0000 front 2024-03-18T15:30:04.895189+0000 (oldest deadline 2024-03-18T15:30:30.794498+0000) 2024-03-18T15:30:35.075 INFO:journalctl@ceph.osd.7.smithi033.stdout:Mar 18 15:30:34 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-7[54818]: 2024-03-18T15:30:34.769+0000 7f480d187700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:50.578091+0000 front 2024-03-18T15:29:50.578050+0000 (oldest deadline 2024-03-18T15:30:15.278156+0000) 2024-03-18T15:30:35.075 INFO:journalctl@ceph.osd.7.smithi033.stdout:Mar 18 15:30:34 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-7[54818]: 2024-03-18T15:30:34.769+0000 7f480d187700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:57.579372+0000 front 2024-03-18T15:29:57.579232+0000 (oldest deadline 2024-03-18T15:30:22.879178+0000) 2024-03-18T15:30:35.075 INFO:journalctl@ceph.osd.7.smithi033.stdout:Mar 18 15:30:34 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-7[54818]: 2024-03-18T15:30:34.769+0000 7f480d187700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:06.280853+0000 front 2024-03-18T15:30:06.280754+0000 (oldest deadline 2024-03-18T15:30:29.780617+0000) 2024-03-18T15:30:35.366 INFO:journalctl@ceph.osd.6.smithi019.stdout:Mar 18 15:30:35 smithi019 podman[92830]: 2024-03-18 15:30:35.014447838 +0000 UTC m=+5.560328601 container died 812afcb2a2c5d5032352fd06de57edbd97b45cd5f81c49099d9bd733e6d9c9fe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-6, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-18T15:30:35.366 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:35 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:35.243+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:39.237875+0000 front 2024-03-18T15:29:39.238431+0000 (oldest deadline 2024-03-18T15:30:05.138032+0000) 2024-03-18T15:30:35.366 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:35 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:35.243+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:50.743126+0000 front 2024-03-18T15:29:50.743226+0000 (oldest deadline 2024-03-18T15:30:13.643377+0000) 2024-03-18T15:30:35.366 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:35 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:35.243+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:56.947757+0000 front 2024-03-18T15:29:56.947820+0000 (oldest deadline 2024-03-18T15:30:22.248013+0000) 2024-03-18T15:30:35.366 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:35 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:35.243+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:06.950015+0000 front 2024-03-18T15:30:06.950486+0000 (oldest deadline 2024-03-18T15:30:31.650112+0000) 2024-03-18T15:30:35.444 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:35 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:35.166+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:41.404783+0000 front 2024-03-18T15:29:41.404661+0000 (oldest deadline 2024-03-18T15:30:04.304572+0000) 2024-03-18T15:30:35.444 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:35 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:35.166+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.207053+0000 front 2024-03-18T15:29:51.206898+0000 (oldest deadline 2024-03-18T15:30:13.507003+0000) 2024-03-18T15:30:35.444 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:35 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:35.166+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:59.809088+0000 front 2024-03-18T15:29:59.809253+0000 (oldest deadline 2024-03-18T15:30:21.509244+0000) 2024-03-18T15:30:35.506 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:35 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:35.075+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.612085+0000 front 2024-03-18T15:29:51.612164+0000 (oldest deadline 2024-03-18T15:30:17.512254+0000) 2024-03-18T15:30:35.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:35 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:35.075+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:57.512824+0000 front 2024-03-18T15:29:57.512967+0000 (oldest deadline 2024-03-18T15:30:21.612923+0000) 2024-03-18T15:30:35.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:35 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:35.075+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:07.414865+0000 front 2024-03-18T15:30:07.415069+0000 (oldest deadline 2024-03-18T15:30:32.714829+0000) 2024-03-18T15:30:35.654 INFO:journalctl@ceph.osd.6.smithi019.stdout:Mar 18 15:30:35 smithi019 podman[92830]: 2024-03-18 15:30:35.477213049 +0000 UTC m=+6.023093849 container cleanup 812afcb2a2c5d5032352fd06de57edbd97b45cd5f81c49099d9bd733e6d9c9fe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-6, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T15:30:35.654 INFO:journalctl@ceph.osd.6.smithi019.stdout:Mar 18 15:30:35 smithi019 bash[92830]: ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-6 2024-03-18T15:30:35.944 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:35 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:35.478+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:49.191521+0000 front 2024-03-18T15:29:49.191660+0000 (oldest deadline 2024-03-18T15:30:12.691483+0000) 2024-03-18T15:30:35.944 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:35 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:35.478+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:59.092785+0000 front 2024-03-18T15:29:59.092921+0000 (oldest deadline 2024-03-18T15:30:21.992976+0000) 2024-03-18T15:30:35.944 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:35 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:35.478+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:04.894647+0000 front 2024-03-18T15:30:04.895189+0000 (oldest deadline 2024-03-18T15:30:30.794498+0000) 2024-03-18T15:30:35.965 INFO:journalctl@ceph.osd.6.smithi019.stdout:Mar 18 15:30:35 smithi019 podman[92849]: 2024-03-18 15:30:35.653191576 +0000 UTC m=+0.626894456 container remove 812afcb2a2c5d5032352fd06de57edbd97b45cd5f81c49099d9bd733e6d9c9fe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-6, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-03-18T15:30:36.067 INFO:journalctl@ceph.osd.7.smithi033.stdout:Mar 18 15:30:35 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-7[54818]: 2024-03-18T15:30:35.775+0000 7f480d187700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:50.578091+0000 front 2024-03-18T15:29:50.578050+0000 (oldest deadline 2024-03-18T15:30:15.278156+0000) 2024-03-18T15:30:36.067 INFO:journalctl@ceph.osd.7.smithi033.stdout:Mar 18 15:30:35 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-7[54818]: 2024-03-18T15:30:35.775+0000 7f480d187700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:57.579372+0000 front 2024-03-18T15:29:57.579232+0000 (oldest deadline 2024-03-18T15:30:22.879178+0000) 2024-03-18T15:30:36.067 INFO:journalctl@ceph.osd.7.smithi033.stdout:Mar 18 15:30:35 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-7[54818]: 2024-03-18T15:30:35.775+0000 7f480d187700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:06.280853+0000 front 2024-03-18T15:30:06.280754+0000 (oldest deadline 2024-03-18T15:30:29.780617+0000) 2024-03-18T15:30:36.367 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:36 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:36.066+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.612085+0000 front 2024-03-18T15:29:51.612164+0000 (oldest deadline 2024-03-18T15:30:17.512254+0000) 2024-03-18T15:30:36.367 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:36 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:36.066+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:57.512824+0000 front 2024-03-18T15:29:57.512967+0000 (oldest deadline 2024-03-18T15:30:21.612923+0000) 2024-03-18T15:30:36.367 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:36 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:36.066+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:07.414865+0000 front 2024-03-18T15:30:07.415069+0000 (oldest deadline 2024-03-18T15:30:32.714829+0000) 2024-03-18T15:30:36.631 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:36 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:36.285+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:39.237875+0000 front 2024-03-18T15:29:39.238431+0000 (oldest deadline 2024-03-18T15:30:05.138032+0000) 2024-03-18T15:30:36.631 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:36 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:36.285+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:50.743126+0000 front 2024-03-18T15:29:50.743226+0000 (oldest deadline 2024-03-18T15:30:13.643377+0000) 2024-03-18T15:30:36.631 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:36 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:36.285+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:56.947757+0000 front 2024-03-18T15:29:56.947820+0000 (oldest deadline 2024-03-18T15:30:22.248013+0000) 2024-03-18T15:30:36.631 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:36 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:36.285+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:06.950015+0000 front 2024-03-18T15:30:06.950486+0000 (oldest deadline 2024-03-18T15:30:31.650112+0000) 2024-03-18T15:30:36.632 INFO:journalctl@ceph.osd.6.smithi019.stdout:Mar 18 15:30:36 smithi019 podman[92938]: 2024-03-18 15:30:36.35197343 +0000 UTC m=+0.038630684 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T15:30:36.632 INFO:journalctl@ceph.osd.6.smithi019.stdout:Mar 18 15:30:36 smithi019 podman[92938]: 2024-03-18 15:30:36.487799039 +0000 UTC m=+0.174456357 container create 7757a8c95655307561ad1d4c51a73d7e7e6485025779399ac727de54085b518b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-6-deactivate, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-03-18T15:30:36.695 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:36 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:36.432+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:49.191521+0000 front 2024-03-18T15:29:49.191660+0000 (oldest deadline 2024-03-18T15:30:12.691483+0000) 2024-03-18T15:30:36.695 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:36 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:36.432+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:59.092785+0000 front 2024-03-18T15:29:59.092921+0000 (oldest deadline 2024-03-18T15:30:21.992976+0000) 2024-03-18T15:30:36.695 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:36 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:36.432+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:04.894647+0000 front 2024-03-18T15:30:04.895189+0000 (oldest deadline 2024-03-18T15:30:30.794498+0000) 2024-03-18T15:30:36.695 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:36 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:36.213+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:41.404783+0000 front 2024-03-18T15:29:41.404661+0000 (oldest deadline 2024-03-18T15:30:04.304572+0000) 2024-03-18T15:30:36.695 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:36 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:36.213+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.207053+0000 front 2024-03-18T15:29:51.206898+0000 (oldest deadline 2024-03-18T15:30:13.507003+0000) 2024-03-18T15:30:36.695 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:36 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:36.213+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:59.809088+0000 front 2024-03-18T15:29:59.809253+0000 (oldest deadline 2024-03-18T15:30:21.509244+0000) 2024-03-18T15:30:36.882 INFO:journalctl@ceph.osd.6.smithi019.stdout:Mar 18 15:30:36 smithi019 podman[92938]: 2024-03-18 15:30:36.774748146 +0000 UTC m=+0.461405443 container init 7757a8c95655307561ad1d4c51a73d7e7e6485025779399ac727de54085b518b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-6-deactivate, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-03-18T15:30:36.882 INFO:journalctl@ceph.osd.6.smithi019.stdout:Mar 18 15:30:36 smithi019 podman[92938]: 2024-03-18 15:30:36.786236913 +0000 UTC m=+0.472894237 container start 7757a8c95655307561ad1d4c51a73d7e7e6485025779399ac727de54085b518b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-6-deactivate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T15:30:36.882 INFO:journalctl@ceph.osd.6.smithi019.stdout:Mar 18 15:30:36 smithi019 podman[92938]: 2024-03-18 15:30:36.881859333 +0000 UTC m=+0.568516670 container attach 7757a8c95655307561ad1d4c51a73d7e7e6485025779399ac727de54085b518b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-6-deactivate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T15:30:37.070 INFO:journalctl@ceph.osd.7.smithi033.stdout:Mar 18 15:30:36 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-7[54818]: 2024-03-18T15:30:36.814+0000 7f480d187700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:50.578091+0000 front 2024-03-18T15:29:50.578050+0000 (oldest deadline 2024-03-18T15:30:15.278156+0000) 2024-03-18T15:30:37.071 INFO:journalctl@ceph.osd.7.smithi033.stdout:Mar 18 15:30:36 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-7[54818]: 2024-03-18T15:30:36.814+0000 7f480d187700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:57.579372+0000 front 2024-03-18T15:29:57.579232+0000 (oldest deadline 2024-03-18T15:30:22.879178+0000) 2024-03-18T15:30:37.071 INFO:journalctl@ceph.osd.7.smithi033.stdout:Mar 18 15:30:36 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-7[54818]: 2024-03-18T15:30:36.814+0000 7f480d187700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:06.280853+0000 front 2024-03-18T15:30:06.280754+0000 (oldest deadline 2024-03-18T15:30:29.780617+0000) 2024-03-18T15:30:37.331 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:37 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:37.070+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.612085+0000 front 2024-03-18T15:29:51.612164+0000 (oldest deadline 2024-03-18T15:30:17.512254+0000) 2024-03-18T15:30:37.331 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:37 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:37.070+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:57.512824+0000 front 2024-03-18T15:29:57.512967+0000 (oldest deadline 2024-03-18T15:30:21.612923+0000) 2024-03-18T15:30:37.331 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:37 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:37.070+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:07.414865+0000 front 2024-03-18T15:30:07.415069+0000 (oldest deadline 2024-03-18T15:30:32.714829+0000) 2024-03-18T15:30:37.588 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:37 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:37.326+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:39.237875+0000 front 2024-03-18T15:29:39.238431+0000 (oldest deadline 2024-03-18T15:30:05.138032+0000) 2024-03-18T15:30:37.588 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:37 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:37.326+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:50.743126+0000 front 2024-03-18T15:29:50.743226+0000 (oldest deadline 2024-03-18T15:30:13.643377+0000) 2024-03-18T15:30:37.588 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:37 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:37.326+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:56.947757+0000 front 2024-03-18T15:29:56.947820+0000 (oldest deadline 2024-03-18T15:30:22.248013+0000) 2024-03-18T15:30:37.588 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:37 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:37.326+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:06.950015+0000 front 2024-03-18T15:30:06.950486+0000 (oldest deadline 2024-03-18T15:30:31.650112+0000) 2024-03-18T15:30:37.589 INFO:journalctl@ceph.osd.6.smithi019.stdout:Mar 18 15:30:37 smithi019 podman[92938]: 2024-03-18 15:30:37.236316691 +0000 UTC m=+0.922973988 container died 7757a8c95655307561ad1d4c51a73d7e7e6485025779399ac727de54085b518b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-6-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-18T15:30:37.694 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:37 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:37.397+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:49.191521+0000 front 2024-03-18T15:29:49.191660+0000 (oldest deadline 2024-03-18T15:30:12.691483+0000) 2024-03-18T15:30:37.694 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:37 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:37.397+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:59.092785+0000 front 2024-03-18T15:29:59.092921+0000 (oldest deadline 2024-03-18T15:30:21.992976+0000) 2024-03-18T15:30:37.694 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:37 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:37.397+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:04.894647+0000 front 2024-03-18T15:30:04.895189+0000 (oldest deadline 2024-03-18T15:30:30.794498+0000) 2024-03-18T15:30:37.695 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:37 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:37.259+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:41.404783+0000 front 2024-03-18T15:29:41.404661+0000 (oldest deadline 2024-03-18T15:30:04.304572+0000) 2024-03-18T15:30:37.695 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:37 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:37.259+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.207053+0000 front 2024-03-18T15:29:51.206898+0000 (oldest deadline 2024-03-18T15:30:13.507003+0000) 2024-03-18T15:30:37.695 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:37 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:37.259+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:59.809088+0000 front 2024-03-18T15:29:59.809253+0000 (oldest deadline 2024-03-18T15:30:21.509244+0000) 2024-03-18T15:30:37.942 DEBUG:teuthology.orchestra.run.smithi019:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@osd.6.service' 2024-03-18T15:30:38.074 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T15:30:38.074 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-18T15:30:38.074 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-18T15:30:38.074 DEBUG:teuthology.orchestra.run.smithi033:> sudo systemctl stop ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@osd.7 2024-03-18T15:30:38.109 INFO:journalctl@ceph.osd.7.smithi033.stdout:Mar 18 15:30:37 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-7[54818]: 2024-03-18T15:30:37.794+0000 7f480d187700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:50.578091+0000 front 2024-03-18T15:29:50.578050+0000 (oldest deadline 2024-03-18T15:30:15.278156+0000) 2024-03-18T15:30:38.109 INFO:journalctl@ceph.osd.7.smithi033.stdout:Mar 18 15:30:37 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-7[54818]: 2024-03-18T15:30:37.794+0000 7f480d187700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:57.579372+0000 front 2024-03-18T15:29:57.579232+0000 (oldest deadline 2024-03-18T15:30:22.879178+0000) 2024-03-18T15:30:38.109 INFO:journalctl@ceph.osd.7.smithi033.stdout:Mar 18 15:30:37 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-7[54818]: 2024-03-18T15:30:37.794+0000 7f480d187700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:06.280853+0000 front 2024-03-18T15:30:06.280754+0000 (oldest deadline 2024-03-18T15:30:29.780617+0000) 2024-03-18T15:30:38.110 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:38 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:38.034+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.612085+0000 front 2024-03-18T15:29:51.612164+0000 (oldest deadline 2024-03-18T15:30:17.512254+0000) 2024-03-18T15:30:38.110 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:38 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:38.034+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:57.512824+0000 front 2024-03-18T15:29:57.512967+0000 (oldest deadline 2024-03-18T15:30:21.612923+0000) 2024-03-18T15:30:38.110 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:38 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:38.034+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:07.414865+0000 front 2024-03-18T15:30:07.415069+0000 (oldest deadline 2024-03-18T15:30:32.714829+0000) 2024-03-18T15:30:38.400 INFO:journalctl@ceph.osd.7.smithi033.stdout:Mar 18 15:30:38 smithi033 systemd[1]: Stopping Ceph osd.7 for d4dc2ba6-e538-11ee-95c9-87774f69a715... 2024-03-18T15:30:38.694 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:38 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:38.400+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:49.191521+0000 front 2024-03-18T15:29:49.191660+0000 (oldest deadline 2024-03-18T15:30:12.691483+0000) 2024-03-18T15:30:38.694 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:38 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:38.400+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:59.092785+0000 front 2024-03-18T15:29:59.092921+0000 (oldest deadline 2024-03-18T15:30:21.992976+0000) 2024-03-18T15:30:38.694 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:38 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:38.400+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:04.894647+0000 front 2024-03-18T15:30:04.895189+0000 (oldest deadline 2024-03-18T15:30:30.794498+0000) 2024-03-18T15:30:38.695 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:38 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:38.276+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:41.404783+0000 front 2024-03-18T15:29:41.404661+0000 (oldest deadline 2024-03-18T15:30:04.304572+0000) 2024-03-18T15:30:38.695 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:38 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:38.276+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.207053+0000 front 2024-03-18T15:29:51.206898+0000 (oldest deadline 2024-03-18T15:30:13.507003+0000) 2024-03-18T15:30:38.695 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:38 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:38.276+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:59.809088+0000 front 2024-03-18T15:29:59.809253+0000 (oldest deadline 2024-03-18T15:30:21.509244+0000) 2024-03-18T15:30:38.715 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:38 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:38.287+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:39.237875+0000 front 2024-03-18T15:29:39.238431+0000 (oldest deadline 2024-03-18T15:30:05.138032+0000) 2024-03-18T15:30:38.715 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:38 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:38.287+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:50.743126+0000 front 2024-03-18T15:29:50.743226+0000 (oldest deadline 2024-03-18T15:30:13.643377+0000) 2024-03-18T15:30:38.715 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:38 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:38.287+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:56.947757+0000 front 2024-03-18T15:29:56.947820+0000 (oldest deadline 2024-03-18T15:30:22.248013+0000) 2024-03-18T15:30:38.716 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:38 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:38.287+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:06.950015+0000 front 2024-03-18T15:30:06.950486+0000 (oldest deadline 2024-03-18T15:30:31.650112+0000) 2024-03-18T15:30:38.756 INFO:journalctl@ceph.osd.7.smithi033.stdout:Mar 18 15:30:38 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-7[54818]: 2024-03-18T15:30:38.400+0000 7f4810b93700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T15:30:38.757 INFO:journalctl@ceph.osd.7.smithi033.stdout:Mar 18 15:30:38 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-7[54818]: 2024-03-18T15:30:38.400+0000 7f4810b93700 -1 osd.7 109 *** Got signal Terminated *** 2024-03-18T15:30:38.757 INFO:journalctl@ceph.osd.7.smithi033.stdout:Mar 18 15:30:38 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-7[54818]: 2024-03-18T15:30:38.400+0000 7f4810b93700 -1 osd.7 109 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T15:30:39.256 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:39 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:39.008+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.612085+0000 front 2024-03-18T15:29:51.612164+0000 (oldest deadline 2024-03-18T15:30:17.512254+0000) 2024-03-18T15:30:39.257 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:39 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:39.008+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:57.512824+0000 front 2024-03-18T15:29:57.512967+0000 (oldest deadline 2024-03-18T15:30:21.612923+0000) 2024-03-18T15:30:39.257 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:39 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:39.008+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:07.414865+0000 front 2024-03-18T15:30:07.415069+0000 (oldest deadline 2024-03-18T15:30:32.714829+0000) 2024-03-18T15:30:39.257 INFO:journalctl@ceph.osd.7.smithi033.stdout:Mar 18 15:30:38 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-7[54818]: 2024-03-18T15:30:38.780+0000 7f480d187700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:50.578091+0000 front 2024-03-18T15:29:50.578050+0000 (oldest deadline 2024-03-18T15:30:15.278156+0000) 2024-03-18T15:30:39.257 INFO:journalctl@ceph.osd.7.smithi033.stdout:Mar 18 15:30:38 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-7[54818]: 2024-03-18T15:30:38.780+0000 7f480d187700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:57.579372+0000 front 2024-03-18T15:29:57.579232+0000 (oldest deadline 2024-03-18T15:30:22.879178+0000) 2024-03-18T15:30:39.257 INFO:journalctl@ceph.osd.7.smithi033.stdout:Mar 18 15:30:38 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-7[54818]: 2024-03-18T15:30:38.780+0000 7f480d187700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:06.280853+0000 front 2024-03-18T15:30:06.280754+0000 (oldest deadline 2024-03-18T15:30:29.780617+0000) 2024-03-18T15:30:39.694 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:39 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:39.440+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:49.191521+0000 front 2024-03-18T15:29:49.191660+0000 (oldest deadline 2024-03-18T15:30:12.691483+0000) 2024-03-18T15:30:39.694 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:39 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:39.440+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:59.092785+0000 front 2024-03-18T15:29:59.092921+0000 (oldest deadline 2024-03-18T15:30:21.992976+0000) 2024-03-18T15:30:39.694 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:39 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:39.440+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:04.894647+0000 front 2024-03-18T15:30:04.895189+0000 (oldest deadline 2024-03-18T15:30:30.794498+0000) 2024-03-18T15:30:39.695 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:39 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:39.440+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-03-18T15:30:13.095972+0000 front 2024-03-18T15:30:13.095864+0000 (oldest deadline 2024-03-18T15:30:38.995906+0000) 2024-03-18T15:30:39.695 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:39 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:39.287+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:41.404783+0000 front 2024-03-18T15:29:41.404661+0000 (oldest deadline 2024-03-18T15:30:04.304572+0000) 2024-03-18T15:30:39.695 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:39 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:39.287+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.207053+0000 front 2024-03-18T15:29:51.206898+0000 (oldest deadline 2024-03-18T15:30:13.507003+0000) 2024-03-18T15:30:39.695 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:39 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:39.287+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:59.809088+0000 front 2024-03-18T15:29:59.809253+0000 (oldest deadline 2024-03-18T15:30:21.509244+0000) 2024-03-18T15:30:39.715 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:39 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:39.243+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:39.237875+0000 front 2024-03-18T15:29:39.238431+0000 (oldest deadline 2024-03-18T15:30:05.138032+0000) 2024-03-18T15:30:39.715 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:39 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:39.243+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:50.743126+0000 front 2024-03-18T15:29:50.743226+0000 (oldest deadline 2024-03-18T15:30:13.643377+0000) 2024-03-18T15:30:39.715 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:39 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:39.243+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:56.947757+0000 front 2024-03-18T15:29:56.947820+0000 (oldest deadline 2024-03-18T15:30:22.248013+0000) 2024-03-18T15:30:39.715 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:39 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:39.243+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:06.950015+0000 front 2024-03-18T15:30:06.950486+0000 (oldest deadline 2024-03-18T15:30:31.650112+0000) 2024-03-18T15:30:40.257 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:40 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:40.034+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.612085+0000 front 2024-03-18T15:29:51.612164+0000 (oldest deadline 2024-03-18T15:30:17.512254+0000) 2024-03-18T15:30:40.257 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:40 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:40.034+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:57.512824+0000 front 2024-03-18T15:29:57.512967+0000 (oldest deadline 2024-03-18T15:30:21.612923+0000) 2024-03-18T15:30:40.257 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:40 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:40.034+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:07.414865+0000 front 2024-03-18T15:30:07.415069+0000 (oldest deadline 2024-03-18T15:30:32.714829+0000) 2024-03-18T15:30:40.257 INFO:journalctl@ceph.osd.7.smithi033.stdout:Mar 18 15:30:39 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-7[54818]: 2024-03-18T15:30:39.809+0000 7f480d187700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:50.578091+0000 front 2024-03-18T15:29:50.578050+0000 (oldest deadline 2024-03-18T15:30:15.278156+0000) 2024-03-18T15:30:40.257 INFO:journalctl@ceph.osd.7.smithi033.stdout:Mar 18 15:30:39 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-7[54818]: 2024-03-18T15:30:39.809+0000 7f480d187700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:57.579372+0000 front 2024-03-18T15:29:57.579232+0000 (oldest deadline 2024-03-18T15:30:22.879178+0000) 2024-03-18T15:30:40.257 INFO:journalctl@ceph.osd.7.smithi033.stdout:Mar 18 15:30:39 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-7[54818]: 2024-03-18T15:30:39.809+0000 7f480d187700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:06.280853+0000 front 2024-03-18T15:30:06.280754+0000 (oldest deadline 2024-03-18T15:30:29.780617+0000) 2024-03-18T15:30:40.257 INFO:journalctl@ceph.osd.7.smithi033.stdout:Mar 18 15:30:39 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-7[54818]: 2024-03-18T15:30:39.809+0000 7f480d187700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-03-18T15:30:14.084168+0000 front 2024-03-18T15:30:14.084342+0000 (oldest deadline 2024-03-18T15:30:39.284848+0000) 2024-03-18T15:30:40.694 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:40 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:40.416+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:49.191521+0000 front 2024-03-18T15:29:49.191660+0000 (oldest deadline 2024-03-18T15:30:12.691483+0000) 2024-03-18T15:30:40.694 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:40 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:40.416+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:59.092785+0000 front 2024-03-18T15:29:59.092921+0000 (oldest deadline 2024-03-18T15:30:21.992976+0000) 2024-03-18T15:30:40.694 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:40 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:40.416+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:04.894647+0000 front 2024-03-18T15:30:04.895189+0000 (oldest deadline 2024-03-18T15:30:30.794498+0000) 2024-03-18T15:30:40.695 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:40 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:40.416+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-03-18T15:30:13.095972+0000 front 2024-03-18T15:30:13.095864+0000 (oldest deadline 2024-03-18T15:30:38.995906+0000) 2024-03-18T15:30:40.695 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:40 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:40.254+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:41.404783+0000 front 2024-03-18T15:29:41.404661+0000 (oldest deadline 2024-03-18T15:30:04.304572+0000) 2024-03-18T15:30:40.695 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:40 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:40.254+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.207053+0000 front 2024-03-18T15:29:51.206898+0000 (oldest deadline 2024-03-18T15:30:13.507003+0000) 2024-03-18T15:30:40.695 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:40 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:40.254+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:59.809088+0000 front 2024-03-18T15:29:59.809253+0000 (oldest deadline 2024-03-18T15:30:21.509244+0000) 2024-03-18T15:30:40.715 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:40 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:40.256+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:39.237875+0000 front 2024-03-18T15:29:39.238431+0000 (oldest deadline 2024-03-18T15:30:05.138032+0000) 2024-03-18T15:30:40.715 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:40 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:40.256+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:50.743126+0000 front 2024-03-18T15:29:50.743226+0000 (oldest deadline 2024-03-18T15:30:13.643377+0000) 2024-03-18T15:30:40.715 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:40 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:40.256+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:56.947757+0000 front 2024-03-18T15:29:56.947820+0000 (oldest deadline 2024-03-18T15:30:22.248013+0000) 2024-03-18T15:30:40.716 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:40 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:40.256+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:06.950015+0000 front 2024-03-18T15:30:06.950486+0000 (oldest deadline 2024-03-18T15:30:31.650112+0000) 2024-03-18T15:30:40.716 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:40 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:40.256+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-03-18T15:30:17.552108+0000 front 2024-03-18T15:30:17.552227+0000 (oldest deadline 2024-03-18T15:30:39.852162+0000) 2024-03-18T15:30:41.256 INFO:journalctl@ceph.osd.7.smithi033.stdout:Mar 18 15:30:40 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-7[54818]: 2024-03-18T15:30:40.814+0000 7f480d187700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:50.578091+0000 front 2024-03-18T15:29:50.578050+0000 (oldest deadline 2024-03-18T15:30:15.278156+0000) 2024-03-18T15:30:41.257 INFO:journalctl@ceph.osd.7.smithi033.stdout:Mar 18 15:30:40 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-7[54818]: 2024-03-18T15:30:40.814+0000 7f480d187700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:57.579372+0000 front 2024-03-18T15:29:57.579232+0000 (oldest deadline 2024-03-18T15:30:22.879178+0000) 2024-03-18T15:30:41.257 INFO:journalctl@ceph.osd.7.smithi033.stdout:Mar 18 15:30:40 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-7[54818]: 2024-03-18T15:30:40.814+0000 7f480d187700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:06.280853+0000 front 2024-03-18T15:30:06.280754+0000 (oldest deadline 2024-03-18T15:30:29.780617+0000) 2024-03-18T15:30:41.257 INFO:journalctl@ceph.osd.7.smithi033.stdout:Mar 18 15:30:40 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-7[54818]: 2024-03-18T15:30:40.814+0000 7f480d187700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-03-18T15:30:14.084168+0000 front 2024-03-18T15:30:14.084342+0000 (oldest deadline 2024-03-18T15:30:39.284848+0000) 2024-03-18T15:30:41.257 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:41 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:41.017+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.612085+0000 front 2024-03-18T15:29:51.612164+0000 (oldest deadline 2024-03-18T15:30:17.512254+0000) 2024-03-18T15:30:41.257 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:41 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:41.017+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:57.512824+0000 front 2024-03-18T15:29:57.512967+0000 (oldest deadline 2024-03-18T15:30:21.612923+0000) 2024-03-18T15:30:41.257 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:41 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:41.017+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:07.414865+0000 front 2024-03-18T15:30:07.415069+0000 (oldest deadline 2024-03-18T15:30:32.714829+0000) 2024-03-18T15:30:41.257 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:41 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:41.017+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-03-18T15:30:16.215900+0000 front 2024-03-18T15:30:16.215789+0000 (oldest deadline 2024-03-18T15:30:40.915902+0000) 2024-03-18T15:30:41.694 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:41 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:41.410+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:49.191521+0000 front 2024-03-18T15:29:49.191660+0000 (oldest deadline 2024-03-18T15:30:12.691483+0000) 2024-03-18T15:30:41.694 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:41 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:41.410+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:59.092785+0000 front 2024-03-18T15:29:59.092921+0000 (oldest deadline 2024-03-18T15:30:21.992976+0000) 2024-03-18T15:30:41.694 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:41 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:41.410+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:04.894647+0000 front 2024-03-18T15:30:04.895189+0000 (oldest deadline 2024-03-18T15:30:30.794498+0000) 2024-03-18T15:30:41.694 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:41 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:41.410+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-03-18T15:30:13.095972+0000 front 2024-03-18T15:30:13.095864+0000 (oldest deadline 2024-03-18T15:30:38.995906+0000) 2024-03-18T15:30:41.695 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:41 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:41.209+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:41.404783+0000 front 2024-03-18T15:29:41.404661+0000 (oldest deadline 2024-03-18T15:30:04.304572+0000) 2024-03-18T15:30:41.695 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:41 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:41.209+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.207053+0000 front 2024-03-18T15:29:51.206898+0000 (oldest deadline 2024-03-18T15:30:13.507003+0000) 2024-03-18T15:30:41.695 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:41 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:41.209+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:59.809088+0000 front 2024-03-18T15:29:59.809253+0000 (oldest deadline 2024-03-18T15:30:21.509244+0000) 2024-03-18T15:30:41.716 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:41 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:41.261+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:39.237875+0000 front 2024-03-18T15:29:39.238431+0000 (oldest deadline 2024-03-18T15:30:05.138032+0000) 2024-03-18T15:30:41.716 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:41 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:41.261+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:50.743126+0000 front 2024-03-18T15:29:50.743226+0000 (oldest deadline 2024-03-18T15:30:13.643377+0000) 2024-03-18T15:30:41.716 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:41 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:41.261+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:56.947757+0000 front 2024-03-18T15:29:56.947820+0000 (oldest deadline 2024-03-18T15:30:22.248013+0000) 2024-03-18T15:30:41.716 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:41 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:41.261+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:06.950015+0000 front 2024-03-18T15:30:06.950486+0000 (oldest deadline 2024-03-18T15:30:31.650112+0000) 2024-03-18T15:30:41.716 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:41 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:41.261+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-03-18T15:30:17.552108+0000 front 2024-03-18T15:30:17.552227+0000 (oldest deadline 2024-03-18T15:30:39.852162+0000) 2024-03-18T15:30:42.257 INFO:journalctl@ceph.osd.7.smithi033.stdout:Mar 18 15:30:41 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-7[54818]: 2024-03-18T15:30:41.848+0000 7f480d187700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:50.578091+0000 front 2024-03-18T15:29:50.578050+0000 (oldest deadline 2024-03-18T15:30:15.278156+0000) 2024-03-18T15:30:42.257 INFO:journalctl@ceph.osd.7.smithi033.stdout:Mar 18 15:30:41 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-7[54818]: 2024-03-18T15:30:41.848+0000 7f480d187700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:57.579372+0000 front 2024-03-18T15:29:57.579232+0000 (oldest deadline 2024-03-18T15:30:22.879178+0000) 2024-03-18T15:30:42.257 INFO:journalctl@ceph.osd.7.smithi033.stdout:Mar 18 15:30:41 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-7[54818]: 2024-03-18T15:30:41.848+0000 7f480d187700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:06.280853+0000 front 2024-03-18T15:30:06.280754+0000 (oldest deadline 2024-03-18T15:30:29.780617+0000) 2024-03-18T15:30:42.257 INFO:journalctl@ceph.osd.7.smithi033.stdout:Mar 18 15:30:41 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-7[54818]: 2024-03-18T15:30:41.848+0000 7f480d187700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-03-18T15:30:14.084168+0000 front 2024-03-18T15:30:14.084342+0000 (oldest deadline 2024-03-18T15:30:39.284848+0000) 2024-03-18T15:30:42.257 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:42 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:42.038+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.612085+0000 front 2024-03-18T15:29:51.612164+0000 (oldest deadline 2024-03-18T15:30:17.512254+0000) 2024-03-18T15:30:42.258 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:42 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:42.038+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:57.512824+0000 front 2024-03-18T15:29:57.512967+0000 (oldest deadline 2024-03-18T15:30:21.612923+0000) 2024-03-18T15:30:42.258 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:42 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:42.038+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:07.414865+0000 front 2024-03-18T15:30:07.415069+0000 (oldest deadline 2024-03-18T15:30:32.714829+0000) 2024-03-18T15:30:42.258 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:42 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:42.038+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-03-18T15:30:16.215900+0000 front 2024-03-18T15:30:16.215789+0000 (oldest deadline 2024-03-18T15:30:40.915902+0000) 2024-03-18T15:30:42.382 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:42 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:42.258+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:41.404783+0000 front 2024-03-18T15:29:41.404661+0000 (oldest deadline 2024-03-18T15:30:04.304572+0000) 2024-03-18T15:30:42.382 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:42 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:42.258+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.207053+0000 front 2024-03-18T15:29:51.206898+0000 (oldest deadline 2024-03-18T15:30:13.507003+0000) 2024-03-18T15:30:42.382 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:42 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:42.258+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:59.809088+0000 front 2024-03-18T15:29:59.809253+0000 (oldest deadline 2024-03-18T15:30:21.509244+0000) 2024-03-18T15:30:42.678 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:42 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:42.381+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:49.191521+0000 front 2024-03-18T15:29:49.191660+0000 (oldest deadline 2024-03-18T15:30:12.691483+0000) 2024-03-18T15:30:42.678 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:42 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:42.381+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:59.092785+0000 front 2024-03-18T15:29:59.092921+0000 (oldest deadline 2024-03-18T15:30:21.992976+0000) 2024-03-18T15:30:42.678 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:42 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:42.381+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:04.894647+0000 front 2024-03-18T15:30:04.895189+0000 (oldest deadline 2024-03-18T15:30:30.794498+0000) 2024-03-18T15:30:42.678 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:42 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:42.381+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-03-18T15:30:13.095972+0000 front 2024-03-18T15:30:13.095864+0000 (oldest deadline 2024-03-18T15:30:38.995906+0000) 2024-03-18T15:30:42.715 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:42 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:42.222+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:39.237875+0000 front 2024-03-18T15:29:39.238431+0000 (oldest deadline 2024-03-18T15:30:05.138032+0000) 2024-03-18T15:30:42.715 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:42 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:42.222+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:50.743126+0000 front 2024-03-18T15:29:50.743226+0000 (oldest deadline 2024-03-18T15:30:13.643377+0000) 2024-03-18T15:30:42.716 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:42 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:42.222+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:56.947757+0000 front 2024-03-18T15:29:56.947820+0000 (oldest deadline 2024-03-18T15:30:22.248013+0000) 2024-03-18T15:30:42.716 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:42 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:42.222+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:06.950015+0000 front 2024-03-18T15:30:06.950486+0000 (oldest deadline 2024-03-18T15:30:31.650112+0000) 2024-03-18T15:30:42.716 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:42 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:42.222+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-03-18T15:30:17.552108+0000 front 2024-03-18T15:30:17.552227+0000 (oldest deadline 2024-03-18T15:30:39.852162+0000) 2024-03-18T15:30:43.061 INFO:journalctl@ceph.osd.7.smithi033.stdout:Mar 18 15:30:42 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-7[54818]: 2024-03-18T15:30:42.810+0000 7f480d187700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:50.578091+0000 front 2024-03-18T15:29:50.578050+0000 (oldest deadline 2024-03-18T15:30:15.278156+0000) 2024-03-18T15:30:43.061 INFO:journalctl@ceph.osd.7.smithi033.stdout:Mar 18 15:30:42 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-7[54818]: 2024-03-18T15:30:42.810+0000 7f480d187700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:57.579372+0000 front 2024-03-18T15:29:57.579232+0000 (oldest deadline 2024-03-18T15:30:22.879178+0000) 2024-03-18T15:30:43.062 INFO:journalctl@ceph.osd.7.smithi033.stdout:Mar 18 15:30:42 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-7[54818]: 2024-03-18T15:30:42.810+0000 7f480d187700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:06.280853+0000 front 2024-03-18T15:30:06.280754+0000 (oldest deadline 2024-03-18T15:30:29.780617+0000) 2024-03-18T15:30:43.062 INFO:journalctl@ceph.osd.7.smithi033.stdout:Mar 18 15:30:42 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-7[54818]: 2024-03-18T15:30:42.810+0000 7f480d187700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-03-18T15:30:14.084168+0000 front 2024-03-18T15:30:14.084342+0000 (oldest deadline 2024-03-18T15:30:39.284848+0000) 2024-03-18T15:30:43.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:43 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:43.061+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.612085+0000 front 2024-03-18T15:29:51.612164+0000 (oldest deadline 2024-03-18T15:30:17.512254+0000) 2024-03-18T15:30:43.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:43 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:43.061+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:57.512824+0000 front 2024-03-18T15:29:57.512967+0000 (oldest deadline 2024-03-18T15:30:21.612923+0000) 2024-03-18T15:30:43.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:43 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:43.061+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:07.414865+0000 front 2024-03-18T15:30:07.415069+0000 (oldest deadline 2024-03-18T15:30:32.714829+0000) 2024-03-18T15:30:43.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:43 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:43.061+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-03-18T15:30:16.215900+0000 front 2024-03-18T15:30:16.215789+0000 (oldest deadline 2024-03-18T15:30:40.915902+0000) 2024-03-18T15:30:43.694 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:43 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:43.379+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:49.191521+0000 front 2024-03-18T15:29:49.191660+0000 (oldest deadline 2024-03-18T15:30:12.691483+0000) 2024-03-18T15:30:43.695 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:43 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:43.379+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:59.092785+0000 front 2024-03-18T15:29:59.092921+0000 (oldest deadline 2024-03-18T15:30:21.992976+0000) 2024-03-18T15:30:43.695 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:43 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:43.379+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:04.894647+0000 front 2024-03-18T15:30:04.895189+0000 (oldest deadline 2024-03-18T15:30:30.794498+0000) 2024-03-18T15:30:43.695 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:43 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:43.379+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-03-18T15:30:13.095972+0000 front 2024-03-18T15:30:13.095864+0000 (oldest deadline 2024-03-18T15:30:38.995906+0000) 2024-03-18T15:30:43.695 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:43 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:43.264+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:41.404783+0000 front 2024-03-18T15:29:41.404661+0000 (oldest deadline 2024-03-18T15:30:04.304572+0000) 2024-03-18T15:30:43.695 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:43 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:43.264+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.207053+0000 front 2024-03-18T15:29:51.206898+0000 (oldest deadline 2024-03-18T15:30:13.507003+0000) 2024-03-18T15:30:43.695 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:43 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:43.264+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:59.809088+0000 front 2024-03-18T15:29:59.809253+0000 (oldest deadline 2024-03-18T15:30:21.509244+0000) 2024-03-18T15:30:43.695 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:43 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:43.264+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-03-18T15:30:12.511708+0000 front 2024-03-18T15:30:12.511569+0000 (oldest deadline 2024-03-18T15:30:43.112056+0000) 2024-03-18T15:30:43.715 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:43 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:43.224+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:39.237875+0000 front 2024-03-18T15:29:39.238431+0000 (oldest deadline 2024-03-18T15:30:05.138032+0000) 2024-03-18T15:30:43.715 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:43 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:43.224+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:50.743126+0000 front 2024-03-18T15:29:50.743226+0000 (oldest deadline 2024-03-18T15:30:13.643377+0000) 2024-03-18T15:30:43.715 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:43 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:43.224+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:56.947757+0000 front 2024-03-18T15:29:56.947820+0000 (oldest deadline 2024-03-18T15:30:22.248013+0000) 2024-03-18T15:30:43.715 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:43 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:43.224+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:06.950015+0000 front 2024-03-18T15:30:06.950486+0000 (oldest deadline 2024-03-18T15:30:31.650112+0000) 2024-03-18T15:30:43.716 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:43 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:43.224+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-03-18T15:30:17.552108+0000 front 2024-03-18T15:30:17.552227+0000 (oldest deadline 2024-03-18T15:30:39.852162+0000) 2024-03-18T15:30:44.043 INFO:journalctl@ceph.osd.7.smithi033.stdout:Mar 18 15:30:43 smithi033 podman[74426]: 2024-03-18 15:30:43.7651316 +0000 UTC m=+5.490201061 container died 32f063c628701d13084860b9a7b2dae5611ecb0c82deb747709850ce29825ba1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307) 2024-03-18T15:30:44.402 INFO:journalctl@ceph.osd.7.smithi033.stdout:Mar 18 15:30:44 smithi033 podman[74426]: 2024-03-18 15:30:44.125654267 +0000 UTC m=+5.850723720 container cleanup 32f063c628701d13084860b9a7b2dae5611ecb0c82deb747709850ce29825ba1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-7, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5) 2024-03-18T15:30:44.402 INFO:journalctl@ceph.osd.7.smithi033.stdout:Mar 18 15:30:44 smithi033 bash[74426]: ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-7 2024-03-18T15:30:44.402 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:44 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:44.081+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.612085+0000 front 2024-03-18T15:29:51.612164+0000 (oldest deadline 2024-03-18T15:30:17.512254+0000) 2024-03-18T15:30:44.402 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:44 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:44.081+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:57.512824+0000 front 2024-03-18T15:29:57.512967+0000 (oldest deadline 2024-03-18T15:30:21.612923+0000) 2024-03-18T15:30:44.402 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:44 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:44.081+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:07.414865+0000 front 2024-03-18T15:30:07.415069+0000 (oldest deadline 2024-03-18T15:30:32.714829+0000) 2024-03-18T15:30:44.403 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:44 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:44.081+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-03-18T15:30:16.215900+0000 front 2024-03-18T15:30:16.215789+0000 (oldest deadline 2024-03-18T15:30:40.915902+0000) 2024-03-18T15:30:44.694 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:44 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:44.378+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:49.191521+0000 front 2024-03-18T15:29:49.191660+0000 (oldest deadline 2024-03-18T15:30:12.691483+0000) 2024-03-18T15:30:44.694 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:44 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:44.378+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:59.092785+0000 front 2024-03-18T15:29:59.092921+0000 (oldest deadline 2024-03-18T15:30:21.992976+0000) 2024-03-18T15:30:44.695 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:44 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:44.378+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:04.894647+0000 front 2024-03-18T15:30:04.895189+0000 (oldest deadline 2024-03-18T15:30:30.794498+0000) 2024-03-18T15:30:44.695 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:44 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:44.378+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-03-18T15:30:13.095972+0000 front 2024-03-18T15:30:13.095864+0000 (oldest deadline 2024-03-18T15:30:38.995906+0000) 2024-03-18T15:30:44.695 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:44 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:44.285+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:41.404783+0000 front 2024-03-18T15:29:41.404661+0000 (oldest deadline 2024-03-18T15:30:04.304572+0000) 2024-03-18T15:30:44.695 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:44 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:44.285+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.207053+0000 front 2024-03-18T15:29:51.206898+0000 (oldest deadline 2024-03-18T15:30:13.507003+0000) 2024-03-18T15:30:44.695 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:44 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:44.285+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:59.809088+0000 front 2024-03-18T15:29:59.809253+0000 (oldest deadline 2024-03-18T15:30:21.509244+0000) 2024-03-18T15:30:44.695 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:44 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:44.285+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-03-18T15:30:12.511708+0000 front 2024-03-18T15:30:12.511569+0000 (oldest deadline 2024-03-18T15:30:43.112056+0000) 2024-03-18T15:30:44.715 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:44 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:44.269+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:39.237875+0000 front 2024-03-18T15:29:39.238431+0000 (oldest deadline 2024-03-18T15:30:05.138032+0000) 2024-03-18T15:30:44.715 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:44 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:44.269+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:50.743126+0000 front 2024-03-18T15:29:50.743226+0000 (oldest deadline 2024-03-18T15:30:13.643377+0000) 2024-03-18T15:30:44.715 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:44 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:44.269+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:56.947757+0000 front 2024-03-18T15:29:56.947820+0000 (oldest deadline 2024-03-18T15:30:22.248013+0000) 2024-03-18T15:30:44.715 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:44 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:44.269+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:06.950015+0000 front 2024-03-18T15:30:06.950486+0000 (oldest deadline 2024-03-18T15:30:31.650112+0000) 2024-03-18T15:30:44.716 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:44 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:44.269+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-03-18T15:30:17.552108+0000 front 2024-03-18T15:30:17.552227+0000 (oldest deadline 2024-03-18T15:30:39.852162+0000) 2024-03-18T15:30:44.748 INFO:journalctl@ceph.osd.7.smithi033.stdout:Mar 18 15:30:44 smithi033 podman[74443]: 2024-03-18 15:30:44.401984619 +0000 UTC m=+0.631970510 container remove 32f063c628701d13084860b9a7b2dae5611ecb0c82deb747709850ce29825ba1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-7, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, ceph=True) 2024-03-18T15:30:45.006 INFO:journalctl@ceph.osd.7.smithi033.stdout:Mar 18 15:30:44 smithi033 podman[74528]: 2024-03-18 15:30:44.651766294 +0000 UTC m=+0.018039665 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T15:30:45.006 INFO:journalctl@ceph.osd.7.smithi033.stdout:Mar 18 15:30:44 smithi033 podman[74528]: 2024-03-18 15:30:44.869448866 +0000 UTC m=+0.235722214 container create db56507f2428eff731ea0f74c7e35ae0059e3035c82e24995c8d4db626261bdf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-7-deactivate, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-18T15:30:45.400 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:45 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:45.064+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.612085+0000 front 2024-03-18T15:29:51.612164+0000 (oldest deadline 2024-03-18T15:30:17.512254+0000) 2024-03-18T15:30:45.400 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:45 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:45.064+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:57.512824+0000 front 2024-03-18T15:29:57.512967+0000 (oldest deadline 2024-03-18T15:30:21.612923+0000) 2024-03-18T15:30:45.400 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:45 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:45.064+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:07.414865+0000 front 2024-03-18T15:30:07.415069+0000 (oldest deadline 2024-03-18T15:30:32.714829+0000) 2024-03-18T15:30:45.400 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:45 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:45.064+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-03-18T15:30:16.215900+0000 front 2024-03-18T15:30:16.215789+0000 (oldest deadline 2024-03-18T15:30:40.915902+0000) 2024-03-18T15:30:45.400 INFO:journalctl@ceph.osd.7.smithi033.stdout:Mar 18 15:30:45 smithi033 podman[74528]: 2024-03-18 15:30:45.097051857 +0000 UTC m=+0.463325230 container init db56507f2428eff731ea0f74c7e35ae0059e3035c82e24995c8d4db626261bdf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-7-deactivate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, ceph=True) 2024-03-18T15:30:45.400 INFO:journalctl@ceph.osd.7.smithi033.stdout:Mar 18 15:30:45 smithi033 podman[74528]: 2024-03-18 15:30:45.10058966 +0000 UTC m=+0.466863010 container start db56507f2428eff731ea0f74c7e35ae0059e3035c82e24995c8d4db626261bdf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-7-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS) 2024-03-18T15:30:45.400 INFO:journalctl@ceph.osd.7.smithi033.stdout:Mar 18 15:30:45 smithi033 podman[74528]: 2024-03-18 15:30:45.197036436 +0000 UTC m=+0.563309797 container attach db56507f2428eff731ea0f74c7e35ae0059e3035c82e24995c8d4db626261bdf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-7-deactivate, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-18T15:30:45.694 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:45 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:45.398+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:49.191521+0000 front 2024-03-18T15:29:49.191660+0000 (oldest deadline 2024-03-18T15:30:12.691483+0000) 2024-03-18T15:30:45.694 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:45 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:45.398+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:59.092785+0000 front 2024-03-18T15:29:59.092921+0000 (oldest deadline 2024-03-18T15:30:21.992976+0000) 2024-03-18T15:30:45.695 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:45 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:45.398+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:04.894647+0000 front 2024-03-18T15:30:04.895189+0000 (oldest deadline 2024-03-18T15:30:30.794498+0000) 2024-03-18T15:30:45.695 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:45 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:45.398+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-03-18T15:30:13.095972+0000 front 2024-03-18T15:30:13.095864+0000 (oldest deadline 2024-03-18T15:30:38.995906+0000) 2024-03-18T15:30:45.695 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:45 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:45.238+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:41.404783+0000 front 2024-03-18T15:29:41.404661+0000 (oldest deadline 2024-03-18T15:30:04.304572+0000) 2024-03-18T15:30:45.695 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:45 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:45.238+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.207053+0000 front 2024-03-18T15:29:51.206898+0000 (oldest deadline 2024-03-18T15:30:13.507003+0000) 2024-03-18T15:30:45.695 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:45 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:45.238+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:59.809088+0000 front 2024-03-18T15:29:59.809253+0000 (oldest deadline 2024-03-18T15:30:21.509244+0000) 2024-03-18T15:30:45.695 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:45 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:45.238+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-03-18T15:30:12.511708+0000 front 2024-03-18T15:30:12.511569+0000 (oldest deadline 2024-03-18T15:30:43.112056+0000) 2024-03-18T15:30:45.699 INFO:journalctl@ceph.osd.7.smithi033.stdout:Mar 18 15:30:45 smithi033 podman[74528]: 2024-03-18 15:30:45.400104372 +0000 UTC m=+0.766377729 container died db56507f2428eff731ea0f74c7e35ae0059e3035c82e24995c8d4db626261bdf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T15:30:45.715 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:45 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:45.302+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:39.237875+0000 front 2024-03-18T15:29:39.238431+0000 (oldest deadline 2024-03-18T15:30:05.138032+0000) 2024-03-18T15:30:45.715 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:45 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:45.302+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:50.743126+0000 front 2024-03-18T15:29:50.743226+0000 (oldest deadline 2024-03-18T15:30:13.643377+0000) 2024-03-18T15:30:45.715 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:45 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:45.302+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:56.947757+0000 front 2024-03-18T15:29:56.947820+0000 (oldest deadline 2024-03-18T15:30:22.248013+0000) 2024-03-18T15:30:45.715 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:45 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:45.302+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:06.950015+0000 front 2024-03-18T15:30:06.950486+0000 (oldest deadline 2024-03-18T15:30:31.650112+0000) 2024-03-18T15:30:45.715 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:45 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:45.302+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-03-18T15:30:17.552108+0000 front 2024-03-18T15:30:17.552227+0000 (oldest deadline 2024-03-18T15:30:39.852162+0000) 2024-03-18T15:30:45.964 INFO:journalctl@ceph.osd.7.smithi033.stdout:Mar 18 15:30:45 smithi033 podman[74554]: 2024-03-18 15:30:45.942652746 +0000 UTC m=+0.533475291 container remove db56507f2428eff731ea0f74c7e35ae0059e3035c82e24995c8d4db626261bdf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-7-deactivate, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0) 2024-03-18T15:30:45.970 DEBUG:teuthology.orchestra.run.smithi033:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@osd.7.service' 2024-03-18T15:30:46.059 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T15:30:46.059 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-18T15:30:46.059 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-03-18T15:30:46.059 DEBUG:teuthology.orchestra.run.smithi104:> sudo systemctl stop ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@osd.8 2024-03-18T15:30:46.256 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:46 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:46.056+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.612085+0000 front 2024-03-18T15:29:51.612164+0000 (oldest deadline 2024-03-18T15:30:17.512254+0000) 2024-03-18T15:30:46.257 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:46 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:46.056+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:57.512824+0000 front 2024-03-18T15:29:57.512967+0000 (oldest deadline 2024-03-18T15:30:21.612923+0000) 2024-03-18T15:30:46.257 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:46 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:46.056+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:07.414865+0000 front 2024-03-18T15:30:07.415069+0000 (oldest deadline 2024-03-18T15:30:32.714829+0000) 2024-03-18T15:30:46.257 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:46 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:46.056+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-03-18T15:30:16.215900+0000 front 2024-03-18T15:30:16.215789+0000 (oldest deadline 2024-03-18T15:30:40.915902+0000) 2024-03-18T15:30:46.365 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:46 smithi104 systemd[1]: Stopping Ceph osd.8 for d4dc2ba6-e538-11ee-95c9-87774f69a715... 2024-03-18T15:30:46.366 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:46 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:46.223+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:41.404783+0000 front 2024-03-18T15:29:41.404661+0000 (oldest deadline 2024-03-18T15:30:04.304572+0000) 2024-03-18T15:30:46.366 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:46 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:46.223+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.207053+0000 front 2024-03-18T15:29:51.206898+0000 (oldest deadline 2024-03-18T15:30:13.507003+0000) 2024-03-18T15:30:46.366 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:46 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:46.223+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:59.809088+0000 front 2024-03-18T15:29:59.809253+0000 (oldest deadline 2024-03-18T15:30:21.509244+0000) 2024-03-18T15:30:46.366 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:46 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:46.223+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-03-18T15:30:12.511708+0000 front 2024-03-18T15:30:12.511569+0000 (oldest deadline 2024-03-18T15:30:43.112056+0000) 2024-03-18T15:30:46.694 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:46 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:46.365+0000 7fb008e02700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T15:30:46.694 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:46 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:46.365+0000 7fb008e02700 -1 osd.8 109 *** Got signal Terminated *** 2024-03-18T15:30:46.694 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:46 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:46.365+0000 7fb008e02700 -1 osd.8 109 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T15:30:46.695 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:46 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:46.439+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:49.191521+0000 front 2024-03-18T15:29:49.191660+0000 (oldest deadline 2024-03-18T15:30:12.691483+0000) 2024-03-18T15:30:46.695 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:46 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:46.439+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:59.092785+0000 front 2024-03-18T15:29:59.092921+0000 (oldest deadline 2024-03-18T15:30:21.992976+0000) 2024-03-18T15:30:46.695 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:46 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:46.439+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:04.894647+0000 front 2024-03-18T15:30:04.895189+0000 (oldest deadline 2024-03-18T15:30:30.794498+0000) 2024-03-18T15:30:46.695 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:46 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:46.439+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-03-18T15:30:13.095972+0000 front 2024-03-18T15:30:13.095864+0000 (oldest deadline 2024-03-18T15:30:38.995906+0000) 2024-03-18T15:30:46.715 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:46 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:46.311+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:39.237875+0000 front 2024-03-18T15:29:39.238431+0000 (oldest deadline 2024-03-18T15:30:05.138032+0000) 2024-03-18T15:30:46.715 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:46 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:46.311+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:50.743126+0000 front 2024-03-18T15:29:50.743226+0000 (oldest deadline 2024-03-18T15:30:13.643377+0000) 2024-03-18T15:30:46.715 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:46 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:46.311+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:56.947757+0000 front 2024-03-18T15:29:56.947820+0000 (oldest deadline 2024-03-18T15:30:22.248013+0000) 2024-03-18T15:30:46.716 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:46 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:46.311+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:06.950015+0000 front 2024-03-18T15:30:06.950486+0000 (oldest deadline 2024-03-18T15:30:31.650112+0000) 2024-03-18T15:30:46.716 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:46 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:46.311+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-03-18T15:30:17.552108+0000 front 2024-03-18T15:30:17.552227+0000 (oldest deadline 2024-03-18T15:30:39.852162+0000) 2024-03-18T15:30:47.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:47 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:47.019+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.612085+0000 front 2024-03-18T15:29:51.612164+0000 (oldest deadline 2024-03-18T15:30:17.512254+0000) 2024-03-18T15:30:47.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:47 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:47.019+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:57.512824+0000 front 2024-03-18T15:29:57.512967+0000 (oldest deadline 2024-03-18T15:30:21.612923+0000) 2024-03-18T15:30:47.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:47 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:47.019+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:07.414865+0000 front 2024-03-18T15:30:07.415069+0000 (oldest deadline 2024-03-18T15:30:32.714829+0000) 2024-03-18T15:30:47.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:47 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:47.019+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-03-18T15:30:16.215900+0000 front 2024-03-18T15:30:16.215789+0000 (oldest deadline 2024-03-18T15:30:40.915902+0000) 2024-03-18T15:30:47.617 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:47 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:47.274+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:39.237875+0000 front 2024-03-18T15:29:39.238431+0000 (oldest deadline 2024-03-18T15:30:05.138032+0000) 2024-03-18T15:30:47.618 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:47 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:47.274+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:50.743126+0000 front 2024-03-18T15:29:50.743226+0000 (oldest deadline 2024-03-18T15:30:13.643377+0000) 2024-03-18T15:30:47.618 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:47 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:47.274+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:56.947757+0000 front 2024-03-18T15:29:56.947820+0000 (oldest deadline 2024-03-18T15:30:22.248013+0000) 2024-03-18T15:30:47.618 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:47 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:47.274+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:06.950015+0000 front 2024-03-18T15:30:06.950486+0000 (oldest deadline 2024-03-18T15:30:31.650112+0000) 2024-03-18T15:30:47.618 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:47 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:47.274+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-03-18T15:30:17.552108+0000 front 2024-03-18T15:30:17.552227+0000 (oldest deadline 2024-03-18T15:30:39.852162+0000) 2024-03-18T15:30:47.694 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:47 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:47.416+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:49.191521+0000 front 2024-03-18T15:29:49.191660+0000 (oldest deadline 2024-03-18T15:30:12.691483+0000) 2024-03-18T15:30:47.694 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:47 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:47.416+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:59.092785+0000 front 2024-03-18T15:29:59.092921+0000 (oldest deadline 2024-03-18T15:30:21.992976+0000) 2024-03-18T15:30:47.695 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:47 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:47.416+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:04.894647+0000 front 2024-03-18T15:30:04.895189+0000 (oldest deadline 2024-03-18T15:30:30.794498+0000) 2024-03-18T15:30:47.695 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:47 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:47.416+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-03-18T15:30:13.095972+0000 front 2024-03-18T15:30:13.095864+0000 (oldest deadline 2024-03-18T15:30:38.995906+0000) 2024-03-18T15:30:47.695 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:47 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:47.416+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2024-03-18T15:30:21.896847+0000 front 2024-03-18T15:30:21.896877+0000 (oldest deadline 2024-03-18T15:30:47.196970+0000) 2024-03-18T15:30:47.695 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:47 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:47.196+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:41.404783+0000 front 2024-03-18T15:29:41.404661+0000 (oldest deadline 2024-03-18T15:30:04.304572+0000) 2024-03-18T15:30:47.695 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:47 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:47.196+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.207053+0000 front 2024-03-18T15:29:51.206898+0000 (oldest deadline 2024-03-18T15:30:13.507003+0000) 2024-03-18T15:30:47.695 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:47 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:47.196+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:59.809088+0000 front 2024-03-18T15:29:59.809253+0000 (oldest deadline 2024-03-18T15:30:21.509244+0000) 2024-03-18T15:30:47.695 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:47 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:47.196+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-03-18T15:30:12.511708+0000 front 2024-03-18T15:30:12.511569+0000 (oldest deadline 2024-03-18T15:30:43.112056+0000) 2024-03-18T15:30:48.323 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:48 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:48.225+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:39.237875+0000 front 2024-03-18T15:29:39.238431+0000 (oldest deadline 2024-03-18T15:30:05.138032+0000) 2024-03-18T15:30:48.323 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:48 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:48.225+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:50.743126+0000 front 2024-03-18T15:29:50.743226+0000 (oldest deadline 2024-03-18T15:30:13.643377+0000) 2024-03-18T15:30:48.323 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:48 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:48.225+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:56.947757+0000 front 2024-03-18T15:29:56.947820+0000 (oldest deadline 2024-03-18T15:30:22.248013+0000) 2024-03-18T15:30:48.324 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:48 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:48.225+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:06.950015+0000 front 2024-03-18T15:30:06.950486+0000 (oldest deadline 2024-03-18T15:30:31.650112+0000) 2024-03-18T15:30:48.324 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:48 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:48.225+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-03-18T15:30:17.552108+0000 front 2024-03-18T15:30:17.552227+0000 (oldest deadline 2024-03-18T15:30:39.852162+0000) 2024-03-18T15:30:48.324 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:48 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:48.225+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2024-03-18T15:30:25.655158+0000 front 2024-03-18T15:30:25.654982+0000 (oldest deadline 2024-03-18T15:30:47.354642+0000) 2024-03-18T15:30:48.444 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:48 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:48.411+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:49.191521+0000 front 2024-03-18T15:29:49.191660+0000 (oldest deadline 2024-03-18T15:30:12.691483+0000) 2024-03-18T15:30:48.444 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:48 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:48.411+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:59.092785+0000 front 2024-03-18T15:29:59.092921+0000 (oldest deadline 2024-03-18T15:30:21.992976+0000) 2024-03-18T15:30:48.444 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:48 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:48.411+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:04.894647+0000 front 2024-03-18T15:30:04.895189+0000 (oldest deadline 2024-03-18T15:30:30.794498+0000) 2024-03-18T15:30:48.444 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:48 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:48.411+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-03-18T15:30:13.095972+0000 front 2024-03-18T15:30:13.095864+0000 (oldest deadline 2024-03-18T15:30:38.995906+0000) 2024-03-18T15:30:48.445 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:48 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:48.411+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2024-03-18T15:30:21.896847+0000 front 2024-03-18T15:30:21.896877+0000 (oldest deadline 2024-03-18T15:30:47.196970+0000) 2024-03-18T15:30:48.445 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:48 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:48.173+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:41.404783+0000 front 2024-03-18T15:29:41.404661+0000 (oldest deadline 2024-03-18T15:30:04.304572+0000) 2024-03-18T15:30:48.445 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:48 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:48.173+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.207053+0000 front 2024-03-18T15:29:51.206898+0000 (oldest deadline 2024-03-18T15:30:13.507003+0000) 2024-03-18T15:30:48.445 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:48 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:48.173+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:59.809088+0000 front 2024-03-18T15:29:59.809253+0000 (oldest deadline 2024-03-18T15:30:21.509244+0000) 2024-03-18T15:30:48.445 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:48 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:48.173+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-03-18T15:30:12.511708+0000 front 2024-03-18T15:30:12.511569+0000 (oldest deadline 2024-03-18T15:30:43.112056+0000) 2024-03-18T15:30:48.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:48 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:48.030+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.612085+0000 front 2024-03-18T15:29:51.612164+0000 (oldest deadline 2024-03-18T15:30:17.512254+0000) 2024-03-18T15:30:48.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:48 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:48.030+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:57.512824+0000 front 2024-03-18T15:29:57.512967+0000 (oldest deadline 2024-03-18T15:30:21.612923+0000) 2024-03-18T15:30:48.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:48 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:48.030+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:07.414865+0000 front 2024-03-18T15:30:07.415069+0000 (oldest deadline 2024-03-18T15:30:32.714829+0000) 2024-03-18T15:30:48.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:48 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:48.030+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-03-18T15:30:16.215900+0000 front 2024-03-18T15:30:16.215789+0000 (oldest deadline 2024-03-18T15:30:40.915902+0000) 2024-03-18T15:30:48.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:48 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:48.030+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2024-03-18T15:30:25.016948+0000 front 2024-03-18T15:30:25.016983+0000 (oldest deadline 2024-03-18T15:30:47.916909+0000) 2024-03-18T15:30:49.444 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:49 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:49.380+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:49.191521+0000 front 2024-03-18T15:29:49.191660+0000 (oldest deadline 2024-03-18T15:30:12.691483+0000) 2024-03-18T15:30:49.444 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:49 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:49.380+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:59.092785+0000 front 2024-03-18T15:29:59.092921+0000 (oldest deadline 2024-03-18T15:30:21.992976+0000) 2024-03-18T15:30:49.444 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:49 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:49.380+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:04.894647+0000 front 2024-03-18T15:30:04.895189+0000 (oldest deadline 2024-03-18T15:30:30.794498+0000) 2024-03-18T15:30:49.445 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:49 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:49.380+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-03-18T15:30:13.095972+0000 front 2024-03-18T15:30:13.095864+0000 (oldest deadline 2024-03-18T15:30:38.995906+0000) 2024-03-18T15:30:49.445 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:49 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:49.380+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2024-03-18T15:30:21.896847+0000 front 2024-03-18T15:30:21.896877+0000 (oldest deadline 2024-03-18T15:30:47.196970+0000) 2024-03-18T15:30:49.445 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:49 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:49.191+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:41.404783+0000 front 2024-03-18T15:29:41.404661+0000 (oldest deadline 2024-03-18T15:30:04.304572+0000) 2024-03-18T15:30:49.445 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:49 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:49.191+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.207053+0000 front 2024-03-18T15:29:51.206898+0000 (oldest deadline 2024-03-18T15:30:13.507003+0000) 2024-03-18T15:30:49.445 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:49 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:49.191+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:59.809088+0000 front 2024-03-18T15:29:59.809253+0000 (oldest deadline 2024-03-18T15:30:21.509244+0000) 2024-03-18T15:30:49.445 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:49 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:49.191+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-03-18T15:30:12.511708+0000 front 2024-03-18T15:30:12.511569+0000 (oldest deadline 2024-03-18T15:30:43.112056+0000) 2024-03-18T15:30:49.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:49 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:49.015+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.612085+0000 front 2024-03-18T15:29:51.612164+0000 (oldest deadline 2024-03-18T15:30:17.512254+0000) 2024-03-18T15:30:49.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:49 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:49.015+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:57.512824+0000 front 2024-03-18T15:29:57.512967+0000 (oldest deadline 2024-03-18T15:30:21.612923+0000) 2024-03-18T15:30:49.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:49 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:49.015+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:07.414865+0000 front 2024-03-18T15:30:07.415069+0000 (oldest deadline 2024-03-18T15:30:32.714829+0000) 2024-03-18T15:30:49.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:49 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:49.015+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-03-18T15:30:16.215900+0000 front 2024-03-18T15:30:16.215789+0000 (oldest deadline 2024-03-18T15:30:40.915902+0000) 2024-03-18T15:30:49.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:49 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:49.015+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2024-03-18T15:30:25.016948+0000 front 2024-03-18T15:30:25.016983+0000 (oldest deadline 2024-03-18T15:30:47.916909+0000) 2024-03-18T15:30:49.715 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:49 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:49.222+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:39.237875+0000 front 2024-03-18T15:29:39.238431+0000 (oldest deadline 2024-03-18T15:30:05.138032+0000) 2024-03-18T15:30:49.716 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:49 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:49.222+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:50.743126+0000 front 2024-03-18T15:29:50.743226+0000 (oldest deadline 2024-03-18T15:30:13.643377+0000) 2024-03-18T15:30:49.716 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:49 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:49.222+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:56.947757+0000 front 2024-03-18T15:29:56.947820+0000 (oldest deadline 2024-03-18T15:30:22.248013+0000) 2024-03-18T15:30:49.716 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:49 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:49.222+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:06.950015+0000 front 2024-03-18T15:30:06.950486+0000 (oldest deadline 2024-03-18T15:30:31.650112+0000) 2024-03-18T15:30:49.716 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:49 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:49.222+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-03-18T15:30:17.552108+0000 front 2024-03-18T15:30:17.552227+0000 (oldest deadline 2024-03-18T15:30:39.852162+0000) 2024-03-18T15:30:49.716 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:49 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:49.222+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2024-03-18T15:30:25.655158+0000 front 2024-03-18T15:30:25.654982+0000 (oldest deadline 2024-03-18T15:30:47.354642+0000) 2024-03-18T15:30:50.444 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:50 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:50.372+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:49.191521+0000 front 2024-03-18T15:29:49.191660+0000 (oldest deadline 2024-03-18T15:30:12.691483+0000) 2024-03-18T15:30:50.444 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:50 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:50.372+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:59.092785+0000 front 2024-03-18T15:29:59.092921+0000 (oldest deadline 2024-03-18T15:30:21.992976+0000) 2024-03-18T15:30:50.445 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:50 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:50.372+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:04.894647+0000 front 2024-03-18T15:30:04.895189+0000 (oldest deadline 2024-03-18T15:30:30.794498+0000) 2024-03-18T15:30:50.445 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:50 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:50.372+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-03-18T15:30:13.095972+0000 front 2024-03-18T15:30:13.095864+0000 (oldest deadline 2024-03-18T15:30:38.995906+0000) 2024-03-18T15:30:50.445 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:50 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8[54873]: 2024-03-18T15:30:50.372+0000 7fb0013ee700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2024-03-18T15:30:21.896847+0000 front 2024-03-18T15:30:21.896877+0000 (oldest deadline 2024-03-18T15:30:47.196970+0000) 2024-03-18T15:30:50.445 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:50 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:50.187+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:41.404783+0000 front 2024-03-18T15:29:41.404661+0000 (oldest deadline 2024-03-18T15:30:04.304572+0000) 2024-03-18T15:30:50.445 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:50 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:50.187+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.207053+0000 front 2024-03-18T15:29:51.206898+0000 (oldest deadline 2024-03-18T15:30:13.507003+0000) 2024-03-18T15:30:50.445 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:50 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:50.187+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:59.809088+0000 front 2024-03-18T15:29:59.809253+0000 (oldest deadline 2024-03-18T15:30:21.509244+0000) 2024-03-18T15:30:50.445 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:50 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:50.187+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-03-18T15:30:12.511708+0000 front 2024-03-18T15:30:12.511569+0000 (oldest deadline 2024-03-18T15:30:43.112056+0000) 2024-03-18T15:30:50.445 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:50 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:50.187+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2024-03-18T15:30:25.913763+0000 front 2024-03-18T15:30:25.913855+0000 (oldest deadline 2024-03-18T15:30:49.413885+0000) 2024-03-18T15:30:50.465 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:50 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:50.181+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:39.237875+0000 front 2024-03-18T15:29:39.238431+0000 (oldest deadline 2024-03-18T15:30:05.138032+0000) 2024-03-18T15:30:50.465 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:50 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:50.181+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:50.743126+0000 front 2024-03-18T15:29:50.743226+0000 (oldest deadline 2024-03-18T15:30:13.643377+0000) 2024-03-18T15:30:50.465 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:50 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:50.181+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:56.947757+0000 front 2024-03-18T15:29:56.947820+0000 (oldest deadline 2024-03-18T15:30:22.248013+0000) 2024-03-18T15:30:50.465 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:50 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:50.181+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:06.950015+0000 front 2024-03-18T15:30:06.950486+0000 (oldest deadline 2024-03-18T15:30:31.650112+0000) 2024-03-18T15:30:50.465 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:50 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:50.181+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-03-18T15:30:17.552108+0000 front 2024-03-18T15:30:17.552227+0000 (oldest deadline 2024-03-18T15:30:39.852162+0000) 2024-03-18T15:30:50.466 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:50 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:50.181+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2024-03-18T15:30:25.655158+0000 front 2024-03-18T15:30:25.654982+0000 (oldest deadline 2024-03-18T15:30:47.354642+0000) 2024-03-18T15:30:50.506 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:50 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:50.045+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.612085+0000 front 2024-03-18T15:29:51.612164+0000 (oldest deadline 2024-03-18T15:30:17.512254+0000) 2024-03-18T15:30:50.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:50 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:50.045+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:57.512824+0000 front 2024-03-18T15:29:57.512967+0000 (oldest deadline 2024-03-18T15:30:21.612923+0000) 2024-03-18T15:30:50.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:50 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:50.045+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:07.414865+0000 front 2024-03-18T15:30:07.415069+0000 (oldest deadline 2024-03-18T15:30:32.714829+0000) 2024-03-18T15:30:50.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:50 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:50.045+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-03-18T15:30:16.215900+0000 front 2024-03-18T15:30:16.215789+0000 (oldest deadline 2024-03-18T15:30:40.915902+0000) 2024-03-18T15:30:50.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:50 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:50.045+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2024-03-18T15:30:25.016948+0000 front 2024-03-18T15:30:25.016983+0000 (oldest deadline 2024-03-18T15:30:47.916909+0000) 2024-03-18T15:30:51.465 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:51 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:51.198+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:39.237875+0000 front 2024-03-18T15:29:39.238431+0000 (oldest deadline 2024-03-18T15:30:05.138032+0000) 2024-03-18T15:30:51.466 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:51 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:51.198+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:50.743126+0000 front 2024-03-18T15:29:50.743226+0000 (oldest deadline 2024-03-18T15:30:13.643377+0000) 2024-03-18T15:30:51.466 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:51 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:51.198+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:56.947757+0000 front 2024-03-18T15:29:56.947820+0000 (oldest deadline 2024-03-18T15:30:22.248013+0000) 2024-03-18T15:30:51.466 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:51 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:51.198+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:06.950015+0000 front 2024-03-18T15:30:06.950486+0000 (oldest deadline 2024-03-18T15:30:31.650112+0000) 2024-03-18T15:30:51.467 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:51 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:51.198+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-03-18T15:30:17.552108+0000 front 2024-03-18T15:30:17.552227+0000 (oldest deadline 2024-03-18T15:30:39.852162+0000) 2024-03-18T15:30:51.467 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:51 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:51.198+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2024-03-18T15:30:25.655158+0000 front 2024-03-18T15:30:25.654982+0000 (oldest deadline 2024-03-18T15:30:47.354642+0000) 2024-03-18T15:30:51.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:51 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:51.068+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.612085+0000 front 2024-03-18T15:29:51.612164+0000 (oldest deadline 2024-03-18T15:30:17.512254+0000) 2024-03-18T15:30:51.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:51 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:51.068+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:57.512824+0000 front 2024-03-18T15:29:57.512967+0000 (oldest deadline 2024-03-18T15:30:21.612923+0000) 2024-03-18T15:30:51.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:51 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:51.068+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:07.414865+0000 front 2024-03-18T15:30:07.415069+0000 (oldest deadline 2024-03-18T15:30:32.714829+0000) 2024-03-18T15:30:51.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:51 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:51.068+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-03-18T15:30:16.215900+0000 front 2024-03-18T15:30:16.215789+0000 (oldest deadline 2024-03-18T15:30:40.915902+0000) 2024-03-18T15:30:51.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:51 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:51.068+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2024-03-18T15:30:25.016948+0000 front 2024-03-18T15:30:25.016983+0000 (oldest deadline 2024-03-18T15:30:47.916909+0000) 2024-03-18T15:30:51.694 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:51 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:51.205+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:41.404783+0000 front 2024-03-18T15:29:41.404661+0000 (oldest deadline 2024-03-18T15:30:04.304572+0000) 2024-03-18T15:30:51.695 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:51 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:51.205+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.207053+0000 front 2024-03-18T15:29:51.206898+0000 (oldest deadline 2024-03-18T15:30:13.507003+0000) 2024-03-18T15:30:51.695 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:51 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:51.205+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:59.809088+0000 front 2024-03-18T15:29:59.809253+0000 (oldest deadline 2024-03-18T15:30:21.509244+0000) 2024-03-18T15:30:51.695 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:51 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:51.205+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-03-18T15:30:12.511708+0000 front 2024-03-18T15:30:12.511569+0000 (oldest deadline 2024-03-18T15:30:43.112056+0000) 2024-03-18T15:30:51.695 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:51 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:51.205+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2024-03-18T15:30:25.913763+0000 front 2024-03-18T15:30:25.913855+0000 (oldest deadline 2024-03-18T15:30:49.413885+0000) 2024-03-18T15:30:52.032 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:51 smithi104 podman[73923]: 2024-03-18 15:30:51.743167428 +0000 UTC m=+5.489949241 container died b039a05751e2381732ae128bee0e82d3f65858166ddaca40c5967a59a4783ee4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T15:30:52.415 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:52 smithi104 podman[73923]: 2024-03-18 15:30:52.130980046 +0000 UTC m=+5.877761851 container cleanup b039a05751e2381732ae128bee0e82d3f65858166ddaca40c5967a59a4783ee4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-03-18T15:30:52.415 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:52 smithi104 bash[73923]: ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8 2024-03-18T15:30:52.415 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:52 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:52.248+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:41.404783+0000 front 2024-03-18T15:29:41.404661+0000 (oldest deadline 2024-03-18T15:30:04.304572+0000) 2024-03-18T15:30:52.415 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:52 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:52.248+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.207053+0000 front 2024-03-18T15:29:51.206898+0000 (oldest deadline 2024-03-18T15:30:13.507003+0000) 2024-03-18T15:30:52.416 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:52 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:52.248+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:59.809088+0000 front 2024-03-18T15:29:59.809253+0000 (oldest deadline 2024-03-18T15:30:21.509244+0000) 2024-03-18T15:30:52.416 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:52 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:52.248+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-03-18T15:30:12.511708+0000 front 2024-03-18T15:30:12.511569+0000 (oldest deadline 2024-03-18T15:30:43.112056+0000) 2024-03-18T15:30:52.416 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:52 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:52.248+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2024-03-18T15:30:25.913763+0000 front 2024-03-18T15:30:25.913855+0000 (oldest deadline 2024-03-18T15:30:49.413885+0000) 2024-03-18T15:30:52.465 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:52 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:52.181+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:39.237875+0000 front 2024-03-18T15:29:39.238431+0000 (oldest deadline 2024-03-18T15:30:05.138032+0000) 2024-03-18T15:30:52.465 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:52 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:52.181+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:50.743126+0000 front 2024-03-18T15:29:50.743226+0000 (oldest deadline 2024-03-18T15:30:13.643377+0000) 2024-03-18T15:30:52.465 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:52 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:52.181+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:56.947757+0000 front 2024-03-18T15:29:56.947820+0000 (oldest deadline 2024-03-18T15:30:22.248013+0000) 2024-03-18T15:30:52.466 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:52 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:52.181+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:06.950015+0000 front 2024-03-18T15:30:06.950486+0000 (oldest deadline 2024-03-18T15:30:31.650112+0000) 2024-03-18T15:30:52.466 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:52 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:52.181+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-03-18T15:30:17.552108+0000 front 2024-03-18T15:30:17.552227+0000 (oldest deadline 2024-03-18T15:30:39.852162+0000) 2024-03-18T15:30:52.466 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:52 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:52.181+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2024-03-18T15:30:25.655158+0000 front 2024-03-18T15:30:25.654982+0000 (oldest deadline 2024-03-18T15:30:47.354642+0000) 2024-03-18T15:30:52.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:52 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:52.046+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.612085+0000 front 2024-03-18T15:29:51.612164+0000 (oldest deadline 2024-03-18T15:30:17.512254+0000) 2024-03-18T15:30:52.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:52 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:52.046+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:57.512824+0000 front 2024-03-18T15:29:57.512967+0000 (oldest deadline 2024-03-18T15:30:21.612923+0000) 2024-03-18T15:30:52.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:52 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:52.046+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:07.414865+0000 front 2024-03-18T15:30:07.415069+0000 (oldest deadline 2024-03-18T15:30:32.714829+0000) 2024-03-18T15:30:52.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:52 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:52.046+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-03-18T15:30:16.215900+0000 front 2024-03-18T15:30:16.215789+0000 (oldest deadline 2024-03-18T15:30:40.915902+0000) 2024-03-18T15:30:52.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:52 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:52.047+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2024-03-18T15:30:25.016948+0000 front 2024-03-18T15:30:25.016983+0000 (oldest deadline 2024-03-18T15:30:47.916909+0000) 2024-03-18T15:30:52.694 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:52 smithi104 podman[73940]: 2024-03-18 15:30:52.415551065 +0000 UTC m=+0.674592494 container remove b039a05751e2381732ae128bee0e82d3f65858166ddaca40c5967a59a4783ee4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T15:30:53.019 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:52 smithi104 podman[74029]: 2024-03-18 15:30:52.656466195 +0000 UTC m=+0.017406033 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T15:30:53.019 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:52 smithi104 podman[74029]: 2024-03-18 15:30:52.76639108 +0000 UTC m=+0.127330928 container create 3dc1e94f00ed766061997496f8252f301351331c0f6a10a083363e3a694af402 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8-deactivate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-03-18T15:30:53.019 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:53 smithi104 podman[74029]: 2024-03-18 15:30:53.019391542 +0000 UTC m=+0.380331373 container init 3dc1e94f00ed766061997496f8252f301351331c0f6a10a083363e3a694af402 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True) 2024-03-18T15:30:53.309 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:53 smithi104 podman[74029]: 2024-03-18 15:30:53.022646829 +0000 UTC m=+0.383586658 container start 3dc1e94f00ed766061997496f8252f301351331c0f6a10a083363e3a694af402 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8-deactivate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240307, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T15:30:53.309 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:53 smithi104 podman[74029]: 2024-03-18 15:30:53.102554959 +0000 UTC m=+0.463494795 container attach 3dc1e94f00ed766061997496f8252f301351331c0f6a10a083363e3a694af402 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-18T15:30:53.310 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:53 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:53.259+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:41.404783+0000 front 2024-03-18T15:29:41.404661+0000 (oldest deadline 2024-03-18T15:30:04.304572+0000) 2024-03-18T15:30:53.310 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:53 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:53.260+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.207053+0000 front 2024-03-18T15:29:51.206898+0000 (oldest deadline 2024-03-18T15:30:13.507003+0000) 2024-03-18T15:30:53.310 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:53 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:53.260+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:59.809088+0000 front 2024-03-18T15:29:59.809253+0000 (oldest deadline 2024-03-18T15:30:21.509244+0000) 2024-03-18T15:30:53.310 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:53 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:53.260+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-03-18T15:30:12.511708+0000 front 2024-03-18T15:30:12.511569+0000 (oldest deadline 2024-03-18T15:30:43.112056+0000) 2024-03-18T15:30:53.310 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:53 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:53.260+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2024-03-18T15:30:25.913763+0000 front 2024-03-18T15:30:25.913855+0000 (oldest deadline 2024-03-18T15:30:49.413885+0000) 2024-03-18T15:30:53.466 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:53 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:53.134+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:39.237875+0000 front 2024-03-18T15:29:39.238431+0000 (oldest deadline 2024-03-18T15:30:05.138032+0000) 2024-03-18T15:30:53.466 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:53 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:53.134+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:50.743126+0000 front 2024-03-18T15:29:50.743226+0000 (oldest deadline 2024-03-18T15:30:13.643377+0000) 2024-03-18T15:30:53.466 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:53 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:53.134+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:56.947757+0000 front 2024-03-18T15:29:56.947820+0000 (oldest deadline 2024-03-18T15:30:22.248013+0000) 2024-03-18T15:30:53.466 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:53 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:53.134+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:06.950015+0000 front 2024-03-18T15:30:06.950486+0000 (oldest deadline 2024-03-18T15:30:31.650112+0000) 2024-03-18T15:30:53.467 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:53 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:53.134+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-03-18T15:30:17.552108+0000 front 2024-03-18T15:30:17.552227+0000 (oldest deadline 2024-03-18T15:30:39.852162+0000) 2024-03-18T15:30:53.467 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:53 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:53.134+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2024-03-18T15:30:25.655158+0000 front 2024-03-18T15:30:25.654982+0000 (oldest deadline 2024-03-18T15:30:47.354642+0000) 2024-03-18T15:30:53.506 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:53 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:53.085+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.612085+0000 front 2024-03-18T15:29:51.612164+0000 (oldest deadline 2024-03-18T15:30:17.512254+0000) 2024-03-18T15:30:53.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:53 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:53.085+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:57.512824+0000 front 2024-03-18T15:29:57.512967+0000 (oldest deadline 2024-03-18T15:30:21.612923+0000) 2024-03-18T15:30:53.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:53 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:53.085+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:07.414865+0000 front 2024-03-18T15:30:07.415069+0000 (oldest deadline 2024-03-18T15:30:32.714829+0000) 2024-03-18T15:30:53.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:53 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:53.085+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-03-18T15:30:16.215900+0000 front 2024-03-18T15:30:16.215789+0000 (oldest deadline 2024-03-18T15:30:40.915902+0000) 2024-03-18T15:30:53.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:53 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:53.085+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2024-03-18T15:30:25.016948+0000 front 2024-03-18T15:30:25.016983+0000 (oldest deadline 2024-03-18T15:30:47.916909+0000) 2024-03-18T15:30:53.573 INFO:journalctl@ceph.osd.8.smithi104.stdout:Mar 18 15:30:53 smithi104 podman[74029]: 2024-03-18 15:30:53.309819716 +0000 UTC m=+0.670759545 container died 3dc1e94f00ed766061997496f8252f301351331c0f6a10a083363e3a694af402 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-8-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-03-18T15:30:53.872 DEBUG:teuthology.orchestra.run.smithi104:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@osd.8.service' 2024-03-18T15:30:53.912 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T15:30:53.912 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-03-18T15:30:53.913 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-03-18T15:30:53.913 DEBUG:teuthology.orchestra.run.smithi019:> sudo systemctl stop ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@osd.9 2024-03-18T15:30:54.466 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:54 smithi019 systemd[1]: Stopping Ceph osd.9 for d4dc2ba6-e538-11ee-95c9-87774f69a715... 2024-03-18T15:30:54.466 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:54 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:54.172+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:39.237875+0000 front 2024-03-18T15:29:39.238431+0000 (oldest deadline 2024-03-18T15:30:05.138032+0000) 2024-03-18T15:30:54.466 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:54 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:54.172+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:50.743126+0000 front 2024-03-18T15:29:50.743226+0000 (oldest deadline 2024-03-18T15:30:13.643377+0000) 2024-03-18T15:30:54.466 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:54 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:54.172+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:56.947757+0000 front 2024-03-18T15:29:56.947820+0000 (oldest deadline 2024-03-18T15:30:22.248013+0000) 2024-03-18T15:30:54.466 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:54 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:54.172+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:06.950015+0000 front 2024-03-18T15:30:06.950486+0000 (oldest deadline 2024-03-18T15:30:31.650112+0000) 2024-03-18T15:30:54.466 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:54 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:54.172+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-03-18T15:30:17.552108+0000 front 2024-03-18T15:30:17.552227+0000 (oldest deadline 2024-03-18T15:30:39.852162+0000) 2024-03-18T15:30:54.466 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:54 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:54.172+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2024-03-18T15:30:25.655158+0000 front 2024-03-18T15:30:25.654982+0000 (oldest deadline 2024-03-18T15:30:47.354642+0000) 2024-03-18T15:30:54.500 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:54 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:54.212+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:41.404783+0000 front 2024-03-18T15:29:41.404661+0000 (oldest deadline 2024-03-18T15:30:04.304572+0000) 2024-03-18T15:30:54.500 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:54 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:54.212+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.207053+0000 front 2024-03-18T15:29:51.206898+0000 (oldest deadline 2024-03-18T15:30:13.507003+0000) 2024-03-18T15:30:54.500 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:54 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:54.212+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:59.809088+0000 front 2024-03-18T15:29:59.809253+0000 (oldest deadline 2024-03-18T15:30:21.509244+0000) 2024-03-18T15:30:54.500 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:54 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:54.212+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-03-18T15:30:12.511708+0000 front 2024-03-18T15:30:12.511569+0000 (oldest deadline 2024-03-18T15:30:43.112056+0000) 2024-03-18T15:30:54.500 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:54 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:54.212+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2024-03-18T15:30:25.913763+0000 front 2024-03-18T15:30:25.913855+0000 (oldest deadline 2024-03-18T15:30:49.413885+0000) 2024-03-18T15:30:54.506 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:54 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:54.080+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.612085+0000 front 2024-03-18T15:29:51.612164+0000 (oldest deadline 2024-03-18T15:30:17.512254+0000) 2024-03-18T15:30:54.506 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:54 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:54.080+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:57.512824+0000 front 2024-03-18T15:29:57.512967+0000 (oldest deadline 2024-03-18T15:30:21.612923+0000) 2024-03-18T15:30:54.506 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:54 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:54.080+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:07.414865+0000 front 2024-03-18T15:30:07.415069+0000 (oldest deadline 2024-03-18T15:30:32.714829+0000) 2024-03-18T15:30:54.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:54 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:54.080+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-03-18T15:30:16.215900+0000 front 2024-03-18T15:30:16.215789+0000 (oldest deadline 2024-03-18T15:30:40.915902+0000) 2024-03-18T15:30:54.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:54 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:54.080+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2024-03-18T15:30:25.016948+0000 front 2024-03-18T15:30:25.016983+0000 (oldest deadline 2024-03-18T15:30:47.916909+0000) 2024-03-18T15:30:54.965 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:54 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:54.511+0000 7f2c41400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T15:30:54.965 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:54 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:54.511+0000 7f2c41400700 -1 osd.9 109 *** Got signal Terminated *** 2024-03-18T15:30:54.965 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:54 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:54.511+0000 7f2c41400700 -1 osd.9 109 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T15:30:55.465 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:55 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:55.211+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:39.237875+0000 front 2024-03-18T15:29:39.238431+0000 (oldest deadline 2024-03-18T15:30:05.138032+0000) 2024-03-18T15:30:55.466 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:55 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:55.211+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:50.743126+0000 front 2024-03-18T15:29:50.743226+0000 (oldest deadline 2024-03-18T15:30:13.643377+0000) 2024-03-18T15:30:55.466 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:55 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:55.211+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:56.947757+0000 front 2024-03-18T15:29:56.947820+0000 (oldest deadline 2024-03-18T15:30:22.248013+0000) 2024-03-18T15:30:55.466 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:55 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:55.211+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:06.950015+0000 front 2024-03-18T15:30:06.950486+0000 (oldest deadline 2024-03-18T15:30:31.650112+0000) 2024-03-18T15:30:55.466 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:55 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:55.211+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-03-18T15:30:17.552108+0000 front 2024-03-18T15:30:17.552227+0000 (oldest deadline 2024-03-18T15:30:39.852162+0000) 2024-03-18T15:30:55.466 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:55 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:55.211+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2024-03-18T15:30:25.655158+0000 front 2024-03-18T15:30:25.654982+0000 (oldest deadline 2024-03-18T15:30:47.354642+0000) 2024-03-18T15:30:55.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:55 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:55.049+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.612085+0000 front 2024-03-18T15:29:51.612164+0000 (oldest deadline 2024-03-18T15:30:17.512254+0000) 2024-03-18T15:30:55.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:55 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:55.049+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:57.512824+0000 front 2024-03-18T15:29:57.512967+0000 (oldest deadline 2024-03-18T15:30:21.612923+0000) 2024-03-18T15:30:55.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:55 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:55.049+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:07.414865+0000 front 2024-03-18T15:30:07.415069+0000 (oldest deadline 2024-03-18T15:30:32.714829+0000) 2024-03-18T15:30:55.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:55 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:55.049+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-03-18T15:30:16.215900+0000 front 2024-03-18T15:30:16.215789+0000 (oldest deadline 2024-03-18T15:30:40.915902+0000) 2024-03-18T15:30:55.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:55 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:55.049+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2024-03-18T15:30:25.016948+0000 front 2024-03-18T15:30:25.016983+0000 (oldest deadline 2024-03-18T15:30:47.916909+0000) 2024-03-18T15:30:55.694 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:55 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:55.213+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:41.404783+0000 front 2024-03-18T15:29:41.404661+0000 (oldest deadline 2024-03-18T15:30:04.304572+0000) 2024-03-18T15:30:55.695 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:55 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:55.213+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.207053+0000 front 2024-03-18T15:29:51.206898+0000 (oldest deadline 2024-03-18T15:30:13.507003+0000) 2024-03-18T15:30:55.695 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:55 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:55.213+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:59.809088+0000 front 2024-03-18T15:29:59.809253+0000 (oldest deadline 2024-03-18T15:30:21.509244+0000) 2024-03-18T15:30:55.695 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:55 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:55.213+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-03-18T15:30:12.511708+0000 front 2024-03-18T15:30:12.511569+0000 (oldest deadline 2024-03-18T15:30:43.112056+0000) 2024-03-18T15:30:55.695 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:55 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:55.213+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2024-03-18T15:30:25.913763+0000 front 2024-03-18T15:30:25.913855+0000 (oldest deadline 2024-03-18T15:30:49.413885+0000) 2024-03-18T15:30:56.465 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:56 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:56.206+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:39.237875+0000 front 2024-03-18T15:29:39.238431+0000 (oldest deadline 2024-03-18T15:30:05.138032+0000) 2024-03-18T15:30:56.466 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:56 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:56.206+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:50.743126+0000 front 2024-03-18T15:29:50.743226+0000 (oldest deadline 2024-03-18T15:30:13.643377+0000) 2024-03-18T15:30:56.466 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:56 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:56.206+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:56.947757+0000 front 2024-03-18T15:29:56.947820+0000 (oldest deadline 2024-03-18T15:30:22.248013+0000) 2024-03-18T15:30:56.466 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:56 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:56.206+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:06.950015+0000 front 2024-03-18T15:30:06.950486+0000 (oldest deadline 2024-03-18T15:30:31.650112+0000) 2024-03-18T15:30:56.466 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:56 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:56.206+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-03-18T15:30:17.552108+0000 front 2024-03-18T15:30:17.552227+0000 (oldest deadline 2024-03-18T15:30:39.852162+0000) 2024-03-18T15:30:56.466 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:56 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:56.206+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2024-03-18T15:30:25.655158+0000 front 2024-03-18T15:30:25.654982+0000 (oldest deadline 2024-03-18T15:30:47.354642+0000) 2024-03-18T15:30:56.506 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:56 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:56.084+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.612085+0000 front 2024-03-18T15:29:51.612164+0000 (oldest deadline 2024-03-18T15:30:17.512254+0000) 2024-03-18T15:30:56.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:56 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:56.084+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:57.512824+0000 front 2024-03-18T15:29:57.512967+0000 (oldest deadline 2024-03-18T15:30:21.612923+0000) 2024-03-18T15:30:56.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:56 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:56.085+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:07.414865+0000 front 2024-03-18T15:30:07.415069+0000 (oldest deadline 2024-03-18T15:30:32.714829+0000) 2024-03-18T15:30:56.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:56 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:56.085+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-03-18T15:30:16.215900+0000 front 2024-03-18T15:30:16.215789+0000 (oldest deadline 2024-03-18T15:30:40.915902+0000) 2024-03-18T15:30:56.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:56 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:56.085+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2024-03-18T15:30:25.016948+0000 front 2024-03-18T15:30:25.016983+0000 (oldest deadline 2024-03-18T15:30:47.916909+0000) 2024-03-18T15:30:56.694 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:56 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:56.238+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:41.404783+0000 front 2024-03-18T15:29:41.404661+0000 (oldest deadline 2024-03-18T15:30:04.304572+0000) 2024-03-18T15:30:56.695 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:56 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:56.238+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.207053+0000 front 2024-03-18T15:29:51.206898+0000 (oldest deadline 2024-03-18T15:30:13.507003+0000) 2024-03-18T15:30:56.695 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:56 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:56.238+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:59.809088+0000 front 2024-03-18T15:29:59.809253+0000 (oldest deadline 2024-03-18T15:30:21.509244+0000) 2024-03-18T15:30:56.695 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:56 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:56.238+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-03-18T15:30:12.511708+0000 front 2024-03-18T15:30:12.511569+0000 (oldest deadline 2024-03-18T15:30:43.112056+0000) 2024-03-18T15:30:56.695 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:56 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:56.238+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2024-03-18T15:30:25.913763+0000 front 2024-03-18T15:30:25.913855+0000 (oldest deadline 2024-03-18T15:30:49.413885+0000) 2024-03-18T15:30:57.465 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:57 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:57.158+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:39.237875+0000 front 2024-03-18T15:29:39.238431+0000 (oldest deadline 2024-03-18T15:30:05.138032+0000) 2024-03-18T15:30:57.466 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:57 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:57.158+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:50.743126+0000 front 2024-03-18T15:29:50.743226+0000 (oldest deadline 2024-03-18T15:30:13.643377+0000) 2024-03-18T15:30:57.466 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:57 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:57.158+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:56.947757+0000 front 2024-03-18T15:29:56.947820+0000 (oldest deadline 2024-03-18T15:30:22.248013+0000) 2024-03-18T15:30:57.466 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:57 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:57.158+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:06.950015+0000 front 2024-03-18T15:30:06.950486+0000 (oldest deadline 2024-03-18T15:30:31.650112+0000) 2024-03-18T15:30:57.466 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:57 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:57.158+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-03-18T15:30:17.552108+0000 front 2024-03-18T15:30:17.552227+0000 (oldest deadline 2024-03-18T15:30:39.852162+0000) 2024-03-18T15:30:57.466 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:57 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:57.158+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2024-03-18T15:30:25.655158+0000 front 2024-03-18T15:30:25.654982+0000 (oldest deadline 2024-03-18T15:30:47.354642+0000) 2024-03-18T15:30:57.506 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:57 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:57.120+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.612085+0000 front 2024-03-18T15:29:51.612164+0000 (oldest deadline 2024-03-18T15:30:17.512254+0000) 2024-03-18T15:30:57.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:57 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:57.120+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:57.512824+0000 front 2024-03-18T15:29:57.512967+0000 (oldest deadline 2024-03-18T15:30:21.612923+0000) 2024-03-18T15:30:57.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:57 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:57.120+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:07.414865+0000 front 2024-03-18T15:30:07.415069+0000 (oldest deadline 2024-03-18T15:30:32.714829+0000) 2024-03-18T15:30:57.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:57 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:57.120+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-03-18T15:30:16.215900+0000 front 2024-03-18T15:30:16.215789+0000 (oldest deadline 2024-03-18T15:30:40.915902+0000) 2024-03-18T15:30:57.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:57 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:57.120+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2024-03-18T15:30:25.016948+0000 front 2024-03-18T15:30:25.016983+0000 (oldest deadline 2024-03-18T15:30:47.916909+0000) 2024-03-18T15:30:57.694 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:57 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:57.199+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:41.404783+0000 front 2024-03-18T15:29:41.404661+0000 (oldest deadline 2024-03-18T15:30:04.304572+0000) 2024-03-18T15:30:57.694 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:57 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:57.199+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.207053+0000 front 2024-03-18T15:29:51.206898+0000 (oldest deadline 2024-03-18T15:30:13.507003+0000) 2024-03-18T15:30:57.694 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:57 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:57.199+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:59.809088+0000 front 2024-03-18T15:29:59.809253+0000 (oldest deadline 2024-03-18T15:30:21.509244+0000) 2024-03-18T15:30:57.694 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:57 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:57.199+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-03-18T15:30:12.511708+0000 front 2024-03-18T15:30:12.511569+0000 (oldest deadline 2024-03-18T15:30:43.112056+0000) 2024-03-18T15:30:57.695 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:57 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:57.199+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2024-03-18T15:30:25.913763+0000 front 2024-03-18T15:30:25.913855+0000 (oldest deadline 2024-03-18T15:30:49.413885+0000) 2024-03-18T15:30:58.465 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:58 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:58.170+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:39.237875+0000 front 2024-03-18T15:29:39.238431+0000 (oldest deadline 2024-03-18T15:30:05.138032+0000) 2024-03-18T15:30:58.466 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:58 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:58.170+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:50.743126+0000 front 2024-03-18T15:29:50.743226+0000 (oldest deadline 2024-03-18T15:30:13.643377+0000) 2024-03-18T15:30:58.466 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:58 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:58.170+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:56.947757+0000 front 2024-03-18T15:29:56.947820+0000 (oldest deadline 2024-03-18T15:30:22.248013+0000) 2024-03-18T15:30:58.466 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:58 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:58.170+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:06.950015+0000 front 2024-03-18T15:30:06.950486+0000 (oldest deadline 2024-03-18T15:30:31.650112+0000) 2024-03-18T15:30:58.466 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:58 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:58.170+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-03-18T15:30:17.552108+0000 front 2024-03-18T15:30:17.552227+0000 (oldest deadline 2024-03-18T15:30:39.852162+0000) 2024-03-18T15:30:58.466 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:58 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:58.170+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2024-03-18T15:30:25.655158+0000 front 2024-03-18T15:30:25.654982+0000 (oldest deadline 2024-03-18T15:30:47.354642+0000) 2024-03-18T15:30:58.506 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:58 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:58.077+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.612085+0000 front 2024-03-18T15:29:51.612164+0000 (oldest deadline 2024-03-18T15:30:17.512254+0000) 2024-03-18T15:30:58.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:58 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:58.077+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:57.512824+0000 front 2024-03-18T15:29:57.512967+0000 (oldest deadline 2024-03-18T15:30:21.612923+0000) 2024-03-18T15:30:58.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:58 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:58.077+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:07.414865+0000 front 2024-03-18T15:30:07.415069+0000 (oldest deadline 2024-03-18T15:30:32.714829+0000) 2024-03-18T15:30:58.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:58 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:58.077+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-03-18T15:30:16.215900+0000 front 2024-03-18T15:30:16.215789+0000 (oldest deadline 2024-03-18T15:30:40.915902+0000) 2024-03-18T15:30:58.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:58 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:58.077+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2024-03-18T15:30:25.016948+0000 front 2024-03-18T15:30:25.016983+0000 (oldest deadline 2024-03-18T15:30:47.916909+0000) 2024-03-18T15:30:58.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:58 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:58.077+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.19:6822 osd.6 since back 2024-03-18T15:30:33.218261+0000 front 2024-03-18T15:30:33.218133+0000 (oldest deadline 2024-03-18T15:30:57.918012+0000) 2024-03-18T15:30:58.694 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:58 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:58.230+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:41.404783+0000 front 2024-03-18T15:29:41.404661+0000 (oldest deadline 2024-03-18T15:30:04.304572+0000) 2024-03-18T15:30:58.694 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:58 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:58.230+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.207053+0000 front 2024-03-18T15:29:51.206898+0000 (oldest deadline 2024-03-18T15:30:13.507003+0000) 2024-03-18T15:30:58.694 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:58 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:58.230+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:59.809088+0000 front 2024-03-18T15:29:59.809253+0000 (oldest deadline 2024-03-18T15:30:21.509244+0000) 2024-03-18T15:30:58.695 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:58 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:58.230+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-03-18T15:30:12.511708+0000 front 2024-03-18T15:30:12.511569+0000 (oldest deadline 2024-03-18T15:30:43.112056+0000) 2024-03-18T15:30:58.695 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:58 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:58.230+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2024-03-18T15:30:25.913763+0000 front 2024-03-18T15:30:25.913855+0000 (oldest deadline 2024-03-18T15:30:49.413885+0000) 2024-03-18T15:30:59.407 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:59 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:59.123+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:39.237875+0000 front 2024-03-18T15:29:39.238431+0000 (oldest deadline 2024-03-18T15:30:05.138032+0000) 2024-03-18T15:30:59.407 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:59 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:59.123+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:50.743126+0000 front 2024-03-18T15:29:50.743226+0000 (oldest deadline 2024-03-18T15:30:13.643377+0000) 2024-03-18T15:30:59.407 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:59 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:59.123+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:56.947757+0000 front 2024-03-18T15:29:56.947820+0000 (oldest deadline 2024-03-18T15:30:22.248013+0000) 2024-03-18T15:30:59.407 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:59 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:59.123+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:06.950015+0000 front 2024-03-18T15:30:06.950486+0000 (oldest deadline 2024-03-18T15:30:31.650112+0000) 2024-03-18T15:30:59.407 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:59 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:59.123+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-03-18T15:30:17.552108+0000 front 2024-03-18T15:30:17.552227+0000 (oldest deadline 2024-03-18T15:30:39.852162+0000) 2024-03-18T15:30:59.407 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:30:59 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9[66333]: 2024-03-18T15:30:59.123+0000 7f2c37c00700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2024-03-18T15:30:25.655158+0000 front 2024-03-18T15:30:25.654982+0000 (oldest deadline 2024-03-18T15:30:47.354642+0000) 2024-03-18T15:30:59.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:59 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:59.126+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.612085+0000 front 2024-03-18T15:29:51.612164+0000 (oldest deadline 2024-03-18T15:30:17.512254+0000) 2024-03-18T15:30:59.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:59 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:59.126+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:57.512824+0000 front 2024-03-18T15:29:57.512967+0000 (oldest deadline 2024-03-18T15:30:21.612923+0000) 2024-03-18T15:30:59.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:59 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:59.126+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:07.414865+0000 front 2024-03-18T15:30:07.415069+0000 (oldest deadline 2024-03-18T15:30:32.714829+0000) 2024-03-18T15:30:59.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:59 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:59.126+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-03-18T15:30:16.215900+0000 front 2024-03-18T15:30:16.215789+0000 (oldest deadline 2024-03-18T15:30:40.915902+0000) 2024-03-18T15:30:59.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:59 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:59.126+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2024-03-18T15:30:25.016948+0000 front 2024-03-18T15:30:25.016983+0000 (oldest deadline 2024-03-18T15:30:47.916909+0000) 2024-03-18T15:30:59.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:30:59 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:30:59.126+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.19:6822 osd.6 since back 2024-03-18T15:30:33.218261+0000 front 2024-03-18T15:30:33.218133+0000 (oldest deadline 2024-03-18T15:30:57.918012+0000) 2024-03-18T15:30:59.694 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:59 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:59.260+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:41.404783+0000 front 2024-03-18T15:29:41.404661+0000 (oldest deadline 2024-03-18T15:30:04.304572+0000) 2024-03-18T15:30:59.694 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:59 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:59.260+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.207053+0000 front 2024-03-18T15:29:51.206898+0000 (oldest deadline 2024-03-18T15:30:13.507003+0000) 2024-03-18T15:30:59.695 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:59 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:59.260+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:59.809088+0000 front 2024-03-18T15:29:59.809253+0000 (oldest deadline 2024-03-18T15:30:21.509244+0000) 2024-03-18T15:30:59.695 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:59 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:59.260+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-03-18T15:30:12.511708+0000 front 2024-03-18T15:30:12.511569+0000 (oldest deadline 2024-03-18T15:30:43.112056+0000) 2024-03-18T15:30:59.695 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:30:59 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:30:59.260+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2024-03-18T15:30:25.913763+0000 front 2024-03-18T15:30:25.913855+0000 (oldest deadline 2024-03-18T15:30:49.413885+0000) 2024-03-18T15:31:00.465 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:31:00 smithi019 podman[93055]: 2024-03-18 15:31:00.064902501 +0000 UTC m=+5.667178732 container died a3f72f1982efd5168d9446bd415a08a2a77d7d2e900587bef89f2f774b99777c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-18T15:31:00.506 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:31:00 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:31:00.150+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.612085+0000 front 2024-03-18T15:29:51.612164+0000 (oldest deadline 2024-03-18T15:30:17.512254+0000) 2024-03-18T15:31:00.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:31:00 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:31:00.150+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:57.512824+0000 front 2024-03-18T15:29:57.512967+0000 (oldest deadline 2024-03-18T15:30:21.612923+0000) 2024-03-18T15:31:00.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:31:00 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:31:00.150+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:07.414865+0000 front 2024-03-18T15:30:07.415069+0000 (oldest deadline 2024-03-18T15:30:32.714829+0000) 2024-03-18T15:31:00.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:31:00 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:31:00.150+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-03-18T15:30:16.215900+0000 front 2024-03-18T15:30:16.215789+0000 (oldest deadline 2024-03-18T15:30:40.915902+0000) 2024-03-18T15:31:00.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:31:00 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:31:00.150+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2024-03-18T15:30:25.016948+0000 front 2024-03-18T15:30:25.016983+0000 (oldest deadline 2024-03-18T15:30:47.916909+0000) 2024-03-18T15:31:00.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:31:00 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:31:00.150+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.19:6822 osd.6 since back 2024-03-18T15:30:33.218261+0000 front 2024-03-18T15:30:33.218133+0000 (oldest deadline 2024-03-18T15:30:57.918012+0000) 2024-03-18T15:31:00.694 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:00 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:00.243+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:41.404783+0000 front 2024-03-18T15:29:41.404661+0000 (oldest deadline 2024-03-18T15:30:04.304572+0000) 2024-03-18T15:31:00.694 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:00 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:00.243+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.207053+0000 front 2024-03-18T15:29:51.206898+0000 (oldest deadline 2024-03-18T15:30:13.507003+0000) 2024-03-18T15:31:00.694 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:00 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:00.243+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:59.809088+0000 front 2024-03-18T15:29:59.809253+0000 (oldest deadline 2024-03-18T15:30:21.509244+0000) 2024-03-18T15:31:00.694 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:00 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:00.243+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-03-18T15:30:12.511708+0000 front 2024-03-18T15:30:12.511569+0000 (oldest deadline 2024-03-18T15:30:43.112056+0000) 2024-03-18T15:31:00.695 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:00 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:00.243+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2024-03-18T15:30:25.913763+0000 front 2024-03-18T15:30:25.913855+0000 (oldest deadline 2024-03-18T15:30:49.413885+0000) 2024-03-18T15:31:00.923 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:31:00 smithi019 podman[93055]: 2024-03-18 15:31:00.653718624 +0000 UTC m=+6.255994900 container cleanup a3f72f1982efd5168d9446bd415a08a2a77d7d2e900587bef89f2f774b99777c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T15:31:00.923 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:31:00 smithi019 bash[93055]: ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9 2024-03-18T15:31:01.215 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:31:00 smithi019 podman[93074]: 2024-03-18 15:31:00.921208111 +0000 UTC m=+0.837280330 container remove a3f72f1982efd5168d9446bd415a08a2a77d7d2e900587bef89f2f774b99777c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-18T15:31:01.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:31:01 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:31:01.190+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.612085+0000 front 2024-03-18T15:29:51.612164+0000 (oldest deadline 2024-03-18T15:30:17.512254+0000) 2024-03-18T15:31:01.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:31:01 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:31:01.190+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:57.512824+0000 front 2024-03-18T15:29:57.512967+0000 (oldest deadline 2024-03-18T15:30:21.612923+0000) 2024-03-18T15:31:01.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:31:01 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:31:01.190+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:07.414865+0000 front 2024-03-18T15:30:07.415069+0000 (oldest deadline 2024-03-18T15:30:32.714829+0000) 2024-03-18T15:31:01.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:31:01 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:31:01.190+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-03-18T15:30:16.215900+0000 front 2024-03-18T15:30:16.215789+0000 (oldest deadline 2024-03-18T15:30:40.915902+0000) 2024-03-18T15:31:01.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:31:01 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:31:01.190+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2024-03-18T15:30:25.016948+0000 front 2024-03-18T15:30:25.016983+0000 (oldest deadline 2024-03-18T15:30:47.916909+0000) 2024-03-18T15:31:01.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:31:01 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:31:01.190+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.19:6822 osd.6 since back 2024-03-18T15:30:33.218261+0000 front 2024-03-18T15:30:33.218133+0000 (oldest deadline 2024-03-18T15:30:57.918012+0000) 2024-03-18T15:31:01.596 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:01 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:01.255+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:41.404783+0000 front 2024-03-18T15:29:41.404661+0000 (oldest deadline 2024-03-18T15:30:04.304572+0000) 2024-03-18T15:31:01.596 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:01 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:01.255+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.207053+0000 front 2024-03-18T15:29:51.206898+0000 (oldest deadline 2024-03-18T15:30:13.507003+0000) 2024-03-18T15:31:01.596 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:01 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:01.255+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:59.809088+0000 front 2024-03-18T15:29:59.809253+0000 (oldest deadline 2024-03-18T15:30:21.509244+0000) 2024-03-18T15:31:01.596 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:01 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:01.255+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-03-18T15:30:12.511708+0000 front 2024-03-18T15:30:12.511569+0000 (oldest deadline 2024-03-18T15:30:43.112056+0000) 2024-03-18T15:31:01.596 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:01 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:01.255+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2024-03-18T15:30:25.913763+0000 front 2024-03-18T15:30:25.913855+0000 (oldest deadline 2024-03-18T15:30:49.413885+0000) 2024-03-18T15:31:01.596 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:01 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:01.255+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.19:6822 osd.6 since back 2024-03-18T15:30:29.414560+0000 front 2024-03-18T15:30:29.414721+0000 (oldest deadline 2024-03-18T15:31:00.614742+0000) 2024-03-18T15:31:01.801 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:31:01 smithi019 podman[93160]: 2024-03-18 15:31:01.428427612 +0000 UTC m=+0.032839143 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T15:31:01.801 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:31:01 smithi019 podman[93160]: 2024-03-18 15:31:01.67074457 +0000 UTC m=+0.275156134 container create 2912c5cb6c9ebbbbfd5558290ad49ca355abd52051b62c663316b7b3ab1c3374 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-18T15:31:02.215 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:31:01 smithi019 podman[93160]: 2024-03-18 15:31:01.92432626 +0000 UTC m=+0.528737831 container init 2912c5cb6c9ebbbbfd5558290ad49ca355abd52051b62c663316b7b3ab1c3374 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, CEPH_POINT_RELEASE=) 2024-03-18T15:31:02.215 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:31:01 smithi019 podman[93160]: 2024-03-18 15:31:01.935769356 +0000 UTC m=+0.540180944 container start 2912c5cb6c9ebbbbfd5558290ad49ca355abd52051b62c663316b7b3ab1c3374 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9-deactivate, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-03-18T15:31:02.216 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:31:02 smithi019 podman[93160]: 2024-03-18 15:31:02.015970433 +0000 UTC m=+0.620382020 container attach 2912c5cb6c9ebbbbfd5558290ad49ca355abd52051b62c663316b7b3ab1c3374 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-03-18T15:31:02.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:31:02 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:31:02.172+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.612085+0000 front 2024-03-18T15:29:51.612164+0000 (oldest deadline 2024-03-18T15:30:17.512254+0000) 2024-03-18T15:31:02.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:31:02 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:31:02.172+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:57.512824+0000 front 2024-03-18T15:29:57.512967+0000 (oldest deadline 2024-03-18T15:30:21.612923+0000) 2024-03-18T15:31:02.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:31:02 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:31:02.172+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:07.414865+0000 front 2024-03-18T15:30:07.415069+0000 (oldest deadline 2024-03-18T15:30:32.714829+0000) 2024-03-18T15:31:02.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:31:02 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:31:02.172+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-03-18T15:30:16.215900+0000 front 2024-03-18T15:30:16.215789+0000 (oldest deadline 2024-03-18T15:30:40.915902+0000) 2024-03-18T15:31:02.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:31:02 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:31:02.172+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2024-03-18T15:30:25.016948+0000 front 2024-03-18T15:30:25.016983+0000 (oldest deadline 2024-03-18T15:30:47.916909+0000) 2024-03-18T15:31:02.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:31:02 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:31:02.172+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.19:6822 osd.6 since back 2024-03-18T15:30:33.218261+0000 front 2024-03-18T15:30:33.218133+0000 (oldest deadline 2024-03-18T15:30:57.918012+0000) 2024-03-18T15:31:02.694 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:02 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:02.244+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:41.404783+0000 front 2024-03-18T15:29:41.404661+0000 (oldest deadline 2024-03-18T15:30:04.304572+0000) 2024-03-18T15:31:02.694 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:02 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:02.244+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.207053+0000 front 2024-03-18T15:29:51.206898+0000 (oldest deadline 2024-03-18T15:30:13.507003+0000) 2024-03-18T15:31:02.694 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:02 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:02.244+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:59.809088+0000 front 2024-03-18T15:29:59.809253+0000 (oldest deadline 2024-03-18T15:30:21.509244+0000) 2024-03-18T15:31:02.694 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:02 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:02.244+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-03-18T15:30:12.511708+0000 front 2024-03-18T15:30:12.511569+0000 (oldest deadline 2024-03-18T15:30:43.112056+0000) 2024-03-18T15:31:02.695 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:02 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:02.244+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2024-03-18T15:30:25.913763+0000 front 2024-03-18T15:30:25.913855+0000 (oldest deadline 2024-03-18T15:30:49.413885+0000) 2024-03-18T15:31:02.695 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:02 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:02.244+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.19:6822 osd.6 since back 2024-03-18T15:30:29.414560+0000 front 2024-03-18T15:30:29.414721+0000 (oldest deadline 2024-03-18T15:31:00.614742+0000) 2024-03-18T15:31:02.715 INFO:journalctl@ceph.osd.9.smithi019.stdout:Mar 18 15:31:02 smithi019 podman[93160]: 2024-03-18 15:31:02.381963617 +0000 UTC m=+0.986375221 container died 2912c5cb6c9ebbbbfd5558290ad49ca355abd52051b62c663316b7b3ab1c3374 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-9-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5) 2024-03-18T15:31:03.252 DEBUG:teuthology.orchestra.run.smithi019:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@osd.9.service' 2024-03-18T15:31:03.377 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T15:31:03.377 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-03-18T15:31:03.377 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-03-18T15:31:03.377 DEBUG:teuthology.orchestra.run.smithi033:> sudo systemctl stop ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@osd.10 2024-03-18T15:31:03.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:31:03 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:31:03.196+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.612085+0000 front 2024-03-18T15:29:51.612164+0000 (oldest deadline 2024-03-18T15:30:17.512254+0000) 2024-03-18T15:31:03.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:31:03 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:31:03.196+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:57.512824+0000 front 2024-03-18T15:29:57.512967+0000 (oldest deadline 2024-03-18T15:30:21.612923+0000) 2024-03-18T15:31:03.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:31:03 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:31:03.196+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:07.414865+0000 front 2024-03-18T15:30:07.415069+0000 (oldest deadline 2024-03-18T15:30:32.714829+0000) 2024-03-18T15:31:03.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:31:03 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:31:03.196+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-03-18T15:30:16.215900+0000 front 2024-03-18T15:30:16.215789+0000 (oldest deadline 2024-03-18T15:30:40.915902+0000) 2024-03-18T15:31:03.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:31:03 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:31:03.196+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2024-03-18T15:30:25.016948+0000 front 2024-03-18T15:30:25.016983+0000 (oldest deadline 2024-03-18T15:30:47.916909+0000) 2024-03-18T15:31:03.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:31:03 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:31:03.196+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.19:6822 osd.6 since back 2024-03-18T15:30:33.218261+0000 front 2024-03-18T15:30:33.218133+0000 (oldest deadline 2024-03-18T15:30:57.918012+0000) 2024-03-18T15:31:03.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:31:03 smithi033 systemd[1]: Stopping Ceph osd.10 for d4dc2ba6-e538-11ee-95c9-87774f69a715... 2024-03-18T15:31:03.694 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:03 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:03.206+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:41.404783+0000 front 2024-03-18T15:29:41.404661+0000 (oldest deadline 2024-03-18T15:30:04.304572+0000) 2024-03-18T15:31:03.694 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:03 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:03.206+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.207053+0000 front 2024-03-18T15:29:51.206898+0000 (oldest deadline 2024-03-18T15:30:13.507003+0000) 2024-03-18T15:31:03.694 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:03 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:03.206+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:59.809088+0000 front 2024-03-18T15:29:59.809253+0000 (oldest deadline 2024-03-18T15:30:21.509244+0000) 2024-03-18T15:31:03.694 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:03 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:03.206+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-03-18T15:30:12.511708+0000 front 2024-03-18T15:30:12.511569+0000 (oldest deadline 2024-03-18T15:30:43.112056+0000) 2024-03-18T15:31:03.694 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:03 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:03.206+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2024-03-18T15:30:25.913763+0000 front 2024-03-18T15:30:25.913855+0000 (oldest deadline 2024-03-18T15:30:49.413885+0000) 2024-03-18T15:31:03.694 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:03 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:03.206+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.19:6822 osd.6 since back 2024-03-18T15:30:29.414560+0000 front 2024-03-18T15:30:29.414721+0000 (oldest deadline 2024-03-18T15:31:00.614742+0000) 2024-03-18T15:31:04.006 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:31:03 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:31:03.677+0000 7fc4a7bc4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T15:31:04.006 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:31:03 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:31:03.677+0000 7fc4a7bc4700 -1 osd.10 109 *** Got signal Terminated *** 2024-03-18T15:31:04.007 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:31:03 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:31:03.677+0000 7fc4a7bc4700 -1 osd.10 109 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T15:31:04.444 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:04 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:04.211+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:41.404783+0000 front 2024-03-18T15:29:41.404661+0000 (oldest deadline 2024-03-18T15:30:04.304572+0000) 2024-03-18T15:31:04.444 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:04 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:04.211+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.207053+0000 front 2024-03-18T15:29:51.206898+0000 (oldest deadline 2024-03-18T15:30:13.507003+0000) 2024-03-18T15:31:04.444 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:04 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:04.211+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:59.809088+0000 front 2024-03-18T15:29:59.809253+0000 (oldest deadline 2024-03-18T15:30:21.509244+0000) 2024-03-18T15:31:04.445 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:04 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:04.211+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-03-18T15:30:12.511708+0000 front 2024-03-18T15:30:12.511569+0000 (oldest deadline 2024-03-18T15:30:43.112056+0000) 2024-03-18T15:31:04.445 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:04 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:04.211+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2024-03-18T15:30:25.913763+0000 front 2024-03-18T15:30:25.913855+0000 (oldest deadline 2024-03-18T15:30:49.413885+0000) 2024-03-18T15:31:04.445 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:04 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:04.211+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.19:6822 osd.6 since back 2024-03-18T15:30:29.414560+0000 front 2024-03-18T15:30:29.414721+0000 (oldest deadline 2024-03-18T15:31:00.614742+0000) 2024-03-18T15:31:04.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:31:04 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:31:04.183+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.612085+0000 front 2024-03-18T15:29:51.612164+0000 (oldest deadline 2024-03-18T15:30:17.512254+0000) 2024-03-18T15:31:04.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:31:04 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:31:04.183+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:57.512824+0000 front 2024-03-18T15:29:57.512967+0000 (oldest deadline 2024-03-18T15:30:21.612923+0000) 2024-03-18T15:31:04.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:31:04 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:31:04.183+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:07.414865+0000 front 2024-03-18T15:30:07.415069+0000 (oldest deadline 2024-03-18T15:30:32.714829+0000) 2024-03-18T15:31:04.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:31:04 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:31:04.183+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-03-18T15:30:16.215900+0000 front 2024-03-18T15:30:16.215789+0000 (oldest deadline 2024-03-18T15:30:40.915902+0000) 2024-03-18T15:31:04.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:31:04 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:31:04.183+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2024-03-18T15:30:25.016948+0000 front 2024-03-18T15:30:25.016983+0000 (oldest deadline 2024-03-18T15:30:47.916909+0000) 2024-03-18T15:31:04.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:31:04 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:31:04.183+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.19:6822 osd.6 since back 2024-03-18T15:30:33.218261+0000 front 2024-03-18T15:30:33.218133+0000 (oldest deadline 2024-03-18T15:30:57.918012+0000) 2024-03-18T15:31:05.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:31:05 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:31:05.202+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.612085+0000 front 2024-03-18T15:29:51.612164+0000 (oldest deadline 2024-03-18T15:30:17.512254+0000) 2024-03-18T15:31:05.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:31:05 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:31:05.202+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:57.512824+0000 front 2024-03-18T15:29:57.512967+0000 (oldest deadline 2024-03-18T15:30:21.612923+0000) 2024-03-18T15:31:05.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:31:05 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:31:05.202+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:07.414865+0000 front 2024-03-18T15:30:07.415069+0000 (oldest deadline 2024-03-18T15:30:32.714829+0000) 2024-03-18T15:31:05.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:31:05 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:31:05.202+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-03-18T15:30:16.215900+0000 front 2024-03-18T15:30:16.215789+0000 (oldest deadline 2024-03-18T15:30:40.915902+0000) 2024-03-18T15:31:05.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:31:05 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:31:05.202+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2024-03-18T15:30:25.016948+0000 front 2024-03-18T15:30:25.016983+0000 (oldest deadline 2024-03-18T15:30:47.916909+0000) 2024-03-18T15:31:05.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:31:05 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:31:05.202+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.19:6822 osd.6 since back 2024-03-18T15:30:33.218261+0000 front 2024-03-18T15:30:33.218133+0000 (oldest deadline 2024-03-18T15:30:57.918012+0000) 2024-03-18T15:31:05.694 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:05 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:05.240+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:41.404783+0000 front 2024-03-18T15:29:41.404661+0000 (oldest deadline 2024-03-18T15:30:04.304572+0000) 2024-03-18T15:31:05.695 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:05 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:05.240+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.207053+0000 front 2024-03-18T15:29:51.206898+0000 (oldest deadline 2024-03-18T15:30:13.507003+0000) 2024-03-18T15:31:05.695 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:05 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:05.240+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:59.809088+0000 front 2024-03-18T15:29:59.809253+0000 (oldest deadline 2024-03-18T15:30:21.509244+0000) 2024-03-18T15:31:05.695 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:05 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:05.240+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-03-18T15:30:12.511708+0000 front 2024-03-18T15:30:12.511569+0000 (oldest deadline 2024-03-18T15:30:43.112056+0000) 2024-03-18T15:31:05.695 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:05 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:05.240+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2024-03-18T15:30:25.913763+0000 front 2024-03-18T15:30:25.913855+0000 (oldest deadline 2024-03-18T15:30:49.413885+0000) 2024-03-18T15:31:05.695 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:05 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:05.240+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.19:6822 osd.6 since back 2024-03-18T15:30:29.414560+0000 front 2024-03-18T15:30:29.414721+0000 (oldest deadline 2024-03-18T15:31:00.614742+0000) 2024-03-18T15:31:06.506 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:31:06 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:31:06.202+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.612085+0000 front 2024-03-18T15:29:51.612164+0000 (oldest deadline 2024-03-18T15:30:17.512254+0000) 2024-03-18T15:31:06.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:31:06 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:31:06.202+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:57.512824+0000 front 2024-03-18T15:29:57.512967+0000 (oldest deadline 2024-03-18T15:30:21.612923+0000) 2024-03-18T15:31:06.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:31:06 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:31:06.202+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:07.414865+0000 front 2024-03-18T15:30:07.415069+0000 (oldest deadline 2024-03-18T15:30:32.714829+0000) 2024-03-18T15:31:06.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:31:06 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:31:06.202+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-03-18T15:30:16.215900+0000 front 2024-03-18T15:30:16.215789+0000 (oldest deadline 2024-03-18T15:30:40.915902+0000) 2024-03-18T15:31:06.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:31:06 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:31:06.202+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2024-03-18T15:30:25.016948+0000 front 2024-03-18T15:30:25.016983+0000 (oldest deadline 2024-03-18T15:30:47.916909+0000) 2024-03-18T15:31:06.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:31:06 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:31:06.202+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.19:6822 osd.6 since back 2024-03-18T15:30:33.218261+0000 front 2024-03-18T15:30:33.218133+0000 (oldest deadline 2024-03-18T15:30:57.918012+0000) 2024-03-18T15:31:06.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:31:06 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:31:06.202+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.33:6820 osd.7 since back 2024-03-18T15:30:43.119122+0000 front 2024-03-18T15:30:43.119167+0000 (oldest deadline 2024-03-18T15:31:06.019092+0000) 2024-03-18T15:31:06.694 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:06 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:06.213+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:41.404783+0000 front 2024-03-18T15:29:41.404661+0000 (oldest deadline 2024-03-18T15:30:04.304572+0000) 2024-03-18T15:31:06.694 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:06 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:06.213+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.207053+0000 front 2024-03-18T15:29:51.206898+0000 (oldest deadline 2024-03-18T15:30:13.507003+0000) 2024-03-18T15:31:06.694 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:06 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:06.213+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:59.809088+0000 front 2024-03-18T15:29:59.809253+0000 (oldest deadline 2024-03-18T15:30:21.509244+0000) 2024-03-18T15:31:06.695 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:06 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:06.213+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-03-18T15:30:12.511708+0000 front 2024-03-18T15:30:12.511569+0000 (oldest deadline 2024-03-18T15:30:43.112056+0000) 2024-03-18T15:31:06.695 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:06 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:06.213+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2024-03-18T15:30:25.913763+0000 front 2024-03-18T15:30:25.913855+0000 (oldest deadline 2024-03-18T15:30:49.413885+0000) 2024-03-18T15:31:06.695 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:06 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:06.213+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.19:6822 osd.6 since back 2024-03-18T15:30:29.414560+0000 front 2024-03-18T15:30:29.414721+0000 (oldest deadline 2024-03-18T15:31:00.614742+0000) 2024-03-18T15:31:07.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:31:07 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:31:07.245+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.612085+0000 front 2024-03-18T15:29:51.612164+0000 (oldest deadline 2024-03-18T15:30:17.512254+0000) 2024-03-18T15:31:07.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:31:07 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:31:07.245+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:57.512824+0000 front 2024-03-18T15:29:57.512967+0000 (oldest deadline 2024-03-18T15:30:21.612923+0000) 2024-03-18T15:31:07.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:31:07 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:31:07.245+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:07.414865+0000 front 2024-03-18T15:30:07.415069+0000 (oldest deadline 2024-03-18T15:30:32.714829+0000) 2024-03-18T15:31:07.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:31:07 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:31:07.245+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-03-18T15:30:16.215900+0000 front 2024-03-18T15:30:16.215789+0000 (oldest deadline 2024-03-18T15:30:40.915902+0000) 2024-03-18T15:31:07.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:31:07 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:31:07.245+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2024-03-18T15:30:25.016948+0000 front 2024-03-18T15:30:25.016983+0000 (oldest deadline 2024-03-18T15:30:47.916909+0000) 2024-03-18T15:31:07.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:31:07 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:31:07.245+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.19:6822 osd.6 since back 2024-03-18T15:30:33.218261+0000 front 2024-03-18T15:30:33.218133+0000 (oldest deadline 2024-03-18T15:30:57.918012+0000) 2024-03-18T15:31:07.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:31:07 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:31:07.245+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.33:6820 osd.7 since back 2024-03-18T15:30:43.119122+0000 front 2024-03-18T15:30:43.119167+0000 (oldest deadline 2024-03-18T15:31:06.019092+0000) 2024-03-18T15:31:07.694 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:07 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:07.211+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:41.404783+0000 front 2024-03-18T15:29:41.404661+0000 (oldest deadline 2024-03-18T15:30:04.304572+0000) 2024-03-18T15:31:07.694 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:07 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:07.211+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.207053+0000 front 2024-03-18T15:29:51.206898+0000 (oldest deadline 2024-03-18T15:30:13.507003+0000) 2024-03-18T15:31:07.694 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:07 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:07.211+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:59.809088+0000 front 2024-03-18T15:29:59.809253+0000 (oldest deadline 2024-03-18T15:30:21.509244+0000) 2024-03-18T15:31:07.695 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:07 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:07.211+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-03-18T15:30:12.511708+0000 front 2024-03-18T15:30:12.511569+0000 (oldest deadline 2024-03-18T15:30:43.112056+0000) 2024-03-18T15:31:07.695 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:07 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:07.211+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2024-03-18T15:30:25.913763+0000 front 2024-03-18T15:30:25.913855+0000 (oldest deadline 2024-03-18T15:30:49.413885+0000) 2024-03-18T15:31:07.695 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:07 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:07.211+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.19:6822 osd.6 since back 2024-03-18T15:30:29.414560+0000 front 2024-03-18T15:30:29.414721+0000 (oldest deadline 2024-03-18T15:31:00.614742+0000) 2024-03-18T15:31:07.695 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:07 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:07.211+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6820 osd.7 since back 2024-03-18T15:30:41.715820+0000 front 2024-03-18T15:30:41.715852+0000 (oldest deadline 2024-03-18T15:31:07.015645+0000) 2024-03-18T15:31:08.444 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:08 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:08.184+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:41.404783+0000 front 2024-03-18T15:29:41.404661+0000 (oldest deadline 2024-03-18T15:30:04.304572+0000) 2024-03-18T15:31:08.444 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:08 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:08.184+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.207053+0000 front 2024-03-18T15:29:51.206898+0000 (oldest deadline 2024-03-18T15:30:13.507003+0000) 2024-03-18T15:31:08.444 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:08 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:08.184+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:59.809088+0000 front 2024-03-18T15:29:59.809253+0000 (oldest deadline 2024-03-18T15:30:21.509244+0000) 2024-03-18T15:31:08.444 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:08 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:08.184+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-03-18T15:30:12.511708+0000 front 2024-03-18T15:30:12.511569+0000 (oldest deadline 2024-03-18T15:30:43.112056+0000) 2024-03-18T15:31:08.445 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:08 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:08.184+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2024-03-18T15:30:25.913763+0000 front 2024-03-18T15:30:25.913855+0000 (oldest deadline 2024-03-18T15:30:49.413885+0000) 2024-03-18T15:31:08.445 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:08 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:08.184+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.19:6822 osd.6 since back 2024-03-18T15:30:29.414560+0000 front 2024-03-18T15:30:29.414721+0000 (oldest deadline 2024-03-18T15:31:00.614742+0000) 2024-03-18T15:31:08.445 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:08 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:08.184+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6820 osd.7 since back 2024-03-18T15:30:41.715820+0000 front 2024-03-18T15:30:41.715852+0000 (oldest deadline 2024-03-18T15:31:07.015645+0000) 2024-03-18T15:31:08.506 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:31:08 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:31:08.244+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.612085+0000 front 2024-03-18T15:29:51.612164+0000 (oldest deadline 2024-03-18T15:30:17.512254+0000) 2024-03-18T15:31:08.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:31:08 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:31:08.244+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:57.512824+0000 front 2024-03-18T15:29:57.512967+0000 (oldest deadline 2024-03-18T15:30:21.612923+0000) 2024-03-18T15:31:08.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:31:08 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:31:08.244+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-03-18T15:30:07.414865+0000 front 2024-03-18T15:30:07.415069+0000 (oldest deadline 2024-03-18T15:30:32.714829+0000) 2024-03-18T15:31:08.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:31:08 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:31:08.244+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-03-18T15:30:16.215900+0000 front 2024-03-18T15:30:16.215789+0000 (oldest deadline 2024-03-18T15:30:40.915902+0000) 2024-03-18T15:31:08.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:31:08 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:31:08.244+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2024-03-18T15:30:25.016948+0000 front 2024-03-18T15:30:25.016983+0000 (oldest deadline 2024-03-18T15:30:47.916909+0000) 2024-03-18T15:31:08.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:31:08 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:31:08.244+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.19:6822 osd.6 since back 2024-03-18T15:30:33.218261+0000 front 2024-03-18T15:30:33.218133+0000 (oldest deadline 2024-03-18T15:30:57.918012+0000) 2024-03-18T15:31:08.507 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:31:08 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10[61902]: 2024-03-18T15:31:08.244+0000 7fc4a41b8700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.33:6820 osd.7 since back 2024-03-18T15:30:43.119122+0000 front 2024-03-18T15:30:43.119167+0000 (oldest deadline 2024-03-18T15:31:06.019092+0000) 2024-03-18T15:31:09.486 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:31:09 smithi033 podman[74636]: 2024-03-18 15:31:09.187406676 +0000 UTC m=+5.610248309 container died b597756955a7ec09cb5cdc2403b040b13c1014a8d95476ba6f88bffdf4974d04 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T15:31:09.694 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:09 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:09.230+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:41.404783+0000 front 2024-03-18T15:29:41.404661+0000 (oldest deadline 2024-03-18T15:30:04.304572+0000) 2024-03-18T15:31:09.694 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:09 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:09.230+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.207053+0000 front 2024-03-18T15:29:51.206898+0000 (oldest deadline 2024-03-18T15:30:13.507003+0000) 2024-03-18T15:31:09.694 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:09 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:09.230+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:59.809088+0000 front 2024-03-18T15:29:59.809253+0000 (oldest deadline 2024-03-18T15:30:21.509244+0000) 2024-03-18T15:31:09.695 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:09 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:09.230+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-03-18T15:30:12.511708+0000 front 2024-03-18T15:30:12.511569+0000 (oldest deadline 2024-03-18T15:30:43.112056+0000) 2024-03-18T15:31:09.695 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:09 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:09.230+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2024-03-18T15:30:25.913763+0000 front 2024-03-18T15:30:25.913855+0000 (oldest deadline 2024-03-18T15:30:49.413885+0000) 2024-03-18T15:31:09.695 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:09 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:09.230+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.19:6822 osd.6 since back 2024-03-18T15:30:29.414560+0000 front 2024-03-18T15:30:29.414721+0000 (oldest deadline 2024-03-18T15:31:00.614742+0000) 2024-03-18T15:31:09.695 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:09 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:09.230+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6820 osd.7 since back 2024-03-18T15:30:41.715820+0000 front 2024-03-18T15:30:41.715852+0000 (oldest deadline 2024-03-18T15:31:07.015645+0000) 2024-03-18T15:31:09.756 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:31:09 smithi033 podman[74636]: 2024-03-18 15:31:09.485710654 +0000 UTC m=+5.908552284 container cleanup b597756955a7ec09cb5cdc2403b040b13c1014a8d95476ba6f88bffdf4974d04 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-03-18T15:31:09.757 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:31:09 smithi033 bash[74636]: ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10 2024-03-18T15:31:09.757 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:31:09 smithi033 podman[74666]: 2024-03-18 15:31:09.703735328 +0000 UTC m=+0.519232608 container remove b597756955a7ec09cb5cdc2403b040b13c1014a8d95476ba6f88bffdf4974d04 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-18T15:31:10.415 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:31:10 smithi033 podman[74756]: 2024-03-18 15:31:09.928883042 +0000 UTC m=+0.017933996 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T15:31:10.415 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:31:10 smithi033 podman[74756]: 2024-03-18 15:31:10.046595384 +0000 UTC m=+0.135646330 container create 11474d9ff74d41ff691af04dfbc5484e29c8aa196f3e1b486f402b5bc8fa8445 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10-deactivate, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-18T15:31:10.415 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:31:10 smithi033 podman[74756]: 2024-03-18 15:31:10.215679597 +0000 UTC m=+0.304730555 container init 11474d9ff74d41ff691af04dfbc5484e29c8aa196f3e1b486f402b5bc8fa8445 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T15:31:10.415 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:31:10 smithi033 podman[74756]: 2024-03-18 15:31:10.218928713 +0000 UTC m=+0.307979669 container start 11474d9ff74d41ff691af04dfbc5484e29c8aa196f3e1b486f402b5bc8fa8445 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240307, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-03-18T15:31:10.416 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:31:10 smithi033 podman[74756]: 2024-03-18 15:31:10.415616707 +0000 UTC m=+0.504667664 container attach 11474d9ff74d41ff691af04dfbc5484e29c8aa196f3e1b486f402b5bc8fa8445 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10-deactivate, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-18T15:31:10.444 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:10 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:10.187+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:41.404783+0000 front 2024-03-18T15:29:41.404661+0000 (oldest deadline 2024-03-18T15:30:04.304572+0000) 2024-03-18T15:31:10.444 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:10 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:10.187+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.207053+0000 front 2024-03-18T15:29:51.206898+0000 (oldest deadline 2024-03-18T15:30:13.507003+0000) 2024-03-18T15:31:10.444 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:10 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:10.187+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:59.809088+0000 front 2024-03-18T15:29:59.809253+0000 (oldest deadline 2024-03-18T15:30:21.509244+0000) 2024-03-18T15:31:10.445 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:10 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:10.187+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-03-18T15:30:12.511708+0000 front 2024-03-18T15:30:12.511569+0000 (oldest deadline 2024-03-18T15:30:43.112056+0000) 2024-03-18T15:31:10.445 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:10 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:10.187+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2024-03-18T15:30:25.913763+0000 front 2024-03-18T15:30:25.913855+0000 (oldest deadline 2024-03-18T15:30:49.413885+0000) 2024-03-18T15:31:10.445 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:10 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:10.187+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.19:6822 osd.6 since back 2024-03-18T15:30:29.414560+0000 front 2024-03-18T15:30:29.414721+0000 (oldest deadline 2024-03-18T15:31:00.614742+0000) 2024-03-18T15:31:10.445 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:10 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:10.187+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6820 osd.7 since back 2024-03-18T15:30:41.715820+0000 front 2024-03-18T15:30:41.715852+0000 (oldest deadline 2024-03-18T15:31:07.015645+0000) 2024-03-18T15:31:10.756 INFO:journalctl@ceph.osd.10.smithi033.stdout:Mar 18 15:31:10 smithi033 podman[74756]: 2024-03-18 15:31:10.517172498 +0000 UTC m=+0.606223463 container died 11474d9ff74d41ff691af04dfbc5484e29c8aa196f3e1b486f402b5bc8fa8445 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-10-deactivate, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-03-18T15:31:11.093 DEBUG:teuthology.orchestra.run.smithi033:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@osd.10.service' 2024-03-18T15:31:11.140 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T15:31:11.141 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-03-18T15:31:11.141 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-03-18T15:31:11.141 DEBUG:teuthology.orchestra.run.smithi104:> sudo systemctl stop ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@osd.11 2024-03-18T15:31:11.444 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:11 smithi104 systemd[1]: Stopping Ceph osd.11 for d4dc2ba6-e538-11ee-95c9-87774f69a715... 2024-03-18T15:31:11.445 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:11 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:11.196+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:41.404783+0000 front 2024-03-18T15:29:41.404661+0000 (oldest deadline 2024-03-18T15:30:04.304572+0000) 2024-03-18T15:31:11.445 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:11 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:11.196+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.207053+0000 front 2024-03-18T15:29:51.206898+0000 (oldest deadline 2024-03-18T15:30:13.507003+0000) 2024-03-18T15:31:11.445 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:11 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:11.196+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:59.809088+0000 front 2024-03-18T15:29:59.809253+0000 (oldest deadline 2024-03-18T15:30:21.509244+0000) 2024-03-18T15:31:11.445 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:11 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:11.196+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-03-18T15:30:12.511708+0000 front 2024-03-18T15:30:12.511569+0000 (oldest deadline 2024-03-18T15:30:43.112056+0000) 2024-03-18T15:31:11.445 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:11 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:11.196+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2024-03-18T15:30:25.913763+0000 front 2024-03-18T15:30:25.913855+0000 (oldest deadline 2024-03-18T15:30:49.413885+0000) 2024-03-18T15:31:11.445 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:11 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:11.196+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.19:6822 osd.6 since back 2024-03-18T15:30:29.414560+0000 front 2024-03-18T15:30:29.414721+0000 (oldest deadline 2024-03-18T15:31:00.614742+0000) 2024-03-18T15:31:11.445 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:11 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:11.196+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6820 osd.7 since back 2024-03-18T15:30:41.715820+0000 front 2024-03-18T15:30:41.715852+0000 (oldest deadline 2024-03-18T15:31:07.015645+0000) 2024-03-18T15:31:11.944 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:11 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:11.448+0000 7f5e6a6c8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T15:31:11.944 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:11 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:11.448+0000 7f5e6a6c8700 -1 osd.11 109 *** Got signal Terminated *** 2024-03-18T15:31:11.944 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:11 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:11.448+0000 7f5e6a6c8700 -1 osd.11 109 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T15:31:12.694 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:12 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:12.232+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:41.404783+0000 front 2024-03-18T15:29:41.404661+0000 (oldest deadline 2024-03-18T15:30:04.304572+0000) 2024-03-18T15:31:12.694 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:12 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:12.232+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.207053+0000 front 2024-03-18T15:29:51.206898+0000 (oldest deadline 2024-03-18T15:30:13.507003+0000) 2024-03-18T15:31:12.695 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:12 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:12.232+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:59.809088+0000 front 2024-03-18T15:29:59.809253+0000 (oldest deadline 2024-03-18T15:30:21.509244+0000) 2024-03-18T15:31:12.695 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:12 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:12.232+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-03-18T15:30:12.511708+0000 front 2024-03-18T15:30:12.511569+0000 (oldest deadline 2024-03-18T15:30:43.112056+0000) 2024-03-18T15:31:12.695 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:12 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:12.232+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2024-03-18T15:30:25.913763+0000 front 2024-03-18T15:30:25.913855+0000 (oldest deadline 2024-03-18T15:30:49.413885+0000) 2024-03-18T15:31:12.695 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:12 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:12.232+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.19:6822 osd.6 since back 2024-03-18T15:30:29.414560+0000 front 2024-03-18T15:30:29.414721+0000 (oldest deadline 2024-03-18T15:31:00.614742+0000) 2024-03-18T15:31:12.695 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:12 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:12.232+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6820 osd.7 since back 2024-03-18T15:30:41.715820+0000 front 2024-03-18T15:30:41.715852+0000 (oldest deadline 2024-03-18T15:31:07.015645+0000) 2024-03-18T15:31:13.694 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:13 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:13.203+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:41.404783+0000 front 2024-03-18T15:29:41.404661+0000 (oldest deadline 2024-03-18T15:30:04.304572+0000) 2024-03-18T15:31:13.695 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:13 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:13.203+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.207053+0000 front 2024-03-18T15:29:51.206898+0000 (oldest deadline 2024-03-18T15:30:13.507003+0000) 2024-03-18T15:31:13.695 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:13 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:13.203+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:59.809088+0000 front 2024-03-18T15:29:59.809253+0000 (oldest deadline 2024-03-18T15:30:21.509244+0000) 2024-03-18T15:31:13.695 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:13 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:13.203+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-03-18T15:30:12.511708+0000 front 2024-03-18T15:30:12.511569+0000 (oldest deadline 2024-03-18T15:30:43.112056+0000) 2024-03-18T15:31:13.695 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:13 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:13.203+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2024-03-18T15:30:25.913763+0000 front 2024-03-18T15:30:25.913855+0000 (oldest deadline 2024-03-18T15:30:49.413885+0000) 2024-03-18T15:31:13.695 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:13 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:13.203+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.19:6822 osd.6 since back 2024-03-18T15:30:29.414560+0000 front 2024-03-18T15:30:29.414721+0000 (oldest deadline 2024-03-18T15:31:00.614742+0000) 2024-03-18T15:31:13.695 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:13 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:13.203+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6820 osd.7 since back 2024-03-18T15:30:41.715820+0000 front 2024-03-18T15:30:41.715852+0000 (oldest deadline 2024-03-18T15:31:07.015645+0000) 2024-03-18T15:31:14.694 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:14 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:14.233+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:41.404783+0000 front 2024-03-18T15:29:41.404661+0000 (oldest deadline 2024-03-18T15:30:04.304572+0000) 2024-03-18T15:31:14.694 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:14 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:14.233+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.207053+0000 front 2024-03-18T15:29:51.206898+0000 (oldest deadline 2024-03-18T15:30:13.507003+0000) 2024-03-18T15:31:14.694 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:14 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:14.233+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:59.809088+0000 front 2024-03-18T15:29:59.809253+0000 (oldest deadline 2024-03-18T15:30:21.509244+0000) 2024-03-18T15:31:14.694 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:14 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:14.233+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-03-18T15:30:12.511708+0000 front 2024-03-18T15:30:12.511569+0000 (oldest deadline 2024-03-18T15:30:43.112056+0000) 2024-03-18T15:31:14.695 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:14 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:14.233+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2024-03-18T15:30:25.913763+0000 front 2024-03-18T15:30:25.913855+0000 (oldest deadline 2024-03-18T15:30:49.413885+0000) 2024-03-18T15:31:14.695 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:14 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:14.233+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.19:6822 osd.6 since back 2024-03-18T15:30:29.414560+0000 front 2024-03-18T15:30:29.414721+0000 (oldest deadline 2024-03-18T15:31:00.614742+0000) 2024-03-18T15:31:14.695 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:14 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:14.233+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6820 osd.7 since back 2024-03-18T15:30:41.715820+0000 front 2024-03-18T15:30:41.715852+0000 (oldest deadline 2024-03-18T15:31:07.015645+0000) 2024-03-18T15:31:15.694 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:15 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:15.265+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:41.404783+0000 front 2024-03-18T15:29:41.404661+0000 (oldest deadline 2024-03-18T15:30:04.304572+0000) 2024-03-18T15:31:15.694 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:15 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:15.265+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.207053+0000 front 2024-03-18T15:29:51.206898+0000 (oldest deadline 2024-03-18T15:30:13.507003+0000) 2024-03-18T15:31:15.694 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:15 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:15.265+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:59.809088+0000 front 2024-03-18T15:29:59.809253+0000 (oldest deadline 2024-03-18T15:30:21.509244+0000) 2024-03-18T15:31:15.695 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:15 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:15.265+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-03-18T15:30:12.511708+0000 front 2024-03-18T15:30:12.511569+0000 (oldest deadline 2024-03-18T15:30:43.112056+0000) 2024-03-18T15:31:15.695 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:15 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:15.265+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2024-03-18T15:30:25.913763+0000 front 2024-03-18T15:30:25.913855+0000 (oldest deadline 2024-03-18T15:30:49.413885+0000) 2024-03-18T15:31:15.695 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:15 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:15.265+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.19:6822 osd.6 since back 2024-03-18T15:30:29.414560+0000 front 2024-03-18T15:30:29.414721+0000 (oldest deadline 2024-03-18T15:31:00.614742+0000) 2024-03-18T15:31:15.695 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:15 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:15.265+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6820 osd.7 since back 2024-03-18T15:30:41.715820+0000 front 2024-03-18T15:30:41.715852+0000 (oldest deadline 2024-03-18T15:31:07.015645+0000) 2024-03-18T15:31:16.694 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:16 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:16.299+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-18T15:29:41.404783+0000 front 2024-03-18T15:29:41.404661+0000 (oldest deadline 2024-03-18T15:30:04.304572+0000) 2024-03-18T15:31:16.695 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:16 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:16.299+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6804 osd.1 since back 2024-03-18T15:29:51.207053+0000 front 2024-03-18T15:29:51.206898+0000 (oldest deadline 2024-03-18T15:30:13.507003+0000) 2024-03-18T15:31:16.695 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:16 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:16.299+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2024-03-18T15:29:59.809088+0000 front 2024-03-18T15:29:59.809253+0000 (oldest deadline 2024-03-18T15:30:21.509244+0000) 2024-03-18T15:31:16.695 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:16 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:16.299+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6812 osd.4 since back 2024-03-18T15:30:12.511708+0000 front 2024-03-18T15:30:12.511569+0000 (oldest deadline 2024-03-18T15:30:43.112056+0000) 2024-03-18T15:31:16.695 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:16 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:16.299+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2024-03-18T15:30:25.913763+0000 front 2024-03-18T15:30:25.913855+0000 (oldest deadline 2024-03-18T15:30:49.413885+0000) 2024-03-18T15:31:16.695 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:16 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:16.299+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.19:6822 osd.6 since back 2024-03-18T15:30:29.414560+0000 front 2024-03-18T15:30:29.414721+0000 (oldest deadline 2024-03-18T15:31:00.614742+0000) 2024-03-18T15:31:16.695 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:16 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:16.299+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.33:6820 osd.7 since back 2024-03-18T15:30:41.715820+0000 front 2024-03-18T15:30:41.715852+0000 (oldest deadline 2024-03-18T15:31:07.015645+0000) 2024-03-18T15:31:16.695 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:16 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11[61597]: 2024-03-18T15:31:16.299+0000 7f5e66cbc700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.104:6820 osd.8 since back 2024-03-18T15:30:50.416898+0000 front 2024-03-18T15:30:50.416938+0000 (oldest deadline 2024-03-18T15:31:15.717084+0000) 2024-03-18T15:31:17.323 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:17 smithi104 podman[74147]: 2024-03-18 15:31:17.000162586 +0000 UTC m=+5.668723009 container died 8eaa5539e49ed89a63dce211f94cc12f08b5d4ad8696226fb12a3c52793205b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T15:31:17.324 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:17 smithi104 podman[74147]: 2024-03-18 15:31:17.32334883 +0000 UTC m=+5.991909230 container cleanup 8eaa5539e49ed89a63dce211f94cc12f08b5d4ad8696226fb12a3c52793205b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-03-18T15:31:17.697 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:17 smithi104 bash[74147]: ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11 2024-03-18T15:31:17.698 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:17 smithi104 podman[74164]: 2024-03-18 15:31:17.541127038 +0000 UTC m=+0.538875555 container remove 8eaa5539e49ed89a63dce211f94cc12f08b5d4ad8696226fb12a3c52793205b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True) 2024-03-18T15:31:18.194 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:17 smithi104 podman[74254]: 2024-03-18 15:31:17.834450238 +0000 UTC m=+0.017797878 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T15:31:18.194 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:17 smithi104 podman[74254]: 2024-03-18 15:31:17.933475531 +0000 UTC m=+0.116823181 container create 9bd5a4708e15945ddbec482093b318ccc87dfef69425fc2757a08fd15b6e6981 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5) 2024-03-18T15:31:18.194 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:18 smithi104 podman[74254]: 2024-03-18 15:31:18.160555891 +0000 UTC m=+0.343903529 container init 9bd5a4708e15945ddbec482093b318ccc87dfef69425fc2757a08fd15b6e6981 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240307) 2024-03-18T15:31:18.195 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:18 smithi104 podman[74254]: 2024-03-18 15:31:18.163736206 +0000 UTC m=+0.347083833 container start 9bd5a4708e15945ddbec482093b318ccc87dfef69425fc2757a08fd15b6e6981 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11-deactivate, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T15:31:18.694 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:18 smithi104 podman[74254]: 2024-03-18 15:31:18.227122463 +0000 UTC m=+0.410470093 container attach 9bd5a4708e15945ddbec482093b318ccc87dfef69425fc2757a08fd15b6e6981 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T15:31:18.694 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:18 smithi104 podman[74254]: 2024-03-18 15:31:18.455068709 +0000 UTC m=+0.638416340 container died 9bd5a4708e15945ddbec482093b318ccc87dfef69425fc2757a08fd15b6e6981 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11-deactivate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T15:31:18.988 DEBUG:teuthology.orchestra.run.smithi104:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@osd.11.service' 2024-03-18T15:31:19.014 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:18 smithi104 podman[74282]: 2024-03-18 15:31:18.972105303 +0000 UTC m=+0.512010322 container remove 9bd5a4708e15945ddbec482093b318ccc87dfef69425fc2757a08fd15b6e6981 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-osd-11-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-18T15:31:19.014 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:18 smithi104 systemd[1]: ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@osd.11.service: Deactivated successfully. 2024-03-18T15:31:19.014 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:18 smithi104 systemd[1]: Stopped Ceph osd.11 for d4dc2ba6-e538-11ee-95c9-87774f69a715. 2024-03-18T15:31:19.014 INFO:journalctl@ceph.osd.11.smithi104.stdout:Mar 18 15:31:18 smithi104 systemd[1]: ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@osd.11.service: Consumed 11.656s CPU time. 2024-03-18T15:31:19.028 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T15:31:19.028 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-03-18T15:31:19.028 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-03-18T15:31:19.028 DEBUG:teuthology.orchestra.run.smithi019:> sudo systemctl stop ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@mds.a 2024-03-18T15:31:19.465 INFO:journalctl@ceph.mds.a.smithi019.stdout:Mar 18 15:31:19 smithi019 systemd[1]: Stopping Ceph mds.a for d4dc2ba6-e538-11ee-95c9-87774f69a715... 2024-03-18T15:31:19.838 INFO:journalctl@ceph.mds.a.smithi019.stdout:Mar 18 15:31:19 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-a[73284]: 2024-03-18T15:31:19.627+0000 7f51c7800700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T15:31:19.838 INFO:journalctl@ceph.mds.a.smithi019.stdout:Mar 18 15:31:19 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-a[73284]: 2024-03-18T15:31:19.627+0000 7f51c7800700 -1 mds.a *** got signal Terminated *** 2024-03-18T15:31:23.965 INFO:journalctl@ceph.mds.a.smithi019.stdout:Mar 18 15:31:23 smithi019 podman[93277]: 2024-03-18 15:31:23.558573888 +0000 UTC m=+4.077442532 container died a8b0d96b5a0e94c7b33d7fc240b564844c8270741a9da1240f02ee77d01cff6c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-a, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T15:31:24.397 INFO:journalctl@ceph.mds.a.smithi019.stdout:Mar 18 15:31:24 smithi019 podman[93277]: 2024-03-18 15:31:24.129250842 +0000 UTC m=+4.648119511 container cleanup a8b0d96b5a0e94c7b33d7fc240b564844c8270741a9da1240f02ee77d01cff6c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-a, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2) 2024-03-18T15:31:24.398 INFO:journalctl@ceph.mds.a.smithi019.stdout:Mar 18 15:31:24 smithi019 bash[93277]: ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-a 2024-03-18T15:31:24.715 INFO:journalctl@ceph.mds.a.smithi019.stdout:Mar 18 15:31:24 smithi019 podman[93303]: 2024-03-18 15:31:24.396729659 +0000 UTC m=+0.815911784 container remove a8b0d96b5a0e94c7b33d7fc240b564844c8270741a9da1240f02ee77d01cff6c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T15:31:24.974 DEBUG:teuthology.orchestra.run.smithi019:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@mds.a.service' 2024-03-18T15:31:25.103 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T15:31:25.103 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-03-18T15:31:25.103 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-03-18T15:31:25.103 DEBUG:teuthology.orchestra.run.smithi019:> sudo systemctl stop ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@mds.d 2024-03-18T15:31:25.649 INFO:journalctl@ceph.mds.d.smithi019.stdout:Mar 18 15:31:25 smithi019 systemd[1]: Stopping Ceph mds.d for d4dc2ba6-e538-11ee-95c9-87774f69a715... 2024-03-18T15:31:25.965 INFO:journalctl@ceph.mds.d.smithi019.stdout:Mar 18 15:31:25 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-d[72277]: 2024-03-18T15:31:25.647+0000 7fce42e00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T15:31:25.965 INFO:journalctl@ceph.mds.d.smithi019.stdout:Mar 18 15:31:25 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-d[72277]: 2024-03-18T15:31:25.647+0000 7fce42e00700 -1 mds.d *** got signal Terminated *** 2024-03-18T15:31:27.849 INFO:journalctl@ceph.mds.d.smithi019.stdout:Mar 18 15:31:27 smithi019 podman[93408]: 2024-03-18 15:31:27.576032819 +0000 UTC m=+2.062436480 container died 4dbf9e96e798199a089092d680debe016213431657387c3868326059b73d6b80 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-d, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-03-18T15:31:28.167 INFO:journalctl@ceph.mds.d.smithi019.stdout:Mar 18 15:31:27 smithi019 podman[93408]: 2024-03-18 15:31:27.974272576 +0000 UTC m=+2.460676247 container cleanup 4dbf9e96e798199a089092d680debe016213431657387c3868326059b73d6b80 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-d, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-18T15:31:28.167 INFO:journalctl@ceph.mds.d.smithi019.stdout:Mar 18 15:31:27 smithi019 bash[93408]: ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-d 2024-03-18T15:31:28.465 INFO:journalctl@ceph.mds.d.smithi019.stdout:Mar 18 15:31:28 smithi019 podman[93421]: 2024-03-18 15:31:28.166867806 +0000 UTC m=+0.572906034 container remove 4dbf9e96e798199a089092d680debe016213431657387c3868326059b73d6b80 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-18T15:31:28.681 DEBUG:teuthology.orchestra.run.smithi019:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@mds.d.service' 2024-03-18T15:31:28.805 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T15:31:28.805 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-03-18T15:31:28.806 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-03-18T15:31:28.806 DEBUG:teuthology.orchestra.run.smithi019:> sudo systemctl stop ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@mds.g 2024-03-18T15:31:29.215 INFO:journalctl@ceph.mds.g.smithi019.stdout:Mar 18 15:31:28 smithi019 systemd[1]: Stopping Ceph mds.g for d4dc2ba6-e538-11ee-95c9-87774f69a715... 2024-03-18T15:31:29.965 INFO:journalctl@ceph.mds.g.smithi019.stdout:Mar 18 15:31:29 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-g[72921]: 2024-03-18T15:31:29.683+0000 7f0025600700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T15:31:29.965 INFO:journalctl@ceph.mds.g.smithi019.stdout:Mar 18 15:31:29 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-g[72921]: 2024-03-18T15:31:29.683+0000 7f0025600700 -1 mds.g *** got signal Terminated *** 2024-03-18T15:31:33.681 INFO:journalctl@ceph.mds.g.smithi019.stdout:Mar 18 15:31:33 smithi019 podman[93530]: 2024-03-18 15:31:33.379193896 +0000 UTC m=+4.107119558 container died 939a29143c15835a69062bbfc9bdf99b705204671b92a06c26da1cc6a8de37c1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-g, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T15:31:33.965 INFO:journalctl@ceph.mds.g.smithi019.stdout:Mar 18 15:31:33 smithi019 podman[93530]: 2024-03-18 15:31:33.78578265 +0000 UTC m=+4.513708335 container cleanup 939a29143c15835a69062bbfc9bdf99b705204671b92a06c26da1cc6a8de37c1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-g, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T15:31:33.965 INFO:journalctl@ceph.mds.g.smithi019.stdout:Mar 18 15:31:33 smithi019 bash[93530]: ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-g 2024-03-18T15:31:34.331 INFO:journalctl@ceph.mds.g.smithi019.stdout:Mar 18 15:31:34 smithi019 podman[93544]: 2024-03-18 15:31:34.103539011 +0000 UTC m=+0.706248629 container remove 939a29143c15835a69062bbfc9bdf99b705204671b92a06c26da1cc6a8de37c1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-g, org.label-schema.build-date=20240307, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-03-18T15:31:34.358 DEBUG:teuthology.orchestra.run.smithi019:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@mds.g.service' 2024-03-18T15:31:34.473 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T15:31:34.473 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-03-18T15:31:34.473 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-03-18T15:31:34.473 DEBUG:teuthology.orchestra.run.smithi019:> sudo systemctl stop ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@mds.j 2024-03-18T15:31:34.965 INFO:journalctl@ceph.mds.j.smithi019.stdout:Mar 18 15:31:34 smithi019 systemd[1]: Stopping Ceph mds.j for d4dc2ba6-e538-11ee-95c9-87774f69a715... 2024-03-18T15:31:35.465 INFO:journalctl@ceph.mds.j.smithi019.stdout:Mar 18 15:31:35 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-j[71933]: 2024-03-18T15:31:35.186+0000 7efda1c00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T15:31:35.465 INFO:journalctl@ceph.mds.j.smithi019.stdout:Mar 18 15:31:35 smithi019 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-j[71933]: 2024-03-18T15:31:35.186+0000 7efda1c00700 -1 mds.j *** got signal Terminated *** 2024-03-18T15:31:38.966 INFO:journalctl@ceph.mds.j.smithi019.stdout:Mar 18 15:31:38 smithi019 podman[93649]: 2024-03-18 15:31:38.580620571 +0000 UTC m=+3.635640510 container died a7249e0bbf372003304b247baf96d946ea71f04b7ff0de5863a98bd6bbbcc057 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-j, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-03-18T15:31:39.249 INFO:journalctl@ceph.mds.j.smithi019.stdout:Mar 18 15:31:39 smithi019 podman[93649]: 2024-03-18 15:31:39.072790637 +0000 UTC m=+4.127810596 container cleanup a7249e0bbf372003304b247baf96d946ea71f04b7ff0de5863a98bd6bbbcc057 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-j, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T15:31:39.249 INFO:journalctl@ceph.mds.j.smithi019.stdout:Mar 18 15:31:39 smithi019 bash[93649]: ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-j 2024-03-18T15:31:39.715 INFO:journalctl@ceph.mds.j.smithi019.stdout:Mar 18 15:31:39 smithi019 podman[93675]: 2024-03-18 15:31:39.248378728 +0000 UTC m=+0.648156526 container remove a7249e0bbf372003304b247baf96d946ea71f04b7ff0de5863a98bd6bbbcc057 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-j, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-18T15:31:39.786 DEBUG:teuthology.orchestra.run.smithi019:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@mds.j.service' 2024-03-18T15:31:39.917 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T15:31:39.918 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-03-18T15:31:39.918 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-03-18T15:31:39.918 DEBUG:teuthology.orchestra.run.smithi033:> sudo systemctl stop ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@mds.b 2024-03-18T15:31:40.219 INFO:journalctl@ceph.mds.b.smithi033.stdout:Mar 18 15:31:39 smithi033 systemd[1]: Stopping Ceph mds.b for d4dc2ba6-e538-11ee-95c9-87774f69a715... 2024-03-18T15:31:40.490 INFO:journalctl@ceph.mds.b.smithi033.stdout:Mar 18 15:31:40 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-b[67599]: 2024-03-18T15:31:40.218+0000 7f55fa9a7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T15:31:40.490 INFO:journalctl@ceph.mds.b.smithi033.stdout:Mar 18 15:31:40 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-b[67599]: 2024-03-18T15:31:40.218+0000 7f55fa9a7700 -1 mds.b *** got signal Terminated *** 2024-03-18T15:31:43.819 INFO:journalctl@ceph.mds.b.smithi033.stdout:Mar 18 15:31:43 smithi033 podman[74862]: 2024-03-18 15:31:43.523536205 +0000 UTC m=+3.403257651 container died 79906bb04e4463e396726f62ab3386b98ad98a96ef6db02808826ad54bea588a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-b, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD) 2024-03-18T15:31:43.819 INFO:journalctl@ceph.mds.b.smithi033.stdout:Mar 18 15:31:43 smithi033 podman[74862]: 2024-03-18 15:31:43.818833407 +0000 UTC m=+3.698554815 container cleanup 79906bb04e4463e396726f62ab3386b98ad98a96ef6db02808826ad54bea588a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-b, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-18T15:31:44.176 INFO:journalctl@ceph.mds.b.smithi033.stdout:Mar 18 15:31:43 smithi033 bash[74862]: ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-b 2024-03-18T15:31:44.176 INFO:journalctl@ceph.mds.b.smithi033.stdout:Mar 18 15:31:44 smithi033 podman[74889]: 2024-03-18 15:31:44.053339034 +0000 UTC m=+0.527222096 container remove 79906bb04e4463e396726f62ab3386b98ad98a96ef6db02808826ad54bea588a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-b, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-18T15:31:44.200 DEBUG:teuthology.orchestra.run.smithi033:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@mds.b.service' 2024-03-18T15:31:44.242 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T15:31:44.242 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-03-18T15:31:44.242 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-03-18T15:31:44.243 DEBUG:teuthology.orchestra.run.smithi033:> sudo systemctl stop ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@mds.e 2024-03-18T15:31:44.506 INFO:journalctl@ceph.mds.e.smithi033.stdout:Mar 18 15:31:44 smithi033 systemd[1]: Stopping Ceph mds.e for d4dc2ba6-e538-11ee-95c9-87774f69a715... 2024-03-18T15:31:45.006 INFO:journalctl@ceph.mds.e.smithi033.stdout:Mar 18 15:31:44 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-e[66984]: 2024-03-18T15:31:44.597+0000 7fed61627700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T15:31:45.007 INFO:journalctl@ceph.mds.e.smithi033.stdout:Mar 18 15:31:44 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-e[66984]: 2024-03-18T15:31:44.597+0000 7fed61627700 -1 mds.e *** got signal Terminated *** 2024-03-18T15:31:47.782 INFO:journalctl@ceph.mds.e.smithi033.stdout:Mar 18 15:31:47 smithi033 podman[74997]: 2024-03-18 15:31:47.51695258 +0000 UTC m=+3.043750443 container died 59af355be2aee87c099525cd3c86f53b1e7feb5cd39a2a02c180c0d12dd4c596 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-e, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T15:31:48.124 INFO:journalctl@ceph.mds.e.smithi033.stdout:Mar 18 15:31:47 smithi033 podman[74997]: 2024-03-18 15:31:47.872913965 +0000 UTC m=+3.399711822 container cleanup 59af355be2aee87c099525cd3c86f53b1e7feb5cd39a2a02c180c0d12dd4c596 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-e, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-03-18T15:31:48.125 INFO:journalctl@ceph.mds.e.smithi033.stdout:Mar 18 15:31:47 smithi033 bash[74997]: ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-e 2024-03-18T15:31:48.257 DEBUG:teuthology.orchestra.run.smithi033:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@mds.e.service' 2024-03-18T15:31:48.297 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T15:31:48.297 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-03-18T15:31:48.297 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-03-18T15:31:48.297 DEBUG:teuthology.orchestra.run.smithi033:> sudo systemctl stop ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@mds.h 2024-03-18T15:31:48.756 INFO:journalctl@ceph.mds.h.smithi033.stdout:Mar 18 15:31:48 smithi033 systemd[1]: Stopping Ceph mds.h for d4dc2ba6-e538-11ee-95c9-87774f69a715... 2024-03-18T15:31:48.757 INFO:journalctl@ceph.mds.h.smithi033.stdout:Mar 18 15:31:48 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-h[67294]: 2024-03-18T15:31:48.619+0000 7fd431ba6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T15:31:48.757 INFO:journalctl@ceph.mds.h.smithi033.stdout:Mar 18 15:31:48 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-h[67294]: 2024-03-18T15:31:48.619+0000 7fd431ba6700 -1 mds.h *** got signal Terminated *** 2024-03-18T15:31:51.796 INFO:journalctl@ceph.mds.h.smithi033.stdout:Mar 18 15:31:51 smithi033 podman[75116]: 2024-03-18 15:31:51.51505332 +0000 UTC m=+2.997452584 container died 10fcda2e0d01b9d8509174d765797d74ae0f24036efbdb9df414a583d970e07b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-h, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T15:31:52.130 INFO:journalctl@ceph.mds.h.smithi033.stdout:Mar 18 15:31:51 smithi033 podman[75116]: 2024-03-18 15:31:51.895982291 +0000 UTC m=+3.378381546 container cleanup 10fcda2e0d01b9d8509174d765797d74ae0f24036efbdb9df414a583d970e07b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-h, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-03-18T15:31:52.130 INFO:journalctl@ceph.mds.h.smithi033.stdout:Mar 18 15:31:51 smithi033 bash[75116]: ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-h 2024-03-18T15:31:52.270 DEBUG:teuthology.orchestra.run.smithi033:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@mds.h.service' 2024-03-18T15:31:52.309 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T15:31:52.309 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-03-18T15:31:52.309 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-03-18T15:31:52.309 DEBUG:teuthology.orchestra.run.smithi033:> sudo systemctl stop ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@mds.k 2024-03-18T15:31:52.756 INFO:journalctl@ceph.mds.k.smithi033.stdout:Mar 18 15:31:52 smithi033 systemd[1]: Stopping Ceph mds.k for d4dc2ba6-e538-11ee-95c9-87774f69a715... 2024-03-18T15:31:52.756 INFO:journalctl@ceph.mds.k.smithi033.stdout:Mar 18 15:31:52 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-k[66659]: 2024-03-18T15:31:52.632+0000 7f6d7184a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T15:31:52.757 INFO:journalctl@ceph.mds.k.smithi033.stdout:Mar 18 15:31:52 smithi033 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-k[66659]: 2024-03-18T15:31:52.632+0000 7f6d7184a700 -1 mds.k *** got signal Terminated *** 2024-03-18T15:31:55.758 INFO:journalctl@ceph.mds.k.smithi033.stdout:Mar 18 15:31:55 smithi033 podman[75237]: 2024-03-18 15:31:55.507729418 +0000 UTC m=+2.978258166 container died 9e9cbccb620682120f0bbfec97aa1333b8797c9193aa243427cc147fa9cfb385 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-k, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307) 2024-03-18T15:31:55.758 INFO:journalctl@ceph.mds.k.smithi033.stdout:Mar 18 15:31:55 smithi033 podman[75237]: 2024-03-18 15:31:55.758178367 +0000 UTC m=+3.228707157 container cleanup 9e9cbccb620682120f0bbfec97aa1333b8797c9193aa243427cc147fa9cfb385 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-k, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-18T15:31:56.164 INFO:journalctl@ceph.mds.k.smithi033.stdout:Mar 18 15:31:55 smithi033 bash[75237]: ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-k 2024-03-18T15:31:56.164 INFO:journalctl@ceph.mds.k.smithi033.stdout:Mar 18 15:31:55 smithi033 podman[75251]: 2024-03-18 15:31:55.925663436 +0000 UTC m=+0.407499286 container remove 9e9cbccb620682120f0bbfec97aa1333b8797c9193aa243427cc147fa9cfb385 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-k, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-18T15:31:56.187 DEBUG:teuthology.orchestra.run.smithi033:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@mds.k.service' 2024-03-18T15:31:56.225 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T15:31:56.225 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-03-18T15:31:56.225 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-03-18T15:31:56.225 DEBUG:teuthology.orchestra.run.smithi104:> sudo systemctl stop ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@mds.c 2024-03-18T15:31:56.535 INFO:journalctl@ceph.mds.c.smithi104.stdout:Mar 18 15:31:56 smithi104 systemd[1]: Stopping Ceph mds.c for d4dc2ba6-e538-11ee-95c9-87774f69a715... 2024-03-18T15:31:56.806 INFO:journalctl@ceph.mds.c.smithi104.stdout:Mar 18 15:31:56 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-c[66920]: 2024-03-18T15:31:56.534+0000 7f2b8d0f8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T15:31:56.806 INFO:journalctl@ceph.mds.c.smithi104.stdout:Mar 18 15:31:56 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-c[66920]: 2024-03-18T15:31:56.534+0000 7f2b8d0f8700 -1 mds.c *** got signal Terminated *** 2024-03-18T15:31:59.945 INFO:journalctl@ceph.mds.c.smithi104.stdout:Mar 18 15:31:59 smithi104 podman[74362]: 2024-03-18 15:31:59.515420042 +0000 UTC m=+3.097610402 container died 57fbc4a22e683530fe9224218e84f8dc7ff25508eb6de5061e62bbb11587bf2d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-c, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux ) 2024-03-18T15:32:00.268 INFO:journalctl@ceph.mds.c.smithi104.stdout:Mar 18 15:32:00 smithi104 podman[74362]: 2024-03-18 15:32:00.051058045 +0000 UTC m=+3.633248404 container cleanup 57fbc4a22e683530fe9224218e84f8dc7ff25508eb6de5061e62bbb11587bf2d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-c, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-03-18T15:32:00.268 INFO:journalctl@ceph.mds.c.smithi104.stdout:Mar 18 15:32:00 smithi104 bash[74362]: ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-c 2024-03-18T15:32:00.470 DEBUG:teuthology.orchestra.run.smithi104:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@mds.c.service' 2024-03-18T15:32:00.509 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T15:32:00.510 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-03-18T15:32:00.510 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-03-18T15:32:00.510 DEBUG:teuthology.orchestra.run.smithi104:> sudo systemctl stop ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@mds.f 2024-03-18T15:32:00.830 INFO:journalctl@ceph.mds.f.smithi104.stdout:Mar 18 15:32:00 smithi104 systemd[1]: Stopping Ceph mds.f for d4dc2ba6-e538-11ee-95c9-87774f69a715... 2024-03-18T15:32:01.194 INFO:journalctl@ceph.mds.f.smithi104.stdout:Mar 18 15:32:00 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-f[66607]: 2024-03-18T15:32:00.829+0000 7f5a41e17700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T15:32:01.194 INFO:journalctl@ceph.mds.f.smithi104.stdout:Mar 18 15:32:00 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-f[66607]: 2024-03-18T15:32:00.829+0000 7f5a41e17700 -1 mds.f *** got signal Terminated *** 2024-03-18T15:32:03.805 INFO:journalctl@ceph.mds.f.smithi104.stdout:Mar 18 15:32:03 smithi104 podman[74493]: 2024-03-18 15:32:03.519651802 +0000 UTC m=+2.794994451 container died 19039ff9850c38be0733eff47fb0ece02c1a98e3c2d5be54be6d43fbffaa0c2c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-f, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS) 2024-03-18T15:32:04.122 INFO:journalctl@ceph.mds.f.smithi104.stdout:Mar 18 15:32:03 smithi104 podman[74493]: 2024-03-18 15:32:03.896313504 +0000 UTC m=+3.171656144 container cleanup 19039ff9850c38be0733eff47fb0ece02c1a98e3c2d5be54be6d43fbffaa0c2c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-f, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True) 2024-03-18T15:32:04.122 INFO:journalctl@ceph.mds.f.smithi104.stdout:Mar 18 15:32:03 smithi104 bash[74493]: ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-f 2024-03-18T15:32:04.255 DEBUG:teuthology.orchestra.run.smithi104:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@mds.f.service' 2024-03-18T15:32:04.292 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T15:32:04.292 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-03-18T15:32:04.292 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-03-18T15:32:04.293 DEBUG:teuthology.orchestra.run.smithi104:> sudo systemctl stop ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@mds.i 2024-03-18T15:32:04.695 INFO:journalctl@ceph.mds.i.smithi104.stdout:Mar 18 15:32:04 smithi104 systemd[1]: Stopping Ceph mds.i for d4dc2ba6-e538-11ee-95c9-87774f69a715... 2024-03-18T15:32:04.695 INFO:journalctl@ceph.mds.i.smithi104.stdout:Mar 18 15:32:04 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-i[67232]: 2024-03-18T15:32:04.599+0000 7f2e92015700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T15:32:04.695 INFO:journalctl@ceph.mds.i.smithi104.stdout:Mar 18 15:32:04 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-i[67232]: 2024-03-18T15:32:04.599+0000 7f2e92015700 -1 mds.i *** got signal Terminated *** 2024-03-18T15:32:07.807 INFO:journalctl@ceph.mds.i.smithi104.stdout:Mar 18 15:32:07 smithi104 podman[74616]: 2024-03-18 15:32:07.509265008 +0000 UTC m=+2.999970808 container died 128cfe8487678eda64be790cae928f6249e791c30f525500e1fd51604bb891d9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-i, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True) 2024-03-18T15:32:07.807 INFO:journalctl@ceph.mds.i.smithi104.stdout:Mar 18 15:32:07 smithi104 podman[74616]: 2024-03-18 15:32:07.807318259 +0000 UTC m=+3.298024062 container cleanup 128cfe8487678eda64be790cae928f6249e791c30f525500e1fd51604bb891d9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-i, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-18T15:32:08.130 INFO:journalctl@ceph.mds.i.smithi104.stdout:Mar 18 15:32:07 smithi104 bash[74616]: ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-i 2024-03-18T15:32:08.130 INFO:journalctl@ceph.mds.i.smithi104.stdout:Mar 18 15:32:08 smithi104 podman[74630]: 2024-03-18 15:32:08.016950196 +0000 UTC m=+0.499469384 container remove 128cfe8487678eda64be790cae928f6249e791c30f525500e1fd51604bb891d9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-i, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, org.label-schema.build-date=20240307, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-03-18T15:32:08.154 DEBUG:teuthology.orchestra.run.smithi104:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@mds.i.service' 2024-03-18T15:32:08.189 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T15:32:08.189 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-03-18T15:32:08.189 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-03-18T15:32:08.189 DEBUG:teuthology.orchestra.run.smithi104:> sudo systemctl stop ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@mds.l 2024-03-18T15:32:08.444 INFO:journalctl@ceph.mds.l.smithi104.stdout:Mar 18 15:32:08 smithi104 systemd[1]: Stopping Ceph mds.l for d4dc2ba6-e538-11ee-95c9-87774f69a715... 2024-03-18T15:32:08.944 INFO:journalctl@ceph.mds.l.smithi104.stdout:Mar 18 15:32:08 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-l[66300]: 2024-03-18T15:32:08.501+0000 7ff067a29700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T15:32:08.944 INFO:journalctl@ceph.mds.l.smithi104.stdout:Mar 18 15:32:08 smithi104 ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-l[66300]: 2024-03-18T15:32:08.501+0000 7ff067a29700 -1 mds.l *** got signal Terminated *** 2024-03-18T15:32:11.853 INFO:journalctl@ceph.mds.l.smithi104.stdout:Mar 18 15:32:11 smithi104 podman[74736]: 2024-03-18 15:32:11.512371437 +0000 UTC m=+3.104420617 container died 4c7a166e007e4dc338410efaed5df891fa433459fad2a8246ae28fc3637e052e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-l, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS) 2024-03-18T15:32:12.128 INFO:journalctl@ceph.mds.l.smithi104.stdout:Mar 18 15:32:11 smithi104 podman[74736]: 2024-03-18 15:32:11.927178316 +0000 UTC m=+3.519227474 container cleanup 4c7a166e007e4dc338410efaed5df891fa433459fad2a8246ae28fc3637e052e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-l, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T15:32:12.128 INFO:journalctl@ceph.mds.l.smithi104.stdout:Mar 18 15:32:11 smithi104 bash[74736]: ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715-mds-l 2024-03-18T15:32:12.243 DEBUG:teuthology.orchestra.run.smithi104:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d4dc2ba6-e538-11ee-95c9-87774f69a715@mds.l.service' 2024-03-18T15:32:12.277 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T15:32:12.277 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-03-18T15:32:12.277 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 --force --keep-logs 2024-03-18T15:32:12.621 INFO:teuthology.orchestra.run.smithi019.stdout:Deleting cluster with fsid: d4dc2ba6-e538-11ee-95c9-87774f69a715 2024-03-18T15:32:17.678 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 --force --keep-logs 2024-03-18T15:32:17.881 INFO:teuthology.orchestra.run.smithi033.stdout:Deleting cluster with fsid: d4dc2ba6-e538-11ee-95c9-87774f69a715 2024-03-18T15:32:20.500 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 --force --keep-logs 2024-03-18T15:32:20.692 INFO:teuthology.orchestra.run.smithi104.stdout:Deleting cluster with fsid: d4dc2ba6-e538-11ee-95c9-87774f69a715 2024-03-18T15:32:23.163 DEBUG:teuthology.orchestra.run.smithi019:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-18T15:32:23.273 DEBUG:teuthology.orchestra.run.smithi033:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-18T15:32:23.312 DEBUG:teuthology.orchestra.run.smithi104:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-18T15:32:23.340 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-18T15:32:23.343 DEBUG:teuthology.misc:Transferring archived files from smithi019:/var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/crash to /home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608866/remote/smithi019/crash 2024-03-18T15:32:23.344 DEBUG:teuthology.orchestra.run.smithi019:> sudo tar c -f - -C /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/crash -- . 2024-03-18T15:32:23.463 INFO:teuthology.orchestra.run.smithi019.stderr:tar: /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/crash: Cannot open: No such file or directory 2024-03-18T15:32:23.463 INFO:teuthology.orchestra.run.smithi019.stderr:tar: Error is not recoverable: exiting now 2024-03-18T15:32:23.468 DEBUG:teuthology.misc:Transferring archived files from smithi033:/var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/crash to /home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608866/remote/smithi033/crash 2024-03-18T15:32:23.469 DEBUG:teuthology.orchestra.run.smithi033:> sudo tar c -f - -C /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/crash -- . 2024-03-18T15:32:23.519 INFO:teuthology.orchestra.run.smithi033.stderr:tar: /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/crash: Cannot open: No such file or directory 2024-03-18T15:32:23.519 INFO:teuthology.orchestra.run.smithi033.stderr:tar: Error is not recoverable: exiting now 2024-03-18T15:32:23.521 DEBUG:teuthology.misc:Transferring archived files from smithi104:/var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/crash to /home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608866/remote/smithi104/crash 2024-03-18T15:32:23.522 DEBUG:teuthology.orchestra.run.smithi104:> sudo tar c -f - -C /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/crash -- . 2024-03-18T15:32:23.560 INFO:teuthology.orchestra.run.smithi104.stderr:tar: /var/lib/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/crash: Cannot open: No such file or directory 2024-03-18T15:32:23.561 INFO:teuthology.orchestra.run.smithi104.stderr:tar: Error is not recoverable: exiting now 2024-03-18T15:32:23.562 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-18T15:32:23.562 DEBUG:teuthology.orchestra.run.smithi019:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'mons down, quorum' | egrep -v 'out of quorum' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | head -n 1 2024-03-18T15:32:23.673 INFO:teuthology.orchestra.run.smithi019.stdout:2024-03-18T15:16:16.808370+0000 mon.a (mon.0) 1101 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-03-18T15:32:23.674 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-18T15:32:23.674 DEBUG:teuthology.orchestra.run.smithi019:> sudo egrep '\[SEC\]' /var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'mons down, quorum' | egrep -v 'out of quorum' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | head -n 1 2024-03-18T15:32:23.769 DEBUG:teuthology.orchestra.run.smithi019:> sudo egrep '\[ERR\]' /var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'mons down, quorum' | egrep -v 'out of quorum' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | head -n 1 2024-03-18T15:32:23.854 DEBUG:teuthology.orchestra.run.smithi019:> sudo egrep '\[WRN\]' /var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'mons down, quorum' | egrep -v 'out of quorum' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | head -n 1 2024-03-18T15:32:23.943 INFO:teuthology.orchestra.run.smithi019.stdout:2024-03-18T15:16:16.808370+0000 mon.a (mon.0) 1101 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-03-18T15:32:23.944 INFO:tasks.cephadm:Compressing logs... 2024-03-18T15:32:23.945 DEBUG:teuthology.orchestra.run.smithi019:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-18T15:32:23.949 DEBUG:teuthology.orchestra.run.smithi033:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-18T15:32:23.951 DEBUG:teuthology.orchestra.run.smithi104:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-18T15:32:23.979 INFO:teuthology.orchestra.run.smithi104.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-18T15:32:23.979 INFO:teuthology.orchestra.run.smithi104.stderr:: No such file or directory 2024-03-18T15:32:23.979 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph-mds.l.log 2024-03-18T15:32:23.979 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph.audit.log 2024-03-18T15:32:23.980 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph-mds.l.log: gzip -5 --verbose -- /var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph-mds.i.log 2024-03-18T15:32:23.980 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph-osd.11.log 2024-03-18T15:32:23.981 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph-mds.i.log: gzip -5 --verbose -- /var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph-mgr.z.log 2024-03-18T15:32:23.981 INFO:teuthology.orchestra.run.smithi104.stderr: 92.5% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-18T15:32:23.981 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph-osd.8.log 2024-03-18T15:32:23.981 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph-osd.5.log 2024-03-18T15:32:23.981 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph-mds.f.log 2024-03-18T15:32:23.982 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph.cephadm.log 2024-03-18T15:32:23.983 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph-osd.5.log: /var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph-mds.f.log: 90.3% -- replaced with /var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph.audit.log.gz 2024-03-18T15:32:23.984 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph.log 2024-03-18T15:32:23.984 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph.cephadm.log: 83.4% -- replaced with /var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph.cephadm.log.gz 2024-03-18T15:32:23.985 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph-osd.2.log 2024-03-18T15:32:23.986 INFO:teuthology.orchestra.run.smithi033.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-03-18T15:32:23.986 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-18T15:32:23.986 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph-osd.10.log 2024-03-18T15:32:23.986 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph.audit.log 2024-03-18T15:32:23.986 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph-mds.b.log 2024-03-18T15:32:23.986 INFO:teuthology.orchestra.run.smithi033.stderr:gzip/var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph.audit.log: -5 --verbose -- /var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph-osd.4.log 2024-03-18T15:32:23.986 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph-mds.b.log: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-18T15:32:23.986 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph-osd.1.log 2024-03-18T15:32:23.989 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph-mds.k.log 2024-03-18T15:32:23.989 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph-osd.4.log: /var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph-mds.h.log 2024-03-18T15:32:23.989 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph-mds.k.log: gzip -5 --verbose -- /var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph-osd.7.log 2024-03-18T15:32:23.990 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph-mds.e.log 2024-03-18T15:32:23.990 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph-osd.7.log: 90.3% -- replaced with /var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph.audit.log.gz 2024-03-18T15:32:23.990 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph-mgr.y.log 2024-03-18T15:32:23.992 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph.log: 87.5% -- replaced with /var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph.log.gz 2024-03-18T15:32:23.992 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph-mds.c.log 2024-03-18T15:32:23.992 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph-osd.2.log: 91.9% -- replaced with /var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph-mgr.z.log.gz 2024-03-18T15:32:23.992 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph-volume.log 2024-03-18T15:32:24.001 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph.cephadm.log 2024-03-18T15:32:24.003 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph-mon.c.log 2024-03-18T15:32:24.012 INFO:teuthology.orchestra.run.smithi019.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-03-18T15:32:24.012 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-18T15:32:24.013 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph-osd.6.log 2024-03-18T15:32:24.014 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph-mon.a.log 2024-03-18T15:32:24.016 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph-mgr.y.log: 91.9%gzip -5 --verbose -- /var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph.log 2024-03-18T15:32:24.016 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph.cephadm.log: 83.4% -- replaced with /var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph.cephadm.log.gz 2024-03-18T15:32:24.017 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph.audit.log 2024-03-18T15:32:24.017 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph-mds.g.log 2024-03-18T15:32:24.018 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph-mds.j.log 2024-03-18T15:32:24.020 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph-osd.3.log 2024-03-18T15:32:24.021 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph-volume.log 2024-03-18T15:32:24.021 INFO:teuthology.orchestra.run.smithi033.stderr: -- replaced with /var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph-mgr.y.log.gz 2024-03-18T15:32:24.022 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph-mgr.x.log 2024-03-18T15:32:24.022 INFO:teuthology.orchestra.run.smithi019.stderr: 90.2% -- replaced with /var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph.audit.log.gz 2024-03-18T15:32:24.022 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph-osd.0.log 2024-03-18T15:32:24.025 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph-mgr.x.log: /var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph-osd.0.log: gzip 93.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-18T15:32:24.025 INFO:teuthology.orchestra.run.smithi019.stderr: -5 --verbose -- /var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph.cephadm.log 2024-03-18T15:32:24.027 INFO:teuthology.orchestra.run.smithi019.stderr: 94.5% -- replaced with /var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph-mds.g.log.gz 2024-03-18T15:32:24.027 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph.log 2024-03-18T15:32:24.027 INFO:teuthology.orchestra.run.smithi019.stderr: 94.6% -- replaced with /var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph-mds.j.log.gz 2024-03-18T15:32:24.029 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph-mon.b.log 2024-03-18T15:32:24.029 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph.cephadm.log: 83.9% -- replaced with /var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph.cephadm.log.gz 2024-03-18T15:32:24.030 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph-volume.log: /var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph-mon.c.log: 90.9% -- replaced with /var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph-volume.log.gz 2024-03-18T15:32:24.031 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph-volume.log 2024-03-18T15:32:24.034 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph-osd.9.log 2024-03-18T15:32:24.034 INFO:teuthology.orchestra.run.smithi019.stderr: 87.5% -- replaced with /var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph.log.gz 2024-03-18T15:32:24.036 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph-volume.log: /var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph-osd.9.log: gzip -5 --verbose -- /var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph-mds.d.log 2024-03-18T15:32:24.037 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph-volume.log: 87.5% -- replaced with /var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph.log.gz 2024-03-18T15:32:24.038 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph-mds.a.log 2024-03-18T15:32:24.040 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph-mds.d.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.log 2024-03-18T15:32:24.042 INFO:teuthology.orchestra.run.smithi104.stderr: 93.5% -- replaced with /var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph-mds.i.log.gz 2024-03-18T15:32:24.044 INFO:teuthology.orchestra.run.smithi104.stderr: 93.4% -- replaced with /var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph-mds.f.log.gz 2024-03-18T15:32:24.046 INFO:teuthology.orchestra.run.smithi104.stderr: 94.5% -- replaced with /var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph-mds.l.log.gz 2024-03-18T15:32:24.051 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph-mon.b.log: 93.4% -- replaced with /var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph-mds.h.log.gz 2024-03-18T15:32:24.052 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph-mds.a.log: /var/log/ceph/ceph-client.admin.log: 92.2% -- replaced with /var/log/ceph/ceph-client.admin.log.gz 2024-03-18T15:32:24.056 INFO:teuthology.orchestra.run.smithi019.stderr: 90.8% -- replaced with /var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph-volume.log.gz 2024-03-18T15:32:24.066 INFO:teuthology.orchestra.run.smithi033.stderr: 90.9% -- replaced with /var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph-volume.log.gz 2024-03-18T15:32:24.067 INFO:teuthology.orchestra.run.smithi033.stderr: 94.3% -- replaced with /var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph-mds.k.log.gz 2024-03-18T15:32:24.073 INFO:teuthology.orchestra.run.smithi033.stderr: 93.9% -- replaced with /var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph-mds.e.log.gz 2024-03-18T15:32:24.118 INFO:teuthology.orchestra.run.smithi019.stderr: 94.5% -- replaced with /var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph-mds.d.log.gz 2024-03-18T15:32:24.119 INFO:teuthology.orchestra.run.smithi104.stderr: 93.0% -- replaced with /var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph-mds.c.log.gz 2024-03-18T15:32:24.127 INFO:teuthology.orchestra.run.smithi019.stderr: 94.4% -- replaced with /var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph-mds.a.log.gz 2024-03-18T15:32:24.286 INFO:teuthology.orchestra.run.smithi033.stderr: 93.3% -- replaced with /var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph-mds.b.log.gz 2024-03-18T15:32:24.386 INFO:teuthology.orchestra.run.smithi033.stderr: 91.8% -- replaced with /var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph-mon.b.log.gz 2024-03-18T15:32:24.400 INFO:teuthology.orchestra.run.smithi019.stderr: 89.7% -- replaced with /var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph-mgr.x.log.gz 2024-03-18T15:32:24.591 INFO:teuthology.orchestra.run.smithi104.stderr: 91.8% -- replaced with /var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph-mon.c.log.gz 2024-03-18T15:32:24.944 INFO:teuthology.orchestra.run.smithi019.stderr: 90.4% -- replaced with /var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph-mon.a.log.gz 2024-03-18T15:32:25.662 INFO:teuthology.orchestra.run.smithi104.stderr: 92.0% -- replaced with /var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph-osd.11.log.gz 2024-03-18T15:32:26.211 INFO:teuthology.orchestra.run.smithi033.stderr: 92.0% -- replaced with /var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph-osd.10.log.gz 2024-03-18T15:32:26.440 INFO:teuthology.orchestra.run.smithi104.stderr: 91.7% -- replaced with /var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph-osd.2.log.gz 2024-03-18T15:32:26.445 INFO:teuthology.orchestra.run.smithi104.stderr: 92.2% -- replaced with /var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph-osd.8.log.gz 2024-03-18T15:32:26.480 INFO:teuthology.orchestra.run.smithi033.stderr: 92.4% -- replaced with /var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph-osd.1.log.gz 2024-03-18T15:32:26.522 INFO:teuthology.orchestra.run.smithi033.stderr: 92.1% -- replaced with /var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph-osd.7.log.gz 2024-03-18T15:32:26.523 INFO:teuthology.orchestra.run.smithi019.stderr: 91.9% -- replaced with /var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph-osd.0.log.gz 2024-03-18T15:32:26.821 INFO:teuthology.orchestra.run.smithi019.stderr: 92.3% -- replaced with /var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph-osd.6.log.gz 2024-03-18T15:32:27.281 INFO:teuthology.orchestra.run.smithi033.stderr: 92.0% -- replaced with /var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph-osd.4.log.gz 2024-03-18T15:32:27.284 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-03-18T15:32:27.284 INFO:teuthology.orchestra.run.smithi033.stderr:real 0m3.318s 2024-03-18T15:32:27.284 INFO:teuthology.orchestra.run.smithi033.stderr:user 0m11.092s 2024-03-18T15:32:27.284 INFO:teuthology.orchestra.run.smithi033.stderr:sys 0m0.347s 2024-03-18T15:32:27.314 INFO:teuthology.orchestra.run.smithi019.stderr: 91.7% -- replaced with /var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph-osd.3.log.gz 2024-03-18T15:32:27.330 INFO:teuthology.orchestra.run.smithi104.stderr: 91.8% -- replaced with /var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph-osd.5.log.gz 2024-03-18T15:32:27.332 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-03-18T15:32:27.332 INFO:teuthology.orchestra.run.smithi104.stderr:real 0m3.366s 2024-03-18T15:32:27.332 INFO:teuthology.orchestra.run.smithi104.stderr:user 0m10.470s 2024-03-18T15:32:27.332 INFO:teuthology.orchestra.run.smithi104.stderr:sys 0m0.373s 2024-03-18T15:32:27.713 INFO:teuthology.orchestra.run.smithi019.stderr: 92.0% -- replaced with /var/log/ceph/d4dc2ba6-e538-11ee-95c9-87774f69a715/ceph-osd.9.log.gz 2024-03-18T15:32:27.717 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-03-18T15:32:27.717 INFO:teuthology.orchestra.run.smithi019.stderr:real 0m3.736s 2024-03-18T15:32:27.717 INFO:teuthology.orchestra.run.smithi019.stderr:user 0m12.860s 2024-03-18T15:32:27.717 INFO:teuthology.orchestra.run.smithi019.stderr:sys 0m1.020s 2024-03-18T15:32:27.718 INFO:tasks.cephadm:Archiving logs... 2024-03-18T15:32:27.719 DEBUG:teuthology.misc:Transferring archived files from smithi019:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608866/remote/smithi019/log 2024-03-18T15:32:27.719 DEBUG:teuthology.orchestra.run.smithi019:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-18T15:32:29.745 DEBUG:teuthology.misc:Transferring archived files from smithi033:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608866/remote/smithi033/log 2024-03-18T15:32:29.746 DEBUG:teuthology.orchestra.run.smithi033:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-18T15:32:31.532 DEBUG:teuthology.misc:Transferring archived files from smithi104:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608866/remote/smithi104/log 2024-03-18T15:32:31.533 DEBUG:teuthology.orchestra.run.smithi104:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-18T15:32:33.215 INFO:tasks.cephadm:Removing cluster... 2024-03-18T15:32:33.215 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 --force 2024-03-18T15:32:33.559 INFO:teuthology.orchestra.run.smithi019.stdout:Deleting cluster with fsid: d4dc2ba6-e538-11ee-95c9-87774f69a715 2024-03-18T15:32:34.217 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 --force 2024-03-18T15:32:34.421 INFO:teuthology.orchestra.run.smithi033.stdout:Deleting cluster with fsid: d4dc2ba6-e538-11ee-95c9-87774f69a715 2024-03-18T15:32:34.808 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid d4dc2ba6-e538-11ee-95c9-87774f69a715 --force 2024-03-18T15:32:34.997 INFO:teuthology.orchestra.run.smithi104.stdout:Deleting cluster with fsid: d4dc2ba6-e538-11ee-95c9-87774f69a715 2024-03-18T15:32:35.267 INFO:tasks.cephadm:Removing cephadm ... 2024-03-18T15:32:35.267 DEBUG:teuthology.orchestra.run.smithi019:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-18T15:32:35.327 DEBUG:teuthology.orchestra.run.smithi033:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-18T15:32:35.344 DEBUG:teuthology.orchestra.run.smithi104:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-18T15:32:35.359 INFO:tasks.cephadm:Teardown complete 2024-03-18T15:32:35.360 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-18T15:32:35.371 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-18T15:32:35.371 DEBUG:teuthology.orchestra.run.smithi019:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-18T15:32:35.379 DEBUG:teuthology.orchestra.run.smithi033:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-18T15:32:35.387 DEBUG:teuthology.orchestra.run.smithi104:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-18T15:32:35.557 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-18T15:32:35.569 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-18T15:32:35.570 DEBUG:teuthology.orchestra.run.smithi019:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-18T15:32:35.578 DEBUG:teuthology.orchestra.run.smithi033:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-18T15:32:35.581 DEBUG:teuthology.orchestra.run.smithi104:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-18T15:32:35.594 INFO:teuthology.orchestra.run.smithi033.stderr:bash: line 1: ntpq: command not found 2024-03-18T15:32:35.594 INFO:teuthology.orchestra.run.smithi104.stderr:bash: line 1: ntpq: command not found 2024-03-18T15:32:35.599 INFO:teuthology.orchestra.run.smithi104.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-18T15:32:35.599 INFO:teuthology.orchestra.run.smithi104.stdout:=============================================================================== 2024-03-18T15:32:35.599 INFO:teuthology.orchestra.run.smithi104.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 9 +1978us[+1978us] +/- 96ms 2024-03-18T15:32:35.599 INFO:teuthology.orchestra.run.smithi104.stdout:^* hv02.front.sepia.ceph.com 3 6 377 10 -1598us[-1605us] +/- 73ms 2024-03-18T15:32:35.599 INFO:teuthology.orchestra.run.smithi104.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 7 +246us[ +246us] +/- 66ms 2024-03-18T15:32:35.600 INFO:teuthology.orchestra.run.smithi104.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T15:32:35.601 INFO:teuthology.orchestra.run.smithi033.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-18T15:32:35.601 INFO:teuthology.orchestra.run.smithi033.stdout:=============================================================================== 2024-03-18T15:32:35.601 INFO:teuthology.orchestra.run.smithi033.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 8 +1942us[+1961us] +/- 96ms 2024-03-18T15:32:35.601 INFO:teuthology.orchestra.run.smithi033.stdout:^* hv02.front.sepia.ceph.com 3 6 377 6 -1636us[-1618us] +/- 73ms 2024-03-18T15:32:35.601 INFO:teuthology.orchestra.run.smithi033.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 10 +213us[ +231us] +/- 66ms 2024-03-18T15:32:35.601 INFO:teuthology.orchestra.run.smithi033.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T15:32:35.628 INFO:teuthology.orchestra.run.smithi019.stderr:bash: line 1: ntpq: command not found 2024-03-18T15:32:35.638 INFO:teuthology.orchestra.run.smithi019.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-18T15:32:35.638 INFO:teuthology.orchestra.run.smithi019.stdout:=============================================================================== 2024-03-18T15:32:35.639 INFO:teuthology.orchestra.run.smithi019.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 8 +1926us[+1910us] +/- 96ms 2024-03-18T15:32:35.639 INFO:teuthology.orchestra.run.smithi019.stdout:^* hv02.front.sepia.ceph.com 3 6 377 7 -1633us[-1649us] +/- 73ms 2024-03-18T15:32:35.639 INFO:teuthology.orchestra.run.smithi019.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 9 +206us[ +189us] +/- 66ms 2024-03-18T15:32:35.639 INFO:teuthology.orchestra.run.smithi019.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T15:32:35.640 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-18T15:32:35.652 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-18T15:32:35.653 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-18T15:32:35.673 DEBUG:teuthology.orchestra.run.smithi019:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-18T15:32:35.928 DEBUG:teuthology.orchestra.run.smithi033:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-18T15:32:36.015 DEBUG:teuthology.orchestra.run.smithi104:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-18T15:32:36.087 DEBUG:teuthology.orchestra.run.smithi019:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-18T15:32:36.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T15:32:36.199 DEBUG:teuthology.orchestra.run.smithi033:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-18T15:32:36.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T15:32:36.236 DEBUG:teuthology.orchestra.run.smithi104:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-18T15:32:36.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T15:32:36.266 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-18T15:32:36.277 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-18T15:32:36.315 INFO:teuthology.task.internal:Duration was 2193.519199 seconds 2024-03-18T15:32:36.316 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-18T15:32:36.350 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-18T15:32:36.351 DEBUG:teuthology.orchestra.run.smithi019:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-18T15:32:36.359 DEBUG:teuthology.orchestra.run.smithi033:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-18T15:32:36.361 DEBUG:teuthology.orchestra.run.smithi104:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-18T15:32:36.404 INFO:teuthology.orchestra.run.smithi104.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-18T15:32:36.412 INFO:teuthology.orchestra.run.smithi033.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-18T15:32:36.526 INFO:teuthology.orchestra.run.smithi019.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-18T15:32:36.813 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-18T15:32:36.813 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi019.front.sepia.ceph.com 2024-03-18T15:32:36.813 DEBUG:teuthology.orchestra.run.smithi019:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-18T15:32:36.881 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi033.front.sepia.ceph.com 2024-03-18T15:32:36.881 DEBUG:teuthology.orchestra.run.smithi033:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-18T15:32:36.910 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi104.front.sepia.ceph.com 2024-03-18T15:32:36.910 DEBUG:teuthology.orchestra.run.smithi104:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-18T15:32:36.932 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-18T15:32:36.932 DEBUG:teuthology.orchestra.run.smithi019:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-18T15:32:36.940 DEBUG:teuthology.orchestra.run.smithi033:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-18T15:32:36.953 DEBUG:teuthology.orchestra.run.smithi104:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-18T15:32:37.039 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-18T15:32:37.039 DEBUG:teuthology.orchestra.run.smithi019:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-18T15:32:37.045 DEBUG:teuthology.orchestra.run.smithi033:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-18T15:32:37.047 DEBUG:teuthology.orchestra.run.smithi104:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-18T15:32:37.578 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-18T15:32:37.591 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-18T15:32:37.591 DEBUG:teuthology.orchestra.run.smithi019:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-18T15:32:37.690 DEBUG:teuthology.orchestra.run.smithi033:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-18T15:32:37.723 DEBUG:teuthology.orchestra.run.smithi104:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-18T15:32:37.751 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-18T15:32:37.762 DEBUG:teuthology.orchestra.run.smithi019:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-18T15:32:37.770 DEBUG:teuthology.orchestra.run.smithi033:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-18T15:32:37.772 DEBUG:teuthology.orchestra.run.smithi104:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-18T15:32:37.800 INFO:teuthology.orchestra.run.smithi033.stdout:kernel.core_pattern = core 2024-03-18T15:32:37.815 INFO:teuthology.orchestra.run.smithi104.stdout:kernel.core_pattern = core 2024-03-18T15:32:37.877 INFO:teuthology.orchestra.run.smithi019.stdout:kernel.core_pattern = core 2024-03-18T15:32:37.952 DEBUG:teuthology.orchestra.run.smithi019:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-18T15:32:38.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T15:32:38.009 DEBUG:teuthology.orchestra.run.smithi033:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-18T15:32:38.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T15:32:38.025 DEBUG:teuthology.orchestra.run.smithi104:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-18T15:32:38.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T15:32:38.039 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-18T15:32:38.050 INFO:teuthology.task.internal:Transferring archived files... 2024-03-18T15:32:38.051 DEBUG:teuthology.misc:Transferring archived files from smithi019:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608866/remote/smithi019 2024-03-18T15:32:38.051 DEBUG:teuthology.orchestra.run.smithi019:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-18T15:32:38.174 DEBUG:teuthology.misc:Transferring archived files from smithi033:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608866/remote/smithi033 2024-03-18T15:32:38.174 DEBUG:teuthology.orchestra.run.smithi033:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-18T15:32:38.219 DEBUG:teuthology.misc:Transferring archived files from smithi104:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608866/remote/smithi104 2024-03-18T15:32:38.220 DEBUG:teuthology.orchestra.run.smithi104:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-18T15:32:38.258 INFO:teuthology.task.internal:Removing archive directory... 2024-03-18T15:32:38.258 DEBUG:teuthology.orchestra.run.smithi019:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-18T15:32:38.266 DEBUG:teuthology.orchestra.run.smithi033:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-18T15:32:38.268 DEBUG:teuthology.orchestra.run.smithi104:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-18T15:32:38.320 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-18T15:32:38.331 INFO:teuthology.task.internal:Not uploading archives. 2024-03-18T15:32:38.331 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-18T15:32:38.433 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-18T15:32:38.433 DEBUG:teuthology.orchestra.run.smithi019:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-18T15:32:38.441 DEBUG:teuthology.orchestra.run.smithi033:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-18T15:32:38.444 DEBUG:teuthology.orchestra.run.smithi104:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-18T15:32:38.458 INFO:teuthology.orchestra.run.smithi104.stdout: 262162 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 18 15:32 /home/ubuntu/cephtest 2024-03-18T15:32:38.458 INFO:teuthology.orchestra.run.smithi033.stdout: 262162 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 18 15:32 /home/ubuntu/cephtest 2024-03-18T15:32:38.494 INFO:teuthology.orchestra.run.smithi019.stdout: 393229 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 18 15:32 /home/ubuntu/cephtest 2024-03-18T15:32:38.498 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-18T15:32:38.510 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-18T15:32:38.567 INFO:teuthology.nuke:Checking targets against current locks 2024-03-18T15:32:38.594 DEBUG:teuthology.nuke:shortname: smithi019 2024-03-18T15:32:38.595 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-18T15:32:38.620 DEBUG:teuthology.nuke:shortname: smithi033 2024-03-18T15:32:38.620 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-18T15:32:38.626 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi019.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608866', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-18 14:47:06.901319', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMi36RYgvG4AraWlJgExqmzfxDhqmHf8HPuvRZse4mMaEql3bjgV0xyq7VJJUIa319dm8A6Liu5CDhgt73TBJrk='} 2024-03-18T15:32:38.648 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi033.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608866', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-18 14:47:06.902738', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNr0GJOBuYiogmoLTbOrsspKiHUxCKK/NaMD1IBqLwCKxp+zCGAt81olsCTp/rSf0yudJmvPJRMj/tOTHKA4Sic='} 2024-03-18T15:32:38.653 DEBUG:teuthology.nuke:shortname: smithi104 2024-03-18T15:32:38.653 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-18T15:32:38.676 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi104.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608866', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-18 14:47:06.899618', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOJddSLJuC3b7IoFH5QS0IDfmdWmvBFATT9soZcSavH79wPqV8wo5g+uJXY8/ezG7NBZcD2rDiLcirZcvp4ZZZ4='} 2024-03-18T15:32:38.687 INFO:teuthology.orchestra.console.smithi019:Power off 2024-03-18T15:32:38.687 DEBUG:teuthology.orchestra.console.smithi019:pexpect command: ipmitool -H smithi019.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-18T15:32:38.711 DEBUG:teuthology.orchestra.console.smithi019:power off output: Chassis Power Control: Down/Off 2024-03-18T15:32:38.712 DEBUG:teuthology.orchestra.console.smithi019:pexpect command: ipmitool -H smithi019.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T15:32:38.732 INFO:teuthology.orchestra.console.smithi033:Power off 2024-03-18T15:32:38.732 DEBUG:teuthology.orchestra.console.smithi033:pexpect command: ipmitool -H smithi033.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-18T15:32:38.743 DEBUG:teuthology.orchestra.console.smithi019:check power output: Chassis Power is on 2024-03-18T15:32:38.756 DEBUG:teuthology.orchestra.console.smithi033:power off output: Chassis Power Control: Down/Off 2024-03-18T15:32:38.757 DEBUG:teuthology.orchestra.console.smithi033:pexpect command: ipmitool -H smithi033.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T15:32:38.774 INFO:teuthology.orchestra.console.smithi104:Power off 2024-03-18T15:32:38.774 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-18T15:32:38.785 DEBUG:teuthology.orchestra.console.smithi033:check power output: Chassis Power is on 2024-03-18T15:32:38.797 DEBUG:teuthology.orchestra.console.smithi104:power off output: Chassis Power Control: Down/Off 2024-03-18T15:32:38.798 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T15:32:38.826 DEBUG:teuthology.orchestra.console.smithi104:check power output: Chassis Power is on 2024-03-18T15:32:42.744 DEBUG:teuthology.orchestra.console.smithi019:pexpect command: ipmitool -H smithi019.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T15:32:42.786 DEBUG:teuthology.orchestra.console.smithi033:pexpect command: ipmitool -H smithi033.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T15:32:42.827 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T15:32:42.856 DEBUG:teuthology.orchestra.console.smithi019:check power output: Chassis Power is on 2024-03-18T15:32:42.896 DEBUG:teuthology.orchestra.console.smithi033:check power output: Chassis Power is on 2024-03-18T15:32:42.940 DEBUG:teuthology.orchestra.console.smithi104:check power output: Chassis Power is on 2024-03-18T15:32:46.858 DEBUG:teuthology.orchestra.console.smithi019:pexpect command: ipmitool -H smithi019.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T15:32:46.897 DEBUG:teuthology.orchestra.console.smithi033:pexpect command: ipmitool -H smithi033.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T15:32:46.941 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T15:32:46.969 DEBUG:teuthology.orchestra.console.smithi019:check power output: Chassis Power is on 2024-03-18T15:32:47.008 DEBUG:teuthology.orchestra.console.smithi033:check power output: Chassis Power is on 2024-03-18T15:32:47.052 DEBUG:teuthology.orchestra.console.smithi104:check power output: Chassis Power is on 2024-03-18T15:32:50.973 DEBUG:teuthology.orchestra.console.smithi019:pexpect command: ipmitool -H smithi019.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T15:32:51.008 DEBUG:teuthology.orchestra.console.smithi033:pexpect command: ipmitool -H smithi033.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T15:32:51.053 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T15:32:51.084 DEBUG:teuthology.orchestra.console.smithi019:check power output: Chassis Power is off 2024-03-18T15:32:51.118 DEBUG:teuthology.orchestra.console.smithi033:check power output: Chassis Power is off 2024-03-18T15:32:51.164 DEBUG:teuthology.orchestra.console.smithi104:check power output: Chassis Power is off 2024-03-18T15:32:51.185 INFO:teuthology.orchestra.console.smithi019:Power off completed 2024-03-18T15:32:51.219 INFO:teuthology.orchestra.console.smithi033:Power off completed 2024-03-18T15:32:51.265 INFO:teuthology.orchestra.console.smithi104:Power off completed 2024-03-18T15:32:51.371 INFO:teuthology.run:Summary data: client.0-kernel-sha1: 3845b7ac715a3ff198582638d199a90e30e254da description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/fsync-tester}} duration: 2193.519199371338 failure_reason: '"2024-03-18T15:16:16.808370+0000 mon.a (mon.0) 1101 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY)" in cluster log' flavor: default owner: scheduled_pdonnell@teuthology success: false 2024-03-18T15:32:51.372 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-18T15:32:51.467 INFO:teuthology.run:FAIL